CI Workflows

CI never applies changes to a cluster. It validates, scans and publishes; Flux owns delivery from main. Seven workflow files live in .github/workflows/, and exactly six jobs — all of them in ci.yaml — can block a merge.

What blocks a merge

Branch protection on main requires these six contexts, and nothing else:

Two protection settings matter as much as the list:

• strict: false — since 2026-08-29 a conflict-free branch merges on green without being up to date with main. It was true until then, and every merge cost a rebase-and-rerun cycle with no conflict in sight. What that gives up is the semantic conflict, where two PRs each pass alone and break combined; these six jobs render and validate the whole tree, so the next PR surfaces it within minutes.
• enforce_admins: true — there is no gh pr merge --admin escape hatch, for anybody. A stuck check gets re-run, not bypassed.

Required approving reviews are set to 0: the gates are mechanical, not social, which is the whole reason they have to be trustworthy.

Neither setting lives in a file, so nothing in CI can catch them drifting — .doc-claims.yaml reads repo config, not the GitHub API. The strict value above was wrong on this page for two weeks after the flip for exactly that reason. Re-read it with gh api repos/Smana/cloud-native-ref/branches/main/protection rather than trusting the prose.

Renovate merges itself

Patch and minor dependency updates carry automerge: true in .github/renovate.json, so GitHub squash-merges them the moment the sixth required check reports success. No human step. Majors never automerge and still wait for someone.

Three properties above are what make that safe rather than reckless, and all three have to hold:

1. The six checks are the only gate, and they render and validate the entire repository — not a diff.
2. strict: false means a queue of Renovate PRs does not serialize behind a rebase-and-rerun each.
3. GitHub’s native auto-merge waits for branch protection, it does not bypass it. enforce_admins and the conversation-resolution requirement apply to an automatic merge exactly as they do to a manual one.

minimumReleaseAge: "1 day" holds each PR back a day first, so a release yanked hours after publication never reaches main.

One cost is accepted knowingly: Renovate calls 0.2.x → 0.3.0 a minor, and on a 0.x project that is routinely a breaking change — the semantic-router chart restructuring its values schema is this repo’s own scar. Automerge covers those too. The answer when a package proves untrustworthy is to bound it with allowedVersions, the way the two rules at the bottom of renovate.json already do, rather than to narrow the automerge rule.

The one carve-out

The Grafana plugin pins in observability/base/victoria-metrics-k8s-stack/vm-common-helm-values-configmap.yaml never automerge, at any update type. They are tracked from GitHub releases and installed from the Grafana.com catalog, and those publish independently — a tag can exist on GitHub days before, or without ever, reaching the catalog. Renovate has no catalog datasource, so it cannot see the gap: the bump is green on all six checks and CrashLoopBackOff on the cluster, because the plugin installer is a startup module and Grafana refuses to start rather than start without the plugin. That takes observability → tooling → apps down with it.

It has happened twice — #1959 reverted by #1980, #1981 reverted by #1982 — which is why the ConfigMap carries the check to run at the point someone is next asked to accept a bump:

curl -s https://grafana.com/api/plugins/<plugin-id>/versions | jq -r '.items[].version'

The exclusion is matched by file, so a third plugin added to that same list inherits it without anyone remembering to.

This is the shape to watch for when adding a dependency: not “is this package risky”, but does CI observe the same source the cluster installs from? Where it does not, green means nothing, and the package belongs in the carve-out.

ci.yaml — the six jobs

Runs on every pull request targeting main, with no path filter.

pre-commit 🛃

Runs the Terraform pre-commit hooks — terraform_fmt, terraform_validate, terraform_tflint — across every OpenTofu stack. jdx/mise-action installs tool versions from mise.toml, then the job calls pre-commit directly — the same command a contributor runs locally, so there is no separate container pipeline to keep in sync.

# Reproduce the job locally
mise install
pre-commit run --all-files

PCT_TFPATH=tofu tells the pre-commit-terraform hooks to call tofu instead of terraform. GITHUB_TOKEN is also set, so tflint --init authenticates when it fetches its ruleset rather than hitting GitHub’s anonymous rate limit.

The job first writes three placeholder certificate files into the OpenBao cluster stack’s gitignored .tls/ directory: tofu validate needs those files to exist, and the real certificates are never in Git.

security-scan 🔒

Three scanners. Trivy and Checkov write SARIF, which the sarif-upload job sends to the GitHub Security tab (see Token scopes):

kubernetes-validation ☸

First task ci:test, which runs every suite scripts/ci/tests/run.sh discovers and fails the job on any failing suite. The manifest gate runs even then, so a failing suite never hides its verdict.

Then the hard manifest gate — ./scripts/ci/validate-manifests.sh, as task ci:validate. It renders the repository the way Flux does (every Kustomize overlay with its postBuild vars substituted, every HelmRelease through helm template with its own values and postRenderers), then applies two gates to the rendered output: flux schema validate with skipMissingSchemas: false, so an unknown Kind fails the build rather than being skipped, and polaris audit. See Validation for why both properties are load-bearing.

render-diff 📝

Renders the PR head and the merge base and posts a sticky PR comment showing exactly which rendered resources the change adds, modifies or removes. It is a required check — the job has to succeed — but its content is informational: a diff showing a hundred changed resources passes exactly like a diff showing none. It exists so a reviewer sees the real effect of a values change rather than the YAML that produced it.

shellcheck 💻

shellcheck -x -S warning over every scripts/**/*.sh.

links 🔗

./scripts/ci/validate-links.sh resolves every relative Markdown link target in the repository: git ls-files '*.md', then each ](target) checked relative to the file holding it. .linkcheck-allow exists for known pre-existing breaks and is currently empty — the goal state. Never add an entry to route around a break your own change introduced.

The same job then runs ./scripts/ci/validate-doc-claims.sh, which checks the specific claims pinned in .doc-claims.yaml against the configuration they describe — it lives in this job rather than its own so the required-check list on main does not have to change.

Token scopes

Every job that checks out the PR runs PR code, so the workflow token is contents: read. Write scopes sit only in jobs that never check out or run PR code: each one downloads an artifact and hands it to a pinned action. Neither is a required check.

Path-filtered workflows

None of these is a required check. They run only when their paths change.

The documentation site

docs-check.yml builds the site on every PR that touches it. Two things make that build a real gate even though it is not a required check:

• refLinksErrorLevel: ERROR in website/hugo.yaml turns any unresolved internal relref into a build failure, so a renamed page cannot silently 404.
• ./scripts/ci/verify-doc-paths.sh asserts that every backticked repository path written in the site’s prose still exists. It walks git ls-files, so it only sees tracked files — run it after git add, or a brand-new page passes without ever being checked.

docs.yml runs the same build on push and publishes Hugo’s output, but its path filter is narrower than docs-check.yml’s — a change to scripts/ci/verify-doc-paths.sh alone triggers the PR check but not a deploy. Its concurrency group never cancels an in-flight deploy: a half-published site is worse than a slightly stale one.

Container images

Two workflows, split by trigger so that no write scope ever reaches a pull request. check-container-images.yml runs on pull requests only: workflow-wide contents: read, no registry login, each changed image built with push: false. It still executes the PR’s Dockerfiles — with a read-only token, the same trust class ci.yaml grants its pre-commit hooks over PR code. build-container-images.yml runs on push to main and manual dispatch — never on a pull request — and holds packages: write and security-events: write on the build-and-push job that logs in to GHCR, pushes each changed image, Trivy-scans it and uploads SARIF.

Two files rather than one with event-gated steps, because permissions: cannot be conditional: a workflow triggered by pull_request injects every declared scope into the PR run’s token, whatever the steps gate.

Images published from main are tagged main-<sha>, latest, and the image’s own version read from its Dockerfile ARG (the v* tags). The four in-repo consumers — the openbao-snapshot CronJob, headlamp-plugin-app, token-exchange-proxy and pev2 — pin that v* tag, which is mutable unless GHCR version immutability is enabled for the package; nothing in this repository deploys latest.

Disabled workflows

terramate-preview.yaml and terramate-drift-detection.yaml are fully commented out and live in .github/workflows-disabled/, a directory GitHub does not execute. They were moved there rather than left in place because a workflow file with no valid name/on/jobs key still gets queued and recorded as a permanently failed run on every push.

Pre-commit hooks

pip install pre-commit
pre-commit install
pre-commit run --all-files

check-added-large-files caps a file at 1000 KB, which is the constraint the diagram export budget in scripts/docs/export-diagrams.sh is set below.

Self-hosted GitHub runners

Runner scale sets run in-cluster (tooling/base/gha-runners/) and are off by default — commented out of tooling/aws-0/kustomization.yaml. When enabled they give private-endpoint access, lower latency, no egress charges for heavy builds, and secrets via External Secrets rather than long-lived tokens in a workflow.