Workforce Identity Federation restores per-user Kubernetes RBAC on GKE

Status: Accepted Date: 2026-09-02 Deciders: Platform Team

Context

ADR-0026 accepted that gcp-0 would run Headlamp with unsafeUseServiceAccountToken: true: every admitted user reaches the API server as the single headlamp ServiceAccount, and the only authorisation control is allowed-group: admin on oauth2-proxy — an all-or-nothing gate in front of a shared, privileged identity. aws-0 has none of this problem: EKS is told to trust ZITADEL directly (identity_providers in opentofu/aws/eks/init/variables.tfvars), Headlamp forwards the user’s id_token, and security/base/rbac/admin.yaml binds Group: admin to cluster-admin per user.

ADR-0026 considered Workforce Identity Federation and dismissed it in one sentence:

Headlamp cannot forward a ZITADEL id_token into it. It solves human kubectl access, not this.

That dismissal rested on an unstated assumption: that Headlamp itself had to perform the STS token exchange. It does not. Headlamp’s -proxy-auth-token-header flag makes it forward whatever bearer token an upstream proxy hands it — Headlamp only needs to receive a token that works, not to mint one. That means the exchange can happen entirely in front of Headlamp, in a small dedicated proxy, and Headlamp never has to speak Google’s STS protocol at all. The first half of ADR-0026’s sentence is still true; the second half is the premise this record corrects.

Decision Drivers

• Restore per-user Kubernetes RBAC on gcp-0, matching the model aws-0 already has, rather than living with a single shared identity indefinitely.
• No new standing GCP IAM authority for a Headlamp user. Whatever authenticates the user to the cluster must not also become a general GCP credential.
• Reuse the existing ZITADEL groups pipeline. groups-from-roles.js already produces the claim aws-0’s RBAC consumes; the GKE side should consume the same claim rather than stand up a second authorisation source.
• Minimal new moving parts, for a cluster that is torn down and rebuilt routinely — a heavyweight standing component is a cost paid on every rebuild, not once.

Considered Options

Option 1: Workforce Identity Federation + a token-exchange shim

An IAM-only GCP feature (no in-cluster component, no fleet registration, no Connect gateway) exchanges a ZITADEL id_token for a short-lived Google access token scoped to a workforce pool. A small proxy (token-exchange-proxy, provider-neutral, RFC 8693) sits between oauth2-proxy and Headlamp: it reads the id_token oauth2-proxy forwards, exchanges it at Google STS, and injects the result as X-Gke-Token. Headlamp forwards that header verbatim (-proxy-auth-token-header=X-Gke-Token) as the bearer token to the API server, which resolves it to a principal://…/workforcePools/<pool>/subject/<sub> identity and a principalSet://…/group/<role> group — an ordinary ClusterRoleBinding on that group authorises exactly as Group: admin does on aws-0.

Pros:

• No in-cluster component beyond one small, provider-neutral shim; no fleet registration, no Connect gateway, no GCP IAM roles granted to the principal.
• Reuses the ZITADEL groups claim already driving aws-0’s RBAC — same role, same cluster-admin, different spelling.
• The federated token is inert in GCP itself (measured: 403 on Resource Manager and Cloud Storage) — compromising it does not hand out a cloud credential, only cluster access scoped by Kubernetes RBAC.

Cons:

• One new component to own: a small Go proxy, in this repo’s threat model, that briefly holds user tokens in memory.
• A new organisation-level GCP resource (the workforce pool) whose name is effectively permanent — renaming is delete-and-recreate with a 30-day soft-delete window, and every RBAC group string embeds the name.

Option 2: Pinniped Concierge (impersonation-proxy mode)

ADR-0026’s originally-named upgrade path. Apache-2.0, no licence cost, GKE explicitly supported, and the impersonation-proxy strategy exists precisely because managed control planes take no --oidc-issuer-url flag. It would preserve per-user RBAC and keep gcp-0 structurally identical to aws-0.

Pros:

• Purpose-built for exactly this gap; would make gcp-0 and aws-0 use the same authentication model, not two different ones.
• No GCP-specific IAM resource to own.

Cons:

• Needs a Concierge (and a Supervisor to federate ZITADEL) as new standing in-cluster infrastructure, plus a LoadBalancer/ClusterIP with its own certificates — for a cluster that is deleted after every validation run.
• Per Tremolo’s own comparison, Pinniped has no story for a non-OIDC-native client like Headlamp: it would still need an impersonating proxy in front of it, which is Option 3 below layered on top rather than instead of it.

Option 3: An impersonating proxy in front of Headlamp (kube-oidc-proxy and forks)

oauth2-proxy authenticates the human as today, then hands off to an impersonating reverse proxy (kube-oidc-proxy, or a maintained fork — TremoloSecurity, banyansecurity, sspreitzer, or OpenUnison’s bundled one) that sets Impersonate-User / Impersonate-Group against the real API server. Kubernetes RBAC would see the real user, with no GCP-side component at all.

Pros:

• Keeps Kubernetes RBAC and needs no GCP org changes — the smallest change on the GCP side of any option considered.

Cons:

• Cannot work with in-cluster Headlamp at all. An impersonating proxy must sit behind Headlamp — Headlamp itself needs to forward Impersonate-* headers to the API server it talks to — but Headlamp cannot be pointed at a different in-cluster API endpoint (kubernetes-sigs/headlamp#1460, open since October 2023), and its OIDC mode does not emit impersonation headers either (#4198). This is a structural blocker, not a configuration gap.

Option 4: Google Workspace identities + Google Groups for RBAC

Authenticate operators as real Google Workspace humans and bind Google Groups in Kubernetes RBAC — natively supported by GKE, no shim, no workforce pool.

Pros:

• Zero custom code; GKE’s own documented path for Google-identity RBAC.

Cons:

• GKE demands a cloud-platform-scoped token to authenticate at all (measured: userinfo.email scope alone is rejected with 401). For a real Workspace human that scope is a full, general-purpose GCP credential — not the inert token Option 1 produces — verified: the same scope returns HTTP 200 against Resource Manager for a real user.
• Moves group membership and identity out of ZITADEL for this one cloud, splitting the group source of truth the two clouds already share.

Decision Outcome

Chosen option: “Option 1 — Workforce Identity Federation + a token-exchange shim”

Rationale: Option 3 is not actually available — it cannot work with Headlamp’s in-cluster deployment mode, full stop. Option 2 does not remove the need for something like Option 3’s proxy in front of Headlamp, so it does not avoid that blocker either; it only adds Pinniped’s own standing infrastructure on top. Option 4 works but hands every logged-in operator a general cloud-platform GCP credential to reach a Kubernetes dashboard, which is a strictly worse blast radius than Option 1’s inert federated token. Option 1 is the only option that is both available (measured working end-to-end against a live cluster) and least-privileged (the credential it produces can do nothing in GCP itself).

The following was measured against a live gcp-0 on 2026-09-02, then torn down; the design rests on this evidence rather than on the product documentation alone:

The last row is the one that makes Option 1 preferable to Option 4 on its own: the token is requested with cloud-platform scope, which reads as broad, but a workforce principal starts with zero IAM role bindings, and scope is not authority. The token authenticates to the cluster and can do nothing in Google Cloud. A Workspace human’s equivalent token (Option 4) carries whatever that person can do in the organisation.

Architecture:

browser
  │
  ▼
oauth2-proxy ──── OIDC authorization-code against ZITADEL
  │               emits X-Forwarded-Groups; forwards the raw id_token upstream
  ▼
token-exchange shim  (new, ~40 lines, provider-neutral, no GCP defaults compiled in)
  │  POST https://sts.googleapis.com/v1/token
  │    subject_token = the ZITADEL id_token
  │    audience      = //iam.googleapis.com/…/workforcePools/<pool>/providers/zitadel
  │  ← short-lived Google federated access token (1h)
  │  sets X-Gke-Token, strips Authorization
  ▼
Headlamp ──── -proxy-auth-token-header=X-Gke-Token
  │           forwards it as Authorization: Bearer, never validates it itself
  ▼
GKE API server
  authenticates → principal://…/subject/<zitadel-sub>
  groups        → principalSet://…/group/admin
  authorises    → ClusterRoleBinding in Git (security/gcp-0/rbac/admin.yaml)

Nothing Google-hosted sits in the request path to the cluster; the only outbound call is the shim’s own STS exchange, and it is unauthenticated — it presents the user’s token and needs no service account of its own.

Consequences

Positive

• Per-user Kubernetes RBAC is restored on gcp-0: the API server can tell users apart again, and security/gcp-0/rbac/admin.yaml authorises the same way security/base/rbac/admin.yaml does on aws-0 — same ZITADEL role, same cluster-admin, a principalSet:// group instead of a bare group name.
• allowed-group: admin on oauth2-proxy becomes defence in depth rather than the whole authorisation model, and the headlamp ServiceAccount’s own ClusterRoleBinding was reduced off cluster-admin in the same change — the blast radius of a Headlamp compromise is now one user’s RBAC for up to an hour, not everything the shared SA could do indefinitely.
• No GCP IAM role is granted anywhere in this design; the federated token is provably inert outside the cluster.

Negative

• A workforce pool is an organisation-level GCP resource with a soft-deleted, effectively-permanent name — every RBAC group string embeds it, so renaming is a coordinated break-and-fix across security/gcp-0/rbac/*, not a config edit.

  • Mitigation: named once, documented as permanent in opentofu/gcp/workforce-identity/variables.tfvars, and pinned by the workforce-pool-id doc claim (see Implementation Notes) so the docs cannot drift from it silently.

• The audience the workforce pool provider trusts is a single OIDC client — headlamp-proxy, the app whose token the shim presents.

  It was originally the whole ZITADEL project, on the reasoning that any token the project issued would then be exchangeable and the pool could be created before a single OIDC app existed. That never worked. Requesting the project audience scope makes aud multi-valued; OIDC Core 3.1.3.7 then requires azp to be present and to equal the trusted client id; Google STS enforces exactly that; and azp is always the client the token was issued to — never the project. Every exchange returned invalid_grant from the day it shipped until 2026-09-12, while oauth2-proxy, the shim, Headlamp and every Flux resource reported healthy.

  Narrowing to one client is therefore not a regression from the original design but the only form of it that functions — and it is the tighter grant besides. The client id is not knowable at apply time, so reconcile_workforce_audience in scripts/provision/zitadel-oidc-clients.sh sets it once the app exists, and the tofu resource keeps lifecycle.ignore_changes on the field so the next apply cannot revert it.

• A new component (the shim) briefly holds user tokens in memory, which is why it runs default-deny CiliumNetworkPolicy, restricted PSS, and a never-log-a-token rule.

Neutral

• aws-0 is untouched. It keeps trusting ZITADEL directly at the EKS control plane; the two clouds now reach the same RBAC outcome by genuinely different mechanisms; ADR-0007’s cloud-abstraction stance treats that as acceptable divergence rather than something to unify.

Implementation Notes

• opentofu/gcp/workforce-identity/ — new stack, organisation-scoped (a second GCP cluster would share the pool), creates google_iam_workforce_pool.zitadel and its OIDC provider. The pool id is workforce_pool_id = "ogenki-zitadel" — permanent by construction (see Consequences), so treat a change to that value as a breaking rename, not a config edit.
• workforce_pool_id is threaded into gke/configure’s flux_cluster_vars (opentofu/gcp/gke/configure/kubernetes.tf) so security/gcp-0/rbac/admin.yaml can substitute it into the principalSet:// group name; scripts/ci/flux-schema/check-substitution.py fails the build if that wiring is missing, rather than letting Flux silently substitute an empty string.
• tooling/gcp-0/headlamp/token-exchange.yaml — the shim Deployment + Service.
• tooling/gcp-0/headlamp/oauth2-proxy.yaml — upstream points at the shim; pass-authorization-header: true forwards the id_token to it.
• tooling/gcp-0/headlamp/headlamp-proxy-auth.yaml — unsafeUseServiceAccountToken: false; -proxy-auth-token-header=X-Gke-Token added; the chart’s default clusterRoleName: cluster-admin on the headlamp ServiceAccount’s own ClusterRoleBinding reduced to view.

References

• ADR-0026 — the record this supersedes
• ADR-0024 — where ZITADEL runs; this record is about what GKE’s API server will trust, which ADR-0024 does not cover
• ADR-0027 — the per-cloud/singleton classification this design follows: the workforce pool is a gcp-0-owned resource, not a cross-cloud one
• ADR-0002 — the AWS identity model this diverges from
• docs/superpowers/specs/2026-09-02-gke-rbac-headlamp-design.md — the full design, including the shim’s failure-handling rules and the false-trail writeup on an earlier, wrong RBAC-scoping conclusion