Migrate the identity provider

ZITADEL is a primary-cloud singleton (ADR-0027): one directory, hosted on AWS by default, that relocates rather than duplicates. You need this page in one situation — you bootstrapped on one cloud and now want the other to host the identity provider. It is not part of a normal deploy.

Three things move, not one

This is the part worth reading. ZITADEL’s state lives in three places, and copying only the obvious one produces a cluster that starts, looks healthy, and authenticates nobody:

Move only the database and you get the compounding failure: the restored rows point every OIDC client at the old domain, and the one tool that fixes that — the setup scripts — needs the PAT you left behind. Recovery is minting a fresh PAT through the console, which requires logging in, which is what the stale configuration broke.

The procedure

Examples are AWS → GCP; reverse the cloud arguments to go the other way.

1. Freeze a seed on the source cloud — same step as Restore a database:

./scripts/ops/k8s/cnpg-promote-seed.sh --cluster xplane-zitadel --namespace security \
  --cloud aws --bucket eu-west-3-ogenki-cnpg-backups --apply

2. Copy the seed to the target cloud’s backup bucket, then verify it:

aws s3 sync s3://eu-west-3-ogenki-cnpg-backups/zitadel-20260902/ /tmp/zitadel-seed/
gcloud storage rsync --recursive /tmp/zitadel-seed/ \
  gs://<gcp-project>-ogenki-cnpg-backups/zitadel-20260902/

./scripts/ops/k8s/cnpg-promote-seed.sh --verify-seed zitadel-20260902 \
  --cloud gcp --bucket <gcp-project>-ogenki-cnpg-backups

3. Copy the admin PAT:

. scripts/lib/cloud-secret-store.sh

CLOUD=aws REGION=eu-west-3
pat="$(store_read zitadel/iam-admin-pat)"
[ -n "$pat" ] || { echo "source PAT is empty or absent — stop" >&2; exit 1; }

CLOUD=gcp GCP_PROJECT=<gcp-project>
store_write zitadel-iam-admin-pat <<< "$pat"

4. Copy the five consumer client secrets:

for key in \
  observability-victoria-metrics-k8s-stack-grafana-envvars \
  headlamp-envvars \
  security-flux-ui-oidc \
  headlamp-oauth2-proxy \
  harbor-oidc
do
  CLOUD=aws REGION=eu-west-3
  val="$(store_read "$key")"
  [ -n "$val" ] || { echo "MISSING on source: $key" >&2; continue; }

  CLOUD=gcp GCP_PROJECT=<gcp-project>
  store_write "$key" <<< "$val"
  echo "copied: $key"
done

5. Flip the two gates, in the same commit. Both live on the GCP side — AWS has no gate: whenever an aws-0 cluster exists, it hosts ZITADEL.

Do not set deploy_identity_provider in variables.tfvars. It is derived from primary_cloud and passed as a -var, which wins over the file — adding it there changes nothing and reports no error. ./scripts/ci/validate-idp-topology.sh fails in CI if the two gates disagree.

6. Deploy:

cd opentofu && terramate script run deploy
flux reconcile kustomization zitadel -n flux-system

7. Converge configuration to the new domain. The seed carried identity and the app rows, but every redirect URI and the Google IdP’s clientId still name the source domain. Run the steps from Set up single sign-on against the target cluster — expect [STALE]/[updated] on fields naming the old domain and [ok] elsewhere, since client IDs and secrets are unchanged by the move. That same run rewrites Harbor’s harbor-oidc entry; force it through with flux reconcile helmrelease harbor -n tooling --with-source rather than waiting.

8. Verify a login at any consumer on the target cluster, with the same Google account. The seed carried the IdP link and the role grant, so you should land as the same user with the same roles. The Google OAuth client needs the target cluster’s callback URI listed once — a single OAuth client accepts every cluster’s callback, so if both were added at first setup, nothing changes there.

Related

• ADR-0027 — why one directory relocates rather than duplicating
• ADR-0024 — what makes it deployable on either cloud
• Set up single sign-on — the scripts step 7 runs
• Restore a database — the seed mechanism