OpenBao cross-cloud failover

The active OpenBao runs on AWS and serves both clusters. Its durable form is the lineage (ADR-0033): the multi-region seal key alias/openbao-seal, five bootstrap secrets, and the snapshot bucket eu-west-3-ogenki-openbao-snapshot, to be mirrored into ogenki-435905-ogenki-openbao-snapshot by a Storage Transfer job at 05:00 UTC, one hour after the snapshot CronJob’s 04:00 UTC run.

diff <(aws s3 ls s3://eu-west-3-ogenki-openbao-snapshot/ | awk '{print $3, $4}') \
     <(gcloud storage ls -l gs://ogenki-435905-ogenki-openbao-snapshot/ \
         | awk '/\.snap$/ {n=split($3,p,"/"); print $1, p[n]}')

key=$(aws s3api list-objects-v2 --bucket eu-west-3-ogenki-openbao-snapshot \
  --query 'sort_by(Contents, &LastModified)[-1].Key' --output text)
aws s3 cp "s3://eu-west-3-ogenki-openbao-snapshot/${key}" /tmp/mirror.snap
gcloud storage cp /tmp/mirror.snap "gs://ogenki-435905-ogenki-openbao-snapshot/${key}"

What this survives, and what it does not

Consumers tolerate the gap: External Secrets keeps the last synced Secrets and cert-manager renews 15 days before expiry. Only new secrets and certificates wait, so this procedure is manual and measured in tens of minutes.

Preconditions

All of this is peacetime work — read it now, not during an incident. Three items need a reachable eu-west-3, so they cannot be done once AWS is down, which is exactly the failure this procedure is for.

Item 3 — gcp-0’s ClusterIssuer connects by openbao.security.svc.cluster.local in both postures, so a certificate carrying only bao.priv.gcp.ogenki.io fails with x509: certificate is valid for bao.priv.gcp.ogenki.io, not openbao.security.svc.cluster.local:

gcloud secrets versions access latest --secret openbao-priv-gcp-server-cert \
  --project ogenki-435905 | jq -r .cert | openssl x509 -noout -ext subjectAltName

Item 4 — the restored store is the AWS one, so the GCP entries must hold the AWS lineage’s credentials. There is no cross-cloud copy tool; these two commands are it, and they must be re-run whenever the AWS lineage’s root token or recovery keys change:

aws secretsmanager get-secret-value --region eu-west-3 \
  --secret-id openbao/cloud-native-ref/tokens/root --query SecretString --output text \
  | gcloud secrets versions add openbao-priv-gcp-root-token --project ogenki-435905 --data-file=-
aws secretsmanager get-secret-value --region eu-west-3 \
  --secret-id openbao/cloud-native-ref/tokens/recovery --query SecretString --output text \
  | gcloud secrets versions add openbao-priv-gcp-recovery-keys --project ogenki-435905 --data-file=-

Failover, AWS → GCP

1. Measure the loss. The newest mirrored object is the data you will have:

   gcloud storage ls -l gs://ogenki-435905-ogenki-openbao-snapshot/ | sort -k2 | tail -1

   Names are <UTC timestamp>-<seal>.snap. The trailing segment is the seal that encrypted the object, and only a node running that seal can unwrap it — which is why the standby uses the AWS key. Expect -awskms on every mirrored object. One with no seal segment predates the scheme and will not be selected; container-images/openbao-snapshot/README.md has the retag.

2. Deploy the standby with the AWS seal. In opentofu/gcp/openbao/cluster/variables.tfvars set:

   seal_provider       = "awskms"
   aws_seal_kms_key_id = "<opentofu/aws/openbao/lineage output seal_key_id>"
   aws_seal_region     = "eu-west-1"
   aws_seal_role_arn   = "<opentofu/shared/aws-gcp-federation output openbao_standby_seal_role_arn>"

   then run the two GCP OpenBao stacks by directory, in this order:

   TM_CLOUD=gcp terramate -C opentofu/gcp/openbao/cluster    script run deploy
   TM_CLOUD=gcp terramate -C opentofu/gcp/openbao/management script run deploy
   By directory, not terramate script run deploy from opentofu/. The root deploy reaches the shared/* stacks first, and their state lives in eu-west-3 — the region the outage is in. tofu init fails there and the run stops before it ever reaches OpenBao. TM_CLOUD=gcp is still required, or the provisioner defaults to aws and prints [skip].

   The management stack’s rehydrate step restores from the GCS bucket. Its output is what tells you it worked, in this order: Restoring snapshot <object>, then The restored snapshot was taken N day(s) ago. — the lineage/check_timestamp marker, read back from inside the restored store, which is why it is an alarm rather than a gate: it can only be read after the restore has been applied. Then PKI issuer present: subject=... and Rehydrate complete;.

3. Verify.

   export VAULT_ADDR=https://bao.priv.gcp.ogenki.io:8200 VAULT_CACERT=opentofu/gcp/openbao/management/.tls/ca.pem
   bao status                     # Initialized true, Sealed false, no operator input
   curl -s --cacert "$VAULT_CACERT" "$VAULT_ADDR/v1/pki_private_issuer/ca/pem" | openssl x509 -noout -subject

   Expected: a subject whose CN is Ogenki AWS Intermediate CA (OpenSSL 3 prints subject=CN = Ogenki AWS Intermediate CA, O = Ogenki, C = FR; older builds print it without the spaces). That one command is the only thing separating “the AWS lineage restored here” from “the GCP node’s own pre-existing PKI answered” — both leave a healthy bao status. If the CN is GCP’s own intermediate instead of the AWS one, the restore did not take: the node is serving its own mount, and every secret and certificate you are about to depend on is the wrong one. Stop and re-read the rehydrate output rather than continuing to step 4.

   Then restart the instance and repeat bao status: it must come back unsealed on its own, with no operator input. The node is managed by google_compute_instance_group_manager.openbao with base_instance_name = "openbao-dev", so its real name carries a MIG-generated random suffix and is not knowable in advance — look it up, and supply the zone:

   zone=europe-west4-a
   name=$(gcloud compute instances list --project ogenki-435905 \
     --filter="name~'^openbao-dev-' AND zone:${zone}" --format='value(name)')
   echo "$name"
   gcloud compute instances stop  "$name" --zone "$zone" --project ogenki-435905
   gcloud compute instances start "$name" --zone "$zone" --project ogenki-435905
   bao status

   The MIG will not race you while it is stopped: compute.tf sets no auto_healing_policies, deliberately — the comment there records the 2026-08-25 incident that decided it. Auto-healing would RECREATE the instance and wipe the auto_delete = true data disk holding the Raft store.

4. Repoint the surviving cluster. For gcp-0 itself nothing changes: its openbao Service is the local form — security/gcp-0/openbao/kustomization.yaml lists ../../base/openbao-endpoint/local — and its ClusterIssuer already connects by the neutral in-cluster name, provided the server certificate from the preconditions above carries that SAN.

   For any other cluster still running, switch security/<cluster>/openbao/kustomization.yaml to ../../base/openbao-endpoint/remote and set that cluster’s openbao_target_ip. Read the address from the GCP cluster stack rather than guessing it — google_compute_address.openbao allocates it dynamically:

   (cd opentofu/gcp/openbao/cluster && tofu output -raw internal_ip)

   openbao_target_ip is a per-cluster variable set in that cluster’s own configure stack, not a repo-wide value. Each cluster has its own source of truth, and both exist:

   Cluster	Declared in	Emitted into
   aws-0	opentofu/aws/eks/configure/variables.tf	eks-aws-0-vars, in opentofu/aws/eks/configure/kubernetes.tf
   gcp-0	opentofu/gcp/gke/configure/variables.tf	gke-gcp-0-vars, in opentofu/gcp/gke/configure/kubernetes.tf

   So the step is executable in both directions — AWS-consuming-GCP included, which is the primary one for this design. Two things are worth knowing before editing anything mid-incident:

   • The key existing is not the same as it being set. Both variables default to "" in the normal posture, and Flux substitutes an undefined variable to the empty string too — so a value left at that default and a missing key render identically: tailscale.com/tailnet-ip: "" in security/base/openbao-endpoint/remote/service.yaml, schema-valid, silently wrong, a Service annotated with nothing. Put the address in that stack’s variables.tfvars, apply, then check the ConfigMap actually carries it (kubectl -n flux-system get cm eks-aws-0-vars -o yaml).
   • The missing-key half is gated. scripts/ci/flux-schema/check-substitution.py fails the build when a Kustomization applies a ${var} its own cluster’s ConfigMap does not define, so ./scripts/ci/validate-manifests.sh catches that regression in CI rather than at 3am. It cannot catch an empty value; only the check above can.

   Commit; Flux reconciles; External Secrets and cert-manager pick up the new endpoint on their next interval.

Failback, GCP → AWS

The mirror only runs one way. Copying GCS back over S3 is deliberate and manual, one object at a time: a standby’s snapshot holds the AWS lineage’s data plus whatever was written during the incident, and it must not silently become the newest object in the AWS history.

1. Take a final snapshot on the GCP node, then copy exactly one object back. The failover’s restore replaced the token store with the AWS lineage’s, so the valid root token here is the AWS one — the value pre-staged into the GCP entry by the preconditions:

   export VAULT_ADDR=https://bao.priv.gcp.ogenki.io:8200
   export VAULT_CACERT=opentofu/gcp/openbao/management/.tls/ca.pem
   VAULT_TOKEN=$(gcloud secrets versions access latest \
     --secret openbao-priv-gcp-root-token --project ogenki-435905 | jq -r .token)
   
   VAULT_TOKEN="$VAULT_TOKEN" VAULT_CACERT="$VAULT_CACERT" CLOUD=gcp \
     ./scripts/provision/openbao-snapshot.sh save \
     -a "$VAULT_ADDR" -b ogenki-435905-ogenki-openbao-snapshot -s /tmp/bao.snap

   VAULT_CACERT is not optional — the script’s own usage says “Set it; do not skip verify”, and this chain is in no system trust store by default.

   The object just written is now the newest. Select it the way the tooling does — latest_snapshot() in scripts/provision/openbao-config.sh sorts GCS objects by name — and copy that one object, by name, into S3:

   newest=$(gcloud storage ls gs://ogenki-435905-ogenki-openbao-snapshot/ \
     | sed 's#.*/##' | grep '\.snap$' | sort | tail -n1)
   echo "$newest"     # confirm this is the snapshot you just took
   gcloud storage cp "gs://ogenki-435905-ogenki-openbao-snapshot/${newest}" /tmp/back.snap
   aws s3 cp /tmp/back.snap "s3://eu-west-3-ogenki-openbao-snapshot/${newest}"

2. Redeploy AWS (terramate script run deploy from opentofu/); its rehydrate restores that object. The root deploy is the right command here, unlike step 2 of the failover, because failback only begins once eu-west-3 is reachable again — so the shared stacks’ S3 backend resolves.

3. Retire the standby — do not flip its seal. The GCP node is holding AWS-sealed data, and restoring seal_provider = "gcpckms" on that stack is the single change that dead-ends it. Flipping the variable edits the instance template; the MIG’s PROACTIVE / REPLACE update policy (opentofu/gcp/openbao/cluster/compute.tf) replaces the running instance; the replacement boots with an empty Raft store; rehydrate goes looking for the newest object in gs://ogenki-435905-ogenki-openbao-snapshot/ — and every object in that bucket is AWS-sealed by now: the mirrored ones by construction, and the standby’s own because it ran under seal_provider = "awskms". A gcpckms node cannot unwrap any of them.

   How that surfaces, and it is no longer a stranding. rehydrate reads this node’s seal from /v1/sys/seal-status, compares it with the newest object’s name segment, and refuses before bao operator init — the irreversible step — naming both seals. The deploy stops with a legible error and an untouched store, rather than the node coming back sealed with nothing to diagnose it by. It is still the wrong move: you are left with a replaced instance, an empty Raft store, and step 4’s decision to make anyway.

   Destroy the standby instead, by directory, and skip its pre-destroy snapshot:

   TM_OPENBAO_SKIP_SNAPSHOT=true TM_CLOUD=gcp \
     terramate -C opentofu/gcp/openbao/cluster script run destroy

   TM_OPENBAO_SKIP_SNAPSHOT=true is the point of this step, not a shortcut. The default pre-destroy snapshot would write one more AWS-sealed object, as the newest in the GCP bucket — so the next GCP-only deploy would rehydrate straight back into the dead end above. Nothing is lost by skipping it: step 1 already carried this node’s data into the AWS lineage, which is where it now belongs.

   By directory, too, and for a second reason: TM_CLOUD=gcp terramate script run --reverse destroy from opentofu/ sweeps the whole GCP lane — gcp/gke/configure, gcp/gke/init and gcp/network are all in terramate list --run-order — tearing down the cluster you just failed over to and are still running on.

4. Before gcp-0 runs GCP-only again, decide which writes to discard. Every object in the GCP snapshot bucket is AWS-sealed by now — the mirrored ones by construction, the standby’s own because it ran under seal_provider = "awskms". What has changed is that the bucket is no longer ambiguous: objects are named <UTC timestamp>-<seal>.snap, so a fresh gcpckms node reads its own seal, sees the mismatch and refuses before bao operator init:

   ERROR: SEAL MISMATCH -- refusing to initialise or restore. Nothing has changed yet.
   ERROR:   this node's seal : gcpckms
   ERROR:   newest object    : 2026-09-02T041500Z-awskms.snap
   ERROR:                      sealed 'awskms'

   Nothing needs moving aside and nothing needs re-stamping. That refusal mutates neither the bucket nor the node, so this step is no longer bucket surgery performed mid-incident. What is left is the one judgement the tooling will not make for you.

   Read the bucket. The seal segment is in the name, so the listing is the whole answer:

   gcloud storage ls gs://ogenki-435905-ogenki-openbao-snapshot/ | sed 's#.*/##' | sort

   The newest object a gcpckms node can unwrap is the last -gcpckms.snap, from before the failover, and restoring it discards every write after it. Those discarded writes belong to the AWS lineage, not this one — the standby was serving the restored AWS store — and step 1 above already carried them into eu-west-3-ogenki-openbao-snapshot, which is where they belong. That is what makes the skip correct here rather than a loss: gcp-0 is going back to being authoritative for itself, and its own last GCP-sealed snapshot is exactly the store it should hold. Confirm that object is the one you expect, then accept the skip on the stack that runs rehydrate:

   OPENBAO_SNAPSHOT_SKIP_FOREIGN_SEAL=true TM_CLOUD=gcp \
     terramate -C opentofu/gcp/openbao/management script run deploy

   It logs which object it skipped past and which it restored. Set it for that one invocation only — it is not a default precisely because skipping a newer snapshot discards data.

   If the listing shows no -gcpckms.snap object at all — a young GCP lineage, or one whose snapshots passed the bucket’s 120-day expiry (lifecycle_rule in opentofu/gcp/openbao/lineage/main.tf) — the flag cannot help, and rehydrate refuses rather than falling through to a plain init, which would overwrite this lineage’s stored root token and recovery keys. That case is a deliberate fresh GCP lineage, not a restore.

5. Revert the openbao_target_ip / overlay changes from step 4 of the failover.

Starting a GCP-only lineage

GCP’s snapshot bucket also holds the AWS mirror, and every mirrored object is AWS-sealed. A gcpckms node that finds no object under its own seal therefore has no way in. rehydrate refuses the foreign seal, and even with OPENBAO_SNAPSHOT_SKIP_FOREIGN_SEAL=true it refuses to initialise, because that would overwrite a lineage’s stored keys on a guess. The first boot of a GCP-only lineage says so out loud instead:

OPENBAO_NEW_LINEAGE=true TM_CLOUD=gcp \
  terramate -C opentofu/gcp/openbao/management script run deploy

The command above presumes the GCP cluster stack is already up, typically from a full deploy that stopped at the seal refusal. The switch also sits behind the recovery-keys pre-flight: on a project where openbao-priv-gcp-recovery-keys has no readable version, the switch is unreachable and rehydrate refuses before it ever reads OPENBAO_NEW_LINEAGE.

Once the deploy that printed the proceed arm’s warning finishes, force-replace the break-glass user:

tofu -chdir=opentofu/gcp/openbao/management apply -parallelism=1 \
  -var-file=variables.tfvars \
  -replace=module.store_of_record.vault_generic_endpoint.admin_user

Keep both flags. Without -var-file, secret_owning_apps falls back to [] and the plan destroys every app persona; -parallelism=1 is what the stack’s own deploy uses against the OpenBao 2.6 write deadlock.

vault_generic_endpoint.admin_user sets disable_read = true (the password is never readable back, so a read would always look like drift). That also means a plain apply against the new, empty OpenBao sees no drift on this resource and never recreates auth/userpass/users/admin — the tofu state still has it recorded from the lineage that just got discarded. Without the -replace, openbao-priv-gcp-admin-credentials keeps publishing a password for a break-glass user that no longer exists, and there is no way back in once OIDC is also unconfigured.

The switch has five limits:

• It is honoured only when no top-level object carries the node’s own seal, and every top-level snapshot’s name carries a -<seal> segment. A snapshot whose name has no seal segment (a legacy <timestamp>.snap or a hand-named one) is refused, because its seal is unknown.
• Snapshots moved aside under a prefix are not examined. Never move one under a prefix in this bucket to get past the refusal, because that only hides it from the check. Retag it to <timestamp>-<seal>.snap, or move it to another bucket.
• It is never honoured together with OPENBAO_SNAPSHOT_KEY.
• It replaces the stored root token and recovery keys.
• These are the same two entries the awskms standby reads its pre-copied AWS keys from. After a GCP-only run, re-copy the AWS lineage’s keys before relying on the standby. The previous versions remain in Secret Manager (gcloud secrets versions list openbao-priv-gcp-recovery-keys --project ogenki-435905).

Every later boot restores the newest -gcpckms object — until the daily S3→GCS mirror makes an -awskms object the newest, which it does whenever the AWS lineage’s last snapshot postdates GCP’s, and the deploy then refuses on the seal. Name the object to restore with OPENBAO_SNAPSHOT_KEY=<timestamp>-gcpckms.snap; the name is checked against the listing before anything changes, and the gcp-0 rebuild of 2026-10-17 restored this way. OPENBAO_SNAPSHOT_SKIP_FOREIGN_SEAL=true, which the refusal message suggests, takes the newest -gcpckms object without naming it.

Drill record

Every executed failover is recorded with the snapshot object, the measured RPO, and the time from step 2 to step 3 — in 2026-09-02-openbao-store-of-record-verification.md under docs/superpowers/specs/ (a repository path — verification notes are not published). No failover has been executed yet, so that note does not exist: it is written by the first one, and /verify-spec creates it post-merge.