Teardown

Same shape as the AWS teardown, with GCP’s own traps. Two of them cost a manual cleanup on the 2026-08-27 gcp-0 teardown and are now handled by the destroy workflow itself.

TM_CLOUD defaults to aws, and it gates the destroy jobs exactly as it gates deploy: without TM_CLOUD=gcp each one prints [skip] and exits 0. A teardown that appears to succeed instantly did nothing at all — check for [skip] lines before believing the cluster is gone.

GKE only

cd opentofu/gcp/gke/init
TM_CLOUD=gcp terramate script run destroy

Seven jobs, defined in opentofu/gcp/gke/init/workflows.tm.hcl:

confirm + init — scripts/ops/teardown/terramate-destroy-confirm.sh, then tofu init. Init runs before anything is destroyed on purpose: a lock file predating a new provider must fail here, not once resources have started disappearing.
stage2-seed-databases — promotes every backed-up CNPG database to a destroy-day seed and refreshes its <app>-pre-destroy alias; the same shared hook the AWS lane runs (scripts/ops/k8s/cnpg-pre-destroy-seed.sh). Warns and continues per database, never gates the teardown; CNPG_SKIP_PRE_DESTROY_SEED=true skips it. Unlike AWS, no gcp-0 claim restores from these yet — none sets objectStoreRecovery, so every rebuild runs initdb. This job builds the first gcp aliases (harbor’s, and rooms’ when the agent platform runs) that the claim-path switch noted in tooling/gcp-0/harbor/sqlinstance.yaml needs.
stage2-reclaim-volumes — reclaims CSI-provisioned PD disks while the cluster still exists (see below).
stage2-destroy-addons — destroys the gke/configure stack (Gateway API CRDs, Cilium, Flux). Never gates the cluster deletion.
stage1-destroy-cluster — destroys the cluster itself. This one is allowed to fail loudly: it is the billable resource.
stage2-sweep-orphaned-disks — the backstop for whatever step 3 could not reclaim in time, and it runs after the cluster is gone precisely so nothing is still attached (why).
stage2-reconcile-state — drops any stage-2 state left behind, now that the cluster holding those objects is provably gone.

Full teardown

cd opentofu
TM_CLOUD=gcp terramate script run --reverse destroy

Destroys every stack — GKE, OpenBao, Network — in reverse dependency order, with a single confirmation cached for 10 minutes.

Expect a second prompt. The cache counts from the first answer and is never refreshed, and stage2-seed-databases can run for several minutes per database before the cluster delete even starts. A later stack’s confirm then asks again, or exits 1 when there is no tty. Answer it, or set TM_DESTROY_CONFIRMED=true for the run.

The two leaks this workflow now prevents

Orphaned Persistent Disks

Deleting the cluster with PVCs still bound skips the reclaim entirely: the PD CSI controller dies with the cluster, and every PVC-backed disk is orphaned in the project with nothing left referencing it. Nothing reports this — tofu destroy says “Destroy complete” and the disks keep billing. Three survived the 2026-08-27 teardown (20/10/5 GB), the GCP replay of an EBS leak EKS already had a step for.

stage2-reclaim-volumes calls scripts/ops/k8s/reclaim-csi-volumes.sh, shared with the AWS teardown because every step of it is plain Kubernetes — what it does, step by step, is on the AWS teardown.

It is not sufficient on its own, and used to claim otherwise. The reclaim can only run while the cluster exists, and when it runs out of time it warns and exits 0 — it used to say a “cloud-side sweep” would catch the rest, and no such step existed. The leak therefore repeated: 3 disks on 2026-08-27, 8 disks / 43 GB on 2026-08-28, on a cluster with more stateful workloads (two CNPG databases, VictoriaMetrics, VictoriaLogs, runlore, Harbor).

stage2-sweep-orphaned-disks is that backstop, made real. It runs after the cluster is deleted — before then, a disk still attached to a draining node is not yet unattached and would be skipped — and removes only disks that are both unattached and carry the GKE CSI driver’s own marker in their description:

"storage.gke.io/created-by": "pd.csi.storage.gke.io"

so a hand-made disk is never touched. It logs each deletion with the PVC it came from, and never fails the teardown.

Run it by hand against any project at any time:

./scripts/ops/gcp/sweep-orphaned-disks.sh --project <project>          # dry run
./scripts/ops/gcp/sweep-orphaned-disks.sh --project <project> --apply

AWS has the same leak and no equivalent step. eks-prepare-destroy.sh shares the same in-cluster reclaim and the same failure mode, and 62 EBS volumes (~518 GB) were swept by hand after a 2026-07-21 rebuild. Nothing sweeps them automatically yet.

This step deletes PVC data unconditionally, regardless of the reclaim policy a PV was created with. There is no flag that skips it. Back up anything you need before running the destroy.

It never gates the teardown. The control-plane endpoint is private, so the usual reason to be running destroy at all is that the cluster or the tailnet is unreachable — the script says so and exits 0. Disks it misses stay in the project; list them with:

gcloud compute disks list --project <project> \
  --filter="-users:*" --format="table(name,sizeGb,zone)"

`containerNotEmpty` on the Cloud DNS zone

external-dns writes a record into the private zone for every HTTPRoute, and nothing removes them when the cluster goes — the controller that owned them went with it. Cloud DNS then refuses to delete a non-empty zone:

Error 400: The container is not empty., containerNotEmpty

which lands at the end of the network destroy, after the rest of the VPC is already gone, leaving the stack half torn down. On 2026-08-27 that meant deleting twelve records by hand.

The network stack’s destroy now runs scripts/ops/gcp/purge-dns-records.sh first, reading the zone name from state rather than re-deriving it. Apex NS and SOA are left alone: Cloud DNS will not delete them separately and removes them with the zone. Safe to re-run — it exits 0 when the zone is already gone or already empty.

What is not deleted

Secrets in Secret Manager outlive the cluster, by design — they are what a rebuild reads back. See ADR-0023. List and remove them by hand once you are certain:

gcloud secrets list --project <project> --format='value(name)'

The three hand-created bootstrap prerequisites — the state bucket, the Cloud KMS key ring, and the Tailscale OAuth client — are also left alone. Cloud KMS key rings cannot be deleted at all.

Everything Crossplane created

tofu destroy does not remove these, and it cannot: Crossplane is in the cluster, so it dies with it. Anything it provisioned in GCP is simply orphaned. On the 2026-08-28 gcp-0 teardown that was 3 Buckets, 7 BucketIAMMembers and 5 ProjectIAMMembers.

The buckets survive on purpose rather than by oversight — their managementPolicies are [Observe, Create, Update, LateInitialize] with no Delete, which is the constitution’s “no deletion permissions for stateful services” applied to object storage. Deleting the claim does not delete the bucket; nothing does, except a human.

gcloud storage buckets list --project <project> --format='value(name)'

The IAM members cost nothing and reference principals that no longer exist, so they are noise rather than a leak. The buckets bill.

Keep <project>-ogenki-cnpg-backups. It holds the CNPG seeds — the destroy-day dated prefixes and the <app>-pre-destroy aliases the stage2-seed-databases job refreshes on every teardown. No gcp-0 claim restores from them yet, but they are the only data the claim-path switch can restore once it lands; delete the bucket and that switch starts from empty databases. It is ~100 MB.

The other two are a judgement call. <project>-ogenki-llm-models held 18.1 GB of model weights after one preload — keeping it makes a later LLM run skip the HuggingFace download entirely; deleting it saves a few tens of cents a month. <project>-ogenki-harbor is empty unless images were pushed.

Verifying nothing is left

Teardown is not finished because a script said so. Check the project:

gcloud container clusters list --project <project>
gcloud compute instances list --project <project>
gcloud compute disks list --project <project>
gcloud compute addresses list --project <project>
gcloud dns managed-zones list --project <project>
gcloud compute networks list --project <project>   # only `default` should remain

# Expected to REMAIN, not to be empty: the Crossplane-created buckets and the
# bootstrap prerequisites. Confirm the list matches what you meant to keep --
# in particular that <project>-ogenki-cnpg-backups is still there.
gcloud storage buckets list --project <project> --format='value(name)'

The workforce pool needs `--show-deleted`, or it lies to you

# WRONG -- this reports an empty org even when a pool name is still reserved.
gcloud iam workforce-pools list --organization <org> --location global

# RIGHT
gcloud iam workforce-pools list --organization <org> --location global --show-deleted

Workforce pools soft-delete with a 30-day purge. tofu destroy removes the pool and empties the state, but the name stays reserved — and list hides deleted pools by default, so every check reports the org clean while the next apply fails on its very first resource:

Error 409: Pool locations/global/workforcePools/ogenki-zitadel already exists.

Measured 2026-09-02: a bootstrap died nine seconds in, on the first stack, for exactly this — after a sweep had declared the org empty.

Undeleting a pool does not restore its providers. They are separate objects with their own lifecycle, so a pool undeleted by hand comes back trusting nothing, and the failure surfaces much later as invalid_grant on every login rather than as an error during the rebuild.

You should not normally have to do any of this: the workforce-identity stack’s deploy runs scripts/ops/gcp/adopt-workforce-pool.sh first, which undeletes the pool and its provider and imports both into state, so a rebuild inside the 30-day window just works. It is a no-op on a genuinely fresh org. The manual commands are here for when you are diagnosing rather than deploying.

The pool name is deliberately stable across rebuilds rather than unique per build, because it is embedded verbatim in every Kubernetes RBAC group string — a per-build name would change every binding every build and leave a trail of tombstones. Surviving the tombstone is the cheaper trade.

Non-interactive

TM_DESTROY_CONFIRMED=true skips the interactive y/n prompt. It exists for CI; skip it on a first manual teardown so you get the confirmation.

Verify the cluster