Verify the cluster

Everything below is read-only. Run it top to bottom after a deploy, or dip into whichever section you are suspicious of.

1. Authenticate

gcloud auth login                       # once per machine
gcloud config set project ogenki-435905

gcloud container clusters get-credentials gcp-0 \
  --location europe-west4-a --project ogenki-435905

kubectl get nodes

The control plane is private. kubectl only works from the tailnet, so tailscale status must show you connected before any of this. A hanging kubectl with no error is almost always a dropped tailnet, not a broken cluster.

To keep this out of your usual kubeconfig, point KUBECONFIG at a scratch file before get-credentials:

export KUBECONFIG=/tmp/gcp0.kubeconfig

2. GitOps layer

flux get kustomizations           # everything True, bar the suspended ones
flux get helmreleases -A
flux get sources all -A

Two Kustomizations report no status at all, and that is correct — they are spec.suspend: true:

Suspended	Why
`llm-platform`	opt-in; see `clusters/gcp-0-llm-platform/README.md`
`security-openbao-snapshot`	opt-in
`zitadel`	only when this cluster consumes an IdP rather than hosting one (ADR-0024)

Anything else not True is a real failure. Start with the message:

kubectl get kustomization -A \
  -o custom-columns=NAME:.metadata.name,READY:.status.conditions[0].status,MSG:.status.conditions[0].message

A dependency '…' is not ready message is a cascade, not a cause — walk to the root of the chain and fix that one.

3. Secrets

The single most common cause of a stuck cluster, and the least self-evident: a missing or unreadable secret surfaces as a HelmRelease timing out ten minutes later, naming only the HelmRelease.

./scripts/secret-store.sh check --cloud gcp --project ogenki-435905

That lists every key the cluster’s ExternalSecrets ask for and whether it exists. For the in-cluster side:

kubectl get externalsecrets -A

SecretSyncedError has two usual causes, and they look identical from the cluster:

the secret does not exist — secret-store.sh check says so;
External Secrets cannot read it — the grant is per-secret, never project-wide, so a newly created secret is unreadable until it is granted:

gcloud secrets get-iam-policy <name> --project ogenki-435905

4. Certificates and networking

kubectl get certificates -A          # all True
kubectl get gateways -A              # PROGRAMMED=True, each with an address
kubectl get httproute -A

Private hostnames resolve only over the tailnet; public ones resolve from anywhere through Route 53 (ADR-0019).

dig +short auth.gcp.cloud.ogenki.io          # public, from anywhere
dig +short grafana.priv.gcp.ogenki.io        # private, needs the tailnet

5. Browse it

Everything private is *.priv.gcp.ogenki.io and needs the tailnet. Start at the homepage, which links the rest.

Private services present the OpenBao private CA, so a browser that does not trust it shows a certificate warning, and curl fails with “unable to establish a secure connection” rather than an HTTP error. That is the PKI working, not a broken service — the certificates are real, they are simply not from a public CA.

Trust the chain, or use curl -k when you only care whether the service answers:

curl -sSk -o /dev/null -w '%{http_code}\n' https://grafana.priv.gcp.ogenki.io/login

ZITADEL is the exception: it is public, on Route 53, with a Let’s Encrypt certificate, so it verifies normally and needs no -k.

Service	URL
Homepage	`https://home.priv.gcp.ogenki.io`
Grafana	`https://grafana.priv.gcp.ogenki.io`
Harbor	`https://harbor.priv.gcp.ogenki.io`
Headlamp	`https://headlamp.priv.gcp.ogenki.io`
Flux UI	`https://flux-ui-gcp-0.priv.gcp.ogenki.io`
VictoriaMetrics	`https://vm.priv.gcp.ogenki.io/vmui`
VictoriaLogs	`https://vl.priv.gcp.ogenki.io/select/vmui`
VictoriaTraces	`https://vt.priv.gcp.ogenki.io/select/vmui`
Alertmanager	`https://vmalertmanager-gcp-0.priv.gcp.ogenki.io`
pev2	`https://pev2.priv.gcp.ogenki.io`
podinfo	`https://podinfo.priv.gcp.ogenki.io`
App Wizard	`https://app-wizard.priv.gcp.ogenki.io`
ZITADEL	`https://auth.gcp.cloud.ogenki.io` (public)

Credentials

Grafana’s admin login, and every other generated credential, lives in Secret Manager. Read one with:

gcloud secrets versions access latest \
  --secret=observability-victoria-metrics-k8s-stack-grafana-envvars \
  --project=ogenki-435905 | jq -r .GF_SECURITY_ADMIN_USER

ZITADEL’s first admin is in zitadel-envvars (ZITADEL_FIRSTINSTANCE_ORG_HUMAN_USERNAME / …_PASSWORD). Prefer SSO once the OIDC clients are registered:

./scripts/zitadel-oidc-clients.sh sync --cluster gcp-0 --cloud gcp \
  --project ogenki-435905 --apply

6. Observability actually receiving data

A green Grafana proves the deployment, not the pipeline. Check that data is arriving:

# metrics: how many series, and are the scrapes healthy
kubectl exec -n observability deploy/vmsingle-victoria-metrics-k8s-stack -- \
  wget -qO- 'http://localhost:8428/api/v1/query?query=count(up)'

# logs
kubectl logs -n observability deploy/victoria-logs --tail=5

In Grafana, the Kubernetes / Views / Global dashboard should show nodes and pods within a minute of loading. An empty panel with a healthy datasource usually means the time range is wrong rather than the stack — event-driven components need 6–12h.

7. Alerting reaches Slack

kubectl get vmalert -A
kubectl get secret flux-slack-app -n flux-system      # must exist

Flux’s Alert is eventSeverity: error, so a healthy cluster sends nothing. Silence is the expected state; it is not evidence the wiring works. To prove the path, look for past deliveries in the notification controller:

kubectl logs -n flux-system deploy/notification-controller --tail=50 | grep -i slack

8. Tear it down

Do not leave it running.

cd opentofu/gcp/gke/init
TM_GCP_ENABLED=true terramate script run destroy

Then the rest, and verify the project is empty — see Teardown.

Teardown