Verify the cluster

Everything below is read-only. Run it top to bottom after a deploy, or dip into whichever section you are suspicious of.

1. Authenticate

gcloud auth login                       # once per machine
gcloud config set project ogenki-435905

gcloud container clusters get-credentials gcp-0 \
  --location europe-west4-a --project ogenki-435905

kubectl get nodes

curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  https://container.googleapis.com/v1/projects/ogenki-435905/zones/europe-west4-a/clusters/gcp-0

export CLOUDSDK_AUTH_ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
gcloud container clusters get-credentials gcp-0 --location europe-west4-a --project ogenki-435905
kubectl get nodes

To keep this out of your usual kubeconfig, point KUBECONFIG at a scratch file before get-credentials:

export KUBECONFIG=/tmp/gcp0.kubeconfig

2. GitOps layer

flux get kustomizations           # everything True, bar the suspended ones
flux get helmreleases -A
flux get sources all -A

These Kustomizations report no status at all, and that is correct — they are spec.suspend: true:

Anything else not True is a real failure. Start with the message:

kubectl get kustomization -A \
  -o custom-columns=NAME:.metadata.name,READY:.status.conditions[0].status,MSG:.status.conditions[0].message

A dependency '…' is not ready message is a cascade, not a cause — walk to the root of the chain and fix that one.

3. Secrets

The single most common cause of a stuck cluster, and the least self-evident: a missing or unreadable secret surfaces as a HelmRelease timing out ten minutes later, naming only the HelmRelease.

./scripts/provision/secret-store.sh check --cloud gcp --project ogenki-435905

That lists every key the cluster’s ExternalSecrets ask for and whether it exists. For the in-cluster side:

kubectl get externalsecrets -A

SecretSyncedError has two usual causes, and they look identical from the cluster:

• the secret does not exist — secret-store.sh check says so;
• External Secrets cannot read it — the grant is per-secret, never project-wide, so a newly created secret is unreadable until it is granted:

gcloud secrets get-iam-policy <name> --project ogenki-435905

4. Certificates and networking

kubectl get certificates -A          # all True
kubectl get gateways -A              # PROGRAMMED=True, each with an address
kubectl get httproute -A

Private hostnames resolve only over the tailnet; public ones resolve from anywhere through Route 53 (ADR-0019).

dig +short auth.gcp.cloud.ogenki.io          # public, from anywhere
dig +short grafana.priv.gcp.ogenki.io        # private, needs the tailnet

5. Browse it

Everything private is *.priv.gcp.ogenki.io and needs the tailnet. Start at the homepage, which links the rest.

curl -sSk -o /dev/null -w '%{http_code}\n' https://grafana.priv.gcp.ogenki.io/login

Credentials

Grafana’s admin login, and every other generated credential, lives in Secret Manager. Read one with:

gcloud secrets versions access latest \
  --secret=observability-victoria-metrics-k8s-stack-grafana-envvars \
  --project=ogenki-435905 | jq -r .GF_SECURITY_ADMIN_USER

ZITADEL’s first admin is in zitadel-envvars (ZITADEL_FIRSTINSTANCE_ORG_HUMAN_USERNAME / …_PASSWORD). Prefer SSO.

Single sign-on

Setting SSO up is a deploy step, not a verification one — it is five ordered commands and it has its own page: Set up single sign-on.

To check it is working, open any of the services above: you should be offered a Google button rather than a username field.

6. Observability actually receiving data

A green Grafana proves the deployment, not the pipeline. Check that data is arriving:

# metrics: how many series, and are the scrapes healthy
kubectl exec -n observability deploy/vmsingle-victoria-metrics-k8s-stack -- \
  wget -qO- 'http://localhost:8428/api/v1/query?query=count(up)'

# logs
kubectl logs -n observability deploy/victoria-logs --tail=5

In Grafana, the Kubernetes / Views / Global dashboard should show nodes and pods within a minute of loading. An empty panel with a healthy datasource usually means the time range is wrong rather than the stack — event-driven components need 6–12h.

7. Alerting reaches Slack

kubectl get vmalert -A
kubectl get secret flux-slack-app -n flux-system      # must exist

Flux’s Alert is eventSeverity: error, so a healthy cluster sends nothing. Silence is the expected state; it is not evidence the wiring works. To prove the path, look for past deliveries in the notification controller:

kubectl logs -n flux-system deploy/notification-controller --tail=50 | grep -i slack

8. Tear it down

Do not leave it running.

cd opentofu/gcp/gke/init
TM_CLOUD=gcp terramate script run destroy

Then the rest, and verify the project is empty — see Teardown.