Troubleshooting

Generic Kubernetes advice is available everywhere. This page carries the failures specific to this platform, most of which have cost real debugging time and several of which fail silently.

Start here on any timeout

Check the network policy first. Default-deny is the norm here, so a missing egress rule is the most common cause of a connection that hangs rather than refuses.

kubectl get ciliumnetworkpolicy -n <ns>
hubble observe --pod <ns>/<pod> --verdict DROPPED --last 50

Gateways stuck “Waiting for controller”

The most confusing failure in the platform, because nothing crashes.

cilium-operator probes for the Gateway API CRDs once, at startup. If any are missing it permanently disables its Gateway API controller — no crash, no alert, no retry. The symptoms cascade: GatewayClass shows ACCEPTED=Unknown, Gateways never become programmed, HTTPRoutes have no status.parents, and every App claim that owns a route sits READY=False.

Confirm:

kubectl logs -n kube-system -l io.cilium/app=operator | grep "Required GatewayAPI resources"

Recover:

kubectl rollout restart -n kube-system deployment/cilium-operator

The durable fix is adding the missing CRD to gateway_api_crds_urls in opentofu/eks/configure/locals.tf. Flux applies the full CRD directory, but only after Cilium is already running — which is why the operator can start before the CRD exists.

CiliumNetworkPolicy traps

Four failure modes that all present as “the policy looks right but traffic is dropped”.

DNS L7 inspection is mandatory for toFQDNs. The kube-dns egress rule must include toPorts.rules.dns.matchPattern: "*". Without it Cilium proxies the query but never sees the response IPs, so the toFQDNs allowlist has no addresses to match and every follow-up TCP connection is silently denied — while DNS itself keeps working, which is what makes it confusing.

matchPattern: "*" does not span dots. It is a single-segment glob: *.huggingface.co matches cdn.huggingface.co but not cas-bridge.xet.huggingface.co. CDN topology fans out quickly, so toFQDNs becomes a maintenance chase.

toEntities: world excludes link-local addresses. And toCIDR alone does not match host-network endpoints. The EKS Pod Identity Agent at 169.254.170.23:80 runs on the node’s host network, so Cilium classifies the destination as the host entity and a toCIDR: 169.254.170.23/32 rule silently fails. Use toEntities: ["host"] scoped to TCP 80. The symptom is an AWS SDK reporting Connect timeout on endpoint URL: 'http://169.254.170.23/v1/credentials'.

Diagnostic order: hubble observe --verdict DROPPED → reverse-IP the dropped address with cilium fqdn cache list -o json on that node’s agent → check rules.dns on the kube-dns rule → check glob depth → check link-local.

A Flux dependency failure cascades

One real error produces a dozen alarming messages. A Kustomization health-checking a namespaced resource whose namespace does not exist reports namespaces "X" not found, and everything depending on it reports dependency … not ready.

Fix the root, not the cascade. flux get kustomizations and look for the one whose message is not “dependency not ready”.

Flux source sharding

This repository shards Flux controllers. A GitRepository or HelmRepository placed under an application directory inherits sharding.fluxcd.io/key=apps, and the default-shard HelmChart then cannot find it — the source reports Ready while the consumer reports source not found.

Sources live under flux-sources, which carries no shard label.

App claim rejected at apply time

CEL validation rejects invalid claims at the API server, and the message names the field.

App stuck not ready

An App reports Ready only when its underlying resources genuinely are. Work down the chain:

kubectl describe app <name> -n <ns>
kubectl get deployment,cronjob -l app.kubernetes.io/name=<name> -n <ns>
kubectl get pods -l app.kubernetes.io/name=<name> -n <ns>
kubectl logs <pod> -n <ns>          # -c <container> for a sidecar or init container

Then the surrounding resources:

kubectl get externalsecret -n <ns>          # secrets syncing?
kubectl get httproute,gateway -n <ns>       # route accepted?
kubectl get ciliumnetworkpolicy -n <ns>     # policy in place?
kubectl get sqlinstance,kvstore -n <ns>     # data services ready?

Pods failing to start

ImagePullBackOff — check image.repository and image.tag; for a private registry confirm imagePullSecrets names a docker-registry Secret in the same namespace; use Always for mutable tags and IfNotPresent for pinned ones.

CrashLoopBackOff or never becoming ready — readOnlyRootFilesystem is on by default, so an application that writes outside /tmp needs a volume via persistence or extraVolumes. The default probe port is service.port and the default paths are /healthz and /readyz; override the probe path or type if yours differ. For slow boots add a startup probe rather than inflating initialDelaySeconds.

Crossplane

<Kind>/<name> namespace not specified — managed resources are namespaced in provider-aws v2; add metadata.namespace.

no matches for kind <Kind> — the CRD is not activated. Provider packages ship many CRDs but only those listed in infrastructure/base/crossplane/providers/activation-policy.yaml are installed.

XR reconcile loops on failed waiting for … Informer to sync — the Crossplane service account lacks RBAC for a third-party Kind. Check with kubectl auth can-i --as=system:serviceaccount:crossplane-system:crossplane list <plural> -A first; if that returns yes, restart the controller.

OpenBao

bao status hangs against 127.0.0.1 — that is a core deadlock, not a VPN problem. See OpenBao for the 2.6 line’s known issue and the serialisation that works around it.

Reading on

• Networking — the datapath and gateway model
• GitOps — the dependency graph behind cascading failures
• Developer platform — what each App field renders