PKI & Secrets

Every internal TLS certificate on this platform — Gateway API listeners, service-to-service TLS — traces back to one private PKI hosted in OpenBao. Nothing here is a public CA: bao.priv.aws.ogenki.io and everything it signs is only meaningful inside the tailnet.

The three-tier chain

Root CA  →  Intermediate CA  →  Issuer CA (OpenBao pki_private_issuer)  →  Leaf certificates

A Root CA at the top, an Intermediate CA in the middle, end-entity leaf certificates at the bottom. The Root CA issues only to the Intermediate; the Intermediate is what OpenBao’s pki_private_issuer mount imports as its signing certificate and uses to issue every leaf, which keeps revocation/rotation scoped to the tier that actually changed.

Building the chain

The root CA is generated once, on an offline medium, with openssl — EC keys (secp384r1 for the CAs, prime256v1 for OpenBao’s own leaf) rather than RSA.

openssl ecparam -genkey -name secp384r1 -out root-ca-key.pem
openssl req -x509 -new -nodes -key root-ca-key.pem -sha384 -days 3653 -out root-ca.pem

Its private key never leaves that medium. A new intermediate per lineage is signed there too, and only the intermediate’s own certificate and key come back:

cat > intermediate-ca.cnf <<'EOF'
[ v3_req ]
basicConstraints = critical, CA:TRUE, pathlen:0
keyUsage = critical, digitalSignature, keyCertSign, cRLSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
EOF
openssl ecparam -genkey -name secp384r1 -out intermediate-ca-key.pem
openssl req -new -key intermediate-ca-key.pem \
  -subj "/CN=Ogenki AWS Intermediate CA/O=Ogenki/C=FR" -out intermediate-ca.csr
openssl x509 -req -in intermediate-ca.csr -CA root-ca.pem -CAkey root-ca-key.pem \
  -CAcreateserial -out intermediate-ca.pem -days 1827 -sha384 \
  -extfile intermediate-ca.cnf -extensions v3_req
openssl verify -CAfile root-ca.pem intermediate-ca.pem

The intermediate’s certificate and key go to Secrets Manager as {"bundle": "..."} under certificates/priv.aws.ogenki.io/intermediate-ca; the certificates-only chain goes to certificates/priv.aws.ogenki.io/ca-chain as {"ca": "..."}. opentofu/aws/openbao/management/pki.tf imports the bundle as the mount’s issuer:

resource "vault_pki_secret_backend_config_ca" "pki" {
  backend    = vault_mount.pki.path
  pem_bundle = jsondecode(data.aws_secretsmanager_secret_version.intermediate_ca.secret_string)["bundle"]
}

OpenBao’s own server certificate (the one terminating TLS on bao.priv.aws.ogenki.io:8200) is a leaf signed by that intermediate, generated once before the cluster exists and stored in Secrets Manager for opentofu/aws/openbao/cluster/ to consume at bootstrap. It is issued on the same offline medium, immediately after the intermediate and while its key is still to hand:

cat > server.cnf <<'EOF'
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:bao.priv.aws.ogenki.io, DNS:bao.priv.gcp.ogenki.io, DNS:openbao.security.svc.cluster.local, DNS:openbao.security.svc
EOF
openssl ecparam -genkey -name prime256v1 -out server-key.pem && chmod 600 server-key.pem
openssl req -new -key server-key.pem \
  -subj "/CN=bao.priv.aws.ogenki.io/O=Ogenki/C=FR" -out server.csr
openssl x509 -req -in server.csr -CA intermediate-ca.pem -CAkey intermediate-ca-key.pem \
  -CAcreateserial -out server.pem -days 825 -sha256 \
  -extfile server.cnf -extensions v3_req
cat intermediate-ca.pem root-ca.pem > ca-chain.pem
openssl verify -CAfile root-ca.pem -untrusted intermediate-ca.pem server.pem
openssl x509 -in server.pem -noout -ext subjectAltName

openssl verify must print server.pem: OK and the SAN line must list all four names before anything is written to a secret store — a leaf short of one name fails at a different layer for each name it lacks, and the cheapest place to catch that is here.

The CN stays the node’s own address (bao.priv.aws.ogenki.io), so nothing already trusting that name has to change; the other three names ride along as SANs. The GCP leaf is issued exactly the same way, against openbao-priv-gcp-intermediate-ca rather than the AWS intermediate and with /CN=bao.priv.gcp.ogenki.io — the SAN list is identical, because either node may answer for either address during a failover.

Two details worth carrying forward if you regenerate it:

• The key is EC P-256, matching the EC P-384 CAs above, and openssl writes key files world-readable by default — chmod 600 it, since this key terminates TLS for every OpenBao client.
• The SAN list has no IP address — the load-bearing property, and the one the four names above do not imply: a client connecting to a Raft peer by private IP (rather than by one of those names) cannot verify TLS against it. The names cover every way a client may legitimately connect: the node’s own address, the other cloud’s node during a failover, and the neutral in-cluster Service in both its forms.
• On this reference platform the deployed certificates do not carry that list yet. certificates/priv.aws.ogenki.io/openbao predates it and holds only bao.priv.aws.ogenki.io; openbao-priv-gcp-server-cert holds only bao.priv.gcp.ogenki.io. Each is fixed by re-issuing that cloud’s leaf with the SAN list above, under that cloud’s own intermediate. Until then, cert-manager cannot verify openbao.security.svc.cluster.local and its ClusterIssuer fails with x509: certificate is valid for bao.priv.<cloud>.ogenki.io, not openbao.security.svc.cluster.local.

Storing the chain

Three secrets carry the result. Build the JSON payloads with jq --rawfile so the PEM newlines survive — a shell-interpolated "$(cat …)" collapses them and OpenBao rejects the bundle:

jq -n --rawfile c intermediate-ca.pem --rawfile k intermediate-ca-key.pem '{bundle: ($c + $k)}' > intermediate.json
jq -n --rawfile ca ca-chain.pem '{ca: $ca}' > chain.json
jq -n --rawfile cert server.pem --rawfile key server-key.pem --rawfile ca ca-chain.pem '{cert: $cert, key: $key, ca: $ca}' > server.json

Which verb depends on whether the secret already exists, and on AWS the answer differs per secret — create-secret on an existing name fails with ResourceExistsException, and put-secret-value on a missing one fails with ResourceNotFoundException. Check first rather than guess:

aws secretsmanager list-secrets --region eu-west-3 \
  --query 'SecretList[?contains(Name, `priv.aws.ogenki.io`)].Name' --output text

aws secretsmanager put-secret-value --region eu-west-3 \
  --secret-id certificates/priv.aws.ogenki.io/intermediate-ca --secret-string file://intermediate.json
aws secretsmanager create-secret --region eu-west-3 \
  --name certificates/priv.aws.ogenki.io/ca-chain --secret-string file://chain.json
aws secretsmanager put-secret-value --region eu-west-3 \
  --secret-id certificates/priv.aws.ogenki.io/openbao --secret-string file://server.json

Prefer put-secret-value wherever the name exists: it adds a version and leaves the previous one recoverable, which matters because the shape changes — intermediate-ca moves from {cert, key} to {bundle}, and only the new shape satisfies jsondecode(...)["bundle"] in pki.tf.

Then destroy the key material that does not belong in a secret store. The intermediate key exists only inside intermediate-ca’s bundle from here on, and the leaf key only inside openbao:

shred -u intermediate-ca-key.pem server-key.pem intermediate.json server.json 2>/dev/null \
  || rm -f intermediate-ca-key.pem server-key.pem intermediate.json server.json

The root key is not in that list and must never be — it stays on the offline medium.

Committing the root certificate

The root certificate is public, and the weekly restore drill verifies a restored chain against it with openssl verify -CAfile. Reading it from the repository rather than from a cloud secret store is deliberate: the one job whose value is being usable during an outage should not depend on the outage being over. From the ceremony directory, with REPO pointing at your clone:

REPO=~/Sources/cloud-native-ref
cp root-ca.pem "$REPO/.github/openbao-root-ca.pem"
cd "$REPO"
git check-ignore .github/openbao-root-ca.pem   # exit 1 == not ignored == correct
git add .github/openbao-root-ca.pem
git ls-files --error-unmatch .github/openbao-root-ca.pem   # errors if still untracked
git commit -m "chore(pki): commit the offline root certificate for the restore drill"

Only root-ca.pem is copied. root-ca-key.pem sits beside it under a name one character longer and never leaves the offline medium — it is the only file in the ceremony that can issue a certificate, and the whole design rests on it never reaching a networked store.

Trusting the CA on your machine

Every private service is served with a certificate from this chain, and no system trust store knows the offline root. Until you import it, browsers and curl reject *.priv.aws.ogenki.io and *.priv.gcp.ogenki.io outright — the failure looks like a broken deployment rather than a missing trust anchor.

This has to be redone whenever the root changes, which in practice means after a cluster rebuild that regenerated the PKI, or when a cluster moves to a new private domain.

Fetch the chain with openbao-config.sh ca — it knows where the secret lives on each cloud, and that AWS stores it as JSON under a .ca key while GCP stores raw PEM:

# aws-0
./scripts/provision/openbao-config.sh ca --region eu-west-3 \
  --root-ca-secret-name certificates/priv.aws.ogenki.io/ca-chain \
  --ca-output-file /tmp/ogenki-aws-ca.pem

# gcp-0
./scripts/provision/openbao-config.sh ca --cloud gcp --project ogenki-435905 \
  --root-ca-secret-name openbao-priv-gcp-ca-chain \
  --ca-output-file /tmp/ogenki-gcp-ca.pem

Check you got a certificate and not an error page before importing anything:

openssl x509 -in /tmp/ogenki-aws-ca.pem -noout -subject -dates

Then add it to the system trust store:

# Arch / Fedora (p11-kit)
sudo trust anchor --store /tmp/ogenki-aws-ca.pem

# Debian / Ubuntu
sudo cp /tmp/ogenki-aws-ca.pem /usr/local/share/ca-certificates/ogenki-aws.crt
sudo update-ca-certificates

# macOS
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain /tmp/ogenki-aws-ca.pem

Both clouds now chain to the same offline root (ADR-0033), so one import covers both — that root certificate is also committed as .github/openbao-root-ca.pem. This was two imports until the AWS ceremony ran on 2026-09-05, when the AWS chain still descended from its own root; if you are looking at an older cluster, import both files above and you are covered under either state.

Nothing else on your machine needs the file afterwards. The OpenBao management stack fetches its own copy into a gitignored .tls/ directory at apply time — that one exists so the Vault provider can verify the server at plan time, not for your browser.

cert-manager: issuing from the PKI

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: openbao
  namespace: security
spec:
  vault:
    server: https://openbao.security.svc.cluster.local:8200
    path: pki_private_issuer/sign/ogenki
    caBundleSecretRef:
      name: openbao-ca
      key: ca.crt
    auth:
      kubernetes:
        mountPath: /v1/auth/jwt/${cluster_name}
        role: cert-manager
        serviceAccountRef:
          name: cert-manager
          audiences:
            - openbao

A ClusterIssuer reaches OpenBao by the neutral in-cluster name openbao.security.svc.cluster.local — an ExternalName Service in security/base/openbao-endpoint/ that a cluster points at its own cloud’s load balancer (local form) or, through the Tailscale operator’s egress ProxyGroup, at the other cloud’s (remote form). It authenticates with a projected ServiceAccount token against jwt/<cluster>: cert-manager requests a 10-minute token with audience openbao for its own ServiceAccount and POSTs it with the role name. No AppRole, no SecretID, nothing synced. The CA bundle is still an ExternalSecret, from certificates/<domain>/ca-chain.

Neither the PKI mount nor the auth mount needs a namespace: field on the issuer — both live in OpenBao’s root namespace (see OpenBao).

A Certificate object requesting one of these leaves looks like any other cert-manager request:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: foobar
spec:
  secretName: foobar-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  commonName: foobar.priv.aws.ogenki.io
  dnsNames:
    - foobar.priv.aws.ogenki.io
    - foobar.security.svc.cluster.local
  issuerRef:
    name: openbao
    kind: ClusterIssuer
    group: cert-manager.io

This is also what terminates TLS at the Gateway API layer: Gateway listeners reference a Secret cert-manager keeps populated from this same issuer, so rotation is automatic — cert-manager renews renewBefore the expiry and the Gateway picks up the new Secret without a redeploy.

External Secrets: the other direction

Where cert-manager pulls certificates out of OpenBao’s PKI, External Secrets Operator pulls arbitrary credentials into the cluster. Most of them now come from OpenBao itself — see Secrets for the two mounts, who may read each one, and what a developer writes to give an application a secret.

What remains on the cloud’s managed store is the bootstrap tier: the CA chain above, OpenBao’s own server certificate, the root token and recovery keys — every value that has to be read before OpenBao has an API — plus a few runtime-generated database credentials. That tier is served by the store below, which is the one this section describes and the only one that predates the move:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: clustersecretstore
spec:
  provider:
    aws:
      region: ${region}
      service: SecretsManager

On gcp-0 the same store name is backed by GCP Secret Manager instead (security/gcp-0/openbao/clustersecretstore.yaml), with no auth block at all — deliberately: under GKE Workload Identity the controller’s ServiceAccount is itself a Google principal, so there is no key to mount, rotate, or leak:

spec:
  provider:
    gcpsm:
      projectID: ${project_id}

Why the managed store was originally chosen over OpenBao — cost, lifecycle, and the bootstrap circularity — is recorded in ADR-0025; the shared store name and the dash-grammar keys that let one ExternalSecret work on both clouds are ADR-0023. Of those three reasons only the circularity still binds, and it binds only on the bootstrap tier — which is why that tier, and nothing else, is still here.

This is the platform’s concrete instance of the constitution’s Secrets Management rule: no hardcoded credentials in a manifest, HelmRelease, or Crossplane composition — everything resolves through a ClusterSecretStore at reconcile time, refreshed on an interval (refreshInterval: 1h is typical) rather than baked in once.

Rotation

Nothing here rotates itself yet. The intermediate is a manual re-import into pki_private_issuer when it approaches its days expiry; leaf certificates rotate automatically through cert-manager’s renewBefore. Treat the Root/Intermediate chain’s expiry the same way as any other operational calendar item — there’s no alert wired to it today.