OpenBao

Foundations covers how the OpenBao cluster is provisioned — five SPOT nodes on Raft, KMS auto-unseal, RAID-0 NVMe. This page covers what runs on top of that cluster: opentofu/openbao/management/ layers namespaces, auth methods, the PKI, and policies onto it, and this is the operational surface every other security page and the Access guide build on.

One secret predates both stacks: OpenBao’s own server certificate — the leaf terminating TLS on bao.priv.cloud.ogenki.io:8200, see PKI & Secrets — is generated offline and read from Secrets Manager at certificates/priv.cloud.ogenki.io/openbao (the openbao_certificates_secret_name default in opentofu/openbao/cluster/variables.tf, set explicitly in that stack’s variables.tfvars) by opentofu/openbao/cluster/ before the management stack in this page ever runs.

Namespace layout

Namespaces are tenancy boundaries, not folders. Earlier revisions nested the PKI and operator logins under admin/admin/pki, which didn’t hold up: an admin namespace is a role, not a tenant, and cluster-wide operations (sys/storage/raft/*, audit devices, seal operations) are root-only regardless — a snapshot agent parked in a child namespace could never reach them. The current layout, verified against opentofu/openbao/management/namespaces.tf:

• Root namespace holds every shared platform service: the PKI mount (pki_private_issuer), the snapshot-agent and cert-manager AppRoles, and the userpass operator login. One login now carries both platform policies, instead of one login per namespace.
• app is the only tenant namespace defined today. It holds a secret/ kv-v2 mount, reachable through its own AppRole (vault_auth_backend.approle_app) — a worked example for future tenants, not yet consumed by anything.
• Cluster-wide endpoints such as sys/storage/raft/* are callable only from root — the API rejects them from any child namespace with a 404 unsupported path, no matter what the token’s policy grants.

Operator login

Human operators authenticate with userpass, not the root token — the root token is retired after initial setup (see below). The backend and user are provisioned by Terraform (opentofu/openbao/management/auth.tf), not created by hand:

export VAULT_ADDR=https://bao.priv.cloud.ogenki.io:8200
export VAULT_CACERT=opentofu/openbao/management/.tls/ca.pem   # written by `openbao-config.sh ca`
bao login -method=userpass username=admin

Use VAULT_CACERT, never VAULT_SKIP_VERIFY — a security reference that tells readers to skip TLS verification undercuts itself, and the real CA chain is one command away. The password is generated by the management stack and published to Secrets Manager:

aws secretsmanager get-secret-value \
  --secret-id openbao/cloud-native-ref/users/admin \
  --query SecretString --output text | jq

The admin login carries both the admin and pki-admin policies. It has no token_bound_cidrs, unlike the machine AppRoles below — the only route to the API is the internal NLB, so the network is already constrained, and a CIDR bind on the one break-glass credential buys nothing against the risk of locking yourself out of the secrets store.

AppRole: machine authentication

AppRole assigns a RoleID/SecretID pair to a workload so it can authenticate without a human-held token. One approle auth backend in root hosts every machine role — snapshot-agent (Raft snapshot capability) and cert-manager (PKI issuance) — each bound to a scoped policy:

resource "vault_auth_backend" "approle" {
  type = "approle"
  path = "approle"
}

resource "vault_approle_auth_backend_role" "snapshot" {
  backend           = vault_auth_backend.approle.path
  role_name         = "snapshot-agent"
  token_policies    = [vault_policy.snapshot.name]
  token_bound_cidrs = var.allowed_cidr_blocks
}

The underlying policy is scoped to exactly the path the role needs — the snapshot policy grants nothing beyond sys/storage/raft/snapshot:

path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}

Each role’s credentials are minted once by Terraform and published to Secrets Manager — but not under one shared path pattern per role:

then synced into the cluster by External Secrets — see PKI & Secrets. The tenant-namespace app AppRole has no minted SecretID, and so no Secrets Manager entry, at all: nothing consumes it yet, and an unused live credential is worse than none (opentofu/openbao/management/auth.tf) — mint one by hand with bao write -f -namespace=app auth/approle/role/app/secret-id only when something needs it.

Nothing mints an AppRole SecretID by hand outside that flow; a credential created outside the stack that manages every other credential drifts by construction.

Cluster initialisation

Initialisation is not a day-2 operation — it happens once, when the Raft cluster first comes up, and is automated rather than run by hand: terramate script run deploy calls scripts/openbao-config.sh (init subcommand — see Commands for the full script table), which runs bao operator init -recovery-shares=1 -recovery-threshold=1 and writes the result to two separate Secrets Manager entries:

Two details here are load-bearing:

• The recovery keys are persisted at all, not just echoed to stdout and discarded. Without them, bao operator generate-root is impossible — a lost or revoked root token would leave the cluster unrecoverable, and the restore path below could never authenticate.
• They live in a different secret than the root token. Storing both together makes the pair only as strong as whichever secret leaks first.

-recovery-shares=1 -recovery-threshold=1 fits an automated flow — one share, one holder. For anything longer-lived than a demo cluster, raise both and distribute the shares to separate holders instead of one Secrets Manager entry.

Sanity checks once the cluster is up:

bao status                       # Initialized: true, Sealed: false
bao operator raft list-peers     # ha mode only — lists every Raft voter

Backup and restore

Raft’s own snapshot mechanism, automated end to end — nothing here is a manual bao operator raft snapshot save run by a human on a schedule.

Backup. A CronJob in the security namespace (manifests under security/base/openbao-snapshot/) uses the snapshot-agent AppRole to save a Raft snapshot and ship it to S3. Its EKS Pod Identity role deliberately has no secretsmanager permission: a daily backup pod able to read the material that regenerates a root token would be a privilege escalation, not a convenience. Trigger one manually with:

kubectl create job --namespace security --from=cronjob/openbao-snapshot manual-openbao-snapshot-$(date +%s)

Restore. scripts/openbao-snapshot.sh (restore subcommand) fetches the newest snapshot from S3, mints a temporary root token from the recovery key, restores, and checks that secret/check_timestamp is recent enough to rule out restoring a stale backup over a good cluster. Run it as an operator, never as the CronJob — it needs RECOVERY_KEYS_SECRET_ID and AWS credentials that can read that secret, which the snapshot job’s Pod Identity role is deliberately denied.

The script itself does export HOME=/snapshot, unconditionally, before the first line of restore runs — the CronJob gets that for free from a mounted emptyDir, but a bare operator shell does not, and the recovery-token nonce-file write under set -e aborts the whole restore the instant $HOME isn’t writable. Create it before invoking the script:

sudo mkdir -p /snapshot && sudo chown "$(id -u):$(id -g)" /snapshot

The block below is self-contained — every variable the script needs is exported here, not assumed left over from the Operator Login section above:

export VAULT_ADDR="https://bao.priv.cloud.ogenki.io:8200"
export VAULT_CACERT=opentofu/openbao/management/.tls/ca.pem
export APPROLE_ROLE_ID=... APPROLE_SECRET_ID=...
export RECOVERY_KEYS_SECRET_ID="openbao/cloud-native-ref/tokens/recovery"
./scripts/openbao-snapshot.sh restore -a "${VAULT_ADDR}" \
  -b eu-west-3-ogenki-openbao-snapshot -s /tmp/bao.snap -d 8

Prerequisites worth stating plainly:

• Raft storage is required. bao operator raft snapshot save|restore and sys/storage/raft/configuration are Raft-only endpoints — dev mode runs the file backend, and neither backup nor restore works there.
• The script only automates a recovery threshold of 1. A higher threshold makes it exit and tell you to run bao operator generate-root by hand with the required number of shares.
• The restore path is not itself tested in CI. There is no workflow that restores the latest snapshot into a throwaway cluster and asserts its contents — until there is, treat the restore procedure as a hypothesis. The OpenBaoSnapshotStale and OpenBaoSnapshotJobFailed VMRules at least confirm the backup half keeps working.