Phase 7 — Vault HA + External Secrets Operator (ESO)¶

Deploy HashiCorp Vault (two-tier HA pattern) and External Secrets Operator so all cluster credentials are stored in Vault and synced to Kubernetes Secrets via ESO — no plaintext tokens in Git or K8s manifests.

Architecture¶

Active deployment: Standalone

This cluster uses the Standalone topology for Main Vault (single pod, storage "file"). The Raft HA topology is documented below as the production-correct alternative.

Standalone (active)¶

┌─────────────────────────────────────────────────────┐
│  Root Vault (standalone, 1 pod)                     │
│  Namespace: vault-root                              │
│  Storage: Raft (1 node, 5Gi)                        │
│  Seal: AUTO after first init (DIY unsealer)         │
│  Purpose: Transit engine only — issues unseal key   │
│  No ingress — cluster-internal only                 │
│  Init keys: ~/hashicorp_vault/vault-root-init-mod-auh1-dev.json     │
│                                                     │
│  ┌─────────────────────────────────────────────┐   │
│  │  vault-root-unsealer (Deployment, 1 pod)    │   │
│  │  Polls every 15s → vault status exit code   │   │
│  │  If sealed: reads keys from K8s Secret      │   │
│  │  (optional volume mount, hot-reloads)       │   │
│  └─────────────────────────────────────────────┘   │
└────────────────────┬────────────────────────────────┘
                     │ Transit auto-unseal
                     ▼
┌─────────────────────────────────────────────────────┐
│  Main Vault (standalone, 1 pod)                     │
│  Namespace: vault                                   │
│  Storage: File (1 × 10Gi csi-rbd-sc)               │
│  Seal: AUTO via Root Vault Transit                  │
│  Purpose: All application secrets (KV v2)           │
│  Ingress: vault.mod.auh1.dev.dir                    │
│           vault.cl1.sq4.aegis.internal              │
└─────────────────────────────────────────────────────┘
                     │
        ESO ClusterSecretStore (Kubernetes auth)
                     │
        ExternalSecret CRs → K8s Secrets (auto-rotated)

Component	Chart	Version	Namespace	Seal
Root Vault	`vault`	`0.32.0`	`vault-root`	Manual (5 shares / threshold 3)
Main Vault standalone (active)	`vault`	`0.32.0`	`vault`	Auto (Transit → Root Vault)
ESO	`external-secrets`	`0.14.4`	`external-secrets`	—

Raft HA (production-correct alternative)¶

Not deployed — documented for production reference

Raft HA is the production-correct topology. It is not used in this cluster (single-zone, non-critical workloads). Apply these differences to migrate.

┌─────────────────────────────────────────────────────┐
│  Root Vault (unchanged — same as standalone)        │
└────────────────────┬────────────────────────────────┘
                     │ Transit auto-unseal
                     ▼
┌─────────────────────────────────────────────────────┐
│  Main Vault HA (5-node Raft)                        │
│  Namespace: vault                                   │
│  Storage: Raft (5 × 10Gi csi-rbd-sc)               │
│  Seal: AUTO via Root Vault Transit                  │
│  PDB: minAvailable: 3 (survives 2 node failures)    │
│  Anti-affinity: 1 pod per node (required)           │
│  Purpose: All application secrets (KV v2)           │
└─────────────────────────────────────────────────────┘

Differences from standalone:

Item	Standalone (active)	Raft HA (production)
`server.standalone.enabled`	`true`	`false`
`server.ha.enabled`	`false`	`true`, `replicas: 5`
Storage backend	`storage "file"`	`storage "raft"` + 5× `retry_join`
PDB	None	`minAvailable: 3` in `manifests/vault/vault-pdb.yaml`
Anti-affinity	None	`requiredDuringScheduling` — 1 pod per node
Node failure tolerance	0 (single pod)	2 of 5 (quorum = 3)
Manual unseal events	Rare (Transit handles it)	Rare (Transit handles it)
Data durability	Single PVC — PVC failure = data loss	Replicated across 5 nodes

Additional values needed for Raft HA:

server:
  ha:
    enabled: true
    replicas: 5
    raft:
      enabled: true
      setNodeId: true
      config: |
        # ... same listener and seal blocks ...
        storage "raft" {
          path = "/vault/data"
          retry_join { leader_api_addr = "https://vault-0.vault-internal:8200"
                       leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt" }
          retry_join { leader_api_addr = "https://vault-1.vault-internal:8200"
                       leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt" }
          retry_join { leader_api_addr = "https://vault-2.vault-internal:8200"
                       leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt" }
          retry_join { leader_api_addr = "https://vault-3.vault-internal:8200"
                       leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt" }
          retry_join { leader_api_addr = "https://vault-4.vault-internal:8200"
                       leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt" }
        }
        service_registration "kubernetes" {}

  disruptionBudget:
    enabled: false   # managed in manifests/vault/vault-pdb.yaml

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: vault
          topologyKey: kubernetes.io/hostname

Component	Chart	Version	Namespace	Seal
Root Vault	`vault`	`0.32.0`	`vault-root`	Manual (5 shares / threshold 3)
Main Vault HA	`vault`	`0.32.0`	`vault`	Auto (Transit → Root Vault)
ESO	`external-secrets`	`0.14.4`	`external-secrets`	—

Why two Vaults?¶

Question	Answer
Why not just manual unseal on Main Vault?	Manual unseal is 1 operation for 1 pod — manageable. Root Vault + Transit is kept because it eliminates that 1 manual step after any pod restart, making recovery fully automatic.
Why not cloud KMS?	Air-gapped environment — no AWS/Azure/GCP KMS reachable.
Why is Root Vault manually unsealed?	Root Vault is a dedicated single-purpose service that is never restarted in steady state. Manual unseal of 1 pod is acceptable; it is not a secret store for applications.
Why standalone and not Raft for Main Vault?	This is a non-HA air-gap environment. Raft requires 3+ pods for quorum, adding complexity without benefit on a single-tenant cluster. Standalone uses `storage "file"` — simpler config, simpler recovery, no peer coordination needed.
Where are unseal keys stored?	`~/hashicorp_vault/vault-root-init-mod-auh1-dev.json` on bastion. Production requirement: distribute 5 key shares to 5 separate operators — no single person holds all shares. `threshold=3` means any 3 of 5 can unseal.
What happens if Root Vault node fails?	Main Vault pod already running continues operating (unseal key in memory). A restarted Main Vault pod cannot auto-unseal until Root Vault is back. See troubleshooting → Root Vault Node Failure Recovery.
Production-correct alternative?	In production, Main Vault would be 5-node Raft with PDB `minAvailable: 3` and pod anti-affinity. Standalone is a known deviation — acceptable here because workload data is non-critical and the cluster is single-zone.

Production Standards Applied¶

Standard	Implementation
Transit auto-unseal	Root Vault Transit — no manual unseal on restart
Transit token	Static K8s Secret, `transit/` policy only (least-privilege)
TLS	cert-manager `internal-ca-issuer`, auto-rotated
Ingress	Both `vault.mod.auh1.dev.dir` and `vault.cl1.sq4.aegis.internal`
Storage	`csi-rbd-sc` (Ceph RBD, RWO) — 10Gi Main Vault, 5Gi Root
Audit logging	Enabled post-init via CLI (`vault audit enable file`) — not in HCL config
Root Vault auto-recovery	DIY unsealer Deployment — polls every 15s, auto-unseals on restart
Unseal keys	K8s Secret (`vault-root-unseal-keys`), RBAC-locked to unsealer SA only — never in Git
PSA namespace label	`baseline` — required because vault-helm StatefulSet declares `IPC_LOCK`
Pod security hardening	Manual `restricted`-equivalent: `runAsNonRoot`, `allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]`, `seccompProfile: RuntimeDefault` — set in values file, not enforced by PSA
mlock	`disable_mlock = true` in HCL — IPC_LOCK not requested or used; no security loss (swap disabled on all nodes)
Deviation	Standalone (1 pod, file storage) — production would use 5-node Raft with PDB and anti-affinity

Prerequisites¶

Item	Required state
ArgoCD	Running — Reloader ✅ (Phase 8 done)
cert-manager	Running with `internal-ca-issuer` ClusterIssuer
Harbor	`dockerhub/hashicorp/vault:1.21.2` and `ghcr/external-secrets/external-secrets:v0.14.4` present
GitLab Helm registry	`vault` 0.32.0 + `external-secrets` 0.14.4 charts pushed
DNS	`vault.mod.auh1.dev.dir` resolves to HAProxy VIP
`harbor-pull-secret`	Manually copied to `vault-root` and `vault` namespaces (see below)

Pre-step — Namespace creation, PSA labels, and harbor-pull-secret¶

Namespace creation and PSA labeling are fully automated via managedNamespaceMetadata in each ArgoCD Application yaml. No manual kubectl create namespace or kubectl label commands needed — ArgoCD creates the namespace with the correct labels on first sync.

harbor-pull-secret must be present before vault pods can pull images. Copy it to both namespaces once before the first ArgoCD sync:

for NS in vault-root vault; do
  kubectl create namespace ${NS} 2>/dev/null || true
  kubectl get secret harbor-pull-secret -n argocd -o json \
    | python3 -c "import sys,json; s=json.load(sys.stdin); s['metadata']={'name':s['metadata']['name']}; print(json.dumps(s))" \
    | kubectl apply -n ${NS} -f -
done

kubectl get secret harbor-pull-secret -n vault-root
kubectl get secret harbor-pull-secret -n vault

After ESO + Step 7.11b: no more manual copy

Once the ClusterExternalSecret is running, ESO owns and rotates harbor-pull-secret in these namespaces automatically. The manual copy above is only required for the initial bootstrap before ESO exists.

Why baseline and not restricted?

PSA restricted blocks the IPC_LOCK capability at the admission level — the pod never reaches the container runtime. This happens even though we set disable_mlock = true in the Vault HCL config, because the vault-helm StatefulSet template still declares IPC_LOCK in its container spec.

baseline allows IPC_LOCK while still blocking host namespaces, privilege escalation, and all other dangerous capabilities. There is no security loss on Kubernetes nodes where swap is disabled.

Security context is manually hardened to compensate

Using baseline PSA means the cluster no longer enforces restricted security policy on these namespaces. To compensate, the values file explicitly sets restricted-equivalent security context on every vault pod:

server:
  securityContext:                      # pod-level
    runAsNonRoot: true
    runAsUser: 100
    runAsGroup: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault

  containerSecurityContext:             # container-level
    allowPrivilegeEscalation: false
    capabilities:
      drop: [ALL]                       # drops IPC_LOCK too — mlock disabled in HCL
    seccompProfile:
      type: RuntimeDefault

Net security posture:

Control	Enforced by
No privilege escalation	`containerSecurityContext.allowPrivilegeEscalation: false`
No extra capabilities	`capabilities.drop: [ALL]`
Non-root user	`runAsNonRoot: true`, `runAsUser: 100`
Seccomp profile	`seccompProfile.type: RuntimeDefault`
mlock disabled	`disable_mlock = true` in HCL (swap is off on all nodes)
IPC_LOCK allowed by PSA	Yes — but Vault does not request or use it (`disable_mlock = true`)

The only gap vs PSA restricted: baseline does not validate these settings at admission — a misconfigured pod could still run without them. This is accepted because the values file is Git-controlled and ArgoCD-managed.

Vault Helm Chart 0.32.0 — Known Gotchas¶

Discovered empirically during deployment. These are not in official docs.

Correct environment variable keys¶

The chart uses two separate keys — extraEnv does not exist in 0.32.0 and is silently ignored:

Key	Type	Use case
`server.extraEnvironmentVars`	dict (`key: value`)	Plain env vars (paths, flags)
`server.extraSecretEnvironmentVars`	list of objects	Values injected from K8s Secrets

server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-tls/ca.crt   # ← dict, not list

  extraSecretEnvironmentVars:                           # ← list of objects
    - envName: VAULT_TOKEN
      secretName: vault-transit-token
      secretKey: token

To verify what keys the chart actually accepts:

helm pull vault --repo https://helm.releases.hashicorp.com --version 0.32.0
tar -xzf vault-0.32.0.tgz
grep -n "extraEnvironment\|extraEnv\|extraSecret" vault/values.yaml

Transit seal — inject token via `VAULT_TOKEN` env var (not in HCL)¶

vault-helm 0.32.0 performs a plain cp of the HCL config file — no envsubst. Variable substitution does not work in the seal stanza:

# WRONG — ${...} passed literally, vault sees it as the literal string
seal "transit" { token = "${VAULT_TRANSIT_TOKEN}" }   # → 403 invalid token

# WRONG — env() is not a valid HCL function in this context
seal "transit" { token = env("VAULT_TOKEN") }         # → HCL parse error

Correct approach: omit token = entirely from the seal stanza. Vault's Transit seal automatically reads the VAULT_TOKEN environment variable. Inject the token via extraSecretEnvironmentVars with envName: VAULT_TOKEN.

server:
  extraSecretEnvironmentVars:
    - envName: VAULT_TOKEN         # Transit seal reads this automatically
      secretName: vault-transit-token
      secretKey: token

  ha:
    raft:
      config: |
        seal "transit" {
          address    = "https://vault-root.vault-root.svc.cluster.local:8200"
          key_name   = "autounseal"
          mount_path = "transit/"
          tls_ca_cert = "/vault/userconfig/vault-tls/ca.crt"
          # token = ...   ← DO NOT ADD — vault reads VAULT_TOKEN env var
        }

`service_registration` must be removed for standalone¶

service_registration "kubernetes" {} enables Kubernetes pod annotations for Vault HA leader/standby state. It is only valid when the storage backend supports HA (Raft, Consul). With storage "file" it causes an immediate crash:

service_registration is configured, but storage does not support HA

This block must be absent from standalone HCL config. It is safe to include in Raft HA config.

PDB — do not add one for standalone¶

The chart creates a PDB by default when ha.enabled: true. With standalone.enabled: true the chart PDB is not created. Do not add a custom PDB manifest either — a PDB with any minAvailable > 0 on a 1-pod deployment permanently blocks node drains.

Step 7.1 — Push Charts to GitLab Helm Registry¶

From your Mac:

cd /Users/tarun.mittal/Documents/096_RKE/scripts/harbor-mirror/helm-charts
export GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)

python3 gitlab-push-charts.py -c gitlab-charts.yaml --only vault
python3 gitlab-push-charts.py -c gitlab-charts.yaml --only external-secrets

Verify:

curl -sk --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/14/packages?package_type=helm" \
  | python3 -m json.tool | grep '"name"'
# Expected: "vault", "external-secrets" in output

Step 7.2 — Mirror Image to Harbor¶

cd /Users/tarun.mittal/Documents/096_RKE/scripts/harbor-mirror

python3 core42-mirror-image.py --image hashicorp/vault:1.21.2
python3 core42-mirror-image.py --image ghcr.io/external-secrets/external-secrets:v0.14.4

Verify:

curl -sk -u 'robot$ask:<password>' \
  "https://harbor.cl1.sq4.aegis.internal/api/v2.0/projects/hashicorp/repositories" \
  | python3 -m json.tool | grep '"name"'

Step 7.3 — Pre-requisite: TLS Certificates (GitOps, sync-wave -1)¶

Both Vault deployments need a TLS secret before pods start. These are managed as Certificate CRs in the manifests/ directory with sync-wave: "-1" so cert-manager issues them before the StatefulSet starts (wave 0).

Files already in mod-ask-devops:

manifests/vault-root/vault-root-tls-certificate.yaml → secret vault-root-tls in vault-root
manifests/vault/vault-tls-certificate.yaml → secret vault-tls in vault

The Main Vault TLS cert covers all 5 pod hostnames via the wildcard SAN *.vault-internal:

dnsNames:
  - vault
  - vault.vault
  - vault.vault.svc
  - vault.vault.svc.cluster.local
  - vault-internal
  - "*.vault-internal"         ← covers vault-0 through vault-4
  - vault.mod.auh1.dev.dir
  - vault.cl1.sq4.aegis.internal

Why *.vault-internal?

Vault Raft peers communicate peer-to-peer via vault-N.vault-internal:8201. A single wildcard SAN covers all 5 peers without listing each hostname individually. Adding replicas later does not require a cert re-issue.

Step 7.4 — Commit and Deploy Root Vault (ArgoCD)¶

All files for Root Vault and Main Vault are already committed to mod-ask-devops:

applications/cluster-addons/vault/
├── values-mod-auh1-dev-ask.yaml                    ← Helm values
└── manifests/
    └── vault-tls-certificate.yaml                  ← cert-manager Certificate CR

applications/cluster-addons/vault-root/
├── values-mod-auh1-dev-ask.yaml                    ← Helm values
└── manifests/
    ├── vault-root-tls-certificate.yaml             ← cert-manager Certificate CR
    └── vault-root-unsealer.yaml                    ← SA + Role + RoleBinding + Deployment

environment/auh1/dev/02-argocd-application/cluster-addons/vault-root.yaml
environment/auh1/dev/02-argocd-application/cluster-addons/vault.yaml

Why a manifests/ subdirectory inside the application folder?

ArgoCD's path source applies every yaml file in the target directory as a Kubernetes manifest. If the path pointed directly at applications/cluster-addons/vault/, ArgoCD would try to apply values-mod-auh1-dev-ask.yaml as a K8s resource and fail. The manifests/ subdirectory keeps the ArgoCD path source cleanly separated from the Helm values file.

No PDB for standalone

No PDB manifest is present. A PDB with minAvailable: 3 on a 1-pod deployment permanently blocks node drains — do not add one.

harbor-pull-secret is a manual prerequisite — not in Git

The secret contains registry credentials and must never be committed. It is created once manually (see Pre-step above) and lives only in the cluster. The pre-step must be re-run for any new cluster or after namespace deletion.

Push to GitLab — ArgoCD's cluster-addons ApplicationSet picks up vault-root and vault Applications automatically.

Deploy Root Vault FIRST — do not sync Main Vault until Transit is configured

Main Vault depends on Root Vault's Transit seal. If Main Vault pods start before the Transit token K8s Secret exists, they will stay sealed and restart continuously.

Correct sequence: 1. Sync vault-root Application only 2. Initialize and unseal Root Vault (Steps 7.5–7.6) 3. Create the vault-transit-token K8s Secret (Step 7.7) 4. Sync vault Application (Step 7.8)

Trigger Root Vault sync from ArgoCD UI or CLI:

argocd app sync vault-root --assumeYes
kubectl get pods -n vault-root -w
# Wait for vault-root-0 Running (will be 0/1 — sealed, expected)

Step 7.5 — Initialize Root Vault¶

# Initialize (5 shares, threshold 3) — tee ensures output is saved AND visible
kubectl exec -n vault-root vault-root-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-root-tls/ca.crt \
  vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json | tee ~/hashicorp_vault/vault-root-init-mod-auh1-dev.json

# Verify file has content
ls -la ~/hashicorp_vault/vault-root-init-mod-auh1-dev.json

Store the init file securely

This file contains all 5 unseal key shares and the root token. It must not remain on bastion disk in production — encrypt it (GPG, Vault, HSM) and distribute the 5 key shares to 5 separate operators. In this deployment, ~/hashicorp_vault/vault-root-init-mod-auh1-dev.json on bastion is the storage location.

Step 7.6 — Unseal Root Vault¶

Root Vault has 1 pod, requires 3 of 5 key shares:

UNSEAL_1=$(python3 -c "import sys,json; d=json.load(open('${HOME}/hashicorp_vault/vault-root-init-mod-auh1-dev.json')); print(d['unseal_keys_b64'][0])")
UNSEAL_2=$(python3 -c "import sys,json; d=json.load(open('${HOME}/hashicorp_vault/vault-root-init-mod-auh1-dev.json')); print(d['unseal_keys_b64'][1])")
UNSEAL_3=$(python3 -c "import sys,json; d=json.load(open('${HOME}/hashicorp_vault/vault-root-init-mod-auh1-dev.json')); print(d['unseal_keys_b64'][2])")
ROOT_TOKEN=$(python3 -c "import sys,json; d=json.load(open('${HOME}/hashicorp_vault/vault-root-init-mod-auh1-dev.json')); print(d['root_token'])")

kubectl exec -n vault-root vault-root-0 -- vault operator unseal ${UNSEAL_1}
kubectl exec -n vault-root vault-root-0 -- vault operator unseal ${UNSEAL_2}
kubectl exec -n vault-root vault-root-0 -- vault operator unseal ${UNSEAL_3}

# Verify
kubectl exec -n vault-root vault-root-0 -- vault status
# Expected: Initialized=true, Sealed=false

Step 7.6b — Create Unseal Keys Secret (one-time, post-init)¶

After initializing Root Vault, store 3 of the 5 key shares in a K8s Secret so the DIY unsealer can auto-recover after any future pod restart.

UNSEAL_1=$(python3 -c "import json; d=json.load(open('${HOME}/hashicorp_vault/vault-root-init-mod-auh1-dev.json')); print(d['unseal_keys_b64'][0])")
UNSEAL_2=$(python3 -c "import json; d=json.load(open('${HOME}/hashicorp_vault/vault-root-init-mod-auh1-dev.json')); print(d['unseal_keys_b64'][1])")
UNSEAL_3=$(python3 -c "import json; d=json.load(open('${HOME}/hashicorp_vault/vault-root-init-mod-auh1-dev.json')); print(d['unseal_keys_b64'][2])")

kubectl create secret generic vault-root-unseal-keys \
  -n vault-root \
  --from-literal=key1=${UNSEAL_1} \
  --from-literal=key2=${UNSEAL_2} \
  --from-literal=key3=${UNSEAL_3}

This Secret is NEVER committed to Git

vault-root-unseal-keys contains real key material — it lives only in the cluster. The unsealer Deployment in manifests/vault-root/vault-root-unsealer.yaml IS in Git; the Secret itself is not. The Secret is created manually once, after Root Vault init.

How the unsealer works — no pod restart needed

The unsealer mounts the Secret as a volume (not env vars) with optional: true. This means: - The unsealer pod starts immediately even before the Secret exists - Kubernetes hot-reloads mounted secret volumes within ~60s of Secret creation - The shell loop re-reads the key files on every 15s iteration — picks up new values automatically - No pod restart needed when the Secret is created

RBAC lock-down: the unsealer ServiceAccount can only get the vault-root-unseal-keys Secret — no other secrets in the namespace are accessible to it.

Vault exit codes: - 0 = unsealed and active — loop skips - 1 = error (not yet initialized, connection refused) — loop skips - 2 = sealed — loop sends 3 unseal keys

Step 7.7 — Configure Root Vault Transit Engine¶

This creates the Transit key and a minimal-privilege token that Main Vault uses for auto-unseal. The token is stored in a K8s Secret in the vault namespace.

# Login to Root Vault
kubectl exec -n vault-root vault-root-0 -- \
  env VAULT_TOKEN=${ROOT_TOKEN} vault login ${ROOT_TOKEN}

# Enable Transit secrets engine
kubectl exec -n vault-root vault-root-0 -- \
  env VAULT_TOKEN=${ROOT_TOKEN} \
  vault secrets enable transit

# Create auto-unseal key (aes256-gcm96)
kubectl exec -n vault-root vault-root-0 -- \
  env VAULT_TOKEN=${ROOT_TOKEN} \
  vault write -f transit/keys/autounseal

# Create least-privilege policy — transit encrypt/decrypt only
kubectl exec -n vault-root vault-root-0 -- \
  env VAULT_TOKEN=${ROOT_TOKEN} \
  vault policy write autounseal-policy - << 'POLICY'
path "transit/encrypt/autounseal"  { capabilities = ["update"] }
path "transit/decrypt/autounseal"  { capabilities = ["update"] }
POLICY

# Create a periodic token (never expires, renewable)
TRANSIT_TOKEN=$(kubectl exec -n vault-root vault-root-0 -- \
  env VAULT_TOKEN=${ROOT_TOKEN} \
  vault token create \
    -policy=autounseal-policy \
    -period=87600h \
    -format=json | python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")

echo "Transit token: ${TRANSIT_TOKEN}"

# Enable audit logging (post-init — not in HCL config file)
kubectl exec -n vault-root vault-root-0 -- \
  env VAULT_TOKEN=${ROOT_TOKEN} \
  vault audit enable file file_path=/vault/logs/audit.log

Create the K8s Secret that Main Vault reads for auto-unseal:

kubectl create namespace vault 2>/dev/null || true

kubectl create secret generic vault-transit-token \
  -n vault \
  --from-literal=token=${TRANSIT_TOKEN}

kubectl get secret vault-transit-token -n vault -o jsonpath='{.data.token}' | base64 -d
# Should print the transit token

Why a static K8s Secret for the transit token?

At the time Main Vault starts, ESO is not yet deployed — there is no mechanism to inject a Vault-sourced secret into a pod that needs that same Vault to unseal. The transit token K8s Secret is a deliberate bootstrap exception. Once ESO is running and Main Vault is initialized, the token can be rotated and the secret updated via kubectl patch.

Step 7.8 — Deploy Main Vault (ArgoCD)¶

Main Vault Application was committed in Step 7.4 alongside Root Vault. Now that vault-transit-token secret exists, sync it:

argocd app sync vault --assumeYes
kubectl get pods -n vault -w
# Wait for vault-0 Running (0/1 — sealed pre-init, expected)

The pod transitions from 0/1 Running to 1/1 Running automatically once initialized. No manual unseal needed — Root Vault Transit handles it.

Step 7.9 — Initialize Main Vault¶

With Transit auto-unseal, initialization only needs to run once — no unseal keys are required (Transit handles that):

# Wait for vault-0 to be Running (not necessarily Ready)
kubectl exec -n vault vault-0 -- vault status 2>/dev/null | grep -E "Initialized|Sealed"

# Initialize — use tee so output is BOTH displayed and saved
# Shell > redirect creates an empty file before the command runs;
# if kubectl exec fails silently, the file stays empty and keys are lost.
kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  vault operator init \
  -recovery-shares=5 \
  -recovery-threshold=3 \
  -format=json | tee ~/hashicorp_vault/vault-main-init-mod-auh1-dev.json

# Verify file has content (must be > 0 bytes)
ls -la ~/hashicorp_vault/vault-main-init-mod-auh1-dev.json
python3 -c "import json; d=json.load(open('${HOME}/hashicorp_vault/vault-main-init-mod-auh1-dev.json')); print('root_token:', d['root_token'][:8], '...')"

Recovery keys, not unseal keys

When Transit auto-unseal is active, vault operator init generates recovery keys not unseal keys. Recovery keys are used only to regenerate a root token if it is lost — normal unseal is automatic via Transit.

Wait for the pod to become Ready:

kubectl get pods -n vault -w
# Expected: vault-0 — 1/1 Running

# Quick verify
kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  vault status
# Expected: Initialized=true, Sealed=false, Storage Type=file

Step 7.10 — Configure Main Vault (GitOps bootstrap Job)¶

Vault configuration (KV v2, Kubernetes auth, ESO policy/role) is managed as a PostSync ArgoCD Job that re-runs on every sync. All commands are idempotent — re-applying does no harm.

Files in applications/cluster-addons/vault/manifests/:

File	Purpose
`vault-bootstrap-config.yaml`	ConfigMap — idempotent bootstrap shell script
`vault-bootstrap-job.yaml`	PostSync Job — reads token from `vault-bootstrap-token` secret

Pre-step — Create the bootstrap token secret (one-time, not in Git)¶

The Job needs a VAULT_TOKEN to authenticate to Vault. Create a K8s Secret from the root token (use root token only for initial bootstrap — rotate to a limited admin token in production):

MAIN_ROOT=$(python3 -c "import json; d=json.load(open('${HOME}/hashicorp_vault/vault-main-init-mod-auh1-dev.json')); print(d['root_token'])")

kubectl create secret generic vault-bootstrap-token \
  -n vault \
  --from-literal=token=${MAIN_ROOT}

This secret is never committed to Git

vault-bootstrap-token contains a high-privilege token. It lives only in the cluster. In production, replace the root token with a dedicated admin token that has only sys/ write capabilities.

VAULT_CACERT required on every exec

Main Vault has TLS enabled. Every kubectl exec command must pass env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt or the CLI returns x509: certificate signed by unknown authority.

Sync and verify¶

Push the new manifest files and sync the vault Application. The PostSync Job runs automatically after sync completes:

# Push (from Mac)
git add applications/cluster-addons/vault/manifests/
git commit -m "feat: vault bootstrap job (idempotent PostSync)"
git push

# Sync (from bastion)
argocd app sync vault --assumeYes

# Watch the bootstrap job
kubectl get jobs -n vault -w
kubectl logs -n vault job/vault-bootstrap -f

Expected log output:

=== Vault bootstrap starting ===
KV v2 enabled at secret/          (or: already enabled — skipping)
Kubernetes auth enabled            (or: already enabled — skipping)
Kubernetes auth configured
eso-read policy written
eso role written
=== Vault bootstrap complete ===

Verify after job completes:

MAIN_ROOT=$(python3 -c "import json; d=json.load(open('${HOME}/hashicorp_vault/vault-main-init-mod-auh1-dev.json')); print(d['root_token'])")

kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault secrets list
# Expected: cubbyhole/, identity/, secret/, sys/

kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault auth list
# Expected: kubernetes/, token/

Enable audit logging (one-time, manual)¶

Audit logging is enabled via CLI — not included in the bootstrap Job because the audit log path is environment-specific:

kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault audit enable file file_path=/vault/data/audit.log

Adding per-app policies (pattern)¶

For each application that needs secrets from Vault, add its policy to bootstrap.sh in the ConfigMap. The Job re-runs on the next argocd app sync vault and applies the new policy idempotently:

# Example: add a new app policy to vault-bootstrap-config.yaml
vault policy write myapp-read - << 'POLICY'
path "secret/data/myapp/*" { capabilities = ["read"] }
path "secret/metadata/myapp/*" { capabilities = ["read", "list"] }
POLICY

vault write auth/kubernetes/role/myapp \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=myapp \
  policies=myapp-read \
  ttl=1h

Step 7.10b — Verify Vault Access¶

Before deploying ESO, confirm Main Vault is fully operational — CLI, UI, and a basic KV read/write. This is the go/no-go gate before handing off to ESO.

CLI access (kubectl exec)¶

MAIN_ROOT=$(python3 -c "import json; d=json.load(open('${HOME}/hashicorp_vault/vault-main-init-mod-auh1-dev.json')); print(d['root_token'])")

# Status check
kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  vault status
# Expected: Initialized=true, Sealed=false, Storage Type=file

# Auth check
kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault token lookup
# Expected: policies=[default root], display_name=root

KV read/write smoke test¶

# Write a test secret
kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault kv put secret/test/smoke key=hello

# Read it back
kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault kv get secret/test/smoke
# Expected: key = hello

# Clean up
kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault kv delete secret/test/smoke

UI access (ingress)¶

Open in browser: https://vault.mod.auh1.dev.dir

Login method: Token
Token: value of ${MAIN_ROOT} from init file

Expected: Vault UI loads, secret/ KV v2 mount visible in left sidebar.

Certificate trust

The UI cert is issued by internal-ca-issuer. If your browser shows an untrusted cert warning, import the internal CA into your OS trust store or accept the exception.

Verify Kubernetes auth is wired up¶

kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault auth list
# Expected: kubernetes/ entry present

kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault read auth/kubernetes/role/eso
# Expected: bound_service_account_names=[external-secrets], policies=[eso-read]

✅ All checks pass → proceed to Step 7.11 (ESO).

Step 7.11 — Deploy ESO via ArgoCD¶

Both files are already committed in mod-ask-devops:

applications/cluster-addons/external-secrets/
└── values-mod-auh1-dev-ask.yaml

environment/auh1/dev/02-argocd-application/cluster-addons/
└── external-secrets.yaml

Pre-step — Verify image exists in Harbor¶

ESO uses the ghcr/external-secrets/external-secrets project in Harbor. Verify the image is present before pushing:

# From bastion
curl -sk -u 'robot$ask:<password>' \
  "https://harbor.cl1.sq4.aegis.internal/api/v2.0/projects/ghcr/repositories/external-secrets%2Fexternal-secrets/artifacts?page_size=10" \
  | python3 -m json.tool | grep '"name"'
# Expected: "v0.14.4" tag visible

If the image is missing, mirror it first:

cd /Users/tarun.mittal/Documents/096_RKE/scripts/harbor-mirror
python3 core42-mirror-image.py --image ghcr.io/external-secrets/external-secrets:v0.14.4

Pre-step — Namespace creation and PSA labeling (fully automated via ArgoCD)¶

No kubectl create namespace or kubectl label commands needed. The ArgoCD Application uses managedNamespaceMetadata to create the namespace and apply PSA labels in the same sync:

syncPolicy:
  syncOptions:
    - CreateNamespace=true
  managedNamespaceMetadata:
    labels:
      pod-security.kubernetes.io/enforce: baseline
      pod-security.kubernetes.io/warn: baseline

This applies to all three applications: external-secrets, vault, and vault-root. Adding a new app namespace with baseline PSA is the same pattern — add managedNamespaceMetadata to its ArgoCD Application yaml and push.

Pre-step — Bootstrap harbor-pull-secret (one-time only for external-secrets namespace)¶

The external-secrets namespace is a chicken-and-egg exception: ESO must pull its own image from Harbor to start, but ESO is what manages harbor-pull-secret via the ClusterExternalSecret going forward. A one-time manual copy is required for this namespace only. After Step 7.11b is complete, ESO maintains it automatically.

# Bastion — run once before pushing
kubectl get secret harbor-pull-secret -n argocd -o json \
  | python3 -c "import sys,json; s=json.load(sys.stdin); s['metadata']={'name':s['metadata']['name']}; print(json.dumps(s))" \
  | kubectl apply -n external-secrets -f -

kubectl get secret harbor-pull-secret -n external-secrets
# Expected: secret/harbor-pull-secret

vault and vault-root do NOT need this manual step

Those namespaces already have harbor-pull-secret from initial deployment. After Step 7.11b completes, ESO takes ownership of those secrets too — future rotations only require updating Vault, not touching any namespace manually.

Push and sync¶

# Push from Mac
cd /Users/tarun.mittal/Documents/modgpt/mod-ask-devops
git add applications/cluster-addons/external-secrets/ \
        environment/auh1/dev/02-argocd-application/cluster-addons/external-secrets.yaml
git commit -m "feat: deploy ESO v0.14.4 via ArgoCD"
git push

# ArgoCD auto-syncs (automated syncPolicy) — or trigger manually:
argocd app sync external-secrets --assumeYes

# Verify pods
kubectl get pods -n external-secrets -w
# Expected: 3 pods Running — external-secrets (controller), webhook, cert-controller

Verify ESO is healthy¶

kubectl get pods -n external-secrets
# NAME                                                READY   STATUS    RESTARTS
# external-secrets-<hash>                             1/1     Running   0
# external-secrets-webhook-<hash>                     1/1     Running   0
# external-secrets-cert-controller-<hash>             1/1     Running   0

kubectl get crds | grep external-secrets.io
# Expected: clustersecretstores, externalsecrets, secretstores, etc.

Step 7.11b — Harbor Pull Secret via ESO (ClusterExternalSecret)¶

Production-correct pattern

Instead of manually copying harbor-pull-secret to every namespace, store Harbor robot account credentials in Vault once and let ESO create the secret automatically in every namespace that needs it via a ClusterExternalSecret. Adding a new app namespace only requires adding the namespace to the selector list — no manual copy.

Why a ClusterExternalSecret?¶

Manual copy	ClusterExternalSecret (label-based)
`kubectl get secret ... \\| kubectl apply -n <ns>` per namespace	Add `harbor-pull-secret: "true"` label to namespace → ESO creates it automatically
Edit an explicit namespace list for each new app	No list — label the namespace, ESO picks it up
Credentials duplicated in every namespace	Single source of truth in Vault
Rotation: re-copy to every namespace	Rotation: `vault kv put secret/harbor/pull-secret password=<new>` → ESO propagates within `refreshInterval`
Drift: namespace might have stale credentials	ESO reconciles continuously

Chicken-and-egg bootstrap note¶

external-secrets namespace is the one exception: it needs harbor-pull-secret to pull the ESO image before ESO runs. The manual pre-step in Step 7.11 covers this once. After ESO is running, the ClusterExternalSecret manages it going forward (including after namespace deletion/recreation).

Step 1 — Store Harbor credentials in Vault¶

Harbor robot account credentials are stored in ~/harbor-images/harbor-robot.json on bastion1. Read directly from the file — token never appears in terminal history.

# Run on bastion1 from ~/harbor-images/
cd ~/harbor-images

MAIN_ROOT=$(python3 -c "import json; d=json.load(open('${HOME}/hashicorp_vault/vault-main-init-mod-auh1-dev.json')); print(d['root_token'])")

HARBOR_USER=$(python3 -c "import json; d=json.load(open('harbor-robot.json')); print(d.get('name', d.get('username')))")
HARBOR_PASS=$(python3 -c "import json; d=json.load(open('harbor-robot.json')); print(d.get('secret', d.get('token', d.get('password'))))")

kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault kv put secret/harbor/pull-secret \
    username="${HARBOR_USER}" \
    password="${HARBOR_PASS}"

Verify (username only — do not print password):

kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault kv get -field=username secret/harbor/pull-secret
# Expected: robot$ask

Init file is on bastion1, not bastion2

vault-main-init-mod-auh1-dev.json is at ~/hashicorp_vault/ on bastion1. If running from bastion2, either SSH to bastion1 first or fetch the root token via: MAIN_ROOT=$(ssh cloud-user@bastion1 "python3 -c \"import json; print(json.load(open('${HOME}/hashicorp_vault/vault-main-init-mod-auh1-dev.json'))['root_token'])\"")

Credential rotation

When you rotate the Harbor robot account token, update harbor-robot.json on bastion1 then re-run the HARBOR_PASS + vault kv put commands above. ESO propagates the new secret to all namespaces within refreshInterval: 1h — no kubectl commands needed.

Step 2 — Update bootstrap policy¶

The ESO bootstrap policy already allows secret/data/apps/* and secret/data/argocd/*. Harbor credentials live at secret/harbor/* — add this path to the eso-read policy in applications/cluster-addons/vault/manifests/vault-bootstrap-config.yaml:

The updated policy section (already applied in the committed ConfigMap):

path "secret/data/harbor/*"    { capabilities = ["read"] }

After committing vault-bootstrap-config.yaml, sync the vault app to re-run the bootstrap Job.

Step 3 — ClusterExternalSecret (GitOps)¶

File: applications/cluster-addons/external-secrets/manifests/harbor-pull-secret-ces.yaml

This manifest is included in the ESO ArgoCD Application via a third source. It is applied in the same sync as the ESO Helm chart — the ClusterExternalSecret CRD comes from the chart, so the manifest applies cleanly in the same wave.

The secret will show Not Ready until Step 7.12 (ClusterSecretStore) is created. Once vault-backend ClusterSecretStore is READY, ESO begins syncing immediately.

Any namespace labeled harbor-pull-secret: "true" automatically gets the secret. No list to maintain — just add the label to the ArgoCD Application's managedNamespaceMetadata.

# After pushing (Step 7.11 push includes this file):
kubectl get clustersecretstore vault-backend
# Wait for READY=True (after Step 7.12)

kubectl get externalsecret -n vault
kubectl get externalsecret -n vault-root
kubectl get externalsecret -n external-secrets
# Expected: NAME=harbor-pull-secret, STATUS=SecretSynced, READY=True

# Verify secret was created
kubectl get secret harbor-pull-secret -n vault
kubectl get secret harbor-pull-secret -n vault-root

Step 7.12 — ClusterSecretStore (GitOps, sync-wave 1)¶

The ClusterSecretStore is managed as a GitOps manifest at applications/cluster-addons/external-secrets/manifests/vault-backend-css.yaml. It is deployed automatically by the external-secrets ArgoCD Application in sync-wave 1, after the ESO CRDs are installed (wave 0 / default) and before the ClusterExternalSecret (wave 2).

Sync wave ordering within the external-secrets Application:

Wave	Resource	Depends on
0 (default)	ESO Helm chart — CRDs, Deployments, RBAC	nothing
1	`ClusterSecretStore/vault-backend`	ESO CRDs
2	`ClusterExternalSecret/harbor-pull-secret`	ClusterSecretStore

No kubectl apply needed — push the manifest and ArgoCD syncs it.

Verify after sync:

kubectl get clustersecretstore vault-backend
# Expected: READY=True

kubectl get clustersecretstore vault-backend -o jsonpath='{.status.conditions[0].message}'
# Expected: Vault health check succeeded (or similar)

Why vault.vault.svc.cluster.local not vault.mod.auh1.dev.dir?

ESO connects to Vault from inside the cluster. Using the internal service FQDN avoids routing through the ingress, HAProxy, and external DNS — faster and more reliable. The CA cert comes directly from the vault-tls secret issued by cert-manager.

ClusterSecretStore will show Not Ready until Vault Kubernetes auth is configured

The eso-read policy and eso Kubernetes auth role are created by the Vault bootstrap Job (PostSync on the vault Application). If ESO syncs before the vault bootstrap Job has run, the store will show Not Ready temporarily and recover automatically once the Job completes.

Step 7.12b — End-to-End Test (verified 2026-06-11)¶

Run after every fresh deployment or ESO upgrade to confirm the full chain is healthy.

# 1. ClusterSecretStore — must be READY=True before anything else works
kubectl get clustersecretstore vault-backend
# NAME            AGE     STATUS   CAPABILITIES   READY
# vault-backend   3m33s   Valid    ReadWrite      True

# 2. ClusterExternalSecret — must be READY=True, store=vault-backend
kubectl get clusterexternalsecret harbor-pull-secret
# NAME                 STORE           REFRESH INTERVAL   READY
# harbor-pull-secret   vault-backend   1h                 True

# 3. harbor-pull-secret present in all target namespaces
kubectl get secret harbor-pull-secret -n vault
kubectl get secret harbor-pull-secret -n vault-root
kubectl get secret harbor-pull-secret -n external-secrets
# Expected: all 3 — TYPE=kubernetes.io/dockerconfigjson, DATA=1

# 4. Confirm secret content has correct registry and username
kubectl get secret harbor-pull-secret -n vault \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | python3 -m json.tool
# Expected:
# {
#   "auths": {
#     "harbor.cl1.sq4.aegis.internal": {
#       "username": "robot$ask",
#       "password": "...",
#       "auth": "..."
#     }
#   }
# }

All checks passed (2026-06-11):

Check	Result
`ClusterSecretStore/vault-backend`	READY=True, Valid, ReadWrite
`ClusterExternalSecret/harbor-pull-secret`	READY=True, store=vault-backend
`harbor-pull-secret` in `vault`	✅ kubernetes.io/dockerconfigjson
`harbor-pull-secret` in `vault-root`	✅ kubernetes.io/dockerconfigjson
`harbor-pull-secret` in `external-secrets`	✅ kubernetes.io/dockerconfigjson

How to test label-based automatic distribution (verified 2026-06-11)¶

Create a temporary namespace with the label and watch ESO react:

kubectl create namespace eso-test
kubectl label namespace eso-test harbor-pull-secret=true

kubectl get secret -n eso-test -w
# Expected: harbor-pull-secret appears within seconds — no manifest, no kubectl copy
# NAME                 TYPE                             DATA   AGE
# harbor-pull-secret   kubernetes.io/dockerconfigjson   1      4s

# Clean up
kubectl delete namespace eso-test

Verified result (2026-06-11): secret appeared in 4 seconds after label was applied.

Onboarding any new app namespace to Harbor:

# In the ArgoCD Application yaml — nothing else needed
syncPolicy:
  managedNamespaceMetadata:
    labels:
      harbor-pull-secret: "true"

ArgoCD creates the namespace with the label → ESO creates harbor-pull-secret within seconds. No list to maintain, no manual copy steps.

How to test credential rotation¶

# On bastion1 — update Vault with new token (reads from harbor-robot.json after rotation)
cd ~/harbor-images
HARBOR_PASS=$(python3 -c "import json; d=json.load(open('harbor-robot.json')); print(d.get('secret', d.get('token', d.get('password'))))")

kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault kv put secret/harbor/pull-secret \
    username='robot$ask' \
    password="${HARBOR_PASS}"

# ESO propagates within refreshInterval (1h), or force immediate refresh:
kubectl annotate clusterexternalsecret harbor-pull-secret \
  force-sync=$(date +%s) --overwrite

# Verify new token in one namespace
kubectl get secret harbor-pull-secret -n vault \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | python3 -m json.tool

Step 7.13 — ArgoCD Repo Credential (intentionally NOT in ESO/Vault)¶

Design decision: ArgoCD repo credentials must NOT be Vault/ESO-managed

The circular dependency problem:

Vault sealed
  → ESO cannot read secret/argocd/gitlab-creds
    → ArgoCD repo credential missing or stale
      → ArgoCD cannot pull GitOps manifests from GitLab
        → Cannot sync the Vault fix via ArgoCD
          → DEADLOCK — platform unrecoverable without manual intervention

ArgoCD is the bootstrap layer — it deploys and recovers everything else including Vault. The dependency chain must flow strictly downward:

ArgoCD  (bootstrap — must survive Vault outage)
   ↓ deploys
Vault + ESO  (secrets layer)
   ↓ provides secrets to
Applications  (app layer)

ArgoCD cannot sit below the thing it is responsible for recovering.

What to put in Vault/ESO and what not to:

Secret	Where	Reason
ArgoCD GitLab repo credential	Static K8s Secret (manual, not in Git)	Bootstrap layer — must survive Vault outage
`harbor-pull-secret`	ESO → Vault	Infrastructure secret, non-critical if briefly stale
App database passwords, API keys	ESO → Vault	Application layer — correct home for Vault
Vault transit token	Static K8s Secret	Bootstrap exception — Vault needs this to unseal

ArgoCD repo credential stays as a static K8s Secret. It is:

Created once manually (not in Git — it contains a live PAT token)
Rotated manually: kubectl patch secret -n argocd <name> --type=merge -p '{"stringData":{"password":"<new-token>"}}'
Never deleted — losing it locks ArgoCD out of GitLab

To find and verify the current credential:

kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repo-creds
# Shows the current static repo credential — verify it exists before any maintenance

# Verify ArgoCD can still reach GitLab
argocd repo list --grpc-web
# Expected: https://gitlab.cl1.sq4.aegis.internal/  STATUS=Successful

Summary Checklist¶

Pre-flight - [x] vault-root and vault namespaces labeled pod-security.kubernetes.io/enforce=baseline - [x] harbor-pull-secret copied to vault-root namespace - [x] harbor-pull-secret copied to vault namespace

Root Vault - [x] vault chart pushed to GitLab Helm registry (0.32.0) - [x] hashicorp/vault:1.21.2 image mirrored to Harbor - [x] Root Vault deployed — vault-root-0 Running - [x] Root Vault initialized — ~/hashicorp_vault/vault-root-init-mod-auh1-dev.json saved securely - [x] Root Vault unsealed (manually, first time only) — Sealed: false - [x] vault-root-unseal-keys K8s Secret created (3 key shares, never in Git) - [x] DIY unsealer pod Running — verified auto-recovery in 16s (tested 2026-06-11) - [x] Transit engine enabled, autounseal key created - [x] autounseal-policy created (encrypt/decrypt only) - [x] Transit token created (periodic, 87600h) - [x] Audit logging enabled on Root Vault

Main Vault - [x] vault-transit-token K8s Secret created in vault namespace - [x] Main Vault deployed standalone — vault-0 1/1 Running - [x] Main Vault initialized — ~/hashicorp_vault/vault-main-init-mod-auh1-dev.json saved securely - [x] vault status shows Initialized=true, Sealed=false, Storage Type=file - [x] UI accessible at https://vault.mod.auh1.dev.dir (fix: backend-protocol: HTTPS annotation required) - [x] KV v2 enabled at secret/ - [x] Kubernetes auth configured (kubernetes_host=https://kubernetes.default.svc) - [x] eso-read policy created (note: requires -i flag on kubectl exec for heredoc stdin) - [x] eso Kubernetes auth role created - [x] vault-bootstrap-token K8s Secret created (not in Git) - [x] Bootstrap Job + ConfigMap committed to applications/cluster-addons/vault/manifests/ - [ ] argocd/gitlab-creds stored in Vault - [x] Audit logging enabled on Main Vault (vault audit enable file file_path=/vault/data/audit.log)

ESO - [ ] external-secrets chart pushed to GitLab Helm registry (0.14.4) - [x] ghcr/external-secrets/external-secrets:v0.14.4 image present in Harbor (verified 2026-06-11) - [ ] harbor-pull-secret manually copied to external-secrets namespace (one-time bootstrap) - [ ] external-secrets namespace labeled pod-security.kubernetes.io/enforce=baseline - [x] ESO values file committed — applications/cluster-addons/external-secrets/values-mod-auh1-dev-ask.yaml - [x] ESO ArgoCD Application committed — environment/auh1/dev/02-argocd-application/cluster-addons/external-secrets.yaml - [ ] ESO deployed — 3 pods Running (controller, webhook, cert-controller)

Harbor Pull Secret via ESO - [x] Harbor robot credentials stored in Vault at secret/harbor/pull-secret (read from ~/harbor-images/harbor-robot.json on bastion1, version 3) - [x] Bootstrap policy updated — secret/data/harbor/* added to eso-read - [x] ClusterExternalSecret committed — applications/cluster-addons/external-secrets/manifests/harbor-pull-secret-ces.yaml - [x] managedNamespaceMetadata added to vault, vault-root, external-secrets ArgoCD Applications — no manual kubectl label needed - [x] ClusterSecretStore committed — applications/cluster-addons/external-secrets/manifests/vault-backend-css.yaml (sync-wave 1) - [ ] ClusterSecretStore/vault-backend — READY=True (after ArgoCD sync) - [ ] harbor-pull-secret synced to vault, vault-root, external-secrets namespaces via ESO

ArgoCD Repo Credential - [x] Intentionally kept as static K8s Secret — circular dependency: ArgoCD cannot depend on Vault (which it deploys and recovers)

Troubleshooting¶

Pod Scheduling and Startup¶

Symptom	Cause	Fix
`vault-root-0` or `vault-N` → `Pending` with `PSA policy violation`	PSA `restricted` blocks `IPC_LOCK` at admission	Label namespaces `baseline`: `kubectl label namespace vault pod-security.kubernetes.io/enforce=baseline --overwrite` (same for `vault-root`)
`vault-root-0` stuck `0/1 Running` (readiness probe failing, not initialized)	Sealed — expected state after first deploy	Run Step 7.6 unseal commands
Main Vault pods stuck `0/1 Running` after sync	`vault-transit-token` secret missing in `vault` namespace	Run Step 7.7 `kubectl create secret` command
`ImagePullBackOff` on vault pods	`harbor-pull-secret` not in vault namespace	Run Pre-step harbor-pull-secret copy commands; ArgoCD cannot propagate secrets
Node drain blocked — pod vault-0 cannot be evicted	A PDB with `minAvailable > 0` was applied on standalone (1 pod) — always violated	Delete any PDB in the `vault` namespace: `kubectl delete pdb -n vault --all`

Configuration Errors¶

Symptom	Cause	Fix
`service_registration is configured, but storage does not support HA` — pod CrashLoopBackOff	`service_registration "kubernetes" {}` present in HCL but `storage "file"` does not support HA — only valid with Raft/Consul	Remove `service_registration "kubernetes" {}` from standalone config
Main Vault pods `CrashLoopBackOff` with `HCL parse error: Unknown token IDENT env`	`env()` function used in seal stanza HCL — not supported	Remove `token = env(...)` from seal stanza; use `extraSecretEnvironmentVars` with `envName: VAULT_TOKEN`
Main Vault pods `CrashLoopBackOff` with `403 permission denied` on Transit seal	`${VAULT_TRANSIT_TOKEN}` passed literally — vault-helm 0.32.0 does not envsubst	Remove `token =` from seal stanza entirely; Vault reads `VAULT_TOKEN` env var automatically
`VAULT_CACERT` and/or `VAULT_TOKEN` missing from pod env	Wrong key `extraEnv` or `server.extraEnv` used (silently ignored in chart 0.32.0)	Use `server.extraEnvironmentVars` (dict) for plain values; `server.extraSecretEnvironmentVars` (list) for K8s Secret values
`x509: certificate signed by unknown authority` in `kubectl exec` commands	VAULT_CACERT not set in exec environment	Either `extraEnvironmentVars.VAULT_CACERT` is missing from values, or pass `env VAULT_CACERT=<path>` explicitly in the exec command
`audit {}` block causes HCL parse error	Audit devices cannot be configured via HCL config file — CLI only	Remove `audit {}` block from HCL config; enable via `vault audit enable file file_path=/vault/logs/audit.log` post-init
`path is already in use` when running `vault secrets enable transit`	Transit was already enabled in a previous run	Skip — verify: `kubectl exec -n vault-root vault-root-0 -- env VAULT_TOKEN=<tok> vault secrets list`
`policy parameter not supplied or empty` on `vault policy write`	`kubectl exec` without `-i` flag — heredoc stdin not passed into container	Add `-i` flag: `kubectl exec -i -n vault vault-0 -- vault policy write eso-read - << 'POLICY'`

Root Vault Auto-Recovery (verified)¶

Symptom	Cause	Fix
`storage.raft.fsm: raft FSM db file has wider permissions than needed: needed=-rw------- existing=-rw-rw----`	Raft data file on PVC has group-write bit set — Vault warns but continues normally	Harmless in this context — PVC is pod-exclusive. No action needed.

Verified recovery sequence (tested 2026-06-11):

vault-root-0 deleted
  → pod restarted (sealed, exit code 2)
  → DIY unsealer detected within 15s
  → sent 3 unseal keys
  → Root Vault active at 16s
  → Transit engine remounted automatically
  → Main Vault unaffected (already holds unseal key in memory)

Initialization and Unsealing¶

Symptom	Cause	Fix
Init file is 0 bytes after `kubectl exec ... > file`	Shell `>` creates/truncates the file BEFORE the command runs. If `kubectl exec` fails (pod not ready, TLS error), stdout is empty → file stays empty. Second attempt finds vault already initialized → error to stderr → file empty again	Always use `tee`: `kubectl exec ... vault operator init -format=json \\| tee file.json`. Output goes to both terminal and file; if command fails, you see the error instead of a silent empty file

Symptom	Cause	Fix
`PermissionError: /root/vault-root-init-mod-auh1-dev.json`	Running as `cloud-user`; `/root/` is not writable	Use `~/hashicorp_vault/` (cloud-user home). Always write init files to `${HOME}/hashicorp_vault/`.
Main Vault pods log `stored unseal keys are supported, but none were found`	Expected pre-init state — vault awaiting first `vault operator init`	Run Step 7.9 initialization. This message is normal for sealed-but-not-initialized pods.
Root Vault pod restarted — Main Vault new pods cannot unseal	Root Vault sealed after restart; DIY unsealer recovers automatically within ~15s	Wait for unsealer loop to detect and unseal. If unseal keys secret missing, re-create it (Step 7.6b). Already-running Main Vault pods are unaffected.
Unsealer logs `Root Vault sealed but unseal keys not yet available — waiting`	`vault-root-unseal-keys` Secret not yet created	Run Step 7.6b to create the secret. The unsealer hot-reloads volume within ~60s — no pod restart needed.

Raft and Cluster¶

Symptom	Cause	Fix
Raft peers not joining after init	TLS cert missing or `ca.crt` SAN mismatch	`kubectl get secret vault-tls -n vault -o jsonpath='{.data}'` — verify `ca.crt`, `tls.crt`, `tls.key` all present; check SANs include `*.vault-internal`
Only 1 of 5 pods shows `1/1 Running` after init	Remaining pods haven't joined Raft yet	Wait 30–60s; check logs: `kubectl logs -n vault vault-1` for retry_join activity

UI / Ingress¶

Symptom	Cause	Fix
`HTTP ERROR 400` on `https://vault.mod.auh1.dev.dir`	nginx terminates TLS and forwards plain HTTP to Vault, but Vault listener has `tls_disable = 0` — it expects HTTPS	Add `nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"` to ingress annotations in values file

ESO¶

Symptom	Cause	Fix
`ClusterSecretStore READY=False`	K8s auth misconfigured	`vault read auth/kubernetes/config` — `kubernetes_host` must be `https://kubernetes.default.svc`
`ExternalSecret SecretSyncedError`	ESO SA not bound to Vault role	Verify `bound_service_account_names=external-secrets` and `bound_service_account_namespaces=external-secrets` in `auth/kubernetes/role/eso`