Main Vault — HA Cluster with Transit Auto-Unseal

Prerequisites

• Root Vault deployed and Transit token saved — Vault Root
• Vault chart and images mirrored to GitLab — Vault Root → Phase 2
• ArgoCD App of Apps root application created — App of Apps → Root Application
• vault namespace and TLS cert exist (from cert-manager)

---

What is Main Vault?

Main Vault is the production-facing HA cluster that applications use for secrets management. It runs as a 3-node Raft cluster and uses Transit auto-unseal — the Root Vault decrypts the unseal key on every pod restart automatically. No human intervention required.

Root Vault (standalone, vault-root namespace)
  └── Transit engine
        └── Auto-unseals → Main Vault (3-node Raft, vault namespace)
                                └── Applications get secrets from here

	Root Vault	Main Vault
Purpose	Auto-unseal only	All application secrets
Replicas	1 (standalone)	3 (HA Raft)
TLS	Disabled (tls_disable=1)	Terminated at ingress
Ingress	None	vault.cl1.sq4.aegis.internal
Managed by	Helm (manual bootstrap)	ArgoCD

---

Architecture

flowchart TD
    argocd["ArgoCD\n(reads apps/vault.yaml)"]
    gitlab_chart["GitLab Helm registry\n.../packages/helm/stable · chart: vault"]
    gitlab_git["GitLab Git\ncluster-addons/values/vault/vault-stg.yaml"]

    vault0["vault-0\n(Raft leader)"]
    vault1["vault-1\n(Raft follower)"]
    vault2["vault-2\n(Raft follower)"]

    root["Root Vault\nvault-root namespace\n(Transit auto-unseal)"]
    ceph["Ceph Block\n10Gi per pod"]
    ingress["NGINX Ingress\nvault.cl1.sq4.aegis.internal"]

    argocd -->|"pulls chart"| gitlab_chart
    argocd -->|"reads values"| gitlab_git
    argocd -->|"deploys"| vault0 & vault1 & vault2
    vault0 & vault1 & vault2 -->|"auto-unseal"| root
    vault0 & vault1 & vault2 -->|"stores data"| ceph
    ingress -->|"routes HTTPS"| vault0

---

Phase 1 — Prerequisites Check

# Verify vault namespace exists
kubectl get namespace vault

# Verify TLS cert is issued and ready
kubectl get certificate -n vault vault-tls
# Expected: READY = True

kubectl get secret -n vault vault-tls
# Expected: TYPE = kubernetes.io/tls

# Verify Root Vault is unsealed and running
kubectl exec -n vault-root vault-root-0 -- vault status | grep -E "Sealed|HA Mode"
# Expected: Sealed: false

# Verify transit token is saved
cat ~/vault-transit-token.txt
# Expected: hvs.XXXXXXXXXXXX

If any check fails — fix it before continuing.

---

Phase 2 — Create Values File

mkdir -p ~/cluster-addons/values/vault

cat > ~/cluster-addons/values/vault/vault-stg.yaml << 'EOF'
server:
  image:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault
    tag: "1.21.2"

  ha:
    enabled: true
    replicas: 3    # This deployment: 3 nodes = 3 replicas. Production: use 5 — see note below.
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address     = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-2.vault-internal:8200"
          }
        }

        seal "transit" {
          address         = "http://vault-root.vault-root.svc.cluster.local:8200"
          token           = "REPLACE_WITH_TRANSIT_TOKEN"
          key_name        = "vault-unseal"
          mount_path      = "transit/"
        }

        service_registration "kubernetes" {}

  dataStorage:
    enabled: true
    size: 10Gi
    storageClass: ceph-block

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: vault
              component: server
          topologyKey: kubernetes.io/hostname

  podDisruptionBudget:
    maxUnavailable: 1

  # OnDelete prevents ArgoCD from auto-rolling Vault pods on every sync.
  # Vault HA requires manual coordination during upgrades to maintain Raft quorum.
  # To upgrade: delete one pod at a time, wait for it to re-join before deleting the next.
  updateStrategyType: OnDelete

  # preStop sleep gives Ceph CSI time to register the RBD detach operation before
  # the pod is killed — prevents "operation with given Volume ID already exists" on restart.
  lifecycleHooks:
    preStop:
      exec:
        command:
          - sh
          - -c
          - sleep 10

  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    hosts:
      - host: vault.cl1.sq4.aegis.internal
        paths: []
    tls:
      - secretName: vault-tls
        hosts:
          - vault.cl1.sq4.aegis.internal

injector:
  enabled: true
  image:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault-k8s
    tag: "1.7.2"
  agentImage:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault
    tag: "1.21.2"

ui:
  enabled: true
EOF

Step 2.2 — Substitute the Transit Token

TRANSIT_TOKEN=$(cat ~/vault-transit-token.txt)
sed -i "s/REPLACE_WITH_TRANSIT_TOKEN/${TRANSIT_TOKEN}/" \
  ~/cluster-addons/values/vault/vault-stg.yaml

# Verify token was substituted
grep "token" ~/cluster-addons/values/vault/vault-stg.yaml
# Expected: token = "hvs.XXXXXXXXXXXX" (not REPLACE_WITH_TRANSIT_TOKEN)

Production Raft cluster sizing — use 5 replicas

Raft quorum requires a majority of nodes (n/2 + 1). This deployment runs 3 replicas to match 3 worker nodes, but production should run 5 replicas:

Replicas	Quorum needed	Can lose	Recommendation
3	2/3	1 node	✅ Non-production
5	3/5	2 nodes	✅✅ Production

To scale to 5 in production:

1. Add 2 more worker nodes to the cluster
2. Change replicas: 3 → replicas: 5
3. Add 2 more retry_join blocks:

   retry_join {
     leader_api_addr = "http://vault-3.vault-internal:8200"
   }
   retry_join {
     leader_api_addr = "http://vault-4.vault-internal:8200"
   }
   

4. Update podDisruptionBudget: maxUnavailable: 1 → maxUnavailable: 2
5. Keep required anti-affinity — each pod on its own node for true fault isolation

---

Phase 3 — Create the ArgoCD Application YAML

mkdir -p ~/cluster-addons/apps/vault
cat > ~/cluster-addons/apps/vault/vault.yaml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.io
spec:
  project: default
  sources:
    # Source 1: Helm chart from GitLab Helm HTTP registry
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/1/packages/helm/stable'
      chart: vault
      targetRevision: 0.32.0
      helm:
        valueFiles:
          - $values/values/vault/vault-stg.yaml
    # Source 2: Values file from Git repo (internal service URL)
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git'
      targetRevision: HEAD
      ref: values
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: vault
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
  # Ignore caBundle — the vault-agent-injector injects its own TLS cert into
  # the MutatingWebhookConfiguration at runtime. Without this, ArgoCD marks
  # vault-agent-injector-cfg as OutOfSync on every sync.
  ignoreDifferences:
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      name: vault-agent-injector-cfg
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
EOF

Why vault-agent-injector-cfg shows OutOfSync

The vault-agent-injector-cfg MutatingWebhookConfiguration is self-mutating — the injector pod writes its own TLS caBundle into the resource after it starts. ArgoCD detects this runtime change as a diff against the Helm template.

The ignoreDifferences block tells ArgoCD to ignore the caBundle field specifically — everything else is still tracked and enforced.

---

Phase 4 — Commit and Push

cd ~/cluster-addons
git add apps/vault/ values/vault/vault-stg.yaml
git commit -m "feat: add main Vault HA cluster"
git push origin main

ArgoCD detects the new file within ~3 minutes and creates the vault Application automatically:

argocd app list --grpc-web
kubectl get pods -n vault -w

Expected pods (all 0/1 until initialized):

NAME                                    READY   STATUS
vault-0                                 0/1     Running
vault-1                                 0/1     Running
vault-2                                 0/1     Running
vault-agent-injector-xxxxxxxxx-xxxxx    1/1     Running

---

Phase 5 — Initialize Main Vault

Main Vault needs to be initialized once. Because it uses Transit auto-unseal, there are no Shamir unseal keys — only recovery keys for emergency use.

# vault.cl1.sq4.aegis.internal resolves via BIND DNS to the ingress VIP (100.115.2.100) — no hosts entry needed

# Initialize — recovery keys are for emergency root token regeneration only
kubectl exec -n vault vault-0 -- vault operator init \
  -recovery-shares=5 \
  -recovery-threshold=3 \
  -format=json > ~/vault-main-init.json

cat ~/vault-main-init.json

Expected output:

{
  "unseal_keys_b64": [],           ← empty — Transit handles unsealing
  "recovery_keys_b64": ["...", "...", "...", "...", "..."],
  "root_token": "hvs.XXXXXXXXXXXX"
}

Save vault-main-init.json immediately

This file contains the root token and recovery keys. Copy it off the bastion immediately:

# On your admin workstation (not the bastion)
scp cloud-user@bastion1:~/vault-main-init.json ./vault-main-init.json

Recovery keys are needed if Root Vault is ever lost and you need to regenerate a root token for Main Vault.

---

Phase 6 — Verify All Pods Auto-Unsealed

After initialization, Root Vault Transit decrypts the unseal key and all pods come up automatically:

kubectl get pods -n vault

Expected — all 1/1 Running:

NAME                                    READY   STATUS
vault-0                                 1/1     Running   ← auto-unsealed
vault-1                                 1/1     Running   ← auto-unsealed
vault-2                                 1/1     Running   ← auto-unsealed
vault-agent-injector-xxxxxxxxx-xxxxx    1/1     Running

Check Raft cluster health:

kubectl exec -n vault vault-0 -- vault status

Expected:

Sealed:     false
HA Mode:    active
HA Cluster: https://vault-0.vault-internal:8201

Check all peers joined:

kubectl exec -n vault vault-0 -- sh -c \
  "VAULT_TOKEN=\$(cat ~/vault-main-init.json | python3 -c \"import sys,json; print(json.load(sys.stdin)['root_token'])\") vault operator raft list-peers"

Expected:

Node       Address                        State       Voter
----       -------                        -----       -----
vault-0    vault-0.vault-internal:8201    leader      true
vault-1    vault-1.vault-internal:8201    follower    true
vault-2    vault-2.vault-internal:8201    follower    true

Access the UI at https://vault.cl1.sq4.aegis.internal — login with the root_token from ~/vault-main-init.json.

---

Day-2 — Manual Pod Upgrade (OnDelete Strategy)

Because updateStrategyType: OnDelete is set, ArgoCD will NOT automatically restart pods when the values file changes. After pushing a change, manually roll pods one at a time:

# Step 1: Push the change → ArgoCD updates the StatefulSet spec
git push origin main

# Step 2: Delete pods one at a time — wait for each to rejoin before the next
kubectl delete pod vault-0 -n vault
kubectl get pods -n vault -w   # wait for vault-0 → 1/1 Running

kubectl delete pod vault-1 -n vault
kubectl get pods -n vault -w   # wait for vault-1 → 1/1 Running

kubectl delete pod vault-2 -n vault
kubectl get pods -n vault -w   # wait for vault-2 → 1/1 Running

Never delete more than 1 pod at a time

3-node Raft requires 2/3 nodes for quorum. Deleting 2 pods simultaneously → quorum lost → cluster freezes.

---

Common Issues

Symptom	Cause	Fix
Pods stuck 0/1 after push	Vault not initialized yet	Run Phase 5 vault operator init
stored unseal keys are supported, but none were found	Expected on first boot — not initialized	Run vault operator init
Pods 0/1 after init	Root Vault sealed	Unseal Root Vault: kubectl exec -n vault-root vault-root-0 -- vault operator unseal ...
vault-2 stuck ContainerCreating	Ceph RBD stale CSI lock	Restart CSI node plugin on affected node — see Issue 22
vault-agent-injector-cfg OutOfSync	Self-mutating webhook (caBundle)	Already handled by ignoreDifferences in Application YAML
chart-example.local in ingress HOSTS	Wrong ingress key (hostname: instead of hosts:)	Use hosts: [{host: vault.cl1.sq4.aegis.internal, paths: []}]