Skip to content

KB-01 — Operating Raft / Quorum-Based StatefulSet Clusters

Applies to: any consensus-based clustered app deployed as a StatefulSet — Vault (integrated Raft storage), and upcoming ones like Hatchet, etcd, Consul, RabbitMQ (quorum queues), Patroni/PostgreSQL, NATS JetStream, Redis (cluster/Sentinel).

Why this KB exists: these apps are not stateless Deployments. Restarting them the "normal" way (delete all pods, let them roll) can break quorum and take the whole cluster down. The rules below are the same for every Raft/quorum app — learn them once, apply them everywhere.

1. The core idea: quorum

A Raft (or any majority-consensus) cluster stays available only while a strict majority of members are up and can talk to each other. That majority is the quorum.

Cluster size (N) Quorum (⌊N/2⌋+1) Failures tolerated Notes
1 1 0 No HA — any restart = downtime
3 2 1 Minimum for HA. Zero headroom during maintenance.
5 3 2 Production recommendation — survives a failure and a maintenance roll
7 4 3 Large clusters; more replication overhead

Always use an ODD number. Even sizes (4, 6) add a member without adding fault tolerance and increase split-brain risk. 4 tolerates only 1 failure — same as 3 — but costs more.

Why 3 is risky in production

With 3 nodes you tolerate exactly 1 failure. During a rolling upgrade you intentionally take one node down — now you are at 2/3, the quorum minimum, with no margin. If a second node hiccups (OOM, node drain, network blip) mid-roll, you lose quorum and the cluster goes read-only / unavailable. With 5 nodes you can roll one node and survive an unplanned failure at the same time. Production = 5.

2. StatefulSets use OnDelete — and why

Most Raft apps ship their Helm chart with:

spec:
  updateStrategy:
    type: OnDelete      # NOT RollingUpdate

What it means: when you change the pod spec (e.g. a new image), Kubernetes updates the StatefulSet template but does nothing to the running pods. Each pod keeps its old spec until you manually kubectl delete pod <name>, at which point it is recreated with the new spec.

Why charts choose it: the default RollingUpdate deletes/recreates pods automatically in reverse-ordinal order, often faster than the cluster can re-converge. For a consensus cluster that means:

  • A new pod may not have rejoined Raft / re-synced before the next one is taken down → cascading quorum loss.
  • The controller might restart the leader at an unsafe moment.

OnDelete hands the timing to a human (or a controlled script) who rolls one member at a time, verifying health between each.

Consequence for image migrations / upgrades

Changing the image in values and helm upgrade/ArgoCD sync updates the spec only. The live pods keep running the old image until you manually delete them one by one. A StatefulSet showing OnDelete will sit "OutOfSync but Healthy" indefinitely — that is expected, not a bug.

3. The safe rolling procedure (one member at a time) 
APP=vault NS=vault            # adapt per app

# 1. Identify the leader vs followers/standbys — roll FOLLOWERS first, LEADER last
kubectl get pods -n $NS -l app.kubernetes.io/name=$APP
# (for vault: `vault status` on each shows HA Mode: active | standby)

# 2. Delete ONE follower/standby
kubectl delete pod ${APP}-2 -n $NS

# 3. WAIT until it is fully back and rejoined quorum before touching the next:
#    - pod 1/1 Running
#    - app reports the member rejoined / caught up
#    e.g. for vault:
kubectl exec -n $NS ${APP}-2 -- vault status        # Sealed: false, HA Mode: standby
#    for raft membership:
kubectl exec -n $NS ${APP}-0 -- vault operator raft list-peers   # all voters, state=follower/leader

# 4. Repeat for the next follower (${APP}-1) ...

# 5. LAST: delete the LEADER (${APP}-0). The cluster elects a new leader,
#    then the old leader rejoins as a follower.

Golden rule: never delete two members at once, and never delete the next one until the previous has rejoined quorum. Deleting 2 of 3 drops you to 1/3 → below quorum → outage.

4. App-specific notes

Vault (HashiCorp) — integrated Raft + auto-unseal

  • Each pod seals on restart. With Transit auto-unseal (Root Vault), a recreated pod unseals itself automatically — no manual vault operator unseal needed.
  • Bootstrap dependency: auto-unseal only works if Root Vault is itself unsealed at that moment. Root Vault (vault-root) is a single standalone pod that seals on its own restart and must be manually unsealed (5 shares / threshold 3 from ~/vault-root-init.json). So: before rolling Main Vault — or after any event that restarted Root Vault — confirm kubectl exec -n vault-root vault-root-0 -- vault status shows Sealed: false. See 18-vault.md.
  • Verify rejoin: vault operator raft list-peers should list all members as voters.

General checklist before rolling ANY Raft app

  1. Is the cluster currently healthy with full quorum? (Don't start a roll while degraded.)
  2. Is any external unseal/bootstrap dependency up? (e.g. Root Vault for Main Vault.)
  3. Do you know which member is the leader? (Roll it last.)
  4. Are PodDisruptionBudgets set so an accidental drain can't evict a quorum? (minAvailable: N-1.)
5. Production sizing & HA rules (summary)
  • Size = 5 (odd) for production consensus clusters; 3 is non-production / minimum.
  • PodDisruptionBudget minAvailable: <quorum> (e.g. 3 for a 5-node cluster) so node drains can't break consensus.
  • podAntiAffinity (required, not preferred) so no two members share a node — a single node loss must never take 2 members.
  • Spread across failure domains (zones/racks) where available.
  • OnDelete retained; roll via the procedure above (or a tested operator/Job, never a blind kubectl rollout restart).
  • Persistent storage per member (Ceph RBD / fast SSD), never emptyDir.

Deviation

This deployment runs 3-node Vault — the documented minimum, not the production target. For real deployments — and for the upcoming Raft apps (Hatchet, etc.) — provision 5 members with the PDB + anti-affinity rules above.