KMS Release Runbook

Copy-paste runbook to promote a new knowledge-management-service build into mod-auh1-dev-ask. KMS is pinned by image digest (not a floating staging tag) and delivered via ArgoCD. This page is KMS-specific; the general rationale is in Release Strategy.

Every KMS promotion — check for config renames AND restructures

Some KMS builds rename or restructure config that comes from Vault and/or the chart TOML. Read the @devops promotion note — but don't trust it to be complete (it often flags only the env-var key, not an accompanying TOML change). The app validates its whole config at startup, so any missed field crashes the migration job / API even after the digest is live. See Step 5.

---

Release variables

Set these once at the top of your shell; every step below references them. Values shown are the current release — replace for the next one.

NEW_COMMIT="a0cdbd1760e4fec40b404c14992bf11ebfe74431"   # full git SHA from CI
NEW_SHORT="a0cdbd17"
NEW_DIGEST="sha256:7c4f5cd7732e5982f3b887a73bb53a4d18efb1bb3d01adbdcde3370b52fa750e"
REL_DATE="2026-07-07"
PREV_DIGEST="sha256:fc2911c541a91e3500533a7e0060fd2118f0c2264709ffaf43c7dada1f9080c1"  # for rollback
IMAGE="ask-knowledge-management-service"
ARGO_APP="mod-auh1-dev-ask.ask.knowledge-management-service"

Path	What it is
ACR source	ai71uaenprodask71acr01.azurecr.io/ask-knowledge-management-service (no ask/ prefix)
Harbor dest	harbor.cl1.sq4.aegis.internal/ask/ask-knowledge-management-service
Mirror config (local-only — copy to bastion)	096_RKE/scripts/harbor-mirror/mirror-config.yaml
Devops overlay	mod-ask-devops/applications/knowledge-management-service/values-mod-auh1-dev-ask.yaml
Vault secret path	secret/ask/knowledge-management-service/auh1-dev (KV v2, provider vault)
K8s Secret / ExternalSecret target	knowledge-management-service-secret

---

Step 1 — Verify the digest against ACR (bastion)

Never trust a hand-typed digest — confirm it fresh (or read it from the CI push output).

cd ~/harbor-images
CREDS="$(jq -r .username acr-creds.json):$(jq -r .password acr-creds.json)"
export https_proxy=http://10.96.137.37:3128
skopeo inspect --creds="$CREDS" \
  "docker://ai71uaenprodask71acr01.azurecr.io/${IMAGE}:${NEW_COMMIT}" \
  | python3 -c "import sys,json;print(json.load(sys.stdin)['Digest'])"
# must equal $NEW_DIGEST

---

Step 2 — Update mirror-config.yaml (local-only)

Add the new digest as staging, demote the previous to staging-prev (keep it — rollback must not require re-mirroring). Already applied for a0cdbd17. For the next release, edit the KMS block:

- { name: ask-knowledge-management-service, source: acr, project: ask, tags: ["staging", "<REL_DATE>"],       digest: "<NEW_DIGEST>" }
- { name: ask-knowledge-management-service, source: acr, project: ask, tags: ["staging-prev", "<prev-date>"], digest: "<PREV_DIGEST>" }

mirror-config.yaml is local-only — NOT in GitLab

Edit it on the Mac; commit locally for history if you like, but don't push. Copy the file to the bastion (Step 3) so its copy matches before mirroring.

---

Step 3 — Copy the config to the bastion

mirror-config.yaml is not in GitLab — copy it manually to the bastion (its ~/harbor-images/ copy must match before mirroring):

scp ~/Documents/096_RKE/scripts/harbor-mirror/mirror-config.yaml cloud-user@bastion1:~/harbor-images/

---

Step 4 — Mirror ACR → Harbor (bastion)

cd ~/harbor-images
./core42-mirror-image.py --only ${IMAGE} --dry-run
./core42-mirror-image.py --only ${IMAGE}

Fully air-gapped: stage the image, don't pull from ACR

The command above pulls from ACR through the forward proxy. In a fully air-gapped environment the bastion has no upstream access — the image must be manually copied in (staged into the mirror cache under /opt/data/ask/…), and the mirror script then pushes from the cache to Harbor (--push-only). Same mirror-config.yaml, no ACR pull.

Confirm the mirror script compares digests, not just tag existence

An older skip_existing skipped a re-copy when the tag already existed even if the digest differed. Use the fixed script; if a [SKIP] looks wrong, pull the ref directly to verify.

---

Step 5 — Config renames / restructures (required when flagged)

A promotion can change config in two places — a build may touch either or both:

Where	What	Reaches the pod as
Vault (env-injected)	fields at secret/ask/knowledge-management-service/auh1-dev (provider vault) — extracted as-is into knowledge-management-service-secret, injected via envFrom	env var, e.g. DOCUMENT_UNDERSTANDING__MODEL_CLIENT__API_KEY
Chart TOML (Git)	the configToml block in mod-ask-knowledge-management-service/helm/environments/values-mod-auh1-dev-ask.yaml (ArgoCD source 1, targetRevision: main) → rendered into the ConfigMap	config path, e.g. document_understanding.model_client.base_url

The @devops note may only mention part of it

Promotion notes often flag the env-var/secret rename but omit an accompanying TOML restructure. The app validates its whole config at startup (pydantic/dynaconf), so a missing field crashes with ValidationError: Field required / DynaconfFormatError even after the Vault key is fixed. Watch the first crash log (or grep the new build's default.toml) and fix every referenced path.

a0cdbd17 (2026-07): [document_understanding] → model_client (BOTH places)

Old	New	Fix
document_understanding.foundry_key (env)	document_understanding.model_client.api_key	Vault (a) below
document_understanding.foundry_endpoint (TOML)	document_understanding.model_client.base_url	chart TOML (b) below
—	document_understanding.model_client.timeout	defaulted to 60

(a) Vault — add the new env-injected key with the same value (keep the old until verified):

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
P=secret/ask/knowledge-management-service/auh1-dev
vault kv get "$P" | grep -iE "DOCUMENT_UNDERSTANDING|FOUNDRY"          # find the old field name
VAL=$(vault kv get -field=DOCUMENT_UNDERSTANDING__FOUNDRY_KEY "$P")
vault kv patch "$P" DOCUMENT_UNDERSTANDING__MODEL_CLIENT__API_KEY="$VAL"
ES=$(kubectl get externalsecret -n ask -o name | grep -i knowledge | head -1)
kubectl -n ask annotate $ES force-sync="$(date +%s)" --overwrite       # re-sync the Secret

(b) Chart TOML — edit the configToml block (source 1), then commit + push:

# before
[mod-auh1-dev-ask.document_understanding]
foundry_endpoint = "http://foundry-litellm.foundry.svc.cluster.local:4000/v1"
# after
[mod-auh1-dev-ask.document_understanding.model_client]
base_url = "http://foundry-litellm.foundry.svc.cluster.local:4000/v1"

cd mod-ask-knowledge-management-service   # git author: tarun.mittal@ai71.ai
git add helm/environments/values-mod-auh1-dev-ask.yaml
git commit -m "fix(kms): document_understanding -> model_client for <build>"
git push

After both parts: sync ArgoCD, rollout restart the deployments, and re-run the migration Job (delete it so Helm/ArgoCD recreates — a rollout restart won't re-run a Job). Remove the old Vault key once the new build is verified.

---

Step 6 — Bump the devops overlay digest + commit

Edit image.digest in the overlay → $NEW_DIGEST (already applied for a0cdbd17). Commit and push only after Step 4 succeeds — pushing before Harbor has the digest causes ImagePullBackOff.

What this step is, and why it's the one that actually deploys

The overlay applications/knowledge-management-service/values-mod-auh1-dev-ask.yaml is the ArgoCD-tracked Helm values for KMS, living in the devops repo in GitLab. ArgoCD continuously reconciles the cluster to match it (GitOps — Git is the source of truth):

image:
  tag: "staging"            # label only — ignored once digest is set
  digest: "sha256:7c4f5cd7…"  # ← the pin: exactly which image bytes run

• This is the pointer that decides what's deployed. Mirroring (Step 4) only copies the image into Harbor; the running pods keep serving the old digest until this pin changes.
• digest, not tag, forces the change. staging is reused every build, and with imagePullPolicy: IfNotPresent kubelet never re-pulls the same tag → it silently serves the old image. A new digest is a real, unambiguous change.
• Commit + push, because ArgoCD reads Git — not your laptop. The push creates the diff ArgoCD detects; Step 7's sync then applies it → the Deployment's pod spec gets the new digest → kubelet pulls exactly that image from Harbor → pods roll.
• Order matters — mirror first. If you push this before the image is in Harbor, ArgoCD deploys a digest that doesn't exist yet → ImagePullBackOff.

Step 4  mirror   →  image EXISTS in Harbor
Step 6  overlay  →  cluster is TOLD to use it   (git push = the trigger)   ← this step
Step 7  sync     →  ArgoCD applies it → pods roll to the new digest

cd mod-ask-devops   # git author: tarun.mittal@ai71.ai, no co-author
git add applications/knowledge-management-service/values-mod-auh1-dev-ask.yaml
git commit -m "chore(kms): deploy ${NEW_SHORT} (${REL_DATE})"
git push

---

Step 7 — ArgoCD sync

argocd login argocd.cl1.sq4.aegis.internal --username admin \
  --password "$(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}'|base64 -d)" \
  --insecure --grpc-web
argocd app sync ${ARGO_APP} --grpc-web

---

Step 8 — Verify the digest is live (API + all 3 workers)

kubectl -n ask get pods -l app.kubernetes.io/instance=knowledge-management-service \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[*].imageID}{"\n"}{end}'
# deployments: knowledge-management-service (API), -generic-worker, -ingestion-worker, -live-ingestion-worker

---

Step 9 — Check worker entrypoints in the new build (KMS gotcha)

A digest bump can change internal script layout (this exact thing broke the workers once: generic_worker.py → singleton_worker.py + a WorkerType arg).

POD=$(kubectl -n ask get pod -l app.kubernetes.io/name=knowledge-management-service-generic-worker -o name | head -1)
kubectl -n ask exec $POD -- find /app -iname "*worker*.py"
# if the layout diverged from the chart's command:/args:, update the KMS chart values → re-sync

---

Step 10 — Rollout restart (config / env changes)

KMS config is ConfigMap-mounted and the parser caches it (@lru_cache); an envFrom secret change also needs a fresh process. Restart the API + all workers:

kubectl -n ask rollout restart deploy -l app.kubernetes.io/instance=knowledge-management-service
kubectl -n ask rollout status  deploy -l app.kubernetes.io/instance=knowledge-management-service

---

Verification checklist

Run after every KMS release, cheapest first:

• [ ] Pods on the new imageID (Step 8), all Running.
• [ ] Migration job succeeded: kubectl -n ask get job | grep knowledge.
• [ ] Worker steady-state: kubectl -n ask logs deploy/knowledge-management-service-ingestion-worker --tail=40 | grep -iE "step run|🪓".
• [ ] Foundry models reachable: embedding qwen/qwen3-embedding-0-6b, VLM qwen/qwen3-5-122b-a10b-kms, rerank nvidia/llama-nemotron-rerank-1b-v2.
• [ ] Document-understanding auth works (the Step 5 rename) — ingest a doc that triggers VLM/OCR; confirm no 401/auth errors in the ingestion-worker logs. Reachability alone won't catch this.
• [ ] Ceph S3 round-trip (PUT/GET/LIST/DELETE) — S3 Wiring.
• [ ] Weaviate (vector) + OpenSearch (lexical) both up.
• [ ] End-to-end ingest a test doc → Indexed N/N chunks, doc → READY.

Once green, remove the superseded Vault field (Step 5):

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
P=secret/ask/knowledge-management-service/auh1-dev
# read all fields, drop the old key, write the full set back (works on any Vault CLI)
vault kv get -format=json "$P" | jq '.data.data | del(."DOCUMENT_UNDERSTANDING__FOUNDRY_KEY")' > /tmp/kms.json
vault kv put "$P" @/tmp/kms.json && rm -f /tmp/kms.json
# resync + restart so the stale env var is removed from the pods
ES=$(kubectl get externalsecret -n ask -o name | grep -i knowledge | head -1)
kubectl -n ask annotate $ES force-sync="$(date +%s)" --overwrite
kubectl -n ask rollout restart deploy -l app.kubernetes.io/instance=knowledge-management-service

Until you do this, the Extra config fields … FOUNDRY_KEY warning is harmless (the app ignores it).

---

Rollback

Revert the overlay digest to $PREV_DIGEST (still in Harbor and mirror-config.yaml as staging-prev), commit/push, re-sync, and restart:

# set image.digest back to $PREV_DIGEST in the overlay, then:
git add applications/knowledge-management-service/values-mod-auh1-dev-ask.yaml
git commit -m "revert(kms): roll back to previous digest"; git push
argocd app sync ${ARGO_APP} --grpc-web
kubectl -n ask rollout restart deploy -l app.kubernetes.io/instance=knowledge-management-service

If you rolled back and had applied a Vault key rename, keep both keys until the roll-forward — the previous build still reads the old key name.

---

This release — a0cdbd17 (2026-07-07) — ✅ COMPLETE

Item	Value / status
Commit	a0cdbd1760e4fec40b404c14992bf11ebfe74431
Digest	sha256:7c4f5cd7732e5982f3b887a73bb53a4d18efb1bb3d01adbdcde3370b52fa750e
Previous (rollback)	sha256:fc2911c5… (git 98efd55, 2026-07-01)
mirror-config.yaml	✅ updated (Mac)
Devops overlay digest	✅ bumped
Vault env-var rename	✅ DOCUMENT_UNDERSTANDING__FOUNDRY_KEY → …MODEL_CLIENT__API_KEY (Step 5a)
Chart TOML restructure	✅ document_understanding.foundry_endpoint → model_client.base_url (Step 5b; pushed d3cb146)
Mirror ACR→Harbor / overlay commit / ArgoCD sync	✅
Old Vault FOUNDRY_KEY removed (warning cleared)	✅
End-to-end verify — VLM/OCR + vector (Weaviate) + lexical (OpenSearch) → doc READY	✅