ASK Core Service — Deployment Runbook (live)¶

Status: IN PROGRESS — empirical runbook

This is a live runbook we execute together, step by step, to deploy the ASK core-service into the air-gap. We run first, document second: real command output and surprises are captured in the Findings log at the bottom as we go. Once the deployment is green, this page becomes the verified basis for the generic in-repo Helm chart (localHelm) section of Add an Add-on Application.

Source repo (on the Mac): /Users/tarun.mittal/Documents/modgpt/ask-core-service-development — the chart lives in helm/.

What core-service is (read from the chart)

A single Helm chart that renders many workloads:

Piece	Detail
API	`deployment.yaml` — health at `/api/v1/health/live` & `/ready`, `service` (default LoadBalancer), optional `ingress`, HPA (autoscaling on)
Workers	bulk-operations (3 replicas, 4Gi each, HPA→20), billing-metering, cron-run, cron-scheduler — each its own Deployment + ServiceAccount
DB bootstrap	`job-postgres-init.yaml` — given a Postgres host + admin creds, creates the DB + role (`postgresInit.*`)
Config	`configmap-config-toml` (Dynaconf TOML, env-selected) + `configmap-models`; app reads `core-service-secret` for env secrets
Auth	`zitadel` subchart (OCI dependency, `condition: zitadel.enabled`) — SSO/IAM
Storage	expects Azure Blob (`storage.type: azure`) for CDN assets

Defaults that do not fit this air-gap (must override)¶

Default	Why it breaks here	Air-Gap override (planned)
`service.type: LoadBalancer`	no cloud LB	`ClusterIP` + ingress
`nodeSelector: node.kubernetes.io/role: ask71` + matching toleration	our nodes have no such label/taint → Pending forever	drop nodeSelector/tolerations
`image.repository: …azurecr.io/…` (api + per-worker)	ACR unreachable / x86	mirror amd64 → `harbor.cl1.sq4.aegis.internal/ask/*`
worker resources (4Gi ×3, HPA→20)	air-gap can't fit	v1: workers off; later small + HPA off
`postgresInit.connection` → Azure Postgres	unreachable	air-gap Postgres
`storage.type: azure`, Azure account keys	no Azure	stub/disable for v1
`zitadel.enabled` (OCI subchart)	air-gap OCI pull; repo URL is a placeholder	enabled + installed (required): vendor the wrapper chart, mirror its images, share the CNPG Postgres
`ingress.host: chart-example.local`, `ingress.enabled: false`	—	`core-service.ask.mod.auh1.dev.dir` + `sso.ask.mod.auh1.dev.dir` (zitadel), cert-manager TLS

Strategy — minimal first, then iterate

v1 goal (this runbook): the API pod Running + Ready and zitadel (IAM) installed + reachable, both backed by a air-gap CloudNativePG Postgres (DBs bootstrapped by the chart's PreSync init jobs), reachable at https://core-service.ask.mod.auh1.dev.dir/api/v1/health/ready and https://sso.ask.mod.auh1.dev.dir. Workers off and Azure storage stubbed for v1 — add those next.

Zitadel is in scope for v1 (required)

Auth is not optional here — core-service calls zitadel for IAM, so zitadel must be vendored, deployed, and seeded before the API is useful. The only things deferred to later iterations are the workers and real object storage.

This keeps the blast radius small (one app + its IAM + one DB) and lets us uncover one failure at a time — the whole point of doing this before writing the generic guide.

Phases

Each phase = run together → paste output → ✅ verify / ❌ diagnose → record in the Findings log → next.

Phase 0 — Local recon (render the chart, see what breaks) ⬅️ start here¶

Goal: render core-service locally on the Mac to discover its real shape, its dependency behaviour, and what it references — before touching the cluster.

cd /Users/tarun.mittal/Documents/modgpt/ask-core-service-development

helm version                                   # confirm helm is installed
helm dependency list helm/                     # shows zitadel dep + whether it's "ok" or "missing"
helm lint helm/ -f helm/environments/values-ai71-uaen-stg-ask71.yaml

# try to render with the staging env file (expect a dependency error — that's a finding)
helm template core-service helm/ \
  -f helm/environments/values-ai71-uaen-stg-ask71.yaml > /tmp/core-render-stg.yaml 2>/tmp/core-render-stg.err
echo "exit=$?"; tail -20 /tmp/core-render-stg.err

We're looking for: does it need the zitadel dep to render? what image refs appear? does nodeSelector/LoadBalancer show up? which secrets/configmaps are referenced? Paste the output and we'll read it together.

Phase 1 — Target mapping & v1 scope decisions¶

From the Phase-0 render, finalise the Azure→air-gap override table above and lock the v1 scope: workers off, zitadel ON, storage stubbed. Output: an agreed list of values to override.

Phase 2 — Get the chart into GitLab¶

Create modgpt/ask-core-service (or reuse the ASK repo), push the helm/ chart. Confirms ArgoCD can later read it from git (repo-creds prefix already covers modgpt/*).

Phase 3 — Images: mirror amd64 → Harbor registry ✅ decided (mirror)¶

Mirror these amd64 images (via skopeo copy --dest-creds) → harbor.cl1.sq4.aegis.internal/ask/*, then override the image refs + add harbor-pull-secret in the air-gap values:

core-service API (v1) — source [TBD — you'll provide host + creds]
zitadel server + zitadel login UI — exact refs come from the vendored wrapper chart (Phase 4)
postgres:16-alpine — used by both PreSync init jobs (postgresInit, zitadelPostgresInit)

(Worker images deferred until we enable workers.)

Phase 4 — Vendor + deploy zitadel (IAM) — required¶

Why we must vendor (verified in Phase 0): helm template fails while the zitadel dep is declared in Chart.yaml but absent from charts/ — even with zitadel.enabled=false (missing in charts/ directory: zitadel). The condition gates whether the subchart is rendered, not whether it must be present. Since zitadel is required, we vendor the real chart and configure it.

Steps:

Get the wrapper chart (you provide it) — real source per Chart.yaml: oci://registry.gitlab.com/ai71.ai/devops/devops-helm-charts/zitadel:1.0.1.
Mirror → internal registry: helm push zitadel-1.0.1.tgz oci://harbor.cl1.sq4.aegis.internal/modgpt/charts.
Point Chart.yaml at it: replace the placeholder repository: with oci://harbor.cl1.sq4.aegis.internal/modgpt/charts.
Vendor: cd helm && helm dependency build → commit charts/zitadel-1.0.1.tgz + Chart.lock to git (so ArgoCD renders it offline).
Mirror zitadel images (amd64) → Harbor ask/ (Phase 3): the zitadel server, the login UI, and postgres:16-alpine. Capture exact refs from the vendored chart's values.yaml once it's in hand.
Configure for the air-gap — see the contract below.

Zitadel config contract (air-gap — adapted from the prod-mufti env reference):

Knob (`zitadel.zitadel.zitadel.…`)	prod-mufti	air-gap value
`configmapConfig.ExternalSecure` / `ExternalDomain`	`true` / `sso.ask.mufti…`	`true` / `sso.ask.mod.auh1.dev.dir`
`configmapConfig.TLS.Enabled`	`false` (TLS at ingress)	`false`
`configmapConfig.Database.Postgres.Host/Port`	Azure PG	CNPG `…-rw.ask.svc.cluster.local` / `5432`
`…Database.Postgres.Database` / `User.Username` / `Admin.Username`	`zitadel` / `zitadel` / `postgres`	same
`…User.SSL.Mode` / `Admin.SSL.Mode`	`prefer`	TBD vs CNPG (`require`/`disable`)
`masterkeySecretName`	`zitadel-secret-kv`	`zitadel-secret-kv` (VSO)
`login.enabled` + `login.ingress` (`/ui/v2/login`)	on, host `sso…`	on, host `sso.ask.mod.auh1.dev.dir`
`service` / main `ingress` (gRPC backend, host `sso…`)	ClusterIP:8080	ClusterIP:8080, air-gap ingress class + cert-manager TLS
`tolerations` (`node.kubernetes.io/role: ask`)	set	clear
`zitadelPostgresInit.enabled` (PreSync hook, waves −2/−1)	`true` → Azure PG	`true` → CNPG (admin creds via secret)

Init-job ordering (verified from the env): the chart runs zitadelPostgresInit and core-service postgresInit as ArgoCD PreSync hooks (sync-waves -2/-1) — they create the zitadel and core-service DBs/roles before the workloads sync. postgresInit.seedData then seeds the default org + superadmin user into zitadel. This is a clean GitOps fit (no manual DB steps once secrets exist).

Phase 5 — Air-Gap Postgres via CloudNativePG ✅ decided (provision)¶

Deploy the CloudNativePG operator + a Postgres Cluster (Ceph-backed ceph-block, in ask or a dedicated cnpg namespace). CNPG generates a superuser secret; feed its host/port/admin creds to the chart's postgresInit (enable it) to create the core-service DB + role. Production-leaning: HA + backups via CNPG.

Phase 6 — Secrets (Vault → VSO)¶

Store all secrets in Vault (secret/ask/*) and render them with VSO into the ask namespace — no plaintext in values. Three secrets are referenced by the chart:

Secret	Keys (from the env reference)	Consumed by
`core-service-secret`	`DATABASE__PASSWORD` + app env (`STORAGE__`, `IAM_SERVICE__PAT`, `BILLING_METERING__`, `NOTIFICATIONS__*`, …)	API + workers (`envFrom`), `postgresInit` (user pw)
`zitadel-secret-kv`	`DATABASE__PASSWORD`, `ADMIN_PASSWORD`, masterkey	zitadel (DB user pw, first-instance admin, masterkey), `zitadelPostgresInit` (user pw)
`zitadel-psql-secret`	`db-admin-password`	both init jobs (Postgres admin pw) — this is the CNPG superuser password

The exact core-service-secret key list we'll confirm from the configToml @format {env[...]} references during Phase 7.

Phase 7 — Production values overlay¶

Create helm/environments/values-ai71-uaen-prod-modgpt.yaml in the sanitized repo. This file contains every air-gap-specific override — no secrets (those come from Vault via VSO). The full template is in the Repo Sanitization guide; the decisions that drive each override are summarised below.

Override decisions (all made during Phases 0–6)¶

Override	From default	To (stg)	Why decided
`image.repository`	`ai71uaenprodask71acr01.azurecr.io/core-service`	`harbor.cl1.sq4.aegis.internal/ask/core-service`	ACR unreachable; air-gap requires Harbor registry
`imagePullSecrets`	`[]`	`[harbor-pull-secret]`	Private Harbor registry requires pull secret
`service.type`	`LoadBalancer`	`ClusterIP`	No cloud LB in air-gap
`ingress`	disabled	enabled — `core-service.ask.mod.auh1.dev.dir`, cert-manager TLS, `rke2-nginx` class	Air-Gap ingress + internal CA
`nodeSelector` / `tolerations`	`node.kubernetes.io/role: ask71`	`{}` / `[]`	Label/taint does not exist on air-gap nodes → pod Pending
Workers (`bulkOperationsWorker`, etc.)	`enabled: true`	`enabled: false`	v1 scope — resources too large for air-gap; enable in Phase 11
`postgresInit` connection	Azure Postgres host	`ask-pg-rw.ask.svc.cluster.local` (CNPG)	No Azure; CNPG provisioned in Phase 5
`zitadelPostgresInit` connection	Azure Postgres host	`ask-pg-rw.ask.svc.cluster.local` (CNPG)	Same
`storage.type`	`azure`	`stub`	No Azure Blob; Ceph S3 wired in Phase 11
`zitadel.enabled`	`false`	`true`	Required for auth — cannot stub
`zitadel…ExternalDomain`	prod domain	`sso.ask.mod.auh1.dev.dir`	Air-Gap DNS
`zitadel…SSL.Mode`	`prefer`	`disable`	CNPG internal TLS not required for air-gap
`zitadel…tolerations`	`node.kubernetes.io/role: ask`	`[]`	Clear cloud scheduling constraints
`migrations.enabled`	`false`	`true`	PreSync Job runs `alembic upgrade head` on every sync

PreSync hook wave ordering¶

All PreSync Jobs run before any Deployment or StatefulSet starts. Wave ordering ensures each step has its prerequisite ready:

Wave	Hook	What it does	Enabled by
−2	`zitadelPostgresInit` ConfigMap + `postgresInit` ConfigMap	Creates DB connection ConfigMaps used by the init Jobs	`postgresInit.argocd.enabled: true`
−1	`zitadelPostgresInit` Job	Creates `zitadel` DB + role on CNPG	`zitadelPostgresInit.enabled: true`
−1	`postgresInit` Job	Creates `core-service` DB + role on CNPG	`postgresInit.enabled: true`
0	`migrations` Job	`alembic upgrade head` — applies schema to `core-service` DB	`migrations.enabled: true`
−1	`postgresInit` seed Job	Seeds default org + superadmin user into `core-service` DB (requires schema — runs after migrations complete within wave −1 ordering)	`postgresInit.seedData.enabled: true`
5 (default)	All Deployments (zitadel server, login UI, core-service API)	Workloads start against a fully initialised + migrated DB	—

Seed data depends on migrations

postgresInit.seedData inserts rows that require the schema to exist. Confirm the seed Job's sync-wave annotation in the vendored chart is ≥ 1 (after migrations at wave 0). If it's set to −1 (same as the DB init Job), override it in values-ai71-uaen-prod-modgpt.yaml:

postgresInit:
  argocd:
    seedJobAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "1"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

Migration toggle¶

# migrations — PreSync Job (helm/templates/job-migrations.yaml)
# enabled: true  → alembic upgrade head runs on every ArgoCD sync
# enabled: false → no Job rendered; use ONLY for emergency rollbacks
migrations:
  enabled: true

Default in values.yaml is false (safe for local dev). Always true in values-ai71-uaen-prod-modgpt.yaml.

Phase 8 — Local validation against the cluster¶

helm template with the air-gap overlay, then helm install --dry-run --server-side (or helm install into a scratch namespace) to catch admission/RBAC issues before GitOps.

Phase 9 — Deploy¶

First green can be a direct helm install to prove values; then wire into GitOps as modgpt.ask.core-service (the in-repo-chart / localHelm pattern we're validating). Record exactly how ArgoCD renders the vendored subchart and runs the PreSync init-job hooks (waves −2/−1) ahead of the workloads — this is the key data for the how-to.

Phase 10 — Verify + post-deploy SSO setup¶

Cluster verification:

# PreSync Jobs completed
kubectl get jobs -n ask

# Zitadel + login UI running
kubectl get pods -n ask | grep -i zitadel

# API healthy
curl -sk https://core-service.ask.mod.auh1.dev.dir/api/v1/health/ready   # → 200
curl -sk https://sso.ask.mod.auh1.dev.dir                            # → Zitadel login page

Post-deploy SSO setup (one-time — standalone Zitadel, no external IDP):

Zitadel seedData creates the default org and superadmin, but the OIDC client for the frontend and the PAT for core-service must be created manually in Zitadel once it's up:

Log into https://sso.ask.mod.auh1.dev.dir as the seeded superadmin (password from zitadel-secret-kv.ADMIN_PASSWORD).
Create OIDC client for the frontend: Projects → New Project (ask) → New Application → type Web → PKCE → set redirect URI (e.g. https://ask.mod.auh1.dev.dir/auth/callback) → copy Client ID.
Create PAT for core-service: Service Users → New → generate PAT → copy token.

Store both in Vault:

vault kv put secret/ask/frontend oidc_client_id=<id>
vault kv put secret/ask/core-service/iam iam_service_pat=<token>

Update core-service-secret VSO VaultStaticSecret to include IAM_SERVICE__PAT → VSO re-renders → Reloader restarts the API pod.

See Air-Gap Alignment → SSO for full detail.

Phase 11 — Iterate¶

Turn on workers (small replicas/resources), wire real object storage (Ceph RGW S3 instead of Azure Blob), and tune the HPA. Each becomes a follow-up section + a Findings entry.

Findings log

Real output and decisions, recorded as we run. This is the empirical record that will seed the generic localHelm how-to.

Phase	Date	Finding / decision	Action
0	2026-05-30	helm v4.1.3 on the Mac; `helm lint` passes (warns on missing dep only).	—
0	2026-05-30	zitadel dep `missing`, repo is placeholder `your-internal-registry.example.com`. `helm template` fails even with `zitadel.enabled=false` → `missing in charts/ directory: zitadel`. Condition gates render, not presence.	Must vendor the chart (can't just disable) — see Phase 4.
0	2026-05-30	Renders 5 Deployments (API + 4 workers, all ON), 1 Service LoadBalancer, 1 HPA, 2 ConfigMap, 5 ServiceAccount, 1 helm-test Pod (busybox `wget`).	v1: disable workers; Service → ClusterIP.
0	2026-05-30	API Deployment carries `nodeSelector: node.kubernetes.io/role: ask71` + matching toleration; workers have empty nodeSelector.	clear top-level `nodeSelector`/`tolerations` (else API pod Pending).
0	2026-05-30	All 5 images are ACR `:latest` — `core-service`, `…-billing-metering-worker`, `…-bulk-operations-worker`, `…-cron-scheduler-worker`, `…-job-run-worker`.	mirror amd64 (v1: API only) → Harbor registry.
0	2026-05-30	Every workload `envFrom: secretRef: core-service-secret` + mounts the configToml at `/app/config/override.toml`; Stakater reloader annotation present.	provide `core-service-secret` via VSO (no plaintext).
0	2026-05-30	No postgres-init Job rendered with stg values (`postgresInit` off → expects an existing Azure Postgres).	air-gap: enable `postgresInit` against a air-gap Postgres, or pre-create the DB.
1	2026-05-30	Decision: Postgres via CloudNativePG (provision operator + Ceph-backed Cluster); enable chart `postgresInit` against it.	Phase 5
1	2026-05-30	Decision: images mirrored from the upstream amd64 registry via skopeo → Harbor `ask/`.	Phase 3
1	2026-05-30	Cluster recon: no CNPG, no non-gitlab Postgres (install operator). `rke2-metrics-server` running → HPA OK. SCs: `ceph-block` (default RBD), `ceph-filesystem` (RWX), `ceph-bucket` (RGW S3 — candidate for storage later). `ask` ns absent.	CNPG operator → cluster-addon; PG `Cluster` CR → `ask` ns
1	2026-05-30	Requirement change: zitadel must be enabled + installed (not stubbed). Must vendor the wrapper chart — user to provide `oci://registry.gitlab.com/ai71.ai/devops/devops-helm-charts/zitadel:1.0.1`.	Phase 4 rewritten to vendor+deploy
1	2026-05-31	Alembic migrations are NOT automatic. `Dockerfile` copies `alembic/` into the image but `CMD` only starts uvicorn. Migrations must be triggered separately.	Decision: PreSync Helm Job (sync-wave 0) — see Phase 7
1	2026-05-31	New template `helm/templates/job-migrations.yaml` added to chart. Toggle: `migrations.enabled` (default `false` in `values.yaml`; set `true` in `values-ai71-uaen-prod-modgpt.yaml`). Wave order: DB init (−1) → migrations (0) → workloads (5).	✅ implemented in chart
1	2026-05-30	Zitadel config contract read from `values-ai71-uaen-prod-mufti.yaml`: shares Postgres (DB `zitadel`/user `zitadel`/admin `postgres`); secrets `zitadel-secret-kv` (masterkey, DB+admin pw) & `zitadel-psql-secret` (PG admin pw); `login` UI subchart; `ExternalDomain` + ingress; init jobs are ArgoCD PreSync hooks (waves −2/−1); `postgresInit.seedData` seeds default org + superadmin.	drives Phase 4/6/7

➡️ Related: ASK Onboarding · Multi-Microservice Pattern · Add an Add-on Application (gets the verified localHelm section after this runbook).

Release strategy (2-source: chart base + devops overlay)¶

The core-service ArgoCD Application is multi-source — separating what the app is (chart) from which build is deployed (overlay), so releases happen in the devops repo without touching the chart repo.

Layer	Repo / file	Changes on	Owner
Base config (configToml, DB-init, zitadel, secrets)	chart `ask-core-service` → `helm/environments/values-mod-auh1-dev-ask.yaml`	chart/config changes (rare)	app team
Release overlay (image tag, replicas, tuning)	devops `mod-ask-devops` → `applications/core-service/values-mod-auh1-dev-ask.yaml`	every release / promotion	ops / GitOps

The Application stacks them as two value files (Helm applies left-to-right, so the overlay wins):

valueFiles:
  - environments/values-mod-auh1-dev-ask.yaml                         # base (chart repo)
  - $values/applications/core-service/values-mod-auh1-dev-ask.yaml    # overlay (devops repo) — wins

Ship a release — edit only the devops overlay:

# in mod-ask-devops: applications/core-service/values-mod-auh1-dev-ask.yaml
#   image:
#     tag: "staging-<new-sha>"     ← bump here
git commit -am "release(core-service): deploy staging-<sha>" && git push   # ArgoCD auto-syncs

Promote across environments: each env has its own overlay with its own image.tag; promotion = bump the tag in the target env's overlay. Argo CD Image Updater can automate this.

Same pattern as assistant-service

Both services use this 2-source chart-base + devops-overlay split. Keeping release-managed values (image tags) out of the chart repo decouples the deployment lifecycle from the app source. See Assistant Service Deployment Guide → §10.