Core Service — Deployment Runbook (mod-auh1-dev)¶

Deploys core-service (FastAPI) + Zitadel (IAM/SSO) into the ask namespace via ArgoCD. Workers disabled for v1 — images are ready in Harbor (tag: staging).

Config aligned with: ai71-pp-auh1-prod-ask production reference.

Access (post-deploy): - API: https://core-service.mod.auh1.dev.dir/api/v1/health/ready - IAM: https://sso.mod.auh1.dev.dir - API docs: https://core-service.mod.auh1.dev.dir/docs

Architecture¶

ask namespace
├── core-service (API)                   FastAPI — /ask71/v2/svc + /api/v1/health
├── core-service-zitadel                 IAM/SSO — gRPC :8080 (fullnameOverride)
├── core-service-zitadel-login           Zitadel login UI — /ui/v2/login
├── core-service-redis (standalone)      Bitnami Redis, no auth (single-node dev)
│
│   PreSync Jobs (every ArgoCD sync):
├── zitadel-postgres-init                CREATE DATABASE zitadel OWNER zitadel
├── core-service-postgres-init           CREATE DATABASE core-service OWNER core-service
└── core-service-migrations              alembic upgrade head
│
└── core-pg (CNPG, 3-node)              Shared PostgreSQL — already deployed
    (mod-auh1-dev-ask.ask.postgresql.core)

Why fullnameOverride: core-service-zitadel matters: The Zitadel Kubernetes service name is core-service-zitadel.ask.svc.cluster.local. The TOML config iam_service.base_url must exactly match this — the two are linked. If fullnameOverride changes, the base_url must change with it.

Images¶

Image	Harbor Path	Tag	Status
core-service API	`harbor.cl1.sq4.aegis.internal/ask/core-service`	`staging`	✅ mirrored
bulk-ops-worker	`harbor.cl1.sq4.aegis.internal/ask/core-service-bulk-operations-worker`	`staging`	✅ mirrored (enable in v2)
billing-worker	`harbor.cl1.sq4.aegis.internal/ask/core-service-billing-metering-worker`	`staging`	✅ mirrored (enable in v2)
cron-scheduler	`harbor.cl1.sq4.aegis.internal/ask/core-service-cron-scheduler-worker`	`staging`	✅ mirrored (enable in v2)
job-run-worker	`harbor.cl1.sq4.aegis.internal/ask/core-service-job-run-worker`	`staging`	✅ mirrored (enable in v2)
zitadel server	`harbor.cl1.sq4.aegis.internal/ghcr/zitadel/zitadel`	`v4.5.0`	🔴 must mirror
zitadel login	`harbor.cl1.sq4.aegis.internal/ghcr/zitadel/zitadel-login`	`v4.5.0`	🔴 must mirror
postgres:16-alpine	`harbor.cl1.sq4.aegis.internal/dockerhub/postgres`	`16-alpine`	✅ mirrored
redis	`harbor.cl1.sq4.aegis.internal/bitnamilegacy/redis`	`8.2.1-debian-12-r0`	✅ mirrored

Files¶

File	Purpose
`applications/core-service/values-mod-auh1-dev-ask.yaml`	Helm values — images, ingress, DB init, Zitadel, TOML configmap
`applications/core-service/core-service-secrets.yaml`	PushSecrets (auto-gen) + ExternalSecrets → `core-service-secret` + `zitadel-secret`
`environment/auh1/dev/02-argocd-application/ask/argocd-application-core-service.yaml`	Multi-source ArgoCD Application

Prerequisites¶

1 — Mirror Zitadel images to Harbor¶

Both images are now enabled in mirror-config.yaml. Run from Mac:

cd ~/harbor-mirror
python3 core42-mirror-image.py --config mirror-config.yaml \
  --only zitadel/zitadel zitadel/zitadel-login

Verify:

ROBOT=$(cat ~/harbor-images/harbor-robot.json | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['name'])")
TOKEN=$(cat ~/harbor-images/harbor-robot.json | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['secret'])")
HARBOR=harbor.cl1.sq4.aegis.internal

curl -sk -u "${ROBOT}:${TOKEN}" \
  "https://${HARBOR}/v2/ghcr/zitadel/zitadel/tags/list"
# Expected: {"tags":["v4.5.0"]}

curl -sk -u "${ROBOT}:${TOKEN}" \
  "https://${HARBOR}/v2/ghcr/zitadel/zitadel-login/tags/list"
# Expected: {"tags":["v4.5.0"]}

2 — Prepare the chart repo in Core42 GitLab¶

The chart is in-repo and must be sanitized + pushed to Core42 GitLab.

Step A — Update Chart.yaml dependency URL:

# Edit: ask-core-service-development/helm/Chart.yaml
# Change the placeholder:
#   repository: "oci://your-internal-registry.example.com/devops-helm-charts"
# To the Core42 GitLab OCI registry:
#   repository: "oci://registry.gitlab.cl1.sq4.aegis.internal/modgpt/charts"

Step B — Get and push the Zitadel wrapper chart:

# The wrapper chart source: oci://registry.gitlab.com/ai71.ai/devops/devops-helm-charts/zitadel:1.0.1
# Get the .tgz from the PP environment or ai71 GitLab, then push to Core42:
helm push zitadel-1.0.1.tgz oci://registry.gitlab.cl1.sq4.aegis.internal/modgpt/charts

Step C — Vendor the subchart:

cd /Users/tarun.mittal/Documents/modgpt/ask-core-service-development/helm
helm dependency build
ls charts/
# Expected: zitadel-1.0.1.tgz  Chart.lock

Step D — Commit uncommitted files:

cd /Users/tarun.mittal/Documents/modgpt/ask-core-service-development
git status
# job-migrations.yaml and helm/values.yaml show as modified/untracked
git add helm/templates/job-migrations.yaml helm/values.yaml helm/charts/ helm/Chart.lock
git commit -m "feat: vendor zitadel subchart + add migration job template"

Step E — Create project in Core42 GitLab and push:

# Create project: https://gitlab.cl1.sq4.aegis.internal/ai71/ask71/ask-core-service

# Push (first time):
cd /Users/tarun.mittal/Documents/modgpt/ask-core-service-development
git remote add core42 https://gitlab.cl1.sq4.aegis.internal/ai71/ask71/ask-core-service.git
git push core42 main

3 — Deploy Redis for core-service¶

No Redis exists in the ask namespace. Deploy a standalone Bitnami Redis:

# The Bitnami Redis image is already in Harbor
# Service name will be: core-service-redis-master.ask.svc.cluster.local
# (matches TOML redis.host in values file)

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
export KUBECONFIG=~/.kube/config   # or however you connect to the cluster

helm install core-service-redis \
  --namespace ask \
  --set image.registry=harbor.cl1.sq4.aegis.internal \
  --set image.repository=bitnamilegacy/redis \
  --set image.tag=8.2.1-debian-12-r0 \
  --set global.security.allowInsecureImages=true \
  --set auth.enabled=false \
  --set architecture=standalone \
  --set master.persistence.size=2Gi \
  --set master.persistence.storageClass=csi-rbd-sc \
  --set global.imagePullSecrets[0]=harbor-pull-secret \
  bitnami/redis
  # Note: use chart from Harbor if Bitnami chart is blocked
  # Or deploy a simple Redis Deployment manifest directly

# Verify
kubectl -n ask get pods | grep core-service-redis
kubectl -n ask get svc | grep core-service-redis
# Expected service: core-service-redis-master  ClusterIP  6379/TCP

Redis service name

Bitnami Redis chart names the master service <release-name>-master. Our release name is core-service-redis, so the service is core-service-redis-master. This matches the TOML redis.host value in values-mod-auh1-dev-ask.yaml.

4 — Seed Vault paths (before first sync)¶

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
vault login $(jq -r '.root_token' ~/hashicorp_vault/vault-main-init-mod-auh1-dev.json)

# Auto-generated (PushSecrets will manage these after first sync, but seed first to avoid 404)
vault kv put secret/ask/core-service/auh1-dev/SECURITY__API_KEY_SALT \
  password="$(openssl rand -hex 32)"

vault kv put secret/ask/core-service/auh1-dev/SECURITY__MASTER_ENCRYPTION_KEY \
  password="$(openssl rand -hex 32)"

vault kv put secret/ask/core-service/auh1-dev/APP__CORE_SERVICES_MACHINE_TOKEN \
  password="$(openssl rand -hex 32)"

vault kv put secret/ask/core-service/auh1-dev/ZITADEL_MASTERKEY \
  password="$(openssl rand -base64 32 | tr -d '\n')"

# Manual placeholders — update after services are up
vault kv put secret/ask/core-service/auh1-dev/IAM_SERVICE__PAT \
  value="placeholder-update-after-zitadel-boot"

vault kv put secret/ask/core-service/auh1-dev/HATCHET__CLIENT_TOKEN \
  value="placeholder-update-from-hatchet-ui"

5 — Verify Vault policy covers ask/core-service/*¶

vault policy read eso-ask-push | grep "core-service"
# Must show: path "secret/data/ask/core-service/*" { capabilities = [...] }

If missing: argocd app sync vault --grpc-web

6 — Add DNS records (Mac `/etc/hosts`)¶

100.115.2.210  core-service.mod.auh1.dev.dir
100.115.2.210  sso.mod.auh1.dev.dir

Deployment¶

# 1 — Push mod-ask-devops to GitLab
cd /Users/tarun.mittal/Documents/modgpt/mod-ask-devops
git add applications/core-service/ \
  environment/auh1/dev/02-argocd-application/ask/argocd-application-core-service.yaml
git commit -m "feat: add core-service deployment (v1 — API + Zitadel)"
git push origin main

# 2 — Sync (ArgoCD auto-sync will pick it up; or trigger manually)
argocd app sync mod-auh1-dev-ask.ask.core-service --grpc-web --prune

# 3 — Watch progress
argocd app get mod-auh1-dev-ask.ask.core-service --grpc-web
kubectl -n ask get pods -w | grep -E "core-service|zitadel"

Sync wave sequence:

Wave	Resource	What happens
0	PushSecrets	Generate SECURITY keys + ZITADEL_MASTERKEY → Vault
1	ExternalSecrets	Pull → `core-service-secret` + `zitadel-secret`
−2	zitadelPostgresInit ConfigMap	DB connection config for Zitadel init job
−1	zitadelPostgresInit Job	`CREATE DATABASE zitadel OWNER zitadel`
−2	postgresInit ConfigMap	DB connection config for core-service init job
−1	postgresInit Job	`CREATE DATABASE core-service OWNER core-service`
0	migrations Job	`alembic upgrade head`
(default)	core-service Deployment	API starts
(default)	core-service-zitadel Deployment	IAM starts
(default)	Ingress resources	TLS cert issued by `internal-ca-issuer`

Verification¶

# Pods
kubectl -n ask get pods | grep -E "core-service|zitadel"

# API health (liveness — just checks the process is up)
curl -sk https://core-service.mod.auh1.dev.dir/api/v1/health/live | python3 -m json.tool

# API readiness (requires DB + Zitadel connection)
curl -sk https://core-service.mod.auh1.dev.dir/api/v1/health/ready | python3 -m json.tool

# Zitadel health
curl -sk https://sso.mod.auh1.dev.dir/debug/healthz

# Databases created
kubectl -n ask exec core-pg-1 -- psql -U postgres -c "\l" | grep -E "core-service|zitadel"

# Migrations applied
kubectl -n ask exec core-pg-1 -- psql -U postgres -d core-service \
  -c "SELECT version_num FROM alembic_version;"

# Secrets synced
kubectl -n ask get externalsecret | grep -E "core-service|zitadel"

Post-Deploy — Zitadel First Boot¶

Open https://sso.mod.auh1.dev.dir

Initial admin: root@zitadel.mod.auh1.dev.dir (or the seeded org admin email) Password: Zitadel@ModAuh1Dev1! (from zitadel-secret.ADMIN_PASSWORD)

First boot takes ~60s

Zitadel runs DB migrations on first start. The pod will be Running but not Ready for ~60s. Watch: kubectl -n ask logs -f deployment/core-service-zitadel

2 — Create service account + PAT for core-service¶

Zitadel UI → Users → Service Users → Create
Name: core-service-svc, username: core-service
Click user → Personal Access Tokens → Add
Copy the generated PAT

3 — Store PAT in Vault + force ESO refresh¶

vault kv put secret/ask/core-service/auh1-dev/IAM_SERVICE__PAT \
  value="<paste-pat-here>"

# Force ESO to pull the updated value
kubectl -n ask annotate externalsecret core-service-secret \
  force-sync=$(date +%s) --overwrite

# Stakater Reloader automatically restarts core-service once secret updates
# Monitor:
kubectl -n ask get pods | grep core-service

4 — Update Hatchet token¶

# Get token: https://hatchet.mod.auh1.dev.dir → Settings → API Tokens → Create Token
# Name: core-service-mod-auh1-dev

vault kv put secret/ask/core-service/auh1-dev/HATCHET__CLIENT_TOKEN \
  value="<paste-token-here>"

kubectl -n ask annotate externalsecret core-service-secret \
  force-sync=$(date +%s) --overwrite

Credential Summary¶

Secret	Vault Path	K8s Secret	Key
PG superuser	`ask/postgres/core-pg--SUPERUSER--PASSWORD`	`core-pg-superuser`	`password`
core-service DB pw	`ask/postgres/core-service--DATABASE--PASSWORD`	`core-service-db-password`	`password`
Zitadel DB pw	`ask/postgres/zitadel--DATABASE--PASSWORD`	`zitadel-db-password`	`password`
API key salt	`ask/core-service/auh1-dev/SECURITY__API_KEY_SALT`	`core-service-secret`	`SECURITY__API_KEY_SALT`
Master enc key	`ask/core-service/auh1-dev/SECURITY__MASTER_ENCRYPTION_KEY`	`core-service-secret`	`SECURITY__MASTER_ENCRYPTION_KEY`
Machine token	`ask/core-service/auh1-dev/APP__CORE_SERVICES_MACHINE_TOKEN`	`core-service-secret`	`APP__CORE_SERVICES_MACHINE_TOKEN`
Zitadel PAT	`ask/core-service/auh1-dev/IAM_SERVICE__PAT`	`core-service-secret`	`IAM_SERVICE__PAT`
Hatchet token	`ask/core-service/auh1-dev/HATCHET__CLIENT_TOKEN`	`core-service-secret`	`HATCHET__CLIENT_TOKEN`
Zitadel masterkey	`ask/core-service/auh1-dev/ZITADEL_MASTERKEY`	`zitadel-secret`	`masterkey`

Differences vs PP Production Reference¶

Config	PP Production	This deployment	Reason
Chart source	Harbor OCI `registry.pp.gov.ae/helm-charts` chart `ask/core-service`	Git repo `ask-core-service.git` `path: helm`	Chart not yet published to Harbor Helm registry in this env
Postgres	Bitnami `postgresql.ask.svc.cluster.local`	CNPG `core-pg-rw.ask.svc.cluster.local`	CNPG already deployed; production-grade HA
Redis	`assistant-service-redis-master` (from assistant-service subchart)	`core-service-redis-master` (standalone)	assistant-service not yet deployed
Storage	`type: azure` with Azure Blob	`type: stub`	No Azure in the air-gap environment
Workers	`bulkOperationsWorker: enabled: true`	All workers disabled	Enable in v2 once resource headroom confirmed
TLS	External wildcard cert `ask-pp-gov-ae-cert`	cert-manager `internal-ca-issuer`	Uses internal CA
Admin password	From external secrets system	`zitadel-secret.ADMIN_PASSWORD`	Same pattern, different secret backend
`core_services_machine_token`	`"dummy"` (hardcoded in TOML — ⚠️ not a secret)	Vault-backed in `core-service-secret`	Security improvement over prod practice
Ingress routing	PP uses F5 path rewrite: dual-path `/ask71/v2/svc/(.)` + `/(.)` → `rewrite-target: /$1`, `root_path = "/ask71/v2/svc"`	Simple `path: /`, no rewrite annotation, `root_path = ""`	PP routes via shared F5 API gateway at `api.ask.pp.gov.ae`; this env has a dedicated subdomain — no prefix stripping needed

Troubleshooting¶

Symptom	Cause	Fix
`helm template` fails: `missing in charts/ directory: zitadel`	Subchart not vendored	Run `cd helm && helm dependency build`; commit `charts/` + `Chart.lock`
`zitadel pod: CrashLoopBackOff` — masterkey error	ZITADEL_MASTERKEY < 32 chars or wrong secret key name	Secret key must be `masterkey` (check `zitadel-secret`); value must be ≥32 bytes
`core-service pod: 503 on /api/v1/health/ready`	Zitadel not yet ready or IAM_SERVICE__PAT placeholder	Wait for Zitadel to be Running+Ready; update PAT in Vault
`postgresInit job: FATAL: role "core-service" does not exist`	ExternalSecret wave 1 not synced before wave −1 job	Force sync ESO first; check `core-service-db-password` secret exists
`ExternalSecret SYNCED=False: invalid JSON format`	Brand-new Vault path returns 404 (empty JSON)	Seed all paths with the `vault kv put` commands in prereq 4
Redis `dial tcp: connection refused`	`core-service-redis` not deployed or wrong service name	Check `kubectl -n ask get svc \| grep redis`; service must be `core-service-redis-master`
`zitadel ingress: 502 Bad Gateway`	nginx sending HTTP to gRPC backend	Add annotation `nginx.ingress.kubernetes.io/backend-protocol: "GRPC"`
`core-service 404 on any path`	Wrong `root_path` in TOML (leftover from PP)	Confirm `[mod-auh1-dev-ask.fastapi] root_path = ""` in override TOML; also confirm no `rewrite-target` annotation in ingress
Zitadel DB init fails: `database "zitadel" already exists`	CNPG created it via `bootstrap.initdb` (check cluster spec)	Safe to ignore — job exits 0; or add `IF NOT EXISTS` to init SQL

V1 Scope — What's Deferred¶

Feature	Status	When
All 4 workers	Disabled	V2 — images in Harbor, just enable per-worker
Ceph S3 storage	Stubbed	V3 — Ceph RGW bucket + credentials
Notifications (email)	Disabled (`@none`)	When SMTP/Mailchimp configured
Billing/metering	Disabled	After billing-metering-worker + Hatchet cron wired
Frontend OIDC	Manual post-step	Register frontend in Zitadel after SSO is up
Chart in Harbor Helm registry	Git source for now	Publish chart via CI when pipeline is set up