ASK — Application Onboarding¶

Concrete onboarding runbook for the ASK application (a multi-microservice product) onto the cluster. It applies the reusable pattern from Multi-Microservice Applications — read that first for the why; this page is the what to do for ASK.

End state: one ApplicationSet ask (wrapped by a root app modgpt.ask) generating one traceable Application per microservice, all under the modgpt AppProject:

modgpt.ask              (root app — manages the ApplicationSet)
  └─ ApplicationSet/ask
       ├─ modgpt.ask.web
       ├─ modgpt.ask.api
       ├─ modgpt.ask.worker
       ├─ modgpt.ask.postgres
       ├─ modgpt.ask.redis
       └─ modgpt.ask.hatchet

Fill in the real values

The service list, chart names, versions, and resource sizes below are a template — replace them with ASK's actual microservices and charts as you onboard. The structure and steps are the part to follow exactly.

Prerequisites

[ ] GitLab up; you can push to gitlab.cl1.sq4.aegis.internal/modgpt/* — GitLab Deployment
[ ] ArgoCD up; modgpt AppProject exists — App of Apps
[ ] modgpt AppProject sourceRepos allows the ASK repo (use the https://gitlab.cl1.sq4.aegis.internal/modgpt/* wildcard) — AppProject
[ ] Vault reachable for ASK secrets — Main Vault

1. Service inventory (plan first)

Service	Kind	Depends on	Sync wave	Ingress	Notes
`postgres`	stateful	—	0	—	Prod: CloudNativePG/Patroni HA; this deployment: bundled
`redis`	stateful	—	0	—	Prod: Sentinel/cluster
`hatchet`	stateful (Raft)	postgres	1	—	Task orchestrator — KB → Raft Clusters
`api`	stateless	postgres, redis, hatchet	2	internal	REST/gRPC
`worker`	stateless	postgres, redis, hatchet	2	—	Background jobs
`web`	stateless	api	3	`ask.mod.auh1.dev.dir`	Frontend

2. Create the ASK GitLab repo + layout

# create project modgpt/ask in GitLab (API or UI), clone it, then:
ask/
├── bootstrap/ask-appset.yaml        # the ApplicationSet
├── apps/{web,api,worker,postgres,redis,hatchet}/app.yaml
├── charts/ask-service/              # in-house generic chart (Deployment+Service+Ingress+HPA+probes)
└── values/{web,api,worker,...}/<svc>-stg.yaml

3. Mirror images + charts into GitLab (air-gap)

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)

# images → Harbor `ask/` project
for img in ask-web ask-api ask-worker; do
  skopeo copy --dest-creds "root:$GITLAB_TOKEN" \
    docker://<source>/$img:<tag> \
    docker://harbor.cl1.sq4.aegis.internal/ask/$img:<tag>
done
# in-house chart → ASK project HTTP Helm registry (basic auth, not PRIVATE-TOKEN header)
ASK_PID=$(curl -sk --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/modgpt%2Fask" | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])')
curl -sk --request POST "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/$ASK_PID/packages/helm/api/stable/charts" \
  --user "root:$GITLAB_TOKEN" --form "chart=@ask-service-1.0.0.tgz"

4. Namespace, pull secret, and secrets (Vault → VSO)

kubectl create namespace ask

- Pull secret harbor-pull-secret in ask (dockerconfigjson with the argocd-ro token) — same as the other namespaces; the container-images project is private. - App secrets (DB password, API keys) → store in Vault at secret/ask/*, render with a per-namespace VaultAuth + SA + gitlab-ro-style role + VaultStaticSecret. Never put DB passwords in values/. Pattern: GitLab Step 1.6b (VSO render).

vault kv put secret/ask/postgres password='<generate>'
vault kv put secret/ask/api      api_key='<...>'

5. Per-service app.yaml (with sync-wave ordering)

# apps/postgres/app.yaml      (data layer — wave 0)
namespace: ask
type: helm
chart: cloudnative-pg
targetRevision: 0.20.0
valuesPath: values/postgres/postgres-stg.yaml
syncWave: "0"

# apps/hatchet/app.yaml       (Raft orchestrator — wave 1, after postgres)
namespace: ask
type: helm
chart: hatchet
targetRevision: 0.30.0
valuesPath: values/hatchet/hatchet-stg.yaml
syncWave: "1"

# apps/api/app.yaml           (consumer — wave 2)
namespace: ask
type: helm
chart: ask-service
targetRevision: 1.0.0
valuesPath: values/api/api-stg.yaml
syncWave: "2"

…and worker (wave 2), web (wave 3, ingress ask.mod.auh1.dev.dir), redis (wave 0).

Each values/<svc>/<svc>-stg.yaml sets the container-images repo + harbor-pull-secret, resource requests/limits, and liveness/readiness probes.

6. The ASK ApplicationSet + root app

Copy the generic ApplicationSet into bootstrap/ask-appset.yaml, changing: - metadata.name: ask - name template → modgpt.ask.{{ .path.basename }} - helm repoURL → ASK project's HTTP Helm registry (…/projects/$ASK_PID/packages/helm/stable) - keep helm.releaseName: {{ .path.basename }} (dots-in-name gotcha) and the sync-wave passthrough

Then the root app (apply once):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: modgpt.ask, namespace: argocd }
spec:
  project: modgpt
  source: { repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/ask.git, targetRevision: HEAD, path: bootstrap }
  destination: { server: https://kubernetes.default.svc, namespace: argocd }
  syncPolicy: { automated: { prune: true, selfHeal: true } }

7. Go-live checklist

[ ] argocd app list | grep modgpt.ask → all Synced / Healthy, applied in wave order (0→3)
[ ] kubectl get pods -n ask → all Running; stateful pods on persistent Ceph volumes
[ ] hatchet (Raft) sized per KB — prod: 5 members, PDB minAvailable: quorum, OnDelete
[ ] Secrets are VSO-rendered from Vault (no plaintext in values/)
[ ] curl -sk https://ask.mod.auh1.dev.dir/healthz → 200
[ ] Every container has requests/limits + probes — Prod → Resource Limits
[ ] PDBs for stateful services; ingress TLS from cert-manager (ask.mod.auh1.dev.dir)

Day-2

Task	How
Add a microservice	drop `apps/<svc>/app.yaml` + push → `modgpt.ask.<svc>` auto-appears
Upgrade one service	bump `targetRevision`/image tag in its `app.yaml`/values → ArgoCD syncs only that service
Roll back one service	`argocd app rollback modgpt.ask.<svc>` — independent of the rest
Roll a stateful service	one pod at a time, leader last — KB → Raft Clusters
Onboard the next app	new repo + ApplicationSet + root app under `modgpt` — same as this page