Core Service — ArgoCD Sync-Wave & Ordering

This page documents the deployment ordering for core-service — why the resources must be created in a specific sequence, the two bugs we hit without it, and the sync-wave / hook scheme that fixes them.

TL;DR

A separate secrets app provisions core-service-secret / zitadel-secret first. Then the core-service app runs: PreSync creates the databases (DB-init) and bootstraps Zitadel (its own init/setup hooks); the Sync phase applies config, runs migrations (wave 4) and finally the Deployment (wave 5).

---

1. Why ordering matters here

core-service is not a single Deployment — it is a dependency chain:

flowchart LR
    S["🔐 Secrets<br/>(separate app, first)"] --> D1["⚙️ DB-init<br/>create DB + roles"]
    D1 --> Z["⚙️ Zitadel init/setup<br/>schema + instance"]
    Z --> C["📄 Config<br/>config-toml"]
    C --> M["⚙️ Migration<br/>alembic upgrade head"]
    M --> W["🚀 Workloads<br/>Deployment + Service"]

    style S fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style D1 fill:#3a1e5f,stroke:#a94ad9,color:#fff
    style Z fill:#3a1e5f,stroke:#a94ad9,color:#fff
    style C fill:#5f4a1e,stroke:#d9a94a,color:#fff
    style M fill:#3a1e5f,stroke:#a94ad9,color:#fff
    style W fill:#1e5f3a,stroke:#4ad98a,color:#fff

If any step runs before its prerequisites exist, the sync fails:

• DB-init needs the DB-password secrets (core-pg-superuser, core-service-db-password, zitadel-db-password) — provided by the ask-db app.
• Zitadel init/setup (upstream subchart PreSync hooks) need zitadel-secret (masterkey + DB password) and the zitadel database to already exist.
• Migration needs the DB, the config-toml ConfigMap, and core-service-secret.
• Deployment mounts config-toml and consumes core-service-secret via envFrom.

---

2. The two bugs we hit without sync waves

🔴 Bug 1 — app secrets were never synced

core-service-secret and zitadel-secret live in applications/core-service/core-service-secrets.yaml, but the core-service ArgoCD Application's second source was ref: values with no path: — so that file was never applied. The Deployment's envFrom: secretRef: core-service-secret then failed with CreateContainerConfigError, and Zitadel's hooks had no masterkey.

Fix: a dedicated secrets Application (argocd-application-core-service-secrets.yaml) that syncs the secrets file, ordered before the core-service app via app-of-apps sync-waves.

🔴 Bug 2 — migration hook depended on a later phase

The migration Job mounts config-toml and uses core-service-secret, but it was a PreSync hook. PreSync runs before the entire Sync phase where those resources are created — so it could never see them.

Fix: convert migration from PreSync to a Sync hook (wave 4), and wave the Deployment to 5 so workloads come last. DB-init stays in PreSync (see §4).

---

3. Why the secrets need their own app (the Zitadel constraint)

The upstream Zitadel subchart ships its own PreSync hooks:

Job	Helm hook	Needs
zitadel-init	pre-install,pre-upgrade (weight 1)	zitadel DB + zitadel-secret
zitadel-setup	pre-install,pre-upgrade (weight 2)	zitadel-secret (masterkey)

ArgoCD maps pre-install/pre-upgrade → PreSync. Since PreSync runs before the whole Sync phase, a secret created as a normal Sync-phase resource (even at wave 0) would not exist when these hooks run. A directory source inside the core-service app cannot satisfy them.

flowchart TB
    subgraph BAD["❌ secrets inside core-service app (Sync phase)"]
        B1["PreSync: zitadel-setup<br/>needs zitadel-secret"] -.->|"can't see"| B2["Sync wave 0:<br/>zitadel-secret"]
    end
    subgraph GOOD["✅ secrets in a separate app, synced first"]
        G1["app wave 0:<br/>core-service-secrets<br/>(secrets materialised)"] --> G2["app wave 1:<br/>core-service<br/>(PreSync hooks see secrets)"]
    end

    style BAD fill:#5f1e1e,stroke:#d94a4a,color:#fff
    style GOOD fill:#1e5f3a,stroke:#4ad98a,color:#fff

This mirrors how ask-db already provisions the DB-password secrets in its own app before any consumer runs.

---

4. The full scheme — two apps, ordered

App-of-apps ordering (Application sync-waves)

App wave	ArgoCD Application	Provides
0	…ask.postgresql (ask-db)	CNPG cluster + DB-password secrets
0	…ask.core-service-secrets	core-service-secret, zitadel-secret
1	…ask.core-service	the chart (waits for wave 0 apps to be Healthy)

Inside the core-service app (resource phases/waves)

Phase · wave	Resources	Kind
PreSync · −2	DB-connection ConfigMaps + init scripts	hook
PreSync · −1	DB-init Jobs (core-service, zitadel) — create DB + roles	hook
PreSync · 1–2	Zitadel init / setup (upstream subchart)	hook
Sync · 0	ConfigMaps (config-toml, model-config), Zitadel Deployments	normal
Sync · 4	Migration Job (alembic upgrade head)	Sync hook
Sync · 5	core-service Deployment, Service, Ingress	normal
Sync · 6	HorizontalPodAutoscaler	normal — after the Deployment (its scaleTargetRef)

Why DB-init stays in PreSync (not Sync)

The Zitadel init/setup hooks are themselves PreSync, and they need the zitadel database to already exist. So the zitadel DB-init (and, for symmetry, the core-service DB-init) must run earlier in the PreSync phase (wave −1) — before Zitadel's own hooks (wave 1–2). Moving DB-init to the Sync phase would run it after Zitadel's PreSync hooks and break them. DB-init only needs the DB-password secrets from ask-db, which exist before this app syncs — so PreSync is safe.

Why the migration job is a Sync hook (not PreSync)

Migration needs config-toml (Sync wave 0) and core-service-secret. As a Sync hook at wave 4 it runs after config (wave 0) and after the DB exists (PreSync), but before the Deployment (wave 5). The hook semantics (hook-delete-policy: BeforeHookCreation) keep it idempotent across re-syncs.

Annotation examples

env values: DB-init PreSync (creates DB before Zitadel hooks)

postgresInit:
  argocd:
    connectionConfigMapAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "-2"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    jobAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "-1"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

chart values.yaml: migration Sync hook + workload wave

migrations:
  argocd:
    jobAnnotations:
      argocd.argoproj.io/hook: Sync
      argocd.argoproj.io/sync-wave: "4"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

argocd:
  syncWave:
    workload: "5"   # Deployment/Service/Ingress deploy last
    hpa: "6"        # HPA after the Deployment, so scaleTargetRef resolves

app-of-apps: core-service app waits for the secrets app

# argocd-application-core-service.yaml
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"   # after core-service-secrets (wave 0)

---

5. Cross-app dependencies

flowchart LR
    subgraph A0["app wave 0"]
        DB["ask-db<br/>CNPG + DB-password secrets"]
        SEC["core-service-secrets<br/>core-service-secret, zitadel-secret"]
        DB -->|"Vault: db passwords"| SEC
    end
    subgraph A1["app wave 1"]
        CS["core-service<br/>PreSync hooks + workloads"]
    end
    DB --> CS
    SEC --> CS

    style A0 fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style A1 fill:#1e5f3a,stroke:#4ad98a,color:#fff

• core-service-secrets ExternalSecrets pull DB passwords from Vault paths populated by ask-db's PushSecrets — so the secrets app stays Progressing until ask-db has run, and core-service (wave 1) waits for both. This is self-healing via ESO retries.

---

5b. Vault prerequisites (one-time, before first sync)

The secrets app cannot go Healthy until Vault is ready to serve ask/core-service/*:

1. Policy grant — ESO's eso role must be allowed on the core-service paths. Add to eso-ask-push in applications/cluster-addons/vault/manifests/vault-bootstrap-config.yaml:

   path "secret/data/ask/core-service/*"     { capabilities = ["create", "update", "read"] }
   path "secret/metadata/ask/core-service/*" { capabilities = ["read", "list"] }
   

   then argocd app sync vault --grpc-web. Without this, PushSecrets fail with 403 — see Troubleshooting.

2. Seed ALL paths once — the secrets app holds only ExternalSecrets (the PushSecret + ClusterGenerator auto-gen was removed: ESO's IfNotExists existence check on a brand-new Vault path gets Vault's {"errors":[]} 404 body and fails with invalid JSON format, which forces a manual seed anyway — bypassing the generator). Seed every path by hand, exactly like ask-db / hatchet do:

   export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
   # Generated secrets (any strong random value works; PushSecret won't overwrite):
   vault kv put secret/ask/core-service/auh1-dev/SECURITY__API_KEY_SALT           password="$(openssl rand -hex 32)"
   vault kv put secret/ask/core-service/auh1-dev/SECURITY__MASTER_ENCRYPTION_KEY  password="$(openssl rand -base64 32 | tr '+/' '-_')"   # Fernet key — 32 urlsafe-base64 bytes (44 chars)
   vault kv put secret/ask/core-service/auh1-dev/APP__CORE_SERVICES_MACHINE_TOKEN password="$(openssl rand -hex 32)"
   vault kv put secret/ask/core-service/auh1-dev/ZITADEL_MASTERKEY                password="$(openssl rand -hex 16)"   # MUST be exactly 32 bytes (hex 16 = 32 chars)
   # Manual placeholders (filled after Zitadel / Hatchet are up):
   vault kv put secret/ask/core-service/auh1-dev/IAM_SERVICE__PAT      value="placeholder"
   vault kv put secret/ask/core-service/auh1-dev/HATCHET__CLIENT_TOKEN value="placeholder"
   

---

6. Verification

# App-of-apps order: secrets app Healthy before core-service starts
kubectl get applications -n argocd | grep core-service

# Secrets exist before core-service hooks
kubectl get externalsecret -n ask core-service-secret zitadel-secret
kubectl get secret         -n ask core-service-secret zitadel-secret

# Resource waves inside the core-service app
kubectl get job    -n ask -l app.kubernetes.io/instance=core-service
kubectl get deploy core-service -n ask \
  -o jsonpath='{.metadata.annotations.argocd\.argoproj\.io/sync-wave}'; echo   # → 5

In the ArgoCD UI the resource tree shows wave badges; a correct sync runs PreSync (−2 → 2) then Sync (0 → 4 → 5), each wave Healthy before the next.

See also: TOML Config Flow for how config-toml is built and mounted, and Troubleshooting for the ConfigMap-not-found and manifest-generation errors.