Core Service Onboarding Runbook (Zitadel + Super Admin)

End-to-end onboarding to make core-service usable after deploy: create the default org, the CORE_SERVICE service user + PAT, the Super Admin user, register the Zitadel custom domain, and seed the org/user so login + org-discovery work. Adapted from the official AI71 core-service setup guide to this air-gapped cluster (our domains, Vault instead of Azure Key Vault).

Prerequisites

core-service and Zitadel are deployed and healthy (this page is post-deploy onboarding, not deployment). See Deployment Runbook. The [foundry] admin block must be configured or org/auth endpoints 500.

Air-gap deviations from the official guide

Official guide	Here (air-gap)
Azure Key Vault secrets (core-service--…)	Vault secret/ask/core-service/auh1-dev/<KEY> → ESO → core-service-secret
sso.ask.pp.gov.ae / *.edgegroup.ae	sso.ask.mod.auh1.dev.dir
internal ask-core-service-zitadel…	core-service-zitadel.ask.svc.cluster.local
Super Admin superadmin@ask.pp.gov.ae	superadmin@ask.mod.auh1.dev.dir (TLD of this env)

---

1. Zitadel — default org, service user, Super Admin

Log in to the Zitadel Console https://sso.ask.mod.auh1.dev.dir with the Zitadel admin (zitadel-admin@zitadel.sso.ask.mod.auh1.dev.dir).

Screenshots

The images below live in docs/core-service/images/ (01-new-organization.png … 14-assistant-service-pat.png). Drop the screenshots from the official AI71 setup guide there — until then the slots render as broken-image icons. The numbered filenames match each step's order.

Step 1 — Create the default organization

Top bar → click the arrow next to the organization name → New Organization.

Enter the org name ask-default (mandatory; drop any 71 suffix). This is the default org for the ASK application.

Once created, open the organization and copy its Resource Id — it is reused everywhere below (default_org_id, seedData, the Super Admin username).

Step 2 — Create the CORE_SERVICE service user

Organization → Users → Service Users → New.

Enter the details — User Name CORE_SERVICE, Access Token Type Bearer → Create.

Step 3 — Mint the CORE_SERVICE PAT

The service user → Personal Access Tokens (left menu) → New.

Generate a PAT with no expiry → Add. Store it immediately — shown only once. It goes into two Vault keys (Step 5): APP__CORE_SERVICES_MACHINE_TOKEN and IAM_SERVICE__PAT.

Step 4 — Create the Super Admin user

Organization → Users → Users (not Service Users) → New.

Field	Value
Email	superadmin@ask.mod.auh1.dev.dir (need not be deliverable; use this env's TLD)
Username	superadmin@ask.mod.auh1.dev.dir#<ask-default-org-id> (format: <email>#<org-id>)
Given / Family name	Super / Admin
Email Verified	True
Initial Password	a secure password

Keep the Super Admin User ID + the ask-default org id — both are needed for seeding (Step 6).

Step 5 — Verified domain

Organization → Settings → Verified Domains → + New → enter ask.mod.auh1.dev.dir → Add.

Step 6 — Grant IAM Owner

Go to https://sso.ask.mod.auh1.dev.dir/ui/console/instance/members.

• + New → add the CORE_SERVICE service user as IAM Owner.
• + New again → add the Super Admin user as IAM Owner.

Step 7 — Secret generators

Default Settings → Secret Generators → Email verification → set the expiration to 1470 → Save (leave everything else at default).

Zitadel setup complete.

---

2. Assistant Service — extra service user

Newer assistant-service needs its own machine token.

1. Organization → Users → Service Users → + New — User Name ASSISTANT_SERVICE_MACHINE_TOKEN, Access Token Type Bearer.

1. …/ui/console/instance/members → + New → add it as Iam User Manager (not Iam Owner).
2. Back in the org → that service user → Personal Access Tokens → New (no expiry) → store the PAT for assistant-service's IAM__MACHINE_TOKEN (Vault: secret/ask/assistant-service/auh1-dev/…).

---

3. Store the PATs in Vault (not Azure Key Vault)

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal VAULT_SKIP_VERIFY=true
# CORE_SERVICE PAT → both keys (core-service-secrets.yaml maps these into core-service-secret)
vault kv put secret/ask/core-service/auh1-dev/APP__CORE_SERVICES_MACHINE_TOKEN value="<CORE_SERVICE-PAT>"
vault kv put secret/ask/core-service/auh1-dev/IAM_SERVICE__PAT                  value="<CORE_SERVICE-PAT>"
# assistant-service machine token
vault kv put secret/ask/assistant-service/auh1-dev/IAM__MACHINE_TOKEN value="<ASSISTANT-PAT>"
# force ESO resync
kubectl annotate externalsecret -n ask core-service-secret force-sync=$(date +%s) --overwrite

See Zitadel Onboarding for the per-key ESO mapping detail.

---

4. Register the Zitadel Custom Domain (the "Instance not found" fix)

core-service calls Zitadel at its internal URL core-service-zitadel.ask.svc.cluster.local:8080, but the instance is registered for sso.ask.mod.auh1.dev.dir → internal calls 500 with Instance not found. There is no UI/API to add an instance domain — Zitadel adds the configured ExternalDomain via a startup DB migration. So do the two-step toggle in the devops overlay applications/core-service/values-mod-auh1-dev-ask.yaml → zitadel.configmapConfig:

Step 1 — point ExternalDomain at the internal name, push + sync, wait, confirm login breaks:

ExternalSecure: false
ExternalDomain: core-service-zitadel.ask.svc.cluster.local

Step 2 — revert, push + sync:

ExternalSecure: true
ExternalDomain: sso.ask.mod.auh1.dev.dir

After step 2 both domains are in the instance's Custom Domains (Console → Default Settings → Domains). Full procedure + why: Troubleshooting → Zitadel Instance not found.

---

5. Helm chart changes (seed the org + user)

In the devops overlay (applications/core-service/values-mod-auh1-dev-ask.yaml):

1. [mod-auh1-dev-ask.iam_service] default_org_id = the ask-default org Resource Id.
2. postgresInit.seedData.org.externalOrganizationId = the ask-default org id.
3. postgresInit.seedData.user.externalUserId = the Super Admin User ID.

4. On the very first deploy postgresInit.seedData.enabled is false; once the IDs above are set, flip it to true and push — the init job seeds the org + Super Admin into the core_service DB.

5. Set postgresInit.seedData.org.defaultRoleId = 3 (member). organizations.default_role_id is NOT NULL (FK → core_service.roles.id) — it is the role auto-assigned to users joining the org, so least-privilege member is correct (roles: 1 ask71_admin / 2 organization_admin / 3 member). The Super Admin himself still gets seedData.user.roleId: 1 (ask71_admin).

default_role_id — the seed was one column behind the schema

The official guide's raw INSERT (and earlier chart seeds) predate a migration that added organizations.default_role_id as NOT NULL. Seeding without it fails with null value in column "default_role_id" … violates not-null constraint. The wrapper chart now sets it from seedData.org.defaultRoleId. See Troubleshooting → core-service seed fails on default_role_id.

Manual DB seeding (fallback only)

The official guide's raw INSERT into organizations / users / user_roles is no longer needed with the Zitadel-wrapper chart + postgresInit.seedData (the seed job runs all three). Kept for debugging only — and note the schema-current organizations INSERT must include default_role_id:

-- database = "core-service", schema = core_service
INSERT INTO "core_service"."organizations"
  (created_at, created_by_email, deleted_at, external_organization_id, is_personal_workspace,
   logo_object_key, name, primary_domain, state, subscription_plan_id, tenant_subdomain,
   default_role_id, updated_at)                               -- ← default_role_id added vs the official doc
VALUES (NOW(), NULL, NULL, '<ZITADEL-ORG-ID>', false, NULL,
   'ask-default', 'ask.mod.auh1.dev.dir', 'active', 'enterprise', 'admin', 3, NOW());

INSERT INTO "core_service"."users"
  (avatar_object_key, created_at, deleted_at, display_name, email, external_user_id,
   first_name, last_name, last_synced_at, organization_id, state, updated_at)
VALUES (NULL, NOW(), NULL, 'Super Admin', 'superadmin@ask.mod.auh1.dev.dir',
   '<ZITADEL-USER-ID>', 'Super', 'Admin', NULL, '<ZITADEL-ORG-ID>', 'active', NOW());

INSERT INTO "core_service"."user_roles" (created_at, role_id, user_id)
VALUES (NOW(), 1, (SELECT id FROM "core_service"."users"
                   WHERE email = 'superadmin@ask.mod.auh1.dev.dir'));   -- role_id 1 = ask71_admin

---

6. Verify

core-service is exposed on its own ingress host core-service.ask.mod.auh1.dev.dir at path / (it is not under api.ask…/ask71/v2/svc — that host/prefix routes to assistant-service). Verify ingress with kubectl get ingress -n ask.

# in-cluster (most reliable; the app image has no curl, so use python):
kubectl exec -n ask deploy/assistant-service-intelligence -- python3 -c \
"import httpx; print(httpx.get('http://core-service.ask.svc.cluster.local/api/v1/organizations/discovery', \
 params={'email':'superadmin@ask.mod.auh1.dev.dir'}, timeout=10).text)"

# external (via ingress) — now returns the org instead of 404 'Organization not found':
curl -sk 'https://core-service.ask.mod.auh1.dev.dir/api/v1/organizations/discovery?email=superadmin@ask.mod.auh1.dev.dir'
# expect: 200 + [{"organization_id":"…","organization_name":"ask-default", … "organization_state":"Active"}]

Browser login needs every ASK host in /etc/hosts

The Mac browser resolves these hosts itself during login. /etc/hosts must map all ASK hosts to the HAProxy VIP (100.115.2.210), not just the frontend:

100.115.2.210  ask.mod.auh1.dev.dir core-service.ask.mod.auh1.dev.dir api.ask.mod.auh1.dev.dir sso.ask.mod.auh1.dev.dir hatchet.ask.mod.auh1.dev.dir

Without core-service.… and sso.… the frontend loads but login fails (the browser can't reach core-service / Zitadel).

Then log into the frontend https://ask.mod.auh1.dev.dir as the Super Admin → the UI loads the org.

---

7. Feature flags (this env)

Flag	This env	Where
Billing	disabled	core-service feature_flags.billing_enabled=false; assistant BILLING-ENABLED=false; frontend NEXT_PUBLIC_FF_BILLING_METERING_ENABLED=false
Public signup	enabled	core-service feature_flags.public_signup_enabled=true; frontend NEXT_PUBLIC_FF_PUBLIC_USER_SIGNUP_ENABLED=true

(Billing off → keep all three in sync, or the UI/services disagree.)

---

Source

Adapted from the official AI71 core-service setup guide (Zitadel onboarding → Super Admin → custom domain → seed → verify). Related: Zitadel Onboarding, Deployment Runbook, Troubleshooting.

---

8. Enabling models for an organisation

The model picker in the UI (/api/v1/organization-models) only shows models that have been explicitly enabled for the organisation. The global catalog (models.yml) is the source of truth for what can be enabled; organization_models DB table controls what is enabled per org.

How it works

models.yml (baked in image)        ← global catalog — all possible models
       ↓
POST /api/v1/organization-models/enable   ← admin assigns models to an org
       ↓
organization_models table (DB)     ← per-org enabled set
       ↓
GET /api/v1/organization-models    ← what the UI agent picker shows

Check which models are currently enabled

# Get your org's JWT first, then:
curl -sk https://core-service.ask.mod.auh1.dev.dir/api/v1/organization-models \
  -H "Authorization: Bearer <token>" | python3 -m json.tool | grep '"id"'

Enable a model for the org

curl -sk -X POST \
  https://core-service.ask.mod.auh1.dev.dir/api/v1/organization-models/enable \
  -H "Authorization: Bearer <org-admin-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "org_id": "<org-id>",
    "llm_model_id": ["qwen/qwen3-5-122b-a10b"],
    "default_model_id": "qwen/qwen3-5-122b-a10b"
  }'

Disable a model for the org

curl -sk -X DELETE \
  https://core-service.ask.mod.auh1.dev.dir/api/v1/organization-models/disable \
  -H "Authorization: Bearer <org-admin-token>" \
  -H "Content-Type: application/json" \
  -d '{"org_id": "<org-id>", "llm_model_id": "gpt-oss-120b"}'

Air-gap recommended model set

Only models served by the air-gap Foundry should be enabled. Enabling a cloud-only model will show it in the UI but calls will fail at inference time.

Model ID	Type	Foundry served	Enable?
qwen/qwen3-5-122b-a10b	inference	✅	✅ yes
qwen/qwen3-5-122b-a10b	inference	✅	✅ yes (if needed)
qwen/qwen3-embedding-0-6b	embedding	✅	✅ yes
anthropic/claude-*	inference	❌ cloud only	❌ no
grok-*, minimax-*, z-ai/*	inference	❌ cloud only	❌ no

models.yml vs organization_models

Removing a model from models.yml removes it from the global catalog — it can no longer be enabled for any org. Deleting from organization_models (via the disable API) removes it only for that org. Use the disable API first; remove from models.yml only if the model should never appear in any org.