Troubleshooting — ASK Platform (Core42)¶

Scope

Issues on the live Core42 ASK platform (cl1.sq4.aegis.internal) — core-service, KMS, assistant-service, frontend, Hatchet, Foundry, ESO. Pre-production infrastructure issues (VM / RKE2 / Rook-Ceph) are out of scope for this document.

Issue 1 — Hatchet: 500 Internal Server Error on /workflow-runs/trigger — gRPC frame too large / TLS mismatch

Symptom:

KMS update-status?upload_status=queued returns 200, but the document stays queued with workflow_run_id: null. KMS logs show repeated ServiceException: (500) retries against Hatchet. Hatchet API logs show:

ERR API error="rpc error: code = Unavailable desc = connection error:
desc = \"error reading server preface: http2: failed reading the frame payload:
http2: frame too large, note that the frame header looked like an HTTP/1.1 header\""
uri=/api/v1/stable/tenants/.../workflow-runs/trigger status=500

Or earlier form (before restart):

transport: authentication handshake failed: remote error: tls: protocol version not supported

Root cause:

hatchet-shared-config sets SERVER_GRPC_BROADCAST_ADDRESS=hatchet.ask.mod.auh1.dev.dir:443 and SERVER_GRPC_INSECURE=t (plaintext gRPC). Both the API pod and Engine pod load this secret. When the API dispatches a workflow to the engine, it dials hatchet.ask.mod.auh1.dev.dir:443 — which is the Nginx ingress (TLS) — using plaintext gRPC. Nginx returns TLS bytes; the HTTP/2 parser sees them as a malformed frame → 500.

Workers are unaffected because they override the broadcast address with HATCHET_CLIENT_HOST_PORT=hatchet-engine.ask.svc.cluster.local:7070 (direct internal plaintext). The API pod had no such override.

Fix (values file, permanent):

In applications/hatchet/values-mod-auh1-dev-ask.yaml, add an api.env override that takes precedence over envFrom (direct env entries win over envFrom in Kubernetes):

api:
  env:
    - name: SERVER_GRPC_BROADCAST_ADDRESS
      value: "hatchet-engine.ask.svc.cluster.local:7070"

This makes the API dial the engine directly on port 7070 (internal, plaintext) instead of routing through Nginx. Workers are unaffected. JWT tokens embedded in existing secrets are unaffected (those are generated by the setup job, not the running API).

Workaround (temporary, lost on ArgoCD sync):

kubectl set env deploy/hatchet-api -n ask \
  SERVER_GRPC_BROADCAST_ADDRESS=hatchet-engine.ask.svc.cluster.local:7070
kubectl rollout status deploy/hatchet-api -n ask

Verify fix:

After rollout, reset a stuck document and re-trigger:

kubectl exec -i -n ask knowledge-pg-1 -- psql -U postgres -d knowledge_management -c \
  "UPDATE knowledge_service.documents SET processing_status='uploading', workflow_run_id=NULL WHERE id='<doc_id>';"
# Then call PATCH /documents/upload/<doc_id>/update-status?upload_status=queued
# Document should move to processing within a few seconds, then ready.

Also: restarting only hatchet-api (not engine) or only hatchet-engine (not API) can change the error variant (TLS version ↔ frame too large) without fixing it — both pods share the same hatchet-shared-config and the mismatch is structural, not transient.

Issue 2 — assistant-service: Error in interruption listener: Error 111 connecting to localhost:6379

Symptom:

After Application startup complete, the main assistants pod logs a warning every 5 s:

assistant_service - WARNING - Error in interruption listener:
  Error 111 connecting to localhost:6379. Connection refused.

The pod stays Running (health probes are not affected — /health/liveness and /health/readiness return 200 unconditionally with no Redis check). However, the chat interruption signal (used to stop a streaming response mid-flight) silently fails — users cannot cancel a running generation.

Root cause:

AssistantsSettings.redis_host (in src/shared/config/assistant_settings.py:77) defaults to "localhost" in pydantic-settings:

redis_host: Optional[str] = Field(default="localhost", ...)
redis_port: Optional[int] = Field(default=6379, ...)
redis_password: str = Field(...)  # required — no default

The InterruptionListener reads these fields to create its Redis pub/sub client. The TOML setting [mod-auh1-dev-ask.redis] host = "..." feeds Dynaconf for OTHER parts of the app but does NOT reach pydantic-settings. REDIS_HOST env var was never set, so the class uses the "localhost" default, connecting to a non-existent Redis on the pod itself.

REDIS_PASSWORD IS set (from the ESO-synced assistant-service-secret), which is why AssistantsSettings initializes without error — but the host is wrong.

Fix — add REDIS_HOST to envConf.extraEnv:

In helm/environments/values-mod-auh1-dev-ask.yaml:

envConf:
  extraEnv:
    - name: REDIS_HOST
      value: "assistant-service-redis-master.ask.svc.cluster.local"

To patch the running cluster immediately:

kubectl set env deploy -n ask \
  -l app.kubernetes.io/instance=assistant-service \
  REDIS_HOST=assistant-service-redis-master.ask.svc.cluster.local
kubectl rollout status deploy -n ask -l app.kubernetes.io/instance=assistant-service

Confirm after rollout:

kubectl logs -n ask deploy/assistant-service --since=2m | grep -i "interrupt\|redis"
# expect: "Subscribed to chat_interruptions channel" and NO "Error 111" lines

Fifth env-only field in AssistantsSettings

This follows the same pattern as CORE_SERVICE_URL — a pydantic-settings field with a bad default that bypasses the TOML. The full list of env vars required in extraEnv is in the deployment guide §4d.

Issue 3 — assistant-service main + intelligence crash: AssistantsSettings — core_service_url — Field required

Symptom:

Both the main assistants (port 8000) and intelligence (port 8003) deployments crash-loop. Gunicorn worker exits with code 3. Traceback ends in pydantic_settings:

pydantic_core._pydantic_core.ValidationError: 1 validation error for AssistantsSettings
core_service_url
  Field required [type=missing, ...]
Worker failed to boot.

The main crash path: src/assistants/app.py → assistant_routes.py → ... → src/assistants/core/config.py → src/shared/config/assistant_settings.py → AssistantsSettings().

Root cause:

AssistantsSettings is a pydantic-settings class that reads env vars only — it does not read the Dynaconf TOML. The core_service.url TOML entry does not reach it. core_service_url has no default and is required, so the class raises ValidationError at import time → worker exits code 3 before the HTTP server starts.

AssistantsSettings is imported by both src/assistants/app.py (main, port 8000) and src/intelligence/app.py (intelligence, port 8003) — so both crash.

Fix — add CORE_SERVICE_URL env var to all workloads:

In the chart values (envConf.extraEnv applies to every deployment + worker + migration):

envConf:
  extraEnv:
    - name: CORE_SERVICE_URL
      value: "http://core-service.ask.svc.cluster.local/api/v1"

To patch the running cluster immediately (before the chart commit lands):

kubectl set env deploy -n ask \
  -l app.kubernetes.io/instance=assistant-service \
  CORE_SERVICE_URL="http://core-service.ask.svc.cluster.local/api/v1"
kubectl rollout status deploy -n ask -l app.kubernetes.io/instance=assistant-service

General pattern: three config systems in assistant-service

assistant-service has three independent config readers. Only Dynaconf reads the TOML. The two pydantic-settings classes (ToolsSettings, AssistantsSettings) read env vars only — their required fields must be in envConf.extraEnv. See deployment guide §4d for the full list.

Issue 4 — assistant-service: tiktoken ConnectTimeout openaipublic.blob.core.windows.net (air-gap)

Symptom:

Main assistants logs show a warning during startup:

[LLM_REGISTRY] Failed to initialize tiktoken encoding by name; falling back to approximate
token counting. encoding='o200k_base'
error=ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='openaipublic.blob.core.windows.net',
port=443): Max retries exceeded ...

This is expected and harmless in an air-gapped cluster. tiktoken tries to download the o200k_base tokenizer vocabulary file from Azure blob storage on first use. The download times out (no internet), and the service falls back to approximate token counting automatically. No action needed — exact token counts are not critical for the assistant service's core functionality.

Issue 5 — Hatchet workers / assistant-service: Socket closed on gRPC connect (TLS mismatch)

Symptom:

assistant-service workers (or core-service workers) crash-loop or fail to register with Hatchet. Worker logs show:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
  status = StatusCode.UNAVAILABLE
  details = "Socket closed"

The gRPC connection reaches the engine pod but is immediately dropped.

Root cause:

sharedConfig.grpcInsecure: "f" in the Hatchet chart values sets SERVER_GRPC_INSECURE=f in hatchet-shared-config, so the engine serves TLS gRPC on port 7070. All in-cluster clients use client_tls_strategy = "none" (plaintext) — the engine rejects the handshake and closes the socket immediately.

Why grpcInsecure=f is wrong for an air-gapped in-cluster setup:

The external Hatchet gRPC ingress (hatchet.ask.mod.auh1.dev.dir:443) uses nginx.ingress.kubernetes.io/backend-protocol: GRPC (plaintext backend) — NGINX terminates TLS externally. There is no TLS between NGINX and the engine. The engine does not actually have a valid cert for in-cluster traffic. Setting grpcInsecure=t makes the engine serve plaintext on 7070 — correct for in-cluster clients and still correct for the external ingress path (NGINX → backend is already plaintext).

Fix — flip the engine to insecure (plaintext) gRPC:

Confirm the ingress backend is plaintext:

kubectl get ingress -n ask -o yaml | grep backend-protocol
# expect: backend-protocol: GRPC  (NOT GRPCS)

Edit applications/hatchet/values-mod-auh1-dev-ask.yaml in mod-ask-devops:

sharedConfig:
  grpcInsecure: "t"      # was "f" — engine now serves plaintext gRPC on 7070

Sync + roll the engine:

argocd app sync mod-auh1-dev-ask.ask.hatchet --grpc-web
kubectl rollout restart deploy hatchet-engine -n ask
kubectl rollout status  deploy hatchet-engine -n ask

Restart the workers:

# assistant-service workers
kubectl rollout restart deploy -n ask -l app.kubernetes.io/instance=assistant-service
# core-service (if workers enabled)
kubectl rollout restart deploy -n ask -l app.kubernetes.io/instance=core-service

Verify: worker logs show the following sequence — no Socket closed:

STARTING HATCHET...
Connected to an older Hatchet engine that does not support multiple slot types.
Falling back to legacy worker registration. ...
STARTING HATCHET (legacy mode)...
starting runner...

The legacy worker registration fallback is expected with Hatchet v0.71.0 + a newer SDK — the SDK detected the engine version and fell back gracefully. Workers are running. No action needed unless you upgrade the engine.

Client-side HATCHET_CLIENT_HOST_PORT is also required

The Hatchet token embeds the external broadcast address (hatchet.ask.mod.auh1.dev.dir:443). In-cluster pods can't resolve the external hostname. Set HATCHET_CLIENT_HOST_PORT=hatchet-engine.ask.svc.cluster.local:7070 in envConf.extraEnv on each service that uses Hatchet workers — this overrides the address from the token so the SDK connects to the internal service.

Issue 6 — RabbitMQ-1 crash-loops: Peer discovery: no nodes available for auto-clustering

Symptom:

hatchet-rabbitmq-1 keeps restarting with Exit Code 0 (clean exit, not OOMKill). Logs show a repeated loop before exiting:

Peer discovery: no nodes available for auto-clustering; waiting before retrying...
Peer discovery: no nodes available for auto-clustering; waiting before retrying...

hatchet-rabbitmq-0 is healthy and has been running for hours. RabbitMQ-1 cannot join the cluster regardless of how many times it restarts.

Root cause — Erlang cookie mismatch:

RabbitMQ clustering uses Erlang Distribution, which requires every node to share the exact same Erlang cookie. If the cookies differ, Erlang refuses the connection and the joining node cannot discover any peers.

The mismatch occurred because:

RabbitMQ-0 started before existingErlangSecret was configured in the Bitnami chart values. Bitnami auto-generated a random cookie and persisted it to /var/lib/rabbitmq/.erlang.cookie on its PVC.
existingErlangSecret was later added, pointing to an ESO-managed Kubernetes secret pulled from Vault — but the Vault value was a different randomly generated cookie.
RabbitMQ-1 restarted and read its cookie from the Kubernetes secret → mismatch with RabbitMQ-0 → Erlang handshake refused → peer discovery timeout → clean exit → restart loop.

Diagnosis:

# RabbitMQ-0's real cookie (from PVC)
kubectl exec hatchet-rabbitmq-0 -n ask -- cat /var/lib/rabbitmq/.erlang.cookie

# What the Kubernetes secret says (ESO-managed)
kubectl get secret hatchet-rabbitmq-erlang -n ask \
  -o jsonpath='{.data.rabbitmq-erlang-cookie}' | base64 -d
echo ""

# Find the Vault path from the ExternalSecret
kubectl get externalsecret hatchet-rabbitmq-erlang -n ask -o yaml | grep key:

If the two cookie values differ — this is the cause.

Fix:

Update Vault to match RabbitMQ-0's real cookie (do NOT touch RabbitMQ-0 — it has live queue state on its PVC):

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
export VAULT_SKIP_VERIFY=true   # only if bastion doesn't trust the internal CA

vault kv patch secret/ask/hatchet/rabbitmq--ERLANG--COOKIE \
  password="<cookie from RabbitMQ-0>"

# Force ESO to sync immediately (don't wait for 1h refreshInterval)
kubectl annotate externalsecret hatchet-rabbitmq-erlang -n ask \
  force-sync=$(date +%s) --overwrite

# Verify secret updated
kubectl get secret hatchet-rabbitmq-erlang -n ask \
  -o jsonpath='{.data.rabbitmq-erlang-cookie}' | base64 -d
echo ""

# Restart RabbitMQ-1
kubectl delete pod hatchet-rabbitmq-1 -n ask

# Confirm cluster formed
kubectl exec hatchet-rabbitmq-0 -n ask -- rabbitmqctl cluster_status | grep -A5 "Running Nodes"

Lesson learned — pre-seed the Erlang cookie before first deploy:

existingErlangSecret is designed for fresh deployments. The cookie must be seeded in Vault before the first RabbitMQ pod starts so all nodes share it from day one. Retrofitting it onto a running cluster causes this mismatch.

✅ Correct order:  seed cookie in Vault → deploy RabbitMQ → all pods read same cookie
❌ Wrong order:    deploy RabbitMQ → auto-generate cookie → add existingErlangSecret later → mismatch

Issue 7 — Hatchet Engine: remaining connection slots reserved for SUPERUSER (SQLSTATE 53300)

Symptom:

Hatchet engine crash-loops with:

engine failure: could not run with config: could not create rebalance controller partitions job:
failed to connect to `user=hatchet database=hatchet`:
  hatchet-pg-rw.ask.svc.cluster.local: FATAL: remaining connection slots are reserved
  for roles with the SUPERUSER attribute (SQLSTATE 53300)

Root cause:

max_connections (default 100) exhausted by direct connections from Hatchet engine + API replicas. PostgreSQL reserves the last superuser_reserved_connections (default 3) slots for superusers only. The hatchet role is non-superuser → rejected.

Immediate fix (stopgap): raise max_connections in the CNPG Cluster CR:

# applications/ask-db/hatchet/hatchet-pg-cluster.yaml
spec:
  postgresql:
    parameters:
      max_connections: "200"

Commit + push. CNPG applies via rolling restart (primary switchover, no data loss).

Production fix: deploy CNPG Pooler (PgBouncer) CR — see 35 — Hatchet PgBouncer Pooler. Update DATABASE_URL in Vault to point to hatchet-pg-pooler.ask.svc.cluster.local:5432.

Why it happened: no connection pooler between Hatchet and PostgreSQL. 2 engine + 2 API replicas × their internal goroutine pools = 100+ connections at idle.

Issue 8 — ESO PushSecret + generatorRef Vault Unmarshal Loop (ESO 0.14.x)

Symptom:

All PushSecret resources with selector.generatorRef are Errored. ESO logs show a tight reconcile loop (30+ retries/second) with errors like:

error unmarshalling vault secret: invalid character 'X' looking for beginning of value
error unmarshalling vault secret: invalid character 'Y' after top-level value
error unmarshalling vault secret: invalid character 'H' in literal null (expecting 'u')

The offending character changes on every retry (y, G, T, P, C, O, U, p...) because ESO re-generates a new password on each reconcile attempt.

Root cause:

A bug in the ESO 0.14.x Vault provider's PushSecret code path: when selector.generatorRef is used, the generated secret bytes are accidentally passed to the JSON unmarshal call that is supposed to parse the Vault HTTP response. The password is treated as the Vault response body. Vault is never actually called. Restarting the ESO deployment has no effect — it is a code-path bug, not a stale-token issue.

Affected versions: ESO 0.14.x confirmed. May be fixed in later versions.

Fix — manual Vault seed + ExternalSecrets only:

Remove all PushSecret resources from Git (keep ExternalSecret resources).

Seed Vault manually from vault-0:

BOOT_TOKEN=$(kubectl get secret -n vault vault-bootstrap-token \
  -o jsonpath='{.data.token}' | base64 -d)

kubectl exec -n vault vault-0 -- sh -c "
export VAULT_ADDR=https://127.0.0.1:8200
export VAULT_SKIP_VERIFY=true
export VAULT_TOKEN='$BOOT_TOKEN'
gen() { cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 24; }
vault kv put -mount=secret ask/postgres/core-pg--SUPERUSER--PASSWORD           password=\$(gen)
vault kv put -mount=secret ask/postgres/core-service--DATABASE--PASSWORD        password=\$(gen)
vault kv put -mount=secret ask/postgres/zitadel--DATABASE--PASSWORD             password=\$(gen)
vault kv put -mount=secret ask/postgres/assistant-service--DATABASE--PASSWORD   password=\$(gen)
vault kv put -mount=secret ask/postgres/hatchet-pg--SUPERUSER--PASSWORD         password=\$(gen)
vault kv put -mount=secret ask/postgres/hatchet--DATABASE--PASSWORD             password=\$(gen)
vault kv put -mount=secret ask/postgres/knowledge-pg--SUPERUSER--PASSWORD       password=\$(gen)
vault kv put -mount=secret ask/postgres/knowledge-management-service--DATABASE--PASSWORD password=\$(gen)
"

ExternalSecrets read from Vault and create K8s Secrets. CNPG clusters deploy normally.

Security posture: unchanged — passwords are random, live only in Vault, never in Git. The only difference is they were written by an operator rather than by ESO automatically.

Future upgrade path: upgrade ESO to a version with the fix, re-add PushSecrets, and automatic generation is restored.

Issue 9 — Foundry Model Download Job: argument of type 'NoneType' is not iterable

Namespace: foundry
Job: download-model-<model-name>
Symptom:

fatal error: argument of type 'NoneType' is not iterable

Job hits BackoffLimitExceeded immediately — no files are downloaded.

Root Cause The error message is misleading. The actual cause is InvalidAccessKeyId (HTTP 403) returned by Ceph RGW. AWS CLI v2 has a bug in s3errormsg.py where it tries to check if an error message contains a sigv4 hint — but when the RGW error body has an empty <Message/> field, enhance_error_msg receives None and throws TypeError: argument of type 'NoneType' is not iterable instead of surfacing the real 403 error.

Debug log confirms this:

https://s3.cl1.sq4.aegis.internal:443 "GET /foundry-models?list-type=2&..." 403 210
<Error><Code>InvalidAccessKeyId</Code><Message></Message>...</Error>
...
TypeError: argument of type 'NoneType' is not iterable

Diagnosis Steps

# 1. Run with --debug to see the real HTTP response
kubectl logs -n foundry job/<job-name> --previous

# 2. Test credentials from bastion directly
AWS_ACCESS_KEY_ID=<key> AWS_SECRET_ACCESS_KEY=<secret> \
  aws s3 ls s3://foundry-models \
  --endpoint-url https://s3.cl1.sq4.aegis.internal \
  --region us-east-1 \
  --no-verify-ssl

Fix The Kubernetes secret foundry-secret has stale/wrong credentials. Update it with the working credentials:

# Assumes correct creds are already exported on bastion
kubectl create secret generic foundry-secret -n foundry \
  --from-literal=AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  --from-literal=AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  --dry-run=client -o yaml | kubectl apply -f -

Verify:

kubectl get secret foundry-secret -n foundry \
  -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d && echo

Then delete and re-apply the job.

Additional Notes - Model already downloaded? Check the PVC before re-running. If files exist with correct timestamps, the model is complete — the job failure has no functional impact:

kubectl exec -n foundry <vllm-pod> -- ls -lh /models/<model-name>/

aws s3 sync vs aws s3 cp --recursive: Both fail with the same misleading error when credentials are wrong. Always confirm credentials first.
--region us-east-1 required: Ceph RGW requires an explicit region for AWS CLI v2. Add --region us-east-1 or set AWS_DEFAULT_REGION=us-east-1 in the job spec.
DNS resolution from pods: s3.cl1.sq4.aegis.internal resolves to 100.115.1.130 (HAProxy VIP) from within the cluster. TLS handshake completes with -k / --no-verify-ssl. DNS is not the issue.

Issue 10 — core-service: ArgoCD “no such file or directory” for environments/values-mod-auh1-dev-ask.yaml

Symptom:

Failed to load target state: failed to generate manifest for source 1 of 2:
Manifest generation error (cached): failed to execute helm template command:
Error: open <path>/helm/environments/values-mod-auh1-dev-ask.yaml: no such file or directory

Root cause: The ArgoCD Application spec (argocd-application-core-service.yaml in mod-ask-devops) was updated to reference environments/values-mod-auh1-dev-ask.yaml from within the chart source, but the file was committed locally to ask-core-service and not yet pushed to GitLab. ArgoCD pulled the new Application spec but couldn't find the file in the chart repo.

Fix: Push the local commit that adds the env values file:

# On Mac (local repo)
cd /path/to/mod-ask-core-service
git log --oneline -3   # confirm d682812 "feat(lab): add mod-auh1-dev-ask environment values" is there
git push origin main

Then trigger an ArgoCD refresh:

# From bastion
argocd app get mod-auh1-dev-ask.ask.core-service --hard-refresh
# or via UI: click Refresh → Hard Refresh

Why this pattern:
The helm/environments/values-mod-auh1-dev-ask.yaml file lives inside the chart repo (matching the production pattern for all other environments). ArgoCD Source 1 picks it up as part of the Helm path. Source 2 (mod-ask-devops) provides additional devops-level overrides stacked on top. Both must be present before ArgoCD can render manifests.

Prevention: Always push the chart repo commit before pushing the ArgoCD Application spec update that references a new file in that repo.

Issue 11 — core-service: MountVolume.SetUp failed — configmap “core-service-config-toml” not found

Symptom:

MountVolume.SetUp failed for volume "config-toml": configmap "core-service-config-toml" not found

Root cause: The Helm template configmap-config-toml.yaml only creates the ConfigMap when envConf.configToml is non-empty:

{{- if .Values.envConf.configToml }}

If ArgoCD renders the chart with an empty configToml (e.g. because the env values file was missing), the ConfigMap is not created. However, the Deployment template has the same conditional for the volume mount — so if both are rendered from the same sync, they should match.

This error appears when: - The values file referencing configToml was not yet applied (file missing from GitLab) - The ArgoCD Application spec pointed to a non-existent values file — see the "no such file or directory" entry above

Fix: Resolve the manifest generation error first (push the missing env values file). Once ArgoCD can render successfully, the ConfigMap is created and the pod mounts it on next restart.

Issue 12 — core-service: ESO PushSecret 403 permission denied on secret/ask/core-service/*

Symptom:

set secret failed: could not verify if secret exists in store: cannot read secret data from Vault:
Error making API request. URL: GET https://vault.vault.svc.cluster.local:8200/v1/secret/data/ask/core-service/auh1-dev/SECURITY__API_KEY_SALT
Code: 403. Errors: * permission denied

Root cause: ESO authenticates to Vault via the Kubernetes auth role eso, whose attached policy eso-ask-push enumerates each ask/<service>/* path explicitly. When a new service (core-service) is added, its paths are not yet in the policy, so ESO's existence-check GET (PushSecret updatePolicy: IfNotExists) returns 403. This is a known bootstrap step — each new ask/* namespace needs a policy grant.

Fix (GitOps): add the core-service paths to eso-ask-push in applications/cluster-addons/vault/manifests/vault-bootstrap-config.yaml:

path "secret/data/ask/core-service/*"     { capabilities = ["create", "update", "read"] }
path "secret/metadata/ask/core-service/*" { capabilities = ["read", "list"] }

Then re-apply the policy by syncing the vault app (the policy write is idempotent / overwrites):

argocd app sync vault --grpc-web
# verify the policy now lists core-service:
vault policy read eso-ask-push | grep core-service

Then a second error appears — invalid JSON format:

set secret failed: could not write remote ref password to target secretstore vault-backend:
error unmarshalling vault secret: invalid JSON format

This is the brand-new-path bug: ESO's existence check on a path that has never existed gets Vault's {"errors":[]} 404 body, which it cannot unmarshal. Seed every path once (the auto-generated ones too) — afterwards the IfNotExists PushSecrets are no-ops:

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
vault kv put secret/ask/core-service/auh1-dev/SECURITY__API_KEY_SALT           password="$(openssl rand -hex 32)"
vault kv put secret/ask/core-service/auh1-dev/SECURITY__MASTER_ENCRYPTION_KEY  password="$(openssl rand -base64 32 | tr '+/' '-_')"   # Fernet key (32 urlsafe-base64 bytes)
vault kv put secret/ask/core-service/auh1-dev/APP__CORE_SERVICES_MACHINE_TOKEN password="$(openssl rand -hex 32)"
vault kv put secret/ask/core-service/auh1-dev/ZITADEL_MASTERKEY                password="$(openssl rand -hex 16)"   # MUST be exactly 32 bytes
vault kv put secret/ask/core-service/auh1-dev/IAM_SERVICE__PAT      value="placeholder"
vault kv put secret/ask/core-service/auh1-dev/HATCHET__CLIENT_TOKEN value="placeholder"

Order matters: (1) fix the policy → 403 resolved; (2) seed all paths → invalid JSON format resolved; (3) force-sync the secrets app → ExternalSecrets pull and core-service-secret / zitadel-secret materialise. Same class of issue as the hatchet invalid JSON format row above. See Core Service → Sync-Wave & Ordering for the full prerequisite list.

Issue 13 — core-service (Zitadel): docker.io/alpine/k8s / docker.io/wait4x image pull fails (air-gap)

Symptom:

Failed to pull image "docker.io/alpine/k8s:1.33.7": ... not found
# (and/or) docker.io/wait4x/wait4x:3.6

Root cause: The upstream Zitadel subchart uses two tool images from Docker Hub — alpine/k8s (a kubectl helper, used by the setup and cleanup jobs) and wait4x/wait4x (an init-container DB waiter on the zitadel + login Deployments). The air-gapped cluster cannot reach docker.io.

Key detail: these tool images are built from imageRegistry + tools.<x>.image.repository, while the main zitadel/login images use image.repository directly. So overriding imageRegistry redirects only the tool images — the main images (already on Harbor) are untouched.

Fix: in the env values (zitadel.zitadel.imageRegistry):

zitadel:
  zitadel:
    imageRegistry: harbor.cl1.sq4.aegis.internal/dockerhub

This yields harbor.cl1.sq4.aegis.internal/dockerhub/alpine/k8s:<kubeVersion> and …/dockerhub/wait4x/wait4x:3.6. All four tool jobs already carry harbor-pull-secret.

Prerequisite: the two images must exist in Harbor's dockerhub mirror. If the dockerhub project is a pull-through proxy cache, they fetch on demand. Otherwise mirror them once via scripts/harbor-mirror/mirror-config.yaml (dockerhub project) + core42-mirror-image.py --only:

- { name: alpine/k8s,    source: docker.io, project: dockerhub, tags: ["1.33.7"] }
- { name: wait4x/wait4x, source: docker.io, project: dockerhub, tags: ["3.6"] }

(The kubectl tag defaults to the cluster's Kubernetes version — 1.33.7 here.)

Issue 14 — core-service (Zitadel): setup job fails — masterkey must be 32 bytes, but is 44

Symptom (zitadel-setup job log):

migration failed ... error="cannot start key storage: ID= Message=masterkey must be 32 bytes, but is 44" name=03_default_instance
... level=fatal msg="setup failed, skipping cleanup"

Root cause: Zitadel's masterkey must be exactly 32 bytes, read as a raw string. Seeding ZITADEL_MASTERKEY with openssl rand -base64 32 produces a 44-character string (32 bytes of entropy → 44 base64 chars), so Zitadel counts 44 and aborts.

Fix: re-seed with a value that is literally 32 characters, e.g. openssl rand -hex 16 (16 bytes → 32 hex chars = 32 bytes):

vault kv put secret/ask/core-service/auh1-dev/ZITADEL_MASTERKEY password="$(openssl rand -hex 16)"
vault kv get -field=password secret/ask/core-service/auh1-dev/ZITADEL_MASTERKEY | wc -c   # → 33 (32 + newline)

kubectl annotate externalsecret -n ask zitadel-secret force-sync="$(date +%s)" --overwrite
kubectl delete job -n ask core-service-zitadel-setup core-service-zitadel-init 2>/dev/null
# re-sync the core-service app to re-run the hooks

Safe to change: because setup fails before any data is encrypted, the masterkey has no dependent ciphertext yet — re-seeding is harmless. The "never rotate the masterkey" rule only applies after Zitadel has successfully bootstrapped and started encrypting events.

Issue 15 — core-service: Fernet key must be 32 url-safe base64-encoded bytes (runtime, not startup)

Symptom: core-service boots and passes health checks, then later throws when a feature that encrypts data is used (adding a connector, a Foundry API key, org SMTP creds, triggers):

ValueError: Fernet key must be 32 url-safe base64-encoded bytes.

Root cause: SECURITY__MASTER_ENCRYPTION_KEY is passed directly to cryptography.fernet.Fernet(key) (encryption_utils.py src/core_service/utils/encryption_utils.py). Fernet requires the key to be exactly 32 url-safe base64-encoded bytes — i.e. the output of Fernet.generate_key() (a 44-char string). A value like openssl rand -hex 32 (64 hex chars) is not valid. The check is lazy — Fernet(key) runs only inside encrypt_data/decrypt_data, so the app starts and migrates fine and only fails when encryption is first invoked.

Fix: re-seed with a Fernet-format key:

vault kv put secret/ask/core-service/auh1-dev/SECURITY__MASTER_ENCRYPTION_KEY \
  password="$(openssl rand -base64 32 | tr '+/' '-_')"
kubectl annotate externalsecret -n ask core-service-secret force-sync="$(date +%s)" --overwrite

Verify it decodes to 32 bytes: python3 -c "import base64,sys;print(len(base64.urlsafe_b64decode(sys.argv[1]))==32)" "<key>".

Secret length requirements — get these right at seed time

Vault key	Required format	Generate with
`ZITADEL_MASTERKEY`	exactly 32 bytes (raw string)	`openssl rand -hex 16`
`SECURITY__MASTER_ENCRYPTION_KEY`	Fernet key (32 urlsafe-b64 bytes)	`openssl rand -base64 32 \\| tr '+/' '-_'`
`SECURITY__API_KEY_SALT`	any non-empty string	`openssl rand -hex 32`
`APP__CORE_SERVICES_MACHINE_TOKEN`	any opaque token	`openssl rand -hex 32`

The old password-alphanumeric-24 ClusterGenerator (24 chars) could not satisfy either of the two fixed-length keys — another reason this deployment seeds these manually (see Sync-Wave & Ordering).

Issue 16 — core-service: HPA unable to get target's current scale: deployments “core-service” not found

Symptom:

the HPA controller was unable to get the target's current scale: deployments.apps "core-service" not found

Root cause: sync-wave gap. The Deployment was given sync-wave: "5" (so it lands after secrets/config/migration), but the HPA was left at the default wave 0 — so ArgoCD creates the HPA five waves before its scaleTargetRef Deployment exists. The HPA controller then can't find the Deployment, and (worse) the HPA's unhealthy state can stall wave 0, so ArgoCD never advances to wave 5 to create the Deployment → deadlock.

Fix: give the HPA a wave after the Deployment:

# chart values.yaml
argocd:
  syncWave:
    workload: "5"   # Deployment
    hpa: "6"        # HPA — created after the Deployment exists

The HPA template emits argocd.argoproj.io/sync-wave: "6". After re-sync, wave 0 no longer contains the HPA, so the phase chain completes: migration (4) → Deployment (5) → HPA (6), and the HPA resolves its target cleanly.

Rule of thumb: any resource that references another by name/scale (HPA → Deployment, PodMonitor → Service) must sync in a wave ≥ its target's wave.

Issue 17 — core-service: migration job — Dynaconf can't interpolate variable because 'APP__ENV'

Symptom (migration job log):

dynaconf.utils.parse_conf.DynaconfFormatError: Dynaconf can't interpolate variable because 'APP__ENV'
KeyError: 'APP__ENV'

Root cause: config/default.toml contains env = "@format {env[APP__ENV]}", so Dynaconf needs the APP__ENV environment variable at import time. The main Deployment supplies it from .Values.extraEnv, but the migration job template (job-migrations.yaml) only set envFrom (the secret) and CONFIG_MAP_REF — it did not inject extraEnv. So alembic imports config, Dynaconf evaluates default.toml, and APP__ENV is unset → KeyError.

Fix: inject extraEnv into the migration job, same as the Deployment:

# job-migrations.yaml
          env:
            {{- with .Values.extraEnv }}
            {{- toYaml . | nindent 12 }}
            {{- end }}
            ...

With extraEnv: [{name: APP__ENV, value: mod-auh1-dev-ask}] the job resolves the env section and runs alembic upgrade head.

General rule: any Job/initContainer that imports the app's config module needs the same APP__ENV + envFrom secret + config-toml mount as the main container — otherwise Dynaconf fails to interpolate the baked-in @format {env[...]} tokens.

Issue 18 — core-service: Pydantic ValidationError on config (storage / default_org_id / notifications)

Symptom (migration or app startup):

pydantic_core._pydantic_core.ValidationError: 3 validation errors for Config
STORAGE.type        Input should be 'azure', 'minio' or 's3' [input_value='stub']
IAM_SERVICE.DEFAULT_ORG_ID   Field required
NOTIFICATIONS       email_provider is 'smtp' but smtp_host is not set.

Root cause: the app validates the entire config with Pydantic at import time (Config.model_validate(_raw)), so every required field / literal / cross-field rule must hold — even for the migration job. The air-gap override TOML had three invalid values.

Fixes (in envConf.configToml):

Field	Rule (from `src/config/schemas/`)	Deployed value
`storage.type`	`Literal["azure","minio","s3"]` + matching sub-block required	`s3` + `[storage.s3]` (region, bucket_name, endpoint_url → Ceph RGW; keys optional)
`iam_service.default_org_id`	required, `min_length=1` (not in `default.toml`)	placeholder until the real Zitadel org ID is known (set with the PAT after boot)
`notifications.smtp_host`	required when `email_provider="smtp"`	placeholder host (no SMTP relay in the air-gap environment)

Pre-flight: validate the merged config locally before deploying

Catch these without a cluster round-trip — load default.toml + your override.toml through the real loader:

python -m venv .venv && .venv/bin/pip install -q dynaconf pydantic
mkdir -p config && cp <repo>/config/default.toml config/
#  extract envConf.configToml → config/override.toml
export APP__ENV=mod-auh1-dev-ask DATABASE__PASSWORD=x SECURITY__API_KEY_SALT=x \
       SECURITY__MASTER_ENCRYPTION_KEY=x APP__CORE_SERVICES_MACHINE_TOKEN=x \
       IAM_SERVICE__PAT=x HATCHET__CLIENT_TOKEN=x
PYTHONPATH=<repo>/src .venv/bin/python -c "from config import config; print('valid')"

Any missing/invalid field prints the exact Pydantic error — fix the TOML, re-run, then push.

Follow-ups (don't block bring-up, but needed for full function): - Wire Ceph S3 credentials (bucket + S3 user → Vault → core-service-secret → STORAGE__S3__*) for file features. - Replace default_org_id + IAM_SERVICE__PAT placeholders with real values after Zitadel first boot. - Point smtp_host at a real relay if email is needed.

Issue 19 — core-service: migration — schema “core_service” does not exist

Symptom (migration job):

psycopg2.errors.InvalidSchemaName: schema "core_service" does not exist
CREATE TABLE core_service.alembic_version ( ... )

Root cause: the app's database.default_schema = "core_service", so Alembic creates its tables (starting with alembic_version) in the core_service schema — but Alembic does not create the schema itself, it expects it to exist. The postgresInit job created the database, role, and set up public, but not the custom core_service schema.

Fix: have the init job create the schema (owned by the app role). Added postgresInit.db.schema and a guarded CREATE SCHEMA to the init script:

\connect "core-service"
...
CREATE SCHEMA IF NOT EXISTS "core_service" AUTHORIZATION "core-service";

# env values
postgresInit:
  db:
    schema: core_service        # must match database.default_schema in the TOML

Quick unblock (the DB-init job already ran): create the schema by hand, then re-run migration:

kubectl exec -n ask core-pg-1 -- \
  psql -U postgres -d core-service -c 'CREATE SCHEMA IF NOT EXISTS core_service AUTHORIZATION "core-service";'
kubectl delete job -n ask core-service-migrations 2>/dev/null
# re-sync the app to re-run the migration hook

A full re-sync also fixes it permanently — the PreSync DB-init hook re-runs (

CREATE SCHEMA IF NOT
EXISTS

is idempotent) before the migration.

Issue 20 — core-service (and any ai71 service): pod Pending — didn't match Pod's node affinity/selector

Symptom:

0/17 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master},
7 node(s) didn't match Pod's node affinity/selector,
7 node(s) had untolerated taint {nvidia.com/gpu}.

Root cause: the ai71 charts ship nodeSelector: node.kubernetes.io/role: ask71 (a dedicated worker pool from the original ai71 cluster). This cluster's nodes aren't labelled ask71, so nothing matches. The 3 masters and 7 GPU nodes are tainted (correctly excluded); the 7 general k8s-worker nodes are Ready/untainted but unlabelled.

Fix — label the cluster, not the charts. Add the ask71 role label to the 7 worker nodes so the charts' built-in selector matches. This fixes every ai71 service at once (no per-chart nodeSelector edits):

for n in 1 2 3 4 5 6 7; do
  kubectl label node k8s-worker${n}.cl1.sq4.aegis.internal node.kubernetes.io/role=ask71 --overwrite
done

The Pending pod schedules automatically once a matching node exists — no re-sync needed.

Label, don't taint — and the empty-map merge trap

Don't taint the workers ask71: that would force every pod in the namespace (CNPG, hatchet, etc.) to carry the toleration. Masters + GPU nodes are already tainted, so general workloads can only land on the 7 workers regardless — a label is enough.
nodeSelector: {} in an env values file does NOT clear a non-empty chart default — Helm merges maps (keeps base keys), unlike lists (tolerations: [] does replace). So you can't strip a nodeSelector from an override; fix it at the base or label the nodes.

Issue 21 — core-service: org/auth endpoints 500 Foundry is not configured (+ 4 GitOps/config gotchas)

Symptom: core-service boots healthy (/health/ready 200), but every org/auth endpoint — including the frontend login's GET /api/v1/organizations/discovery — returns 500 Internal server error. The pod log shows:

File ".../organization/organization_service.py", line 268, in __init__
    self.foundry_client = get_foundry_client()
File ".../configs/foundry/settings.py", line 23, in get_foundry_client
    raise RuntimeError("Foundry is not configured. Provide [foundry] in TOML or set FOUNDRY_* env vars.")

Root cause: OrganizationService.__init__ eagerly builds the Foundry admin client (the one that provisions per-user/per-org Foundry keys via the LiteLLM admin API). If config.foundry is None it raises — and because the build is in a constructor on the org dependency path, every org/auth request 500s while health stays green (health doesn't touch the org service). core-service was deployed before Foundry existed, so the [foundry] block was never added.

The fix (two parts):

Add a [<env>.foundry] block to core-service's live config (the devops $values overlay — see gotcha 3) with all 5 required fields:

[mod-auh1-dev-ask.foundry]
base_url = "http://foundry-litellm.foundry.svc.cluster.local:4000"   # LiteLLM gateway ROOT (admin at /, not /v1)
api_key = "@format {env[FOUNDRY_ADMIN_KEY]}"                          # see gotcha 2 for the var name
default_tpm = 1000000
default_rpm = 10000
default_max_budget = "@none"

Provide the admin key (= LiteLLM PROXY_MASTER_KEY) to core-service:
devops core-service-secrets.yaml ExternalSecret: add a data entry + template.data mapping to env var FOUNDRY_ADMIN_KEY.
seed Vault: vault kv put secret/ask/core-service/auh1-dev/FOUNDRY__API_KEY value="<master-key>".

The 4 gotchas hit while fixing this (each cost a deploy cycle) Gotcha 1 — FoundryConfig needs 5 fields, not 3. First attempt set only base_url/api_key/ default_tpm → boot ValidationError for default_rpm + default_max_budget. default_max_budget is float | None → use "@none".

Gotcha 2 — Dynaconf section-shadowing via SECTION__KEY env vars. Naming the key env var FOUNDRY__API_KEY (double underscore) makes Dynaconf auto-load it as the FOUNDRY section and replace the whole TOML [foundry] block (the config sees only FOUNDRY.API_KEY, dropping base_url/default_tpm). Use a single-underscore name (FOUNDRY_ADMIN_KEY) so it isn't parsed as the section, and let the TOML block supply the rest. (core-service's loader: Dynaconf(environments=True, merge_enabled=True, env_switcher="APP__ENV").)

Gotcha 3 — the devops $values overlay shadows the chart-repo env file. The core-service ArgoCD Application lists two valueFiles: environments/values-mod-auh1-dev-ask.yaml (chart repo) and $values/applications/core-service/values-mod-auh1-dev-ask.yaml (devops). The overlay is last, so its configToml replaces the chart file's entirely (strings don't merge). Editing the chart env file had no effect — the fix had to go in the devops overlay. ⚠️ For core-service, the authoritative config is the devops overlay; the chart env file is shadowed.

Gotcha 4 — subPath configmap mounts don't update live; ArgoCD force-sync reuses the git cache. - The configToml is mounted at /app/config/override.toml via subPath → a configmap change does not propagate to the running pod. You must kubectl rollout restart deploy/core-service. - argocd.argoproj.io/refresh=hard re-fetches git; a plain force-sync annotation / cached sync may apply a stale revision. After pushing, use hard refresh → sync → restart.

Diagnosing "Synced but config missing" (the decisive test) If ArgoCD shows Synced to your commit but the configmap lacks your change, render isn't coming from the file you edited. Confirm with a dummy marker:

# add a unique comment to the configToml you think is live, push, hard-refresh, sync, then:
kubectl get cm -n ask core-service-config-toml -o jsonpath='{.data}' | grep -c YOUR_MARKER

Marker absent ⇒ you edited a shadowed file (gotcha 3) — find the real valueFile in the Application's spec.sources[].helm.valueFiles (the last one wins).

Verify the fix

kubectl get cm -n ask core-service-config-toml -o jsonpath='{.data}' | grep -ci foundry   # >0
kubectl exec -n ask deploy/core-service -- grep -A6 -i foundry /app/config/override.toml   # block present
kubectl exec -n ask deploy/core-service -- printenv | grep FOUNDRY_ADMIN_KEY               # set
kubectl exec -n ask deploy/assistant-service-intelligence -- python3 -c \
  "import httpx; print(httpx.get('http://core-service.ask.svc.cluster.local/api/v1/organizations/discovery', params={'email':'x@y.z'}, timeout=10).status_code)"   # 200 (or 404 = endpoint works, no org for that email)

Confirmed fixed 2026-06-19

After the [foundry] block landed in the devops overlay + FOUNDRY_ADMIN_KEY in core-service-secret, discovery returned 404 {"detail":"Organization not found"} (was 500 Foundry is not configured). The 404 is the correct response — the endpoint works; that email is just not a member of any org. (Next step is org/user provisioning, not a fix.)

Issue 22 — core-service / Zitadel: Instance not found (unable to set instance using origin)

Symptom (on login / a Zitadel call after discovery):

unable to set instance using origin &{core-service-zitadel.ask.svc.cluster.local:8080 https}
(ExternalDomain is sso.ask.mod.auh1.dev.dir): Instance not found.
... unable to get instance by host: instanceHost core-service-zitadel.ask.svc.cluster.local:8080

Root cause: Zitadel is multi-tenant and resolves the instance by the request Host. The instance is registered for ExternalDomain = sso.ask.mod.auh1.dev.dir, but core-service connects to the internal service core-service-zitadel.ask.svc.cluster.local:8080. core-service splits the two:

[<env>.iam_service]
base_url        = "http://core-service-zitadel.ask.svc.cluster.local:8080"   # where it CONNECTS
public_base_url = "https://sso.ask.mod.auh1.dev.dir"                          # the instance DOMAIN

The auth API (iam_auth_api.py) sends the instance to Zitadel via headers built from public_base_url (x-zitadel-instance-host, x-zitadel-origin, X-Forwarded-Host, x-zitadel-public-host) — so it resolves. But base_http_client (used by POST /api/v1/auth/signup, public signup) omits those headers — the log shows origin &{core-service-zitadel...:8080 https} with an empty publicHost → Zitadel has only the internal Host → Instance not found. (Confirmed failing path 2026-06-19: public user signup.)

✅ Official fix — register the internal domain as a Zitadel Custom Domain (two-step toggle) There is no UI/API to add an instance domain here. Zitadel adds the configured ExternalDomain to the instance's domains via a DB migration on startup. So temporarily point ExternalDomain at the internal service name, let it migrate, then revert. (Per the AI71 core-service setup guide.)

In the zitadel subchart config (devops overlay applications/core-service/values-mod-auh1-dev-ask.yaml):

Step 1 — point ExternalDomain at the internal name, push, sync, wait:

zitadel:
  configmapConfig:
    ExternalSecure: false
    ExternalDomain: core-service-zitadel.ask.svc.cluster.local

# after push + ArgoCD sync, Zitadel redeploys + runs the migration that ADDS the domain.
kubectl rollout status deploy/core-service-zitadel -n ask --timeout=300s
# CONFIRM login is now broken (expected — proves the migration ran):
#   browsing https://sso.ask.mod.auh1.dev.dir should error "instance not found"

Step 2 — revert to the real external domain, push, sync:

zitadel:
  configmapConfig:
    ExternalSecure: true
    ExternalDomain: sso.ask.mod.auh1.dev.dir

After this redeploy, both sso.ask.mod.auh1.dev.dir and core-service-zitadel.ask.svc.cluster.local are in the instance's Custom Domains (visible in Console → Default settings → Domains). core-service's internal calls (signup, discovery) now resolve → no more "Instance not found".

It's a deliberate 2-commit dance

Don't stop after step 1 (login stays broken). The internal domain persists in the DB once migrated, so step 2 restores normal external login while keeping the internal domain registered.

Rejected alternatives (kept for context): adding the domain via Console (not exposed at org level); a hostAlias mapping sso… → Zitadel ClusterIP (works but pins a ClusterIP); base_url=https://sso… (needs internal-CA trust, and httpx ignores SSL_CERT_FILE so it's unreliable without a source change).

Diagnose which call path is missing the header:

kubectl logs -n ask deploy/core-service --tail=80 | grep -iE "instance|origin|zitadel|QUERY-"
# and confirm the instance's known domains in Zitadel (Console → Settings → Domains)

Issue 23 — core-service: seed job fails — null value in column “default_role_id” … not-null constraint

Symptom — the core-service-postgres-seed PreSync job (which inserts the default org + Super Admin so login/discovery work) fails:

ERROR:  null value in column "default_role_id" of relation "organizations" violates not-null constraint
DETAIL: Failing row contains (1, 378447542100166005, ask-default, ask.mod.auh1.dev.dir, active, …, null, …).

Root cause: the chart's seed INSERT into organizations is older than the live DB schema. A later Alembic migration added organizations.default_role_id as NOT NULL (FK → core_service.roles.id), but the seed SQL (and the official AI71 manual-SQL guide it was copied from) never set it → the row inserts NULL → constraint violation. The DB schema (built by the image's migrations) had moved ahead of the hand-written seed.

default_role_id is the org's default member role

It's the role auto-assigned to users joining the org — not the Super Admin's role. Roles: 1 ask71_admin / 2 organization_admin / 3 member. Use 3 (member) — least privilege, so public signups don't become admins. (The Super Admin still gets role_id 1 via seedData.user.roleId.)

Diagnose — find the real DB/schema (the DB is core-service with a hyphen; core_service underscore is the schema inside it) and list the required columns:

kubectl exec -n ask core-pg-1 -- psql -U postgres -c "\l"                      # DB = core-service
kubectl exec -n ask core-pg-1 -- psql -U postgres -d core-service -c "\d core_service.organizations"
kubectl exec -n ask core-pg-1 -- psql -U postgres -d core-service -c "SELECT id, name FROM core_service.roles ORDER BY id;"

Any not null column without a default that the seed omits will fail the same way — default_role_id was the only one (others like state, created_at, is_model_onboarding_completed carry defaults).

Fix — add default_role_id to the seed org INSERT (chart templates/job-postgres-init.yaml) and supply it via seedData.org.defaultRoleId: 3 (chart values.yaml + the devops overlay). Then re-run the seed (the old failed Job + its ConfigMap are stale, so delete first):

kubectl delete job       -n ask core-service-postgres-seed        --ignore-not-found
kubectl delete configmap -n ask core-service-postgres-seed-script --ignore-not-found
kubectl annotate application -n argocd mod-auh1-dev-ask.ask.core-service argocd.argoproj.io/refresh=hard --overwrite
argocd app sync mod-auh1-dev-ask.ask.core-service --grpc-web
kubectl logs -n ask job/core-service-postgres-seed --tail=30          # should complete clean

Verify — three rows land and discovery stops 404-ing:

kubectl exec -n ask core-pg-1 -- psql -U postgres -d core-service -c \
 "SELECT external_organization_id, name, default_role_id FROM core_service.organizations;"
# discovery now returns ask-default instead of 404 'Organization not found'

General lesson — hand-written seed SQL drifts from migration-built schemas

Whenever a chart ships static seed/INSERT SQL but the app builds its schema with migrations in the image, the two drift over time. Always read the live \d <table> before trusting seed SQL, and cover every NOT NULL-without-default column. See Onboarding Runbook §5.

Issue 24 — core-service: login 404 “User could not be found” (Zitadel username suffix wrong)

Symptom — POST /api/v1/auth/login (org_id + email + password) returns 404 {"detail":"User could not be found"}, even though org discovery for the same email returns 200 and the user exists in the core_service DB and in Zitadel.

Root cause — the Super Admin's Zitadel username has the wrong # suffix. core-service builds the Zitadel login name from email + org id (utils/auth_utils.py):

def fetch_username_from_email(email, org_id):  return email + "#" + org_id

and auth_service.login() creates the Zitadel session by that login name (CreateSessionRequestByLoginName(user=User(loginName=email + "#" + org_id))). So it expects the user's Zitadel preferred login name to be <email>#<ORG-id>. If the user was created with the user-id as the suffix (a common slip — the user-id and org-id look alike), Zitadel can't resolve the login name → "User could not be found". Discovery still works because it queries the DB by email (a different path), which is why the two disagree.

Fix — set the username suffix to the org Resource Id:

	Login name
❌ wrong (user-id suffix)	`superadmin@ask.mod.auh1.dev.dir#<USER-id>`
✅ correct (org-id suffix)	`superadmin@ask.mod.auh1.dev.dir#<ORG-id>`

Zitadel Console → the org → Users → the user → modify User Name (or Actions → Rename) → set <email>#<org-id> → Save. Then re-run login → expect 200 (or a password/MFA error, but not 404).

Discovery 200 + login 404 = a username/login-name problem, not a seeding problem

Discovery uses the DB (get_by_email_in_organization); login uses Zitadel (CreateSessionRequestByLoginName). When only login 404s, the DB seed is fine — the mismatch is in the Zitadel login name. The official AI71 guide's username format is <email>#<org-id> for exactly this reason.

Issue 25 — KMS + assistant-service-tooling: every request 401 Unauthorized — iam_client hits /ask71/v2/svc/... → 404

Symptom — authenticated KMS calls (/collections/, /documents/, /retrieve/) and assistant-service-tooling auth-gated endpoints all return 401:

401 Unauthorized: GET request failed: Client error '404 Not Found' for url
'http://core-service.ask.svc.cluster.local/ask71/v2/svc/api/v1/users/me'

The Bearer token is valid (core-service login returned it), but the services can't validate it.

Root cause — iam_client hardcodes the API prefix and appends it to services_base_url.

iam_client is a shared library used by KMS and assistant-service. Its path_constants.py hardcodes:

BASE_URL_V1 = "/ask71/v2/svc/api/v1"

and every request is built as url = services_base_url + BASE_URL_V1 + endpoint. There is no config knob to change BASE_URL_V1 — it is baked into the library.

With services_base_url = "http://core-service.ask.svc.cluster.local" (the internal ClusterIP), the call becomes http://core-service.ask.svc.cluster.local/ask71/v2/svc/api/v1/users/me — a path that core-service does not serve (core-service serves at /api/v1/... directly; the /ask71/v2/svc prefix only exists at the ingress level). Result: 404 → iam_client treats the token as invalid → every request 401s.

Fix — gateway ingress + point services_base_url at it.

Add a second Ingress on core-service that accepts the iam_client path and rewrites the prefix off:

# mod-ask-core-service/helm/templates/ingress-gateway.yaml (values-driven)
# gatewayIngress.enabled: true in the devops overlay
#   host: api.ask.mod.auh1.dev.dir
#   path: /ask71/v2/svc(/|$)(.*)
#   rewrite-target: /$2
#   ssl-redirect: "false"   ← HTTP; avoids internal-CA TLS trust issue for in-cluster callers

Then point both KMS and assistant-service at the gateway host:

# KMS: mod-ask-knowledge-management-service/helm/environments/values-mod-auh1-dev-ask.yaml
[mod-auh1-dev-ask.core_service]
services_base_url = "http://api.ask.mod.auh1.dev.dir"
# fastapi_root_path = default "/ask71/v2/svc" — matches iam_client's hardcoded prefix

# assistant-service: mod-ask-assistant-service/helm/environments/values-mod-auh1-dev-ask.yaml
[mod-auh1-dev-ask.core_service]
url = "http://core-service.ask.svc.cluster.local/api/v1"   # unchanged — direct client
services_base_url = "http://api.ask.mod.auh1.dev.dir"      # iam_client base

Now both iam_client and the native CoreServiceClient build URLs at the gateway that the ingress can rewrite: http://api.ask.mod.auh1.dev.dir/ask71/v2/svc/api/v1/users/me → nginx strips prefix → core-service /api/v1/users/me → 200.

Resync core-service + KMS + assistant-service and roll the pods (subPath configmaps aren't hot-reloaded).

Why HTTPS and not HTTP to the ingress?

iam_client uses httpx.AsyncClient() with default TLS verification — it WILL check the cert. The ingress cert is signed by internal-ca-issuer, which is not in the pod's system trust store by default. Mounting the internal CA cert via a ConfigMap + SSL_CERT_FILE env var lets httpx verify the cert without code changes. Python's ssl module respects SSL_CERT_FILE natively; httpx inherits this. This mirrors the PP pattern exactly (PP uses a publicly-trusted cert; we use the internal CA — same mechanism, different trust anchor).

One-time setup — create the CA ConfigMap on the bastion:

CA_SECRET=$(kubectl get clusterissuer internal-ca-issuer -o jsonpath='{.spec.ca.secretName}')
kubectl get secret -n cert-manager "$CA_SECRET" -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/internal-ca.crt
# Verify it is the CA (should show CA:TRUE)
openssl x509 -in /tmp/internal-ca.crt -noout -text | grep "CA:"
kubectl create configmap internal-ca-cert -n ask --from-file=ca.crt=/tmp/internal-ca.crt \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl get configmap -n ask internal-ca-cert

KMS and assistant-service mount it at /etc/ssl/certs/internal-ca.crt and set SSL_CERT_FILE to that path via extraEnv. SSL_CERT_FILE replaces the system CA bundle — safe in air-gap where no public HTTPS endpoints are called from these pods.

Audit (2026-06-23) — both KMS and assistant-service-tooling affected

Caller	iam_client base URL	Result before fix	Result after fix
KMS	`services_base_url` → `http://core-service.ask.svc.cluster.local`	🔴 401 (404 on `/ask71/v2/svc/...`)	✅ gateway rewrites → `/api/v1/...`
assistant-service-tooling	same `services_base_url`	🔴 401 (same 404)	✅ same fix
frontend	`CORE_SERVICE_URL` + `basePath="/api/v1"` (`proxies/core`)	✅ no prefix	✅ unchanged

Confirmed 2026-06-22 — the two failure codes pinpoint the layer

Proven by creating a fresh testadmin user with the correct email#<org-id> username + a DB row, then logging in → 200 + session token. The two distinct errors map cleanly to the two checks in auth_service.login():

Login result	Failing check	Fix
`401 "User not found in organization"`	DB `get_by_email_in_organization` (line 163)	user missing from `core_service.users` → seed the row
`404 "User could not be found"`	Zitadel `create_session` by loginName (line 183)	Zitadel username ≠ `email#<org-id>` → rename the user
`200` + `sessionToken`	both pass	—

A Console-created user (e.g. via the Email-Verified checkbox) is not auto-synced into the core-service DB, so it needs a manual users + user_roles insert (external_user_id = the Zitadel user id, organization_id = the external org id, role_id 1 = ask71_admin) or login 401s.

Issue 26 — frontend: login redirects to admin.ask.stg.ai71.ai → DNS_PROBE_FINISHED_BAD_CONFIG

Symptom — the frontend loads https://ask.mod.auh1.dev.dir/login (blank form), then after entering an email it redirects the browser to https://admin.ask.stg.ai71.ai (the staging domain), which doesn't resolve in air-gap → Chrome DNS_PROBE_FINISHED_BAD_CONFIG.

Root cause — a baked NEXT_PUBLIC_* value (staging image). We deployed the frontend-stg:staging image. Next.js inlines NEXT_PUBLIC_* at build time (client and server bundles), so the staging build has NEXT_PUBLIC_APP_BASE_URL=https://ask.stg.ai71.ai hardcoded in the JS — no runtime env can override it. The tenant-redirect builds the target host from that baked value (apps/web/lib/auth/utils.server.ts):

getTenantDomainUrl(tenantSubdomain) → `${protocol}://${tenantSubdomain}.${primaryDomain}`
//   primaryDomain = new URL(process.env.NEXT_PUBLIC_APP_BASE_URL).host   // BAKED = ask.stg.ai71.ai

Our ask-default org has tenant_subdomain = admin, so → admin.ask.stg.ai71.ai. (PP avoids this with a frontend-prod image baked for ask.pp.gov.ae.)

Fix — rebuild the image with the air-gap base URL. No runtime workaround exists. AI71 builds with an air-gap config/<env>.env (the build does --build-arg ENV_FILE=config/<env>.env), minimally:

NEXT_PUBLIC_APP_BASE_URL=https://ask.mod.auh1.dev.dir
NEXT_PUBLIC_APP_ENV=production
NEXT_PUBLIC_FF_BILLING_METERING_ENABLED=false     # billing off (match core-service/assistant)
NEXT_PUBLIC_FF_PUBLIC_USER_SIGNUP_ENABLED=true     # public signup on
NEXT_PUBLIC_FF_DISABLE_VOICE_MODE=true             # voice not deployed
NEXT_PUBLIC_FF_DISABLE_SPEECH_TO_TEXT=true
NEXT_PUBLIC_AMPLITUDE_ENABLED=false                # no internet analytics
NEXT_PUBLIC_SENTRY_ENABLED=false                   # no external Sentry
NEXT_PUBLIC_GOOGLE_MAPS_API_KEY=                    # no internet

Then on our side: mirror the new image → Harbor, bump image.repository/tag in values-mod-auh1-dev-ask.yaml, and add ingress host + DNS + cert for the tenant subdomain(s) — admin.ask.mod.auh1.dev.dir (and chat. if used), mirroring PP's multi-host ingress.

General lesson — NEXT_PUBLIC_* is baked, never runtime

Any value referenced as process.env.NEXT_PUBLIC_X is inlined into the bundle at next build. Setting it in the Deployment env does nothing. For air-gap, every public URL/flag/domain must be correct at image build time — so a staging image cannot be re-pointed at an air-gap domain without a rebuild. See Air-Gap Hardcoded-Value Audit.

Issue 27 — core-service login: 404 “membership not found” after ArgoCD sync / pod restart

Symptom — POST /api/v1/auth/login returns 404 {"detail":"membership not found"} even though the user exists in both Zitadel and the core-service DB, and the password is correct (Zitadel audit shows "Password check succeeded").

Root cause — CORE_SERVICE machine user is created without IAM_OWNER. The core-service Helm chart ships two Kubernetes Jobs:

Job	What it does
`core-service-zitadel-init`	Bootstraps Zitadel instance config; creates `core-service` (lowercase) machine user with IAM_OWNER
`core-service-zitadel-setup`	Creates `CORE_SERVICE` (uppercase) machine user + PAT → stores PAT in k8s secret as `IAM_SERVICE__PAT`

The setup job creates CORE_SERVICE but does not grant it IAM_OWNER. The core-service application uses IAM_SERVICE__PAT to call Zitadel management APIs during login — with no membership, every Zitadel API call fails with AUTHZ-cdgFk → core-service surfaces this as 404 "membership not found".

This is a ONE-TIME fix. Once IAM_OWNER is granted via the Zitadel UI:

The membership is persisted in the Zitadel PostgreSQL DB (projections.instance_members4 on core-pg)
ArgoCD syncs do not remove it — the setup job only creates CORE_SERVICE if it doesn't already exist (idempotent); it never deletes existing memberships
Status (2026-06-23): fixed via UI ✅ — membership will survive future syncs and upgrades

Re-apply only if: - Full cluster rebuild where the Zitadel DB (core-pg) is wiped (DR scenario) - Someone explicitly deletes CORE_SERVICE from the Zitadel UI

Diagnosis:

# Confirm CORE_SERVICE has no Zitadel membership
kubectl exec -i -n ask core-pg-1 -- psql -U postgres -d "zitadel" -c \
  "SELECT user_id, roles FROM projections.instance_members4
   WHERE user_id IN ('378447770320635253','377545578533356354');"
# Expected broken state: only 377545578533356354 (core-service lowercase) has {IAM_OWNER}
# CORE_SERVICE (378447770320635253) shows 0 rows

Fix (manual — Zitadel admin UI):

Log into Zitadel admin (https://sso.ask.mod.auh1.dev.dir)
Navigate to IAM → Users → click CORE_SERVICE (ID: 378447770320635253)
Go to Memberships tab → Add → type IAM → select Iam Owner → save

Also fix superadmin if missing:
Same symptom affects user logins. If superadmin or any user shows "membership not found", check their Zitadel org membership: IAM → Organizations → ask-default → Members.

Verify fix:

kubectl exec -i -n ask deploy/assistant-service-intelligence -- env PW="$SUPERPW" python3 - <<'EOF'
import os, httpx
r=httpx.post("http://core-service.ask.svc.cluster.local/api/v1/auth/login", json={
    "org_id":"378447542100166005",
    "email":"superadmin@ask.mod.auh1.dev.dir",
    "password":os.environ["PW"]}, timeout=20)
print(r.status_code, r.json().get("sessionToken","FAILED")[:30])
EOF
# Expected: 200 <token>

Decision (2026-06-23): accepted as a one-time manual step.

The setup job gap is a known deviation from full GitOps automation. For this environment, the IAM_OWNER grant was applied once via the Zitadel UI and will persist for the life of the cluster. No chart change is planned. On DR rebuild, re-apply via the same UI steps above.

Note: There are two machine users in Zitadel: - core-service (377545578533356354) — created by init job, IAM_OWNER, no PAT stored anywhere (unused) - CORE_SERVICE (378447770320635253) — created by setup job, PAT in IAM_SERVICE__PAT secret, no membership by default

The lowercase core-service user is an orphan (unused PAT, no reference in any secret). Safe to delete for security hygiene once confirmed no Zitadel process recreates it.

Issue 28 — ASK Agent Narrates Intent Instead of Calling Tools (BAAI/bge-large-en-v1.5 400 Error)

Symptom:

The ASK assistant responds with reasoning text like "I need to search the knowledge base..." or "I should use the knowledge_base_search tool..." but never actually invokes any tool. The conversation completes in ~2s with completion_tokens under 200 — no tool call round-trips.

Logs (assistant-service pod):

WARNING - Foundry embedding error: Error code: 400 - {'error': {'message':
  '/embeddings: Invalid model name passed in model=BAAI/bge-large-en-v1.5.
  Call `/v1/models` to view available models for your key.'}}
WARNING - Tool search failed, falling back to full MCP config: Error code: 400 ...
INFO - Sending LLM request: ... mcps_count=8 ... prompt_tokens=20815

Root cause:

The assistant-service has a Tool Search feature: before each LLM call it embeds the user query and tool descriptions to semantically select only the relevant 2-3 tools to inject into the prompt (rather than all 19+). This embedding call goes to Foundry.

The embedding model for tool search is set in ToolsSettings.foundry_tool_embedding_model_name (defined in src/tooling/config/settings.py). This is a pydantic-settings class that reads directly from environment variables — it does NOT read Dynaconf TOML.

The existing TOML override tool_embedding_model = "qwen/qwen3-embedding-0-6b" covers the Dynaconf config tree but never reaches ToolsSettings. The field fell back to its default: BAAI/bge-large-en-v1.5 — which is not registered in Foundry → 400 error.

When tool search fails, the service falls back to injecting the full MCP tool list (19+ tools, ~20k prompt tokens). With this bloated context the Qwen3-122B model narrates its intent rather than emitting structured tool calls.

Fix:

Add FOUNDRY_TOOL_EMBEDDING_MODEL_NAME as an explicit env var in the assistant-service values file (pydantic-settings reads this as the field name in uppercase):

# mod-ask-assistant-service/helm/environments/values-mod-auh1-dev-ask.yaml
envConf:
  extraEnv:
    - name: FOUNDRY_TOOL_EMBEDDING_MODEL_NAME
      value: "qwen/qwen3-embedding-0-6b"

Key lesson:

ToolsSettings (tooling service) is a separate pydantic-settings class — it reads ENV vars only. Dynaconf TOML overrides ([mod-auh1-dev-ask.foundry]) never reach it. Any model reference in src/tooling/config/settings.py must be overridden via extraEnv, not TOML.

Verification:

After ArgoCD sync + pod restart, logs should show:

INFO - Creating embeddings for N tools...
INFO - Search query for tool search: <query>
INFO - Sending LLM request: ... mcps_count=2 ...   ← only relevant tools injected

Issue 29 — ASK Agent Silent Empty Response (chunks_count=0) After Tool Call — TOOL message without tool_call_id

Symptom:

Agent receives a message, tool search works correctly, tool call is generated (79 completion tokens), but the user sees no response in the UI. Intelligence pod logs show:

WARNING - [HISTORY_GROUPING] Found TOOL message without tool_call_id. This may cause LLM failures. Ignoring message.
# (repeated 18+ times per request)
...
INFO - LLM response completed ... chunks_count=0
INFO - No response or chunks available, skipping citation generation

Token usage log repeats the same value (e.g. completion_tokens=79) every ~1.4 seconds for 20+ seconds, then chunks_count=0 — no content ever reaches the user.

Root cause:

The intelligence service's HISTORY_GROUPING function drops all TOOL messages that lack a tool_call_id from the conversation history it sends to the LLM. This happens when tool result messages are reloaded from the database — the tool_call_id linkage is not persisted or reconstructed correctly.

With 18+ tool results dropped, the LLM receives a corrupted conversation: - It sees its own prior tool calls (assistant messages with tool_calls field) - But no corresponding tool results (those were dropped) - OpenAI-format chat requires every tool_calls in an assistant message to be followed by a tool message with matching tool_call_id

The LLM generates a new tool call (79 tokens). The intelligence service tries to parse and execute it, but the malformed history context causes a loop — usage events stream for 20+ seconds with no content chunks, then the agentic loop exits with chunks_count=0.

Affected layer: Code bug in src/intelligence/modules/nodes/ (scratchpad node / history grouping). Not fixable via config or Helm values.

Workaround (users):

Start a new conversation (click New Chat). The bug only triggers on long conversations where tool calls happened in prior turns. A fresh conversation has no stale tool messages in history, so the agentic loop completes normally.

Permanent fix (upstream):

The HISTORY_GROUPING function must preserve tool_call_id when persisting tool result messages to the database and restore it when reconstructing history. Dropping orphaned TOOL messages silently corrupts the LLM's view of the conversation. The correct behaviour is either:

Persist tool_call_id and restore it on reload (correct fix), or
Drop the paired assistant tool_calls message too when the tool result is orphaned (preserves history coherence at the cost of losing that exchange)

Diagnostic commands:

# Confirm the bug is active — look for HISTORY_GROUPING warnings
kubectl logs -n ask deploy/assistant-service-intelligence --since=5m | grep "HISTORY_GROUPING"

# Confirm chunks_count=0 pattern
kubectl logs -n ask deploy/assistant-service-intelligence --since=5m | grep "chunks_count=0"

# Confirm no content reached the client
kubectl logs -n ask deploy/assistant-service-intelligence --since=5m | grep "No response or chunks"

Key lesson:

chunks_count=0 with non-zero completion_tokens always means the model generated something (usually a tool call) but the intelligence service could not stream content back to the client. Check for HISTORY_GROUPING warnings in the same request window — if present, this bug is the cause.

And the agent will make actual tool calls instead of narrating intent.