Assistant Service — Integration Tests & Verification¶

Use this page after any deployment or config change to verify each workload and its dependencies. Tests are ordered from innermost (infra) to outermost (API).

flowchart TB
    T0["0 · All pods Running"] --> T1["1 · DB — schema + tables"]
    T1 --> T2["2 · Redis — RediSearch + ReJSON"]
    T2 --> T3["3 · Hatchet worker registered"]
    T3 --> T4["4 · core-service reachable"]
    T4 --> T5["5 · Foundry LLM gateway"]
    T5 --> T6["6 · KMS (tooling)"]
    T6 --> T7["7 · End-to-end via ingress"]

    style T0 fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style T7 fill:#1e5f3a,stroke:#4ad98a,color:#fff

0. Pre-flight — all pods Running¶

kubectl get pods -n ask -l app.kubernetes.io/instance=assistant-service -o wide

Expected: every workload shows Running with restart count 0 (or low after initial config fixes).

Workload	Container	Port	Expected status
`assistant-service`	assistants API	8002	Running, probe ✅
`assistant-service-intelligence`	RAG / agents	8003	Running, probe ✅
`assistant-service-tooling`	MCP ToolBus	8001	Running, probe ✅
`assistant-service-generic-worker`	Hatchet worker	—	Running
`assistant-service-migration` (Job)	alembic	—	Completed

If any pod is CrashLoopBackOff or restarting, get its logs before continuing:

kubectl logs -n ask deploy/<workload-name> --since=3m | tail -60
kubectl logs -n ask deploy/<workload-name> -p 2>/dev/null | tail -60   # previous crash

1. Database — migration ran, schema exists¶

The assistant-service-migration Job runs alembic upgrade head at sync-wave 1, before the deployments start.

# 1a. Confirm the migration job completed
kubectl get job -n ask | grep -i assistant
# expect: assistant-service-migration  1/1  <age>

# 1b. Verify the schema and a sample table exist
kubectl exec -n ask core-pg-1 -- \
  psql -U postgres -d assistant-service -c '\dn'
# expect: schema "assistant_service"

kubectl exec -n ask core-pg-1 -- \
  psql -U postgres -d assistant-service -c '\dt assistant_service.*' | head -20
# expect: tables like conversations, assistants, messages, ...

Fail path: migration job not found → check ArgoCD sync-wave ordering; the job is PreSync wave -1 and the ConfigMaps are wave -2. If the ConfigMap assistant-service-postgres-init-script is missing, the DB-init Job also didn't run (database not created) — re-sync the app.

2. Redis — connectivity and module check¶

assistant-service uses Redis for chat-session cache and MCP state. The bitnami 8.2.1 image bundles RediSearch + ReJSON; the chart loads them via commonConfiguration.

# 2a. Confirm Redis pod is Running
kubectl get pod -n ask -l app.kubernetes.io/name=redis | grep assistant

# 2b. Ping from the main assistants pod
REDIS_PASS=$(kubectl get secret -n ask assistant-service-secret \
  -o jsonpath='{.data.REDIS_PASSWORD}' | base64 -d)
kubectl exec -n ask deploy/assistant-service -- \
  redis-cli -h assistant-service-redis-master.ask.svc.cluster.local \
  -a "$REDIS_PASS" PING
# expect: PONG

# 2c. Confirm RediSearch and ReJSON modules are loaded
kubectl exec -n ask assistant-service-redis-master-0 -- \
  redis-cli -a "$REDIS_PASS" MODULE LIST
# expect: entries for "search" (RediSearch) and "ReJSON"

Fail path — modules missing: the redis:8.2.1 image includes modules in /opt/bitnami/redis/lib/redis/modules/. If MODULE LIST returns empty, the commonConfiguration loadmodule lines aren't taking effect — check the redis StatefulSet config and restart the redis pod.

3. Hatchet — worker registered and running¶

The assistant-service-generic-worker connects to hatchet-engine.ask.svc.cluster.local:7070 (plaintext gRPC — requires SERVER_GRPC_INSECURE=t on the engine).

Don't grep for the startup banner on a long-running pod

The STARTING HATCHET / starting runner lines are logged once at boot. On a pod that's been up for hours they've scrolled off, and a --since=5m + grep "hatchet|worker|runner" returns nothing — which looks like a failure but isn't. The worker's runtime logs use a 🪓 emoji and the words rx, step run, finished (none match that grep). Validate against steady-state runtime, not the boot banner.

# 3a (preferred). Steady-state — confirm the worker is RECEIVING and RUNNING dispatched work.
# This only happens over a live gRPC stream to the engine, so it proves registration + connectivity
# more strongly than the startup banner does.
kubectl logs -n ask deploy/assistant-service-generic-worker --tail=40 | \
  grep -iE "step run|finished step|refreshed|🪓"

✅ Healthy output (the built-in refresh_usage_counts cron fires every 10 min):

🪓 -- rx: start step run: <id>/assistant_refresh_usage_counts:refresh_usage_counts
🪓 -- run: start step: assistant_refresh_usage_counts:refresh_usage_counts/<id>
assistant_service - INFO - Refreshed mv_assistant_usage_counts
🪓 -- finished step run: assistant_refresh_usage_counts:refresh_usage_counts/<id>

Regular 10-minute intervals with no gaps = connected and processing. 0 restarts, 1/1 Running.

# 3b (boot banner — only meaningful on a freshly (re)started pod, within ~5 min of start).
kubectl logs -n ask deploy/assistant-service-generic-worker | \
  grep -iE "STARTING HATCHET|starting runner|legacy" | head
# expect (if caught early enough):
#   STARTING HATCHET...
#   Connected to an older Hatchet engine ... Falling back to legacy worker registration.
#   starting runner...

❌ Fail patterns:

Error	Cause	Fix
`Socket closed`	engine TLS mismatch (`SERVER_GRPC_INSECURE=f`)	`kubectl patch secret hatchet-shared-config -n ask --type=merge -p '{"stringData":{"SERVER_GRPC_INSECURE":"t"}}' && kubectl rollout restart deploy/hatchet-engine -n ask`
`invalid JWT` / pod crash on start	`HATCHET__CLIENT_TOKEN` is a random placeholder, not a real JWT	`TOK=$(kubectl get secret -n ask hatchet-client-config -o jsonpath='{.data.HATCHET_CLIENT_TOKEN}' \| base64 -d) && vault kv patch secret/ask/assistant-service/auh1-dev HATCHET__CLIENT_TOKEN="$TOK"`
`could not resolve host: hatchet.ask...`	token embeds the external hostname, `HATCHET_CLIENT_HOST_PORT` override missing	confirm `HATCHET_CLIENT_HOST_PORT=hatchet-engine.ask.svc.cluster.local:7070` is set in `envConf.extraEnv`

# 3c. Verify on the Hatchet dashboard (workers tab)
# Open: https://hatchet.ask.mod.auh1.dev.dir → Workers
# Expect: "assistant-service-generic-worker" shown as connected

# 3d. List registered workflows via Hatchet API
HATCHET_TOKEN=$(kubectl get secret -n ask hatchet-client-config \
  -o jsonpath='{.data.HATCHET_CLIENT_TOKEN}' | base64 -d)
curl -sk https://hatchet.ask.mod.auh1.dev.dir/api/v1/workflows \
  -H "Authorization: Bearer $HATCHET_TOKEN" | python3 -m json.tool | grep -i name | head -20
# expect: workflow definitions from assistant-service (e.g. usage-count, bulk ops)

4. core-service — reachability and auth¶

All three app workloads (assistants, intelligence, tooling) call core-service for auth/IAM. AssistantsSettings.core_service_url (env var CORE_SERVICE_URL) must point to it.

# 4a. Confirm CORE_SERVICE_URL reached the pods
kubectl exec -n ask deploy/assistant-service -- env | grep CORE_SERVICE_URL
kubectl exec -n ask deploy/assistant-service-intelligence -- env | grep CORE_SERVICE_URL
# expect: CORE_SERVICE_URL=http://core-service.ask.svc.cluster.local/api/v1

# 4b. HTTP reachability from intelligence pod (app image has NO curl — use httpx)
kubectl exec -n ask deploy/assistant-service-intelligence -- python3 -c \
"import httpx; r=httpx.get('http://core-service.ask.svc.cluster.local/api/v1/health/ready', timeout=10); print(r.status_code, r.text[:200])"
# expect: 200 {"status":"ok"} or similar

The app images ship no curl

assistant-service, -intelligence, and -tooling are built on python:3.12-slim-trixie with no curl/wget. Any in-pod HTTP check must use Python httpx (an app dependency, always present), as above. curl from the bastion is fine — it only fails inside kubectl exec.

Fail path: Connection refused → core-service pod is down. connection timed out → DNS resolution failure or network policy blocking the pod. Check kubectl get pods -n ask -l app.kubernetes.io/name=core-service.

5. Foundry / LLM gateway — connectivity and model availability¶

intelligence calls Foundry for chat completions, embeddings, and reranking. For initial bring-up a dummy endpoint is used (AI calls fail lazily — pods start fine).

# 5a. Confirm the Foundry service is reachable from intelligence (httpx — no curl in image)
# NOTE: LiteLLM's /health REQUIRES the api key (401 "No api key passed in." if omitted) — that 401
# still PROVES reachability. For a clean no-auth reachability check use /health/liveliness.
kubectl exec -n ask deploy/assistant-service-intelligence -- python3 -c \
"import httpx; r=httpx.get('http://foundry-litellm.foundry.svc.cluster.local:4000/health/liveliness', timeout=10); print(r.status_code, r.text[:120])"
# expect: 200  (a 401 from the plain /health endpoint also means 'reachable, needs key' — not a failure)

# 5b. List models served by Foundry
FOUNDRY_KEY=$(kubectl get secret -n ask assistant-service-secret \
  -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d)
kubectl exec -n ask deploy/assistant-service-intelligence -- env FK="$FOUNDRY_KEY" python3 -c \
"import os,httpx; r=httpx.get('http://foundry-litellm.foundry.svc.cluster.local:4000/v1/models', headers={'Authorization':'Bearer '+os.environ['FK']}, timeout=15); print(r.status_code); [print(m['id']) for m in r.json().get('data',[])]"
# expect: 200 then the model ids — qwen/qwen3-5-122b-a10b, Qwen/Qwen3-4B-Instruct-2507, z-ai/glm-5,
#         qwen/qwen3-embedding-0-6b, nvidia/llama-nemotron-rerank-1b-v2
#         (qwen/qwen3-5-4b no longer directly served — compat alias redirects to Qwen3-4B-Instruct-2507)

!!! danger "Inject the key from the Secret — don't read it from the pod env"
    These tests pass the key via `env FK=$FOUNDRY_KEY` (read fresh from the Secret), **not**
    `os.environ['AI71_FOUNDRY_KEY']`. The running pod's env is captured at start, so right after a
    Vault key change it still holds the **old placeholder** until you `rollout restart`. Injecting
    the fresh key makes the test correct regardless of restart state.

!!! bug "Use `python3 -c`, not a `python3 -` heredoc — stdin isn't forwarded"
    `kubectl exec` only forwards stdin with `-i`. A `python3 - <<'EOF'` **without `-i`** reads an
    empty program and exits printing **nothing** (silent, looks like a hang). Pass the code as a
    `-c` argument instead (newlines inside the quoted string are fine), as below.

```bash
# 5c. Embedding smoke test — fresh key injected from the Secret
FOUNDRY_KEY=$(kubectl get secret -n ask assistant-service-secret \
  -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d)
kubectl exec -n ask deploy/assistant-service-intelligence -- env FK="$FOUNDRY_KEY" python3 -c \
"import os,httpx
r=httpx.post('http://foundry-litellm.foundry.svc.cluster.local:4000/v1/embeddings', headers={'Authorization':'Bearer '+os.environ['FK']}, json={'model':'qwen/qwen3-embedding-0-6b','input':'hello world'}, timeout=30)
print('status:', r.status_code)
d=r.json()
print('dims:', len(d['data'][0]['embedding']) if 'data' in d else d)"
# expect: status 200, dims: <N>   (404 → model id wrong; 401 → key stale/placeholder)

# 5d. Chat completion smoke test — fresh key injected from the Secret
FOUNDRY_KEY=$(kubectl get secret -n ask assistant-service-secret \
  -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d)
kubectl exec -n ask deploy/assistant-service-intelligence -- env FK="$FOUNDRY_KEY" python3 -c \
"import os,httpx
r=httpx.post('http://foundry-litellm.foundry.svc.cluster.local:4000/v1/chat/completions', headers={'Authorization':'Bearer '+os.environ['FK']}, json={'model':'qwen/qwen3-5-122b-a10b','messages':[{'role':'user','content':'ping'}],'max_tokens':5}, timeout=60)
print('status:', r.status_code)
print(r.json()['choices'][0]['message']['content'] if r.status_code==200 else r.text[:300])"

Then roll the pods so the APP uses the real key

The tests above inject the key, but the app code still reads it from its captured env. After confirming Foundry works, restart so the real key is live for actual traffic:

kubectl rollout restart deploy/assistant-service \
  deploy/assistant-service-intelligence deploy/assistant-service-tooling -n ask

# 5e. Reranker smoke test (Cohere-style /v1/rerank) — fresh key injected from the Secret
FOUNDRY_KEY=$(kubectl get secret -n ask assistant-service-secret \
  -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d)
kubectl exec -n ask deploy/assistant-service-intelligence -- env FK="$FOUNDRY_KEY" python3 -c \
"import os,httpx
r=httpx.post('http://foundry-litellm.foundry.svc.cluster.local:4000/v1/rerank',
  headers={'Authorization':'Bearer '+os.environ['FK']},
  json={'model':'nvidia/llama-nemotron-rerank-1b-v2','query':'What is the capital of France?',
        'documents':['Berlin is the capital of Germany.','Paris is the capital of France.','The Eiffel Tower is a famous landmark.'],'top_n':3},
  timeout=60)
print('status',r.status_code)
print([(x['index'],round(x['relevance_score'],3)) for x in r.json()['results']] if r.status_code==200 else r.text[:300])"
# expect: 200, the Paris doc (index 1) ranked first with the highest score

Foundry is wired (2026-06-17)

The real in-cluster LiteLLM gateway and OSS models are configured — see Foundry / LLM Endpoint. 404 model not found here means a model name drifted back to a cloud default; 401 means AI71_FOUNDRY_KEY is missing/stale in the Secret. Foundry gateway capability verified 2026-06-17: chat (5d), embeddings (5c), rerank (5e — model is served and ranks correctly), VLM (KMS). Note: 5e tests the gateway; assistant-service code does not actually call the rerank endpoint ([foundry.reranker] is unused — see model-references Layer 3).

6. KMS — knowledge base reachability (tooling)¶

The assistant-service-tooling ToolBus calls KMS for knowledge-base search tools. KMS may not be deployed yet — calls are lazy; the pod still starts.

# 6a. Confirm KMS_BASE_URL is set on the tooling pod
kubectl exec -n ask deploy/assistant-service-tooling -- env | grep KMS_BASE_URL
# expect: KMS_BASE_URL=http://knowledge-management-service.ask.svc.cluster.local

# 6b. Reachability check (httpx — no curl in image; only meaningful when KMS is deployed)
kubectl exec -n ask deploy/assistant-service-tooling -- python3 -c \
"import httpx,sys
try:
    r=httpx.get('http://knowledge-management-service.ask.svc.cluster.local/health', timeout=8)
    print('✅ KMS reachable', r.status_code)
except Exception as e:
    print('⚠️ KMS not reachable (expected if not deployed):', type(e).__name__)"

7. End-to-end — API smoke test via ingress¶

# 7a. Health checks on all three HTTP services
for ep in \
  "https://api.ask.mod.auh1.dev.dir/ask71/v2/astsvc/health/readiness" \
  "https://api.ask.mod.auh1.dev.dir/ask71/v2/astsvc/health/liveness"; do
  echo -n "$ep → "
  curl -sk "$ep" | head -c 80
  echo
done
# expect: 200 {"status":"ok"} or {"status":"healthy"}

How to get a CORE_SERVICE_JWT

The assistant-service API validates a user session JWT that core-service signs with the shared CORE_SERVICE__JWT_SECRET (HS, jwt_ttl_seconds = 300). It represents a logged-in user — there is no shortcut that bypasses the IAM chain. Two ways to obtain one:

A — Browser (easiest). Log into the ASK frontend https://ask.mod.auh1.dev.dir/ (Zitadel), open DevTools → Network, click any call to api.ask.mod.auh1.dev.dir, and copy the Authorization: Bearer <token> value. Use it within 5 min (300s TTL).

B — Mint via the IAM chain (scriptable). Replicates trigger_execution_service: iam_client.auth.get_user_id_by_token(machine_token, impersonate_user_id=<user_id>) → iam_client.jwt_service.create_jwt(user_details, impersonate_user_id=<user_id>). Needs the Zitadel machine token (IAM_SERVICE__PAT in core-service-secret) and a real user_id / org_id (default org 377526562817377089). More setup; use only for repeatable automation.

# 7b. Authenticated API smoke test — list assistants
CORE_SERVICE_JWT="<bearer-from-step-A-or-B>"

curl -sk https://api.ask.mod.auh1.dev.dir/ask71/v2/astsvc/api/v1/assistants \
  -H "Authorization: Bearer $CORE_SERVICE_JWT" | python3 -m json.tool | head -30
# expect: 200 {"items": [...]}  (may be empty list if no assistants created yet)
# 401 → token expired (>300s) or IAM/jwt_secret mismatch between core-service and assistant-service

# 7c. Direct service health checks (in-cluster service DNS, bypasses ingress; httpx — no curl)
kubectl exec -n ask deploy/assistant-service -- python3 -c \
"import httpx
for u in ['http://assistant-service.ask.svc.cluster.local:80/health/readiness',
          'http://assistant-service-intelligence.ask.svc.cluster.local:8003/health/readiness',
          'http://assistant-service-tooling.ask.svc.cluster.local:8001/health/mcp']:
    try: print(httpx.get(u, timeout=8).status_code, u)
    except Exception as e: print('ERR', type(e).__name__, u)"

8. Per-workload dependency matrix (quick reference)¶

Workload	Hard deps (pod won't start without)	Lazy / per-request deps
migration	core-pg reachable, `DATABASE__PASSWORD` set, `APP_ENV` set	—
assistant-service (main)	core-pg, redis, secret synced, `CORE_SERVICE_URL` env, `APP_ENV`	core-service (auth), intelligence (per-req), tooling (per-req)
assistant-service-intelligence	core-pg, redis, secret synced, `CORE_SERVICE_URL` env, `APP_ENV`	Foundry (LLM calls), tooling (tool exec), core-service
assistant-service-tooling	core-pg, redis, secret synced, `AI71_FOUNDRY_ENDPOINT` env, `KMS_BASE_URL` env, `APP_ENV`	KMS (knowledge tools), Foundry (tool LLM), intelligence
assistant-service-generic-worker	real JWT `HATCHET__CLIENT_TOKEN`, `HATCHET_CLIENT_HOST_PORT` override, engine `grpcInsecure=t`, core-pg, redis	—

Config env var checklist — must be in envConf.extraEnv (applied to all workloads)

- APP_ENV: "mod-auh1-dev-ask"             # Dynaconf env_switcher
- APP_ENVIRONMENT: "mod-auh1-dev-ask"     # settings.py file selection
- DATABASE__PASSWORD                       # from assistant-service-db-password secretKeyRef
- AI71_FOUNDRY_ENDPOINT                   # ToolsSettings (tooling) — env only, not TOML
- KMS_BASE_URL                            # ToolsSettings (tooling) — env only, not TOML
- CORE_SERVICE_URL                        # AssistantsSettings (main + intelligence) — env only, not TOML
- HATCHET_CLIENT_HOST_PORT                # override external broadcast address in JWT token
- REDIS_HOST                              # AssistantsSettings.redis_host — defaults "localhost"; breaks InterruptionListener

REDIS_HOST missing does NOT crash the pod (field has a default) but breaks chat stream interrupts silently. Missing any others → the affected workload crashes at import (gunicorn exits code 3).

9. Common failure patterns (quick lookup)¶

Symptom	Workload	Root cause	Fix
`AssistantsSettings — core_service_url — Field required`	main, intelligence	`CORE_SERVICE_URL` not in env	add to `envConf.extraEnv`; `kubectl set env deploy -n ask -l app.kubernetes.io/instance=assistant-service CORE_SERVICE_URL=...`
`ToolsSettings — foundry_base_url — Field required`	tooling	`AI71_FOUNDRY_ENDPOINT` not in env	add to `envConf.extraEnv`
`Socket closed` (gRPC)	generic-worker	Hatchet engine TLS mismatch	`kubectl patch secret hatchet-shared-config -n ask --type=merge -p '{"stringData":{"SERVER_GRPC_INSECURE":"t"}}'` + restart engine
`invalid JWT` / crash at import	generic-worker	`HATCHET__CLIENT_TOKEN` is a placeholder	copy real JWT from `hatchet-client-config` → Vault → force-sync ExternalSecret
Health probe timeout (`context deadline exceeded`)	intelligence	pod still starting (slow ML imports) OR pydantic crash before HTTP starts	check logs; if pydantic crash, look for `Field required` in previous-container logs
`Error in interruption listener: Error 111 connecting to localhost:6379`	main	`REDIS_HOST` not set — `AssistantsSettings.redis_host` defaults to `"localhost"`	add `REDIS_HOST=assistant-service-redis-master.ask.svc.cluster.local` to `envConf.extraEnv`
`tiktoken ConnectTimeout openaipublic.blob.core.windows.net`	main	air-gap — can't download tokenizer vocab	harmless warning, falls back to approximate counts; ignore
`MODULE LIST` empty on Redis	redis	`loadmodule` config not applied	check redis commonConfiguration; restart redis pod
DB `relation "assistant_service.*" does not exist`	main, intelligence	migration Job didn't run or failed	check migration job logs; manually run `alembic upgrade head` if needed