E2E Testing Without the Frontend

An ordered sequence to validate the whole ASK platform — every model and every feature — using only kubectl/API calls, while the frontend is blocked on its air-gap rebuild. Run the stages in order; each builds on the previous, so a failure localises the break (model → service → feature).

How this stays accurate

Deep request bodies are read from each service's live /openapi.json (Stage 3) rather than hand-written — that is the authoritative contract for the running image. Routes below are confirmed from source; exact field names for POST bodies should be taken from /openapi.json.

---

Stage 0 — Setup (keys + endpoints)

# Foundry virtual key (model calls) and admin key (alias inspection)
KEY=$(kubectl get secret -n ask assistant-service-secret -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d)
ADMIN=$(kubectl get secret -n ask core-service-secret -o jsonpath='{.data.FOUNDRY_ADMIN_KEY}' | base64 -d)
# core-service machine token (for auth-gated core APIs), if seeded:
PAT=$(kubectl get secret -n ask core-service-secret -o jsonpath='{.data.APP__CORE_SERVICES_MACHINE_TOKEN}' | base64 -d 2>/dev/null)

In-cluster service base URLs (no TLS, no ingress):

Service	In-cluster base
core-service	http://core-service.ask.svc.cluster.local
assistant — intelligence	http://assistant-service-intelligence:8003
assistant — tooling	http://assistant-service-tooling:8001
KMS	http://knowledge-management-service.ask.svc.cluster.local
Foundry (LiteLLM)	http://foundry-litellm.foundry.svc.cluster.local:4000

Run probes from a pod with python3 (app images lack curl)

All snippets kubectl exec -i … deploy/assistant-service-intelligence -- python3 — that pod has httpx. Heredocs must use -i.

---

Stage 1 — All models (the foundation)

Every feature ultimately calls one of these. Green here first.

kubectl exec -i -n ask deploy/assistant-service-intelligence -- env KEY="$KEY" python3 - <<'EOF'
import os, httpx
B="http://foundry-litellm.foundry.svc.cluster.local:4000"; H={"Authorization":"Bearer "+os.environ["KEY"]}
def chat(m):
    r=httpx.post(B+"/v1/chat/completions",headers=H,
      json={"model":m,"messages":[{"role":"user","content":"Reply OK"}],"max_tokens":5},timeout=90)
    return r.status_code
print("chat  qwen3-5-122b-a10b     ", chat("qwen/qwen3-5-122b-a10b"))
print("chat  qwen3-4b-instruct      ", chat("Qwen/Qwen3-4B-Instruct-2507"))
print("chat  gpt-oss-120b          ", chat("qwen/qwen3-5-122b-a10b"))
print("chat  qwen3-5-122b-a10b-kms ", chat("qwen/qwen3-5-122b-a10b-kms"))
r=httpx.post(B+"/v1/embeddings",headers=H,json={"model":"qwen/qwen3-embedding-0-6b","input":"hi"},timeout=60)
print("embed qwen3-embedding-0-6b  ", r.status_code, "dim="+str(len(r.json()["data"][0]["embedding"])) if r.status_code==200 else "")
r=httpx.post(B+"/v1/rerank",headers=H,json={"model":"nvidia/llama-nemotron-rerank-1b-v2","query":"capital of UAE","documents":["Abu Dhabi is the UAE capital","Bananas are yellow"]},timeout=60)
print("rerank llama-nemotron       ", r.status_code)
EOF

Pass: all chat 200, embed 200 dim=1024, rerank 200.

---

Stage 2 — Service health (each process up)

kubectl exec -i -n ask deploy/assistant-service-intelligence -- python3 - <<'EOF'
import httpx
checks = {
  "core-service":      "http://core-service.ask.svc.cluster.local/api/v1/health/ready",
  "kms":               "http://knowledge-management-service.ask.svc.cluster.local/health/",
  "assistant-intel":   "http://assistant-service-intelligence:8003/health",
  "assistant-tooling": "http://assistant-service-tooling:8001/health/mcp",
}
for name,url in checks.items():
    try: print(f"{name:18s}", httpx.get(url,timeout=8).status_code)
    except Exception as e: print(f"{name:18s} ERR {e}")
EOF

Verified output (2026-06-22):

core-service       200
kms                200
assistant-intel    200
assistant-tooling  200

Each service has a DIFFERENT health path — a uniform path lies

Confirmed from /openapi.json (Stage 3):

Service	Health path	Notes
core-service	/api/v1/health/ready (+ /live)
KMS	/health/	other paths need auth → 401, not down
assistant-intel	/health (+ /health/liveness, /health/readiness)
assistant-tooling	/health/mcp	no plain /health — only the MCP health route exists, so /health 404s by design

A loop that hits /api/v1/health/ready on all four returns 200/404/401/404 even when all are healthy — only core uses that path.

---

Stage 3 — Enumerate exact endpoints (authoritative)

Pull each service's OpenAPI to get the exact paths + request schemas for the deeper stages (no guessing):

for svc in \
  "core-service|http://core-service.ask.svc.cluster.local" \
  "kms|http://knowledge-management-service.ask.svc.cluster.local" \
  "assistant-intel|http://assistant-service-intelligence:8003" \
  "assistant-tooling|http://assistant-service-tooling:8001"; do
  name=${svc%%|*}; base=${svc##*|}
  echo "== $name =="
  kubectl exec -i -n ask deploy/assistant-service-intelligence -- env B="$base" python3 - <<'EOF'
import os, httpx
try:
    spec=httpx.get(os.environ["B"]+"/openapi.json",timeout=10).json()
    for p,methods in sorted(spec.get("paths",{}).items()):
        print("  ",",".join(m.upper() for m in methods), p)
except Exception as e: print("  (no /openapi.json:",e,")")
EOF
done

Use the printed paths/bodies as the source of truth for Stages 4–6.

---

Stage 4 — core-service (IAM / org / model catalog)

Real routes (from openapi): discovery GET /api/v1/organizations/discovery, login POST /api/v1/auth/login, model catalog GET /api/v1/models/ (+ /enabled, /providers, /disabled), orgs GET /api/v1/organizations.

Test user (verified 2026-06-23): testadmin@ask.mod.auh1.dev.dir / org 378447542100166005.

# 4a. org discovery — no auth
kubectl exec -i -n ask deploy/assistant-service-intelligence -- python3 - <<'EOF'
import httpx
r=httpx.get("http://core-service.ask.svc.cluster.local/api/v1/organizations/discovery",
            params={"email":"testadmin@ask.mod.auh1.dev.dir"},timeout=10)
print("discovery", r.status_code, r.text[:200])
EOF

Pass: 200 + the ask-default org.

The catalog/org endpoints are auth-gated — get a session via login (verified schema LoginDTO = org_id + email + password, a simple password grant). This login also bypasses the broken frontend and proves the login backend itself.

The login username MUST be email#<org-id> in Zitadel — or login 404s

auth_service.login() builds the Zitadel login name as email + "#" + org_id (utils/auth_utils.py:fetch_username_from_email). If the user's Zitadel preferred login name uses a different suffix (e.g. the user-id), Zitadel returns 404 "User could not be found" even though the DB row is fine. The two error codes localise the failure exactly:

Login result	Failing check	Fix
401 "User not found in organization"	DB lookup get_by_email_in_organization	seed/insert the core_service.users row
404 "User could not be found"	Zitadel create_session by loginName	rename the Zitadel user to email#<org-id>
200 + sessionToken	both pass ✓	—

Full detail: Troubleshooting → login 404.

# 4b. LOGIN → carry the session → model catalog (L2), in one run
export PW='<testadmin password from Vault>'
kubectl exec -i -n ask deploy/assistant-service-intelligence -- env PW="$PW" python3 - <<'EOF'
import os, httpx
CS="http://core-service.ask.svc.cluster.local"
with httpx.Client(base_url=CS, timeout=20) as c:
    r=c.post("/api/v1/auth/login", json={
        "org_id":"378447542100166005",
        "email":"testadmin@ask.mod.auh1.dev.dir",
        "password":os.environ["PW"]})
    print("login", r.status_code, r.text[:160])
    tok=None
    try: tok=r.json().get("sessionToken") or r.json().get("access_token")
    except Exception: pass
    h={"Authorization":"Bearer "+tok} if tok else {}
    m=c.get("/api/v1/models/enabled", headers=h)
    print("models/enabled", m.status_code, m.text[:400])
EOF

Verified (2026-06-23): login → 200 with {"sessionId","sessionToken","userId","organizationId",...}. The sessionToken is the Bearer token for all downstream auth-gated calls.

iam_client auth proof (2026-06-23)

After the iam_client 401 fix (gateway ingress + SSL_CERT_FILE + internal CA ConfigMap), passing the sessionToken to KMS GET /collections/ and core-service GET /api/v1/models/enabled both return 403 Insufficient permissions (not 401) — confirming the full chain: token → iam_client → core-service /api/v1/users/me → 200 → service applies authz → 403. A 401 here means iam_client is still broken; 403 means auth passed.

models/enabled requires a superadmin or model-admin role — testadmin (ask71_admin role) gets 403. The machine PAT (APP__CORE_SERVICES_MACHINE_TOKEN) also returns 401 "Token Validation Failed" here — it is not a user session token and cannot be resolved by users/me. Use superadmin credentials to pull the full catalog, or confirm model availability via Foundry /v1/models (Stage 1) which uses the Foundry key.

Superadmin verified (2026-06-23): superadmin@ask.mod.auh1.dev.dir login 200 → models/enabled 200 → catalog is populated (38 inference / 2 embedding across all orgs) but all disabled by default for new orgs.

---

4c. Enable models for the org (superadmin, one-time)

kubectl exec -i -n ask deploy/assistant-service-intelligence -- env PW="$SUPERPW" python3 - <<'EOF'
import os, httpx, json
CS="http://core-service.ask.svc.cluster.local"
with httpx.Client(base_url=CS, timeout=20) as c:
    r=c.post("/api/v1/auth/login", json={
        "org_id":"378447542100166005",
        "email":"superadmin@ask.mod.auh1.dev.dir",
        "password":os.environ["PW"]})
    tok=r.json().get("sessionToken")
    h={"Authorization":"Bearer "+tok}
    r=c.post("/api/v1/models/enable", headers=h, json={
        "model_id": ["qwen3-5-122b-a10b", "gpt-oss-120b"],
        "default_model_id": "qwen3-5-122b-a10b",
        "org_id": "378447542100166005"
    })
    print("enable", r.status_code, r.text[:200])
    m=c.get("/api/v1/models/enabled", headers=h)
    print("enabled after:", m.status_code, m.text[:400])
EOF

Verified (2026-06-23):

enable 200  → {"message":"Models enabled successfully.","enabled_models_count":2}

enabled after: 200
{"inference":[
  {"id":"gpt-oss-120b","provider":"OpenAI","name":"GPT-OSS 120B","version":"oss-120b",
   "description":"Open-source GPT model with 120B parameters for advanced tasks.",
   "type":"inference","supported_languages":["English"],"logo_file_name":"openai.svg",
   "logo_path":"https://cdn.ask.mod.auh1.dev.dir/static/logos/llm_models/openai.svg",
   "is_default":false,"is_recommended":true},
  {"id":"qwen3-5-122b-a10b", ... (truncated — 400-char limit hit)}
 ],
 "embedding":[]}

Enabled catalog → Foundry model mapping for this org:

Catalog ID	Name	Foundry model	Role
qwen3-5-122b-a10b	QWEN-3-5-122B	qwen/qwen3-5-122b-a10b	default chat
gpt-oss-120b	GPT-OSS 120B	qwen/qwen3-5-122b-a10b	code / data analysis

Embedding (BAAI/bge-m3) skipped — embedding is wired server-side via the TOML, not user-selectable.

cdn_base_url is required for /api/v1/models/ (not /models/enabled)

GET /models/ and /models/disabled build logo URLs via get_public_cdn_url() which raises ValueError: Storage CDN base URL is not configured. if cdn_base_url is absent from the [storage] TOML block. Add it to the devops overlay (mod-ask-devops/applications/core-service/values-mod-auh1-dev-ask.yaml), NOT the chart-repo env file (that's shadowed). Value: "https://cdn.ask.mod.auh1.dev.dir" (no /static suffix — the code appends the path). /models/enabled still returns 200 even without it because the enabled list is cached differently.

Console-created users need a DB row

A user made in the Zitadel Console (Email-Verified checkbox) is not auto-synced to the core-service DB → login 401 until you insert it:

INSERT INTO core_service.users
  (external_user_id, organization_id, email, display_name, first_name, last_name, state, created_at, updated_at)
VALUES ('<ZITADEL-USER-ID>','378447542100166005','<email>','<name>','<first>','<last>','active',NOW(),NOW());
INSERT INTO core_service.user_roles (created_at, role_id, user_id)
SELECT NOW(), 1, id FROM core_service.users WHERE email='<email>';   -- role_id 1 = ask71_admin

---

Stage 5 — KMS (ingest → retrieve → rerank)

Full pipeline: create collection → upload document → Hatchet processes (embed+VLM) → retrieve with rerank. All non-health KMS routes are auth-gated (org context via sessionToken).

5a. Health + route inventory

kubectl exec -i -n ask deploy/assistant-service-intelligence -- python3 - <<'EOF'
import httpx, json
KMS="http://knowledge-management-service.ask.svc.cluster.local"
h=httpx.get(KMS+"/health/",timeout=8)
print("health", h.status_code, h.text[:200])
s=httpx.get(KMS+"/openapi.json",timeout=10).json()
paths=list(s.get("paths",{}).keys())
print("\nROUTES:")
for p in sorted(paths): print(" ", p)
EOF

Verified (2026-06-23):

health 200 {"status":"HEALTHY","database":{"status":"UP","error":null}}

ROUTES:
  /admin/collection/{collection_id}    /admin/document/{document_id}
  /chunks/document/{document_id}       /chunks/neighbours
  /chunks/search                       /chunks/{chunk_id}
  /collections/                        /collections/is_shared
  /collections/search                  /collections/{collection_id}
  /collections/{collection_id}/actors  /collections/{collection_id}/restore
  /collections/{collection_id}/share   /databases/
  /databases/types                     /databases/{connection_id}
  /databases/{connection_id}/actors    /databases/{connection_id}/share
  /documents/                          /documents/cancel-processing
  /documents/collection/{collection_id}/documents/content
  /documents/download-urls             /documents/file_formats
  /documents/import                    /documents/move
  /documents/queued/count              /documents/retry-processing
  /documents/search                    /documents/sync
  /documents/update/initiate           /documents/upload/initiate
  /documents/upload/{document_id}/update-status
  /documents/{document_id}             /documents/{document_id}/actors
  /documents/{document_id}/restore     /documents/{document_id}/share
  /documents/{document_id}/summary     /folders/
  /folders/children                    /folders/tree
  /folders/{folder_id}                 /folders/{folder_id}/actors
  /folders/{folder_id}/move            /folders/{folder_id}/share
  /health/                             /organizations/me
  /organizations/onboard               /retrieve/
  /tables/document/{document_id}       /tables/{table_id}

---

Stage 6 — assistant-service (chat + tools)

Real routes (openapi):

Concern	Pod	Route
Chat (sync)	intelligence :8003	POST /orchestration/invoke
Chat (stream)	intelligence :8003	POST /orchestration/stream
Orchestrator	intelligence :8003	POST /orchestrator/call, POST /orchestrator/process
Tool invoke	tooling :8001	POST /toolbus/call (+ POST /toolbus/prepare)
Tool catalog	tooling :8001	GET /catalog/my-tools, GET /catalog/connectors
KMS proxy	tooling :8001	GET /kms/collections, GET /kms/documents

• Chat (POST /orchestration/invoke) is the main flow — exercises [foundry] llm_model (orchestrator) and, by message type, the small LLM / code-gen / data-viz models. Needs auth + assistant/conversation context; pull the exact body from the schema:

  kubectl exec -i -n ask deploy/assistant-service-intelligence -- python3 - <<'EOF'
  import httpx, json
  s=httpx.get("http://assistant-service-intelligence:8003/openapi.json",timeout=10).json()
  print(json.dumps(s["paths"]["/orchestration/invoke"]["post"].get("requestBody",{}),indent=2)[:900])
  EOF
  

• Tools (POST /toolbus/call on tooling :8001) drives a specific tool:
• data-analysis → [tooling.data_analysis] * (gpt-oss-120b)
• data visualization → [foundry] data_visualization_llm (qwen3-5-122b-a10b) — see the discrepancy note in Model Aliases §4
• KMS proxy (GET /kms/collections) → assistant reaching KMS

Deep chat flows are async (Hatchet)

A real chat run is queued through Hatchet and streamed (/orchestration/stream). For a scripted check, drive /orchestration/invoke with the schema body and read the response. The repos' tests/intelligence/tests_integration suites are the canonical E2E for these flows.

---

Stage 7 — Feature → model → test matrix

Feature (tracker)	Model	Stage that proves it
Orchestrator	qwen3-5-122b-a10b	1 (model) + 6 (chat)
Guardrails (policy + safety)	Qwen/Qwen3-4B-Instruct-2507 (alias qwen3-4b-instruct-2507)	1 (model) + 6 (send a tripping message → blocked)
Data visualization	qwen3-5-122b-a10b *	1 + 6 (data-viz tool)
Data-analysis tools	gpt-oss-120b	1 + 6 (tooling)
KMS embedding	qwen3-embedding-0-6b	1 (dim=1024) + 5 (search)
KMS OCR / VLM	qwen3-5-122b-a10b-kms	1 (model) + 5 (ingest an image)
KMS reranker	llama-nemotron-rerank-1b-v2	1 (model) + 5 (search ranked)
Mem0 / Voice / Meta-Agent	NA (disabled)	confirm off, not erroring

* tracker sheet lists gpt-oss-120b for data-viz — config says qwen3-5-122b-a10b; confirm which the tool actually calls.

---

Recommended run order

1. Stage 1 (models) — must be green first.
2. Stage 2 (health) — every service up.
3. Stage 3 (openapi) — capture exact bodies once.
4. Stage 4 (core discovery) — IAM/DB wiring.
5. Stage 5 (KMS search) — embed + rerank in context.
6. Stage 6 (assistant chat + one tool) — orchestrator + tool models.

If a stage fails, re-run Stage 1 for that feature's model to isolate model vs service. Per-service request detail: core-service Onboarding §6, assistant Integration Tests, 37 KMS Deploy.

---

Related

• Model Aliases & Testing — alias inventory, per-model probes (Stage 1 detail)
• LLM → Service Config Mappings — feature → model reference