Lazy Dependency Test Matrix (core-service · assistant-service · KMS)¶
Lazy dependencies pass health checks and let the pod boot, then fail only when the feature is
exercised. A pod can be Running 1/1 with a broken lazy dependency. This page is the standing
checklist of every lazy item across the three ASK services and how to functionally test it — use it
after any deploy, secret change, or dependency bring-up.
Why this matters
"All pods Running" ≠ working. The S3 InvalidAccessKeyId, a stale Foundry key, a missing
REDIS_HOST, or an undeployed KMS all leave the pods green while the feature silently fails.
Core-service¶
| Lazy item | Backs | Test | Status |
|---|---|---|---|
Ceph S3 (ask-core-service) |
avatar / org-logo uploads | boto3 round-trip with ask-core-service keys, or upload an avatar via API |
⚠️ verify |
Zitadel / IAM (IAM_SERVICE__PAT) |
login + user/org mgmt | curl -k https://sso.../auth/v1/users/me -H "Authorization: Bearer $PAT"; do a real login |
✅ onboarded |
| Hatchet workers (bulk-ops, billing-metering, cron-scheduler, job-run) | async jobs | trigger each workflow / Hatchet dashboard shows runs | ✅ wired |
Assistant-service¶
| Lazy item | Backs | Test | Status |
|---|---|---|---|
Foundry chat (qwen3-5-122b-a10b) |
chat completions | integration test 5d | ✅ |
Foundry embeddings (qwen3-embedding-0-6b) |
RAG / memory | test 5c (1024-dim) | ✅ |
Foundry reranker (llama-nemotron-rerank) |
RAG rerank | POST …/v1/rerank (Cohere-style) with AI71_FOUNDRY_KEY |
🟡 gateway only — direct call ranks correctly, but assistant-service does not call it ([foundry.reranker] config is unused); the real post-answer path is hardcoded azure/gpt-5-chat (❌, Layer 3). So this validates Foundry's capability, not an app feature. |
KMS (KMS_BASE_URL) |
knowledge-base tools (tooling) | test 6 — now that KMS is deployed | ⚠️ re-test |
core-service (CORE_SERVICE_URL) |
per-request auth | test 4 | ✅ |
Redis InterruptionListener (REDIS_HOST) |
chat-stream cancel | log Subscribed to chat_interruptions; cancel a stream |
⚠️ verify log |
| Hatchet generic-worker | usage-count workflows | test 3 | ✅ |
| tiktoken / Daytona / MCP gateway | tokenizer / code-exec / MCP | fallback or disabled by design | n/a |
KMS¶
| Lazy item | Backs | Test | Status |
|---|---|---|---|
Ceph S3 (ask-kms) |
document storage | round-trip, S3 wiring §5B | ✅ verified |
Foundry embeddings (qwen3-embedding-0-6b) |
chunk embedding on ingest | embedding call (§5); ingest a doc | ⚠️ embed-call ✅, e2e ❌ |
Foundry VLM (qwen3-5-122b-a10b-kms) |
visual doc understanding | multimodal /v1/chat/completions with an image (read text back); or ingest an image/PDF |
✅ direct test passed (read image text); e2e via ingest pending |
| Weaviate | vector search + upsert | ready+auth check; ingest then search | ✅ up; search ❌ |
| Hatchet workers (ingestion / generic / live-ingestion) | ingest / metrics / connector workflows | upload doc → kms_document_ingestion runs |
⚠️ dispatch untested |
| core-service (auth) | API authz | authenticated KMS API call | ⚠️ untested |
Foundry reranker (nvidia/llama-nemotron-rerank-1b-v2) |
search result ranking | POST /v1/rerank with AI71_FOUNDRY_KEY; or run a retrieval query and confirm ranked results |
⚠️ untested — model served, config enabled (candidate_pool_multiplier=2.0) |
| Azure DI | alt OCR | disabled by design (parsing_strategy = generic) |
n/a |
Runnable now (no JWT required)¶
# A. KMS API health (in-cluster; no curl in image → httpx)
kubectl exec -n ask deploy/knowledge-management-service -- python3 -c \
"import httpx;print(httpx.get('http://knowledge-management-service.ask.svc.cluster.local/health/',timeout=8).status_code)"
# B. Weaviate ready + apikey auth (KMS vector store)
WV=$(kubectl get secret -n ask knowledge-management-service-secret -o jsonpath='{.data.VECTOR_DB__API_KEY}' | base64 -d)
kubectl exec -n ask deploy/knowledge-management-service -i -- env WV="$WV" python3 - <<'EOF'
import os, httpx
b="http://weaviate.ask.svc.cluster.local"
print("ready:", httpx.get(b+"/v1/.well-known/ready", timeout=8).status_code)
print("auth :", httpx.get(b+"/v1/meta", headers={"Authorization":"Bearer "+os.environ["WV"]}, timeout=8).status_code)
EOF
# C. assistant-service tooling → KMS reachability (now that KMS is up)
kubectl exec -n ask deploy/assistant-service-tooling -- python3 -c \
"import httpx;print(httpx.get('http://knowledge-management-service.ask.svc.cluster.local/health/',timeout=8).status_code)"
# D. KMS S3 round-trip (creds + bucket RWX) — see 40-kms-s3-wiring.md §5B
# E. KMS Foundry VLM — multimodal read of a generated image (no JWT)
FK=$(kubectl get secret -n ask knowledge-management-service-secret -o jsonpath='{.data.DOCUMENT_UNDERSTANDING__FOUNDRY_KEY}' | base64 -d)
kubectl exec -n ask deploy/knowledge-management-service-ingestion-worker -i -- env FK="$FK" python3 - <<'EOF'
import os, io, base64, httpx
from PIL import Image, ImageDraw
img = Image.new("RGB", (240, 80), "white"); ImageDraw.Draw(img).text((10, 30), "VLM TEST 4271", fill="black")
buf = io.BytesIO(); img.save(buf, format="PNG"); b64 = base64.b64encode(buf.getvalue()).decode()
r = httpx.post("http://foundry-litellm.foundry.svc.cluster.local:4000/v1/chat/completions",
headers={"Authorization": "Bearer " + os.environ["FK"]},
json={"model": "qwen/qwen3-5-122b-a10b-kms",
"messages": [{"role": "user", "content": [
{"type": "text", "text": "What text appears in this image? Answer briefly."},
{"type": "image_url", "image_url": {"url": "data:image/png;base64," + b64}}]}],
"max_tokens": 40}, timeout=90)
print("status:", r.status_code); print(r.json()["choices"][0]["message"]["content"] if r.status_code==200 else r.text[:400])
EOF
Expected: A 200, B ready 200 / auth 200, C 200, E 200 + a reply describing the image text.
These clear the KMS API, Weaviate auth, the assistant→KMS lazy path, S3, and the VLM — without a user token.
End-to-end document ingest (JWT-gated) — the highest-value test¶
A real document upload exercises five lazy items at once: S3 (store/fetch) + Foundry embeddings +
Foundry VLM + Hatchet workflow dispatch + Weaviate upsert. KMS uses a multi-step upload
(POST /upload/initiate → presigned S3 PUT → confirm → kms_document_ingestion workflow), API root
/ask71/v2/kmsvc. Prerequisites:
- core-service JWT for the org (see assistant integration §7b — browser DevTools).
- KMS org onboarded (
POST /organizations/onboard) and a collection to upload into.
JWT="<core-service-jwt>"
BASE="https://api.ask.mod.auh1.dev.dir/ask71/v2/kmsvc"
# 1. confirm org context
curl -sk "$BASE/organizations/me" -H "Authorization: Bearer $JWT" | python3 -m json.tool
# 2. initiate an upload (returns a presigned S3 URL + document id) — payload shape per the API
curl -sk -X POST "$BASE/documents/upload/initiate" -H "Authorization: Bearer $JWT" \
-H 'Content-Type: application/json' -d '{"file_name":"sample.pdf","collection_id":"<id>"}'
# 3. PUT the file to the returned presigned URL (this is the S3 write)
# curl -sk -X PUT "<presigned-url>" --data-binary @sample.pdf
# 4. watch the ingestion worker run the workflow (S3 fetch → VLM → embed → Weaviate upsert)
kubectl logs -n ask deploy/knowledge-management-service-ingestion-worker --tail=60 \
| grep -iE "kms_document_ingestion|s3|vlm|embed|weaviate|upsert|error"
# 5. search it back (exercises Weaviate query + embeddings)
curl -sk -X POST "$BASE/documents/search" -H "Authorization: Bearer $JWT" \
-H 'Content-Type: application/json' -d '{"query":"<text from the doc>","collection_id":"<id>"}'
Healthy = the worker logs show download-from-S3 → VLM/parse → 1024-dim embed → Weaviate upsert, and
the search returns the chunk. Confirm growth Ceph-side: radosgw-admin bucket stats --bucket=ask-knowledge-management.
Exact payloads
upload/initiate, search, and collection creation field names come from the KMS OpenAPI —
fetch it from a running pod to be exact:
kubectl exec -n ask deploy/knowledge-management-service -- python3 -c "import httpx;print(httpx.get('http://localhost:8000/openapi.json',timeout=8).text)" | python3 -m json.tool | grep -A3 upload
Priority order¶
- KMS end-to-end ingest — clears five lazy items in one test (highest value).
- assistant-service → KMS (
KMS_BASE_URL) — newly unblocked now that KMS is live. - assistant-service Redis interruption —
Subscribed to chat_interruptionsin logs. - core-service S3 — confirm the
ask-core-serviceupload path. - Foundry reranker — gateway capability confirmed (direct call), but not wired into
assistant-service (
[foundry.reranker]unused; real path hardcoded toazure/gpt-5-chat). No app-level test applies until that Layer-3 hardcode is fixed (alias or source patch).