LLM → Service Config Mappings¶

A plain-English map of every AI feature across the three ASK services, which ones use Foundry, which model each one calls, and whether that model actually exists in our air-gapped setup. Written assuming no prior LLM knowledge. Sources: Foundry, Model References, Hardcoded Audit, KMS Config & Models.

0. The basics (read this first)¶

Model = the "AI brain." You send it text (or an image) and it produces an answer, a number list (an embedding), or a ranking. Different models are good at different jobs.
Three kinds of model we use:
Chat / reasoning model — writes answers, reasons, generates code. (Also vision models that can read images.)
Embedding model — turns text into a list of numbers so we can find "similar" text (powers search).
Reranker model — given a question and some documents, sorts them by relevance.
vLLM = the engine that runs the big chat & vision models on GPUs.
Infinity = the engine that runs the embedding & reranker models.
LiteLLM = the front desk / gateway. Every app talks only to LiteLLM at one address (foundry-litellm...:4000); LiteLLM forwards the request to vLLM or Infinity behind the scenes. "Uses Foundry" = the feature calls this LiteLLM gateway.
"Available?" = is that exact model actually loaded in our Foundry? We are air-gapped (no internet), so cloud models — GPT-5, Claude, Gemini, Grok — are NOT available. Only the open-source (OSS) models we host are. A feature pointed at a cloud model fails when used (often silently — see the audit).

flowchart LR
    APP["a feature in\nAssistant / KMS"] -->|"asks for a model"| LL["LiteLLM\n(front desk / gateway)"]
    LL -->|chat / vision| VLLM["vLLM\n(GPU engine)"]
    LL -->|embeddings / rerank| INF["Infinity\n(engine)"]
    CORE["core-service"] -.->|"admin: makes API keys,\nNOT inference"| LL

What each piece actually is¶

Foundry — the whole "AI engine room" of the platform (the foundry namespace). It's not one program but a stack: a gateway (LiteLLM) + a chat/vision engine (vLLM) + an embedding/rerank engine (Infinity) + support bits (a database, Redis, S3 for model weights). When a doc says a feature "uses Foundry," it means the request goes into this stack. (Foundry is AI71/TII's name for their internal model-serving platform — think "our private OpenAI.")
vLLM — the engine that runs the big language/vision models on GPUs. It loads a model's multi-gigabyte "weights" (its brain) into GPU memory, takes a prompt, and generates the answer. Built to be fast and serve many users at once. Runs our chat/vision models (qwen3-5-122b-a10b, …-kms, Qwen/Qwen3-4B-Instruct-2507, gpt-oss-120b).
Infinity — a smaller, specialized engine for embeddings and reranking (the "math" models, not the "writing" models). These are small and high-throughput, so they run on their own engine instead of tying up vLLM's GPUs. Runs qwen3-embedding-0-6b and llama-nemotron-rerank-1b-v2.
LiteLLM — the front desk. Apps never talk to vLLM/Infinity directly; they send every request to one address (foundry-litellm…:4000) in one standard format (the OpenAI API). LiteLLM sees which model was asked for and routes it to the right engine, and checks the API key.

Restaurant analogy

Foundry = the whole restaurant.
LiteLLM = the waiter — you only talk to the waiter, in one language, with one order.
vLLM = the main kitchen — cooks the big dishes (chat, reasoning, reading images).
Infinity = the prep station — fast small jobs (embeddings, reranking).
the models (qwen…, gpt-oss…, nemotron…) = the recipes the kitchens know.
core-service = the host who issues your membership card (API key) and shows the menu — but doesn't cook.

Chat reply: app → waiter (LiteLLM) → main kitchen (vLLM) runs qwen/qwen3-5-122b-a10b → answer. KB search: app → waiter (LiteLLM) → prep station (Infinity) runs qwen3-embedding-0-6b → numbers used to find matching documents.

These are loaded in our Foundry and confirmed working (tested 2026-06-17):

Model id (what apps ask for)	Job	Engine	Notes
`qwen/qwen3-5-122b-a10b`	Chat / reasoning / vision	vLLM	the flagship; also reads images
`qwen/qwen3-5-122b-a10b-kms`	Chat / vision — KMS only	vLLM	separate copy so KMS load doesn't slow chat
`Qwen/Qwen3-4B-Instruct-2507`	Small / fast chat	vLLM	lightweight tasks (alias `qwen3-4b-instruct-2507`; old `qwen/qwen3-5-4b` compat-redirects here)
`qwen/qwen3-5-122b-a10b`	Chat / code generation	vLLM	OSS, despite the "openai/" prefix
`qwen/qwen3-embedding-0-6b`	Embeddings (search)	Infinity	1024 numbers per text
`nvidia/llama-nemotron-rerank-1b-v2`	Reranking	Infinity	sorts results by relevance

Everything not on this list (GPT-5, Claude, Gemini, Grok, GLM, MiniMax, BAAI/*, meta-llama/Llama-Guard) is not available here. Guardrails now use qwen-3-4b-it-2507 / qwen3-5-122b-a10b (commit 32aaae8) — ⚠️ bare names 400 on direct Foundry call, verify resolution.

1b. Where each model is configured — the 3 "layers"¶

Every model id is written down somewhere. There are three places ("layers"), and they differ in how easily we can change them in the air-gap. This matters because a feature pointed at an unavailable model can only be fixed at the layer where its id lives.

Layer	Where the model id lives	How we change it	Control
Layer 1 — Environment config (TOML)	`config/<env>.toml` sections (`[foundry]`, `[organization]`, `[document_understanding]`, `[tooling.*]`), delivered through our Helm `envConf.configToml`	edit the values file → GitOps sync	🟢 Easy — fully ours
Layer 2 — Baked catalog (YAML in the image)	`models.yaml` (assistant) / `models.yml` (core-service) shipped inside the container image	can't edit values; mount a corrected file over it via ConfigMap	🟡 Medium
Layer 3 — Hardcoded in source (`.py`)	a model id typed directly into Python code	patch the source repo, OR add a LiteLLM alias on the gateway so the old id resolves to a served one	🔴 Hard

Which layer defines what, per service¶

Assistant-service (all 3 layers)KMS (Layer 1 only — clean)Core-service (Layer 2 catalog; no inference)

Model use	Layer	Defined in	Available?
`llm_model`, `small_llm_model`, `code_generator_model`, `docx_prepare_context_model`, `data_visualization_llm`, `tool_embedding_model`, `reranker.model`, `tooling.data_analysis.*`	L1	values `configToml` → `[foundry]`, `[tooling.data_analysis]`	✅ set to served qwen/gpt-oss
user-selectable model picker + `_TEXT/_VOICE_MODEL_PRIORITY`	L2	`src/shared/llm/llm_models/models.yaml` + `assistants/services/llm_model_service.py`	⚠️ cloud-heavy — override needed
citations `azure/gpt-5-chat`	L3	`intelligence/.../citations/post_answer_citations.py:37`	❌
meta-agent `gpt-5`	L3	`assistants/services/meta_agent_service.py:79`	❌
scratchpad `anthropic/claude-haiku-4-5`	L3	`intelligence/.../scratchpad_node.py:247`	❌
tooling defaults `azure/gpt-5-chat`, `anthropic/claude-3-7-sonnet`, `BAAI/bge-large`	L3	`tooling/config/settings.py:168/172/176/185/462`	❌ unless env-set
KB-tool embedding allow-list `BAAI/bge-m3`	L3	`tooling/.../retrieval/retrieval_tool.py:81`	⚠️ inert (KMS uses its own model)
guardrail models (`Llama-Guard-3-8B`, `gpt-oss-safeguard-120b`, `gpt-oss`)	L3 (default) + per-org override	`src/safety/guardrails/*`	`gpt-oss-safeguard-120b` ✅ (alias added); Llama-Guard ❌ not deployed
token counting `gpt-5-chat`	L3	`tooling/core/llms/utils.py:24`, `runnable.py:96`	benign (tokenizer)

Model use	Layer	Defined in	Available?
`embedding_model_id` = `qwen/qwen3-embedding-0-6b`	L1	values `configToml` → `[organization]`	✅
`vlm_model_name` = `qwen/qwen3-5-122b-a10b-kms`	L1	values `configToml` → `[organization]`	✅
`retriever.reranker.model` = `nvidia/llama-nemotron-rerank-1b-v2`	L1	`[mod-auh1-dev-ask.retriever.reranker]`	✅ enabled by env override (`default.toml` ships `@none`)

KMS has no Layer 2 catalog and no Layer 3 hardcoded model ids — every model comes from L1 config, which is why it was the easiest to get fully air-gap-correct.

Model use	Layer	Defined in	Available?
model catalog (the menu the UI shows)	L2	`src/model_provider/models/models.yml`	⚠️ cloud-heavy — trim to served
Foundry admin (key/team provisioning)	L1 (`FoundryConfig`)	config `foundry` — not a model id	✅ management only

Core-service has no Layer 3 model ids (only example strings in API docs). It doesn't call models.

Rule of thumb

L1 = we control it (easy fix). L2 = baked menu (mount-override). L3 = baked in code (alias or patch). The air-gap pain is almost entirely L2 + L3; L1 is already correct.

2. Assistant-service — feature → model map¶

Feature (what the user experiences)	Uses Foundry?	Model it calls	Available?
Main chat answer	✅ LiteLLM→vLLM	user-picked from the menu; default `qwen/qwen3-5-122b-a10b`	✅ if a served model is picked (the picker also lists cloud models that ❌)
RAG reasoning (intelligence)	✅ vLLM	`qwen/qwen3-5-122b-a10b`	✅
Lightweight tasks / summaries	✅ vLLM	`Qwen/Qwen3-4B-Instruct-2507`	✅
Code generation	✅ vLLM	`qwen/qwen3-5-122b-a10b`	✅
DOCX context prep	✅ vLLM	`qwen/qwen3-5-122b-a10b`	✅
Data visualization	✅ vLLM	`qwen/qwen3-5-122b-a10b`	✅
Tool search (embeddings)	✅ Infinity	`qwen/qwen3-embedding-0-6b`	✅
Knowledge-base search (calls KMS)	✅ (via KMS)	KMS embeds with `qwen/qwen3-embedding-0-6b`	✅ (tool's hardcoded `BAAI` id is ignored by KMS)
RAG reranking (config block)	✅ Infinity	`nvidia/llama-nemotron-rerank-1b-v2`	✅ served — but this build's code doesn't call it
Answer citations	✅ vLLM	`azure/gpt-5-chat` (hardcoded in code)	❌ → citations silently skipped
Meta-agent	✅ vLLM	`gpt-5` (default)	❌ unless given a served model
Data-analysis tool	✅ vLLM	`qwen/qwen3-5-122b-a10b` (our config)	✅ (code default is `azure/gpt-5-chat` ❌ if env not applied)
Scratchpad reasoning	✅ vLLM	`anthropic/claude-haiku-4-5` (default)	❌ if used
Guardrail: llama-guard	✅ vLLM	`meta-llama/Llama-Guard-3-8B`	❌ if enabled
Guardrail: keyword / custom-policy	✅ vLLM	`qwen3-5-122b-a10b` (bare)	⚠️ 400 on Foundry — verify resolution (commit `32aaae8`)
Guardrail: safety filters	✅ vLLM	`qwen-3-4b-it-2507` (bare)	⚠️ 400 on Foundry — verify resolution
Guardrail: OpenAI moderation	✅	a moderation model	❌ air-gap
Image analysis (VLM)	—	`Qwen3-VL-8B` (commented out)	off by design
User memory (mem0)	— (disabled)	LLM + embedder via Foundry; vector store Qdrant. Models from `intelligence.memory_ingestion.{llm_model_name, embedding_model_name}` (PP default `azure/gpt-5-nano` ❌)	🚫 DISABLED — `memory_ingestion=@null` + Qdrant omitted → `mem0` client returns `None`, no calls. To enable: set served models (`Qwen/Qwen3-4B-Instruct-2507` + `qwen/qwen3-embedding-0-6b`, dims 1024) and deploy Qdrant (image already mirrored).
Model picker list	—	catalog (Claude/GPT-5/Grok/…)	mostly ❌ — needs air-gap trim

3. KMS — feature → model map¶

Feature	Uses Foundry?	Model	Available?
Document embedding (when you upload)	✅ Infinity	`qwen/qwen3-embedding-0-6b`	✅
Search query embedding	✅ Infinity	`qwen/qwen3-embedding-0-6b` (the org's model)	✅
Visual document understanding (read image/PDF pages)	✅ vLLM	`qwen/qwen3-5-122b-a10b-kms`	✅
Reranking	✅ Infinity	`nvidia/llama-nemotron-rerank-1b-v2`	✅ (env override enables it; `default.toml` ships `@none` but `[mod-auh1-dev-ask.retriever.reranker]` sets `model + candidate_pool_multiplier = 2.0`)
Keyword extraction / summarization	—	disabled (`@none`)	n/a
Azure Document Intelligence (alt OCR)	❌ external Azure	—	dormant — only if someone switches parsing to `azure`; don't

KMS is the cleanest — every active model is a served one.

4. Core-service — it does NOT run AI¶

Core-service is the identity / org / billing service. It touches Foundry only on the admin plane, never for inference:

Feature	Uses Foundry?	What it does	Available?
Foundry key / team provisioning	✅ Foundry admin API (`FoundryAdminService`)	when an org/user is onboarded, core-service asks Foundry to mint that user's API key + team — this is the key assistant-service later uses to call models	✅ (management, not inference)
Model catalog (`model_provider/models/models.yml`)	—	publishes the list of models the UI offers + which are enabled per org	⚠️ catalog lists cloud models — needs air-gap trim
No chat/embedding calls	—	core-service never sends prompts to a model	—

So: core-service hands out the keys and the menu; assistant-service and KMS do the actual eating.

5. Bottom line — what works, what silently won't¶

✅ Works today (served models, tested): assistant chat/RAG/code/data-viz/embeddings, KMS ingest + embeddings + vision + search, the reranker model itself, core-service key provisioning.

🚫 Disabled by design (off — no model called, safe): user memory (mem0) — memory_ingestion=@null + Qdrant omitted; image analysis (VLM) — commented out. Enabling either needs served-model overrides (+ Qdrant for mem0) first.

❌ Silently broken until fixed (point at unavailable cloud models): - Assistant citations (azure/gpt-5-chat), meta-agent (gpt-5), scratchpad (claude-haiku-4-5), Llama-Guard (not deployed), OpenAI moderation (external API). - ⚠️ Guardrails moved to qwen-3-4b-it-2507 / qwen3-5-122b-a10b (commit 32aaae8) — bare names 400 on direct Foundry call; confirm the resolution layer or switch to prefixed served names. - The model pickers / catalogs in assistant-service (models.yaml) and core-service (models.yml) list cloud models that don't exist here.

The two fixes that clear most of it: 1. One LiteLLM alias on the gateway: azure/gpt-5-chat, anthropic/claude-3-7-sonnet, gpt-5 → qwen/qwen3-5-122b-a10b. Neutralizes the hardcoded-in-code cases at once. (Draft in the Hardcoded Audit.) 2. Trim the catalogs to only served models (and enable only those per org).

See Lazy Dependency Test Matrix for how to test each path.