Model References — How assistant-service Resolves Models¶

assistant-service references LLM/embedding/rerank models in three places. All must point at models the air-gapped Foundry (foundry-litellm) actually serves, or calls 404. For the Foundry endpoint, key, and the list of served models see Foundry / LLM Endpoint.

Layer 1 — TOML ([foundry], [tooling.data_analysis]) — overridable via Helm values. ✅ fixed.
Layer 2 — models.yaml — baked user-selectable catalog (api_model_name). ⚠️ needs ConfigMap override.
Layer 3 — hardcoded in source — cloud model ids written directly in src/, bypassing both 1 and 2. Neither TOML nor a ConfigMap fixes these; they need a source patch (or a LiteLLM alias). See the Layer 3 section below.

flowchart TB
    subgraph L1["Layer 1 — internal/operational"]
        TOML["config/default.toml\n(overridden by Helm envConf.configToml)"]
    end
    subgraph L2["Layer 2 — user-selectable catalog"]
        YAML["src/shared/llm/llm_models/models.yaml\n(baked into the image)"]
    end
    TOML -->|"model id verbatim"| GW["foundry-litellm\n:4000/v1"]
    YAML -->|"api_model_name"| GW
    GW --> VLLM["vLLM / Infinity backends"]

    style L1 fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style L2 fill:#5f1e3a,stroke:#d94a90,color:#fff

Layer 1 — internal / operational models¶

config/default.toml, overridden per-env by the chart's envConf.configToml (helm/environments/values-mod-auh1-dev-ask.yaml). These values are passed verbatim as the model id to LiteLLM, so they must equal Foundry's served ids.

TOML field	Used for	Air-gap value
`[foundry] llm_model`	main chat / reasoning	`qwen/qwen3-5-122b-a10b`
`[foundry] small_llm_model`	lightweight tasks	`Qwen/Qwen3-4B-Instruct-2507`
`[foundry] code_generator_model`	code gen	`qwen/qwen3-5-122b-a10b`
`[foundry] docx_prepare_context_model`	DOCX context	`qwen/qwen3-5-122b-a10b`
`[foundry] data_visualization_llm`	data-viz	`qwen/qwen3-5-122b-a10b`
`[foundry] tool_embedding_model`	tool search embeddings	`qwen/qwen3-embedding-0-6b`
`[foundry.reranker] model`	RAG rerank	`nvidia/llama-nemotron-rerank-1b-v2`
`[tooling.data_analysis] profiler_model / code_generator_model / query_validator_model / column_mapper_model`	data-analysis tool	`qwen/qwen3-5-122b-a10b`

These are validated by integration test 5 (embedding + chat). The upstream default.toml ships azure/gpt-5-chat, anthropic/claude-3-7-sonnet, and BAAI/* defaults that Foundry does not serve — every override above is therefore required. See Foundry → Why every model name had to change.

Layer 2 — user-selectable model catalog (`models.yaml`)¶

src/shared/llm/llm_models/models.yaml is the catalog of models a user picks for their assistant in the UI. It is loaded once at startup from a fixed path baked into the image (Path(__file__).parent / "models.yaml" in src/shared/llm/llm_models/llm_models.py) — there is no env or config override. Each entry is keyed by name; the api_model_name field is what gets sent to Foundry at call time (src/intelligence/.../llm_call_node.py).

- name: "qwen3-5-122b-a10b"          # what the UI shows / the assistant stores
  api_model_name: "..."              # ← what is sent to the gateway
  context_limit: 262000
  supports_vision: true
  is_reasoning: true
  ...

⚠️ Air-gap mismatch (must be corrected)¶

The baked catalog's api_model_name values point at cloud/upstream paths the air-gap Foundry does not serve:

Catalog `name`	baked `api_model_name`	Foundry serves	Match
~~`gpt-oss-120b`~~	(dead entry)	removed from Foundry	❌ decommissioned 2026-07 — remove from catalogue
`qwen3-5-122b-a10b`	`foundry-stg/core42-oicm-auh1-shd/qwen-3-5-122b-a10b`	`qwen/qwen3-5-122b-a10b`	❌
`gpt-5`, `anthropic/claude-`, `grok-`, `glm-`, `minimax-*`, `gemini`	azure / anthropic / xai / z-ai / minimax / google	not served	❌

Why this bites later, not now

Layer 1 works (tested via the gateway directly). But the moment a user creates an assistant and selects a model in the UI, the app sends the catalog's api_model_name → 404 on Foundry. The default public assistant (mujeeb) is affected too. It hasn't surfaced only because no real assistant chat has been driven yet (that needs a user/JWT).

Fix — mount a corrected catalog via ConfigMap (no image rebuild)¶

The chart exposes generic .Values.volumes / .Values.volumeMounts (helm/values.yaml, rendered in helm/templates/deployment.yaml). Mount a ConfigMap-backed models.yaml over the baked file with subPath, containing only the models Foundry serves and the correct api_model_names:

# values-mod-auh1-dev-ask.yaml (illustrative)
volumes:
  - name: models-catalog
    configMap:
      name: assistant-service-models-catalog
volumeMounts:
  - name: models-catalog
    mountPath: /app/src/shared/llm/llm_models/models.yaml
    subPath: models.yaml

The ConfigMap holds an air-gap catalog, e.g.:

models:
  - name: "qwen3-5-122b-a10b"
    api_model_name: "qwen/qwen3-5-122b-a10b"
    context_limit: 262000
    response_max_clamp_value: 16384
    response_min_clamp_value: 256
    provider: "qwen"
    supports_vision: true          # confirm on this Foundry instance
    is_creativity_level_supported: true
    is_reasoning: true
    structured_output_ruleset_name: default
  - name: "qwen3-4b-instruct-2507"
    api_model_name: "Qwen/Qwen3-4B-Instruct-2507"
    context_limit: 32000
    response_max_clamp_value: 8192
    response_min_clamp_value: 256
    provider: "qwen"
    supports_vision: false
    is_creativity_level_supported: false
    is_reasoning: false
  # gpt-oss-120b decommissioned 2026-07 — remove this catalogue entry (roles moved to qwen/qwen3-5-122b-a10b; GLM-5.2 added as glm-5)
    context_limit: 128000
    response_max_clamp_value: 16384
    response_min_clamp_value: 256
    provider: "OpenAI"
    supports_vision: false
    is_creativity_level_supported: true
    is_reasoning: true

Open decisions before finalizing the catalog

Which models are user-selectable — all three served chat models, or a curated subset.
supports_vision — confirm whether qwen3-5-122b-a10b serves vision on this Foundry instance (the upstream catalog marked it true); set the flag to match reality or image uploads will be offered and then fail.
qwen/qwen3-5-122b-a10b-kms is the KMS-dedicated instance — do not add it to the assistant-service catalog.

Alternative: LiteLLM aliasing

Instead of overriding the catalog, the Foundry team could register the baked api_model_name strings as aliases pointing at the OSS backends. Rejected as the default here: it means aliasing azure/… / anthropic/… names to qwen models (confusing), and keeps the mapping out of our GitOps repo. The ConfigMap override is the air-gap-correct, version-controlled approach.

Layer 3 — hardcoded in source¶

Some model ids are written directly in src/, not read from TOML or models.yaml. Neither a values override nor a ConfigMap can change them — they require a source patch in the sanitized repo (or a matching LiteLLM alias on the Foundry side). Found 2026-06-17:

Source location	Hardcoded model	Role	Air-gap impact
`src/intelligence/modules/nodes/citations/post_answer_citations.py:37`	`azure/gpt-5-chat`	post-answer citation LLM (`MarketplaceGeneratorLM`)	404 when citations run → node catches the error and returns chunks unchanged, so citations are silently skipped
`src/intelligence/modules/nodes/scratchpad_node/scratchpad_node.py:247`	`anthropic/claude-haiku-4-5` (default arg)	scratchpad reasoning	404 if the scratchpad path executes and the default isn't overridden by pipeline config
`src/tooling/config/settings.py:168/172/176/462`	`azure/gpt-5-chat`, `anthropic/claude-3-7-sonnet` (pydantic defaults)	tooling data-analysis / misc	404 unless overridden by env (ToolsSettings reads ENV, not the Dynaconf TOML)
`src/assistants/services/llm_model_service.py`	`_TEXT_MODEL_PRIORITY` / `_VOICE_MODEL_PRIORITY` lists (claude/gpt-5/grok/glm/minimax/gemini)	user-facing model-picker priority order	filtered by `get_enabled_models_for_org`; cloud entries are noise if the org only enables served models
`src/intelligence/models/constants.py:35-36`	`anthropic/claude-*` → reasoning-config-file map	reasoning-effort config lookup	lookup miss for non-served models (benign)

The reranker is configured but NOT called in this build

config.foundry.reranker (the [foundry.reranker] block — separate endpoint, model, api_key = @null) appears only in the schema (src/config/schemas/shared/foundry.py RerankerConfig). No code reads it and there is no /rerank / litellm.rerank call anywhere in src/. So although Foundry serves nvidia/llama-nemotron-rerank-1b-v2 (verified via the gateway), this assistant-service build does not invoke it. The actual post-answer relevance work is the citation LLM above (hardcoded azure/gpt-5-chat), not a rerank-model call.

Priority to fix: post_answer_citations.py (azure/gpt-5-chat) is the clearest functional gap — it's on the answer path and 404s every time (silently). The fix is a source edit to a served model (e.g. qwen/qwen3-5-122b-a10b) — or register azure/gpt-5-chat as a LiteLLM alias → qwen on the Foundry gateway, which fixes all hardcoded azure/gpt-5-chat references at once without touching source. The alias approach is pragmatic for the source-hardcoded layer specifically (it's the one case where editing every call site is impractical).

Quick verification¶

# Layer 1 — what the running pod will send for internal calls
kubectl exec -n ask deploy/assistant-service-intelligence -- \
  cat /app/config/mod-auh1-dev-ask.toml | grep -iE "llm_model|reranker|embedding|data_analysis"

# Layer 2 — the catalog the pod actually loaded (after any ConfigMap override)
kubectl exec -n ask deploy/assistant-service-intelligence -- \
  cat /app/src/shared/llm/llm_models/models.yaml | grep -E "name:|api_model_name:"

Every api_model_name printed must appear in Foundry's /v1/models list (see Foundry → Validate).