KMS — Model Reference¶

Every AI/ML call made by the Knowledge Management Service, which model it uses, how it is configured, and its air-gap status.

KMS is the cleanest of the three ASK services from an air-gap perspective: every model comes from Layer 1 config (TOML overrides in values-mod-auh1-dev-ask.yaml) — no baked catalogs, no hardcoded model IDs in source.

Model map (quick reference)¶

Feature	Model called	Foundry served?	Status
Document embedding (ingest)	`qwen/qwen3-embedding-0-6b`	✅ yes	✅ Active
Query embedding (search)	`qwen/qwen3-embedding-0-6b`	✅ yes	✅ Active
VLM / OCR (visual doc parsing)	`qwen/qwen3-5-122b-a10b-kms`	✅ yes (KMS-dedicated instance)	✅ Active
Retrieval reranker	`nvidia/llama-nemotron-rerank-1b-v2`	✅ yes	✅ Active (enabled via env override)
Document summarization	`document_understanding.summarization.model`	❌ not configured	🚫 Disabled (`@none` in default.toml; not overridden)
Summary embedding blend	same as embedding model	N/A	🚫 Disabled (no summary → no blend vector)
Keyword / NER extraction	External NER endpoint (`ner_path` + `ner_token`)	❌ not a Foundry model	🚫 Disabled (`@none` in default.toml; not overridden)

Feature detail¶

1. Document Embedding¶

When: every document ingest — after chunking, each chunk is embedded and stored in Weaviate.

Config key: [organization] embedding_model_id (L1, values-mod-auh1-dev-ask.yaml)

Field	upstream default	Air-gap override
`embedding_model_id`	`BAAI/bge-m3` ❌ (not served)	`qwen/qwen3-embedding-0-6b` ✅
`blend_factor`	`0.75`	unchanged (blending disabled — no summary)

Code path: hatchet_workers/services/embedding_service.py → EmbeddingService.ingest() → embedder.embed_batch(texts, embedding_model_id) → Foundry /v1/embeddings.

The model id is read from org_config.embedding_model_id (the stored org onboarding value, which defaults to [organization] embedding_model_id from config). Each org can have its own embedding model — new orgs onboarded under our config will use qwen/qwen3-embedding-0-6b.

2. Query Embedding¶

When: every retrieval request — the user's search query is embedded to find similar chunks in Weaviate.

Config key: same [organization] embedding_model_id — the retriever uses the collection's stored model, not a separate config key.

Code path: knowledge_management_service/retriever_pipeline/embedding.py → get_query_embedding(query, embedding_model_id, embedding_service) → same Foundry embeddings endpoint.

The collection stores the embedding_model_id used when documents were ingested — retrieval always uses the same model as ingest. Mismatching these (e.g. re-ingesting with a different model) would corrupt search quality.

3. VLM / OCR (Visual Document Parsing)¶

When: during ingest, when the parsing_strategy = "generic" parser detects a visual document (images, scanned PDFs, non-selectable PDF pages) and sends pages to the VLM for text extraction.

Config key: [organization] vlm_model_name (L1)

Field	upstream default	Air-gap override
`vlm_model_name`	`core42-oicm-auh1-shd/qwen-3-5-9b` ❌ (not served)	`qwen/qwen3-5-122b-a10b-kms` ✅
`parsing_strategy`	`quick`	`generic` (handles both text and visual docs)

The -kms suffix is a dedicated vLLM instance (vllm-qwen3-5-122b-a10b-kms-engine-service) isolated from the main assistant-service traffic. This means heavy OCR processing (e.g. large scanned PDFs) does not contend with live user chat.

Code path: document_understanding library (vendored) → DocumentProcessingService.parse() → calls Foundry /v1/chat/completions with the page image and VLM model.

4. Retrieval Reranker¶

When: after vector search — results are passed to the reranker, reordered by relevance, and trimmed using an elbow-cutoff algorithm.

Config key: [retriever.reranker] block (L1)

Field	upstream default	Air-gap value
`reranker`	`@none` (disabled)	full block enabled
`model`	—	`nvidia/llama-nemotron-rerank-1b-v2` ✅
`candidate_pool_multiplier`	—	`2.0` (rerank 2× the requested `top_k`)
`elbow_cutoff_min_k`	—	`5` (keep ≥5 results after elbow)
`elbow_min_points`	—	`5` (min candidates to attempt elbow detection)

The elbow cutoff trims results at the natural relevance drop-off point using max-perpendicular-distance on the score curve. If the reranker call fails, it silently falls back to vector-search ordering (search_results[:top_k]).

Code path: knowledge_management_service/retriever_pipeline/reranking.py → rerank_search_results() → RerankerService.rerank() → Foundry /v1/rerank.

5. Document Summarization¶

When: optional step during ingest — generates a text summary of the full document, stored as document.summary. If a summary exists, it is embedded and blended with each chunk embedding at blend_factor weight.

Config key: [document_understanding] summarization block (L1 — currently @none)

Field	upstream default	Air-gap value
`summarization`	`@none`	`@none` — not configured, disabled
`model`	(e.g. `azure/gpt-5-mini`)	N/A
`fallback_model`	`@none`	N/A
`reasoning_effort`	`@none`	N/A

Impact of being disabled: config.document_understanding.summarization is None → summarize_document_task logs "Summarization is not configured, skipping" and returns immediately. No LLM call is made. document.summary remains None, so summary embedding blending is also skipped (embed step checks if document.summary).

To enable: add a [mod-auh1-dev-ask.document_understanding.summarization] block to the TOML with model = "qwen/qwen3-5-122b-a10b" (or the KMS-dedicated instance). The SummarizationConfig requires model, max_document_chars, and fallback_model/reasoning_effort (both nullable).

6. Keyword / NER Extraction¶

When: optional step during ingest — extracts named entities (keywords) at document, section, and subsection levels from chunk content. Keywords are stored in chunk.chunk_metadata and appended to the embedding text to improve retrieval quality.

Config key: [document_understanding] keyword_extraction block (L1 — currently @none)

Field	upstream default	Air-gap value
`keyword_extraction`	`@none`	`@none` — not configured, disabled
`ner_path`	—	N/A
`ner_token`	—	N/A
`min_score`	—	N/A
`min_length`	—	N/A

This is NOT a Foundry model call — keyword extraction uses a dedicated NER endpoint (ner_path relative to the Foundry origin, with its own ner_token). It does not use the embedding or chat completion endpoint. No NER model is deployed in our Foundry configuration, so this feature cannot be enabled without also deploying a NER inference backend.

Impact of being disabled: config.document_understanding.keyword_extraction is None → extract_keywords_task skips immediately. chunk.chunk_metadata stays empty. Chunks are embedded with content only (no keyword augmentation).

Config of record (`values-mod-auh1-dev-ask.yaml`)¶

[mod-auh1-dev-ask.document_understanding]
foundry_endpoint = "http://foundry-litellm.foundry.svc.cluster.local:4000/v1"
# summarization = @none  (default — disabled)
# keyword_extraction = @none  (default — disabled)

[mod-auh1-dev-ask.organization]
embedding_model_id = "qwen/qwen3-embedding-0-6b"    # 1024-dim; matches Weaviate vectors
vlm_model_name     = "qwen/qwen3-5-122b-a10b-kms"  # KMS-dedicated vLLM instance

[mod-auh1-dev-ask.retriever.reranker]
model                    = "nvidia/llama-nemotron-rerank-1b-v2"
candidate_pool_multiplier = 2.0
elbow_cutoff_min_k        = 5
elbow_min_points          = 5

Default.toml upstream values (overridden above):

Field	Upstream default	Why it breaks without override
`embedding_model_id`	`BAAI/bge-m3`	Not served in Foundry → 404 on every ingest
`vlm_model_name`	`core42-oicm-auh1-shd/qwen-3-5-9b`	Not served → 404 on visual doc parsing
`retriever.reranker`	`@none`	Reranker disabled → lower retrieval quality

Layer classification¶

All KMS model references are Layer 1 — fully controlled by our Helm values TOML.

Layer	KMS state
L1 — config TOML	All active models: embedding, VLM, reranker ✅
L2 — baked catalog	None — KMS has no user-selectable model catalog
L3 — hardcoded in source	None (one exception: Azure DI endpoint in `document_understanding.py` — gated behind `parsing_strategy = AZURE_DOCUMENT_INTELLIGENCE`, which we don't use)

KMS is the cleanest service for air-gap

No L2 catalog overrides needed. No L3 source patches needed. All three active models are set in one TOML block and all three are served by Foundry.

Ingestion pipeline — model call sequence¶

For a typical document ingest, the model calls happen in this order:

1. PARSE   → VLM (qwen/qwen3-5-122b-a10b-kms)     [only for visual/scanned docs]
2. CHUNK   → no model call (text splitting only)
3. SUMMARIZE → LLM (disabled in air-gap)
4. KEYWORD → NER endpoint (disabled in air-gap)
5. EMBED   → embedding model (qwen/qwen3-embedding-0-6b)
             [+ optional summary blend — disabled because step 3 is off]

For retrieval:

1. EMBED QUERY → embedding model (qwen/qwen3-embedding-0-6b)
2. VECTOR SEARCH → Weaviate (no model call)
3. RERANK → reranker (nvidia/llama-nemotron-rerank-1b-v2) [top_k × 2 candidates → elbow cut]

Open items¶

Priority	Item	Status
🟡 Low	Enable document summarization	Needs `[document_understanding.summarization]` block + choose model (`qwen/qwen3-5-122b-a10b` or KMS instance)
📋 Backlog	Enable keyword / NER extraction	Requires dedicated NER model deployment (not a LiteLLM model); no NER backend in Foundry currently
⚠️ Verify	Reranker E2E test	Model is live and config is enabled — but has not been exercised via a full search flow. Test with a retrieval query and confirm ranked results.