Foundry & Models — Overview
Foundry is the LLM gateway layer of the ASK platform. It is a LiteLLM proxy that sits in front of all vLLM inference backends and presents a single OpenAI-compatible API endpoint to the rest of the platform. Every LLM call — whether from a user chat or an internal service — goes through Foundry.
What it does
| Responsibility |
Detail |
| Model routing |
Translates model names (qwen/qwen3-5-122b-a10b, z-ai/glm-5, etc.) to the correct vLLM engine endpoint |
| Per-org key management |
Core Service calls the Foundry admin API (/model/new, /key/generate) to issue scoped API keys per organisation |
| Load balancing |
Routes requests across multiple vLLM replicas when configured; respects TPM/RPM budgets per key |
| Embedding & rerank |
Proxies embedding (qwen/qwen3-embedding-0-6b) and rerank (nvidia/llama-nemotron-rerank-1b-v2) requests alongside chat completions |
| Redis caching |
Caches identical requests in Redis to reduce GPU load for repeated queries |
Served models (air-gap)
| Model name in Foundry |
Backend |
Used by |
qwen/qwen3-5-122b-a10b |
hosted_vllm/Qwen/Qwen3-5-122B-A10B |
assistant-service (main LLM, data viz) |
qwen/qwen3-5-122b-a10b-kms |
hosted_vllm/Qwen/Qwen3-5-122B-A10B |
KMS (dedicated vLLM instance — separate endpoint) |
z-ai/glm-5 · zai-org/glm-5-2 |
hosted_vllm/zai-org/GLM-5.2 |
user-selectable GLM-5.2 (new 2026-07) |
Qwen/Qwen3-4B-Instruct-2507 |
hosted_vllm/Qwen/Qwen3-4B-Instruct-2507 |
assistant-service (small LLM, summaries) |
qwen/qwen3-embedding-0-6b |
hosted_vllm/Qwen/Qwen3-Embedding-0.6B |
assistant-service + KMS embeddings |
nvidia/llama-nemotron-rerank-1b-v2 |
hosted_vllm/nvidia/llama-nemotron-rerank-1b-v2 |
assistant-service + KMS reranking |
foundry-stg/.../qwen-3-5-122b-a10b (alias) |
→ qwen/qwen3-5-122b-a10b |
models.yaml compat |
qwen/qwen3-5-4b, qwen3-5-4b (compat redirect) |
→ Qwen/Qwen3-4B-Instruct-2507 |
backward compat — old small model removed |
Integrations
| System |
Role |
| vLLM engines |
Actual GPU inference — one deployment per model, in-cluster DNS names in foundry namespace |
| Core Service |
Admin API client — provisions per-org LiteLLM keys and queries model availability |
| Assistant Service |
Primary LLM consumer — all chat completions, code generation, embeddings, reranking |
| Knowledge Management Service |
Embedding generation and VLM calls during document ingestion and retrieval |
| Redis |
Response cache — foundry-litellm-redis-master.foundry.svc.cluster.local |
Key endpoint
http://foundry-litellm.foundry.svc.cluster.local:4000
- Chat completions:
POST /v1/chat/completions
- Model list:
GET /v1/models
- Model info (admin):
GET /model/info (requires admin key)
- Add model alias (admin):
POST /model/new — requires STORE_MODEL_IN_DB=True; otherwise edit the ConfigMap directly
Pages in this section
| Page |
Purpose |
| Model Aliases & Testing |
How to add model aliases to the Foundry ConfigMap and verify them; alias fix runbook for aicloud/ and foundry-stg/ prefixes |
| E2E Testing (no frontend) |
End-to-end smoke tests that bypass the UI — direct Foundry calls, assistant-service curl tests |
| Platform Capability Matrix |
Which features work end-to-end in the air-gap environment vs. what requires cloud connectivity |