ASK Platform — Service Flows¶

How each service participates in the four main platform flows. All flows assume the user has already logged in and holds a valid sessionToken.

Services at a Glance¶

Service	Role	Internal address
Frontend	Next.js web UI — chat, admin, document upload	`ask.mod.auh1.dev.dir`
Core Service	Auth proxy, org/user/model CRUD, JWT issuer	`core-service.ask.svc.cluster.local`
Zitadel	IAM / SSO — credential validation, org membership	`zitadel.ask.svc.cluster.local`
Assistant Service	Chat orchestration, RAG coordination, streaming	`assistant-service-intelligence.ask.svc.cluster.local`
KMS	Document ingestion API, retrieval, presigned URLs	`knowledge-management-service.ask.svc.cluster.local`
Hatchet	Async workflow engine — dispatches ingestion jobs	`hatchet-engine.ask.svc.cluster.local:7070` (gRPC)
Ingestion Worker	Pulls file from S3, chunks, embeds, upserts to Weaviate	`knowledge-management-service-ingestion-worker`
Foundry / LiteLLM	LLM proxy — chat model + embedding model	`foundry-litellm.foundry.svc.cluster.local:4000`
Weaviate	Vector store — cosine similarity search	`weaviate.weaviate.svc.cluster.local`
Ceph S3 (RGW)	Object storage — raw documents in `ask-knowledge-management` bucket	`rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local`
Vault + ESO	Secrets injection at pod startup	`vault.vault.svc.cluster.local`
CNPG	PostgreSQL clusters — `core-pg`, `knowledge-pg`, `hatchet-pg`	in-cluster

sequenceDiagram
    actor User
    participant FE as Frontend
    participant CS as Core Service
    participant Z as Zitadel
    participant DB as core-pg (CNPG)

    User->>FE: email + password
    FE->>CS: POST /api/v1/auth/login
    CS->>Z: gRPC — verify credentials + org membership
    Z->>DB: lookup projections.instance_members4 / org_members4
    DB-->>Z: membership record
    Z-->>CS: ✓ valid session
    CS->>DB: fetch org_id, role (ask71_admin / superadmin)
    CS-->>FE: sessionToken (JWT)
    FE-->>User: logged in — token stored in browser

Key points:

sessionToken is a JWT issued by Core Service, not Zitadel. All subsequent API calls send it as Authorization: Bearer <token>.
Core Service calls Zitadel using the IAM_SERVICE__PAT (from Vault via ESO at startup) — the CORE_SERVICE machine user must hold IAM Owner in Zitadel. Without this membership, all logins return 404 "membership not found".
Roles: ask71_admin → org-scoped access. superadmin → full platform + IAM access.

Flow 2 — Chat (no Knowledge Base)¶

sequenceDiagram
    actor User
    participant FE as Frontend
    participant CS as Core Service
    participant AS as Assistant Service
    participant F as Foundry (LiteLLM)
    participant H as Hatchet

    User->>FE: send message
    FE->>CS: POST /chat (Authorization: Bearer token)
    CS->>CS: validate JWT, attach org + user context
    CS->>AS: forward chat request + context
    AS->>AS: build system prompt (persona, tools, model config)
    AS->>F: POST /v1/chat/completions (qwen3-5-122b)
    F-->>AS: stream tokens
    AS-->>CS: stream tokens
    CS-->>FE: stream tokens (SSE)
    FE-->>User: response rendered in UI
    AS-)H: async — usage metering event

Key points:

The model qwen/qwen3-5-122b-a10b is served in-cluster via Foundry/LiteLLM — no external LLM call.
Responses are streamed end-to-end using Server-Sent Events (SSE).
Hatchet receives a fire-and-forget metering event after the response completes — does not block streaming.

Flow 3 — Chat with RAG (Knowledge Base enabled)¶

sequenceDiagram
    actor User
    participant FE as Frontend
    participant CS as Core Service
    participant AS as Assistant Service
    participant KMS as KMS
    participant W as Weaviate
    participant F as Foundry

    User->>FE: message + collection selected
    FE->>CS: POST /chat (with collection_ids)
    CS->>AS: forward + context
    AS->>KMS: POST /retrieve/?query=<user message>
    note over KMS: body: {retriever_parameters, collection_ids}
    KMS->>F: embed query → qwen3-embedding-0.6b (1024-dim)
    F-->>KMS: query vector
    KMS->>W: vector similarity search (cosine, top_k=25)
    W-->>KMS: top-k chunks with scores
    KMS->>KMS: rerank, apply similarity_threshold, character_limit
    KMS-->>AS: {documents: [{document_id, chunks: [{content, id, order}]}]}
    AS->>AS: inject chunks into system prompt as context
    AS->>F: POST /v1/chat/completions (qwen3-5-122b + context)
    F-->>AS: stream tokens (with citations referencing document_id)
    AS-->>FE: streamed response + chunk metadata
    FE-->>User: response with citation links

Key points:

The retrieve call is POST /retrieve/?query=<text> — query is a URL query parameter, not in the JSON body.
Response structure is documents[].chunks[] — not a flat chunks[] array.
similarity_threshold default is 0.7071 (√2/2). Set to 0.0 to disable and return all results.
Citations in the UI reference document_id + chunk.id from the retrieve response.

Flow 4 — Document Upload & Ingestion¶

This flow has two phases: a synchronous user-facing phase and an asynchronous background phase.

Phase A — Sync (user waits for upload confirmation)¶

sequenceDiagram
    actor User
    participant FE as Frontend
    participant CS as Core Service
    participant KMS as KMS
    participant S3 as Ceph S3 (RGW)

    User->>FE: select file + collection
    FE->>CS: POST /documents/upload/initiate
    CS->>KMS: POST /documents/upload/initiate {file_name, file_hash, collection_id}
    KMS->>S3: generate presigned PUT URL
    S3-->>KMS: presigned URL (short TTL)
    KMS-->>CS: {document_id, presigned_url, ...}
    CS-->>FE: document_id + presigned_url

    FE->>S3: PUT file directly (presigned URL) ← bypasses all pods
    S3-->>FE: 200 OK

    FE->>CS: PATCH /documents/upload/{doc_id}/update-status?upload_status=queued
    CS->>KMS: PATCH update-status
    KMS->>KMS: DB: set processing_status = queued
    KMS-)KMS: fire FastAPI background task → trigger Hatchet
    KMS-->>CS: 200 {processing_status: queued}
    CS-->>FE: 200
    FE-->>User: "processing..." + poll for ready

Phase B — Async (background ingestion worker)¶

sequenceDiagram
    participant KMS as KMS (background task)
    participant HA as Hatchet API
    participant HE as Hatchet Engine
    participant W as Ingestion Worker
    participant S3 as Ceph S3 (RGW)
    participant F as Foundry (embedding)
    participant WV as Weaviate
    participant DB as knowledge-pg

    KMS->>HA: POST /workflow-runs/trigger (gRPC → hatchet-engine:7070)
    HA->>HE: gRPC dispatch
    HE->>W: AMQP via RabbitMQ — run ingestion job
    W->>S3: GET raw file from bucket ask-knowledge-management
    S3-->>W: file bytes
    W->>W: parse + chunk (hierarchical strategy)
    W->>F: POST /v1/embeddings (qwen3-embedding-0.6b)
    F-->>W: 1024-dim vectors per chunk
    W->>WV: upsert chunks + vectors
    WV-->>W: ✓ indexed
    W->>DB: UPDATE documents SET processing_status='ready', workflow_run_id=<id>
    DB-->>W: ✓

Key points:

The file goes directly from the browser to Ceph S3 using the presigned URL — it never transits through any application pod. This is by design for large files.
The update-status endpoint only triggers the Hatchet workflow on the uploading → queued transition. Calling it again on an already-queued doc is a no-op.
Hatchet API must dial the engine on hatchet-engine.ask.svc.cluster.local:7070 (internal plaintext gRPC). The shared SERVER_GRPC_BROADCAST_ADDRESS points to the Nginx ingress (TLS), which causes a protocol mismatch — see Troubleshooting.
processing_status lifecycle: uploading → queued → processing → ready (or failed).

Flow 5 — Admin: Org Onboarding & Model Config¶

sequenceDiagram
    actor Admin
    participant FE as Frontend (admin UI)
    participant CS as Core Service
    participant Z as Zitadel
    participant KMS as KMS
    participant F as Foundry
    participant V as Vault + ESO

    Note over Admin,V: Org Onboarding
    Admin->>FE: create new department org
    FE->>CS: POST /organizations/onboard
    CS->>Z: create org + invited users (gRPC management API)
    Z-->>CS: org_id
    CS->>KMS: POST /collections/ (collection scoped to org)
    KMS-->>CS: collection_id
    CS-->>FE: org onboarded

    Note over Admin,V: Model Catalog
    Admin->>FE: view/enable models
    FE->>CS: GET /models
    CS->>F: GET /model/info (LiteLLM)
    F-->>CS: available models + aliases
    CS-->>FE: model list with org-scoped availability

    Note over Admin,V: Secrets (startup-time only)
    V->>CS: IAM_SERVICE__PAT (ESO VaultStaticSecret)
    V->>F: LLM API keys per model

Key points:

Each department gets its own KMS collection. Collection actors (/collections/{id}/actors) are set to the department's Zitadel group — users outside the group get 403 on collection reads.
Model API keys are injected at pod startup via Vault Secrets Operator (ESO VaultStaticSecret) — never in env files or ConfigMaps.
SAML/LDAP for MoD AD integration is not yet configured — pending Zitadel SAML connector setup.

Cross-Cutting Concerns¶

Concern	How it's handled
TLS	All ingress traffic terminates TLS at Nginx. Internal pod-to-pod traffic is HTTP/plaintext.
Secrets	Vault KV v2 + ESO VaultStaticSecret. Secrets injected as k8s Secrets at pod startup. Never in ConfigMaps or env files.
Config	ArgoCD ApplicationSet syncs all Helm values from `mod-ask-devops` Git repo. No manual `kubectl apply` in steady state.
Cert rotation	cert-manager issues internal TLS certs. Stakater Reloader watches cert secrets and restarts affected pods.
Observability	Resource limits/requests on all workloads. Hatchet dashboard at `hatchet.ask.mod.auh1.dev.dir`.

ASK Platform — Service Flows¶

Services at a Glance¶

Flow 1 — Authentication / Login¶

Flow 2 — Chat (no Knowledge Base)¶

Flow 3 — Chat with RAG (Knowledge Base enabled)¶

Flow 4 — Document Upload & Ingestion¶

Phase A — Sync (user waits for upload confirmation)¶

Phase B — Async (background ingestion worker)¶

Flow 5 — Admin: Org Onboarding & Model Config¶

Cross-Cutting Concerns¶