Core Service — Discovery¶

Findings from reading the source repository and Helm chart. This page explains how the application deploys today with the upstream source code, and what we learned that drives the sanitization and air-gap alignment decisions.

Internet Egress Map¶

Every outbound internet call the application makes — at build time, at deploy time, and at runtime. This is the starting point for air-gap alignment.

Full internet dependency by default

Out of the box, core-service requires internet access at every phase: image pulls from Azure ACR, Python packages from private Azure feeds, runtime calls to Azure PaaS services, and outbound AI inference calls. Nothing works air-gapped without the changes documented in Air-Gap Alignment.

Build-time — internet calls during docker build

What	Endpoint	Why
Base image pull	`docker.io/library/python:3.12`	Builder stage base
Base image pull	`docker.io/library/python:3.12-slim`	Runtime stage base
Private Python packages (×5)	`UV_INDEX_CONNECTORS_LIB` — ai71 internal Azure feed	connectors-lib
Private Python packages	`UV_INDEX_IAM_CLIENT` — ai71 internal Azure feed	iam-client
Private Python packages	`UV_INDEX_LOGGER_SDK` — ai71 internal Azure feed	logger-sdk
Private Python packages	`UV_INDEX_FOUNDRY_CLIENT` — ai71 internal Azure feed	foundry-client
Private Python packages	`UV_INDEX_STORAGE_LIB` — ai71 internal Azure feed	storage-lib

Air-gap resolution: images are pre-built externally and mirrored to the Harbor container registry. Build-time internet calls do not happen in this environment — the image is pulled from harbor.cl1.sq4.aegis.internal.

Deploy-time — internet calls during Helm install / ArgoCD sync

What	Endpoint	Why
Application images (×5)	`ai71uaenprodask71acr01.azurecr.io`	Azure ACR — all workload images
Postgres init image	`docker.io/library/postgres:16-alpine`	PreSync job base image
Busybox	`docker.io/library/busybox`	Helm test pod only
Zitadel subchart	`registry.gitlab.com/ai71.ai/devops/devops-helm-charts/zitadel:1.0.1`	OCI Helm dependency
Zitadel images	refs inside vendored zitadel chart	zitadel server + login UI

Air-gap resolution: all images mirrored to harbor.cl1.sq4.aegis.internal/ask/. Zitadel chart vendored into charts/ directory. See Air-Gap Alignment → Images.

Runtime — outbound calls from running pods

What	Endpoint	Protocol	Which pods	Can be stubbed?
AI model inference	AI71 API Gateway (internet)	HTTPS	API, job-run-worker	❌ Core function — must reach gateway or use mock
Azure Postgres	`ai71-uaen-stg-ask71-psql-01.postgres.database.azure.com:5432`	TCP	All pods (DB connection pool)	✅ → CNPG in-cluster
Azure Blob Storage	`*.blob.core.windows.net`	HTTPS	API, bulk-operations-worker	✅ → Ceph RGW S3 (Phase 11) or `stub` type
Mailchimp API	`api.mailchimp.com`	HTTPS	API (notifications)	✅ → disabled in air-gap values
SendGrid API	`api.sendgrid.com`	HTTPS	API (SMTP fallback)	✅ → disabled in air-gap values
Hatchet orchestrator	Hatchet server (internal or external)	gRPC	billing-metering-worker, job-run-worker	✅ → Hatchet deployed in-cluster
Zitadel IAM	Zitadel server	HTTPS	API (`IAM_SERVICE__PAT`)	✅ → Zitadel deployed in-cluster
Redis	Redis cluster	TCP	All workers (task queue)	✅ → Redis deployed in-cluster

AI gateway is the only unavoidable internet dependency at runtime

Every other runtime dependency can be self-hosted (Postgres → CNPG, Blob → Ceph, Zitadel → in-cluster, Hatchet → in-cluster, notifications → disabled). The AI71 API gateway for model inference is the one call that genuinely requires internet egress — unless a local model serving stack (e.g. vLLM) is deployed and the model catalog is redirected to it.

Summary — air-gap egress status

Phase	Internet needed?	Resolution
Build	❌ Not in this environment	Pre-built images mirrored to Harbor registry
Deploy (images)	❌ Not in this environment	All images in `harbor.cl1.sq4.aegis.internal`
Deploy (Helm chart)	❌ Not in this environment	Zitadel vendored; chart in GitLab Helm registry
Runtime — Postgres	❌ Replaced	CloudNativePG on Ceph
Runtime — Object storage	❌ Stubbed (Phase 11 → Ceph)	`stub` type for v1
Runtime — Notifications	❌ Disabled	Keys not provided
Runtime — Hatchet	❌ Replaced	Deployed in-cluster
Runtime — Zitadel	❌ Replaced	Deployed in-cluster
Runtime — Redis	❌ Replaced	Deployed in-cluster
Runtime — AI inference	⚠️ Required	This environment needs egress to AI71 gateway, OR local vLLM

Repository structure

ask-core-service-development/
├── src/                         ← Python application (FastAPI + workers)
│   ├── core_service/            ← main API service
│   ├── billing_metering/        ← billing worker logic
│   ├── model_provider/          ← AI model abstraction
│   └── config/                  ← Dynaconf config reader
├── tests/                       ← pytest test suite
├── alembic/ + alembic.ini       ← DB migration scripts (copied into image)
├── config/
│   ├── default.toml             ← non-secret defaults (version-controlled)
│   ├── local.toml               ← local dev overrides (gitignored)
│   └── test.toml                ← test overrides
├── helm/                        ← Helm chart (the only part the cluster needs)
│   ├── Chart.yaml               ← chart metadata + zitadel OCI dependency
│   ├── values.yaml              ← defaults for all environments
│   ├── templates/               ← Go-templated Kubernetes manifests
│   └── environments/            ← per-environment values files
├── Dockerfile                   ← API image
├── Dockerfile.bulk_operations_worker
├── Dockerfile.billing_metering_worker
├── Dockerfile.cron_scheduler_worker
├── Dockerfile.job_run_worker
├── scripts/                     ← dev/ADO ops scripts
├── security/                    ← Azure DevOps CI pipeline YAML
├── .cruft.json                  ← ADO project template metadata
└── sonar-project.properties     ← SonarCloud config (ai71 org)

How the application image is built

The Dockerfile uses a two-stage build — builder + runtime:

Stage 1 (builder):
  base: python:3.12
  - installs uv
  - runs uv sync with private index creds (mounted as build secrets)
    → pulls packages from ai71's internal package feed:
       UV_INDEX_CONNECTORS_LIB   (connectors-lib)
       UV_INDEX_IAM_CLIENT        (iam-client)
       UV_INDEX_LOGGER_SDK        (logger-sdk)
       UV_INDEX_FOUNDRY_CLIENT    (foundry-client)
       UV_INDEX_STORAGE_LIB       (storage-lib)
    All 5 indexes require REGISTRY_USERNAME + secret-mounted password.
  - produces /app/.venv

Stage 2 (runtime):
  base: python:3.12-slim
  - copies .venv from builder
  - COPY alembic.ini + alembic/     ← migrations IN the image
  - COPY src/                       ← application source IN the image
  - COPY config/default.toml        ← non-secret config IN the image
  - EXPOSE 8000
  - CMD ["uvicorn", "core_service.app:app", "--host", "0.0.0.0", "--port", "8000"]

Key finding: the image starts uvicorn only — it does NOT run alembic upgrade head. Migrations must be triggered separately.

How the application is configured

The app uses Dynaconf with a two-layer config model:

Layer	File	Contains	Committed?
Defaults	`config/default.toml`	all non-secret settings (DB host, pool size, feature flags, etc.)	✅ yes
Secrets	`.env` / environment variables	passwords, tokens, API keys — `@format {env[...]}` references	❌ never committed

@format {env[SOME__KEY]} tells Dynaconf to read the value from the environment at runtime — the config file itself contains no secrets, only the key names. In Kubernetes this translates to envFrom: secretRef: core-service-secret in every workload.

Environment selection — the app reads APP__ENV to pick the TOML section:

[ai71-uaen-stg-ask71.database]
host = "ai71-uaen-stg-ask71-psql-01.postgres.database.azure.com"

The Helm chart injects this via configmap-config-toml.yaml, mounted at /app/config/override.toml.

What the Helm chart deploys

The chart (helm/) renders the following Kubernetes resources when all components are enabled:

Kind	Name	Notes
Deployment	`core-service`	API — 1 replica default, HPA on
Deployment	`core-service-bulk-operations-worker`	3 replicas, 4Gi each, HPA max 20
Deployment	`core-service-billing-metering-worker`	1 replica
Deployment	`core-service-cron-scheduler-worker`	1 replica
Deployment	`core-service-cron-run-worker`	3 replicas (image: `job-run-worker`), HPA max 20
Service	`core-service`	LoadBalancer by default
Ingress	`core-service`	disabled by default
HPA	`core-service`	CPU 80%, min 1 / max 4
ConfigMap	`core-service-config-toml`	Dynaconf TOML override (env-specific)
ConfigMap	`core-service-model-config`	Full AI model catalog (40+ models, see `modelConfig.models` in `values.yaml`)
ServiceAccount	×5	one per workload
Job	`core-service-postgres-role-init`	creates DB + role (PreSync hook)
Job	`core-service-postgres-seed`	seeds org + superadmin (PreSync hook)
Job	`core-service-migrations`	⭐ new — `alembic upgrade head` (PreSync hook)
Pod	`core-service-test-connection`	helm test only (busybox wget)
Subchart	zitadel server + login UI	via `zitadel` OCI dependency

YAML bug — duplicate postgresInit key in values.yaml

values.yaml defines postgresInit: twice at the top level (lines 266 and 339 of the committed file). Go YAML (Helm's parser) silently takes the last occurrence, which is the bare enabled: false. The comprehensive first block (connection, db, secrets, argocd hooks) is overridden and has no effect.

Impact: postgresInit in the base chart has no host/db/secrets wired — every setting must be provided by the environment overlay. The values-ai71-uaen-prod-modgpt.yaml overlay must supply the complete postgresInit block (which it does — see Config Updates). But if you deploy without an overlay, postgresInit silently does nothing even though the first block looks comprehensive.

Fix: the duplicate bare block at line 339 is removed in the sanitized air-gap repo. See Sanitization.

Uncommitted changes in dev repo

The migrations: block in values.yaml and helm/templates/job-migrations.yaml are uncommitted working-tree changes — not yet in the main branch of ask-core-service-development. The committed chart has neither. These must be included when pushing to the lab GitLab repo. See Sanitization → Files added for the reminder.

Helm chart dependency — zitadel

The chart declares a subchart dependency in Chart.yaml:

dependencies:
  - name: zitadel
    version: "1.0.1"
    repository: "oci://your-internal-registry.example.com/devops-helm-charts"
    condition: zitadel.enabled

Critical finding: the repository: value is a placeholder (your-internal-registry.example.com). The chart was clearly designed for air-gap from the start — it expects the consumer to replace this with their internal registry. The upstream source is:

oci://registry.gitlab.com/ai71.ai/devops/devops-helm-charts/zitadel:1.0.1

Second critical finding: condition: zitadel.enabled only controls whether the subchart is rendered — Helm still requires the chart to be present in charts/ even if zitadel.enabled: false. Setting the condition to false without vendoring the chart produces:

Error: found in Chart.yaml, but missing in charts/ directory: zitadel

This means the subchart must be vendored (helm dependency build) before the chart can be rendered or installed at all.

Database migrations

How they are bundled into the image:

COPY alembic.ini ./
COPY alembic/ ./alembic/    ← migration scripts inside the container

How they are NOT triggered:

CMD ["uvicorn", "core_service.app:app", ...]   ← starts API only, no migration step

Migrations must be run explicitly via alembic upgrade head. The repo provides scripts/apply_migrations.sh for local dev (uses uv run alembic upgrade head). In the cluster the correct pattern is a PreSync Helm Job — see Config Updates.

Secrets the application needs

Every workload mounts core-service-secret via envFrom. The secret keys come from @format {env[...]} references in config/default.toml and the per-environment TOML files:

Secret key	Used for
`DATABASE__PASSWORD`	Postgres connection
`SECURITY__API_KEY_SALT`	API key hashing
`SECURITY__MASTER_ENCRYPTION_KEY`	Connector token encryption
`IAM_SERVICE__PAT`	Zitadel personal access token
`STORAGE__ACCOUNT_KEY`	Azure Blob / S3 storage (stub in stg)
`NOTIFICATIONS__MAILCHIMP_API_KEY`	Email notifications
`NOTIFICATIONS__SENDGRID_API_KEY`	SMTP fallback
`BILLING_METERING__*`	Billing/Autumn + Hatchet tokens
`REDIS__PASSWORD`	Redis (empty in this environment — Redis has no auth)
`APP__CORE_SERVICES_MACHINE_TOKEN`	Inter-service auth token

In this environment all of these are stored in Vault (secret/ask/core-service/*) and rendered into the namespace via VSO.

Images

Five container images, one per workload, all on Azure ACR:

Image	Dockerfile	Worker
`ai71uaenprodask71acr01.azurecr.io/core-service`	`Dockerfile`	API
`ai71uaenprodask71acr01.azurecr.io/core-service-bulk-operations-worker`	`Dockerfile.bulk_operations_worker`	Bulk ops
`ai71uaenprodask71acr01.azurecr.io/core-service-billing-metering-worker`	`Dockerfile.billing_metering_worker`	Billing
`ai71uaenprodask71acr01.azurecr.io/core-service-cron-scheduler-worker`	`Dockerfile.cron_scheduler_worker`	Cron scheduler
`ai71uaenprodask71acr01.azurecr.io/core-service-job-run-worker`	`Dockerfile.job_run_worker`	Job runner

All are tagged :latest in the default values.yaml. Additional images used at runtime (not built from this repo):

Image	Used by
`postgres:16-alpine`	`postgresInit` + `zitadelPostgresInit` PreSync Jobs
zitadel server	zitadel subchart (refs in vendored chart)
zitadel login UI	zitadel subchart
`busybox`	helm test pod only

Health checks

The API has both probes configured in values.yaml:

livenessProbe:
  httpGet: { path: /api/v1/health/live, port: http }
  initialDelaySeconds: 15 / periodSeconds: 15

readinessProbe:
  httpGet: { path: /api/v1/health/ready, port: http }
  initialDelaySeconds: 20 / periodSeconds: 10

Readiness requires the app to be fully connected (DB, Redis, IAM). A failing external dependency (Azure Blob, Zitadel unreachable) will fail readiness.

Model configuration (modelConfig)

values.yaml contains a modelConfig section that renders core-service-model-config as a ConfigMap and mounts it into the API pod at APP__MODEL_CONFIG_DIR (/app/config/models):

modelConfig:
  enabled: true
  mountPath: /app/config/models
  configMapName: core-service-model-config
  configMapAutoReloadOnUpdate: true
  models:
    models:
      - id: "anthropic/claude-sonnet-4-5"
        ...  # 40+ entries

The catalog covers:

Provider	Models
Anthropic	Claude Sonnet 4/4.5/4.6, Haiku 4.5, Opus 4.6, internal-thinking variants
OpenAI	GPT-4o, GPT-5, GPT-5.2/5.3/5.4, GPT-5-mini/nano/chat, GPT-OSS-120B
xAI	GROK-4, GROK-4.1-FAST (English + Arabic)
Zhipu AI	GLM-4-7, GLM-5
Minimax	MINIMAX-M2-1, MINIMAX-M2-5
Alibaba Cloud	Qwen2.5-32B, Qwen3-Next-80B, QWEN-3-5-122B
Meta	Llama3.3-70B
AI71	Falcon3-10B/7B, AgriLLM fine-tuned variants (Falcon/Llama/Qwen)
Google	Gemini 2.5 Flash
BAAI	bge-m3 (embedding, default)

Consideration: the model catalog is baked into values.yaml as config — no secrets, safe to deploy as-is. The models listed here are what the API will advertise; actual inference calls go through the upstream AI71 API gateway, so only the gateway credentials (in core-service-secret) need to exist.

Reloader integration

The API Deployment carries two Reloader annotations:

annotations:
  secret.reloader.stakater.com/reload: core-service-secret
  configmap.reloader.stakater.com/reload: core-service-model-config,core-service-config-toml

Restarts trigger on: secret rotation (Vault/VSO cycle), model catalog update, or TOML config change. All three are wired automatically — no extra config needed.

➡️ Related: Sanitization · Air-Gap Alignment · Config Updates · Deployment Runbook