Core Service — Air-Gap Deployment Alignment¶

Every architectural change required to run core-service in an air-gapped customer environment — one where the cluster has no outbound internet access and pulls everything from internal registries. Changes are grouped by concern.

Overview

The upstream deployment assumes:

Assumption	Cloud reality	Air-gap reality
Container images	Pull from Azure ACR (`ai71uaenprodask71acr01.azurecr.io`)	Must mirror to internal registry
Helm subchart (zitadel)	Pull from `registry.gitlab.com/ai71.ai/...`	Must mirror + vendor into chart
Postgres	Azure managed Postgres (PaaS)	Self-hosted — CloudNativePG on Ceph
Object storage	Azure Blob Storage	Self-hosted — Ceph RGW (S3-compatible)
Ingress	Azure Web App Routing (`webapprouting.kubernetes.azure.com`)	`rke2-nginx` ingress class
TLS certificates	External/Azure managed	cert-manager with internal CA
Scheduling	Cloud node labels/taints (`node.kubernetes.io/role: ask71`)	Cleared — no equivalent labels
Load balancer	Azure cloud LB (`service.type: LoadBalancer`)	`ClusterIP` + Ingress
IDP / SSO	Zitadel with upstream IDP connectors (Azure AD, Okta, social login)	Standalone Zitadel — no upstream connectors; local user DB; Zitadel is the IDP

1. Container images

The problem¶

All images reference Azure ACR which is: - Unreachable from an air-gapped cluster - Architecture — production nodes are x86_64/amd64; upstream images are pulled for the native architecture and mirrored to Harbor (no cross-arch build needed)

The solution — mirror to Harbor registry¶

Each image is copied to harbor.cl1.sq4.aegis.internal/ask/<name> using skopeo:

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
GLT=$(vault kv get -field=token secret/gitlab/migration-token)

skopeo copy --dest-creds "root:$GLT" \
  docker://<source-registry>/<image>:<tag> \
  docker://harbor.cl1.sq4.aegis.internal/ask/<image>:<tag>

Images to mirror for v1 (API + auth + DB bootstrap):

Image	Destination
`core-service` API	`…/container-images/core-service:<tag>`
zitadel server	`…/container-images/zitadel:<ver>` (ref from vendored chart)
zitadel login UI	`…/container-images/zitadel-login:<ver>` (ref from vendored chart)
`postgres:16-alpine`	`…/container-images/postgres:16-alpine`

Images deferred to Phase 11 (workers):

Image	When
`core-service-bulk-operations-worker`	when workers enabled
`core-service-billing-metering-worker`	when workers enabled
`core-service-cron-scheduler-worker`	when workers enabled
`core-service-job-run-worker`	when workers enabled

Pull secret¶

The Harbor registry is private. Every namespace that runs these images must have the harbor-pull-secret dockerconfigjson secret:

ARGOCD_RO=$(vault kv get -field=token secret/gitlab/argocd-ro)
kubectl -n ask create secret docker-registry harbor-pull-secret \
  --docker-server=harbor.cl1.sq4.aegis.internal \
  --docker-username=argocd-ro \
  --docker-password="$ARGOCD_RO"

Referenced in values-ai71-uaen-prod-modgpt.yaml:

imagePullSecrets:
  - name: harbor-pull-secret

Production hardening

The pull secret is currently created manually per namespace. Production-correct is a VSO VaultStaticSecret that renders the dockerconfigjson from Vault, propagating automatically and rotating centrally.

2. Helm subchart (zitadel)

The problem¶

The chart declares a subchart dependency with a placeholder repository URL that will never resolve:

repository: "oci://your-internal-registry.example.com/devops-helm-charts"

Helm checks for the chart in charts/ before rendering, even when zitadel.enabled: false. Result without vendoring:

Error: found in Chart.yaml, but missing in charts/ directory: zitadel

The solution — vendor the subchart¶

# 1. Get the chart from the upstream source (you provide the .tgz)
#    upstream: oci://registry.gitlab.com/ai71.ai/devops/devops-helm-charts/zitadel:1.0.1

# 2. Push to our internal OCI registry
helm push zitadel-1.0.1.tgz oci://harbor.cl1.sq4.aegis.internal/modgpt/charts

# 3. Update Chart.yaml to point at the internal registry
#    repository: "oci://harbor.cl1.sq4.aegis.internal/modgpt/charts"

# 4. Vendor the chart into the repo (produces charts/ + Chart.lock)
cd helm && helm dependency build

# 5. Commit the vendored chart
git add charts/ Chart.lock && git commit -m "vendor: zitadel subchart for air-gap"
git push

Once charts/zitadel-1.0.1.tgz and Chart.lock are committed, ArgoCD renders the chart entirely offline — no registry pull at sync time. The Chart.yaml repository: URL is only used during helm dependency build, not at render/install time.

The chart was designed for this

The upstream Chart.yaml includes an explicit AIRGAP: comment documenting exactly this process — mirror the chart, update the repository URL, run helm dependency build. This is an intentional air-gap pattern by the upstream team.

3. Postgres — CloudNativePG instead of Azure managed Postgres

The problem¶

Every environment file hard-codes an Azure-managed Postgres hostname:

ai71-uaen-stg-ask71-psql-01.postgres.database.azure.com

This is a PaaS service — unavailable in an air-gapped environment.

The solution — CloudNativePG (CNPG) operator¶

Deploy the CloudNativePG operator as a cluster add-on (modgpt.cluster-addons.cnpg), then declare a Postgres Cluster CR in the ask namespace backed by Ceph RBD storage:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: ask-pg, namespace: ask }
spec:
  instances: 3                                    # 1 primary + 2 replicas (HA)
  storage: { size: 20Gi, storageClass: ceph-block }
  enableSuperuserAccess: true                     # needed for postgresInit Jobs

CNPG creates the service ask-pg-rw.ask.svc.cluster.local:5432 (→ primary, read/write). Both the chart's postgresInit and zitadelPostgresInit PreSync Jobs point at this service to create their respective DBs and roles.

Two databases on one Postgres server (created by the chart's PreSync hooks):

Database	Role	Created by
`zitadel`	`zitadel`	`zitadelPostgresInit` Job (wave −1)
`core-service`	`core-service`	`postgresInit` Job (wave −1)

The admin password (CNPG superuser) is stored in Vault at secret/ask/zitadel-psql and rendered as zitadel-psql-secret via VSO.

4. Object storage — Ceph RGW (deferred to Phase 11)

The problem¶

The app's default storage is Azure Blob:

[default.storage]
type = "azure"

Azure storage accounts are unreachable in this environment.

v1 — stubbed¶

For v1 the storage type is overridden to stub in values-ai71-uaen-prod-modgpt.yaml:

envConf:
  configToml: |
    [prod.storage]
    type = "stub"

The app starts without storage. Features that upload/download assets (e.g. bulk CSV processing) will fail gracefully — acceptable for initial bring-up.

Phase 11 — Ceph RGW (S3-compatible)¶

The Ceph cluster already exposes an RGW S3 endpoint (ceph-bucket storageclass). In Phase 11:

Create an RGW bucket + access/secret key for core-service.
Store credentials in Vault (secret/ask/core-service/storage), render via VSO.

Override in values-ai71-uaen-prod-modgpt.yaml:

envConf:
  configToml: |
    [prod.storage]
    type = "s3"
    [prod.storage.s3]
    endpoint = "http://rook-ceph-rgw-my-store.rook-ceph.svc.cluster.local"
    bucket = "core-service-assets"
    # access_key and secret_key from env via @format {env[...]}

5. Networking — ingress class + TLS

The problem¶

Production uses Azure Web App Routing as the ingress class:

ingressClassName: webapprouting.kubernetes.azure.com

Not available in RKE2. TLS certificates were managed externally.

The solution¶

Use the RKE2-native nginx ingress class + cert-manager with the internal CA:

# values-ai71-uaen-prod-modgpt.yaml
ingress:
  enabled: true
  className: rke2-nginx
  annotations:
    cert-manager.io/cluster-issuer: internal-ca-issuer
  hosts:
    - host: core-service.ask.mod.auh1.dev.dir
      paths: [{ path: /, pathType: Prefix }]
  tls:
    - secretName: core-service-tls
      hosts: [core-service.ask.mod.auh1.dev.dir]

Same for zitadel (sso.ask.mod.auh1.dev.dir) — replace webapprouting.kubernetes.azure.com with rke2-nginx.

Add both hostnames to /etc/hosts on the bastion (or to the internal DNS):

100.115.2.100  core-service.ask.mod.auh1.dev.dir
100.115.2.100  sso.ask.mod.auh1.dev.dir

6. Service type — ClusterIP

The problem¶

Default service.type: LoadBalancer triggers cloud LB provisioning (Azure/cloud-provider integration). No cloud provider exists in this environment → service stays <pending> indefinitely.

The solution¶

service:
  type: ClusterIP    # traffic enters via Ingress, not a cloud LB

7. Node scheduling constraints

The problem¶

The API Deployment carries cloud-specific scheduling:

nodeSelector:
  node.kubernetes.io/role: ask71    # doesn't exist on the nodes → pod Pending
tolerations:
  - key: node.kubernetes.io/role
    value: ask71
    effect: NoSchedule

The solution¶

nodeSelector: {}    # schedule on any available node
tolerations: []

8. Database SSL mode

The problem¶

Azure managed Postgres requires SSL (SSL.Mode: prefer). CNPG's internal cluster uses self-signed certificates — the init Jobs would fail SSL validation.

The solution¶

zitadel:
  zitadel:
    zitadel:
      configmapConfig:
        Database:
          Postgres:
            User:  { SSL: { Mode: disable } }
            Admin: { SSL: { Mode: disable } }

SSL between pods on the same cluster is acceptable to disable; network-level security is handled by Kubernetes network policies.

9. Migrations — new PreSync Job

The upstream chart has no migration mechanism. A new helm/templates/job-migrations.yaml was added with a toggle flag (see Sanitization and Config Updates):

migrations:
  enabled: true    # in values-ai71-uaen-prod-modgpt.yaml

This runs alembic upgrade head as a PreSync ArgoCD hook at sync-wave 0 — after the DB is created (wave −1) and before any Deployment starts (wave 5).

10. SSO — standalone Zitadel (Zitadel is the IDP)

Decision¶

Zitadel is the IDP — there is no external identity provider. Zitadel runs standalone, manages its own user database, and issues OIDC tokens directly to the frontend and core-service. No upstream connectors (Azure AD, Okta, LDAP) are configured or needed.

Two auth flows that require full SSO¶

Flow	Who	How
User-facing	End users → frontend → Zitadel login UI → OIDC token	`sso.ask.mod.auh1.dev.dir/ui/v2/login` (login subchart)
Service-to-service	`core-service` API → Zitadel API	`IAM_SERVICE__PAT` (Personal Access Token) — backend only

Both flows are needed because the frontend is in scope for v1.

What's configured in `values-ai71-uaen-prod-modgpt.yaml`¶

zitadel:
  enabled: true
  zitadel:
    login:
      enabled: true                    # ← login UI deployed
      ingress:
        enabled: true
        hosts:
          - host: sso.ask.mod.auh1.dev.dir
            paths: [{ path: /ui/v2/login, pathType: Prefix }]
    zitadel:
      configmapConfig:
        ExternalSecure: true
        ExternalDomain: sso.ask.mod.auh1.dev.dir   # ← public SSO URL

Post-deploy: register the frontend as an OIDC client¶

Zitadel's seedData creates the org and superadmin but does not create an OIDC application for the frontend. After Zitadel first boots, a one-time manual step registers the frontend:

Log into Zitadel at https://sso.ask.mod.auh1.dev.dir as the seeded superadmin.
Navigate to Projects → Create Project (e.g. ask).
Add an Application → type Web → authentication method PKCE (recommended) or Code.
Set the Redirect URI to the frontend's callback URL (e.g. https://ask.mod.auh1.dev.dir/auth/callback).
Copy the Client ID (and Client Secret if not PKCE).
Store in Vault: vault kv put secret/ask/frontend oidc_client_id=<id> oidc_client_secret=<secret>
Render into the frontend's namespace via VSO, same pattern as core-service-secret.

The frontend then uses: - Issuer: https://sso.ask.mod.auh1.dev.dir - Client ID / Secret: from Vault via VSO - Scopes: openid profile email (minimum)

PAT for service-to-service

The IAM_SERVICE__PAT in core-service-secret is a separate Zitadel Personal Access Token (not the OIDC client secret). It's created in Zitadel under a service account and gives core-service API-level access to manage users and sessions. Generate it post-deploy the same way: Zitadel UI → Service Users → create → generate PAT → store in Vault.

Summary — what changed vs upstream

Concern	Upstream (cloud)	Air-gap
Images	Azure ACR	Harbor container registry
Helm subchart	`registry.gitlab.com/ai71.ai/...`	Vendored from internal OCI registry
Postgres	Azure managed Postgres	CloudNativePG on Ceph
Storage	Azure Blob	Stubbed v1; Ceph RGW S3 in Phase 11
Ingress class	`webapprouting.kubernetes.azure.com`	`rke2-nginx`
TLS	External/Azure managed	cert-manager + internal CA
Service type	`LoadBalancer`	`ClusterIP`
Node scheduling	`node.kubernetes.io/role: ask71`	Cleared (`{}` / `[]`)
DB SSL	`prefer`	`disable` (intra-cluster)
Migrations	Not automated	PreSync Helm Job (new template)
IDP / SSO	Zitadel + upstream IDP connectors (federated)	Standalone Zitadel — Zitadel is the IDP; local user DB; no upstream connectors

➡️ Related: Discovery · Sanitization · Config Updates · Deployment Runbook

Core Service — Air-Gap Deployment Alignment¶

The problem¶

The solution — mirror to Harbor registry¶

Pull secret¶

The problem¶

The solution — vendor the subchart¶

The problem¶

The solution — CloudNativePG (CNPG) operator¶

The problem¶

v1 — stubbed¶

Phase 11 — Ceph RGW (S3-compatible)¶

The problem¶

The solution¶

The problem¶

The solution¶

The problem¶

The solution¶

The problem¶

The solution¶

Decision¶

Two auth flows that require full SSO¶

What's configured in values-ai71-uaen-prod-modgpt.yaml¶

Post-deploy: register the frontend as an OIDC client¶

What's configured in `values-ai71-uaen-prod-modgpt.yaml`¶