Repo Sanitization — Pushing Dev Repos to the Air-Gap Environment¶

Purpose: a repeatable, auditable process for taking a development repository and pushing only what is relevant and safe to our GitLab (gitlab.cl1.sq4.aegis.internal). Applies to every repo we onboard — ask-core-service is the worked example, but every other service follows the same categories and steps.

Where sanitized repos live

Sanitized repos land in gitlab.cl1.sq4.aegis.internal/modgpt/<repo-name> — the GitLab instance we own and operate. This single GitLab serves two roles simultaneously:

Role	Who uses it	How
Cluster-facing	ArgoCD (automated)	Reads `helm/` chart + `values-ai71-uaen-prod-modgpt.yaml` to deploy workloads to the cluster
Customer-facing	The customer / ops team (human)	Browses, reviews, and updates deployment config via the GitLab UI or git

Each application gets its own repo + its own ApplicationSet (not bundled into cluster-addons which is for platform infrastructure). ArgoCD naming follows the traceable pattern: modgpt.<app>.<service> (e.g. modgpt.ask.core-service).

Dev repo (Azure DevOps / GitHub)       Our GitLab (gitlab.cl1.sq4.aegis.internal)
─────────────────────────────────      ──────────────────────────────────────
ask-core-service-development/          modgpt/ask-core-service
  src/          ✗ stripped               helm/
  tests/        ✗ stripped                 Chart.yaml
  security/     ✗ stripped                 values.yaml
  scripts/…ado  ✗ stripped                 templates/
  helm/         ✅ kept                    environments/
  .cruft.json   ✗ stripped                   values-ai71-uaen-prod-modgpt.yaml  ← environment-specific
  …                                      README.md
                                         CHANGELOG.md

Never push a dev repo directly — always sanitize first

Dev repos contain cloud-provider endpoints, internal author emails, ADO/Azure DevOps SSH keys in template configs, Sonar org keys, ACR image refs, and production environment files. Even @format {env[...]} refs are safe in isolation, but the surrounding config (host names, org IDs, domain names) reveals production infrastructure topology. Strip it before it touches our GitLab.

The principle: push only what the cluster needs to run

The cluster needs one thing from each app repo: the Helm chart (or manifests) to deploy it. Everything else — Python source, tests, CI pipelines, dev tooling, application documentation, ADO scripts — is irrelevant to the cluster and should not be in the cluster-facing repo.

DEV REPO (full)                     AIR-GAP REPO (sanitized)
────────────────────────────────    ────────────────────────────────
src/            (Python source)  ✗
tests/          (test suite)     ✗
alembic/        (DB migrations)  ✗  ← migrations run inside the app
scripts/        (ADO/dev ops)    ✗
docs/ + site/   (MkDocs)         ✗
security/       (CI pipelines)   ✗
.claude/        (AI tooling)     ✗
.cruft.json     (ADO template)   ✗
sonar-project.properties         ✗
pyproject.toml / uv.lock         ✗
config/*.toml   (app config)     ✗  ← baked into the image already

helm/           (chart)          ✅ ← the only thing the cluster needs
helm/environments/values-<env>   ✅  but strip prod env files, keep non-production
Dockerfile*     (image builds)   ✅  if we build images in the air-gap environment; otherwise ✗
README.md                        ✅  useful context
CHANGELOG.md                     ✅  useful context
.gitignore                       ✅

What we remove and why

Category 1 — Application source code (`src/`, `tests/`, `alembic/`)¶

Path	Why removed
`src/`	The compiled/containerised image is what runs; Kubernetes never touches source. Pushing it exposes business logic unnecessarily.
`tests/`	Test infrastructure has no meaning in a cluster repo. Adds confusion and misleads about what "should work" in-cluster.
`alembic/` + `alembic.ini`	The Dockerfile copies these into the image (`COPY alembic/ ./alembic/`) so they are available inside the container. The `CMD` does not run migrations automatically — they are triggered separately via `scripts/apply_migrations.sh` or a Kubernetes Job/init-container. Either way, the cluster repo never needs the raw migration files: they're in the image.

Category 2 — Dev tooling and CI metadata¶

Path	Why removed
`.pre-commit-config.yaml`	Pre-commit hooks are a developer local tool. No meaning in a cluster repo.
`PRE_COMMIT_SETUP.md`	Paired with pre-commit config — irrelevant.
`.coveragerc`	Python test coverage config — irrelevant.
`.tool-versions`	`asdf` version manager pins for local dev — irrelevant.
`pyproject.toml` + `uv.lock`	Python package metadata and dependency lock — the image is already built; nobody re-installs packages from the cluster repo.
`requirements-docs.txt`	MkDocs build requirements — irrelevant.
`sonar-project.properties`	Contains `sonar.organization=ai71`, internal Sonar project key, and the ADO branch name — exposes internal org structure.
`mkdocs.yml` + `docs/` + `site/`	The app's own developer documentation. Exposing it is harmless but irrelevant; `site/` contains built HTML (~10 MB of noise).
`templates/`	Commitizen changelog Jinja templates — dev tooling.
`.cruft.json`	High risk — contains the ADO SSH URL (`git@ssh.dev.azure.com:v3/VentureOne/AI71/ai71_python_template`), internal author name and email, and the Azure DevOps project structure. Remove unconditionally.

Category 3 — CI/CD and security pipelines¶

Path	Why removed
`security/`	Contains Azure DevOps pipeline YAML (`autoupdate-pipeline.yml`, `security-pipeline.yml`, `gitleaks-pipeline.yml`, `zap-framework.yml`). Exposes internal CI/CD structure; this environment doesn't use ADO.
`scripts/download_env_from_ado.sh`	Downloads secrets from Azure DevOps — leaks ADO org name, project path, variable group names.
`scripts/upload_env_to_ado.sh`	Same — writes secrets back to ADO.
`scripts/fetch_sample_dump.py`	Fetches a prod DB dump — exposes prod DB access pattern.
`scripts/publish_gitlab_release.py`	Publishes to the upstream GitLab.com org (`ai71`) — leaks internal release process.
`scripts/release_config_changes.py`	Same concern.
`scripts/run_integration_tests.sh`	References internal service endpoints.
`scripts/test_bulk_ops_e2e.sh`	Same.

Scripts we CAN keep

scripts/apply_migrations.sh, scripts/init_db.sh, scripts/make_migrations.sh, scripts/verify_migrations.py, scripts/validate_env.py — these are cluster-relevant DB bootstrap helpers. Keep them if the deployment uses them; otherwise they're still harmless.

Category 4 — Internal tooling metadata¶

Path	Why removed
`.claude/` + `CLAUDE.md` (repo root)	AI assistant configuration and internal notes about the codebase. Not for customer/air-gap.
`helm/CLAUDE.md`	Same — remove from inside the chart too.
`.env.example`	Key names are safe (they're `@format {env[...]}` references), but the surrounding comments reveal secret categories. Replace with a clean environment-specific version or omit.

Category 5 — Helm chart: production environment files¶

The helm/environments/ directory contains one values file per production deployment:

File	Why removed
`values-ai71-uaen-prod-mufti.yaml`	Azure Postgres host, Azure ACR image refs, Azure Redis host (`redis.azure.net`), Azure storage account name, real external domains (`sso.ask.mufti.ai71services.ai`), Azure ingress class (`webapprouting.kubernetes.azure.com`), real org/user IDs in `seedData`.
`values-ai71-uaen-stg-ask71.yaml`	Azure Postgres host, staging domain names, Azure CDN URL, Azure storage account.
`values-ai71-uaen-dev-ask71.yaml`	Dev environment Azure endpoints.
`values-ai71-mec1-prod-mufti.yaml`	MEC region production endpoints.
`values-ai71-mec1-stg-ask71.yaml`	MEC region staging endpoints.
`values-ai71-pp-auh1-prod-ask.yaml`	Abu Dhabi production endpoints.

All of these are removed. We create a new values-ai71-uaen-prod-modgpt.yaml that references our CNPG Postgres, Harbor registry, internal domains, and Vault-sourced secrets.

Category 6 — Helm `values.yaml`: cloud-specific defaults to override (not remove)¶

values.yaml stays (it's the chart's own defaults), but the air-gap values file overrides these Azure-specific defaults:

Default value	Issue	Air-gap override
`image.repository: ai71uaenprodask71acr01.azurecr.io/core-service`	Unreachable ACR	`harbor.cl1.sq4.aegis.internal/ask/core-service`
`service.type: LoadBalancer`	No cloud LB	`ClusterIP`
`nodeSelector: node.kubernetes.io/role: ask71`	Label doesn't exist on the nodes → pod Pending	`{}` (clear it)
`tolerations: [{key: node.kubernetes.io/role, value: ask71}]`	Paired with above	`[]` (clear it)
`storage.type: azure` + Azure account refs	No Azure	stub/Ceph S3
Worker image repos (ACR)	Unreachable	Harbor registry

Step-by-step process (per repo)

Step 0 — Understand the repo¶

Before touching anything, map it: what does it deploy? What's in helm/ or equivalent? Which env files exist? Run:

find . -maxdepth 2 -not -path "./.git/*" | sort        # full structure
ls helm/environments/ 2>/dev/null                       # env files
grep -rn "azure\|acr\|ado\|ssh.dev.azure" helm/ 2>/dev/null | grep -v ".git"   # cloud refs

Output feeds the category checklist below.

Step 1 — Create the sanitized repo in GitLab¶

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
GLT=$(vault kv get -field=token secret/gitlab/migration-token)
# get the modgpt group ID
GID=$(curl -sk -H "PRIVATE-TOKEN: $GLT" https://gitlab.cl1.sq4.aegis.internal/api/v4/groups/modgpt \
  | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])")
# create (idempotent)
curl -sk --request POST -H "PRIVATE-TOKEN: $GLT" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects" \
  --form "name=<repo-name>" \
  --form "namespace_id=$GID" \
  --form "visibility=private" | python3 -c "import sys,json;d=json.load(sys.stdin);print('id:',d.get('id','error:',d))"

Step 2 — Create a sanitized working copy (never modify the dev repo)¶

cd /tmp
cp -r /path/to/dev-repo/<repo-name> <repo-name>-airgap
cd <repo-name>-airgap

Always work from a copy. The dev repo is the source of truth; changes go back there via the normal dev process.

Step 3 — Remove irrelevant/sensitive content¶

Apply the category table above. For ask-core-service:

# Category 1 — source code
rm -rf src/ tests/ alembic/ alembic.ini

# Category 2 — dev tooling
rm -f .pre-commit-config.yaml PRE_COMMIT_SETUP.md .coveragerc .tool-versions
rm -f pyproject.toml uv.lock requirements-docs.txt sonar-project.properties
rm -rf docs/ site/ templates/ mkdocs.yml .env.example

# Category 3 — CI/CD pipelines and ADO scripts
rm -rf security/
rm -f scripts/download_env_from_ado.sh scripts/upload_env_to_ado.sh \
       scripts/fetch_sample_dump.py scripts/publish_gitlab_release.py \
       scripts/release_config_changes.py scripts/run_integration_tests.sh \
       scripts/test_bulk_ops_e2e.sh

# Category 4 — internal tooling metadata
rm -rf .claude/
rm -f CLAUDE.md helm/CLAUDE.md .cruft.json

# Category 5 — production environment files (ALL of them)
rm -f helm/environments/values-ai71-*.yaml

# Category 6 — Dockerfiles (only needed if building images in the air-gap environment; omit if mirroring)
# rm -f Dockerfile Dockerfile.* config/     ← decide based on whether you build or mirror

Verify before committing

After removal, run a quick scan for leaked internal refs:

grep -rn "azure.com\|ssh.dev.azure\|ai71\.ai\|azurecr\.io\|sonar\.organization\|ai71_python_template" . \
  --include="*.yaml" --include="*.toml" --include="*.json" --include="*.py" --include="*.sh" \
  2>/dev/null | grep -v ".git/"

Zero results expected. Any hit = investigate before pushing.

Step 4 — Create the air-gap values file¶

Replace all production env files with a single helm/environments/values-ai71-uaen-prod-modgpt.yaml. Use the production-mufti file as the structural template (it has the most complete zitadel + postgresInit config), but substitute every external value:

# helm/environments/values-ai71-uaen-prod-modgpt.yaml
# Production environment overrides — Ceph/CNPG, Harbor registry, internal domains
# Secrets are NEVER in this file — they come from Vault via VSO

extraEnv:
  - name: APP__ENV
    value: "prod"

# --- Image overrides (ACR → Harbor registry) ---
image:
  repository: harbor.cl1.sq4.aegis.internal/ask/core-service
  tag: "<tag>"     # pin; never use latest in GitOps
  pullPolicy: IfNotPresent

imagePullSecrets:
  - name: harbor-pull-secret

# --- Remove cloud scheduling constraints ---
nodeSelector: {}
tolerations: []

# --- Service: ClusterIP (no cloud LB) ---
service:
  type: ClusterIP
  port: 80

# --- Ingress ---
ingress:
  enabled: true
  className: rke2-nginx
  annotations:
    cert-manager.io/cluster-issuer: internal-ca-issuer
  hosts:
    - host: core-service.ask.mod.auh1.dev.dir
      paths: [{ path: /, pathType: Prefix }]
  tls:
    - secretName: core-service-tls
      hosts: [core-service.ask.mod.auh1.dev.dir]

# --- Workers: off for v1 ---
bulkOperationsWorker:  { enabled: false }
billingMeteringWorker: { enabled: false }
cronSchedulerWorker:   { enabled: false }
jobRunWorker:          { enabled: false }

# --- Storage: stub (Ceph S3 wired in v2) ---
envConf:
  configToml: |
    [storage]
    type = "stub"

# --- Postgres DB init (runs as PreSync ArgoCD hook) ---
postgresInit:
  enabled: true
  namespace: ask
  connection:
    host: ask-pg-rw.ask.svc.cluster.local    # CNPG service
    port: "5432"
    adminUsername: postgres
  db:
    name: core-service
    user: core-service
  secrets:
    adminPasswordSecretName: zitadel-psql-secret   # shared PG admin — created by VSO
    adminPasswordKey: db-admin-password
    userPasswordSecretName: core-service-secret
    userPasswordKey: DATABASE__PASSWORD
  job:
    image: harbor.cl1.sq4.aegis.internal/dockerhub/postgres
    tag: "16-alpine"
    backoffLimit: 4
    ttlSecondsAfterFinished: 300
    pgSslMode: disable           # CNPG internal; TLS configured at CNPG level
  argocd:
    enabled: true
    connectionConfigMapAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "-2"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    jobAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "-1"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

# --- Zitadel (enabled; vendored in Chart.yaml) ---
zitadel:
  enabled: true
  zitadel:
    replicaCount: 1       # v1: 1 replica
    fullnameOverride: ask-core-service-zitadel
    tolerations: []
    nodeSelector: {}
    service: { type: ClusterIP, port: 8080 }
    ingress:
      enabled: true
      className: rke2-nginx
      annotations:
        cert-manager.io/cluster-issuer: internal-ca-issuer
        nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
      hosts:
        - host: sso.ask.mod.auh1.dev.dir
          paths: [{ path: /, pathType: Prefix }]
      tls:
        - secretName: zitadel-tls
          hosts: [sso.ask.mod.auh1.dev.dir]
    zitadel:
      masterkeySecretName: zitadel-secret-kv
      configmapConfig:
        ExternalSecure: true
        ExternalDomain: sso.ask.mod.auh1.dev.dir
        TLS: { Enabled: false }
        Database:
          Postgres:
            Host: ask-pg-rw.ask.svc.cluster.local
            Port: 5432
            Database: zitadel
            User:   { Username: zitadel,   SSL: { Mode: disable } }
            Admin:  { Username: postgres,  SSL: { Mode: disable } }
    env:
      - name: ZITADEL_DATABASE_POSTGRES_USER_PASSWORD
        valueFrom: { secretKeyRef: { name: zitadel-secret-kv, key: DATABASE__PASSWORD } }
      - name: ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD
        valueFrom: { secretKeyRef: { name: zitadel-psql-secret, key: db-admin-password } }
      - name: ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORDCHANGEREQUIRED
        value: "false"
      - name: ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD
        valueFrom: { secretKeyRef: { name: zitadel-secret-kv, key: ADMIN_PASSWORD } }
    login:
      enabled: true
      replicaCount: 1
      fullnameOverride: ask-core-service-zitadel-login
      ingress:
        enabled: true
        className: rke2-nginx
        hosts:
          - host: sso.ask.mod.auh1.dev.dir
            paths: [{ path: /ui/v2/login, pathType: Prefix }]
  zitadelPostgresInit:
    enabled: true
    namespace: ask
    connection:
      host: ask-pg-rw.ask.svc.cluster.local
      port: "5432"
      adminUsername: postgres
    secrets:
      adminPasswordSecretName: zitadel-psql-secret
      adminPasswordKey: db-admin-password
      userPasswordSecretName: zitadel-secret-kv
      userPasswordKey: DATABASE__PASSWORD
    job:
      image: harbor.cl1.sq4.aegis.internal/dockerhub/postgres
      tag: "16-alpine"
      backoffLimit: 4
      ttlSecondsAfterFinished: 300
      pgSslMode: disable
    argocd:
      enabled: true
      connectionConfigMapAnnotations:
        argocd.argoproj.io/hook: PreSync
        argocd.argoproj.io/sync-wave: "-2"
        argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
      jobAnnotations:
        argocd.argoproj.io/hook: PreSync
        argocd.argoproj.io/sync-wave: "-1"
        argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

Step 5 — Init git and push¶

cd /tmp/<repo-name>-airgap
git init && git checkout -b main
git add .
git status          # review one more time before committing
git commit -m "feat: initial sanitized air-gap deployment repo"
GLT=$(vault kv get -field=token secret/gitlab/migration-token)
git remote add origin "https://root:${GLT}@gitlab.cl1.sq4.aegis.internal/modgpt/<repo-name>.git"
git -c http.sslVerify=false push -u origin main

Step 6 — Verify the push is clean¶

# confirm no production refs landed in GitLab
curl -sk -H "PRIVATE-TOKEN: $GLT" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/modgpt%2F<repo-name>/repository/tree?recursive=true" \
  | python3 -c "import sys,json;[print(f['path']) for f in json.load(sys.stdin)]"
# visually confirm: only helm/, README.md, CHANGELOG.md, .gitignore present

Per-repo checklist

Copy this for each repo. Check every box before git push.

PRE-PUSH SANITIZATION CHECKLIST — <repo-name>  date: ____

REMOVED:
[ ] src/              (application source code)
[ ] tests/            (test suite)
[ ] alembic/ + alembic.ini  (DB migrations)
[ ] pyproject.toml + uv.lock / requirements*.txt
[ ] .pre-commit-config.yaml + PRE_COMMIT_SETUP.md + .coveragerc + .tool-versions
[ ] mkdocs.yml + docs/ + site/ + templates/
[ ] sonar-project.properties
[ ] .cruft.json
[ ] security/ (CI pipeline YAML)
[ ] scripts/download_env_from_ado.sh + upload_env_to_ado.sh
[ ] scripts/fetch_sample_dump.py + publish_gitlab_release.py + release_config_changes.py
[ ] scripts/run_integration_tests.sh + test_bulk_ops_e2e.sh
[ ] .claude/ + CLAUDE.md (root) + helm/CLAUDE.md
[ ] .env.example
[ ] helm/environments/values-ai71-*.yaml  (ALL production env files)

ADDED:
[ ] helm/environments/values-ai71-uaen-prod-modgpt.yaml  (air-gap override values, no secrets)

VERIFIED CLEAN (grep scan output = zero hits):
[ ] grep -rn "azure.com\|azurecr\.io\|ssh.dev.azure\|ai71\.ai\|sonar\.organization\|ai71_python_template" .

DECISIONS RECORDED:
[ ] Dockerfiles: included ☐  excluded ☐  reason: _______________
[ ] config/*.toml: included ☐  excluded ☐  reason: _______________
[ ] scripts remaining: ______________________________

Keeping the air-gap repo in sync with upstream changes

When the dev team ships a new version:

Re-run Step 2 (fresh copy of the updated dev repo).
Re-apply Step 3 removals.
Do not overwrite helm/environments/values-ai71-uaen-prod-modgpt.yaml — it's non-production. Compare the updated helm/values.yaml for any new keys that need air-gap overrides.
Bump the image tag in values-ai71-uaen-prod-modgpt.yaml to the new version.
Commit, push. ArgoCD picks it up within ~3 min.

Repo inventory

Repo	Status	Air-gap project	Notes
`ask-core-service`	🔄 In progress	`modgpt/ask-core-service`	Worked example above
(repo 2)	⬜ Pending
(repo 3)	⬜ Pending
(repo 4)	⬜ Pending
(repo 5)	⬜ Pending
(repo 6)	⬜ Pending

Migration toggle in `values-ai71-uaen-prod-modgpt.yaml`¶

The migration Job is off by default in values.yaml (safe for local dev / first install before a DB exists). Enable it in values-ai71-uaen-prod-modgpt.yaml so every ArgoCD sync automatically runs alembic upgrade head as a PreSync hook at sync-wave 0 — after the DB init Jobs (wave −1) and before any workload starts:

# helm/environments/values-ai71-uaen-prod-modgpt.yaml (append to the file)

# ─── Migrations ─────────────────────────────────────────────────────────────
# enabled: true  → PreSync Job runs alembic upgrade head on every sync
# enabled: false → no Job; use only for emergency rollbacks (manual step)
migrations:
  enabled: true

Wave ordering enforced by the chart:

wave -2  →  zitadelPostgresInit ConfigMaps / postgresInit ConfigMaps
wave -1  →  zitadelPostgresInit Job + postgresInit Job  (DB + roles created)
wave  0  →  migrations Job  (alembic upgrade head — schema applied)
wave  5  →  zitadel Deployment, core-service Deployment, …  (workloads start)

Disable the toggle (enabled: false) only during an emergency rollback where you want to redeploy a previous image without re-running migrations. In all other cases leave it true.