Mirror Images & Charts (from the Bastion)¶

Air-gap mirroring runs from the bastion, never the Mac. The bastion can reach the upstream registries through the cluster forward proxy (10.96.137.37:3128) and the in-cluster Harbor / GitLab; the Mac's VPN only exposes the internal 100.115.x network, so it can reach Harbor/GitLab but not the proxy or the public internet. On the Mac you only edit the manifests; the bastion runs the mirror.

Two tools, two destinations

What	Tool	Manifest	Destination
Container images	`core42-mirror-image.py`	`mirror-config.yaml`	Harbor projects (`ask`, `dockerhub`, `ghcr`, `quay`, `gitlab`, `bitnamilegacy`)
Helm charts → Harbor	`helm-charts/core42-mirror-charts.py`	`helm-charts/mirror-charts.yaml`	Harbor OCI `helm-charts` project
Helm charts → GitLab	`helm-charts/gitlab-push-charts.py`	`helm-charts/gitlab-charts.yaml`	GitLab Helm registry (what ArgoCD pulls)

flowchart LR
    UP["upstream\nACR / Docker Hub / ghcr / quay / Helm repos"]
    UP -->|"proxy 10.96.137.37:3128"| B["bastion\n~/harbor-images"]
    B -->|core42-mirror-image.py| H["Harbor\n(images)"]
    B -->|core42-mirror-charts.py| HO["Harbor OCI\nhelm-charts"]
    HO -->|gitlab-push-charts.py| GL["GitLab Helm registry\n(ArgoCD pulls)"]

Prerequisites (on the bastion, `~/harbor-images/`)¶

cd ~/harbor-images
command -v skopeo helm jq
ls core42-mirror-image.py mirror-config.yaml harbor-robot.json acr-creds.json ghcr-creds.json
ls helm-charts/core42-mirror-charts.py helm-charts/mirror-charts.yaml helm-charts/gitlab-push-charts.py helm-charts/gitlab-charts.yaml

- Harbor creds — harbor-robot.json (robot robot$ask); both manifests read it via auth_file. - ACR / ghcr creds — acr-creds.json / ghcr-creds.json (image manifest only). - Proxy — ON by default in both manifests (features/proxy); reaches ACR + public repos. - GitLab push creds — GITLAB_USER/GITLAB_TOKEN env, or auto-fetched from Vault secret/gitlab/chart-push-sa (needs vault login + VAULT_ADDR).

Mac edits, bastion runs — copy the manifest over

Edit the manifests (mirror-config.yaml / mirror-charts.yaml / gitlab-charts.yaml) on the Mac, then copy them to the bastion (they are not in GitLab) so its ~/harbor-images/ copy matches before running. A stale bastion manifest is a common "why didn't my new entry mirror?" cause.

1. Mirror container images → Harbor¶

Add/verify the image entry in mirror-config.yaml (no sha256:<digest> placeholder — the script skips those):

- { name: ask-knowledge-management-service, source: acr, project: ask, tags: ["staging"] }

Run on the bastion (proxy auto-ON):

cd ~/harbor-images
./core42-mirror-image.py --dry-run                 # always preview first
./core42-mirror-image.py --only knowledge          # mirror one image (substring filter)
./core42-mirror-image.py                            # mirror everything (skip_existing=true)

- --only <substring> filters by image name. [PUSHED] = mirrored, [SKIP] … already in Harbor = present. - Harbor path must match the values file exactly (incl. nested repos like ask/semitechnologies/weaviate). - Two-phase (internet host pulls, air-gap host pushes): --download-only -v then --push-only -v.

Fully air-gapped — no ACR pull from the bastion

The default flow pulls from ACR through the proxy. When the environment is fully air-gapped, the bastion has no upstream access: images are manually copied in and staged into the mirror cache (/opt/data/ask/…), then pushed to Harbor with --push-only. The proxy step is skipped entirely — mirror-config.yaml is unchanged; only the source of the image bytes differs.

Detail + Harbor project layout: 01 — Image Mirroring.

On the Mac the mirror FAILS by design

proxyconnect … i/o timeout (proxy unreachable) or lookup …azurecr.io … i/o timeout (no public DNS) — both mean you're on the Mac. Run on the bastion. /opt/data cache also only exists there; use --no-cache if you must stream.

2. Mirror Helm charts → Harbor OCI¶

Add the chart to helm-charts/mirror-charts.yaml (name + version + upstream repo), then:

cd ~/harbor-images/helm-charts
./core42-mirror-charts.py --dry-run
./core42-mirror-charts.py --only vault              # one chart
./core42-mirror-charts.py                            # all charts (→ harbor.../helm-charts/<name>:<version>)

Charts land as OCI artifacts in Harbor's helm-charts project (skip_existing=true).

3. Push Helm charts → GitLab Helm registry (what ArgoCD pulls)¶

ArgoCD pulls cluster-addon charts from the GitLab HTTP Helm registry, so charts must go Harbor OCI → GitLab. Add the chart to helm-charts/gitlab-charts.yaml (name + version) and push:

cd ~/harbor-images/helm-charts
# creds: env or Vault (secret/gitlab/chart-push-sa)
export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
./gitlab-push-charts.py -c gitlab-charts.yaml --dry-run
./gitlab-push-charts.py -c gitlab-charts.yaml --only external-secrets
./gitlab-push-charts.py -c gitlab-charts.yaml        # push all → GitAir-gap project 14 /packages/helm/stable

- Source = Harbor OCI (Step 2 must have run first), destination = GitLab Helm registry (basic auth username:token, not PRIVATE-TOKEN). - --pull-only (Harbor→cache) / --push-only (cache→GitLab) for the two-phase split. - Verify in GitLab: project → Packages & registries → Helm, version present.

Full chart lifecycle

upstream → (core42-mirror-charts.py) → Harbor OCI → (gitlab-push-charts.py) → GitLab Helm registry → ArgoCD. A new cluster-addon chart needs both chart steps; an app image needs only Step 1.

Quick reference¶

Goal	Command (on bastion)
One image	`./core42-mirror-image.py --only <name>`
All images	`./core42-mirror-image.py`
One chart → Harbor	`helm-charts/core42-mirror-charts.py --only <name>`
One chart → GitLab	`helm-charts/gitlab-push-charts.py -c gitlab-charts.yaml --only <name>`
Preview anything	add `--dry-run`

Image Tagging Convention & Rollback¶

Every image entry in mirror-config.yaml must follow this convention so rollback never requires re-mirroring:

# Current build — tagged "staging" (what ArgoCD deploys) + date stamp
- { name: core-service, source: acr, project: ask, tags: ["staging", "2026-07-01"],
    digest: "sha256:a3e2aa25578bb279ac3dcec68d69a7406434e57d68cddc9a0da95744e41ec550" }

# Previous build — kept as "staging-prev" + date stamp (NEVER remove)
- { name: core-service, source: acr, project: ask, tags: ["staging-prev", "2026-06-17"],
    digest: "sha256:7240129df2425661ee000db9f02bd3a77eea492e7cc323a3c040a0fe49598549" }

Rules: - Never replace the previous entry — add a new one on top and demote the old staging to staging-prev - Always include a date tag alongside staging — it makes it trivial to know what's running and when it was built - Always pin by digest — git SHAs are NOT image digests; get the real digest with:

cd ~/harbor-images
skopeo inspect "docker://ai71uaenprodask71acr01.azurecr.io/<image>:staging" \
  --creds "modgpt-devops:$(python3 -c "import json; print(json.load(open('acr-creds.json'))['password'])")" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['Digest'])"

HTTP proxy required for ACR

Set proxy before running skopeo inspect against ACR:

export https_proxy=http://10.96.137.37:3128

Resolving a PP-Reported Commit to a Verified Digest¶

When someone (e.g. a teammate on Slack) reports "PP is stable on commit <sha>" for a service, that commit is not an image digest — it's the git tag the CI pipeline also pushed to ACR alongside staging. You still have to look up what digest that tag actually resolves to before pinning mirror-config.yaml. Real incident (2026-07-01): two earlier attempts to "fix" a digest by hand (core-service → a3e2aa25...) were simply wrong — not verified against ACR at all. Treat every digest as unverified until you've run skopeo inspect yourself.

1. Split the tag from the commit SHA. Slack often glues a version tag directly onto the git SHA with no separator, e.g. v3.1.0-rc7b8cf159d2ebf64f84f378007880b36680182a8d4. A git commit SHA is always 40 hex characters — count backward from the end to find the boundary:

v3.1.0-rc7 | b8cf159d2ebf64f84f378007880b36680182a8d4   ← 40 hex chars = the SHA

If the split is ambiguous (doesn't cleanly leave exactly 40 hex chars), ask for confirmation rather than guess — a wrong boundary means looking up the wrong tag entirely.

2. Confirm the tag exists in ACR (CI usually pushes both an 8-char short SHA and the full 40-char SHA as tags — skopeo inspect on the wrong-length guess just 404s with manifest unknown):

cd ~/harbor-images
CREDS="modgpt-devops:$(python3 -c "import json;print(json.load(open('acr-creds.json'))['password'])")"
export https_proxy=http://10.96.137.37:3128   # ACR is only reachable through the proxy

skopeo list-tags --creds="$CREDS" docker://ai71uaenprodask71acr01.azurecr.io/<image> \
  | grep -i <short-sha>

Note the source ref has no project prefix — it's {registry}/{name} (e.g. .../assistant-service, not .../ask/assistant-service). project: ask in mirror-config.yaml only applies to the Harbor destination path.

3. Get the real digest, reusing the same JSON-parsing one-liner as the tagging convention above (not a grep '"Digest"' — a multi-arch manifest list has one top-level digest plus a "Digest" line per architecture underneath it; naive grep returns all of them and it's easy to grab the wrong one):

skopeo inspect --creds="$CREDS" "docker://ai71uaenprodask71acr01.azurecr.io/<image>:<full-40-char-sha>" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['Digest'])"

4. Pin it following the Image Tagging Convention above — new entry with staging + a date tag, old digest demoted to staging-prev, comment naming the commit SHA for traceability.

Pinning the config doesn't deploy it

Updating mirror-config.yaml only changes what the next mirror run pushes to Harbor. You still need to run core42-mirror-image.py, then kubectl rollout restart the affected deployment — staging is a floating tag, so imagePullPolicy: IfNotPresent will keep serving the old cached image on each node until forced. Verify with:

kubectl -n ask get pods -l app.kubernetes.io/instance=<service> \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].imageID}{"\n"}{end}'

Rollback Procedure¶

Rollback = point ArgoCD back to the previous image tag. No re-mirroring needed.

Step 1 — Update the values overlay (applications/<service>/values-mod-auh1-dev-ask.yaml):

# Change from:
image:
  tag: "staging"

# To (rollback to previous):
image:
  tag: "staging-prev"
# OR by date:
  tag: "2026-06-17"

Step 2 — Commit and push:

cd ~/modgpt-sc/mod-ask-devops
git add applications/<service>/values-mod-auh1-dev-ask.yaml
git commit -m "rollback(<service>): revert to staging-prev (2026-06-17 build)"
git push

Step 3 — Sync in ArgoCD:

argocd app sync mod-auh1-dev-ask.ask.<service> --grpc-web

Step 4 — Verify pods are running the old image:

kubectl get pods -n ask -l app.kubernetes.io/name=<service> \
  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}'

Service	ArgoCD App	Values file
core-service	`mod-auh1-dev-ask.ask.core-service`	`applications/core-service/values-mod-auh1-dev-ask.yaml`
KMS	`mod-auh1-dev-ask.ask.knowledge-management-service`	`applications/knowledge-management-service/values-mod-auh1-dev-ask.yaml`
assistant-service	`mod-auh1-dev-ask.ask.assistant-service`	`applications/assistant-service/values-mod-auh1-dev-ask.yaml`