Phase 8 — Stakater Reloader¶

Stakater Reloader watches ConfigMaps and Secrets and triggers rolling restarts of Deployments, StatefulSets and DaemonSets when their referenced config changes.

Required for: Any workload that mounts Vault secrets via ESO — when ESO refreshes a Secret, Reloader ensures the pod picks up the new value without manual restart.

Overview¶

Item	Value
Chart	`reloader` `2.2.12`
App version	`v1.4.17`
Image	`harbor.cl1.sq4.aegis.internal/ghcr/stakater/reloader:v1.4.17`
Namespace	`reloader`
Source chart	`https://stakater.github.io/stakater-charts`
Harbor project	`ghcr`

Step 8.1 — Mirror Image to Harbor¶

Run from your Mac in the harbor-mirror directory:

cd /Users/tarun.mittal/Documents/096_RKE/scripts/harbor-mirror

# Mirror reloader image
HARBOR_ROBOT_PASS=$(cat harbor-robot.json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d.get('password',''))") \
python3 core42-mirror-image.py --image ghcr.io/stakater/reloader:v1.4.17

Verify in Harbor:

curl -sk -u "robot\$ask:<password>" \
  "https://harbor.cl1.sq4.aegis.internal/api/v2.0/projects/ghcr/repositories" \
  | python3 -m json.tool | grep reloader

Step 8.2 — Mirror Helm Chart to Harbor (if not already done)¶

Charts are mirrored to Harbor by core42-mirror-charts.py — reloader is already in the manifest. Run from your Mac:

cd /Users/tarun.mittal/Documents/096_RKE/scripts/harbor-mirror/helm-charts

python3 core42-mirror-charts.py --only reloader

Verify chart in Harbor:

helm show chart oci://harbor.cl1.sq4.aegis.internal/helm-charts/reloader \
  --version 2.2.12 --insecure-skip-tls-verify

Step 8.2b — Push Helm Chart to GitLab HTTP Registry¶

Why GitLab HTTP instead of Harbor OCI?

ArgoCD v3.4.2 has a credential-matching bug with multi-level OCI paths (harbor.reg/helm-charts/reloader). The ArgoCD OCI Helm repository URL must be hostname-only, but credential lookup against the Application's full oci:// path fails. GitLab's built-in HTTP Helm package registry works reliably with ArgoCD's existing credential template — no new auth configuration needed.

Production note: Harbor OCI is the correct long-term approach for an air-gapped registry. This GitLab HTTP workaround should be revisited when upgrading ArgoCD beyond v3.4.x.

Step 8.2b.1 — Create a GitAir-gap project for Helm charts¶

In the GitLab UI at https://gitlab.cl1.sq4.aegis.internal:

New project → modgpt / helm-charts (or use an existing project)
Note the project ID (shown in the project settings page header)

Or via API:

GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)

curl -sk --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects?search=helm-charts" \
  | python3 -m json.tool | grep -E '"id"|"path_with_namespace"'

Step 8.2b.2 — Push charts using the mirror script¶

From your Mac:

cd /Users/tarun.mittal/Documents/096_RKE/scripts/harbor-mirror/helm-charts

export GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)

# Edit gitlab-charts.yaml and replace REPLACE_WITH_PROJECT_ID with the real project ID
vi gitlab-charts.yaml

# Dry-run first
python3 gitlab-push-charts.py -c gitlab-charts.yaml --dry-run

# Push reloader
python3 gitlab-push-charts.py -c gitlab-charts.yaml --only reloader

Step 8.2b.3 — Verify chart is in GitLab¶

Check via the GitLab UI: modgpt/helm-charts → Deploy → Package Registry → Helm

Or via API:

curl -sk --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<id>/packages?package_type=helm" \
  | python3 -m json.tool | grep -E '"name"|"version"'

Expected:

"name": "reloader",
"version": "2.2.12"

Step 8.2b.4 — Add GitLab Helm repo to ArgoCD¶

The ArgoCD credential template already covers https://gitlab.cl1.sq4.aegis.internal/ via the argocd-ro service account. Add the Helm registry as a named repo entry:

ARGOCD_TOKEN=$(argocd account generate-token --account argocd-ro --grpc-web 2>/dev/null || \
  vault kv get -field=token secret/gitlab/argocd-ro-token 2>/dev/null)

argocd repo add \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<id>/packages/helm/stable" \
  --type helm \
  --name gitlab-helm-charts \
  --username argocd-ro \
  --password "${ARGOCD_TOKEN}" \
  --grpc-web

argocd repo list --grpc-web | grep gitlab-helm

Expected: STATUS: Successful

Step 8.3 — Create imagePullSecret¶

kubectl create namespace reloader --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret docker-registry harbor-pull-secret \
  -n reloader \
  --docker-server=harbor.cl1.sq4.aegis.internal \
  --docker-username="robot\$ask" \
  --docker-password="<harbor-robot-password>" \
  --dry-run=client -o yaml | kubectl apply -f -

Step 8.4 — Create ArgoCD Application¶

The GitLab HTTP Helm registry URL format:

https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<id>/packages/helm/stable

The reloader.yaml in the mod-ask-devops repo at environment/auh1/dev/02-argocd-application/cluster-addons/reloader.yaml should be:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: reloader
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.io
spec:
  project: mod-auh1-dev-ask.cluster-addons
  destination:
    server: https://kubernetes.default.svc
    namespace: reloader
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
  sources:
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<id>/packages/helm/stable'
      chart: reloader
      targetRevision: "2.2.12"
      helm:
        valueFiles:
          - $values/applications/cluster-addons/reloader/values-mod-auh1-dev-ask.yaml
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/ai71/ask71/mod-ask-devops.git'
      targetRevision: main
      ref: values

Commit and push to mod-ask-devops — ArgoCD will pick it up via selfHeal.

Step 8.5 — Verify¶

# Check ArgoCD app
kubectl get application reloader -n argocd

# Check pods
kubectl get pods -n reloader

# Check logs
kubectl logs -n reloader -l app=reloader --tail=20

Expected:

NAME                        READY   STATUS    RESTARTS   AGE
reloader-<hash>             1/1     Running   0          1m

Logs should show:

Starting Reloader
Reloader started watching for changes...

How Reloader Works with ESO¶

Once Reloader is running, annotate any Deployment that mounts ESO secrets:

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

Or specify the exact secret to watch:

metadata:
  annotations:
    secret.reloader.stakater.com/reload: "my-secret-name"

When ESO refreshes the secret, Reloader triggers a rolling restart automatically. ✅

Helm Values Structure¶

Reloader chart value paths are non-standard

The Stakater Reloader chart splits values across two levels. Getting this wrong causes silently ignored overrides — ArgoCD syncs green but chart defaults are used.

Field	Correct path	Wrong path (silently ignored)
Image repository	`image.repository` (root)	`reloader.image.repository`
Image tag	`image.tag` (root)	`reloader.image.tag`
Image pull secrets	`global.imagePullSecrets` (root)	`reloader.imagePullSecrets`
Pod security context	`reloader.deployment.securityContext`	`reloader.securityContext`
Container security context	`reloader.deployment.containerSecurityContext`	`reloader.containerSecurityContext`

Correct values file (values-mod-auh1-dev-ask.yaml):

image:
  repository: harbor.cl1.sq4.aegis.internal/ghcr/stakater/reloader
  tag: v1.4.17

global:
  imagePullSecrets:
    - name: harbor-pull-secret

reloader:
  ignoreSecrets: false
  ignoreConfigMaps: false
  logLevel: info
  deployment:
    securityContext:
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

Troubleshooting¶

Symptom	Cause	Fix
`ImagePullBackOff`	`harbor-pull-secret` missing or wrong password	Re-create secret — Step 8.3
`ErrImagePull`	Image not in Harbor	Re-run Step 8.1
Pod not restarting on secret change	Missing annotation	Add `reloader.stakater.com/auto: "true"` to Deployment
ArgoCD sync failed — OCI chart not found	Chart not pushed to Harbor	Re-run Step 8.2
ArgoCD `invalid reference: missing registry or repository`	ArgoCD v3.4.x OCI credential bug	Use GitLab HTTP Helm registry (Step 8.2b) instead of Harbor OCI
ArgoCD `401 Unauthorized` on Helm chart pull	`argocd-ro` token not set for the GitLab Helm repo	Re-run Step 8.2b.4 with correct PAT
Image stays `ghcr.io` despite values override	`image.*` nested under `reloader:` instead of root level	Move `image.repository`, `image.tag`, `global.imagePullSecrets` to root level — see table above
Pod fails `restricted` PodSecurity	`containerSecurityContext` at wrong path	Use `reloader.deployment.containerSecurityContext` not `reloader.containerSecurityContext`
ArgoCD 308 redirect error on git fetch	Application `repoURL` uses `http://` — GitLab redirects to HTTPS	Update all Application manifests to `https://` and re-apply
`$values` ref not resolving (chart uses defaults)	ArgoCD credential template is HTTP-only but Application uses HTTPS	Add HTTPS credential template: `argocd repocreds add https://gitlab.../`
ArgoCD credential template not matching	PAT missing `read_repository` scope — git clone fails silently	Recreate PAT with both `read_api` and `read_repository` scopes