Skip to content

36 — ArgoCD Internal CA Trust (ESO → Pod Trust Store)

Problem

The internal root CA certificate (O=ai71, CN=modgpt) was embedded inline in the ArgoCD Helm values file to allow ArgoCD to trust GitLab's TLS certificate when cloning repos.

# BAD — cert PEM in values.yaml
configs:
  tls:
    certificates:
      gitlab.cl1.sq4.aegis.internal: |
        -----BEGIN CERTIFICATE-----
        ...

Problems with this approach:

Concern Impact
Cert rotation Must find + update the YAML, commit, and sync — no automation
Wrong concern Values files configure app behaviour, not trust anchors
Drift risk YAML cert may not match the actual CA in use
Hidden PEM buried inside a Helm values file is easy to miss

Solution — ESO → Pod Trust Store via SSL_CERT_DIR

Vault KV                     ESO                      ArgoCD Pods
secret/platform/internal-ca  ──►  K8s Secret          extraVolume: Secret mount
  cert: <PEM>                     argocd-ca-cert    ──►  at /tmp/internal-ca/ca.crt
                                  ca.crt: <PEM>         SSL_CERT_DIR env var
                                                         Reloader: restart on change
  1. CA cert stored in Vault — single source of truth, rotation-friendly
  2. ESO syncs it as K8s Secret argocd-ca-cert in the argocd namespace
  3. The Secret is mounted into argocd-server and argocd-repo-server at /tmp/internal-ca/
  4. SSL_CERT_DIR=/tmp/internal-ca:/etc/ssl/certs tells Go's TLS to look in BOTH our internal CA directory AND the system bundle — no initContainer, no bundle merging
  5. Reloader watches the Secret and restarts pods automatically when the cert rotates

Why SSL_CERT_DIR instead of SSL_CERT_FILE

SSL_CERT_FILE replaces the system CA bundle entirely — only our CA would be trusted. SSL_CERT_DIR with a colon-separated path ADDS our directory alongside the system bundle. Go reads all .crt/.pem files from each listed directory.

Day-0 Bootstrap (cluster rebuild)

On a fresh cluster before ESO is running, apply the bootstrap Secret once:

kubectl apply -f applications/cluster-addons/argocd/manifests/argocd-ca-cert-bootstrap.yaml

This creates argocd-ca-cert Secret in argocd namespace with the CA cert as base64. ArgoCD pods start with SSL_CERT_DIR trusting GitLab immediately. Once ESO is running, the ExternalSecret (creationPolicy: Merge) merges the Vault-sourced cert into the same Secret and takes over rotation. The bootstrap manifest is kept in Git as documentation and for cluster rebuilds — it contains only the public CA cert (no private key).

ignoreDifferences on argocd-tls-certs-cm

argocd.yaml Application has ignoreDifferences + RespectIgnoreDifferences=true for argocd-tls-certs-cm. Helm never overwrites this ConfigMap. The cert is kept there as a secondary fallback and managed manually (or can be removed once SSL_CERT_DIR is confirmed stable).

Step 36.1 — Store CA Cert in Vault

Extract the cert from the cluster and store it in Vault:

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
export VAULT_SKIP_VERIFY=true   # if bastion doesn't trust internal CA

# Read the CA cert from cert-manager (where internal-ca-issuer stores it)
CA_CERT=$(kubectl get secret -n cert-manager \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' \
  | grep -i "internal-ca\|cluster-ca\|ca-key-pair" | head -1)

kubectl get secret $CA_CERT -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d

# Store in Vault (paste the PEM output above as the value)
vault kv put secret/platform/internal-ca \
  cert="$(kubectl get secret $CA_CERT -n cert-manager \
    -o jsonpath='{.data.tls\.crt}' | base64 -d)"

Verify: 

vault kv get -field=cert secret/platform/internal-ca | openssl x509 -noout -subject -issuer
# Expected: subject=O=ai71, CN=modgpt  issuer=O=ai71, CN=modgpt

Step 36.2 — Deploy ESO ExternalSecret

The manifest at applications/cluster-addons/argocd/manifests/argocd-ca-cert-eso.yaml creates Secret argocd-ca-cert in the argocd namespace.

ArgoCD syncs this as part of the argocd Application (same wave as other manifests).

Step 36.3 — Update ArgoCD Values

Changes to applications/cluster-addons/argocd/values-mod-auh1-dev-ask.yaml:

  • Remove configs.tls.certificates (no more inline cert)
  • Add repoServer.extraVolumes + extraVolumeMounts + extraEnvVars + podAnnotations
  • Add server.extraVolumes + extraVolumeMounts + extraEnvVars + podAnnotations
repoServer:
  podAnnotations:
    reloader.stakater.com/auto: "true"
  extraVolumes:
    - name: internal-ca-cert
      secret:
        secretName: argocd-ca-cert
  extraVolumeMounts:
    - name: internal-ca-cert
      mountPath: /tmp/internal-ca
      readOnly: true
  extraEnvVars:
    - name: SSL_CERT_DIR
      value: /tmp/internal-ca:/etc/ssl/certs

No initContainers needed — SSL_CERT_DIR with colon-separated paths tells Go's TLS to scan both directories. The Secret file /tmp/internal-ca/ca.crt is picked up automatically.

Step 36.4 — Verify

# ESO synced the secret
kubectl get secret argocd-ca-cert -n argocd
kubectl get externalsecret argocd-ca-cert -n argocd

# ArgoCD pods restarted and are healthy
kubectl get pods -n argocd

# ArgoCD can still reach GitLab (no TLS error)
argocd app list   # all apps should show without errors
argocd repo list  # gitlab repo should show Connected

Cert Rotation Procedure

  1. Update Vault: vault kv put secret/platform/internal-ca cert="<new PEM>"
  2. Force ESO sync: kubectl annotate externalsecret argocd-ca-cert -n argocd force-sync=$(date +%s) --overwrite
  3. Reloader detects Secret change → restarts argocd-server + argocd-repo-server automatically
  4. No Git commit required — the cert content is no longer in Git

Why SSL_CERT_DIR (not initContainer bundle merge)

The Bitnami ArgoCD chart renders initContainers and extraVolumes in separate template passes — initContainers cannot reference volumes defined via extraVolumes (Kubernetes validates them as not found). SSL_CERT_DIR with colon-separated paths solves this without any initContainer: Go's TLS scans each listed directory for .crt/.pem files, adding them all to the trust pool alongside the system bundle.