KB-03 — GitOps Architecture Evolution & ApplicationSet Reference

Historical record of how the platform's GitOps architecture evolved, and reference material for ApplicationSet patterns that are no longer used in steady-state but may be needed when creating per-app ApplicationSets for future workloads.

---

Evolution Timeline

Phase 1 — Single shared AppProject + ApplicationSet (auto-discovery)
  AppProject: modgpt (shared by all)
  Application: modgpt.cluster-addons → ApplicationSet → child Applications

Phase 2 — AppProject per app (via ApplicationSet generating AppProject+App pairs)
  ApplicationSet generates: AppProject/modgpt.cluster-addons.<svc> + Application/modgpt.cluster-addons.<svc>
  Abandoned: ApplicationSet can't natively generate AppProject objects cleanly

Phase 3 — Pure App of Apps with per-app AppProjects (ai71 pattern)
  root Application → AppProject+Application pairs directly (no ApplicationSet)
  Each service: AppProject/modgpt.cluster-addons.<svc> + Application/modgpt.cluster-addons.<svc>
  Abandoned: too many files (8 for 4 services), per-service AppProjects add no RBAC value for single operator

Phase 4 — 3-Layer App of Apps (CURRENT)
  Layer 1: AppProject modgpt + Application modgpt (manual)
  Layer 2: AppProject modgpt.cluster-addons + Application modgpt.cluster-addons (GitOps)
  Layer 3: Application per service, no per-service AppProject (GitOps)

The current architecture is documented in 15 — ArgoCD GitOps Architecture.

---

Why Each Phase Was Superseded

Phase 1 → Phase 2: Single shared AppProject

The shared modgpt AppProject had no per-service RBAC boundary. Any app could theoretically deploy to any namespace allowed by the project. Moving to per-app AppProjects was the correct direction but the implementation via ApplicationSet was complex.

Phase 2 → Phase 3: ApplicationSet generating AppProjects

ArgoCD ApplicationSets generate Application objects only — AppProject generation required a workaround (deploying the AppProject as a managed resource of the Application, via a third source pointing to apps/<svc>/appproject.yaml). This worked but added template complexity and a confusing mental model.

Phase 3 → Phase 4: Per-service AppProjects

With 4 services, bootstrap/ had 8 files (4 AppProject + 4 Application). The Layer 3 Application spec already constrains destination.namespace; the Layer 2 AppProject already constrains source repos. Per-service AppProjects provided no additional RBAC boundary for a single operator.

---

When to Add Layer 3 AppProjects

The current architecture intentionally omits AppProjects at Layer 3. Add them back when:

Trigger	Example
Per-service team ownership	Vault team gets ArgoCD sync permission on modgpt.cluster-addons.vault project only — cannot touch vso or reloader
Per-service CI tokens	Vault CI pipeline authenticates with a token scoped to vault's AppProject — cannot deploy outside vault namespace
Compliance audit requirement	Evidence that vault workload is provably prevented at the RBAC level from deploying into reloader namespace — not just by convention

To add: create bootstrap/appproject-<svc>.yaml with sync-wave: "-1" alongside bootstrap/application-<svc>.yaml. The Application then references project: modgpt.cluster-addons.<svc> instead of project: modgpt.cluster-addons.

---

ApplicationSet Reference (Historical)

The ApplicationSet pattern is no longer used for cluster-addons but the knowledge is preserved here for future use cases where dynamic generation is genuinely needed (e.g. multi-cluster deployments, PR preview environments).

The git files generator

Scans a repo for files matching a glob and generates one Application per match:

generators:
  - git:
      repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/<app>.git
      revision: HEAD
      files:
        - path: "apps/*/app.yaml"   # one Application per matching file

Each matched file's contents become template variables via {{ .key }}.

Gotcha 1 — Dots in application names break Helm release names

ArgoCD uses the Application name as the Helm release name by default. Traceable names like modgpt.cluster-addons.vault contain dots — Kubernetes container names are RFC 1123 labels (no dots allowed) → SyncFailed: containers[0].name must not contain dots.

Fix: always set helm.releaseName explicitly in the templatePatch:

templatePatch: |
  spec:
    sources:
      - helm:
          releaseName: {{ .path.basename }}   # short name from directory, no dots

This also ensures the release name matches the original (pre-ApplicationSet) name, so ArgoCD adopts existing resources instead of creating duplicates.

Gotcha 2 — missingkey=error with optional template keys

# DON'T add this:
goTemplateOptions: ["missingkey=error"]

If any app.yaml omits an optional key (e.g. ignoreDifferences) and missingkey=error is set, the generator fails for all apps — producing zero Applications with no clear error. Guard optional keys with hasKey:

templatePatch: |
  {{- if hasKey . "ignoreDifferences" }}
  ignoreDifferences:
    {{- toYaml .ignoreDifferences | nindent 4 }}
  {{- end }}

Gotcha 3 — path vs dirPath collision

The git files generator exposes .path (the config file's path, e.g. apps/vault/app.yaml). A path: key inside app.yaml collides with this variable name. Use dirPath: for directory-type apps:

# apps/pdb/app.yaml
namespace: argocd
type: dir
dirPath: infra/pdb    # ← dirPath, not path

templatePatch — handling helm vs dir type

goTemplate can't conditionally change the YAML structure of sources (adding/removing keys changes the object shape, not just values). Use templatePatch for structural conditionals:

templatePatch: |
  spec:
    sources:
    {{- if eq .type "helm" }}
      - repoURL: <helm-registry>
        chart: {{ .chart }}
        targetRevision: "{{ .targetRevision }}"
        helm:
          releaseName: {{ .path.basename }}
          valueFiles: [ "$values/{{ .valuesPath }}" ]
      - repoURL: <git>
        targetRevision: HEAD
        ref: values
    {{- else if eq .type "dir" }}
      - repoURL: <git>
        targetRevision: HEAD
        path: {{ .dirPath }}
        directory: { recurse: true }
    {{- end }}

---

OCI Helm Registry — Why HTTP is Used Instead

ArgoCD v3.4.2 double-encodes the JWT authentication scope for multi-level OCI paths (e.g. oci://registry.gitlab.cl1.sq4.aegis.internal/modgpt/charts/<chart>). GitLab returns 403 because the encoded scope doesn't match the expected format.

The HTTP package registry (https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<id>/packages/helm/stable) does not have this issue and is used for all ArgoCD Helm chart pulls.

OCI remains valid for direct helm pull and helm push from the CLI. ArgoCD uses HTTP only.

See Issue 39 — ArgoCD OCI Helm 403 for Multi-Level Paths for the full diagnosis.