Update ArgoCD Config — Projects, Applications, App Groups¶

The most important thing to know before touching ArgoCD config: some objects are GitOps (push a file, it self-applies) and some are a manual bootstrap (you run a command once). Editing the wrong layer and waiting for it to "just sync" is the trap most people hit.

All commands run from the bastion (ssh cloud-user@bastion1), where kubectl targets the cluster and ~/cluster-addons is the GitLab repo clone.

The 3-Layer Mental Model

LAYER 1 — Manual Bootstrap (cluster only — kubectl apply)
  AppProject: modgpt            ← tenancy boundary
  Application: modgpt           ← root tile; syncs root-bootstrap/

        ↓ GitOps from here down ↓

LAYER 2 — App Groups (managed by modgpt Application, source: root-bootstrap/)
  AppProject: modgpt.cluster-addons   ← group boundary
  Application: modgpt.cluster-addons  ← app of apps; syncs bootstrap/

  AppProject: modgpt.ask              ← (future) group boundary
  Application: modgpt.ask             ← (future) app of apps

        ↓ GitOps

LAYER 3 — Services (managed by each Layer 2 app, source: bootstrap/)
  Application: modgpt.cluster-addons.vault
  Application: modgpt.cluster-addons.vso
  Application: modgpt.cluster-addons.reloader
  Application: modgpt.cluster-addons.pdb

Layer	Object	Lives in	How to change	Auto-applies?
1	`AppProject: modgpt`	Cluster only	`kubectl apply` — manual	❌
1	`Application: modgpt`	Cluster only	`kubectl apply` — manual	❌
2	`AppProject: modgpt.cluster-addons`	`root-bootstrap/` in cluster-addons git	`git push` → modgpt app syncs	✅
2	`Application: modgpt.cluster-addons`	`root-bootstrap/` in cluster-addons git	`git push` → modgpt app syncs	✅
3	`Application: modgpt.cluster-addons.*`	`bootstrap/` in cluster-addons git	`git push` → modgpt.cluster-addons syncs	✅

No ApplicationSet in cluster-addons

cluster-addons was migrated to a pure App of Apps pattern — no ApplicationSet. Layer 2 and 3 objects are static YAML files in git, not generated from a template. Adding a new service = add bootstrap/application-<svc>.yaml and push.

Layer 1 — AppProject modgpt

The tenancy boundary: which repos apps may pull from, which namespaces they may deploy into.

View¶

kubectl -n argocd get appproject modgpt -o yaml

Current canonical manifest¶

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: modgpt
  namespace: argocd
spec:
  description: "modgpt tenant — root boundary"
  sourceRepos:
    - "https://gitlab.cl1.sq4.aegis.internal/modgpt/*"
    - "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/*/packages/helm/stable"
  destinations:
    - server: "https://kubernetes.default.svc"
      namespace: "*"
  clusterResourceWhitelist:
    - { group: "*", kind: "*" }
  namespaceResourceWhitelist:
    - { group: "*", kind: "*" }

Change it¶

# Edit and apply
kubectl apply -f /tmp/modgpt-project.yaml

# Or patch a single field (e.g. add a new allowed repo)
kubectl -n argocd patch appproject modgpt --type=json \
  -p='[{"op":"add","path":"/spec/sourceRepos/-","value":"https://gitlab.cl1.sq4.aegis.internal/modgpt/new-app.git"}]'

No git push — it is not reconciled from git. The change is live immediately.

Layer 1 — Root Application modgpt

The bootstrap entrypoint. Syncs root-bootstrap/ which contains the Layer 2 AppProject + Application pairs.

View¶

kubectl -n argocd get application modgpt -o yaml

Canonical manifest¶

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.io
spec:
  project: modgpt
  source:
    repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git
    targetRevision: HEAD
    path: root-bootstrap
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }

Change it¶

kubectl apply -f /tmp/modgpt-root-app.yaml

Only touch this to change where the bootstrap comes from — the repo URL, branch, or root-bootstrap path. Everything inside root-bootstrap/ is Layer 2 (GitOps).

Layer 2 — AppProject + Application pairs (App Groups)

These live in root-bootstrap/ and are managed by the modgpt root Application. Change them via git push.

Add a new app group (e.g. ASK)¶

cd ~/cluster-addons

cat > root-bootstrap/appproject-ask.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: modgpt.ask
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  description: "Layer 2 — ASK application group"
  sourceRepos:
    - "https://gitlab.cl1.sq4.aegis.internal/modgpt/ask.git"
    - "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<ASK-PID>/packages/helm/stable"
  destinations:
    - server: "https://kubernetes.default.svc"
      namespace: "*"
  clusterResourceWhitelist:
    - { group: "*", kind: "*" }
  namespaceResourceWhitelist:
    - { group: "*", kind: "*" }
EOF

cat > root-bootstrap/application-ask.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt.ask
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  project: modgpt.ask
  source:
    repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/ask.git
    targetRevision: HEAD
    path: bootstrap
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }
EOF

GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)
git add root-bootstrap/
git commit -m "feat: add modgpt.ask app group (Layer 2)"
git remote set-url origin https://oauth2:${GITLAB_TOKEN}@gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git
git push origin main
# modgpt root app picks it up → creates AppProject + Application automatically

Sync wave order

AppProject gets sync-wave: "-1" so it exists before the Application (wave 0) references it. Without this, ArgoCD creates both in the same wave and the Application fails with "project not found".

Modify an existing app group¶

Edit the YAML in root-bootstrap/ and push — the modgpt root app reconciles it:

cd ~/cluster-addons
vi root-bootstrap/appproject-cluster-addons.yaml   # e.g. add a new sourceRepo
git commit -am "fix: widen cluster-addons AppProject sourceRepos"
git push origin main

Remove an app group¶

git rm root-bootstrap/appproject-ask.yaml root-bootstrap/application-ask.yaml
git commit -m "chore: remove ask app group"
git push origin main
# modgpt app prunes AppProject + Application → which prunes all Layer 3 children

Layer 3 — Service Applications

These live in bootstrap/ in each app group's repo and are managed by their Layer 2 Application.

Add a new service to cluster-addons¶

cd ~/cluster-addons

cat > bootstrap/application-<svc>.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt.cluster-addons.<svc>
  namespace: argocd
spec:
  project: modgpt.cluster-addons
  destination:
    server: https://kubernetes.default.svc
    namespace: <svc>
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]
  sources:
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/1/packages/helm/stable
      chart: <chart>
      targetRevision: "<version>"
      helm:
        releaseName: <svc>
        valueFiles: ["$values/values/<svc>/<svc>-stg.yaml"]
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git
      targetRevision: HEAD
      ref: values
EOF

# Add values file
mkdir -p values/<svc>
vi values/<svc>/<svc>-stg.yaml

GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)
git add bootstrap/application-<svc>.yaml values/<svc>/
git commit -m "feat: add <svc> service to cluster-addons"
git push origin main
# modgpt.cluster-addons app creates modgpt.cluster-addons.<svc> automatically

Upgrade a service¶

vi bootstrap/application-vault.yaml    # bump targetRevision
vi values/vault/vault-stg.yaml         # bump image tags
git commit -am "chore: upgrade vault to 0.33.0"
git push origin main

Remove a service¶

git rm bootstrap/application-<svc>.yaml
git commit -m "chore: remove <svc>"
git push origin main
# modgpt.cluster-addons prunes the Application → prunes Helm resources

Force Reconciliation — Cheat Sheet

Goal	Command
Soft refresh (re-compare to git)	`kubectl -n argocd annotate app <app> argocd.argoproj.io/refresh=normal --overwrite`
Hard refresh (re-generate manifests, clear cache)	`kubectl -n argocd annotate app <app> argocd.argoproj.io/refresh=hard --overwrite`
Trigger sync immediately	`kubectl -n argocd patch app <app> --type merge -p '{"operation":{"sync":{}}}'`
View diff vs git	`argocd app diff <app>`
List all apps + health	`kubectl -n argocd get applications.argoproj.io`
List all AppProjects	`kubectl -n argocd get appproject`

Rollback

Layer	Rollback
Layer 3 (service Application)	`git revert` the `bootstrap/` commit + push
Layer 2 (app group AppProject/Application)	`git revert` the `root-bootstrap/` commit + push
Layer 1 (AppProject modgpt / root Application)	`kubectl apply` the previous manifest
One service to a previous synced state	`argocd app rollback modgpt.cluster-addons.<svc>`

Verification

# Full hierarchy check
kubectl -n argocd get appproject                     # modgpt, modgpt.cluster-addons
kubectl -n argocd get application modgpt             # L1: Synced/Healthy
kubectl -n argocd get application modgpt.cluster-addons  # L2: Synced/Healthy
kubectl -n argocd get applications.argoproj.io | grep cluster-addons  # L3: all Synced/Healthy