GitLab — Deployment & Setup¶

Deploy GitLab in-cluster on RKE2 and make it accessible & secure — Git, container registry, Helm registry, and PyPI. GitLab is the source of truth ArgoCD reconciles from.

How GitLab is hosted

In this environment, GitLab runs inside the RKE2 cluster, deployed via the official gitlab/gitlab Helm chart with all state on Ceph (block for Gitaly / PostgreSQL / Redis; object via RGW/S3 for uploads, LFS, artifacts, and container-registry blobs). It is delivered and reconciled through ArgoCD like any other platform workload.

Property	Value
Deployment	`gitlab/gitlab` Helm chart — in-cluster, namespace `gitlab`
Git + Helm registry	`https://gitlab.cl1.sq4.aegis.internal`
Container registry	`https://registry.gitlab.cl1.sq4.aegis.internal`
Storage	Ceph — block (Gitaly/PostgreSQL/Redis) + object/RGW (uploads, LFS, registry blobs); bundled MinIO disabled
TLS	internal CA (cert-manager)
Ingress	NGINX ingress (ingress VIP `100.115.2.100`)

What this page covers

Architecture — how GitLab is laid out in-cluster on Ceph.
Phase 1 — Deploy GitLab on Kubernetes — the gitlab/gitlab Helm install (bootstrap).
Phase 2 — DNS, registry trust & first login.
The modgpt group, projects, access tokens, and the container / Helm / PyPI registries.
The ArgoCD GitOps credentials so ArgoCD can pull Git manifests and Helm charts.

Architecture

Design: GitLab on Kubernetes with Ceph Object Storage¶

GitLab is deployed inside the cluster using the official gitlab/gitlab Helm chart. It uses Ceph for all persistence:

Ceph Block (ceph-block) — Gitaly (Git repos), PostgreSQL, Redis
Ceph Object (RGW, S3-compatible) — uploads, LFS, artifacts, packages, registry blobs

The bundled MinIO that the chart ships is disabled — Ceph's RADOS Gateway already provides the same S3-compatible API natively, with 3× replication built in. This removes one layer of indirection.

Developer workstation
       │  https://gitlab.cl1.sq4.aegis.internal
       │  https://registry.gitlab.cl1.sq4.aegis.internal
       ▼
100.115.2.100 (nginx ingress — node1)
       │  → gitlab-webservice.gitlab.svc.cluster.local:8080
       │  → gitlab-registry.gitlab.svc.cluster.local:5000
       ▼
Kubernetes cluster (namespace: gitlab)
  ├─ gitlab-webservice   (web UI + Git API)         PDB: minAvailable 1
  ├─ gitlab-workhorse    (sidecar: Git proxy)
  ├─ gitlab-sidekiq      (background jobs)           PDB: minAvailable 1
  ├─ gitlab-gitaly       (Git data storage)          PVC: ceph-block 20Gi
  ├─ gitlab-registry     (Container + Helm OCI)      → object storage via S3
  ├─ gitlab-postgresql   (relational DB)             PVC: ceph-block 10Gi
  └─ gitlab-redis        (cache / sessions)          PVC: ceph-block 2Gi

Object storage (no MinIO pod — direct to Ceph RGW)
  └─ rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80
       ├─ gitlab-artifacts      (CI/CD artifacts)
       ├─ gitlab-lfs            (Git LFS blobs)
       ├─ gitlab-uploads        (user uploads / attachments)
       ├─ gitlab-packages       (Package registry: PyPI, Helm, etc.)
       ├─ gitlab-registry       (Container registry blobs)
       └─ gitlab-backups        (GitLab backups)

Known deviations from production¶

Deviation	Reason	Production-correct
Single webservice replica	Resource constraints	2+ replicas with HPA
Bundled PostgreSQL	Avoids cross-namespace dependency	External PostgreSQL HA cluster
Bundled Redis	Avoids cross-namespace dependency	External Redis Sentinel/Cluster
No GitLab Runner	Not needed for registry-only use	Kubernetes executor, auto-scaling

Bootstrap exception¶

GitLab images pulled from public registry on first deploy

GitLab cannot be bootstrapped from itself — the same pattern as ArgoCD (bootstrapped manually before it could manage itself). On first install, helm install pulls images from registry.gitlab.com. After GitLab is running, all future workloads use registry.gitlab.cl1.sq4.aegis.internal.

Phase 1 — Deploy GitLab on Kubernetes

Step 1.1 — Mirror the GitLab chart into Harbor (air-gap)¶

charts.gitlab.io is not reachable from the cluster — the chart must first be mirrored into Harbor's OCI helm-charts project. This is a one-time step run from the bastion (which has the HTTP proxy 10.96.137.37:3128 to reach upstream). The chart and pinned version are already declared in scripts/harbor-mirror/helm-charts/mirror-charts.yaml (gitlab → 9.11.5):

# On the bastion — mirror the GitLab chart into Harbor OCI (helm pull via proxy → helm push)
cd scripts/harbor-mirror/helm-charts
./core42-mirror-charts.py --only gitlab --dry-run   # preview
./core42-mirror-charts.py --only gitlab             # push to Harbor

Result: the chart lands at oci://harbor.cl1.sq4.aegis.internal/helm-charts/gitlab:9.11.5. Log in to Harbor OCI and confirm (the deploy steps below pull from here, not from the internet):

helm registry login harbor.cl1.sq4.aegis.internal   # robot$ask creds (~/harbor-images/harbor-robot.json)
helm show chart oci://harbor.cl1.sq4.aegis.internal/helm-charts/gitlab --version 9.11.5 | head

Chart version

Chart 9.11.5 = GitLab v18.11.4. To bump, edit the gitlab entry in mirror-charts.yaml and re-run the mirror. Avoid .0 releases of a new major (e.g. v19.0.x) in production.

Step 1.2 — Create namespace and root password secret¶

kubectl create namespace gitlab

# Generate password
GITLAB_ROOT_PASS=$(openssl rand -base64 24 | tr -d '/+=')
echo "Root password: ${GITLAB_ROOT_PASS}"

# Kubernetes secret (GitLab reads this once at first start)
kubectl create secret generic gitlab-initial-root-password \
  -n gitlab \
  --from-literal=password="${GITLAB_ROOT_PASS}"

# Vault (source of truth)
vault kv put secret/gitlab/root-password \
  username="root" \
  password="${GITLAB_ROOT_PASS}"

Step 1.3 — Create TLS certificate¶

Two hostnames only — MinIO is disabled so no MinIO ingress is created:

kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gitlab-tls
  namespace: gitlab
spec:
  secretName: gitlab-tls
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer
  dnsNames:
    - gitlab.cl1.sq4.aegis.internal
    - registry.gitlab.cl1.sq4.aegis.internal
EOF

kubectl get certificate gitlab-tls -n gitlab --watch
# Wait for READY=True, then Ctrl+C

Step 1.3b — (Upgrade) Enable cert-manager auto-provisioning on GitLab ingresses¶

What changed and why

Previously, global.ingress.configureCertmanager: false and a manual Certificate resource (Step 1.3) pre-created the gitlab-tls secret. That works but requires human intervention on every renewal and on new hostnames.

The production-correct path is to add cert-manager.io/cluster-issuer: internal-ca-issuer to global.ingress.annotations. cert-manager then watches all GitLab ingresses and automatically issues and renews TLS certs — the manual Certificate in Step 1.3 is no longer needed.

Why we keep `configureCertmanager: false`¶

Flag	Our value	Why
`global.certmanager.install`	`false`	Never install the bundled cert-manager (Helm sub-chart) — cluster already has the singleton
`global.ingress.configureCertmanager`	`false`	When `true`, the chart creates a namespace-scoped `Issuer` — wrong when we have a `ClusterIssuer`. We use the annotation path instead.
`cert-manager.io/cluster-issuer: internal-ca-issuer`	annotation	Tells cert-manager to use the cluster-wide `ClusterIssuer`. cert-manager sees the ingress TLS block + this annotation → creates + renews the cert automatically.

If GitLab is already running — apply the change via `helm upgrade`¶

1. Edit the values file on the bastion:

# Add the cert-manager annotation to global.ingress.annotations in ~/gitlab-stg.yaml
# The block should look like this:
#   global:
#     ingress:
#       configureCertmanager: false
#       ...
#       annotations:
#         cert-manager.io/cluster-issuer: internal-ca-issuer   # ← add this line
#         nginx.ingress.kubernetes.io/proxy-body-size: "0"
#         ...

2. Delete the manually-created Certificate (cert-manager will re-own it):

# This deletes the Certificate CRD object and the gitlab-tls Secret.
# cert-manager will immediately re-issue it via the ingress annotation.
kubectl delete certificate gitlab-tls -n gitlab
kubectl delete secret gitlab-tls -n gitlab 2>/dev/null || true

3. Apply via helm upgrade:

helm upgrade gitlab /tmp/gitlab-chart/gitlab \
  -n gitlab \
  -f ~/gitlab-stg.yaml \
  --set certmanager.install=false \
  --timeout 10m \
  --wait

4. Verify cert-manager issued the cert automatically:

# cert-manager creates a Certificate for each ingress TLS block with the annotation
kubectl get certificate -n gitlab
# Expected: gitlab-tls   True   <hostname>   ...   (READY=True within ~30 seconds)

kubectl describe certificate gitlab-tls -n gitlab | grep -A5 "Conditions:"
# Expected: Type: Ready, Status: True, Message: Certificate is up to date and has not expired

# Confirm the Secret was issued (not from the deleted manual cert)
kubectl get secret gitlab-tls -n gitlab -o jsonpath='{.metadata.annotations}' | python3 -m json.tool
# Expected: cert-manager.io/issuer-name: internal-ca-issuer (or cluster-issuer equivalent)

5. Confirm GitLab TLS still works:

curl -sk https://gitlab.cl1.sq4.aegis.internal/-/health
# Expected: GitLab OK

If cert issuance takes > 60 seconds

Check kubectl describe certificaterequest -n gitlab and kubectl describe order -n gitlab (for ACME issuers; not applicable here since internal-ca-issuer is a CA issuer). For a CA issuer, cert issuance is synchronous and nearly instant. A long delay usually means the ClusterIssuer name is wrong or the issuer is not Ready.

kubectl get clusterissuer internal-ca-issuer
# Expected: READY=True

Step 1.4 — Create Ceph object store user for GitLab¶

GitLab needs an S3 user on the existing ceph-objectstore. Rook provisions the credentials automatically:

kubectl apply -f - << 'EOF'
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: gitlab
  namespace: rook-ceph
spec:
  store: ceph-objectstore
  displayName: "GitLab Object Storage"
  capabilities:
    user: "*"
    bucket: "*"
    metadata: "*"
    usage: "*"
EOF

# Wait for Rook to create the credentials secret (~15 seconds)
kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph --watch
# Wait for secret to appear, then Ctrl+C

Step 1.5 — Extract credentials and store in Vault¶

S3_ACCESS_KEY=$(kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph -o jsonpath='{.data.AccessKey}' | base64 -d)
S3_SECRET_KEY=$(kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph -o jsonpath='{.data.SecretKey}' | base64 -d)
S3_ENDPOINT="http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local"

# Store in Vault
vault kv put secret/gitlab/ceph-s3 \
  access_key="${S3_ACCESS_KEY}" \
  secret_key="${S3_SECRET_KEY}" \
  endpoint="${S3_ENDPOINT}"

vault kv get secret/gitlab/ceph-s3

Step 1.6 — Create Kubernetes secrets for GitLab¶

GitLab needs two secrets — one for general object storage (uploads/LFS/artifacts/packages), one for the container registry blobs.

These reuse $S3_ACCESS_KEY / $S3_SECRET_KEY from Step 1.5 — re-hydrate if you're in a new shell

The credentials are not generated here — they come from the Rook-created secret (Step 1.4) via the env vars set in Step 1.5. Run Steps 1.4 → 1.5 → 1.6 in the same shell. If you opened a fresh shell (vars empty), the secrets would be created with blank keys. Re-hydrate from Vault first:

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
S3_ACCESS_KEY=$(vault kv get -field=access_key secret/gitlab/ceph-s3)
S3_SECRET_KEY=$(vault kv get -field=secret_key secret/gitlab/ceph-s3)
# sanity: both must be non-empty
echo "access=[${S3_ACCESS_KEY:?empty — fix before continuing}] secret-len=${#S3_SECRET_KEY}"

# S3 connection for uploads, LFS, artifacts, packages
kubectl create secret generic gitlab-s3-connection \
  -n gitlab \
  --from-literal=connection="$(cat << CONNEOF
provider: AWS
region: us-east-1
aws_access_key_id: ${S3_ACCESS_KEY}
aws_secret_access_key: ${S3_SECRET_KEY}
aws_signature_version: 4
host: rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local
endpoint: http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local
path_style: true
CONNEOF
)"

# S3 config for GitLab container registry blobs
kubectl create secret generic gitlab-registry-storage \
  -n gitlab \
  --from-literal=config="$(cat << REGEOF
s3:
  bucket: gitlab-registry
  accesskey: ${S3_ACCESS_KEY}
  secretkey: ${S3_SECRET_KEY}
  regionendpoint: http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local
  region: us-east-1
  pathstyle: true
  secure: false
REGEOF
)"

kubectl get secret gitlab-s3-connection gitlab-registry-storage -n gitlab

The kubectl create secret above is the bootstrap path — production uses VSO

Those two commands write plaintext S3 credentials into Kubernetes Secrets by hand. That is acceptable only to get GitLab running the first time. Per the secrets-in-Vault mandate, the production-correct approach is to keep the credentials solely in Vault (secret/gitlab/ceph-s3) and have the Vault Secrets Operator (VSO) render these two Kubernetes Secrets from Vault — single source of truth, automatic rotation, no plaintext drift. Step 1.6b converts them.

Step 1.6b — (Production) Render the secrets from Vault via VSO¶

Vault already holds the Ceph S3 creds at secret/gitlab/ceph-s3 (fields: access_key, secret_key, endpoint — written in Step 1.5). We make VSO materialize the exact same gitlab-s3-connection and gitlab-registry-storage Secrets from those fields.

1) Vault side — least-privilege policy + a gitlab k8s-auth role:

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal

# Policy: read ONLY secret/gitlab/* (not all of secret/*)
vault policy write gitlab-ro - <<'POL'
path "secret/data/gitlab/*"     { capabilities = ["read"] }
path "secret/metadata/gitlab/*" { capabilities = ["read","list"] }
POL

# k8s auth role bound to a dedicated SA in the gitlab namespace
vault write auth/kubernetes/role/gitlab \
  bound_service_account_names=gitlab-vso \
  bound_service_account_namespaces=gitlab \
  policies=gitlab-ro ttl=1h

2) Kubernetes side — dedicated SA + a gitlab-namespace VaultConnection/VaultAuth:

kubectl create serviceaccount gitlab-vso -n gitlab

kubectl apply -f - <<'YAML'
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata: {name: vault, namespace: gitlab}
spec:
  address: https://vault.cl1.sq4.aegis.internal
  skipTLSVerify: true
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata: {name: vault, namespace: gitlab}
spec:
  method: kubernetes
  mount: kubernetes
  vaultConnectionRef: vault
  kubernetes:
    role: gitlab
    serviceAccount: gitlab-vso
    audiences: ["vault"]
YAML

Why a per-namespace VaultAuth + SA (not the operator's default)

VSO resolves a VaultAuth's serviceAccount in the namespace of the secret that uses it — so referencing the operator's vault-secrets-operator/default VaultAuth from gitlab fails with ServiceAccount "vso-...-controller-manager" not found. Each consuming namespace needs its own VaultAuth + ServiceAccount, bound to a Vault role scoped to just that namespace's secrets. This is also the least-privilege production pattern — gitlab-ro can read only secret/gitlab/*.

3) Two VaultStaticSecrets that template the exact Secret bodies (overwrite: true adopts the bootstrap Secrets in place):

kubectl apply -f - <<'YAML'
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: {name: gitlab-s3-connection, namespace: gitlab}
spec:
  type: kv-v2
  mount: secret
  path: gitlab/ceph-s3
  vaultAuthRef: vault
  refreshAfter: 1h
  destination:
    name: gitlab-s3-connection
    create: true
    overwrite: true               # adopt the existing hand-created Secret
    transformation:
      excludeRaw: true            # write ONLY the templated key, not raw vault fields
      templates:
        connection:
          text: |-
            provider: AWS
            region: us-east-1
            aws_access_key_id: {{ .Secrets.access_key }}
            aws_secret_access_key: {{ .Secrets.secret_key }}
            aws_signature_version: 4
            host: {{ .Secrets.endpoint | trimPrefix "http://" | trimPrefix "https://" }}
            endpoint: {{ .Secrets.endpoint }}
            path_style: true
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: {name: gitlab-registry-storage, namespace: gitlab}
spec:
  type: kv-v2
  mount: secret
  path: gitlab/ceph-s3
  vaultAuthRef: vault
  refreshAfter: 1h
  destination:
    name: gitlab-registry-storage
    create: true
    overwrite: true
    transformation:
      excludeRaw: true
      templates:
        config:
          text: |-
            s3:
              bucket: gitlab-registry
              accesskey: {{ .Secrets.access_key }}
              secretkey: {{ .Secrets.secret_key }}
              regionendpoint: {{ .Secrets.endpoint }}
              region: us-east-1
              pathstyle: true
              secure: false
YAML

4) Verify (the Secrets become VSO-owned, content unchanged → no GitLab restart needed):

kubectl get vaultstaticsecret -n gitlab
kubectl get secret gitlab-s3-connection gitlab-registry-storage -n gitlab \
  -o jsonpath='{range .items[*]}{.metadata.name}{"  managed-by="}{.metadata.labels.app\.kubernetes\.io/managed-by}{"\n"}{end}'
# Expect: managed-by=hashicorp-vso  for both

Zero-downtime cutover

Because the templated output is byte-identical to the bootstrap Secrets, adopting them with overwrite: true does not change the data GitLab reads — pods keep running. Validate the template first by rendering to throwaway *-vso destination names and diff-ing against the live Secrets before pointing VSO at the real names. This VaultAuth + SA + VaultStaticSecret trio is the reusable pattern for every app namespace that needs Vault secrets.

Which GitLab secret belongs to which manager (decision matrix)¶

Not every secret should be a VaultStaticSecret. Pick the manager by where the source of truth lives and what lifecycle the secret has:

Secret	Type	Source of truth	Manager	Why
`gitlab-s3-connection`	Opaque	Vault `secret/gitlab/ceph-s3`	VSO `VaultStaticSecret`	Runtime credential, Vault-sourced → declarative + rotatable
`gitlab-registry-storage`	Opaque	Vault `secret/gitlab/ceph-s3`	VSO `VaultStaticSecret`	Same — registry blob-store creds
`gitlab-tls`	`kubernetes.io/tls`	cert-manager `Certificate` (`internal-ca-issuer`)	cert-manager	Certs need CSR + issuance + auto-renewal — that's cert-manager's job, not Vault's. A VSS here would never renew.
`gitlab-initial-root-password`	Opaque	Vault `secret/gitlab/root-password`	Bootstrap only — leave as-is	Read by the chart only at first install; slated for deletion after first-login rotation (Phase 2 · Step 2.4.5). VSO-managing it would fight that retirement.

Rule of thumb:

✅ VSO — secrets whose source of truth is Vault and that are read at runtime (the two S3 secrets).
🔒 cert-manager — anything kubernetes.io/tls. Let the cert controller own issuance + renewal.
🧹 Leave alone — bootstrap-only secrets (seed-once, then retired). Keep the canonical copy in Vault as the record; don't have an operator keep re-creating a secret you intend to delete.

Step 1.7 — Resource planning: ensure adequate node memory¶

Node memory (in-cluster method only)

The in-cluster GitLab release requests ~2.2 GiB of memory (PostgreSQL HA ×3, pgpool ×2, GitLab pods ×2, Valkey ×3). Ensure the target worker nodes have adequate free memory before installing; the production CPU workers (768 GB–1 TB RAM) have ample headroom. (On the production VM-based GitLab this does not apply.)

Verify memory is freed:

kubectl describe node node1 | grep -A5 'Allocated resources:' | grep memory
kubectl describe node node2 | grep -A5 'Allocated resources:' | grep memory
kubectl describe node node3 | grep -A5 'Allocated resources:' | grep memory
# Expected: memory usage drops from 98% to ~30-40%

Step 1.7b — Write the values file¶

cat > ~/gitlab-stg.yaml << 'EOF'
global:
  hosts:
    domain: cl1.sq4.aegis.internal
    https: true
    gitlab:
      name: gitlab.cl1.sq4.aegis.internal
    registry:
      name: registry.gitlab.cl1.sq4.aegis.internal
  ingress:
    configureCertmanager: false   # keep false — we use ClusterIssuer, not a namespace Issuer
    provider: nginx
    class: nginx
    tls:
      enabled: true
      secretName: gitlab-tls      # cert-manager writes the cert into this named secret
    annotations:
      # cert-manager auto-provisions + renews the cert via our cluster-wide ClusterIssuer
      cert-manager.io/cluster-issuer: internal-ca-issuer
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "300"

  certmanager:
    install: false                # never install the bundled cert-manager — use the cluster singleton
  certmanager-issuer:
    email: admin@cl1.sq4.aegis.internal

  initialRootPassword:
    secret: gitlab-initial-root-password
    key: password

  # ── Object storage: Ceph RGW (no MinIO) ────────────────────────────────────
  minio:
    enabled: false

  appConfig:
    object_store:
      enabled: true
      proxy_download: true
      connection:
        secret: gitlab-s3-connection
        key: connection

    # NOTE: when object_store.enabled=true, individual items must NOT have connection:
    # The connection is inherited from object_store.connection above.
    artifacts:
      enabled: true
      bucket: gitlab-artifacts
    lfs:
      enabled: true
      bucket: gitlab-lfs
    uploads:
      enabled: true
      bucket: gitlab-uploads
    packages:
      enabled: true
      bucket: gitlab-packages
    backups:
      bucket: gitlab-backups
      tmpBucket: gitlab-backups-tmp

# ── Disable what we already have ────────────────────────────────────────────────
# NOTE: certmanager and minio are disabled via global.* — no top-level blocks needed
nginx-ingress:
  enabled: false

prometheus:
  install: false

gitlab-runner:
  install: false

# ── Registry — blobs stored in Ceph via S3 ──────────────────────────────────────
registry:
  enabled: true
  storage:
    secret: gitlab-registry-storage
    key: config
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: "1"
      memory: 512Mi

# ── Scale down to free node memory (only if nodes are memory-constrained) ──────
# NOTE: every "# PROD:" comment is the production-recommended value. The bare value
# is a compromise forced by RAM-constrained pre-production nodes. Revert to PROD once nodes
# are rebalanced (see troubleshooting → Host Memory Overcommit).
gitlab:
  webservice:
    replicaCount: 1      # PROD: 3+  (HA; survive a node loss) with podAntiAffinity + a PDB (minAvailable: 2)
    minReplicas: 1       # PROD: 3   (HPA floor)
    maxReplicas: 1       # PROD: 8-10 (HPA scales on CPU under request load)
    workerProcesses: 1   # CRITICAL on 3.3 GiB nodes — 2 Puma workers OOM at boot eager-load. PROD: 2-4 (≈ vCPUs) on 8 GiB+ nodes
    resources:
      requests:
        cpu: 200m        # PROD: "1"   (Rails is CPU-heavy under real traffic; request ≈ steady use)
        memory: 300Mi    # keep request low to fit 3.3 GiB nodes; Rails uses more at runtime. PROD: 2.5Gi (request ≈ real RSS so the scheduler reserves it)
      limits:
        cpu: "2"         # PROD: "4"
        memory: 3Gi      # boot eager-load spikes to ~2.5Gi then settles ~1.6Gi — 2.5Gi limit OOMs during preload. PROD: 5Gi (headroom for N workers)
    workhorse:           # must set EXPLICITLY — pod request = webservice + workhorse
      resources:
        requests:
          cpu: 10m       # PROD: 100m
          memory: 64Mi   # PROD: 128Mi
        limits:
          cpu: "500m"    # PROD: "1"
          memory: 256Mi  # PROD: 512Mi

  sidekiq:
    replicaCount: 1      # PROD: 2+  (HA background processing) + PDB
    minReplicas: 1       # pin HPA to 1 — default max=10 scales out under CPU and Pending-deadlocks on memory. PROD: minReplicas 2
    maxReplicas: 1       # PROD: 10  (let the HPA scale queues out; cluster must have the memory)
    resources:
      requests:
        cpu: 50m         # PROD: 500m
        memory: 256Mi    # PROD: 2Gi  (request ≈ real RSS)
      limits:
        cpu: "1"         # PROD: "2"
        memory: 3Gi      # sidekiq also eager-loads Rails — same ~2.5Gi boot spike as webservice. PROD: 4Gi

  kas:
    minReplicas: 1       # PROD: 2  (HA agent server) + PDB
    maxReplicas: 1       # PROD: 3
    resources:
      requests:
        cpu: 10m         # PROD: 100m
        memory: 100Mi    # PROD: 256Mi
      limits:
        cpu: "500m"      # PROD: "1"
        memory: 512Mi    # PROD: 1Gi

  gitaly:
    persistence:
      enabled: true
      storageClass: ceph-block   # PROD: fast RWO SSD/NVMe class; Gitaly is latency-sensitive
      size: 20Gi                 # PROD: size to repo growth (100Gi+); cannot shrink later
    resources:
      requests:
        cpu: 100m        # PROD: "1"  (git pack/gc is CPU-heavy)
        memory: 200Mi    # PROD: 2Gi
      limits:
        cpu: "1"         # PROD: "2"
        memory: 512Mi    # PROD: 4Gi
    # PROD HA: run Gitaly Cluster (Praefect + 3 Gitaly nodes) for replicated repos — a single Gitaly is a SPOF

  gitlab-shell:
    # PROD: replicas: 2 + PDB (SSH git access stays up during a node drain)
    resources:
      requests:
        cpu: 10m         # PROD: 100m
        memory: 24Mi     # PROD: 128Mi
      limits:
        cpu: "500m"      # PROD: "1"
        memory: 128Mi    # PROD: 256Mi

  toolbox:
    enabled: true
    resources:
      requests:
        cpu: 50m         # PROD: 100m
        memory: 200Mi    # PROD: 512Mi
      limits:
        cpu: "1"
        memory: 2Gi      # 2Gi needed for initial db:migrate (1381 migrations, Rails app boot ~1.5Gi RSS). PROD: 2Gi is fine; raise if backups run here

# ── PostgreSQL (bundled) ────────────────────────────────────────────────────────
# PROD: do NOT use the bundled single-instance PostgreSQL. Use an EXTERNAL HA
# Postgres (cloud-managed, or Patroni/CloudNativePG: 1 primary + 2 replicas,
# automated failover + PITR backups). Bundled chart = single pod = SPOF + no backups.
postgresql:
  primary:
    persistence:
      storageClass: ceph-block
      size: 10Gi         # PROD (if self-hosting): 100Gi+ on fast SSD, sized to DB growth
  resources:
    limits:
      memory: 1Gi        # PROD: 8Gi+ (Postgres benefits hugely from shared_buffers/cache)
      cpu: "1"           # PROD: "4"

# ── Redis (bundled) ─────────────────────────────────────────────────────────────
# PROD: do NOT use the bundled single Redis. Use EXTERNAL Redis HA (Sentinel or
# cluster, 3+ nodes). Sessions/cache/queues all depend on Redis — a single pod is a SPOF.
redis:
  master:
    persistence:
      storageClass: ceph-block
      size: 2Gi          # PROD: size to working set (8Gi+)
  resources:
    limits:
      memory: 256Mi      # PROD: 2Gi+ with a maxmemory-policy set
EOF

echo "✅ Values file written — $(wc -l < ~/gitlab-stg.yaml) lines"

Why we reduced Puma workers to 1 (`gitlab.webservice.workerProcesses: 1`)¶

GitLab's webservice runs Puma, a multi-process Ruby application server. The chart default is 2 worker processes plus the Puma master. Each Puma worker is a full fork of the Rails application.

The boot-time OOM problem on small nodes

On startup, each Puma worker eager-loads the entire Rails codebase (bootsnap + autoload of all of GitLab). This is a one-time RSS spike to ~2.5 GiB per worker before copy-on-write sharing and GC settle it down to ~1.6 GiB.

With the default 2 workers on our 3.3 GiB worker nodes:

2 workers × ~2.5 GiB boot spike = the pod cgroup blows past any sane memory limit
The container is OOMKilled (exit 137) during preload, on every boot → permanent CrashLoopBackOff
GitLab sets DISABLE_PUMA_WORKER_KILLER=true by default, so nothing recycles a bloated worker either

Fix (two parts, both required):

Change	Value	Reason
`gitlab.webservice.workerProcesses`	`1`	Halves the boot RSS spike — one worker fits in the node's real headroom
`gitlab.webservice.resources.limits.memory`	`3Gi`	The single worker still spikes to ~2.5 GiB at preload; a 2.5 GiB limit OOMs during boot. 3 GiB survives the spike, then settles ~1.6 GiB

This is a resource-constrained compromise — state the production-correct alternative

Production-correct: keep workerProcesses: 2 (or more) for request concurrency and HA, on nodes with 8 GiB+ RAM so the eager-load spike has headroom. One Puma worker means the webservice handles fewer concurrent requests and has no in-pod redundancy.

We run 1 worker only because the pre-production nodes were RAM-constrained. Once the nodes have adequate memory, revert to workerProcesses: 2 and re-run helm upgrade.

Summary of config changes vs chart defaults — and why¶

Every change below is driven by one of three production principles: (A) consume shared platform services, don't re-install them; (B) externalize state to Ceph (HA storage), not pod-local or bundled object storage; (C) fit RAM-constrained pre-production nodes without hiding the production-correct value.

Setting	Chart default	Our value	Principle	Why
`global.certmanager.install`	`true` (bundled)	`false` + sub-chart deleted	A	Cluster cert-manager is a singleton; GitLab consumes it via ingress annotation, doesn't install its own (see above)
`global.ingress.configureCertmanager`	`false`	`false` (keep) + add `cert-manager.io/cluster-issuer` annotation	A	`configureCertmanager: true` creates a namespace `Issuer` — wrong when a `ClusterIssuer` already exists. We use the annotation path instead: cert-manager watches ingresses and auto-provisions certs
`nginx-ingress.enabled`	`true` (bundled)	`false`	A	Cluster already runs ingress-nginx as a shared platform component
`prometheus.install`	`true`	`false`	A	Observability stack is cluster-wide, not per-app
`global.minio.enabled`	`true` (bundled)	`false`	B	MinIO is single-replica ephemeral object storage. We use Ceph RGW (HA, S3-compatible) for artifacts/lfs/uploads/packages/registry/backups
`*.persistence.storageClass`	default SC	`ceph-block`	B	All stateful PVCs (Gitaly, Postgres, Redis) on replicated Ceph RBD, not node-local disk
`gitlab.webservice.workerProcesses`	`2`	`1`	C	Boot-time eager-load OOM on 3.3 GiB nodes (see above)
`webservice.resources.limits.memory`	`2.5Gi`-ish	`3Gi`	C	Survive the ~2.5 GiB preload spike without OOMKill
`sidekiq.{min,max}Replicas`	HPA max `10`	pinned `1`	C	Sidekiq also eager-loads Rails (~2.5 GiB each). Default HPA scales out under CPU and dead-locks the cluster on memory (pods stuck `Pending`)
`webservice.{min,max}Replicas`	HPA-managed	pinned `1`	C	One replica fits memory-constrained nodes; HPA scale-out would risk OOM
`gitlab-runner.install`	`false`	`false`	—	Runners are deployed separately, scoped per-project

GitOps note

In steady state this values file lives in Git and is applied by ArgoCD — not by manual helm install. The manual install here is the bootstrap of the GitLab platform itself; once GitLab is up and hosting its own repos, its Helm release is managed declaratively. The workerProcesses: 1 / pinned-replica compromises are explicitly marked so they can be reverted in the Git-tracked values once nodes are rebalanced.

Step 1.8 — Patch the chart and install GitLab¶

Why we disable the bundled cert-manager dependency chart¶

The GitLab Helm chart bundles cert-manager as a dependency sub-chart (charts/cert-manager). By default, installing GitLab would deploy a second, GitLab-owned cert-manager into the cluster.

Why a bundled cert-manager is wrong for this cluster

Our cluster already runs a cluster-wide cert-manager (installed as a shared platform component — see the cert-manager bootstrap page). Letting the GitLab chart install its own copy causes:

Two cert-manager controllers fighting over the same CRDs — both watch Certificate, Issuer, ClusterIssuer cluster-wide. CRD ownership conflicts and duplicate reconcile loops result in flapping certs and webhook race conditions.
CRD lifecycle coupling — if Helm uninstalls GitLab, it would try to remove cert-manager CRDs that other workloads (ArgoCD, Vault, ingress) depend on, taking down TLS across the whole cluster.
Version drift — the GitLab-pinned cert-manager version may differ from the platform version, giving two incompatible webhook versions.

Production rule: cert-manager is a singleton platform service. Exactly one instance, owned by the platform layer, shared by all tenants. Application charts must consume it, never install it.

We disable it in two layers — both are required:

Layer	What it does	Why it's needed
`global.certmanager.install: false` (+ `--set certmanager.install=false`)	Tells the GitLab chart's templating logic not to render cert-manager resources and not to auto-create an `Issuer`/`Certificate` for the GitLab hosts	Stops GitLab from deploying cert-manager workloads and from managing our TLS secret. We supply the cert manually via `global.ingress.tls.secretName: gitlab-tls`.
Physically removing `charts/cert-manager` + its `Chart.yaml`/`Chart.lock` entries	Deletes the sub-chart from the local chart copy before install	The `install: false` flag alone is not enough — see the Helm bug below.

Why --set certmanager.install=false alone fails — and we must delete the sub-chart

Due to a known Helm bug, a sub-chart's values.schema.json validation fires even when the sub-chart is disabled via a Helm condition. The cert-manager sub-chart schema uses additionalProperties: false and rejects the install key that GitLab's parent chart injects into it. The install aborts with:

Error: values don't meet the specifications of the schema(s) ...
additional properties 'install' not allowed

--disable-openapi-validation does not help — that flag only skips Kubernetes API-server schema validation, not Helm's client-side sub-chart schema validation. The only reliable fix is to physically remove the sub-chart directory and its references in Chart.yaml / Chart.lock from a local copy of the chart, then install from that local copy.

# Pull the chart locally
mkdir -p /tmp/gitlab-chart
helm pull oci://harbor.cl1.sq4.aegis.internal/helm-charts/gitlab --version 9.11.5 -d /tmp/gitlab-chart/ --untar

# Remove cert-manager sub-chart (directory + Chart.yaml/Chart.lock entries)
rm -rf /tmp/gitlab-chart/gitlab/charts/cert-manager

python3 - << 'EOF'
import yaml, pathlib

for fname in ["Chart.yaml", "Chart.lock"]:
    p = pathlib.Path(f"/tmp/gitlab-chart/gitlab/{fname}")
    if not p.exists():
        continue
    data = yaml.safe_load(p.read_text())
    key = "dependencies" if fname == "Chart.yaml" else "dependencies"
    if key in data and data[key]:
        data[key] = [d for d in data[key] if d.get("name") != "cert-manager"]
    p.write_text(yaml.dump(data, default_flow_style=False))
    print(f"✅ patched {fname}")
EOF

echo "✅ cert-manager sub-chart removed"
ls /tmp/gitlab-chart/gitlab/charts/  # cert-manager should NOT appear

Now install from the patched local chart:

helm install gitlab /tmp/gitlab-chart/gitlab \
  -n gitlab \
  -f ~/gitlab-stg.yaml \
  --set certmanager.install=false \
  --timeout 15m \
  --wait

Watch progress in a second terminal:

watch -n5 kubectl get pods -n gitlab

If the release is already in failed state from a previous attempt

helm uninstall gitlab -n gitlab
# Then re-run the helm install above

Step 1.9 — Trigger database migrations¶

The GitLab migrations Job runs as a pre-upgrade Helm hook. On a freshly installed cluster it may time out (if pods were previously Pending due to memory). If you see Init:Error on the webservice pod with "Database has not been initialized yet", run migrations manually from the toolbox:

# Check if migrations job completed
kubectl get jobs -n gitlab

# If the job shows Failed, delete it and run migrations from toolbox
kubectl delete job -n gitlab -l app=migrations 2>/dev/null || true

# Run migrations via toolbox (takes 5-15 minutes on first install)
nohup kubectl exec -n gitlab $(kubectl get pod -n gitlab -l app=toolbox -o jsonpath='{.items[0].metadata.name}') \
  -- gitlab-rake db:migrate > /tmp/migrations.log 2>&1 &
echo "Migrations running PID: $!"

# Tail progress
tail -f /tmp/migrations.log

Registry bucket must be pre-created

Unlike other GitLab buckets (created by the migrations job), the container registry bucket must exist before the registry pod starts or it will CrashLoopBackOff with s3aws: NoSuchBucket. Pre-create all buckets using the GitLab toolbox's built-in aws CLI (which runs inside the cluster and can resolve Ceph RGW's cluster-internal DNS):

S3_ACCESS_KEY=$(kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph -o jsonpath='{.data.AccessKey}' | base64 -d)
S3_SECRET_KEY=$(kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph -o jsonpath='{.data.SecretKey}' | base64 -d)
ENDPOINT='http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local'
TOOLBOX=$(kubectl get pod -n gitlab -l app=toolbox -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n gitlab $TOOLBOX -- bash -c "
  export AWS_ACCESS_KEY_ID='${S3_ACCESS_KEY}'
  export AWS_SECRET_ACCESS_KEY='${S3_SECRET_KEY}'
  export AWS_DEFAULT_REGION='us-east-1'
  for bucket in gitlab-artifacts gitlab-lfs gitlab-uploads gitlab-packages gitlab-registry gitlab-backups gitlab-backups-tmp; do
    aws --endpoint-url '${ENDPOINT}' s3 mb s3://\${bucket} 2>/dev/null || echo \"\${bucket} exists\"
  done
  aws --endpoint-url '${ENDPOINT}' s3 ls
"

Step 1.10 — Wait for webservice to start¶

Once migrations complete, delete the webservice pods so they restart cleanly:

# Check migrations progress
kubectl exec -n gitlab $(kubectl get pod -n gitlab -l app=toolbox -o jsonpath='{.items[0].metadata.name}') \
  -- gitlab-rake db:migrate:status 2>&1 | tail -5
# Expected last line: (in /var/opt/gitlab)   down  ...  or up  ...

# Delete the Init:Error webservice pod — it will be recreated and pass the dependencies check
kubectl delete pod -n gitlab -l app=webservice

# Watch
watch -n5 kubectl get pods -n gitlab

Step 1.11 — Verify all pods are Running¶

kubectl get pods -n gitlab

Expected (all Running/Completed):

NAME                                        READY   STATUS
gitlab-gitaly-0                             1/1     Running
gitlab-gitlab-shell-xxx                     1/1     Running
gitlab-postgresql-0                         2/2     Running
gitlab-redis-master-0                       2/2     Running
gitlab-registry-xxx                         1/1     Running
gitlab-sidekiq-xxx                          1/1     Running
gitlab-toolbox-xxx                          1/1     Running
gitlab-webservice-default-xxx               2/2     Running

No gitlab-minio pod — by design

MinIO is disabled. All object storage (uploads, LFS, artifacts, packages, registry blobs) goes directly to Ceph RGW at rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80. Buckets were pre-created in Step 1.9.

Step 1.10 — Verify web is reachable¶

curl -sk https://gitlab.cl1.sq4.aegis.internal/-/health
# Expected: GitLab OK

Phase 2 — DNS, Registry Trust & Initial Login

Step 2.1 — Update /etc/hosts¶

On Mac:

sudo bash -c 'echo "100.115.2.100 gitlab.cl1.sq4.aegis.internal registry.gitlab.cl1.sq4.aegis.internal" >> /etc/hosts'

On bastion:

sudo bash -c 'echo "100.115.2.100 gitlab.cl1.sq4.aegis.internal registry.gitlab.cl1.sq4.aegis.internal" >> /etc/hosts'

On all cluster nodes (for containerd image pulls):

for node in node1 node2 node3 apiserver; do
  ssh cloud-user@$node "sudo bash -c 'echo \"100.115.2.100 registry.gitlab.cl1.sq4.aegis.internal\" >> /etc/hosts'"
done

Step 2.2 — Update CoreDNS¶

kubectl patch configmap rke2-coredns-rke2-coredns -n kube-system --type merge -p '{
  "data": {
    "Corefile": ".:53 {\n    errors\n    health {\n        lameduck 10s\n    }\n    ready\n    hosts {\n      100.115.2.100 gitlab.cl1.sq4.aegis.internal\n      100.115.2.100 argocd.cl1.sq4.aegis.internal\n      100.115.2.100 vault.cl1.sq4.aegis.internal\n      100.115.2.100 gitlab.cl1.sq4.aegis.internal\n      100.115.2.100 registry.gitlab.cl1.sq4.aegis.internal\n      fallthrough\n    }\n    kubernetes  cluster.local  in-addr.arpa ip6.arpa {\n        pods insecure\n        fallthrough in-addr.arpa ip6.arpa\n        ttl 30\n    }\n    prometheus  0.0.0.0:9153\n    forward  . /etc/resolv.conf\n    cache  30\n    loop\n    reload\n    loadbalance\n}"
  }
}'

kubectl rollout restart deployment/rke2-coredns-rke2-coredns -n kube-system
kubectl rollout status deployment/rke2-coredns-rke2-coredns -n kube-system --timeout=60s

Step 2.3 — Configure containerd to trust the internal CA¶

Why this step is required¶

The GitLab container registry (registry.gitlab.cl1.sq4.aegis.internal) is served over HTTPS with a certificate signed by our private internal CA (internal-ca-issuer). That CA is not in any node's default trust store, so when containerd (the RKE2 container runtime) tries to pull an image from the registry, the TLS handshake fails:

failed to pull and unpack image "registry.gitlab.cl1.sq4.aegis.internal/...":
  failed to resolve reference: failed to do request:
  tls: failed to verify certificate: x509: certificate signed by unknown authority

This is not the same trust used by kubectl, helm, or curl — those use the OS trust store and/or their own flags. Image pulls go through containerd, which has its own per-registry TLS configuration. We must teach containerd (on every node that could schedule a pod pulling from the registry) to trust the internal CA.

Two different trust mechanisms — we configure both, on purpose

Mechanism	File	Who uses it	Why we set it
OS trust store	`/etc/ssl/certs/internal-ca.crt` + `update-ca-certificates`	`curl`, `skopeo`, `git`, any system TLS client on the node	So node-level tooling (e.g. `skopeo copy` during image mirroring in Phase 4) trusts the registry
containerd registry config	`/etc/containerd/certs.d/<registry>/hosts.toml`	containerd image pulls (kubelet → CRI)	So Kubernetes can actually pull images for pods

Adding the CA to the OS store alone is not enough for image pulls — containerd reads its own certs.d host config, not /etc/ssl/certs. This is the #1 reason "the cert is installed but pulls still fail."

Why `certs.d/hosts.toml` and not editing `config.toml`¶

RKE2 manages containerd's main config.toml and regenerates it on every restart — any hand-edit there is wiped. The certs.d/ directory is the supported, drop-in host configuration mechanism (containerd's config_path): RKE2 already points containerd at /etc/containerd/certs.d, and files there survive restarts and upgrades. This is the production-correct, restart-safe way to add per-registry TLS/mirror config.

What each field in `hosts.toml` means¶

server = "https://registry.gitlab.cl1.sq4.aegis.internal"          # the upstream registry this config applies to
[host."https://registry.gitlab.cl1.sq4.aegis.internal"]
  capabilities = ["pull", "resolve", "push"]           # operations allowed against this host
  ca = "/etc/ssl/certs/internal-ca.crt"            # CA bundle to verify the registry's TLS cert

server — the canonical registry endpoint.
capabilities — resolve (look up manifests by tag/digest), pull (download layers), push (upload — needed because nodes also push during the Phase 4 image mirror).
ca — points at the internal CA we just installed; this is what fixes the x509: unknown authority error.

Run it (all nodes)¶

The CA must be installed on every node, because any node may schedule a pod that pulls from the registry. We loop over all worker nodes and the control-plane node:

# Pull the internal CA cert out of the cert-manager-managed secret
CA_CERT=$(kubectl get secret -n cert-manager \
  $(kubectl get clusterissuer internal-ca-issuer -o jsonpath='{.spec.ca.secretName}') \
  -o jsonpath='{.data.tls\.crt}' | base64 -d)

for node in node1 node2 node3 apiserver; do
  echo "=== $node ==="
  # 1) OS trust store (for curl/skopeo/git on the node)
  echo "$CA_CERT" | ssh cloud-user@$node \
    "sudo tee /etc/ssl/certs/internal-ca.crt > /dev/null && sudo update-ca-certificates"
  # 2) containerd per-registry host config (for image pulls/pushes)
  ssh cloud-user@$node "
    sudo mkdir -p /etc/containerd/certs.d/registry.gitlab.cl1.sq4.aegis.internal
    sudo tee /etc/containerd/certs.d/registry.gitlab.cl1.sq4.aegis.internal/hosts.toml > /dev/null << 'EOF2'
server = \"https://registry.gitlab.cl1.sq4.aegis.internal\"
[host.\"https://registry.gitlab.cl1.sq4.aegis.internal\"]
  capabilities = [\"pull\", \"resolve\", \"push\"]
  ca = \"/etc/ssl/certs/internal-ca.crt\"
EOF2
  "
  echo "✅ $node done"
done

No containerd restart needed

containerd reads certs.d/<host>/hosts.toml per request, so the new trust takes effect on the next pull — no systemctl restart (which would briefly disrupt running pods). If you ever do edit the main config.toml, that would require a restart, which is another reason to prefer certs.d.

Verify¶

# From any node — confirm the OS trusts the CA against the live registry endpoint
ssh cloud-user@node1 "curl -sS https://registry.gitlab.cl1.sq4.aegis.internal/v2/ -o /dev/null -w '%{http_code}\n'"
# Expect: 401  (TLS verified OK; 401 = registry reached, auth required — this is success)
#         a TLS error here means the CA is not trusted

# Confirm containerd config is in place on every node
for node in node1 node2 node3 apiserver; do
  ssh cloud-user@$node "test -f /etc/containerd/certs.d/registry.gitlab.cl1.sq4.aegis.internal/hosts.toml && echo \"$node OK\""
done

A 401 is the success signal, not an error

401 Unauthorized from /v2/ means TLS succeeded and you reached the registry's auth gate. A genuine failure looks like curl: (60) SSL certificate problem: unable to get local issuer certificate. Don't chase the 401.

2.4.1 — Pre-flight: confirm the webservice is actually ready¶

The webservice has two containers (webservice + gitlab-workhorse). Rails eager-loads the whole app at boot (~2–3 min), and the ingress returns 502 Bad Gateway until the webservice container passes its readiness probe. Wait for 2/2 before attempting login:

# From bastion — wait for 2/2 Ready
kubectl get pod -n gitlab -l app=webservice -o wide
# READY must show 2/2, STATUS Running

# Confirm the health endpoint through the ingress
curl -sk https://gitlab.cl1.sq4.aegis.internal/-/health
# Expected: GitLab OK   (NOT a 502 page)

If you see 502 Bad Gateway

The webservice container is not Ready yet (1/2), or it is crash-looping. Check:

kubectl get pod -n gitlab -l app=webservice
kubectl logs -n gitlab -l app=webservice -c webservice --tail=30

A repeating restart with exit 137 during "Preloading application" is the boot OOM — see Phase 1 Why we reduced Puma workers and the troubleshooting entry on webservice OOM.

2.4.2 — Retrieve the initial root password¶

The bootstrap password lives in the Kubernetes secret created in Step 1.2:

kubectl get secret gitlab-initial-root-password -n gitlab \
  -o jsonpath='{.data.password}' | base64 -d; echo

2.4.3 — Log in¶

Open https://gitlab.cl1.sq4.aegis.internal (accept the self-signed internal CA cert — or import the internal CA into your browser/keychain to avoid the warning). Log in as:

Username: root
Password: the value from 2.4.2

A fresh GitLab is open by default. Apply these before creating any real projects or tokens. (Admin Area = the wrench//admin icon.)

#	Setting	Where	Why (production rationale)
1	Disable public sign-up	Admin → Settings → General → Sign-up restrictions → uncheck Sign-up enabled → Save	Stops anyone reaching the URL from self-registering. This is the single most important hardening step.
2	Enforce 2FA for all users	Admin → Settings → General → Sign-in restrictions → enable Two-factor authentication (set a grace period)	Credential theft is the #1 Git platform compromise vector.
3	Restrict admin/email domain	Admin → Settings → General → Sign-up restrictions → Allowed domains	Even with sign-up off, scope future invites to your org domain.
4	Set the instance default branch protection	Admin → Settings → Repository → Default branch protection → Fully protected	New repos start with `main` protected (no force-push, no direct push).
5	Disable Gravatar / external avatar fetch	Admin → Settings → General → Account and limit	No outbound calls from an air-gapped environment.

2.4.5 — Rotate the root password into Vault (production mandate)¶

The bootstrap password is a plaintext Kubernetes secret — rotate it and store the new one in Vault

Per the secrets-in-Vault mandate, the bootstrap K8s secret is acceptable only for first login. Immediately:

In the GitLab UI: root → Edit profile → Password → set a new strong password.

Store it in Vault (KV v2), not back into a K8s secret:

# From a host with the vault CLI + token (e.g. bastion)
vault kv put secret/gitlab/root \
  username=root \
  password='<new-strong-password>'

Delete the now-stale bootstrap secret so it can't be used:
```
kubectl delete secret gitlab-initial-root-password -n gitlab
```
The Helm value global.initialRootPassword only seeds the password on first install; deleting the secret afterward does not affect a running GitLab.

DNS is a single-node IP — compromise

gitlab.cl1.sq4.aegis.internal resolves to one node IP (Step 2.1 / 2.2). The ingress controller is a DaemonSet on every node (hostPort 80/443), so any node IP works — but a single A-record is a SPOF. Production-correct: put a LoadBalancer VIP (MetalLB or kube-vip) in front of the ingress and point DNS at the VIP, or use round-robin A-records across all node IPs. Acceptable in this environment because any single node IP reaches the ingress.

Step 2.5 — Create API token and group¶

Why we need an API token¶

The rest of this migration is automated — creating projects, mirroring Git repos (Phase 3), pushing container images and Helm charts (Phases 4–5), and re-pointing ArgoCD (Phase 6). None of that can be done by clicking in the UI at scale; it's driven by curl/git/helm/skopeo calling the GitLab REST API, which requires authentication.

A Personal/Project Access Token is how you authenticate a non-interactive client (a script, CI job, or git push) to GitLab. It is the API equivalent of a password, but:

Scoped — limited to specific permissions (api, read_repository, write_registry…), so a leaked token can't do everything a full login can.
Revocable independently — kill the token without changing the account password.
Auditable — actions are attributable to the named token.

Production: scope tightly and set an expiry — do NOT use a never-expiring api token

The quick-start below uses a broad api scope for convenience during the one-time migration. That is not production-correct. For production:

Practice	Why
Set an expiry (e.g. 7–30 days for a migration token)	A never-expiring token is a permanent liability if leaked. GitLab now requires an expiry by default.
Least-privilege scope	Use `read_repository` + `write_repository` for mirroring, `read_registry`/`write_registry` for images — only add `api` if you actually need project/group management.
Prefer a Group/Project Access Token over a Personal one	Tied to the resource, not to `root`. If root leaves/rotates, automation keeps working; blast radius is one group.
Store the token in Vault, not `~/.bashrc`	`~/.bashrc` is plaintext on disk and leaks into shell history/backups. See the Vault note below.

For this one-time migration a short-lived (e.g. 7-day) api-scoped token is the pragmatic choice; revoke it the moment the migration is verified.

Create it: User icon → Edit profile → Access Tokens → Add new token Name: migration-api, Scopes: api, Expiry: 7 days (revoke after migration).

Why we need the `modgpt` group¶

In GitLab, a group is a namespace + permission boundary that owns projects (repos), container-registry paths, and Helm/package registries. We create the modgpt group as the single home for everything migrated off GitLab, instead of scattering repos under the root user.

Reason	Detail
Stable namespace	All paths become `modgpt/<project>` — repos, `registry.gitlab.cl1.sq4.aegis.internal/modgpt/<image>`, and Helm charts. ArgoCD Applications point at these stable URLs (Phase 6).
RBAC boundary	Permissions are granted at the group level and inherited by every project — you manage access once, not per-repo.
Avoids the `root`-owned anti-pattern	Owning production repos under a personal admin account is fragile (tied to one user) and a security smell. A group is owned by the org, not a person.
Mirrors the GitLab `modgpt` org	Keeps the same path structure so existing references (clone URLs, image refs) translate 1:1 during the cutover.

Group visibility = private

We set visibility: private so projects, images, and charts are not exposed to anonymous/unauthenticated users — correct for an internal platform. (internal and public exist but are wrong for an internal-platform tenancy model.)

Run it¶

# Production-correct: store the token in Vault, not ~/.bashrc
vault kv put secret/gitlab/migration-token token='<paste-token>'

# For the current shell session, export from Vault (or paste once)
export GITLAB_URL="https://gitlab.cl1.sq4.aegis.internal"
export GITLAB_TOKEN="$(vault kv get -field=token secret/gitlab/migration-token)"
export GITLAB_REGISTRY="registry.gitlab.cl1.sq4.aegis.internal"
export GITLAB_ORG="modgpt"

# Create the modgpt group
curl -sk -X POST "${GITLAB_URL}/api/v4/groups" \
  -H "Content-Type: application/json" \
  --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
  -d '{"name":"modgpt","path":"modgpt","visibility":"private"}' \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('Group ID:', d.get('id'), '|', d.get('full_path', d.get('message','')))"

Fallback if you don't have the Vault CLI/token handy

You can export GITLAB_TOKEN='<paste-token>' directly for the migration session. Just don't persist it to ~/.bashrc — let it die with the shell, and revoke the token after Phase 6.

Using PyPI (New Capability)

# Configure pip to use GitLab group-level endpoint (covers all modgpt projects)
pip install \
  --index-url "https://root:${GITLAB_TOKEN}@gitlab.cl1.sq4.aegis.internal/api/v4/groups/modgpt/-/packages/pypi/simple" \
  <package-name>

# Upload a package
pip install twine
cat > ~/.pypirc << PYPIEOF
[distutils]
index-servers = gitlab
[gitlab]
repository = https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PROJECT_ID>/packages/pypi
username = root
password = ${GITLAB_TOKEN}
PYPIEOF

python -m build && twine upload --repository gitlab dist/*

Connect ArgoCD to GitLab (GitOps Credentials)

ArgoCD needs credentials to pull Git manifests and OCI Helm charts from GitLab. This is GitOps setup (needed by any GitLab+ArgoCD install).

Use a DEDICATED read-only token — never the migration token, never plaintext

Not the migration-api token: it's broad (api) and gets revoked after migration → ArgoCD would break. ArgoCD needs a long-lived read-only identity.
Not hand-applied kubectl secrets with the token in plaintext — per the secrets-in-Vault mandate, the token lives in Vault and VSO renders the ArgoCD repo secrets.

Step A — Dedicated read-only Group Access Token¶

GitLab UI: Group modgpt → Settings → Access Tokens → create:

Name: argocd-ro · Role: Reporter · Scopes: read_repository, read_registry · Expiry: 1 year (rotate)

Or via API:

GROUP_ID=$(curl -sk --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLAB_URL/api/v4/groups/$GITLAB_ORG" \
  | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])")
curl -sk -X POST "$GITLAB_URL/api/v4/groups/$GROUP_ID/access_tokens" \
  -H "Content-Type: application/json" --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  -d '{"name":"argocd-ro","scopes":["read_repository","read_registry"],"access_level":20,"expires_at":"2027-05-29"}' \
  | python3 -c "import sys,json;d=json.load(sys.stdin);print('token:', d.get('token', d.get('message')))"

access_level: 20 = Reporter. Copy the returned token (shown once).

Step B — Store the token in Vault¶

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
vault kv put secret/argocd/gitlab-repo token='<argocd-ro-token>'

Step C — VSO auth chain in the `argocd` namespace (least-privilege)¶

Same per-namespace pattern as GitLab (Step 1.6b), scoped to secret/argocd/*:

# Vault policy + role
vault policy write argocd-ro - <<'POL'
path "secret/data/argocd/*"     { capabilities = ["read"] }
path "secret/metadata/argocd/*" { capabilities = ["read","list"] }
POL
vault write auth/kubernetes/role/argocd \
  bound_service_account_names=argocd-vso \
  bound_service_account_namespaces=argocd \
  policies=argocd-ro ttl=1h

# SA + VaultConnection + VaultAuth in argocd ns
kubectl create serviceaccount argocd-vso -n argocd
kubectl apply -f - <<'YAML'
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata: {name: vault, namespace: argocd}
spec: {address: https://vault.cl1.sq4.aegis.internal, skipTLSVerify: true}
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata: {name: vault, namespace: argocd}
spec:
  method: kubernetes
  mount: kubernetes
  vaultConnectionRef: vault
  kubernetes: {role: argocd, serviceAccount: argocd-vso, audiences: ["vault"]}
YAML

Step D — VSO renders the two ArgoCD repo secrets¶

kubectl apply -f - <<'YAML'
# Git manifests — repo-creds template (auth for any repo under the modgpt URL prefix)
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: {name: gitlab-git-creds, namespace: argocd}
spec:
  type: kv-v2
  mount: secret
  path: argocd/gitlab-repo
  vaultAuthRef: vault
  refreshAfter: 1h
  destination:
    name: gitlab-git-creds
    create: true
    labels:
      argocd.argoproj.io/secret-type: repo-creds
    transformation:
      excludeRaw: true
      templates:
        type:     {text: "git"}
        url:      {text: "http://gitlab-webservice-default.gitlab.svc.cluster.local:8080/modgpt/"}
        username: {text: "argocd-ro"}
        password: {text: "{{ .Secrets.token }}"}
---
# Helm OCI charts — repository (NOTE: url is hostname only + enableOCI; see Issue 20)
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: {name: gitlab-helm-repo, namespace: argocd}
spec:
  type: kv-v2
  mount: secret
  path: argocd/gitlab-repo
  vaultAuthRef: vault
  refreshAfter: 1h
  destination:
    name: gitlab-helm-repo
    create: true
    labels:
      argocd.argoproj.io/secret-type: repository
    transformation:
      excludeRaw: true
      templates:
        type:      {text: "helm"}
        name:      {text: "gitlab-helm-oci"}
        url:       {text: "registry.gitlab.cl1.sq4.aegis.internal"}
        enableOCI: {text: "true"}
        username:  {text: "argocd-ro"}
        password:  {text: "{{ .Secrets.token }}"}
YAML

OCI repo url format (troubleshooting Issue 20)

The Helm OCI repository url must be registry.gitlab.cl1.sq4.aegis.internal — hostname only, no oci:// scheme, no path — plus enableOCI: "true". oci://registry.gitlab.cl1.sq4.aegis.internal triggers OCI Helm repository URL should include hostname and port only.

Step D2 — Trust the internal CA in ArgoCD (required for HTTPS connectivity)¶

ArgoCD OCI → Use GitLab HTTP Helm registry instead

ArgoCD v3.4.2 has a double URL-encoding bug for OCI chart paths with multiple / levels (e.g. modgpt/charts/reloader). The JWT auth scope gets encoded as repository%253Amodgpt%252Fcharts%252Freloader%253Apull instead of repository%3Amodgpt%2Fcharts%2Freloader%3Apull — GitLab's strict registry spec rejects the double-encoded scope with 403. Use the GitLab HTTP Helm package registry (Step D3 below) for ArgoCD instead of OCI. The OCI pushes from Phase 5 remain valid for direct helm pull use.

ArgoCD's repo-server verifies TLS via the argocd-tls-certs-cm ConfigMap (keyed by hostname). This covers HTTP Helm repos and HTTPS git — register the internal CA under each GitLab host:

CA=$(kubectl get secret internal-ca-issuer-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d)
for host in gitlab.cl1.sq4.aegis.internal registry.gitlab.cl1.sq4.aegis.internal; do
  kubectl patch configmap argocd-tls-certs-cm -n argocd --type merge \
    --patch "$(printf 'data:\n  %s: |\n%s\n' "$host" "$(echo "$CA" | sed 's/^/    /')")"
done
kubectl rollout restart deploy/argocd-repo-server -n argocd
kubectl rollout status  deploy/argocd-repo-server -n argocd --timeout=120s

OCI-only: OS trust store injection (NOT needed for the HTTP path)

argocd-tls-certs-cm is sufficient for HTTP Helm + HTTPS git (proven: an HTTP Helm repo works with the CM alone). It is not enough for OCI helm registry login — that subprocess reads the OS system trust store, which ArgoCD v3.x doesn't populate. Since we use the HTTP registry, you do not need the OS-trust-store initContainer. If you ever switch ArgoCD to OCI, see the ArgoCD OCI Helm x509 troubleshooting note for the initContainer procedure.

Step E — Verify¶

kubectl get vaultstaticsecret -n argocd
kubectl get secret gitlab-git-creds gitlab-helm-repo -n argocd \
  -o jsonpath='{range .items[*]}{.metadata.name}{"  managed-by="}{.metadata.labels.app\.kubernetes\.io/managed-by}{"\n"}{end}'

argocd repo list --grpc-web
# Expect: gitlab-helm-http  Successful  (HTTP package registry)

If repo list shows Failed but apps still sync

The repo connectivity status can be a stale cache. The authoritative test is app manifest generation. If helm repo update works from inside the repo-server pod, ArgoCD can pull — restart argocd-repo-server to clear cached credentials.

Step D3 — Use GitLab HTTP Helm package registry for ArgoCD (replaces OCI for ArgoCD)¶

Due to the ArgoCD v3.4.2 OCI double-encoding bug, register the GitLab HTTP Helm package registry for ArgoCD instead.

Step D3a — Upload charts to GitLab HTTP Helm registry:

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
GITLAB_URL="https://gitlab.cl1.sq4.aegis.internal"
GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)
PROJECT_ID=$(curl -sk --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "$GITLAB_URL/api/v4/projects/modgpt%2Fcluster-addons" \
  | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])")

# Upload each chart via the GitLab Helm package API
for f in ~/.cache/helm/repository/cert-manager-v1.20.2.tgz \
          ~/.cache/helm/repository/argo-cd-9.5.15.tgz \
          ~/.cache/helm/repository/reloader-2.2.12.tgz \
          ~/.cache/helm/repository/vault-0.32.0.tgz \
          ~/.cache/helm/repository/vault-secrets-operator-0.9.1.tgz; do
  echo "=== $(basename $f) ==="
  curl -sk --request POST "$GITLAB_URL/api/v4/projects/$PROJECT_ID/packages/helm/api/stable/charts" \
    --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
    --form "chart=@${f}" 2>&1 | python3 -c "import sys,json;d=json.load(sys.stdin);print(d)" || echo "done"
done

Step D3b — Register in ArgoCD (VSO-managed, same pattern as GitLab):

Update the gitlab-helm-repo VaultStaticSecret to point at the HTTP registry:

# VaultStaticSecret spec → destination templates
type:     {text: "helm"}
name:     {text: "gitlab-helm-http"}
url:      {text: "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PROJECT_ID>/packages/helm/stable"}
username: {text: "argocd-ro"}
password: {text: "{{ .Secrets.token }}"}

Step D3c — Update Application sources:

Change all chart Application sources from OCI to HTTP:

# Before (OCI — broken)
repoURL: 'oci://registry.gitlab.cl1.sq4.aegis.internal/modgpt/charts/reloader'
chart: reloader
targetRevision: 2.2.12

# After (HTTP — working)
repoURL: 'https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PROJECT_ID>/packages/helm/stable'
chart: reloader
targetRevision: 2.2.12

Common Issues

Symptom	Cause	Fix
Pods stuck `Pending` — `0/7 nodes available: 4 Insufficient memory`	Worker nodes are full — GitLab + Rook-Ceph CSI occupy 96-98% of memory requests on each node	Scale GitLab down to 0 (Step 1.7) to free ~2.2 GiB; GitLab pods schedule automatically once memory is available
`additional properties 'install' not allowed` on `helm install`	Known Helm bug: cert-manager sub-chart `values.schema.json` validates even when sub-chart is disabled via condition	Remove cert-manager sub-chart directory from local chart AND patch `Chart.yaml`/`Chart.lock` (Step 1.8). `--disable-openapi-validation` is NOT sufficient — it only skips Kubernetes API validation, not Helm schema validation
`found in Chart.yaml, but missing in charts/ directory: cert-manager`	`Chart.yaml` still references cert-manager after directory deletion	Patch `Chart.yaml` and `Chart.lock` to remove cert-manager from `dependencies:` list (Step 1.8 python3 script)
`cannot re-use a name that is still in use`	Previous failed/cancelled install left release in failed state	`helm uninstall gitlab -n gitlab` then re-install
`minio.enabled` REMOVALS error	`minio: enabled: false` at chart root is removed in v9 — moved to `global.minio.enabled: false`	Remove the top-level `minio:` block; use only `global.minio.enabled: false`
`When consolidated object storage is enabled, connection must be empty`	Individual `connection:` blocks under each bucket conflict with consolidated mode	Remove `connection:` from each bucket item; define it once at `object_store:` level only
`helm install` timeout with pods still Pending	Not enough memory (see first row)	Scale GitLab down first; if resources are fine, increase `--timeout 20m`
webservice `OOMKilled` (exit 137) during boot, crashloops forever	GitLab eager-loads the whole Rails app at startup, spiking RSS to ~2.5Gi. With 2 Puma workers (default) and/or a 2.5Gi limit, the preload spike hits the cgroup limit → OOM on every boot. `DISABLE_PUMA_WORKER_KILLER=true` (chart default) means nothing recycles the worker.	Set `gitlab.webservice.workerProcesses: 1` (halves RSS) and raise `limits.memory` to `3Gi`. It survives the preload spike then settles ~1.6Gi. Nodes have physical headroom even though requests show 90%+ (check `kubectl top nodes` for real usage).
sidekiq `OOMKilled` during boot	Same Rails eager-load spike as webservice	Raise `gitlab.sidekiq.resources.limits.memory` to `3Gi`
sidekiq scales to 4+ replicas, 3 stuck `Pending` (`Insufficient memory`)	The sidekiq HPA defaults to `max=10`; a CPU burst during initial job processing scales it out, but the memory-overcommitted cluster can't place the surge pods	Pin the HPA: set `gitlab.sidekiq.minReplicas: 1` and `maxReplicas: 1` in values, then `helm upgrade`
New webservice/sidekiq pod stuck `Pending` on `helm upgrade` while old pod still `Running`/crashlooping	Rolling-update surge pod can't schedule because memory requests are overcommitted (90%+) and the old pod still holds its slot	Delete the old pod to free its request: `kubectl delete pod -n gitlab <old-pod>` — the surge pod schedules immediately
Whole install `failed` after running during cluster infra issues (control-plane/CNI/CSI churn)	The Helm hooks (shared-secrets, migrations) ran against a degraded cluster and left partial state	Clean reinstall: `helm uninstall gitlab -n gitlab`; delete the 3 PVCs (`data-gitlab-postgresql-0`, `redis-data-gitlab-redis-master-0`, `repo-data-gitlab-gitaly-0`); KEEP the prereq secrets (`gitlab-initial-root-password`, `gitlab-s3-connection`, `gitlab-registry-storage`, `gitlab-tls`); re-run `helm install`. Fresh DB → clean migrations
Registry push fails with `500`	Ceph RGW bucket doesn't exist	GitLab creates buckets on first use — check `kubectl logs -n gitlab -l app=gitlab-sidekiq`
`cert verify failed` on image push	Node doesn't trust internal CA	Re-run Phase 2 Step 2.3 on affected node
ArgoCD OCI shows `Failed` in `repo list`	Known ArgoCD v3 display bug	Ignore — verify with `argocd app list` and pod status
`helm push` 401 Unauthorized	Helm not logged in to registry	Re-run `helm registry login`
S3 connection error in sidekiq logs	Wrong Ceph RGW endpoint	Verify: `kubectl exec -n gitlab deploy/gitlab-sidekiq -- curl -s http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local`
`Init:Error` on webservice — "Database has not been initialized yet, 1381 pending migrations"	The migrations Job timed out (pods were Pending too long) or the hook didn't fire	Delete failed migration job, run `gitlab-rake db:migrate` from toolbox (Step 1.9), then delete webservice pods to restart them
`registry` pod CrashLoopBackOff — `s3aws: NoSuchBucket`	GitLab registry does NOT auto-create its S3 bucket at startup	Pre-create all buckets via toolbox's `aws` CLI (Step 1.9) before starting registry
sidekiq rolling update deadlock — old RS keeps creating 512Mi pod, new RS can't start	Old ReplicaSet recreates a pod matching the old request every time you delete it	Scale sidekiq Deployment to 0 (`kubectl scale deployment gitlab-sidekiq-all-in-1-v2 -n gitlab --replicas=0`), then let `helm upgrade` take over
webservice still Pending after reducing requests	Workhorse container's requests need to be set EXPLICITLY — otherwise the chart default applies	Set `gitlab.webservice.workhorse.resources.requests.memory: 64Mi` in values, otherwise the scheduler sees a higher total pod request
GitLab migrations job stuck	DB not ready	`kubectl delete job -n gitlab -l app=migrations` — Helm re-creates it