GitLab — In-Cluster Installation on RKE2¶

Standalone runbook to install GitLab inside the RKE2 cluster via the official gitlab/gitlab Helm chart, with all state on Ceph. This page is install-only — for post-install setup (groups, projects, access tokens, DNS/registry trust, ArgoCD GitOps credentials, PyPI), see GitLab — Deployment & Setup.

Prerequisites¶

Confirm every item below before starting — the install assumes they already exist.

#	Prerequisite	Check
1	RKE2 cluster running; `kubectl` configured against it	`kubectl get nodes`
2	NGINX ingress controller present (RKE2 built-in) with the ingress VIP `100.115.2.100`	`kubectl get svc -A \\| grep ingress`
3	cert-manager deployed with the internal-CA `ClusterIssuer` `internal-ca-issuer`	`kubectl get clusterissuer internal-ca-issuer`
4	Rook-Ceph — the `ceph-block` StorageClass (RWO) and the Ceph object store (RGW / S3) are available	`kubectl get sc ceph-block`; `kubectl -n rook-ceph get cephobjectstore`
5	HashiCorp Vault + Vault Secrets Operator (VSO) — for rendering the S3 secrets declaratively (bootstrap can use plain Secrets first)	`kubectl get pods -n vault`; `kubectl get vaultstaticsecret -A`
6	DNS — `gitlab.cl1.sq4.aegis.internal` and `registry.gitlab.cl1.sq4.aegis.internal` resolve to the ingress VIP	`dig +short gitlab.cl1.sq4.aegis.internal`
7	GitLab Helm chart `9.11.5` mirrored to Harbor OCI (`helm-charts/gitlab`) — see Step 1.1	`helm show chart oci://harbor.cl1.sq4.aegis.internal/helm-charts/gitlab --version 9.11.5`
8	Node memory headroom — GitLab requests ~2.2 GiB across worker nodes	`kubectl top nodes`

What gets deployed

The gitlab/gitlab chart into namespace gitlab, with the bundled cert-manager sub-chart disabled (the cluster's cert-manager issues TLS via the internal-ca-issuer ClusterIssuer) and the bundled MinIO disabled (object storage goes to Ceph RGW). Endpoints: https://gitlab.cl1.sq4.aegis.internal (Git + Helm registry) and https://registry.gitlab.cl1.sq4.aegis.internal (container registry).

Architecture¶

Design: GitLab on Kubernetes with Ceph Object Storage¶

GitLab is deployed inside the cluster using the official gitlab/gitlab Helm chart. It uses Ceph for all persistence:

Ceph Block (ceph-block) — Gitaly (Git repos), PostgreSQL, Redis
Ceph Object (RGW, S3-compatible) — uploads, LFS, artifacts, packages, registry blobs

The bundled MinIO that the chart ships is disabled — Ceph's RADOS Gateway already provides the same S3-compatible API natively, with 3× replication built in. This removes one layer of indirection.

Developer workstation
       │  https://gitlab.cl1.sq4.aegis.internal
       │  https://registry.gitlab.cl1.sq4.aegis.internal
       ▼
100.115.2.100 (nginx ingress — node1)
       │  → gitlab-webservice.gitlab.svc.cluster.local:8080
       │  → gitlab-registry.gitlab.svc.cluster.local:5000
       ▼
Kubernetes cluster (namespace: gitlab)
  ├─ gitlab-webservice   (web UI + Git API)         PDB: minAvailable 1
  ├─ gitlab-workhorse    (sidecar: Git proxy)
  ├─ gitlab-sidekiq      (background jobs)           PDB: minAvailable 1
  ├─ gitlab-gitaly       (Git data storage)          PVC: ceph-block 20Gi
  ├─ gitlab-registry     (Container + Helm OCI)      → object storage via S3
  ├─ gitlab-postgresql   (relational DB)             PVC: ceph-block 10Gi
  └─ gitlab-redis        (cache / sessions)          PVC: ceph-block 2Gi

Object storage (no MinIO pod — direct to Ceph RGW)
  └─ rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80
       ├─ gitlab-artifacts      (CI/CD artifacts)
       ├─ gitlab-lfs            (Git LFS blobs)
       ├─ gitlab-uploads        (user uploads / attachments)
       ├─ gitlab-packages       (Package registry: PyPI, Helm, etc.)
       ├─ gitlab-registry       (Container registry blobs)
       └─ gitlab-backups        (GitLab backups)

Known deviations from production¶

Deviation	Reason	Production-correct
Single webservice replica	Resource constraints	2+ replicas with HPA
Bundled PostgreSQL	Avoids cross-namespace dependency	External PostgreSQL HA cluster
Bundled Redis	Avoids cross-namespace dependency	External Redis Sentinel/Cluster
No GitLab Runner	Not needed for registry-only use	Kubernetes executor, auto-scaling

Bootstrap exception¶

GitLab images pulled from public registry on first deploy

GitLab cannot be bootstrapped from itself — the same pattern as ArgoCD (bootstrapped manually before it could manage itself). On first install, helm install pulls images from registry.gitlab.com. After GitLab is running, all future workloads use registry.gitlab.cl1.sq4.aegis.internal.

Installation¶

Step 1.1 — Mirror the GitLab chart into Harbor (air-gap)¶

charts.gitlab.io is not reachable from the cluster — the chart must first be mirrored into Harbor's OCI helm-charts project. This is a one-time step run from the bastion (which has the HTTP proxy 10.96.137.37:3128 to reach upstream). The chart and pinned version are already declared in scripts/harbor-mirror/helm-charts/mirror-charts.yaml (gitlab → 9.11.5):

# On the bastion — mirror the GitLab chart into Harbor OCI (helm pull via proxy → helm push)
cd scripts/harbor-mirror/helm-charts
./core42-mirror-charts.py --only gitlab --dry-run   # preview
./core42-mirror-charts.py --only gitlab             # push to Harbor

Result: the chart lands at oci://harbor.cl1.sq4.aegis.internal/helm-charts/gitlab:9.11.5. Log in to Harbor OCI and confirm (the deploy steps below pull from here, not from the internet):

helm registry login harbor.cl1.sq4.aegis.internal   # robot$ask creds (~/harbor-images/harbor-robot.json)
helm show chart oci://harbor.cl1.sq4.aegis.internal/helm-charts/gitlab --version 9.11.5 | head

Chart version

Chart 9.11.5 = GitLab v18.11.4. To bump, edit the gitlab entry in mirror-charts.yaml and re-run the mirror. Avoid .0 releases of a new major (e.g. v19.0.x) in production.

Step 1.2 — Create namespace and root password secret¶

kubectl create namespace gitlab

# Generate password
GITLAB_ROOT_PASS=$(openssl rand -base64 24 | tr -d '/+=')
echo "Root password: ${GITLAB_ROOT_PASS}"

# Kubernetes secret (GitLab reads this once at first start)
kubectl create secret generic gitlab-initial-root-password \
  -n gitlab \
  --from-literal=password="${GITLAB_ROOT_PASS}"

# Vault (source of truth)
vault kv put secret/gitlab/root-password \
  username="root" \
  password="${GITLAB_ROOT_PASS}"

Step 1.3 — Create TLS certificate¶

Two hostnames only — MinIO is disabled so no MinIO ingress is created:

kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gitlab-tls
  namespace: gitlab
spec:
  secretName: gitlab-tls
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer
  dnsNames:
    - gitlab.cl1.sq4.aegis.internal
    - registry.gitlab.cl1.sq4.aegis.internal
EOF

kubectl get certificate gitlab-tls -n gitlab --watch
# Wait for READY=True, then Ctrl+C

Step 1.3b — (Upgrade) Enable cert-manager auto-provisioning on GitLab ingresses¶

What changed and why

Previously, global.ingress.configureCertmanager: false and a manual Certificate resource (Step 1.3) pre-created the gitlab-tls secret. That works but requires human intervention on every renewal and on new hostnames.

The production-correct path is to add cert-manager.io/cluster-issuer: internal-ca-issuer to global.ingress.annotations. cert-manager then watches all GitLab ingresses and automatically issues and renews TLS certs — the manual Certificate in Step 1.3 is no longer needed.

Why we keep `configureCertmanager: false`¶

Flag	Our value	Why
`global.certmanager.install`	`false`	Never install the bundled cert-manager (Helm sub-chart) — cluster already has the singleton
`global.ingress.configureCertmanager`	`false`	When `true`, the chart creates a namespace-scoped `Issuer` — wrong when we have a `ClusterIssuer`. We use the annotation path instead.
`cert-manager.io/cluster-issuer: internal-ca-issuer`	annotation	Tells cert-manager to use the cluster-wide `ClusterIssuer`. cert-manager sees the ingress TLS block + this annotation → creates + renews the cert automatically.

If GitLab is already running — apply the change via `helm upgrade`¶

1. Edit the values file on the bastion:

# Add the cert-manager annotation to global.ingress.annotations in ~/gitlab-stg.yaml
# The block should look like this:
#   global:
#     ingress:
#       configureCertmanager: false
#       ...
#       annotations:
#         cert-manager.io/cluster-issuer: internal-ca-issuer   # ← add this line
#         nginx.ingress.kubernetes.io/proxy-body-size: "0"
#         ...

2. Delete the manually-created Certificate (cert-manager will re-own it):

# This deletes the Certificate CRD object and the gitlab-tls Secret.
# cert-manager will immediately re-issue it via the ingress annotation.
kubectl delete certificate gitlab-tls -n gitlab
kubectl delete secret gitlab-tls -n gitlab 2>/dev/null || true

3. Apply via helm upgrade:

helm upgrade gitlab /tmp/gitlab-chart/gitlab \
  -n gitlab \
  -f ~/gitlab-stg.yaml \
  --set certmanager.install=false \
  --timeout 10m \
  --wait

4. Verify cert-manager issued the cert automatically:

# cert-manager creates a Certificate for each ingress TLS block with the annotation
kubectl get certificate -n gitlab
# Expected: gitlab-tls   True   <hostname>   ...   (READY=True within ~30 seconds)

kubectl describe certificate gitlab-tls -n gitlab | grep -A5 "Conditions:"
# Expected: Type: Ready, Status: True, Message: Certificate is up to date and has not expired

# Confirm the Secret was issued (not from the deleted manual cert)
kubectl get secret gitlab-tls -n gitlab -o jsonpath='{.metadata.annotations}' | python3 -m json.tool
# Expected: cert-manager.io/issuer-name: internal-ca-issuer (or cluster-issuer equivalent)

5. Confirm GitLab TLS still works:

curl -sk https://gitlab.cl1.sq4.aegis.internal/-/health
# Expected: GitLab OK

If cert issuance takes > 60 seconds

Check kubectl describe certificaterequest -n gitlab and kubectl describe order -n gitlab (for ACME issuers; not applicable here since internal-ca-issuer is a CA issuer). For a CA issuer, cert issuance is synchronous and nearly instant. A long delay usually means the ClusterIssuer name is wrong or the issuer is not Ready.

kubectl get clusterissuer internal-ca-issuer
# Expected: READY=True

Step 1.4 — Create Ceph object store user for GitLab¶

GitLab needs an S3 user on the existing ceph-objectstore. Rook provisions the credentials automatically:

kubectl apply -f - << 'EOF'
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: gitlab
  namespace: rook-ceph
spec:
  store: ceph-objectstore
  displayName: "GitLab Object Storage"
  capabilities:
    user: "*"
    bucket: "*"
    metadata: "*"
    usage: "*"
EOF

# Wait for Rook to create the credentials secret (~15 seconds)
kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph --watch
# Wait for secret to appear, then Ctrl+C

Step 1.5 — Extract credentials and store in Vault¶

S3_ACCESS_KEY=$(kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph -o jsonpath='{.data.AccessKey}' | base64 -d)
S3_SECRET_KEY=$(kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph -o jsonpath='{.data.SecretKey}' | base64 -d)
S3_ENDPOINT="http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local"

# Store in Vault
vault kv put secret/gitlab/ceph-s3 \
  access_key="${S3_ACCESS_KEY}" \
  secret_key="${S3_SECRET_KEY}" \
  endpoint="${S3_ENDPOINT}"

vault kv get secret/gitlab/ceph-s3

Step 1.6 — Create Kubernetes secrets for GitLab¶

GitLab needs two secrets — one for general object storage (uploads/LFS/artifacts/packages), one for the container registry blobs.

These reuse $S3_ACCESS_KEY / $S3_SECRET_KEY from Step 1.5 — re-hydrate if you're in a new shell

The credentials are not generated here — they come from the Rook-created secret (Step 1.4) via the env vars set in Step 1.5. Run Steps 1.4 → 1.5 → 1.6 in the same shell. If you opened a fresh shell (vars empty), the secrets would be created with blank keys. Re-hydrate from Vault first:

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
S3_ACCESS_KEY=$(vault kv get -field=access_key secret/gitlab/ceph-s3)
S3_SECRET_KEY=$(vault kv get -field=secret_key secret/gitlab/ceph-s3)
# sanity: both must be non-empty
echo "access=[${S3_ACCESS_KEY:?empty — fix before continuing}] secret-len=${#S3_SECRET_KEY}"

# S3 connection for uploads, LFS, artifacts, packages
kubectl create secret generic gitlab-s3-connection \
  -n gitlab \
  --from-literal=connection="$(cat << CONNEOF
provider: AWS
region: us-east-1
aws_access_key_id: ${S3_ACCESS_KEY}
aws_secret_access_key: ${S3_SECRET_KEY}
aws_signature_version: 4
host: rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local
endpoint: http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local
path_style: true
CONNEOF
)"

# S3 config for GitLab container registry blobs
kubectl create secret generic gitlab-registry-storage \
  -n gitlab \
  --from-literal=config="$(cat << REGEOF
s3:
  bucket: gitlab-registry
  accesskey: ${S3_ACCESS_KEY}
  secretkey: ${S3_SECRET_KEY}
  regionendpoint: http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local
  region: us-east-1
  pathstyle: true
  secure: false
REGEOF
)"

kubectl get secret gitlab-s3-connection gitlab-registry-storage -n gitlab

The kubectl create secret above is the bootstrap path — production uses VSO

Those two commands write plaintext S3 credentials into Kubernetes Secrets by hand. That is acceptable only to get GitLab running the first time. Per the secrets-in-Vault mandate, the production-correct approach is to keep the credentials solely in Vault (secret/gitlab/ceph-s3) and have the Vault Secrets Operator (VSO) render these two Kubernetes Secrets from Vault — single source of truth, automatic rotation, no plaintext drift. Step 1.6b converts them.

Step 1.6b — (Production) Render the secrets from Vault via VSO¶

Vault already holds the Ceph S3 creds at secret/gitlab/ceph-s3 (fields: access_key, secret_key, endpoint — written in Step 1.5). We make VSO materialize the exact same gitlab-s3-connection and gitlab-registry-storage Secrets from those fields.

1) Vault side — least-privilege policy + a gitlab k8s-auth role:

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal

# Policy: read ONLY secret/gitlab/* (not all of secret/*)
vault policy write gitlab-ro - <<'POL'
path "secret/data/gitlab/*"     { capabilities = ["read"] }
path "secret/metadata/gitlab/*" { capabilities = ["read","list"] }
POL

# k8s auth role bound to a dedicated SA in the gitlab namespace
vault write auth/kubernetes/role/gitlab \
  bound_service_account_names=gitlab-vso \
  bound_service_account_namespaces=gitlab \
  policies=gitlab-ro ttl=1h

2) Kubernetes side — dedicated SA + a gitlab-namespace VaultConnection/VaultAuth:

kubectl create serviceaccount gitlab-vso -n gitlab

kubectl apply -f - <<'YAML'
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata: {name: vault, namespace: gitlab}
spec:
  address: https://vault.cl1.sq4.aegis.internal
  skipTLSVerify: true
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata: {name: vault, namespace: gitlab}
spec:
  method: kubernetes
  mount: kubernetes
  vaultConnectionRef: vault
  kubernetes:
    role: gitlab
    serviceAccount: gitlab-vso
    audiences: ["vault"]
YAML

Why a per-namespace VaultAuth + SA (not the operator's default)

VSO resolves a VaultAuth's serviceAccount in the namespace of the secret that uses it — so referencing the operator's vault-secrets-operator/default VaultAuth from gitlab fails with ServiceAccount "vso-...-controller-manager" not found. Each consuming namespace needs its own VaultAuth + ServiceAccount, bound to a Vault role scoped to just that namespace's secrets. This is also the least-privilege production pattern — gitlab-ro can read only secret/gitlab/*.

3) Two VaultStaticSecrets that template the exact Secret bodies (overwrite: true adopts the bootstrap Secrets in place):

kubectl apply -f - <<'YAML'
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: {name: gitlab-s3-connection, namespace: gitlab}
spec:
  type: kv-v2
  mount: secret
  path: gitlab/ceph-s3
  vaultAuthRef: vault
  refreshAfter: 1h
  destination:
    name: gitlab-s3-connection
    create: true
    overwrite: true               # adopt the existing hand-created Secret
    transformation:
      excludeRaw: true            # write ONLY the templated key, not raw vault fields
      templates:
        connection:
          text: |-
            provider: AWS
            region: us-east-1
            aws_access_key_id: {{ .Secrets.access_key }}
            aws_secret_access_key: {{ .Secrets.secret_key }}
            aws_signature_version: 4
            host: {{ .Secrets.endpoint | trimPrefix "http://" | trimPrefix "https://" }}
            endpoint: {{ .Secrets.endpoint }}
            path_style: true
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: {name: gitlab-registry-storage, namespace: gitlab}
spec:
  type: kv-v2
  mount: secret
  path: gitlab/ceph-s3
  vaultAuthRef: vault
  refreshAfter: 1h
  destination:
    name: gitlab-registry-storage
    create: true
    overwrite: true
    transformation:
      excludeRaw: true
      templates:
        config:
          text: |-
            s3:
              bucket: gitlab-registry
              accesskey: {{ .Secrets.access_key }}
              secretkey: {{ .Secrets.secret_key }}
              regionendpoint: {{ .Secrets.endpoint }}
              region: us-east-1
              pathstyle: true
              secure: false
YAML

4) Verify (the Secrets become VSO-owned, content unchanged → no GitLab restart needed):

kubectl get vaultstaticsecret -n gitlab
kubectl get secret gitlab-s3-connection gitlab-registry-storage -n gitlab \
  -o jsonpath='{range .items[*]}{.metadata.name}{"  managed-by="}{.metadata.labels.app\.kubernetes\.io/managed-by}{"\n"}{end}'
# Expect: managed-by=hashicorp-vso  for both

Zero-downtime cutover

Because the templated output is byte-identical to the bootstrap Secrets, adopting them with overwrite: true does not change the data GitLab reads — pods keep running. Validate the template first by rendering to throwaway *-vso destination names and diff-ing against the live Secrets before pointing VSO at the real names. This VaultAuth + SA + VaultStaticSecret trio is the reusable pattern for every app namespace that needs Vault secrets.

Which GitLab secret belongs to which manager (decision matrix)¶

Not every secret should be a VaultStaticSecret. Pick the manager by where the source of truth lives and what lifecycle the secret has:

Secret	Type	Source of truth	Manager	Why
`gitlab-s3-connection`	Opaque	Vault `secret/gitlab/ceph-s3`	VSO `VaultStaticSecret`	Runtime credential, Vault-sourced → declarative + rotatable
`gitlab-registry-storage`	Opaque	Vault `secret/gitlab/ceph-s3`	VSO `VaultStaticSecret`	Same — registry blob-store creds
`gitlab-tls`	`kubernetes.io/tls`	cert-manager `Certificate` (`internal-ca-issuer`)	cert-manager	Certs need CSR + issuance + auto-renewal — that's cert-manager's job, not Vault's. A VSS here would never renew.
`gitlab-initial-root-password`	Opaque	Vault `secret/gitlab/root-password`	Bootstrap only — leave as-is	Read by the chart only at first install; slated for deletion after first-login rotation (Phase 2 · Step 2.4.5). VSO-managing it would fight that retirement.

Rule of thumb:

✅ VSO — secrets whose source of truth is Vault and that are read at runtime (the two S3 secrets).
🔒 cert-manager — anything kubernetes.io/tls. Let the cert controller own issuance + renewal.
🧹 Leave alone — bootstrap-only secrets (seed-once, then retired). Keep the canonical copy in Vault as the record; don't have an operator keep re-creating a secret you intend to delete.

Step 1.7 — Resource planning: ensure adequate node memory¶

Node memory (in-cluster method only)

The in-cluster GitLab release requests ~2.2 GiB of memory (PostgreSQL HA ×3, pgpool ×2, GitLab pods ×2, Valkey ×3). Ensure the target worker nodes have adequate free memory before installing; the production CPU workers (768 GB–1 TB RAM) have ample headroom. (On the production VM-based GitLab this does not apply.)

Verify memory is freed:

kubectl describe node node1 | grep -A5 'Allocated resources:' | grep memory
kubectl describe node node2 | grep -A5 'Allocated resources:' | grep memory
kubectl describe node node3 | grep -A5 'Allocated resources:' | grep memory
# Expected: memory usage drops from 98% to ~30-40%

Step 1.7b — Write the values file¶

cat > ~/gitlab-stg.yaml << 'EOF'
global:
  hosts:
    domain: cl1.sq4.aegis.internal
    https: true
    gitlab:
      name: gitlab.cl1.sq4.aegis.internal
    registry:
      name: registry.gitlab.cl1.sq4.aegis.internal
  ingress:
    configureCertmanager: false   # keep false — we use ClusterIssuer, not a namespace Issuer
    provider: nginx
    class: nginx
    tls:
      enabled: true
      secretName: gitlab-tls      # cert-manager writes the cert into this named secret
    annotations:
      # cert-manager auto-provisions + renews the cert via our cluster-wide ClusterIssuer
      cert-manager.io/cluster-issuer: internal-ca-issuer
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "300"

  certmanager:
    install: false                # never install the bundled cert-manager — use the cluster singleton
  certmanager-issuer:
    email: admin@cl1.sq4.aegis.internal

  initialRootPassword:
    secret: gitlab-initial-root-password
    key: password

  # ── Object storage: Ceph RGW (no MinIO) ────────────────────────────────────
  minio:
    enabled: false

  appConfig:
    object_store:
      enabled: true
      proxy_download: true
      connection:
        secret: gitlab-s3-connection
        key: connection

    # NOTE: when object_store.enabled=true, individual items must NOT have connection:
    # The connection is inherited from object_store.connection above.
    artifacts:
      enabled: true
      bucket: gitlab-artifacts
    lfs:
      enabled: true
      bucket: gitlab-lfs
    uploads:
      enabled: true
      bucket: gitlab-uploads
    packages:
      enabled: true
      bucket: gitlab-packages
    backups:
      bucket: gitlab-backups
      tmpBucket: gitlab-backups-tmp

# ── Disable what we already have ────────────────────────────────────────────────
# NOTE: certmanager and minio are disabled via global.* — no top-level blocks needed
nginx-ingress:
  enabled: false

prometheus:
  install: false

gitlab-runner:
  install: false

# ── Registry — blobs stored in Ceph via S3 ──────────────────────────────────────
registry:
  enabled: true
  storage:
    secret: gitlab-registry-storage
    key: config
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: "1"
      memory: 512Mi

# ── Scale down to free node memory (only if nodes are memory-constrained) ──────
# NOTE: every "# PROD:" comment is the production-recommended value. The bare value
# is a compromise forced by RAM-constrained pre-production nodes. Revert to PROD once nodes
# are rebalanced (see troubleshooting → Host Memory Overcommit).
gitlab:
  webservice:
    replicaCount: 1      # PROD: 3+  (HA; survive a node loss) with podAntiAffinity + a PDB (minAvailable: 2)
    minReplicas: 1       # PROD: 3   (HPA floor)
    maxReplicas: 1       # PROD: 8-10 (HPA scales on CPU under request load)
    workerProcesses: 1   # CRITICAL on 3.3 GiB nodes — 2 Puma workers OOM at boot eager-load. PROD: 2-4 (≈ vCPUs) on 8 GiB+ nodes
    resources:
      requests:
        cpu: 200m        # PROD: "1"   (Rails is CPU-heavy under real traffic; request ≈ steady use)
        memory: 300Mi    # keep request low to fit 3.3 GiB nodes; Rails uses more at runtime. PROD: 2.5Gi (request ≈ real RSS so the scheduler reserves it)
      limits:
        cpu: "2"         # PROD: "4"
        memory: 3Gi      # boot eager-load spikes to ~2.5Gi then settles ~1.6Gi — 2.5Gi limit OOMs during preload. PROD: 5Gi (headroom for N workers)
    workhorse:           # must set EXPLICITLY — pod request = webservice + workhorse
      resources:
        requests:
          cpu: 10m       # PROD: 100m
          memory: 64Mi   # PROD: 128Mi
        limits:
          cpu: "500m"    # PROD: "1"
          memory: 256Mi  # PROD: 512Mi

  sidekiq:
    replicaCount: 1      # PROD: 2+  (HA background processing) + PDB
    minReplicas: 1       # pin HPA to 1 — default max=10 scales out under CPU and Pending-deadlocks on memory. PROD: minReplicas 2
    maxReplicas: 1       # PROD: 10  (let the HPA scale queues out; cluster must have the memory)
    resources:
      requests:
        cpu: 50m         # PROD: 500m
        memory: 256Mi    # PROD: 2Gi  (request ≈ real RSS)
      limits:
        cpu: "1"         # PROD: "2"
        memory: 3Gi      # sidekiq also eager-loads Rails — same ~2.5Gi boot spike as webservice. PROD: 4Gi

  kas:
    minReplicas: 1       # PROD: 2  (HA agent server) + PDB
    maxReplicas: 1       # PROD: 3
    resources:
      requests:
        cpu: 10m         # PROD: 100m
        memory: 100Mi    # PROD: 256Mi
      limits:
        cpu: "500m"      # PROD: "1"
        memory: 512Mi    # PROD: 1Gi

  gitaly:
    persistence:
      enabled: true
      storageClass: ceph-block   # PROD: fast RWO SSD/NVMe class; Gitaly is latency-sensitive
      size: 20Gi                 # PROD: size to repo growth (100Gi+); cannot shrink later
    resources:
      requests:
        cpu: 100m        # PROD: "1"  (git pack/gc is CPU-heavy)
        memory: 200Mi    # PROD: 2Gi
      limits:
        cpu: "1"         # PROD: "2"
        memory: 512Mi    # PROD: 4Gi
    # PROD HA: run Gitaly Cluster (Praefect + 3 Gitaly nodes) for replicated repos — a single Gitaly is a SPOF

  gitlab-shell:
    # PROD: replicas: 2 + PDB (SSH git access stays up during a node drain)
    resources:
      requests:
        cpu: 10m         # PROD: 100m
        memory: 24Mi     # PROD: 128Mi
      limits:
        cpu: "500m"      # PROD: "1"
        memory: 128Mi    # PROD: 256Mi

  toolbox:
    enabled: true
    resources:
      requests:
        cpu: 50m         # PROD: 100m
        memory: 200Mi    # PROD: 512Mi
      limits:
        cpu: "1"
        memory: 2Gi      # 2Gi needed for initial db:migrate (1381 migrations, Rails app boot ~1.5Gi RSS). PROD: 2Gi is fine; raise if backups run here

# ── PostgreSQL (bundled) ────────────────────────────────────────────────────────
# PROD: do NOT use the bundled single-instance PostgreSQL. Use an EXTERNAL HA
# Postgres (cloud-managed, or Patroni/CloudNativePG: 1 primary + 2 replicas,
# automated failover + PITR backups). Bundled chart = single pod = SPOF + no backups.
postgresql:
  primary:
    persistence:
      storageClass: ceph-block
      size: 10Gi         # PROD (if self-hosting): 100Gi+ on fast SSD, sized to DB growth
  resources:
    limits:
      memory: 1Gi        # PROD: 8Gi+ (Postgres benefits hugely from shared_buffers/cache)
      cpu: "1"           # PROD: "4"

# ── Redis (bundled) ─────────────────────────────────────────────────────────────
# PROD: do NOT use the bundled single Redis. Use EXTERNAL Redis HA (Sentinel or
# cluster, 3+ nodes). Sessions/cache/queues all depend on Redis — a single pod is a SPOF.
redis:
  master:
    persistence:
      storageClass: ceph-block
      size: 2Gi          # PROD: size to working set (8Gi+)
  resources:
    limits:
      memory: 256Mi      # PROD: 2Gi+ with a maxmemory-policy set
EOF

echo "✅ Values file written — $(wc -l < ~/gitlab-stg.yaml) lines"

Why we reduced Puma workers to 1 (`gitlab.webservice.workerProcesses: 1`)¶

GitLab's webservice runs Puma, a multi-process Ruby application server. The chart default is 2 worker processes plus the Puma master. Each Puma worker is a full fork of the Rails application.

The boot-time OOM problem on small nodes

On startup, each Puma worker eager-loads the entire Rails codebase (bootsnap + autoload of all of GitLab). This is a one-time RSS spike to ~2.5 GiB per worker before copy-on-write sharing and GC settle it down to ~1.6 GiB.

With the default 2 workers on our 3.3 GiB worker nodes:

2 workers × ~2.5 GiB boot spike = the pod cgroup blows past any sane memory limit
The container is OOMKilled (exit 137) during preload, on every boot → permanent CrashLoopBackOff
GitLab sets DISABLE_PUMA_WORKER_KILLER=true by default, so nothing recycles a bloated worker either

Fix (two parts, both required):

Change	Value	Reason
`gitlab.webservice.workerProcesses`	`1`	Halves the boot RSS spike — one worker fits in the node's real headroom
`gitlab.webservice.resources.limits.memory`	`3Gi`	The single worker still spikes to ~2.5 GiB at preload; a 2.5 GiB limit OOMs during boot. 3 GiB survives the spike, then settles ~1.6 GiB

This is a resource-constrained compromise — state the production-correct alternative

Production-correct: keep workerProcesses: 2 (or more) for request concurrency and HA, on nodes with 8 GiB+ RAM so the eager-load spike has headroom. One Puma worker means the webservice handles fewer concurrent requests and has no in-pod redundancy.

We run 1 worker only because the pre-production nodes were RAM-constrained. Once the nodes have adequate memory, revert to workerProcesses: 2 and re-run helm upgrade.

Summary of config changes vs chart defaults — and why¶

Every change below is driven by one of three production principles: (A) consume shared platform services, don't re-install them; (B) externalize state to Ceph (HA storage), not pod-local or bundled object storage; (C) fit RAM-constrained pre-production nodes without hiding the production-correct value.

Setting	Chart default	Our value	Principle	Why
`global.certmanager.install`	`true` (bundled)	`false` + sub-chart deleted	A	Cluster cert-manager is a singleton; GitLab consumes it via ingress annotation, doesn't install its own (see above)
`global.ingress.configureCertmanager`	`false`	`false` (keep) + add `cert-manager.io/cluster-issuer` annotation	A	`configureCertmanager: true` creates a namespace `Issuer` — wrong when a `ClusterIssuer` already exists. We use the annotation path instead: cert-manager watches ingresses and auto-provisions certs
`nginx-ingress.enabled`	`true` (bundled)	`false`	A	Cluster already runs ingress-nginx as a shared platform component
`prometheus.install`	`true`	`false`	A	Observability stack is cluster-wide, not per-app
`global.minio.enabled`	`true` (bundled)	`false`	B	MinIO is single-replica ephemeral object storage. We use Ceph RGW (HA, S3-compatible) for artifacts/lfs/uploads/packages/registry/backups
`*.persistence.storageClass`	default SC	`ceph-block`	B	All stateful PVCs (Gitaly, Postgres, Redis) on replicated Ceph RBD, not node-local disk
`gitlab.webservice.workerProcesses`	`2`	`1`	C	Boot-time eager-load OOM on 3.3 GiB nodes (see above)
`webservice.resources.limits.memory`	`2.5Gi`-ish	`3Gi`	C	Survive the ~2.5 GiB preload spike without OOMKill
`sidekiq.{min,max}Replicas`	HPA max `10`	pinned `1`	C	Sidekiq also eager-loads Rails (~2.5 GiB each). Default HPA scales out under CPU and dead-locks the cluster on memory (pods stuck `Pending`)
`webservice.{min,max}Replicas`	HPA-managed	pinned `1`	C	One replica fits memory-constrained nodes; HPA scale-out would risk OOM
`gitlab-runner.install`	`false`	`false`	—	Runners are deployed separately, scoped per-project

GitOps note

In steady state this values file lives in Git and is applied by ArgoCD — not by manual helm install. The manual install here is the bootstrap of the GitLab platform itself; once GitLab is up and hosting its own repos, its Helm release is managed declaratively. The workerProcesses: 1 / pinned-replica compromises are explicitly marked so they can be reverted in the Git-tracked values once nodes are rebalanced.

Step 1.8 — Patch the chart and install GitLab¶

Why we disable the bundled cert-manager dependency chart¶

The GitLab Helm chart bundles cert-manager as a dependency sub-chart (charts/cert-manager). By default, installing GitLab would deploy a second, GitLab-owned cert-manager into the cluster.

Why a bundled cert-manager is wrong for this cluster

Our cluster already runs a cluster-wide cert-manager (installed as a shared platform component — see the cert-manager bootstrap page). Letting the GitLab chart install its own copy causes:

Two cert-manager controllers fighting over the same CRDs — both watch Certificate, Issuer, ClusterIssuer cluster-wide. CRD ownership conflicts and duplicate reconcile loops result in flapping certs and webhook race conditions.
CRD lifecycle coupling — if Helm uninstalls GitLab, it would try to remove cert-manager CRDs that other workloads (ArgoCD, Vault, ingress) depend on, taking down TLS across the whole cluster.
Version drift — the GitLab-pinned cert-manager version may differ from the platform version, giving two incompatible webhook versions.

Production rule: cert-manager is a singleton platform service. Exactly one instance, owned by the platform layer, shared by all tenants. Application charts must consume it, never install it.

We disable it in two layers — both are required:

Layer	What it does	Why it's needed
`global.certmanager.install: false` (+ `--set certmanager.install=false`)	Tells the GitLab chart's templating logic not to render cert-manager resources and not to auto-create an `Issuer`/`Certificate` for the GitLab hosts	Stops GitLab from deploying cert-manager workloads and from managing our TLS secret. We supply the cert manually via `global.ingress.tls.secretName: gitlab-tls`.
Physically removing `charts/cert-manager` + its `Chart.yaml`/`Chart.lock` entries	Deletes the sub-chart from the local chart copy before install	The `install: false` flag alone is not enough — see the Helm bug below.

Why --set certmanager.install=false alone fails — and we must delete the sub-chart

Due to a known Helm bug, a sub-chart's values.schema.json validation fires even when the sub-chart is disabled via a Helm condition. The cert-manager sub-chart schema uses additionalProperties: false and rejects the install key that GitLab's parent chart injects into it. The install aborts with:

Error: values don't meet the specifications of the schema(s) ...
additional properties 'install' not allowed

--disable-openapi-validation does not help — that flag only skips Kubernetes API-server schema validation, not Helm's client-side sub-chart schema validation. The only reliable fix is to physically remove the sub-chart directory and its references in Chart.yaml / Chart.lock from a local copy of the chart, then install from that local copy.

# Pull the chart locally
mkdir -p /tmp/gitlab-chart
helm pull oci://harbor.cl1.sq4.aegis.internal/helm-charts/gitlab --version 9.11.5 -d /tmp/gitlab-chart/ --untar

# Remove cert-manager sub-chart (directory + Chart.yaml/Chart.lock entries)
rm -rf /tmp/gitlab-chart/gitlab/charts/cert-manager

python3 - << 'EOF'
import yaml, pathlib

for fname in ["Chart.yaml", "Chart.lock"]:
    p = pathlib.Path(f"/tmp/gitlab-chart/gitlab/{fname}")
    if not p.exists():
        continue
    data = yaml.safe_load(p.read_text())
    key = "dependencies" if fname == "Chart.yaml" else "dependencies"
    if key in data and data[key]:
        data[key] = [d for d in data[key] if d.get("name") != "cert-manager"]
    p.write_text(yaml.dump(data, default_flow_style=False))
    print(f"✅ patched {fname}")
EOF

echo "✅ cert-manager sub-chart removed"
ls /tmp/gitlab-chart/gitlab/charts/  # cert-manager should NOT appear

Now install from the patched local chart:

helm install gitlab /tmp/gitlab-chart/gitlab \
  -n gitlab \
  -f ~/gitlab-stg.yaml \
  --set certmanager.install=false \
  --timeout 15m \
  --wait

Watch progress in a second terminal:

watch -n5 kubectl get pods -n gitlab

If the release is already in failed state from a previous attempt

helm uninstall gitlab -n gitlab
# Then re-run the helm install above

Step 1.9 — Trigger database migrations¶

The GitLab migrations Job runs as a pre-upgrade Helm hook. On a freshly installed cluster it may time out (if pods were previously Pending due to memory). If you see Init:Error on the webservice pod with "Database has not been initialized yet", run migrations manually from the toolbox:

# Check if migrations job completed
kubectl get jobs -n gitlab

# If the job shows Failed, delete it and run migrations from toolbox
kubectl delete job -n gitlab -l app=migrations 2>/dev/null || true

# Run migrations via toolbox (takes 5-15 minutes on first install)
nohup kubectl exec -n gitlab $(kubectl get pod -n gitlab -l app=toolbox -o jsonpath='{.items[0].metadata.name}') \
  -- gitlab-rake db:migrate > /tmp/migrations.log 2>&1 &
echo "Migrations running PID: $!"

# Tail progress
tail -f /tmp/migrations.log

Registry bucket must be pre-created

Unlike other GitLab buckets (created by the migrations job), the container registry bucket must exist before the registry pod starts or it will CrashLoopBackOff with s3aws: NoSuchBucket. Pre-create all buckets using the GitLab toolbox's built-in aws CLI (which runs inside the cluster and can resolve Ceph RGW's cluster-internal DNS):

S3_ACCESS_KEY=$(kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph -o jsonpath='{.data.AccessKey}' | base64 -d)
S3_SECRET_KEY=$(kubectl get secret rook-ceph-object-user-ceph-objectstore-gitlab \
  -n rook-ceph -o jsonpath='{.data.SecretKey}' | base64 -d)
ENDPOINT='http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local'
TOOLBOX=$(kubectl get pod -n gitlab -l app=toolbox -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n gitlab $TOOLBOX -- bash -c "
  export AWS_ACCESS_KEY_ID='${S3_ACCESS_KEY}'
  export AWS_SECRET_ACCESS_KEY='${S3_SECRET_KEY}'
  export AWS_DEFAULT_REGION='us-east-1'
  for bucket in gitlab-artifacts gitlab-lfs gitlab-uploads gitlab-packages gitlab-registry gitlab-backups gitlab-backups-tmp; do
    aws --endpoint-url '${ENDPOINT}' s3 mb s3://\${bucket} 2>/dev/null || echo \"\${bucket} exists\"
  done
  aws --endpoint-url '${ENDPOINT}' s3 ls
"

Step 1.10 — Wait for webservice to start¶

Once migrations complete, delete the webservice pods so they restart cleanly:

# Check migrations progress
kubectl exec -n gitlab $(kubectl get pod -n gitlab -l app=toolbox -o jsonpath='{.items[0].metadata.name}') \
  -- gitlab-rake db:migrate:status 2>&1 | tail -5
# Expected last line: (in /var/opt/gitlab)   down  ...  or up  ...

# Delete the Init:Error webservice pod — it will be recreated and pass the dependencies check
kubectl delete pod -n gitlab -l app=webservice

# Watch
watch -n5 kubectl get pods -n gitlab

Step 1.11 — Verify all pods are Running¶

kubectl get pods -n gitlab

Expected (all Running/Completed):

NAME                                        READY   STATUS
gitlab-gitaly-0                             1/1     Running
gitlab-gitlab-shell-xxx                     1/1     Running
gitlab-postgresql-0                         2/2     Running
gitlab-redis-master-0                       2/2     Running
gitlab-registry-xxx                         1/1     Running
gitlab-sidekiq-xxx                          1/1     Running
gitlab-toolbox-xxx                          1/1     Running
gitlab-webservice-default-xxx               2/2     Running

No gitlab-minio pod — by design

MinIO is disabled. All object storage (uploads, LFS, artifacts, packages, registry blobs) goes directly to Ceph RGW at rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80. Buckets were pre-created in Step 1.9.

Step 1.10 — Verify web is reachable¶

curl -sk https://gitlab.cl1.sq4.aegis.internal/-/health
# Expected: GitLab OK

After the install¶

Once all pods are Running and the web UI is reachable, continue with the platform setup in GitLab — Deployment & Setup: DNS & registry trust, first login, the modgpt group and projects, access tokens, the container / Helm / PyPI registries, and the ArgoCD GitOps credentials.