Skip to content

Verify the Docs Match the Live Infrastructure

This runbook turns the concrete claims in this documentation (node counts, DNS names, VIPs, image paths, secret names, CPU architecture, Ceph layout) into live probes you run from the bastion, then reports ✅/❌ per check. Run it after any infrastructure change, and before shipping a new docs revision, to catch drift between what the docs say and what the cluster actually is.

Where to run

Run everything from a bastion (ssh -p 2222 cloud-user@bastion1, 100.115.1.23), where kubectl, dig, and the Ceph toolbox are reachable. The script is read-only — it makes no changes to the cluster.

What it checks (and against which doc)

# Doc claim Live check
1 17 nodes: 3 masters + 7 CPU workers + 7 GPU kubectl get nodes
2 Nodes are x86_64/amd64 (not ARM64) kubectl get nodes -o wide → ARCH
2b Kubernetes v1.33 (RKE2), nodes on Rocky Linux 9.8 kubectl version; kubectl get nodes -o wide → OS-IMAGE
3 Core endpoints resolve in DNS dig +short for gitlab / vault / argocd / registry.gitlab / s3 / harbor / ask
4 API VIP reachable (200, or 401 = anonymous auth hardened off — still reachable) curl -k https://100.115.1.50:6443/healthz + kubectl get --raw /healthz
5 GitLab runs in-cluster (namespace gitlab) kubectl get pods -n gitlab shows running GitLab pods
6 App images come from Harbor ask/ (no modgpt/container-images, no arm64) scrape all running images
7 Pull secret harbor-pull-secret present in ask kubectl get secret
8 CA issuer internal-ca-issuer exists kubectl get clusterissuer
9 Namespaces ask / hatchet / argocd / vault exist kubectl get ns
10 CNPG clusters core-pg / hatchet-pg / knowledge-pg healthy kubectl get cluster -n ask
11 Storage classes ceph-block (default) + cephfs kubectl get sc
12 Core app workloads present in ask kubectl get deploy -n ask

The script

Save as verify-docs.sh on the bastion, chmod +x verify-docs.sh, and run ./verify-docs.sh. It prints a per-check result and exits non-zero if any check fails (so it can gate CI).

#!/usr/bin/env bash
# verify-docs.sh — assert the documented facts against the live AEGIS cl1.sq4 cluster.
# Read-only. Run from the bastion. Exit code = number of failed checks.
set -uo pipefail

PASS=0; FAIL=0
ok(){   echo "✅ $*"; PASS=$((PASS+1)); }
bad(){  echo "❌ $*"; FAIL=$((FAIL+1)); }

# ---- expected values (from the docs) -------------------------------------------------
DOMAIN="cl1.sq4.aegis.internal"
EXPECT_NODES=17
EXPECT_MASTERS=3; EXPECT_CPU=7; EXPECT_GPU=7
EXPECT_K8S="1.33"
EXPECT_OS="Rocky Linux 9.8"
API_VIP="100.115.1.50:6443"
DNS_HOSTS=(gitlab vault argocd registry.gitlab s3 harbor)
APP_NS=ask
PULL_SECRET=harbor-pull-secret
CA_ISSUER=internal-ca-issuer
NAMESPACES=(ask hatchet argocd vault)
CNPG=(core-pg hatchet-pg knowledge-pg)
BAD_IMAGE_RE='modgpt/container-images|registry\.gitlab.*/ask/|/arm64|:.*arm64'

echo "== AEGIS ${DOMAIN} — docs ↔ infra conformance =="

# 1 · node count
got=$(kubectl get nodes --no-headers 2>/dev/null | wc -l | tr -d ' ')
[ "$got" = "$EXPECT_NODES" ] && ok "node count = $got" || bad "node count: doc says $EXPECT_NODES, cluster has $got"

# 2 · architecture is amd64
archs=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.nodeInfo.architecture}{"\n"}{end}' | sort -u)
[ "$archs" = "amd64" ] && ok "all nodes amd64 (x86_64)" || bad "unexpected node arch(es): $(echo $archs | tr '\n' ' ')"

# 2b · Kubernetes server version
kv=$(kubectl version -o json 2>/dev/null | grep -o '"gitVersion": *"[^"]*"' | head -1 | cut -d'"' -f4)
echo "$kv" | grep -q "$EXPECT_K8S" && ok "K8s server $kv (~$EXPECT_K8S)" || bad "K8s server $kv (doc says $EXPECT_K8S)"

# 2c · node OS image (all uniform, matching the doc)
oses=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.nodeInfo.osImage}{"\n"}{end}' | sort -u)
if [ "$(echo "$oses" | wc -l | tr -d ' ')" = "1" ] && echo "$oses" | grep -q "$EXPECT_OS"; then
  ok "node OS = $oses"
else
  bad "node OS: got [$(echo "$oses" | tr '\n' ';')], doc says $EXPECT_OS"
fi

# 3 · DNS resolution of core endpoints
for h in "${DNS_HOSTS[@]}"; do
  if dig +short "${h}.${DOMAIN}" | grep -qE '[0-9]'; then ok "DNS ${h}.${DOMAIN}"; else bad "DNS ${h}.${DOMAIN} does not resolve"; fi
done
# app UI is on its own domain
dig +short ask.mod.auh1.dev.dir | grep -qE '[0-9]' && ok "DNS ask.mod.auh1.dev.dir" || bad "DNS ask.mod.auh1.dev.dir does not resolve"

# 4 · API VIP reachable — 200 = open healthz, 401/403 = reachable but anonymous auth hardened off
code=$(curl -ks -o /dev/null -w '%{http_code}' "https://${API_VIP}/healthz" || echo 000)
case "$code" in
  200)     ok "API VIP ${API_VIP} reachable (healthz 200)";;
  401|403) ok "API VIP ${API_VIP} reachable (HTTP $code — anonymous auth hardened off)";;
  *)       bad "API VIP ${API_VIP} unreachable (HTTP $code)";;
esac
# authenticated healthz via kubeconfig should return "ok"
hz=$(kubectl get --raw='/healthz' 2>/dev/null || echo FAIL)
[ "$hz" = "ok" ] && ok "authenticated /healthz = ok" || bad "authenticated /healthz = $hz"

# 5 · GitLab runs in-cluster (namespace gitlab, delivered via ArgoCD)
if kubectl get ns gitlab >/dev/null 2>&1 && [ "$(kubectl get pods -n gitlab --no-headers 2>/dev/null | grep -c Running)" -gt 0 ]; then ok "GitLab in-cluster: namespace gitlab has running pods"; else bad "GitLab 'gitlab' namespace missing or no running pods"; fi

# 6 · image provenance — every running image must be Harbor ask/ or dockerhub/, none stale/arm64
imgs=$(kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{range .spec.initContainers[*]}{.image}{"\n"}{end}{end}' | sort -u)
stale=$(echo "$imgs" | grep -E "$BAD_IMAGE_RE" || true)
[ -z "$stale" ] && ok "image paths clean (Harbor ask//dockerhub/, no arm64/modgpt)" || { bad "stale image refs found:"; echo "$stale" | sed 's/^/     /'; }

# 7 · pull secret present
kubectl get secret "$PULL_SECRET" -n "$APP_NS" >/dev/null 2>&1 && ok "secret $PULL_SECRET in $APP_NS" || bad "secret $PULL_SECRET missing in $APP_NS"

# 8 · CA issuer
kubectl get clusterissuer "$CA_ISSUER" >/dev/null 2>&1 && ok "clusterissuer $CA_ISSUER" || bad "clusterissuer $CA_ISSUER missing"

# 9 · namespaces
for ns in "${NAMESPACES[@]}"; do
  kubectl get ns "$ns" >/dev/null 2>&1 && ok "namespace $ns" || bad "namespace $ns missing"
done

# 10 · CNPG clusters healthy
for c in "${CNPG[@]}"; do
  st=$(kubectl get cluster "$c" -n "$APP_NS" -o jsonpath='{.status.phase}' 2>/dev/null || echo MISSING)
  echo "$st" | grep -qi "healthy" && ok "CNPG $c: $st" || bad "CNPG $c: $st"
done

# 11 · storage classes
kubectl get sc ceph-block >/dev/null 2>&1 && ok "StorageClass ceph-block" || bad "StorageClass ceph-block missing"
def=$(kubectl get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.metadata.name}{end}')
[ "$def" = "ceph-block" ] && ok "default StorageClass = ceph-block" || bad "default StorageClass = '${def:-none}' (doc says ceph-block)"
kubectl get sc cephfs >/dev/null 2>&1 && ok "StorageClass cephfs (RWX)" || bad "StorageClass cephfs missing"

# 12 · core app workloads exist
for d in core-service assistant-service knowledge-management-service frontend; do
  kubectl get deploy "$d" -n "$APP_NS" >/dev/null 2>&1 && ok "deploy $d" || bad "deploy $d missing in $APP_NS"
done

echo "== summary: ${PASS} passed, ${FAIL} failed =="
exit "$FAIL"

Run it in CI / on a schedule

Because it exits non-zero on any mismatch, wire it into a scheduled job (or a pre-release gate). A green run is objective evidence the docs still describe reality; a red run pinpoints exactly which fact drifted.

Hardware facts — a manual / Zabbix checklist

Some documented facts are not visible from Kubernetes and must be confirmed against the infrastructure tooling (Zabbix already inventories Linux OS + iLO + Ceph):

Doc claim How to verify
14 HPE servers (7× DL385 Gen11, 7× DL380a Gen12) Zabbix host inventory / iLO Redfish (/redfish/v1/Systems)
NVMe compute: 192 vCPU, 1.5 TB RAM, 22× 1.6 TB NVMe Zabbix per-host inventory; lscpu, free -h, lsblk on each host
56× NVIDIA H200 NVL (8 per GPU node) nvidia-smi -L on a GPU node; or the NVIDIA device-plugin capacity: kubectl get nodes -o jsonpath='{..nvidia\.com/gpu}'
Ceph Reef v18, 120 OSDs, ~58.4 TiB usable Ceph toolbox: ceph version, ceph osd stat, ceph df
Harbor 2 replicas (VIP 100.115.1.101) ssh -p 2222 cloud-user@100.115.1.25 / .26; Harbor health API
VIPs .50 / .101 / .110 / .2.100 (Keepalived) On infra1/2: ip addr show \| grep 100.115

GPU count via Kubernetes

You can partially confirm the GPU fleet from the cluster: 

kubectl get nodes -l nvidia.com/gpu.present=true \
  -o custom-columns='NODE:.metadata.name,GPUs:.status.capacity.nvidia\.com/gpu'
# expect 7 GPU nodes × 8 = 56 total

Interpreting results

  • All ✅ — the documented facts match the running platform; safe to ship the docs.
  • Any ❌ — either the infra changed (update the docs) or the docs are wrong (fix them). The failing line names the exact claim and the observed value, so the fix is unambiguous.