Resource Inventory — Nodes & Services¶
Live snapshot of node capacity and per-service CPU / memory allocation. Captured: 2026-06-23. Use as baseline before any resource-tuning pass.
Node Inventory¶
Live usage from kubectl top nodes (captured 2026-06-23). Allocated = sum of pod requests scheduled.
| Node (FQDN) | Type | CPUs | RAM (GB) | Live CPU | Live RAM | CPU alloc | RAM req |
|---|---|---|---|---|---|---|---|
| gpu1.cl1.sq4.aegis.internal | GPU | 192 | 3,023 | 3,388m (1%) | 30.4 GB (1%) | 24.6 cores (13%) | 64 GB (2%) |
| gpu2.cl1.sq4.aegis.internal | GPU | 192 | 3,023 | 275m (0%) | 30.8 GB (1%) | 16.6 cores (8%) | 64 GB (2%) |
| gpu3.cl1.sq4.aegis.internal | GPU | 192 | 3,023 | 426m (0%) | 30.4 GB (1%) | 16.6 cores (8%) | 64 GB (2%) |
| gpu4.cl1.sq4.aegis.internal | GPU | 192 | 3,023 | 471m (0%) | 109.5 GB (3%) | 48.6 cores (25%) | 193 GB (6%) |
| gpu5.cl1.sq4.aegis.internal | GPU | 192 | 3,023 | 459m (0%) | 121.7 GB (4%) | 64.6 cores (34%) | 256 GB (8%) |
| gpu6.cl1.sq4.aegis.internal | GPU | 192 | 3,023 | 326m (0%) | 23.3 GB (0%) | 16.6 cores (8%) | 64 GB (2%) |
| gpu7.cl1.sq4.aegis.internal | GPU | 192 | 3,023 | 435m (0%) | 120.8 GB (4%) | 64.6 cores (34%) | 256 GB (8%) |
| k8s-master1.cl1.sq4.aegis.internal | Master | 4 | 15.4 | 212m (10%) | 6.2 GB (57%) | 1.6 cores (79%) | 2.5 GB (22%) |
| k8s-master2.cl1.sq4.aegis.internal | Master | 4 | 15.4 | 174m (8%) | 6.5 GB (59%) | 1.5 cores (73%) | 2.3 GB (20%) |
| k8s-master3.cl1.sq4.aegis.internal | Master | 4 | 15.4 | 128m (6%) | 6.1 GB (55%) | 1.5 cores (73%) | 2.3 GB (20%) |
| k8s-worker1.cl1.sq4.aegis.internal | Worker | 80 | 755 | 254m (0%) | 6.1 GB (0%) | 2.3 cores (3%) | 3.8 GB (0.5%) |
| k8s-worker2.cl1.sq4.aegis.internal | Worker | 80 | 755 | 151m (0%) | 8.5 GB (1%) | 3.5 cores (4%) | 11.4 GB (1.5%) |
| k8s-worker3.cl1.sq4.aegis.internal | Worker | 80 | 755 | 255m (0%) | 5.9 GB (0%) | 3.7 cores (5%) | 7.4 GB (1.0%) |
| k8s-worker4.cl1.sq4.aegis.internal | Worker | 128 | 1,007 | 426m (0%) | 12.4 GB (1%) | 3.9 cores (3%) | 10.8 GB (1.1%) |
| k8s-worker5.cl1.sq4.aegis.internal | Worker | 128 | 1,007 | 314m (0%) | 7.6 GB (0%) | 4.0 cores (3%) | 8.9 GB (0.9%) |
| k8s-worker6.cl1.sq4.aegis.internal | Worker | 128 | 1,007 | 304m (0%) | 9.8 GB (0%) | 5.1 cores (4%) | 14.6 GB (1.5%) |
| k8s-worker7.cl1.sq4.aegis.internal | Worker | 128 | 1,007 | 1,423m (1%) | 9.2 GB (0%) | 4.9 cores (4%) | 10.8 GB (1.1%) |
Masters at 55–59% RAM
k8s-master1/2/3 are running at over 55% actual memory usage on 15.4 GB nodes. No headroom for etcd spikes. Do not schedule workloads on masters.
Worker nodes: < 2% actual memory. GPU nodes: < 5% actual CPU. Headroom is substantial — raise limits freely.
Supporting Infrastructure VMs (non-Kubernetes)¶
KVM VMs on the NVMe compute hosts that provide platform services alongside the RKE2 cluster. All on the Backend network (100.115.1.0/24, VLAN 11); infra1/2 are dual-homed onto the Frontend network for ingress/LB.
| Hostname | IP | Role | KVM host |
|---|---|---|---|
| infra1 | 100.115.1.21 (·2.21) | DNS · NTP · HAProxy · PKI (Step-CA) · Zabbix | kvm1 |
| infra2 | 100.115.1.22 (·2.22) | DNS · NTP · HAProxy · PKI (Step-CA) · Zabbix | kvm2 |
| bastion1 | 100.115.1.23 | Bastion / jump host | kvm1 |
| bastion2 | 100.115.1.24 | Bastion / jump host | kvm3 |
| harbor1 | 100.115.1.25 | Harbor container registry (primary) | kvm2 |
| harbor2 | 100.115.1.26 | Harbor container registry (backup) | kvm3 |
| gitlab1 | 100.115.1.27 | GitLab — Git + Helm registry (SSH port 2222) | kvm3 |
| reposerver1 | 100.115.1.30 | Local RPM mirror (air-gapped packages) | kvm1 |
| daytona1 | 100.115.1.35/24 | Development environments (Daytona) — self-hosted dev workspaces; not part of the ASK platform | — |
| vascan1 | 100.115.1.120 | Vulnerability scanner | kvm1 |
| logserver1 | 100.115.1.121 | Log collector | kvm1 |
daytona1 — new VM (2026-07)
daytona1 (100.115.1.35/24) was provisioned as a standalone development-environment host. It runs outside the ASK application and K8s cluster, so it is not covered by the ASK graceful-shutdown runbook — sequence it with general infrastructure. Access list is still being decided with the infra team.
Service VIPs (Keepalived): K8s API 100.115.1.50:6443 · Harbor 100.115.1.101 · Zabbix 100.115.1.110 · Ingress 100.115.2.100.
Service Resource Inventory (Live — 2026-06-23)¶
Source: kubectl get pods -n ask output. Jobs and migration pods excluded.
Legend: ⚠️ = missing limit or request (needs tuning) · ✅ = fully set
Core Service¶
| Pod | Count | CPU req | CPU limit | Mem req | Mem limit |
|---|---|---|---|---|---|
| core-service | 1 | 300m | ⚠️ none | 500 Mi | 768 Mi |
| core-service-zitadel | 1 | ⚠️ none | ⚠️ none | ⚠️ none | ⚠️ none |
| core-service-zitadel-login | 1 | ⚠️ none | ⚠️ none | ⚠️ none | ⚠️ none |
Assistant Service¶
| Pod | Count | CPU req | CPU limit | Mem req | Mem limit |
|---|---|---|---|---|---|
| assistant-service (API) | 2 | 500m | ⚠️ none | 2 Gi | 4 Gi |
| assistant-service-generic-worker | 1 | 250m | 500m | 512 Mi | 1 Gi ✅ |
| assistant-service-intelligence | 2 | 500m | ⚠️ none | 2 Gi | 10 Gi |
| assistant-service-tooling | 2 | 500m | ⚠️ none | 2 Gi | 4 Gi |
| assistant-service-redis-master | 1 | 1000m | 1500m | 2 Gi | 3 Gi ✅ |
Knowledge Management Service¶
| Pod | Count | CPU req | CPU limit | Mem req | Mem limit |
|---|---|---|---|---|---|
| knowledge-management-service (API) | 1 | 500m | ⚠️ none | 2 Gi | 4 Gi |
| knowledge-management-service-generic-worker | 1 | 500m | ⚠️ none | 2 Gi | 4 Gi |
| knowledge-management-service-ingestion-worker | 1 | 500m | ⚠️ none | 2 Gi | 4 Gi |
| knowledge-management-service-live-ingestion-worker | 1 | 500m | ⚠️ none | 2 Gi | 4 Gi |
Frontend¶
| Pod | Count | CPU req | CPU limit | Mem req | Mem limit |
|---|---|---|---|---|---|
| frontend | 1 | 300m | 1000m | 1 Gi | 1 Gi ✅ |
Hatchet Workflow Engine¶
| Pod | Count | CPU req | CPU limit | Mem req | Mem limit |
|---|---|---|---|---|---|
| hatchet-api | 2 | 100m | 500m | 128 Mi | 512 Mi ✅ |
| hatchet-engine | 2 | 250m | 1000m | 256 Mi | 2 Gi ✅ |
| hatchet-frontend | 2 | 25m | 100m | 32 Mi | 64 Mi ✅ |
| hatchet-pg-pooler | 2 | 50m | 200m | 64 Mi | 128 Mi ✅ |
| hatchet-rabbitmq | 3 | 250m | 500m | 512 Mi | 1 Gi ✅ |
CNPG Database Clusters (per pod × 3 replicas)¶
Tuned 2026-06-23: raised from 1 Gi/2 Gi → 16 Gi/32 Gi (2× PP baseline of 8/16 Gi).
| Cluster | Pods | CPU req | CPU limit | Mem req | Mem limit | shared_buffers |
|---|---|---|---|---|---|---|
| hatchet-pg | 3 | 1000m | 4000m | 16 Gi | 32 Gi ✅ | 8 GB |
| core-pg | 3 | 1000m | 4000m | 16 Gi | 32 Gi ✅ | 8 GB |
| knowledge-pg | 3 | 1000m | 4000m | 16 Gi | 32 Gi ✅ | 8 GB |
Weaviate (Vector DB)¶
| Pod | Count | CPU req | CPU limit | Mem req | Mem limit |
|---|---|---|---|---|---|
| weaviate | 1 | ⚠️ none | ⚠️ none | ⚠️ none | ⚠️ none |
Gaps Requiring Tuning¶
| Priority | Service | Issue |
|---|---|---|
| 🔴 High | weaviate | Zero requests AND limits — completely unbounded |
| 🔴 High | core-service-zitadel / zitadel-login | Zero requests AND limits — completely unbounded |
| 🟠 Medium | core-service | CPU limit missing — unbounded CPU on primary auth service |
| 🟠 Medium | assistant-service API, intelligence, tooling | CPU limit missing — 5 pods with no CPU ceiling |
| 🟠 Medium | knowledge-management-service (all workers) | CPU limit missing — 4 pods with no CPU ceiling |
CPU limits in Kubernetes
Missing CPU limits means a pod can consume unlimited CPU, starving neighbours on the node. Missing CPU requests means the scheduler cannot properly place the pod. Missing memory limits risks OOMKill of other pods on the same node when memory pressure hits.
Total Reserved — CNPG DBs¶
| Per pod | × 9 pods | Free on one worker (755 GB) | |
|---|---|---|---|
| Request | 16 Gi | 144 Gi | 611 GB |
| Limit | 32 Gi | 288 Gi | 467 GB |