ASK Platform — Air-Gapped Deployment (AEGIS cl1.sq4)¶
This documentation covers the end-to-end deployment and operation of the ASK platform in the
air-gapped AEGIS cl1.sq4 production environment (cl1.sq4.aegis.internal). The platform runs
on a bare-metal RKE2 Kubernetes cluster with no outbound internet access — every image,
chart, and package is served from internal registries and mirrors, and all delivery is
GitOps-driven through ArgoCD.
Which site am I on?
This is the production documentation for the live AEGIS cl1.sq4 platform. The
pre-production practice-lab guides (RKE2 on VMware Fusion, and the EC2 x86 lab) live in a
separate site. This site describes only the production environment.
Underlying infrastructure (AEGIS cl1.sq4)¶
AEGIS is a stack of bare-metal HPE servers running Rocky Linux 9.8, deployed and managed entirely through Ansible. GPU nodes run AI/ML workloads bare-metal on Kubernetes; CPU worker VMs handle non-GPU workloads; and all supporting services (DNS, NTP, load balancing, PKI, registry, source control) run in KVM virtual machines hosted on the NVMe compute nodes that also provide Ceph storage.
flowchart TB
subgraph HW["Bare metal · 14× HPE · Rocky Linux 9.8 · Ansible-managed"]
direction LR
NVMe["NVMe Compute\n7× HPE DL385 Gen11\nKVM hypervisor + Ceph host"]
GPUC["GPU Compute\n7× HPE DL380a Gen12\n56× NVIDIA H200 NVL"]
end
VMS["KVM virtual machines · on NVMe compute\ninfra1/2 · bastion1/2 · harbor1/2 · gitlab1\nreposerver1 · daytona1 · k8s-master1-3 · k8s-worker1-7"]
K8S["RKE2 Kubernetes · v1.33 · API VIP 100.115.1.50:6443\n3× control-plane + 7× CPU workers (VMs) + 7× GPU workers (bare-metal)"]
CEPH["Ceph storage · on NVMe compute\n120 OSDs + Mon/Mgr/MDS · RBD · CephFS · S3 (RGW)"]
ING["Frontend ingress · VIP 100.115.2.100"]
NVMe --> VMS
VMS --> K8S
GPUC --> K8S
K8S -->|ceph-csi| CEPH
K8S --> ING| Layer | Detail |
|---|---|
| Servers | 14 bare-metal HPE — 7× DL385 Gen11 (NVMe compute + Ceph), 7× DL380a Gen12 (GPU) |
| NVMe compute | 2× AMD EPYC 9454, 192 vCPU, 1.5 TB DDR5, 22× 1.6 TB U.3 NVMe, 4× 25 GbE — KVM hypervisor + Ceph host |
| GPU compute | 2× Intel Xeon 6740E, 3 TB DDR5, 8× NVIDIA H200 NVL PCIe GPUs each (56 total), 8× 25 GbE — bare-metal K8s workers |
| OS / config | Rocky Linux 9.8, deployed & managed via Ansible; RAID-1 boot (HPE NS204i-u) |
| Virtualisation | KVM on the NVMe compute nodes — 20 VMs (infra, bastion, registry, Git, RKE2 control plane + CPU workers) |
| Networking | Cisco Nexus 9K ToR (vPC pair), LACP over 25 GbE; Backend VLAN 11, Frontend VLAN 12, OOB via iLO |
Infrastructure vs. platform
The bare-metal, networking, and Ceph substrate above is defined by the AEGIS High-Level Design and Technical Design Document. This site documents the ASK platform deployed on top of that substrate.
Platform stack (deployed on AEGIS)¶
| Layer | Technology | Role |
|---|---|---|
| Kubernetes | RKE2 v1.33.1+rke2r1 (Kubernetes 1.33), containerd 2.0.5-k3s1 |
3 master VMs + 7 CPU worker VMs + 7 bare-metal GPU workers; API VIP 100.115.1.50:6443 |
| Ingress | RKE2 built-in NGINX | End-user traffic on the Frontend network (ingress VIP 100.115.2.100) |
| Distributed storage | Ceph Reef v18 (cephadm) | Block (RBD, default SC) for PVCs, filesystem (CephFS) RWX, object (S3 via RGW at s3.cl1.sq4.aegis.internal) |
| Container registry | Harbor (2 replicas, VIP 100.115.1.101) |
Internal registry for all container images (mirrored from source) |
| Git + Helm registry | GitLab (in-cluster, namespace gitlab) |
Source of all charts & manifests that ArgoCD reconciles from |
| GitOps delivery | ArgoCD | 3-layer App-of-Apps; every workload declared in Git |
| Secrets | HashiCorp Vault + External Secrets Operator | Vault is the source of truth; ESO syncs secrets into Kubernetes |
| TLS / internal CA | cert-manager (+ Step-CA infra PKI) | Issues TLS from the internal CA (internal-ca-issuer, O=ai71, CN=modgpt) |
| Databases | CloudNativePG | HA PostgreSQL clusters (core-pg, hatchet-pg, knowledge-pg) |
| Config reload | Stakater Reloader | Restarts workloads on ConfigMap/Secret change |
| Workflow engine | Hatchet | Durable async task/workflow orchestration |
| Search / vectors | Weaviate · Qdrant · OpenSearch | Vector + lexical (BM25) indexing for RAG |
| Model serving | Foundry | Internal LLM endpoint + routing for all AI features |
| Observability | SigNoz + LiteLLM (app) · Zabbix 7.0 (infra) | Traces/metrics/logs, model-proxy telemetry, and hardware/OS monitoring |
Environment at a glance¶
| Item | Value |
|---|---|
| Environment | cl1.sq4 (AEGIS) |
| Cluster domain | cl1.sq4.aegis.internal |
| ASK deployment env | mod-auh1-dev-ask |
| Application namespace | ask (workflow engine in hatchet) |
| Web UI | https://ask.mod.auh1.dev.dir (Frontend VLAN 12) |
| Object storage (S3) | https://s3.cl1.sq4.aegis.internal (Ceph RGW) |
| Git / charts | https://gitlab.cl1.sq4.aegis.internal (SSH port 2222) |
| Backend network | 100.115.1.0/24 (VLAN 11) — K8s API, Ceph, SSH, infra |
| Frontend network | 100.115.2.0/24 (VLAN 12) — application ingress, end-user traffic |
| Network posture | Air-gapped — no outbound internet; images via Harbor, charts via GitLab, packages via local RPM mirror |
Application services¶
| Service | Role |
|---|---|
| Core Service | IAM / org / catalog; bundles Zitadel as the identity provider — everything authenticates through it |
| Assistant Service | Agent/chat service — API + intelligence + tooling + workers |
| Knowledge Management Service (KMS) | Document ingestion + RAG (vector via Weaviate, lexical via OpenSearch) |
| Frontend | The ASK web UI (browser entry point at ask.mod.auh1.dev.dir) |
| Hatchet | Shared workflow engine used by core-service and KMS for async jobs |
How to read this documentation¶
The navigation follows the deployment journey top to bottom:
- Overview — the platform, its service model, URLs, and request flows.
- Platform Deployment — build the foundation in order: infrastructure → base-layer services (GitLab, Vault, cert-manager, ArgoCD, ESO, CNPG, Reloader) → platform onboarding.
- Application Services — deploy each ASK service (core-service, assistant-service, KMS, frontend, Hatchet).
- Foundry & Models — the internal model-serving layer and model routing.
- Operations — day-2 runbooks, observability, and production-hardening considerations.
- Reference — the image/config inventory and knowledge base.
- Troubleshooting — known platform issues and their fixes.
New here?
Start with the Deployment Overview — it is the master index that sequences the base-layer platform services and the application services.