KB-04 — End-to-End DNS & Traffic Flow¶

This page explains how DNS resolution and traffic routing works across every layer of the platform — from a pod making a DNS query to the response arriving at the correct backend service.

Components Involved¶

Component	Role	Location
BIND	Authoritative DNS for the cluster zones	`100.115.1.21` (primary), `100.115.1.22` (secondary)
CoreDNS	In-cluster DNS resolver (ClusterIP `172.31.0.10`)	Pods on k8s-worker1–7 (7 replicas)
Nginx Ingress Controller	HTTP/HTTPS routing by hostname	DaemonSet on all 17 nodes
HAProxy	TLS termination for Ceph RGW S3 only	VIP `100.115.1.130` on Ceph MON nodes
Harbor	Container registry — direct IP, no Ingress	`100.115.1.101` (dedicated IP in BIND)

Harbor does not use Nginx Ingress

Harbor has a dedicated IP (100.115.1.101) registered directly in BIND. kubectl get ingress -A | grep harbor returns nothing — Harbor exposes itself directly via its own service, not through the cluster Ingress Controller. Both harbor.cl1.sq4.aegis.internal and harbor.mod.auh1.dev.dir point to 100.115.1.101.

Architecture Diagram¶

                         ┌─────────────────────────────────────────────────────────┐
                         │                  BIND DNS Servers                       │
                         │  Primary: 100.115.1.21   Secondary: 100.115.1.22       │
                         │  Zones: cl1.sq4.aegis.internal, mod.auh1.dev.dir       │
                         └──────────────────────┬──────────────────────────────────┘
                                                │  authoritative answers
                                                │
                         ┌──────────────────────▼──────────────────────────────────┐
                         │              CoreDNS (172.31.0.10:53)                   │
                         │  ClusterIP service — 7 pods on k8s-worker1–7            │
                         │                                                          │
                         │  Resolution order:                                       │
                         │  1. kubernetes plugin  → cluster.local names            │
                         │  2. hosts block        → static overrides (interim)     │
                         │  3. forward to BIND    → everything else                │
                         └──────────────────────┬──────────────────────────────────┘
                                                │  resolved IP
                                                │
              ┌─────────────────────────────────┴──────────────────────────────────┐
              │                                                                     │
              ▼  Ingress hostnames (gitlab, argocd, vault…)           ▼  S3 endpoint
 ┌────────────────────────────────────────┐              ┌────────────────────────┐
 │  Nginx Ingress Controller              │              │  HAProxy VIP           │
 │  DaemonSet — all 17 nodes             │              │  100.115.1.130         │
 │  Listens: 80/443 on every node        │              │  TLS termination only  │
 │  Routes by Host: header               │              │  → Ceph RGW backends   │
 └──────────────┬─────────────────────────┘              └────────────────────────┘
                │
    ┌───────────┼───────────┐
    ▼           ▼           ▼
 gitlab      argocd      vault
 service     service     service

Scenario 1 — Pod on a Worker Node resolves `gitlab.cl1.sq4.aegis.internal`¶

Pod (k8s-worker3)
    │
    │ 1. DNS query → /etc/resolv.conf → nameserver 172.31.0.10
    │
    ▼
CoreDNS pod (k8s-worker3 — LOCAL, no extra hop)
    │
    │ 2. Check kubernetes plugin → not a cluster.local name → skip
    │ 3. Check hosts block → match: 100.115.1.51 → RETURN
    │    (temporary — until BIND has the record)
    │    OR
    │ 3. forward to BIND 100.115.1.21 → A record → 100.115.1.51/52/53 → RETURN
    │
    ▼
Pod receives: 100.115.1.51 (k8s-master1)
    │
    │ 4. TCP connect to 100.115.1.51:443
    │
    ▼
Nginx Ingress Controller (running on k8s-master1 as DaemonSet)
    │
    │ 5. Reads Host: gitlab.cl1.sq4.aegis.internal header
    │ 6. Matches Ingress rule → routes to gitlab-webservice Service
    │
    ▼
GitLab pod (wherever it's scheduled)

Characteristics: - DNS query stays on the same node — CoreDNS pod is co-located on every worker - Zero extra network hops for DNS - Ingress hit is determined by which master IP DNS returned (round-robin across .51/.52/.53)

Scenario 2 — Pod on a GPU Node resolves `gitlab.cl1.sq4.aegis.internal`¶

Pod (gpu3 — 100.115.1.x)
    │
    │ 1. DNS query → /etc/resolv.conf → nameserver 172.31.0.10
    │
    ▼
CoreDNS ClusterIP 172.31.0.10
    │
    │ ⚠️  kube-proxy load-balances to one of the 7 CoreDNS pods
    │     CoreDNS pods are on worker1–7, NOT on GPU nodes
    │     → EXTRA network hop: gpu3 → k8s-workerX
    │
    ▼
CoreDNS pod (k8s-workerX — REMOTE hop)
    │
    │ 2. Same resolution logic as Scenario 1
    │
    ▼
Pod receives: 100.115.1.51 (or .52 or .53)
    │
    │ 3. TCP connect to 100.115.1.51:443
    │
    ▼
Nginx Ingress Controller (on that master node)
    │
    │ 4. Routes to correct backend service
    │
    ▼
Target pod

Characteristics: - DNS query crosses a node boundary — ~0.1–1ms additional latency per lookup - For AI/ML workloads doing frequent S3 calls or service discovery, this adds up - Mitigation: Add CoreDNS toleration for GPU nodes (or use NodeLocal DNSCache)

Scenario 3 — Pod resolves `s3.cl1.sq4.aegis.internal` (Ceph RGW / S3)¶

Pod (any node)
    │
    │ 1. DNS query → CoreDNS → BIND → 100.115.1.130
    │
    ▼
HAProxy VIP: 100.115.1.130
    │
    │ 2. HAProxy terminates TLS (presents wildcard cert)
    │ 3. Load-balances across 3 Ceph RGW backends
    │
    ▼
Ceph RGW (ceph1/ceph2/ceph3 — 100.115.1.11–13)
    │
    │ 4. S3 API response
    │
    ▼
Pod receives S3 response

Why HAProxy here and not Ingress?

Ceph RGW is not a Kubernetes service — it runs on bare-metal Ceph nodes outside the cluster. Nginx Ingress only routes to Kubernetes Services. HAProxy bridges the gap and also provides TLS termination so the RGW backends can stay on plain HTTP internally.

Scenario 4 — External User (VPN/Bastion) accesses `argocd.cl1.sq4.aegis.internal`¶

User laptop / Bastion
    │
    │ 1. DNS query → corporate resolver → BIND 100.115.1.21
    │ 2. BIND returns A records: 100.115.1.51, 100.115.1.52, 100.115.1.53
    │    OS picks one (round-robin)
    │
    ▼
100.115.1.51 (k8s-master1) port 443
    │
    │ 3. Nginx Ingress terminates TLS (presents wildcard cert from cert-manager)
    │ 4. Routes to argocd-server Service → ArgoCD pod
    │
    ▼
ArgoCD UI in browser

Note: External users rely on BIND — CoreDNS is only for in-cluster pod DNS. Until BIND has the records, external users must use /etc/hosts overrides.

DNS Resolution Order — CoreDNS¶

Query arrives at CoreDNS
        │
        ▼
┌─────────────────────────────────┐
│  1. kubernetes plugin           │  ← *.cluster.local, *.svc.cluster.local
│     (pod/service names)         │     e.g. gitlab-webservice.gitlab.svc.cluster.local
└──────────────┬──────────────────┘
               │ no match
               ▼
┌─────────────────────────────────┐
│  2. hosts block (interim)       │  ← static overrides while BIND is being updated
│     gitlab → 100.115.1.51       │     REMOVE once BIND records are in place
│     argocd → 100.115.1.51       │
│     vault  → 100.115.1.51       │
└──────────────┬──────────────────┘
               │ no match (fallthrough)
               ▼
┌─────────────────────────────────┐
│  3. forward . /etc/resolv.conf  │  ← forwards to BIND 100.115.1.21/22
│     (BIND authoritative)        │     harbor → 100.115.1.101
│                                 │     s3     → 100.115.1.130
└─────────────────────────────────┘

Why CoreDNS Does NOT Run on GPU or Master Nodes¶

Node type	Schedulable for CoreDNS?	Reason
k8s-worker1–7	✅ Yes	No blocking taints
k8s-master1–3	❌ No (currently)	Taint `node-role.kubernetes.io/master` not tolerated
gpu1–7	❌ No	Taint `nvidia.com/gpu:NoSchedule` not tolerated

CoreDNS tolerates node-role.kubernetes.io/control-plane but not node-role.kubernetes.io/master (RKE2 sets both taints on master nodes).

Production fix — add master toleration so CoreDNS runs on 10 nodes (7 workers + 3 masters):

kubectl patch deployment rke2-coredns-rke2-coredns -n kube-system --type=json -p='[
  {"op":"add","path":"/spec/template/spec/tolerations/-","value":
    {"key":"node-role.kubernetes.io/master","operator":"Exists","effect":"NoSchedule"}}
]'
kubectl scale deployment rke2-coredns-rke2-coredns -n kube-system --replicas=10

Summary Table¶

Query source	CoreDNS hop	DNS backend	Traffic lands on
Worker node pod → gitlab/argocd/vault/ask	Local (same node)	hosts block → BIND	Nginx Ingress (master node `.51/.52/.53`)
GPU node pod → gitlab/argocd/vault/ask	Remote (worker node)	hosts block → BIND	Nginx Ingress (master node `.51/.52/.53`)
Master node pod → gitlab/argocd/vault/ask	Remote (worker node)	hosts block → BIND	Nginx Ingress (master node `.51/.52/.53`)
Any pod → harbor	Local or Remote	BIND	Harbor direct (`100.115.1.101`) — no Ingress
Any pod → s3	Local or Remote	BIND	HAProxy VIP (`100.115.1.130`) → Ceph RGW
External user/VPN	N/A (no CoreDNS)	BIND directly	Same as above

DNS Records Reference¶

Zone: `cl1.sq4.aegis.internal`¶

Hostname	IP	Via
`gitlab`	`100.115.1.51/52/53`	Nginx Ingress
`registry`	`100.115.1.51/52/53`	Nginx Ingress
`kas`	`100.115.1.51/52/53`	Nginx Ingress
`argocd`	`100.115.1.51/52/53`	Nginx Ingress
`vault`	`100.115.1.51/52/53`	Nginx Ingress
`harbor`	`100.115.1.101`	Direct (no Ingress)
`s3`	`100.115.1.130`	HAProxy → Ceph RGW

Zone: `mod.auh1.dev.dir`¶

Hostname	IP	Via
`ask`	`100.115.1.51/52/53`	Nginx Ingress
`api`	`100.115.1.51/52/53`	Nginx Ingress
`argocd`	`100.115.1.51/52/53`	Nginx Ingress
`gitlab`	`100.115.1.51/52/53`	Nginx Ingress
`vault`	`100.115.1.51/52/53`	Nginx Ingress
`harbor`	`100.115.1.101`	Direct (no Ingress)
`s3`	`100.115.1.130`	HAProxy → Ceph RGW

KB-04 — End-to-End DNS & Traffic Flow¶

Components Involved¶

Architecture Diagram¶

Scenario 1 — Pod on a Worker Node resolves gitlab.cl1.sq4.aegis.internal¶

Scenario 2 — Pod on a GPU Node resolves gitlab.cl1.sq4.aegis.internal¶

Scenario 3 — Pod resolves s3.cl1.sq4.aegis.internal (Ceph RGW / S3)¶

Scenario 4 — External User (VPN/Bastion) accesses argocd.cl1.sq4.aegis.internal¶