Skip to content

Application Monitoring Plan (SigNoz)

Zabbix already covers infrastructure (nodes, CPU, RAM, disk, network). This plan covers the application layer — the ASK services and their dependencies — using SigNoz (OpenTelemetry-native: traces + metrics + logs in one tool, correlated by trace_id). The goal: confirm app health, catch the failure modes we've actually hit (DB connection exhaustion, silent lazy-dependency failures, probe-race CrashLoops, model 404s), and give per-request root-cause via distributed tracing.

Division of labour

Layer Tool Examples
Infrastructure Zabbix node CPU/RAM/disk, VM health, network, Ceph host metrics
Application SigNoz request traces, RED metrics, app logs, DB pool saturation, LLM calls, queue depth
Boundary rule: if it's inside a pod or between services, it's SigNoz; if it's the box under the pod, it's Zabbix.

1. Architecture

flowchart LR
    subgraph apps["ASK pods (ask ns)"]
      A["core / assistant / KMS / frontend\n(OTel SDK auto-instrumentation)"]
    end
    DEPS["dependency /metrics endpoints\nCNPG · Redis · Weaviate · LiteLLM · Hatchet · ESO · ArgoCD"]
    A -->|"OTLP traces/metrics/logs"| COL["OTel Collector\n(DaemonSet + gateway)"]
    DEPS -->|"prometheus scrape"| COL
    LOGS["pod stdout (filelog)"] --> COL
    COL -->|"OTLP"| SIG["SigNoz\n(ClickHouse + query + alerts)"]
  • App signals — each service runs with OTel auto-instrumentation (env-driven, no code change): opentelemetry-instrument wraps FastAPI, SQLAlchemy/psycopg, httpx, redis. Exports OTLP to the collector.
  • Dependency metrics — the OTel Collector's prometheus receiver scrapes the /metrics endpoints the dependencies already expose (CNPG, Redis exporter, Weaviate, LiteLLM, Hatchet, ESO, ArgoCD).
  • Logs — collector filelog receiver tails pod stdout (structured JSON), enriched with k8sattributes (pod/namespace/labels) and correlated to traces via trace_id.
  • Backend — SigNoz (ClickHouse store, query-service, alertmanager, UI).

Air-gap image mirroring

Mirror all SigNoz + OTel images to Harbor (ClickHouse, signoz, otel-collector, schema-migrator) — all available for linux/amd64. Deploy via the SigNoz Helm chart through ArgoCD (own namespace signoz, own PVCs on ceph-block). No external telemetry egress — everything stays in-cluster.

2. The three signals — what to capture

Signal What Why
Traces every inbound HTTP request + its downstream spans: DB queries, Redis, httpx calls to core/foundry/kms/weaviate, Hatchet steps, LLM calls (model, tokens, latency as span attributes) per-request root cause; see where latency/errors come from across services
Metrics RED (Rate, Errors, Duration) per endpoint; USE (Utilization, Saturation, Errors) for pools/queues; custom (LLM tokens, embedding latency, queue depth, S3 ops) trends, SLOs, saturation alerts
Logs structured app logs shipped + correlated to trace_id; exceptions captured as span events the detail when a trace shows an error

Trace propagation: W3C traceparent across frontend → core-service (auth) → assistant-service → foundry / KMS → weaviate / postgres. One trace shows the whole chat or ingestion flow end-to-end.

3. Integrations to monitor (the dependency edges — where it breaks)

Integration Metrics (how) What to watch Failure mode it catches
PostgreSQL (CNPG: core-pg, hatchet-pg, knowledge-pg) CNPG built-in Prometheus exporter → collector scrape active connections vs max_connections, per-DB connections, query latency, deadlocks, replication lag, DB size DB connection exhaustion (the Hatchet incident), slow queries
SQLAlchemy pools (per app) OTel SQLAlchemy instrumentation + custom gauge pool checked-out vs pool_size + max_overflow (KMS 20+5; assistant; core) pool saturation → request stalls
PgBouncer (Hatchet) pgbouncer exporter client/server conns, wait time, pool saturation connection pile-up
Redis (assistant-service) redis exporter (bitnami) connected_clients vs maxclients, latency, memory, evictions, pubsub channels InterruptionListener pubsub down, memory pressure
Hatchet (engine + workers) Hatchet metrics + worker spans worker registered, queue depth/backlog, workflow success/fail/duration, gRPC stream health workers not consuming, backlog growth, the gRPC-insecure issues
Foundry (LiteLLM / vLLM / Infinity) LiteLLM /metrics + httpx spans on LLM calls call rate/latency/errors per model, token usage, 404 model-not-found, 401, rerank/embedding latency, queue/GPU saturation hardcoded cloud-model 404s, key (401), model overload
Weaviate Weaviate /metrics (PROMETHEUS_MONITORING_ENABLED) query latency, object/vector count, heap, import rate, readiness search degraded, OOM, memberlist issues
Ceph S3 (external RGW) httpx/boto spans + custom counters upload/download success, latency, 403/auth errors, bucket op rate InvalidAccessKeyId / placeholder keys, RGW down
Zitadel / IAM (core-service) core-service spans + Zitadel metrics auth success/fail rate, token-validation latency, PAT validity login failures, expired PAT
Vault + ESO ESO controller /metrics externalsecret_sync_status, sync errors, last sync time ESO 403 / secret not synced (the policy-ordering bug)
ArgoCD argocd metrics app sync status, health, OutOfSync count, sync failures drift, failed syncs
Inter-service (frontend→core→assistant→kms) distributed traces per-hop latency/errors which hop broke a request

4. API calls / endpoints to monitor

Service Endpoint(s) Track
All /health/readiness, /health/liveness, /api/health uptime, probe success, latency
core-service auth/IAM, /api/v1/*, model-catalog auth error rate, p95 latency
assistant-service /api/v1/assistants, chat/stream endpoints (intelligence), tooling /health/mcp chat success rate, stream duration, 5xx
KMS /documents/upload/initiate, /documents/search, /health/ ingestion success, search latency, queued count
Foundry (LiteLLM) /v1/chat/completions, /v1/embeddings, /v1/rerank, /v1/models per-model rate/latency/errors, tokens
frontend /api/health, SSR routes, /login/sso/* page latency, login success, 5xx

For each: RED (request rate, error rate by status class, duration p50/p95/p99). Alert on error-rate and p99 SLO breaches.

5. Connection & DB limits (critical — we've been burned here)

The Hatchet outage was PostgreSQL max_connections exhaustion. Connection saturation is the highest-value thing to monitor on this platform.

PostgreSQL (CNPG)

Metric Source Alert
pg_stat_activity count vs max_connections CNPG exporter ⚠️ > 75%, 🔴 > 90%
connections per database (core/assistant/kms/hatchet/zitadel) CNPG exporter spot the noisy DB
connections per application_name / user pg_stat_activity which service leaks
idle-in-transaction count CNPG exporter leaked transactions
longest query / xact age CNPG exporter runaway query

Capacity math to track: every app's pool_size + max_overflow, ×replicas, ×workers, summed across all DBs on the shared cluster, must stay under max_connections. Example KMS: pool_size 20 + max_overflow 5 = 25 per pod × (API + 3 workers). Dashboard this sum vs the server limit — that's the guardrail that would have caught the Hatchet incident.

Other connection limits

Resource Limit to watch
Redis connected_clients vs maxclients
PgBouncer (Hatchet) pool size, cl_waiting, server-conn saturation
httpx client pools (app→backend) max connections / keepalive; connection wait time
Weaviate query timeout breaches, gRPC conn count
Hatchet worker slots in use vs configured (ingestion_slots, generic_worker_slots, …)
OS / pod file descriptors, ephemeral ports (high-conn services)

6. Per-service health signals

"Pod Running" ≠ healthy. Confirm health with golden signals + service-specific:

Service Golden signals Service-specific health
core-service API RED, p99 latency Zitadel reachable, auth success rate, migration job completed, DB pool < 75%
assistant-service chat RED, stream duration Redis pubsub up (InterruptionListener), Hatchet worker registered, Foundry call success, intelligence/tooling reachable
KMS API RED ingestion workflow success rate, embedding/VLM call success, Weaviate query latency, S3 op success, migration completed
frontend page RED, SSR latency /api/health 200, backend reachability (core/assistant/kms), login success rate
Foundry per-model RED, tokens/s model availability (no 404s), GPU/queue saturation, rerank/embed latency

Plus universal: restart count (probe-race CrashLoops — the tiktoken stall), readiness flapping, OOMKills, migration Job success, ESO secret synced.

7. Alerting (SLO + the failure modes we've seen)

Alert Condition Why (incident)
DB connections high any CNPG cluster > 85% max_connections for 5m Hatchet exhaustion outage
Pool saturation app pool checked-out = max for 2m request stalls
CrashLoop / restart spike pod restarts > 3 in 10m tiktoken probe-race
Readiness flapping readiness fail rate > 20% startup/probe issues
LLM error spike Foundry 404/401 rate > threshold hardcoded cloud-model refs, stale key
S3 errors RGW 403/5xx rate > 0 sustained placeholder/expired ask-kms keys
ESO sync error externalsecret not synced > 10m policy/403 ordering bug
Hatchet backlog queue depth rising, no worker progress worker disconnect / gRPC
Endpoint SLO p99 latency or 5xx rate > SLO (per endpoint) user-facing degradation
Migration failed migration Job failed schema drift → app errors
ArgoCD app Degraded/OutOfSync > 15m deploy drift

Route critical → on-call (alertmanager); warnings → dashboard. (Optionally forward critical alerts to Zabbix so there's one pane of glass, since Zabbix is already the ops tool.)

8. Dashboards

  1. Platform overview — all services up/health, request rate, error rate, p99, ArgoCD sync state.
  2. Per-service — RED, dependency latency, pod restarts, pool usage, logs panel.
  3. Database & connections — connections vs limit per CNPG cluster + per DB + per app; pool saturation; slow queries. (Highest-priority board.)
  4. Foundry / LLM — per-model rate/latency/errors/tokens, embedding & rerank latency, 404/401 counts, GPU/queue saturation.
  5. Ingestion (KMS) — Hatchet workflow success/duration, S3 ops, Weaviate import/query, embedding latency.
  6. Service map — auto-generated from traces (frontend→core→assistant→foundry/kms→weaviate/db).

9. Implementation phases

  1. Deploy SigNoz (own ns, ArgoCD, Harbor-mirrored images, ClickHouse PVCs on ceph-block, retention sized — e.g. 15d traces / 30d metrics / 7d logs).
  2. OTel Collector (DaemonSet for logs/node + gateway for OTLP/scrape) with k8sattributes, filelog, prometheus, otlp receivers.
  3. Dependency metrics first (fastest value, no app change) — enable/scrape CNPG, Redis, Weaviate, LiteLLM, Hatchet, ESO, ArgoCD /metrics. This alone gives the DB-connection dashboard.
  4. App auto-instrumentation — add OTel env (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES) + run via opentelemetry-instrument (image/entrypoint change in the sanitized repos). Traces + RED metrics per service.
  5. Logs — ship structured pod logs, correlate to traces.
  6. Dashboards + alerts — build the boards above; wire alertmanager; (optional) forward criticals to Zabbix.
  7. SLOs — set per-endpoint latency/error objectives; review.

10. Air-gap / production notes

  • No external exporters (no SaaS APM, no Sentry cloud, no public OTel endpoints) — all in-cluster.
  • Mirror images to Harbor (SigNoz, ClickHouse, otel-collector) — bastion mirror workflow.
  • Resource cost — ClickHouse is the heavy component; size PVCs + memory deliberately; set retention.
  • Instrumentation is a source/image change for the apps — fits the sanitized-repo + GitOps flow; roll per service. Dependency-metrics (phase 3) need no app change and deliver the connection/DB guardrail immediately — do that first.
  • This complements, doesn't replace, Zabbix; consider funnelling SigNoz critical alerts into Zabbix for a single alerting pane.