Hatchet — Workers, Job Scheduling & Observability

This page covers how ASK application workers connect to Hatchet, the end-to-end job scheduling path from trigger to execution, and how to observe the state of the system in both the Hatchet UI and from kubectl.

---

Worker Architecture — Who Does the Work

Hatchet itself (hatchet-api, hatchet-engine) is the orchestrator — it does not execute business logic. The actual work is done by worker deployments in the application's own pods. For ASK these live in core-service:

┌─────────────────────────────────────────────────────────────────────────┐
│  ASK core-service (ask namespace)                                       │
│                                                                         │
│  bulk-operations-worker   × 3 replicas  ─────────────────────────────┐ │
│  billing-metering-worker  × 1 replica   ──────────────────────────┐  │ │
│  cron-scheduler-worker    × 1 replica   ───────────────────────┐  │  │ │
│  cron-run-worker          × 3 replicas  ────────────────────┐  │  │  │ │
│                                         (HPA: 3–20)         │  │  │  │ │
└─────────────────────────────────────────────────────────────┼──┼──┼──┼─┘
                                                              │  │  │  │
                          gRPC (TLS) over port 443            │  │  │  │
                          HATCHET__CLIENT_TOKEN auth          │  │  │  │
                                                              ▼  ▼  ▼  ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  Hatchet (ask namespace)                                                │
│                                                                         │
│  hatchet-engine  (gRPC :7070) ◄── receives workflow dispatch           │
│  hatchet-api     (HTTP :8080) ◄── REST for UI + token auth             │
│  rabbitmq                     ◄── task queue (AMQP)                    │
│  hatchet-pg (CNPG)            ◄── state store                          │
└─────────────────────────────────────────────────────────────────────────┘

Workers are long-running Deployments, not Jobs. They connect to Hatchet engine over gRPC on startup, register their workflow handlers, and then block waiting for tasks. Hatchet dispatches step executions to workers over that persistent gRPC connection.

---

Worker Inventory (ASK Core Service)

Worker	Replicas	HPA	Purpose
bulk-operations-worker	3	—	Bulk document/data operations (CSV, batch indexing)
billing-metering-worker	1	—	Usage metering, billing event processing
cron-scheduler-worker	1	—	Registers and manages cron-triggered workflow schedules
cron-run-worker	3–20	✓	Executes cron-triggered job runs

Worker capacity model

Each worker pod has workerSlots (configured in values) — the number of step executions it can handle concurrently. With 3 replicas × 5 slots, bulk-operations-worker handles 15 concurrent steps.

# core-service values.yaml
bulkOperationsWorker:
  replicaCount: 3
  workerSlots: 5          # concurrent step executions per pod
  maxRunsPerOrg: 20       # application-level throttle (not a Hatchet setting)
  resources:
    requests:
      cpu: 1000m
      memory: 4Gi
    limits:
      memory: 4Gi

cronRunWorker:
  replicaCount: 3
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 20       # scales out under load
  resources:
    requests:
      cpu: 500m
      memory: 2Gi
    limits:
      memory: 2Gi

---

Worker Authentication — How the Token is Wired

Workers authenticate to Hatchet using a worker API token (HATCHET__CLIENT_TOKEN). This is not a user JWT — it is a service-level token created during Hatchet setup.

Token generation (one-time, setup job)

When the Hatchet setup job runs hatchet-admin k8s quickstart, it: 1. Generates JWT signing keys + encryption keysets → stores in hatchet-config K8s secret 2. Creates a default worker token → stores in a K8s secret (chart-specific; typically hatchet-worker-token or written to hatchet-config)

How core-service workers receive the token

Vault KV: ask/core-service/<env>/HATCHET__CLIENT_TOKEN
              │
              ▼ (ESO ExternalSecret)
         K8s Secret: core-service-secret
         (namespace: ask)
              │
              ▼ (envFrom.secretRef in Deployment)
         env: HATCHET__CLIENT_TOKEN=<token>
              │
              ▼ (app config: config.toml)
         [ask.hatchet]
         client_token = "@format {env[HATCHET__CLIENT_TOKEN]}"

The token is never hardcoded — it lives in Vault and is pulled by ESO. All workers in a namespace share the same core-service-secret, which aggregates all env vars including the Hatchet token.

Getting the current token

# From Vault
export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
vault kv get secret/ask/core-service/auh1-dev/HATCHET__CLIENT_TOKEN

# From the K8s secret
kubectl -n ask get secret core-service-secret \
  -o jsonpath='{.data.HATCHET__CLIENT_TOKEN}' | base64 -d

Creating a new token via Hatchet UI

1. Open https://hatchet.mod.auh1.dev.dir
2. Navigate to Settings → API Tokens
3. Click Create Token → give it a name (e.g. core-service-auh1-dev)
4. Copy the token and store it in Vault:

vault kv put secret/ask/core-service/auh1-dev/HATCHET__CLIENT_TOKEN \
  value="<token-from-ui>"

1. Force ESO to refresh: kubectl -n ask annotate externalsecret core-service-secret \ force-sync=$(date +%s) --overwrite

---

gRPC Connection Config

Workers use two environment variables to reach the engine:

Env Var	Value in this deployment
HATCHET_CLIENT_TLS_STRATEGY	none (TLS terminated at ingress; gRPC is plain inside cluster)
HATCHET__CLIENT_TOKEN	worker API token

The gRPC endpoint is set in Hatchet's sharedConfig.grpcBroadcastAddress:

sharedConfig:
  grpcBroadcastAddress: "hatchet.mod.auh1.dev.dir:443"
  grpcInsecure: "f"

Workers connect to hatchet.mod.auh1.dev.dir:443 (through nginx ingress, TLS-terminated), then the engine routes their connections internally.

TLS strategy

HATCHET_CLIENT_TLS_STRATEGY: none does NOT mean no TLS. It means the Hatchet SDK does not manage its own mTLS — TLS is handled by the ingress layer (cert-manager / internal CA). This is the correct setting for an ingress-fronted deployment.

---

Job Scheduling — End-to-End Path

Trigger types

Trigger	Where configured	Example
API trigger (on-demand)	Application code calls Hatchet SDK .run()	User uploads a document → triggers bulk-index workflow
Cron	Hatchet UI or SDK .schedule()	Daily billing rollup at 02:00
Event-driven	SDK .on_event()	document.uploaded event triggers processing workflow

Scheduling path (step by step)

1. Trigger
   Application (or cron) → hatchet-api HTTP POST /api/v1/workflows/{id}/trigger

2. Dispatch
   hatchet-engine receives trigger → writes WorkflowRun row to PostgreSQL
   → pushes step execution message to RabbitMQ queue

3. Worker pickup
   Worker pod (long-running) has active gRPC stream open to hatchet-engine
   Engine pops message from RabbitMQ → pushes step dispatch to worker over gRPC
   (uses worker's registered handler for that workflow name)

4. Execution
   Worker runs the step function → reports SUCCESS/FAILURE back to engine over gRPC

5. State update
   Engine writes step result to PostgreSQL (step_runs, workflow_runs tables)
   → triggers next step in DAG (if multi-step)

6. Completion
   All steps complete → WorkflowRun marked SUCCEEDED or FAILED
   → optional webhook/event emitted

DAG (multi-step workflow) execution

Hatchet supports directed acyclic graph workflows where each step can depend on previous steps:

WorkflowRun
  └── Step 1: validate_input       (runs immediately)
        └── Step 2: process_data   (runs after Step 1 succeeds)
              ├── Step 3a: index    (runs in parallel after Step 2)
              └── Step 3b: notify  (runs in parallel after Step 2)

The engine manages the DAG state in PostgreSQL and dispatches each step to available workers as its dependencies complete.

---

Observability — What to Watch

1. Hatchet UI (https://hatchet.mod.auh1.dev.dir)

Section	What it shows
Dashboard	Active workflow runs, recent completions, failure rate
Workflows	All registered workflow definitions + last run status
Runs	All WorkflowRun instances — filter by workflow, status, date
Workers	Active worker connections — which pods are registered, heartbeat time, slots available
Events	Event log — all triggers (cron, API, event-driven)
Crons	Active cron schedules — last run, next run, status
Settings → API Tokens	Manage worker tokens

Workers page is the first place to check when jobs stop processing — it shows whether any worker pods are actually connected. Zero workers = no dispatch possible.

2. kubectl — live state

# All hatchet-related pods
kubectl -n ask get pods | grep -E "hatchet|bulk-operations|billing-metering|cron"

# Worker pod logs (real-time step execution)
kubectl -n ask logs -l app.kubernetes.io/name=bulk-operations-worker -f --tail=100

# Is the worker connected? (look for "worker registered" / "heartbeat" in logs)
kubectl -n ask logs deployment/core-service-bulk-operations-worker --tail=50 | grep -i "hatchet\|worker\|registered\|heartbeat"

# Engine logs — step dispatch + RabbitMQ consumption
kubectl -n ask logs -l app.kubernetes.io/name=engine -f --tail=100

# API logs — incoming triggers
kubectl -n ask logs -l app.kubernetes.io/name=api -f --tail=50

# RabbitMQ queue depths (backlog check)
kubectl -n ask exec hatchet-rabbitmq-0 -- rabbitmqctl list_queues name messages consumers

3. PostgreSQL — workflow run state

# Connect to hatchet DB
kubectl -n ask exec hatchet-pg-1 -- psql -U postgres -d hatchet

-- Recent workflow runs (last 1 hour)
SELECT id, workflow_version_id, status, created_at, finished_at,
       EXTRACT(EPOCH FROM (finished_at - created_at))::int AS duration_s
FROM workflow_runs
WHERE created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC
LIMIT 20;

-- Failed runs
SELECT wr.id, wv.workflow_id, wr.status, wr.error, wr.created_at
FROM workflow_runs wr
JOIN workflow_versions wv ON wr.workflow_version_id = wv.id
WHERE wr.status = 'FAILED'
ORDER BY wr.created_at DESC
LIMIT 10;

-- Step run durations (spot slow steps)
SELECT sr.id, sr.step_id, sr.status, sr.started_at,
       EXTRACT(EPOCH FROM (sr.finished_at - sr.started_at))::int AS duration_s
FROM step_runs sr
WHERE sr.started_at > NOW() - INTERVAL '1 hour'
ORDER BY duration_s DESC NULLS LAST
LIMIT 20;

-- Active (running) step runs
SELECT sr.id, sr.step_id, sr.status, sr.started_at,
       EXTRACT(EPOCH FROM (NOW() - sr.started_at))::int AS age_s
FROM step_runs sr
WHERE sr.status = 'RUNNING'
ORDER BY age_s DESC;

-- Queue depth (pending steps not yet dispatched)
SELECT COUNT(*) AS pending_steps
FROM step_runs
WHERE status = 'PENDING';

4. RabbitMQ — queue health

# Management UI (port-forward — no ingress in this deployment)
kubectl -n ask port-forward svc/hatchet-rabbitmq 15672:15672 &
# Open: http://localhost:15672  login: hatchet / <from hatchet-secret>

# CLI checks
kubectl -n ask exec hatchet-rabbitmq-0 -- rabbitmqctl list_queues name messages consumers ready

# Memory and alarm state
kubectl -n ask exec hatchet-rabbitmq-0 -- rabbitmqctl status | grep -E "memory|alarm|watermark"

# Connections from workers + engine
kubectl -n ask exec hatchet-rabbitmq-0 -- rabbitmqctl list_connections peer_host peer_port state channels

Healthy RabbitMQ output (list_queues):

name                                    messages  consumers
hatchet.task.queue.bulk-operations      0         3
hatchet.task.queue.cron-run             0         3
hatchet.task.queue.billing              0         1

messages = 0 with consumers > 0 = queue being drained normally. messages > 0 with consumers = 0 = workers disconnected — queue backing up.

---

Common Issues — Workers

Symptom	Diagnosis	Fix
Workers page in UI shows no workers	Worker pods not connecting — check token or gRPC address	kubectl logs deployment/bulk-operations-worker | grep -i error; verify HATCHET__CLIENT_TOKEN is valid
Workflow runs stuck in RUNNING	Worker registered but step not completing	Check worker pod logs for the step; look for panics or timeouts
Queue depth growing	Workers too slow or crashed; check list_queues	Scale replicas or check worker resource limits (OOM)
HATCHET_CLIENT_TLS_STRATEGY error	Misconfigured TLS strategy	Set HATCHET_CLIENT_TLS_STRATEGY: none in worker extraEnv
Token 401 Unauthorized	Worker token revoked or wrong env pointing to different Hatchet	Re-generate token via UI, update Vault, force ESO sync
Cron not firing	cron-scheduler-worker crashed or not registered	kubectl get pods | grep cron-scheduler; check it shows Running

---

Viewing Registered Workers — UI Walkthrough

1. Open https://hatchet.mod.auh1.dev.dir
2. Login with admin@example.com / Admin123!!
3. Left sidebar → Workers
4. Each row = one worker pod. Columns:
5. Name: worker process name (set by the SDK, typically {service}-{hostname})
6. Status: ACTIVE (heartbeating) or INACTIVE (disconnected)
7. Last Heartbeat: how long ago the pod checked in
8. Max Runs: workerSlots value for that worker
9. Available Slots: slots not currently executing a step

Correlate pods to workers

The worker Name in the UI includes the pod name (hostname). Use this to correlate a specific worker slot to a specific pod: kubectl -n ask get pods -o wide | grep bulk-operations.

---

Cron Schedule Management

Cron schedules are registered by cron-scheduler-worker on startup. They are stored in Hatchet's PostgreSQL, not as Kubernetes CronJobs.

# View active cron schedules via psql
kubectl -n ask exec hatchet-pg-1 -- psql -U postgres -d hatchet \
  -c "SELECT cron_name, cron_expr, last_run, next_run, status
      FROM workflow_cron_triggers
      ORDER BY next_run;"

Via UI: Left sidebar → Crons — shows name, expression, last/next run, and recent run status.

To trigger a cron workflow manually (without waiting for the schedule):

# Via Hatchet REST API (get workflow ID from UI → Workflows)
curl -s -X POST \
  -H "Authorization: Bearer <HATCHET__CLIENT_TOKEN>" \
  -H "Content-Type: application/json" \
  https://hatchet.mod.auh1.dev.dir/api/v1/workflows/<workflow-id>/trigger \
  -d '{"input": {}}'