Hatchet Workflow Engine Deployment¶

Deploys Hatchet (hatchet-stack chart 0.10.0, app v0.71.0) into the ask namespace via ArgoCD. Hatchet provides distributed background job processing for the ASK platform.

Access: https://hatchet.mod.auh1.dev.dir — login: admin@example.com / Admin123!!

Architecture¶

ask namespace
├── hatchet-api          (REST API + auth — port 8080)
├── hatchet-engine       (workflow scheduler/executor — gRPC 7070)
├── hatchet-frontend     (UI — port 80)
├── setup job            (runs once: hatchet-admin k8s quickstart → creates hatchet-config)
├── worker-token job     (runs once: creates a default worker token)
├── rabbitmq             (bundled bitnami subchart — AMQP message broker)
└── hatchet-pg (CNPG)   (3-node PostgreSQL cluster)

Vault KV v2 (mount: secret)
├── ask/postgres/hatchet-pg--SUPERUSER--PASSWORD
├── ask/postgres/hatchet--DATABASE--PASSWORD
├── ask/hatchet/rabbitmq--PASSWORD
└── ask/hatchet/rabbitmq--ERLANG--COOKIE

Secret flow:

Vault ──► ESO PushSecrets (wave 0) ──► generate & store passwords
      ──► ESO ExternalSecrets (wave 1) ──► K8s secrets
                                              ├── hatchet-pg-superuser  (CNPG superuserSecret)
                                              ├── hatchet-db-password   (CNPG managed role)
                                              ├── hatchet-secret        (api+engine envFrom + rabbitmq auth)
                                              └── hatchet-rabbitmq-erlang (rabbitmq Erlang cookie)

hatchet-secret contains three keys — it overrides the chart defaults in hatchet-shared-config:

Key	Value
`DATABASE_URL`	`postgresql://hatchet:<pass>@hatchet-pg-rw.ask.svc.cluster.local:5432/hatchet`
`RABBITMQ_PASSWORD`	generated password
`SERVER_TASKQUEUE_RABBITMQ_URL`	`amqp://hatchet:<pass>@hatchet-rabbitmq:5672/`

Sync-wave order:

Wave	Resource	Purpose
0	PushSecrets	Generate passwords → Vault (IfNotExists)
1	ExternalSecrets	Pull from Vault → K8s secrets
2	CNPG Cluster	Starts after passwords exist

Prerequisites¶

1 — Mirror images to Harbor¶

cd ~/harbor-mirror
python3 core42-mirror-image.py --config mirror-config.yaml \
  --only hatchet-dev/hatchet/hatchet-api \
         hatchet-dev/hatchet/hatchet-engine \
         hatchet-dev/hatchet/hatchet-frontend \
         hatchet-dev/hatchet/hatchet-migrate \
         hatchet-dev/hatchet/hatchet-admin

Source	Harbor
`ghcr.io/hatchet-dev/hatchet/hatchet-api:v0.71.0`	`harbor.../ghcr/hatchet-dev/hatchet/hatchet-api:v0.71.0`
`ghcr.io/hatchet-dev/hatchet/hatchet-engine:v0.71.0`	`harbor.../ghcr/hatchet-dev/hatchet/hatchet-engine:v0.71.0`
`ghcr.io/hatchet-dev/hatchet/hatchet-frontend:v0.71.0`	`harbor.../ghcr/hatchet-dev/hatchet/hatchet-frontend:v0.71.0`
`ghcr.io/hatchet-dev/hatchet/hatchet-migrate:v0.71.0`	`harbor.../ghcr/hatchet-dev/hatchet/hatchet-migrate:v0.71.0`
`ghcr.io/hatchet-dev/hatchet/hatchet-admin:v0.71.0`	`harbor.../ghcr/hatchet-dev/hatchet/hatchet-admin:v0.71.0`
`docker.io/bitnamilegacy/rabbitmq:4.1.3-debian-12-r1`	`harbor.../bitnamilegacy/rabbitmq:4.1.3-debian-12-r1`
`docker.io/postgres:16-alpine`	`harbor.../dockerhub/postgres:16-alpine`

hatchet-admin is required

hatchet-admin runs k8s quickstart inside the setup job to generate JWT signing keys and encryption keysets, storing them in the hatchet-config K8s secret. Without it the api and engine pods fail with secret "hatchet-config" not found.

2 — Mirror and push Helm chart¶

cd ~/harbor-mirror
python3 helm-charts/core42-mirror-charts.py \
  -c helm-charts/mirror-charts.yaml --only hatchet-stack

python3 helm-charts/gitlab-push-charts.py --only hatchet-stack

3 — Verify Vault policy covers ask/hatchet/*¶

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
vault login $(jq -r '.root_token' ~/hashicorp_vault/vault-main-init-mod-auh1-dev.json)
vault policy read eso-ask-push

Expected output must include:

path "secret/data/ask/hatchet/*"      { capabilities = ["create", "update", "read"] }
path "secret/metadata/ask/hatchet/*"  { capabilities = ["read", "list"] }

If missing: argocd app sync vault --grpc-web

4 — Seed new Vault paths (first-time only)¶

Seed before first ArgoCD sync

ESO PushSecrets with updatePolicy: IfNotExists fail on brand-new Vault paths with invalid JSON format (Vault returns {"errors":[]} on 404). Seed each new path once.

vault kv put secret/ask/hatchet/rabbitmq--PASSWORD \
  password="$(openssl rand -hex 16)"

vault kv put secret/ask/hatchet/rabbitmq--ERLANG--COOKIE \
  password="$(openssl rand -hex 32)"

5 — Create the hatchet database in CNPG (first-time only)¶

initdb only runs on first bootstrap

initdb.database: hatchet in the CNPG cluster spec only executes when the cluster is first created on an empty data directory. If the cluster already existed when you added this config, the database was never created and the setup job will fail at check-db-connection.

Check and create if needed:

# Verify database exists
kubectl -n ask exec hatchet-pg-1 -- psql -U postgres -c "\l hatchet"

# If missing — create it (hatchet role is created by CNPG managed roles)
kubectl -n ask exec hatchet-pg-1 -- psql -U postgres \
  -c "CREATE DATABASE hatchet OWNER hatchet;"

Deployment¶

Files in mod-ask-devops¶

File	Purpose
`applications/ask-db/hatchet/hatchet-pg-cluster.yaml`	CNPG 3-node cluster — `initdb.database: hatchet`, owner `hatchet`
`applications/ask-db/hatchet/hatchet-pg-credentials.yaml`	PushSecrets + ExternalSecrets (hatchet-secret, erlang cookie)
`applications/hatchet/values-mod-auh1-dev-ask.yaml`	Helm values — images, ingress, jobs, RabbitMQ
`environment/auh1/dev/02-argocd-application/ask/argocd-application-hatchet.yaml`	ArgoCD Application (`releaseName: hatchet`)

Key values — jobs¶

api:
  setupJob:
    enabled: true          # enables the Job resource
    image:
      repository: harbor.cl1.sq4.aegis.internal/ghcr/hatchet-dev/hatchet/hatchet-admin
      tag: v0.71.0
  quickstartJob:
    enabled: true          # enables hatchet-admin k8s quickstart → creates hatchet-config secret
  migrationJob:
    enabled: true          # runs DB migrations as initContainer
  seedJob:
    enabled: true          # seeds admin user
  postgresImage:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/postgres
    tag: "16-alpine"       # needed by check-db-connection initContainer

Key values — RabbitMQ¶

rabbitmq:
  image:
    registry: harbor.cl1.sq4.aegis.internal   # registry and repository must be split
    repository: bitnamilegacy/rabbitmq         # bitnami chart splits these separately
    tag: "4.1.3-debian-12-r1"
  global:
    imagePullSecrets:
      - harbor-pull-secret
    security:
      allowInsecureImages: true                # required for bitnamilegacy images
  auth:
    existingPasswordSecret: "hatchet-secret"
    existingSecretPasswordKey: "RABBITMQ_PASSWORD"
    existingErlangSecret: "hatchet-rabbitmq-erlang"  # pinned cookie — prevents sync breakage
    existingSecretErlangKey: "rabbitmq-erlang-cookie"

ArgoCD Application¶

# releaseName is required — without it ArgoCD uses the app name (contains dots)
# which makes Kubernetes service/ingress names invalid (DNS-1035 violation)
helm:
  releaseName: hatchet
  valueFiles:
    - $values/applications/hatchet/values-mod-auh1-dev-ask.yaml

# SYS_PTRACE in setup-job requires privileged PSA
# Production fix: deploy to dedicated namespace (prod reference uses namespace: hatchet)
managedNamespaceMetadata:
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: baseline

Sync order¶

git push origin main

# 1 — Credentials (PushSecrets → CNPG)
argocd app sync mod-auh1-dev-ask.ask.postgresql.hatchet --grpc-web

# 2 — Hatchet chart
argocd app sync mod-auh1-dev-ask.ask.hatchet --grpc-web --prune

Verification¶

# All pods running
kubectl -n ask get pods | grep hatchet
# Expected: api 1/1, engine 1/1, frontend 1/1, rabbitmq 1/1, pg-1/2/3 1/1
# Jobs: Completed (not Failed)

# ExternalSecrets synced
kubectl -n ask get externalsecret | grep hatchet

# hatchet-secret contains all 3 keys
kubectl -n ask get secret hatchet-secret \
  -o jsonpath='{.data}' | python3 -m json.tool | grep -o '"[A-Z_]*"'

# Ingress
kubectl -n ask get ingress | grep hatchet

UI: https://hatchet.mod.auh1.dev.dir

Default login: admin@example.com / Admin123!!

Change default credentials after first login

The default admin password is set via sharedConfig.defaultAdminPassword in values. Change it in the Hatchet UI after first login — do not leave the default in a long-running environment.

Credential Summary¶

Secret	Vault Path	K8s Secret	Consumer
CNPG superuser	`ask/postgres/hatchet-pg--SUPERUSER--PASSWORD`	`hatchet-pg-superuser`	CNPG `superuserSecret`
DB user password	`ask/postgres/hatchet--DATABASE--PASSWORD`	`hatchet-db-password`	CNPG managed role
App secret	`ask/postgres/hatchet--DATABASE--PASSWORD` + `ask/hatchet/rabbitmq--PASSWORD`	`hatchet-secret`	api + engine `envFrom`, rabbitmq `existingPasswordSecret`
Erlang cookie	`ask/hatchet/rabbitmq--ERLANG--COOKIE`	`hatchet-rabbitmq-erlang`	rabbitmq `existingErlangSecret`

Getting the worker/client token¶

Client services (core-service, assistant-service) connect to Hatchet with a client token — a JWT tenant-scoped to the engine. The chart's worker-token job mints a default-tenant token at setup and stores it in the K8s secret hatchet-client-config, key HATCHET_CLIENT_TOKEN.

Read the existing token (preferred — reuse the default-tenant token):

# confirm the key
kubectl get secret -n ask hatchet-client-config -o jsonpath='{.data}' | jq 'keys'   # → ["HATCHET_CLIENT_TOKEN"]

# decode it (a JWT — starts with eyJ)
kubectl get secret -n ask hatchet-client-config -o jsonpath='{.data.HATCHET_CLIENT_TOKEN}' | base64 -d; echo

Mint a fresh token (if you need a new one / a different tenant):

# list tenants, then create a token for one
kubectl exec -n ask deploy/hatchet-api -- /hatchet-admin token create --tenant-id <tenant-id>

Or via the Hatchet dashboard → Settings → API Tokens → Create (add the frontend host to /etc/hosts → ingress IP first).

It must be a JWT

The Hatchet SDK's ClientConfig validates the token is a JWT (eyJ..., three dot-separated segments) and rejects anything else. A random placeholder (e.g. openssl rand -hex 32) fails immediately — and workers that build the client at import crash at startup, so the real token is required before enabling them. Use the value from hatchet-client-config.

Wiring it into a consumer (e.g. assistant-service) — copy the JWT into that service's Vault path:

TOK=$(kubectl get secret -n ask hatchet-client-config -o jsonpath='{.data.HATCHET_CLIENT_TOKEN}' | base64 -d)
vault kv patch secret/ask/assistant-service/auh1-dev HATCHET__CLIENT_TOKEN="$TOK"   # or .../core-service/...
kubectl annotate externalsecret -n ask <service>-secret force-sync="$(date +%s)" --overwrite
kubectl rollout restart deploy -n ask -l app.kubernetes.io/instance=<service>

Troubleshooting¶

Symptom	Cause	Fix
`PushSecret: invalid JSON format`	Brand-new Vault path returns `{"errors":[]}` (404) — ESO cannot parse	Seed path: `vault kv put secret/<path> password="$(openssl rand -hex 16)"` then force-sync
`permission denied` on `ask/hatchet/*`	ESO role missing `eso-ask-push` policy	`argocd app sync vault --grpc-web` — PostSync bootstrap job rewrites policy
`secret "hatchet-config" not found`	`api.quickstartJob.enabled` defaults to `false` — Job runs empty alpine container, never runs `hatchet-admin k8s quickstart`	Set `api.quickstartJob.enabled: true` + `api.setupJob.enabled: true` in values
Setup job pods forbidden: `SYS_PTRACE` violates PodSecurity `baseline`	Chart hardcodes `SYS_PTRACE` in setup-job containers; blocked by namespace `baseline` PSA	Set `pod-security.kubernetes.io/enforce: privileged` on namespace. Production: use dedicated `hatchet` namespace
Setup job image pull fails	Default chart images point to `ghcr.io` — blocked in air-gap	Override `api.setupJob.image`, `api.seedJob.image`, `api.migrationJob.image`, `api.postgresImage` to Harbor
`Ingress … DNS-1035` service name invalid	ArgoCD app name contains dots; used as Helm release name	Set `helm.releaseName: hatchet` in ArgoCD Application
`check-db-connection` exits 2: `database "hatchet" does not exist`	CNPG `initdb.database` only runs on first bootstrap	`kubectl -n ask exec hatchet-pg-1 -- psql -U postgres -c "CREATE DATABASE hatchet OWNER hatchet;"`
Engine: `Exception (403): username or password not allowed` on RabbitMQ	`hatchet-shared-config` hardcodes `SERVER_TASKQUEUE_RABBITMQ_URL=amqp://hatchet:hatchet@…`	Add `SERVER_TASKQUEUE_RABBITMQ_URL` to `hatchet-secret` ExternalSecret template — loaded after `hatchet-shared-config` in `envFrom` so it overrides
RabbitMQ image: `docker.io/harbor…` not found	Bitnami chart prepends `docker.io/` when full hostname is in `repository`	Split into `image.registry: harbor.cl1.sq4.aegis.internal` + `image.repository: bitnamilegacy/rabbitmq`
RabbitMQ `allowInsecureImages` error	Bitnami chart rejects `bitnamilegacy` images without explicit opt-in	Add `rabbitmq.global.security.allowInsecureImages: true`