Skip to content

Hatchet Wiring — Activate Async Jobs for core-service

core-service offloads long-running work (bulk user invites, cron jobs, scheduled tasks) to Hatchet. The Hatchet engine is already deployed in ask; this page wires core-service to it by providing a real client token and (optionally) enabling the worker deployments.

How core-service uses Hatchet

  • The client is lazy — get_hatchet_client() builds it on first use, not at startup (src/core_service/hatchet/client.py). So a placeholder token doesn't break boot; it fails only when a feature dispatches a job.
  • Connection: gRPC to hatchet-engine:7070, TLS strategy none (plaintext, in-cluster).
  • Credential: config.hatchet.client_token ← @format {env[HATCHET__CLIENT_TOKEN]} ← core-service-secret ← Vault. Required (min_length=1) whenever Hatchet is exercised.

Prerequisites

  • [x] core-service deployed + healthy in ask
  • [x] Hatchet engine running: kubectl get svc -n ask hatchet-engine hatchet-api hatchet-frontend
  • [x] Vault reachable + unsealed

Step 1 — Obtain a Hatchet client token

Hatchet often mints a default-tenant API token at bootstrap. Check for it first:

kubectl get secret -n ask | grep -i hatchet
# inspect a likely candidate (token / client-token keys):
kubectl get secret -n ask <hatchet-token-secret> -o jsonpath='{.data}' | jq 'keys'

If a usable token exists, decode it: 

kubectl get secret -n ask <secret> -o jsonpath='{.data.<TOKEN_KEY>}' | base64 -d; echo

Otherwise mint one from the Hatchet dashboard:

  1. Add hatchet.ask.mod.auh1.dev.dir (or the Hatchet ingress host) to /etc/hosts → ingress IP.
  2. Open the Hatchet frontend, select the tenant core-service should use.
  3. Settings → API Tokens → Create → copy the token (shown once).

Tenant alignment

The token is tenant-scoped. Use the same tenant the workers run in. In a single-tenant air-gapped setup that's the default tenant.

Verify: you have a non-empty Hatchet token string.

Step 2 — Wire the token into Vault

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
vault kv put secret/ask/core-service/auh1-dev/HATCHET__CLIENT_TOKEN value="<PASTE-TOKEN>"

# force ESO to re-pull (else refreshes within 1m):
kubectl annotate externalsecret -n ask core-service-secret force-sync="$(date +%s)" --overwrite

The core-service-secret ExternalSecret already maps this path to the HATCHET__CLIENT_TOKEN key (property: value) — no manifest change needed.

Verify: 

kubectl get secret -n ask core-service-secret -o jsonpath='{.data.HATCHET__CLIENT_TOKEN}' | base64 -d | head -c 10; echo '…'
Should not be placeholder. The Reloader restarts the pod on the secret change.

Step 3 — (Optional) enable the worker deployments

The API dispatches jobs; the workers execute them. Workers are disabled by default. Enable the ones you need in both values files (helm/environments/... and mod-ask-devops/...):

bulkOperationsWorker:
  enabled: true          # processes bulk user invites
  image:
    repository: harbor.cl1.sq4.aegis.internal/ask/core-service-bulk-operations-worker
    tag: "staging"
# cronSchedulerWorker / cronRunWorker / billingMeteringWorker similarly

The workers read the same core-service-secret (so they get HATCHET__CLIENT_TOKEN too) and run on the ask71 worker nodes. Mind their resource requests (bulk worker requests 1 CPU / 4Gi) — confirm headroom before enabling several.

Commit + push both files, then re-sync: 

kubectl -n argocd patch application mod-auh1-dev-ask.ask.core-service --type merge \
  -p '{"operation":{"initiatedBy":{"username":"manual"},"sync":{"revision":"HEAD"}}}'

Verify: worker pods reach Running: 

kubectl get pods -n ask -l app.kubernetes.io/instance=core-service | grep worker

Step 4 — Validate

Trigger a feature that dispatches to Hatchet (e.g. a bulk invite), then check both sides:

# core-service: client initialized, no token error
kubectl logs -n ask deploy/core-service --tail=40 | grep -iE "hatchet|HATCHET_CLIENT_TOKEN|dispatch"
# hatchet engine: received the run
kubectl logs -n ask deploy/hatchet-engine --tail=40 | grep -iE "run|workflow|task" | head

Done when: core-service logs Hatchet client initialized (not HATCHET_CLIENT_TOKEN not configured) and the engine shows the workflow run.

What's stored where

Value Source of truth Delivered via Consumed as
Hatchet token Vault secret/ask/core-service/auh1-dev/HATCHET__CLIENT_TOKEN ExternalSecret → core-service-secret env HATCHET__CLIENT_TOKEN → @format → hatchet.client_token

Production / air-gap notes

  • TLS is none (in-cluster plaintext gRPC). Acceptable inside the cluster mesh; if you front Hatchet with TLS, the SDK strategy would need to change in hatchet/client.py.
  • Rotate the token on a schedule; on rotation, re-run Step 2 only.
  • Keep workers disabled until you actually need async throughput — they're heavy.