Deploy OpenSearch for KMS Lexical Indexing (Air-Gap)¶

KMS's document-ingestion pipeline has a lexical (BM25) indexing step that requires an OpenSearch backend. This page documents deploying a single-node OpenSearch via ArgoCD and wiring KMS to it — the fix that unblocked live document ingestion on EDGE #2 (mod-auh1-dev-ask).

Why this exists (the trigger)

The pinned KMS 2026-07-01 build (fc2911…, git 98efd55) has a bug in the live-ingestion workflow: hatchet_workers/tasks/live_ingestion_tasks.py live_index_chunks_for_search uses skip_if=[small_file, skip_lexical_indexing] (AND-combined). For normal-sized files small_file=False, so the step does not skip even when OpenSearch is absent (skip_lexical_indexing=True) → it runs → dies with 424: OpenSearch is not configured → document ends PROCESSING_FAILED (and the retryable error loops forever). The batch workflow is correct (gates on skip_lexical_indexing alone).

OpenSearch is optional by design — the "flag" is simply whether config.opensearch is set. Deploying it makes config.opensearch non-None → the lexical step runs and succeeds instead of hitting the buggy skip. The real fix is an upstream skip_if correction; until that lands, this deploy is the workaround. See Removal path.

Architecture¶

KMS live-ingestion worker (ask ns)
   │  boto3-style OpenSearch client (opensearch-py 3.2.0)
   │  http://opensearch.ask.svc.cluster.local:9200   (no TLS, no auth — dev-grade)
   ▼
Service/opensearch (ask ns, ClusterIP :9200)
   ▼
StatefulSet/opensearch  (single node, opensearchproject/opensearch:2.19.0)
   └── PVC data (csi-rbd-sc, 10Gi)   ← index "kms-lexical-chunks" persists here

Delivered as an ArgoCD Application (mod-auh1-dev-ask.ask.opensearch) under the ask app-of-apps — same GitOps pattern as every other service.

GitOps layout (mod-ask-devops)¶

File	Purpose
`manifests/opensearch/opensearch.yaml`	`Service` + `StatefulSet` (single node) + `volumeClaimTemplate`
`environment/auh1/dev/02-argocd-application/ask/argocd-application-opensearch.yaml`	ArgoCD `Application` (directory source → `manifests/opensearch`), namespace `ask`

The parent app-of-apps mod-auh1-dev-ask.ask recursively syncs .../ask/*.yaml, so committing the Application file auto-creates + syncs it. No manual kubectl apply / argocd app create.

Key manifest choices (dev-grade — see Production considerations)¶

Setting	Value	Why
Replicas	`1` (single node)	dev scale; prod = 3-node with dedicated masters
`DISABLE_SECURITY_PLUGIN`	`true`	HTTP, no auth — avoids TLS/internal-user setup (prod = cert-manager TLS + users)
`node.store.allow_mmap`	`false`	avoids the `vm.max_map_count` sysctl → no privileged initContainer → satisfies the `ask` namespace baseline PSA
`discovery.type`	`single-node`	no cluster bootstrap needed
`OPENSEARCH_JAVA_OPTS`	`-Xms512m -Xmx512m`	modest heap
Storage	`csi-rbd-sc`, 10Gi RWO	index persists across restarts
securityContext	runAsNonRoot 1000, drop ALL caps, seccomp RuntimeDefault	baseline-PSA compliant
Probes	`GET /_cluster/health` (readiness waits for `yellow`)	single-node health is `yellow` (1 unassigned replica) — that's normal

KMS wiring — the `[opensearch]` config block¶

Added to the KMS chart envConf (mod-ask-knowledge-management-service/helm/environments/values-mod-auh1-dev-ask.yaml, the envConf.configToml string). Fields match src/config/schemas/opensearch.py:

[mod-auh1-dev-ask.opensearch]
host = "opensearch.ask.svc.cluster.local"
port = 9200
scheme = "http"
username = "@none"           # dynaconf None; security disabled so no auth
password = "@none"
verify_certs = false
index_name = "kms-lexical-chunks"
number_of_shards = 1
number_of_replicas = 0       # single node → 0 replicas keeps the index green
analyzer = "standard"
query_timeout_sec = 30
retry_attempts = 3
retry_wait_time = 2
bulk_index_batch_size = 100

The index (kms-lexical-chunks) is auto-created by KMS on first lexical-indexing run — no manual bootstrap needed.

Deploy procedure¶

1. Mirror the image (air-gap — from the bastion)¶

skopeo, through the proxy, Docker Hub → Harbor. Use --dest-creds, not --dest-authfile (the Harbor robot JSON is a robot export, not a docker config.json):

cd ~/harbor-images
HCREDS=$(python3 -c "import json,base64; d=json.load(open('harbor-robot.json')); a=d.get('auths'); print(base64.b64decode(list(a.values())[0]['auth']).decode() if a else (d.get('name') or d.get('username'))+':'+(d.get('secret') or d.get('password')))")
HTTPS_PROXY=http://10.96.137.37:3128 HTTP_PROXY=http://10.96.137.37:3128 \
NO_PROXY=harbor.cl1.sq4.aegis.internal,.cl1.sq4.aegis.internal,100.115.0.0/16,.svc,.cluster.local \
skopeo copy --dest-tls-verify=false --dest-creds "$HCREDS" \
  docker://docker.io/opensearchproject/opensearch:2.19.0 \
  docker://harbor.cl1.sq4.aegis.internal/dockerhub/opensearchproject/opensearch:2.19.0

Also add the image to scripts/harbor-mirror/mirror-config.yaml so future re-mirrors keep it.

2. Commit the manifests → ArgoCD deploys¶

Commit manifests/opensearch/opensearch.yaml + argocd-application-opensearch.yaml, push, then let the parent reconcile (or force it):

kubectl -n argocd annotate application mod-auh1-dev-ask.ask argocd.argoproj.io/refresh=hard --overwrite
kubectl -n argocd get application mod-auh1-dev-ask.ask.opensearch
kubectl get statefulset,svc,pvc,pod -n ask | grep -i opensearch   # opensearch-0 → 1/1 Running

3. Wire KMS + restart workers¶

Commit the [opensearch] block to the KMS chart values, then sync + restart (the ConfigMap must update AND the workers must reload it):

# force the (multi-source) KMS app to sync the new commit
kubectl -n argocd patch application mod-auh1-dev-ask.ask.knowledge-management-service --type merge \
  -p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}'
kubectl -n argocd annotate application mod-auh1-dev-ask.ask.knowledge-management-service argocd.argoproj.io/refresh=hard --overwrite
# confirm the ConfigMap picked it up (>0)
kubectl get cm -n ask knowledge-management-service-config-toml -o yaml | grep -c opensearch
# reload it into the running workers
kubectl rollout restart deploy -n ask -l app.kubernetes.io/instance=knowledge-management-service

4. Verify end-to-end¶

Upload a multi-page PDF (≥ the small_file threshold — a 50-page book works), then:

kubectl logs -n ask deploy/knowledge-management-service-live-ingestion-worker -f \
  | grep -iE "lexical|opensearch|index|ready|error"

Success looks like:

opensearch | HEAD .../kms-lexical-chunks [status:200 ...]
opensearch | POST .../_bulk?refresh=wait_for [status:200 ...]
Indexed 406/406 chunks into lexical index kms-lexical-chunks
Updating Document ... processing_status → READY

Operations¶

# cluster health (yellow is normal for single node)
kubectl exec -n ask deploy/knowledge-management-service-live-ingestion-worker -- \
  python3 -c "import urllib.request;print(urllib.request.urlopen('http://opensearch.ask.svc.cluster.local:9200/_cluster/health',timeout=5).read().decode())"

# index doc count
#   GET http://opensearch.ask.svc.cluster.local:9200/kms-lexical-chunks/_count
kubectl logs statefulset/opensearch -n ask --tail=50    # OpenSearch server logs

Data durability: the kms-lexical-chunks index lives on the PVC (csi-rbd-sc), so it survives pod restarts. Deleting the PVC loses the lexical index (docs would need re-ingestion to rebuild it).
Availability: single node — if opensearch-0 restarts, lexical indexing pauses (~1 min) and new ingestions fail the lexical step until it's back. Existing vector search (Weaviate) is unaffected.

Production considerations¶

This is a documented dev-grade deviation

Concern	This deploy (dev)	Production-correct
HA	single node	3-node cluster, dedicated cluster-manager nodes, `number_of_replicas ≥ 1`
Security	plugin disabled, HTTP, no auth	security plugin on, TLS (cert-manager), internal users / RBAC, `scheme=https`, `verify_certs=true` + CA trust
Resources	512m heap	sized to corpus; `bootstrap.memory_lock=true` + memlock ulimits
mmap	`allow_mmap=false` (perf cost)	`vm.max_map_count=262144` at node level, mmap enabled
PDB / anti-affinity	none	PodDisruptionBudget + pod anti-affinity across nodes

Removal path (once the upstream fix lands)¶

OpenSearch is only here to work around the live-workflow skip_if bug. Report it to the KMS team:

live_ingestion_tasks.py live_index_chunks_for_search ANDs small_file with skip_lexical_indexing; it should gate on skip_lexical_indexing alone (like the batch workflow), and OPENSEARCH not configured should be non-retryable.

When a corrected build is pinned, revert to vector-only: 1. Remove the [mod-auh1-dev-ask.opensearch] block from the KMS chart values. 2. Delete manifests/opensearch/ + argocd-application-opensearch.yaml from mod-ask-devops. 3. ArgoCD prunes the StatefulSet/Service; delete the PVC + opensearch leftovers if any.

Gotchas encountered (for next time)¶

KMS app stuck OutOfSync (rev= empty): auto-sync had been left off from the earlier graceful-shutdown exercise. The operation.sync patch with a revision field no-ops on multi-source apps — instead patch spec.syncPolicy.automated={prune,selfHeal} + hard refresh, then rollout restart the workers so they reload the updated ConfigMap.
skopeo --dest-authfile → authentication required: the Harbor robot JSON is a robot export ({name, secret}), not a docker config.json. Use --dest-creds "user:pass".
ArgoCD excludes Endpoints/EndpointSlice (unrelated, but same repo): if you ever front an external service, declare an EndpointSlice and apply it out-of-band — ArgoCD won't manage it.