40 — KMS S3 Wiring (Ceph RGW object storage)¶

KMS stores uploaded source documents and processed artifacts in S3-compatible object storage. In the air-gapped environment that's Ceph RGW (s3.cl1.sq4.aegis.internal, external to the cluster). This page is the end-to-end wiring of the credential chain and how to test it. The bucket ask-knowledge-management and the ask-kms user are the R1 pre-engagement requirement in 37 — KMS Deployment.

Verified 2026-06-17

Round-trip PUT/GET/LIST/DELETE from the ingestion worker against ask-knowledge-management succeeded with the real ask-kms keys (access len=20, secret len=40) — the full chain Vault → ESO → secret → pod env → boto3 → Ceph RGW works. The earlier InvalidAccessKeyId was the bring-up placeholder before the real keys were patched.

How KMS uses S3

The ingestion and live-ingestion workers write the raw uploaded file to the bucket and read it back during the parse → chunk → embed pipeline. The API enqueues; the workers do the I/O.
The client is boto3 (src/knowledge_management/config → [storage.s3]): endpoint_url, access_key_id, secret_access_key, region, verify_ssl, bucket_name.
S3 is lazy — pods boot fine without working keys; only an actual document upload fails. That is why bring-up seeds a placeholder (to satisfy the @format requirement) and the real keys are patched later.

The credential chain (what wires to what)¶

flowchart LR
    CEPH["Ceph RGW\nuser: ask-kms\n(access_key + secret_key)"]
    V["Vault KV v2\nsecret/ask/knowledge-management-service/auh1-dev\nSTORAGE__S3__ACCESS_KEY_ID / _SECRET_ACCESS_KEY"]
    ESO["ExternalSecret\n(dataFrom: extract)"]
    K8S["k8s Secret\nknowledge-management-service-secret"]
    ENV["pod env\n(envConf.secretRef)"]
    TOML["config TOML\n[storage.s3] @format {env[...]}"]
    B["boto3 client → bucket\nask-knowledge-management"]

    CEPH -->|keys handed over| V
    V --> ESO --> K8S --> ENV --> TOML --> B
    B -->|PUT/GET/LIST/DELETE| CEPH

Prerequisites¶

[x] KMS deployed; [storage] type = "s3", with endpoint_url, region, bucket_name set
[x] Ceph RGW reachable from the cluster: getent hosts s3.cl1.sq4.aegis.internal → 100.115.1.130
[x] Bucket ask-knowledge-management created
[x] Access to radosgw-admin (Ceph node) to create the ask-kms user — R1, Ceph team
[x] Vault reachable + the eso-ask-push policy grants secret/data/ask/knowledge-management-service/* (see 38)

Step 1 — Create the `ask-kms` S3 user (Ceph team)¶

The bucket already exists; this creates the dedicated, least-privilege user and links it to the bucket.

# on a Ceph node
radosgw-admin user create \
  --uid=ask-kms \
  --display-name="ASK Knowledge Management Service"
# → copy access_key + secret_key from the JSON "keys" array — these go to Vault (Step 2)

# grant ask-kms access to the existing bucket
radosgw-admin bucket link --bucket=ask-knowledge-management --uid=ask-kms

Required object permissions on the bucket: s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket — nothing wider.

Step 2 — Store the credentials in Vault¶

The keys live only in Vault — never in values or Git.

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
export VAULT_SKIP_VERIFY=true

vault kv patch secret/ask/knowledge-management-service/auh1-dev \
  STORAGE__S3__ACCESS_KEY_ID="<access_key from Ceph>" \
  STORAGE__S3__SECRET_ACCESS_KEY="<secret_key from Ceph>"

Paste the keys EXACTLY — no trailing newline/space

A trailing newline is a top cause of InvalidAccessKeyId / SignatureDoesNotMatch. Use printf if scripting, and verify the length (Step 5) matches what Ceph issued. RGW access keys are typically ~20 chars and secret keys ~40.

Step 3 — ExternalSecret (already wired)¶

KMS uses ESO dataFrom: extract against the single Vault path — so every key at secret/ask/knowledge-management-service/auh1-dev (including the two S3 keys) is pulled into the one k8s Secret. There is no per-key PushSecret to edit (unlike core-service). After a Vault change, force a resync:

kubectl annotate externalsecret -n ask knowledge-management-service-external-secret \
  force-sync=$(date +%s) --overwrite

kubectl get externalsecret -n ask knowledge-management-service-external-secret \
  -o custom-columns='STATUS:.status.conditions[0].reason,READY:.status.conditions[0].status'
# expect: SecretSynced / True

(If this is SecretSyncedError with a Vault 403, the eso-ask-push policy is missing the KMS path — see 38.)

Step 4 — Reference in the config TOML (already set)¶

In helm/environments/values-mod-auh1-dev-ask.yaml → envConf.configToml:

[mod-auh1-dev-ask.storage]
type = "s3"

[mod-auh1-dev-ask.storage.s3]
endpoint_url      = "https://s3.cl1.sq4.aegis.internal/"
bucket_name       = "ask-knowledge-management"
region            = "us-east-1"
verify_ssl        = false      # Ceph RGW uses a self-signed cert
access_key_id     = "@format {env[STORAGE__S3__ACCESS_KEY_ID]}"
secret_access_key = "@format {env[STORAGE__S3__SECRET_ACCESS_KEY]}"

STORAGE__S3__ACCESS_KEY_ID / STORAGE__S3__SECRET_ACCESS_KEY reach the pod as env via envConf.secretRef: knowledge-management-service-secret; @format substitutes them at config load.

TLS verification — known deviation (`verify_ssl: false`)¶

Non-production pattern, documented deviation

We run with verify_ssl: false because the Ceph RGW at s3.cl1.sq4.aegis.internal serves a self-signed certificate. boto3 then skips TLS cert verification — functional, but it violates the "TLS everywhere" production mandate, and every S3 call logs:

InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.cl1.sq4.aegis.internal'

The warning is harmless (not an error) — S3 read/write works. It is the visible symptom of the deviation, not a failure.

Why it's only a bool, not a CA path: the schema is verify_ssl: bool (src/config/schemas/storage.py) → AWS_S3_VERIFY_SSL → boto3 verify=True/False. It cannot carry a CA-bundle path, so trusting the CA must be done via botocore's AWS_CA_BUNDLE env instead.

Production-correct upgrade path (CA trust)¶

Obtain the CA that signed the RGW cert:
```
openssl s_client -connect s3.cl1.sq4.aegis.internal:443 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -subject     # identify the issuer CA
```
If the issuer is the cluster internal CA (cert-manager internal-ca-issuer, see 36 — ArgoCD CA Trust) we already manage it; otherwise capture Ceph's CA PEM.
Mount the CA PEM into the KMS pods (ConfigMap volume), e.g. /etc/ssl/ceph-ca/ca.crt.

Enable verification + point botocore at the CA:

# values: extraEnv
- name: AWS_CA_BUNDLE
  value: /etc/ssl/ceph-ca/ca.crt
# values: configToml
#   [storage.s3] verify_ssl = true

Platform-wide, do it together

core-service uses verify_ssl: false against the same RGW (see core-service Ceph S3 wiring). Harden the CA trust across all S3 consumers (KMS + core-service) in one deliberate task, not piecemeal — same pattern as the registry HTTP→OCI deviation: shipped as a pragmatic fallback with a documented upgrade path.

Silencing the warning in test scripts (cosmetic only, does not change the deviation):

import urllib3; urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

Browser-facing cert trust (presigned URL uploads) — RESOLVED¶

Resolved by distributing the Aegis root CA to clients

Fixed by installing the Aegis CL1.SQ4 CA Root into client trust stores (the CA that signs RGW's cert) — endpoint_url stays s3.cl1.sq4.aegis.internal, no DNS/Ceph change. Full runbook: Fix S3 Browser Cert Trust. The cert-manager-ingress alternative was rejected (needed a Ceph-side hostname registration we couldn't do). The analysis below is retained for context.

Different failure mode than verify_ssl: false above — do not conflate

The verify_ssl: false deviation above is about pods trusting RGW's TLS cert when calling boto3 directly — that's a solved, documented, server-side deviation. This section is about browsers trusting RGW's TLS cert when uploading directly to a presigned URL (KMS/core-service hand the browser a presigned S3 URL; the browser's own TLS stack validates it, with no verify_ssl knob available). This is currently broken and unresolved.

Symptom: browser file uploads in KMS (and core-service) fail with net::ERR_CERT_AUTHORITY_INVALID (Chrome) / Cross-Origin Request Blocked... Status code: (null) (Firefox) when hitting the presigned RGW URL directly.

Root cause, confirmed via openssl s_client:

openssl s_client -connect s3.cl1.sq4.aegis.internal:443 -showcerts </dev/null 2>/dev/null | openssl x509 -noout -text | grep -A2 "Subject Alternative Name"
# X509v3 Subject Alternative Name:
#     DNS:s3.cl1.sq4.aegis.internal, IP Address:100.115.1.130
openssl s_client -connect s3.cl1.sq4.aegis.internal:443 -showcerts </dev/null 2>/dev/null | openssl x509 -noout -issuer
# issuer=O=Aegis CL1.SQ4 CA, CN=Aegis CL1.SQ4 CA Intermediate CA

# compare against the frontend, which browsers already trust:
openssl s_client -connect ask.mod.auh1.dev.dir:443 -showcerts </dev/null 2>/dev/null | openssl x509 -noout -issuer
# issuer=O=ai71, CN=modgpt

RGW's cert is issued by "Aegis CL1.SQ4 CA" — the underlying infra provider's own CA, not trusted by end-user browsers, and not cert-manager's internal-ca-issuer.
The SAN list only covers s3.cl1.sq4.aegis.internal + IP 100.115.1.130 — it does not include s3.mod.auh1.dev.dir.
Per 10-apphaproxy.md, Ceph RGW's DNS entries bypass AppHAProxy entirely by design — s3.mod.auh1.dev.dir is a plain A record pointing directly at 100.115.1.130. Every other *.mod.auh1.dev.dir host gets its browser-trusted cert via TCP passthrough → nginx-ingress → cert-manager internal-ca-issuer (the O=ai71, CN=modgpt issuer above). RGW was never wired into that path, so switching the app-config hostname alone can't fix it — same IP, same cert, either name.

Already tried and reverted — do not re-apply without the actual RGW fix below

storage.s3.endpoint_url was switched from s3.cl1.sq4.aegis.internal to s3.mod.auh1.dev.dir in KMS, core-service (chart + devops overlay) on the assumption the doc's claim of dual-SAN coverage (ceph-infrastructure.md:90) was accurate. It wasn't — confirmed via the SAN dump above. The hostname switch was reverted back to s3.cl1.sq4.aegis.internal in all three files (KMS chart, core-service chart, core-service devops overlay) since it changed nothing about the underlying cert-trust problem.

Production-correct fix (not yet implemented): reuse the exact pattern already proven for the GitLab alias in 10-apphaproxy.md Step 9.9:

Standalone Ingress + backend Service (pointing at RGW 100.115.1.130:443, with nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" and proxy-ssl-verify: "off" since the backend hop still uses RGW's own self-signed cert — internal-network-only, same trust boundary as today's verify_ssl: false).
Annotate with cert-manager.io/cluster-issuer: internal-ca-issuer — cert-manager auto-issues a browser-trusted O=ai71, CN=modgpt cert for s3.mod.auh1.dev.dir, same as every other alias.
Repoint the s3.mod.auh1.dev.dir DNS A record from 100.115.1.130 (direct) to the AppHAProxy VIP 100.115.2.210, so traffic actually flows through the new ingress. Confirmed this doesn't affect GitLab's own S3 usage, which goes through a separate endpoint (100.115.1.10:7480, see ceph-infrastructure.md).
Set unlimited proxy-body-size and long timeouts (large object uploads), matching the GitLab alias annotations.
Only once the new ingress is live and DNS repointed: re-apply the endpoint_url switch to s3.mod.auh1.dev.dir in KMS + core-service (chart + devops overlay) — this time it will actually change which cert is served.

Alternative considered: install a cert-manager-issued cert directly on RGW's own beast frontend (rgw_frontends ssl_certificate/ssl_private_key) — avoids the extra ingress hop, but requires SSH access to the Ceph mon nodes, has no auto-rotation without extra tooling, and mixes an external, non-Kubernetes-managed component into the cert-manager rotation lifecycle. Deferred in favor of the ingress-based approach above, which requires no changes on the Ceph side at all.

Status: RESOLVED via Aegis root CA distribution — see Fix S3 Browser Cert Trust. The ingress/DNS approach described above was not used (RGW hostname registration was owned by another team).

Step 5 — Validate¶

# A. keys present + sane length (real RGW: access ~20, secret ~40)
av=$(kubectl get secret -n ask knowledge-management-service-secret -o jsonpath='{.data.STORAGE__S3__ACCESS_KEY_ID}' | base64 -d)
sv=$(kubectl get secret -n ask knowledge-management-service-secret -o jsonpath='{.data.STORAGE__S3__SECRET_ACCESS_KEY}' | base64 -d)
echo "access_key: len=${#av} prefix=${av:0:4}…   secret_key: len=${#sv}"

# B. round-trip PUT/GET/LIST/DELETE from a worker pod (boto3 ships in the KMS image)
AK="$av"; SK="$sv"
kubectl exec -n ask deploy/knowledge-management-service-ingestion-worker -i -- env AK="$AK" SK="$SK" python3 - <<'EOF'
import os, boto3, botocore
s3 = boto3.client("s3", endpoint_url="https://s3.cl1.sq4.aegis.internal/",
    aws_access_key_id=os.environ["AK"], aws_secret_access_key=os.environ["SK"],
    region_name="us-east-1", verify=False,
    config=botocore.config.Config(s3={"addressing_style":"path"}))
b, k = "ask-knowledge-management", "_healthcheck/kms-write-test.txt"
s3.put_object(Bucket=b, Key=k, Body=b"kms s3 ok");                       print("PUT  ok")
print("GET  ok:", s3.get_object(Bucket=b, Key=k)["Body"].read().decode())
print("LIST ok:", [o["Key"] for o in s3.list_objects_v2(Bucket=b, Prefix="_healthcheck/").get("Contents", [])])
s3.delete_object(Bucket=b, Key=k);                                       print("DEL  ok")
EOF

-i is required for the heredoc to reach the pod's stdin. All four ok = the chain works end to end.

# C. end-to-end: upload a document via the API, watch it land in S3 + ingest
JWT="<core-service-jwt>"   # see assistant-service/integration-tests.md §7b
curl -sk -X POST https://api.ask.mod.auh1.dev.dir/<kms-upload-path> \
  -H "Authorization: Bearer $JWT" -F "file=@./sample.pdf"
kubectl logs -n ask deploy/knowledge-management-service-ingestion-worker --tail=40 \
  | grep -iE "s3|kms_document_ingestion|embed|weaviate|upsert|error"
# Ceph-side: radosgw-admin bucket stats --bucket=ask-knowledge-management  (object count grows)

Troubleshooting¶

Symptom	Meaning	Fix
`InvalidAccessKeyId`	TLS reached RGW, but the access key is unknown to Ceph — placeholder, wrong key, or wrong RGW zone	confirm Ceph created `ask-kms`; re-patch the exact `access_key` from Ceph (Step 2)
`SignatureDoesNotMatch`	access key valid but the secret key is wrong (or a trailing newline)	re-patch the secret key exactly; check `len` in Step 5A
`403 AccessDenied` (on PUT/LIST)	key valid but the user policy lacks a permission	Ceph team: grant Get/Put/Delete/List on the bucket / re-link the bucket
`EndpointConnectionError` / TLS error	pod can't reach `s3.cl1.sq4.aegis.internal`	check `getent hosts` resolves to `100.115.1.130`; `verify_ssl=false` for the self-signed cert
`len=40` access key	still the bring-up placeholder (`openssl rand -hex 20`) — real keys not patched yet	pending R1; patch when Ceph delivers (not a deploy blocker — S3 is lazy)
pods `Running` but uploads fail	expected with placeholder keys — S3 init is lazy	patch real keys, force ESO resync, retest

What's stored where¶

Item	Location
`ask-kms` access/secret keys	Vault `secret/ask/knowledge-management-service/auh1-dev` → `knowledge-management-service-secret` (ESO)
Source documents + artifacts	Ceph RGW bucket `ask-knowledge-management`
Document metadata	`knowledge-pg` (CNPG)
Vectors/embeddings	Weaviate