[parlakisik]

Fixes #66.

What was wrong

• Placeholder secrets in every overlay: deploy/k8s/base/secrets.yaml was a kustomize base resource. Every overlay rendered aex-secrets with JWT_SECRET=REPLACE_ME, Mongo root/root, AEX_API_KEY=dev-api-key and placeholder webhook and signing keys. That includes production.
• The gateway accepted the placeholder: it rejected only an empty JWT_SECRET, so anyone could sign an HS256 token with REPLACE_ME, any tenant_id and scopes: ["*"].
• Worse than the report: kubectl apply -k overwrote a Secret the operator had already created. The README's kubectl create secret step and deploy/aws/deploy-eks.sh both create real secrets and then apply the overlay. So even a careful deployment ended up on the public placeholders.
• Related, from the report's impact section: settlement's tenant-scoped endpoints took tenant_id from the query string or request body and ignored the gateway's authenticated X-Tenant-ID. Any legitimate tenant could read another tenant's balance and usage, or deposit into another tenant's account.

Changes

deploy(k8s): stop shipping placeholder secrets in every overlay

• No Secret in the base: base, dev, staging and production now render no Secret and no placeholder strings, checked with kubectl kustomize on all four. An existing aex-secrets is never overwritten.
• Template: base/secrets.example.yaml documents the keys. No kustomization references it.
• New deploy/k8s/create-secrets.sh: idempotent, creates or updates aex-secrets with every key the manifests reference.
  • Values come from the environment first, then the existing value if it isn't a placeholder, then random generation.
  • Re-running never rotates anything.
  • It rejects short or placeholder JWT and webhook secrets and never logs a value.
  • It keeps an external MONGO_URI.
  • It also creates the mongodb-keyfile Secret, which the StatefulSet mounts but which nothing created before.
• Deploy scripts: EKS, GKE and Cloud Build now call the helper. JWT_SECRET is now always set; no script ever set it before.
• Docs: the k8s README, DEPLOYMENT.md, docs/deployment.md, docs/PROJECT.md and demo/README.md create the Secret before kubectl apply -k. There's also a rotation section for clusters deployed before this change.

fix(gateway): refuse placeholder or weak JWT_SECRET outside development

Unless ENVIRONMENT=development, startup fails when JWT_SECRET is:

• empty;
• shorter than 32 bytes;
• a known placeholder (REPLACE_ME, CHANGE_ME, secret, password and similar, plus values containing replace_me, change_me or do-not-use-in-production);
• a <...> template value.

In development these only log a warning, so local compose and kind keep working. The secret is never logged.

fix(settlement): bind tenant-scoped endpoints to the gateway tenant

On GET /v1/balance, GET /v1/usage, GET /v1/usage/transactions and POST /v1/deposits:

• X-Tenant-ID (set by the gateway, which strips client copies) decides the tenant, and tenant_id becomes optional.
• A different tenant_id returns 403 TENANT_MISMATCH.
• Direct internal calls without the header behave as before.

Action for anyone who deployed with the old manifests

Rotate JWT_SECRET, the Mongo root credentials, WEBHOOK_SECRET and any API keys. Treat tokens signed with REPLACE_ME as compromised. The rotation steps are in deploy/k8s/README.md.

Test plan

• kubectl kustomize renders base, dev, staging and production with 0 Secrets and 0 REPLACE_ME, root:root or dev-api-key hits (before: 1 Secret and 6 placeholder hits each).
• GOWORK=off go build, go vet and go test pass in every module. go test -race passes for aex-settlement. go vet -tags integration,e2e passes in hack/integration and hack/tests.
• New tests: a table test for the gateway secret check (placeholders, short values, templates, dev-only warnings); settlement tenant binding (header only, matching or mismatching param or body → 403, no header → param used).
• create-secrets.sh tested end to end against a fake kubectl: fresh run, re-run (values unchanged), upgrade from the old placeholder Secret, and rejection of placeholder or short values from the environment.
• bash -n passes on every changed script. shellcheck -S warning is clean apart from one SC2155 in deploy-gke.sh that was already on main.

Known follow-ups (not in this PR)

• With ENVIRONMENT unset, the gateway assumes development, and deploy-eks.sh/deploy-gke.sh default to the dev overlay on real clusters. Production and staging should be the safer defaults.
• kubectl apply -k on base and the overlays needs --load-restrictor=LoadRestrictionsNone, because the base references files outside its directory. This was already broken on main.
• Settlement still trusts tenant_id from callers inside the cluster that don't send X-Tenant-ID. A NetworkPolicy or service-to-service auth would close that.
• The ECS and Cloud Run paths still read placeholder-update-me values from AWS and GCP secret seeding.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.