[parlakisik]

Summary

A dead-code sweep found code that was never removed, and some that was never wired up. Wiring it up showed that completed contracts were never settled. Making settlement work then exposed several money-path defects. This PR fixes them.

Seven commits, each self-contained. Reviewing commit by commit is easiest.

1. chore: remove unused code and modules (−1,942 lines)

Candidates came from golang.org/x/tools/cmd/deadcode across all 28 modules, checked on darwin and linux and with the integration,e2e tags. Each was then checked against the specs, open tasks and git history before deletion.

• Modules removed:
  • src/internal/testutil: never imported.
  • src/internal/agentcard and the root src/internal module: replaced by provider-registry's push registration.
• Functions removed:
  • httpclient auth helpers and the NewClientWithRetry/NewClientWithCircuitBreaker duplicates
  • ap2 mandate checks (never called, and no signature check)
  • token-bank's unused auth middleware
  • work-publisher's BidGatewayClient, which called a route that doesn't exist
  • duplicate settlement methods, nats.StreamForSubject and RouteScopes.Keys
  • hack/integration DatabaseCleaner and three deprecated stubs
• Kept on purpose:
  • the internal/certauth offline verifier (pending a decision)
  • certauth RecalculateAll
  • test seams
  • the integration-test harness API
• Also updates the CI matrix, the Makefile and the docs.

2. feat(gateway): configurable CORS origins

The router used CORSAllowAll. It now uses middleware.CORS(cfg.AllowedOrigins), with ALLOWED_ORIGINS read as a comma-separated list (default *). The default behaves exactly as before, including preflight.

3. feat(bid-evaluator): fetch the work spec from work-publisher

• Wires up WORK_PUBLISHER_URL. When a request sends no budget, the evaluator fetches the work. A budget in the request still wins.
• The client's JSON fields were wrong (id/state where work-publisher sends work_id/status).
• A missing work maps to 404 and other upstream failures to 502. A negative max_price gets 400.

4. fix(contract-engine): settle completed contracts

• Bug: contract-engine never notified settlement, and settlement doesn't subscribe to contract.completed. Completed contracts were never paid.
• Fix: contract-engine now posts to /internal/settlement/complete in the background once a completion is saved. A 409 counts as success, and a settlement failure never fails the provider's completion.
• Real consumer: on award, contract-engine looks up the real consumer_id from work-publisher. It used to store "unknown", and settlement is skipped while the consumer is still unknown.
• Failed outcomes are not charged.
• /complete only from AWARDED or EXECUTING. Anything else gets 409. Before this, a FAILED contract could be completed and paid.

5. fix(httpclient): resend the request body on retries

Retried POSTs and PUTs went out with an empty body: the same request was re-sent after its body had been read, and doJSON used io.Pipe, so GetBody was never set. Each attempt now clones the request and gets a fresh body from GetBody. A request with a body but no GetBody is not retried.

6. fix(settlement): settle each contract exactly once

What was wrong:

• A lookup error was read as "not settled".
• Check-then-insert raced, with no unique index on contract_id.
• The execution was saved before balances moved, so a failed update turned retries into a false 409.
• Unknown tenants were created with negative balances.

The fix, without transactions (the compose and ECS stacks run standalone Mongo):

• A unique contract_id enforces one execution per contract.
• Each execution starts PENDING. The steps are: payment outcome (saved once), debit, credit, two ledger entries, SETTLED.
• Each balance change is a single-document atomic update keyed by <contract_id>:debit or <contract_id>:credit, so replaying it is a no-op. Ledger ids are deterministic, and the final change to SETTLED is compare-and-set.

Recovery:

• Settlement work runs on its own context, so a caller disconnecting can't cut it off.
• A resumer drives stranded PENDING executions to SETTLED, safely across replicas. Executions too old to replay safely are left out of the scan and logged at ERROR for manual reconciliation.
• Startup rejects duration settings that contradict each other.

Rules enforced in settlement itself:

• Empty or "unknown" consumer, provider or contract → 400.
• success=false → recorded, no money moves. A later success=true settles once.
• A consumer with no account → 402.

New settings: SETTLEMENT_RESUME_INTERVAL, SETTLEMENT_RESUME_GRACE, SETTLEMENT_PENDING_ALERT_AGE, SETTLEMENT_OP_RETENTION.

7. fix(work-publisher): require a real consumer identity

• Bug: the gateway sets X-Tenant-ID but never X-Consumer-ID, so every submission was stored with consumer default_consumer. With settlement live, that fake tenant would have been charged.
• Work-publisher:
  • Reads X-Tenant-ID first, then X-Consumer-ID for direct service calls. With neither, it returns 401 CONSUMER_ID_REQUIRED.
  • Returns 404 only for real not-found; other store errors become 500.
• Gateway: removes any client-sent X-Consumer-ID.
• Callers updated: hack/integration, hack/tests/e2e_test.sh and the demo UI. The integration SettleContract helper called a route that doesn't exist; its test now checks balances and the 409 replay.

Behaviour changes to note

• Settlement is now active. SETTLEMENT_URL and WORK_PUBLISHER_URL are set for contract-engine in compose, k8s, AWS, GCP and the hack deploy scripts.
• 402 for unfunded consumers: a consumer with no settlement account now gets 402. The demo stacks never deposit, so their settlements return 402.
• Duplicate contract_id rows block startup: settlement fails to start if it cannot create the unique contract_id index. Check existing executions data for duplicates before deploying.
• 401 for direct submissions without identity: a direct work-publisher submission with no consumer identity now gets 401. Requests through the gateway are unaffected.

Test plan

• GOWORK=off go build, go vet and go test pass in every module (src/aex-*, src/internal/*, hack/*).
• go test -race passes for aex-settlement and internal/httpclient.
• go vet -tags integration,e2e passes, and the tagged tests compile, in hack/integration and hack/tests. Plain go vet/go test in hack/tests fails with "matched no packages", which is already the case on main.
• Each change was reviewed by independent agents, including adversarial reviews of the money path (crash at each step, concurrent requests, resumer across replicas, retries).
• Not yet done, needed before merge: run hack/integration and the e2e suite against hack/docker-compose.yml. The new Mongo store code (pipeline update with upsert, duplicate-key mapping, conditional updates, the partial index, and the WithTransaction fallback on standalone) has only been exercised through the in-memory store.

Follow-ups (not in this PR)

• Contract-engine shutdown waits up to 65s, longer than the k8s default grace period of 30s.
• Production ALLOWED_ORIGINS overlay.
• Demo stacks need a deposit step and auth.
• AP2 mandate signature validation.
• Decide whether to wire up or delete the certauth offline verifier, and whether to schedule or remove RecalculateAll.