fix(llm): emit OpenAI strict JSON schemas

[Jordan-Jarvis]

Summary

• serialize typed Pydantic response models with a dedicated OpenAI strict JSON Schema generator
• close every model object with additionalProperties: false
• require every declared property while preserving nullable unions and removing null defaults
• apply the serializer to OpenAI-compatible, LiteLLM, Codex, and direct batch-retain paths

Root cause

HINDSIGHT_API_LLM_STRICT_SCHEMA=true currently sends raw model_json_schema() output. OpenAI-compatible strict structured-output endpoints reject those schemas because objects are not closed and defaulted fields are omitted from required. This was previously identified in the review of #1954 and is reproducible during native retain fact extraction.

The fix does not coerce incoming data or add a fallback. Typed response models are serialized directly through Pydantic into the provider-required strict wire schema.

Validation

• targeted strict-schema tests: 13 passed
• Ruff format/check passed
• repository pre-commit hooks passed
• generated docs skill refreshed by the repository hook
• git diff --check passed

[Jordan-Jarvis]

This change is now the remaining upstream release blocker for a fail-closed production deployment that uses only official vectorize-io/hindsight artifacts. We reproduced the OpenAI strict Structured Outputs rejection against v0.8.4: nested object schemas without additionalProperties: false cause retain/fact extraction to fail. The PR checks are green and the fix covers the shared OpenAI-compatible, LiteLLM, Codex, and batch fact-extraction paths. Could a maintainer please review this for merge and include it in the next signed stable release? We will consume the official multi-platform image digests directly and will not publish a downstream fork image.

[Jordan-Jarvis]

Fresh fail-closed deployment evidence from July 19, 2026 isolates this PR as the blocker:

• bank creation and configuration completed successfully
• native retain fact extraction retried four times
• every attempt received HTTP 400 invalid_json_schema because response format response lacked root additionalProperties: false (param: text.format.schema)
• Hindsight then returned 500 and the calling memory service correctly exposed dependency-unavailable 503

So this is not an OAuth, routing, or bank-configuration failure; it is exactly the strict schema producer corrected here. The PR remains mergeable with all checks green. Please include it in the next signed stable release; downstream qualification will consume the official release artifacts directly.

[Jordan-Jarvis]

Fresh live qualification evidence from July 19, 2026: Will dev promotion run 29668107113 failed its blocking memory acceptance at memory.manual_add. Hindsight retain fact extraction retried four times and received the same provider rejection each time:

Invalid schema for response_format 'response': In context=(), 'additionalProperties' is required to be supplied and to be false.

The failure is on the currently deployed strict-schema Hindsight path and is exactly the producer defect fixed by this PR. We are keeping the gate fail-closed; staging and production did not advance. The downstream organizational fork is archived, so we will not publish another patched fork artifact. Merge into a signed upstream release is now the only production cutover path.

[Jordan-Jarvis]

@benfrank241 @dcbouius could one of you review this strict-schema fix for merge? The PR is clean and green, and two independent fail-closed dev promotions on July 19, 2026 reproduced the exact retain failure it fixes: OpenAI rejects the generated response schema because the root object omits additionalProperties: false. We cannot qualify or promote an official Hindsight artifact until this lands in a signed stable release.

[koriyoshi2041]

Verified the strict-schema serializer closes every object in the real FactExtractionResponse schema, requires all declared fields while preserving nullable unions, and is wired through OpenAI-compatible, LiteLLM, Codex, and batch-retain paths. I also ran the strict-schema/provider/fact-extraction regression set locally: 33 passed.