Skip to content

Make entities required with an empty-list default so literal models stop dropping them#2792

Closed
dimonnld wants to merge 1 commit into
vectorize-io:mainfrom
dimonnld:entities-required-schema
Closed

Make entities required with an empty-list default so literal models stop dropping them#2792
dimonnld wants to merge 1 commit into
vectorize-io:mainfrom
dimonnld:entities-required-schema

Conversation

@dimonnld

Copy link
Copy Markdown
Contributor

Fixes #2749.

Problem

The four extraction models (ExtractedFact, ExtractedFactVerbose, ExtractedFactNoCausal, VerbatimExtractedFact) declared entities as an optional field that was absent from json_schema_extra["required"]. Under a strict JSON schema a literal model resolves the conflict — the field is optional and typed as objects, while the few-shot examples taught bare strings — by omitting it entirely: zero entities, no error. It bites precisely the most schema-conformant models and reproduces with strict-schema both on and off.

The retain parser is lenient (it coerces string entities into Entity(text=...)), so the string-array shape itself survives — the load-bearing defect is the omitted optional field, not the string shape.

Fix

  • entities: list[Entity] = Field(default_factory=list, ...) on all four models, and "entities" added to each model's required array. An empty list stays valid, so lenient parsing never hard-fails, but the field is no longer silently omitted by literal models.
  • Prompts teach the object form. The concise few-shot examples now use [{"text": ...}] objects, and an explicit "array of objects with a text field, never bare strings" instruction is added to the concise/verbatim (base) prompt and the verbose prompt — so non-strict / no-schema paths stop emitting the ambiguous shape.

Tests

test_fact_extraction_entities_schema.py — deterministic, no LLM:

  • each of the four models' generated JSON schema lists entities as required, typed as an array of objects with a text field (the CI regression that would have caught this);
  • omitting entities still parses, defaulting to [] (lenient path intact);
  • every extraction prompt (concise / verbose / verbatim / custom) teaches the object form and shows no bare-string entity examples.

Verified locally: the schema/prompt suite is green, all prompt modes assemble, and the existing test_fact_extraction_fact_type_prompt.py still passes. Note the base prompt is .format()-ed twice (module assembly, then per-request), so the object examples in the few-shot are escaped accordingly and the standalone format instruction is brace-free.

…#2749)

The four extraction models declared entities as optional and absent from
json_schema_extra["required"]. Under a strict JSON schema a literal model
resolves the optional-typed-as-objects field (while examples taught strings)
by omitting it entirely — 0 entities, no error — biting the most
schema-conformant models.

- entities: list[Entity] = Field(default_factory=list, ...) on all four models
  (ExtractedFact, ExtractedFactVerbose, ExtractedFactNoCausal,
  VerbatimExtractedFact), with "entities" added to each required array. Empty
  list stays valid so lenient parsing never hard-fails; the field is no longer
  silently omitted.
- Prompts teach the object form: concise few-shot switched to [{"text": ...}]
  objects, plus an explicit array-of-objects-with-text-field instruction on the
  base (concise/verbatim) and verbose prompts.
- Deterministic schema/prompt regression tests (no LLM).
@dimonnld

Copy link
Copy Markdown
Contributor Author

Closing as superseded. #2749 was already fixed in main by #2830, which took the list[str] (flat strings) approach — main now has entities: list[str] = Field(default_factory=list). This PR used list[Entity] (required + empty-list default), which conflicts with that design choice, and its regression test duplicates the one #2830 added (test_fact_extraction_entities_schema.py). Nothing left to land here.

@dimonnld dimonnld closed this Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Auto-extracted entities are never persisted (v0.8.4): prompt teaches string arrays while Entity requires objects, and the field is optional

1 participant