Parent roadmap
Contributes to #9 ([M8] Add reproducible i18n Prompt evaluation harness and baseline reports).
Why this is a good contributor task
This task is intentionally narrow: it does not require implementing model judging, Prompt authoring, AgentRun orchestration, or release integration. The goal is to establish a small, legally reusable deterministic regression slice for the future i18n evaluation harness.
Scope
Add a small synthetic evaluation fixture set covering response protocol and placeholder preservation. Cases should be original synthetic examples and include provenance/license metadata compatible with public redistribution.
Cover at least:
- printf placeholders such as
%s, %d, positional variants where supported
- Python-style placeholders such as
{name} / {count}
- escaped percent/braces where relevant
- multiple placeholders in one source string
- reordered placeholders when the format permits it
- missing, duplicated, or mutated placeholders as negative outputs
- protocol completeness examples such as missing items, extra items, malformed terminators, or structurally incomplete output
Implement deterministic checks that can report structured pass/fail evidence for these cases. Keep the implementation independent of any specific LLM provider.
Suggested artifact shape
The exact schema may evolve with #9, but fixtures should preserve the fields needed by the M8 design, including:
- stable case ID
- source/target locale
- source text and optional context
- required tokens/assertions
- expected deterministic outcome
- provenance kind and license
- tags/difficulty
Non-goals
- No semantic or style model judge.
- No network/model API calls.
- No Prompt baseline comparison.
- No game text copied from sources with uncertain redistribution rights.
- No release gate integration.
Acceptance criteria
Coordination
If #9 lands a finalized manifest/case schema first, use it directly. If this task lands first, keep fixture structures small and easy to migrate rather than building a competing evaluation framework.
Parent roadmap
Contributes to #9 (
[M8] Add reproducible i18n Prompt evaluation harness and baseline reports).Why this is a good contributor task
This task is intentionally narrow: it does not require implementing model judging, Prompt authoring, AgentRun orchestration, or release integration. The goal is to establish a small, legally reusable deterministic regression slice for the future i18n evaluation harness.
Scope
Add a small synthetic evaluation fixture set covering response protocol and placeholder preservation. Cases should be original synthetic examples and include provenance/license metadata compatible with public redistribution.
Cover at least:
%s,%d, positional variants where supported{name}/{count}Implement deterministic checks that can report structured pass/fail evidence for these cases. Keep the implementation independent of any specific LLM provider.
Suggested artifact shape
The exact schema may evolve with #9, but fixtures should preserve the fields needed by the M8 design, including:
Non-goals
Acceptance criteria
Coordination
If #9 lands a finalized manifest/case schema first, use it directly. If this task lands first, keep fixture structures small and easy to migrate rather than building a competing evaluation framework.