Skip to content

[M8][contributor] Add JSON-schema validation for public i18n eval cases #15

Description

@AllenXiao95

Parent roadmap

Contributes to #9 ([M8] Add reproducible i18n Prompt evaluation harness and baseline reports).

Goal

Provide a small schema-validation layer for public evaluation cases so malformed fixtures fail before any model or scoring code runs.

Scope

Define a versioned schema for the stable core fields already described by M8:

  • schema_version
  • stable id
  • source/target locale
  • domain
  • source text
  • optional context
  • glossary/required tokens
  • accepted references when present
  • assertions
  • provenance kind/license/source metadata
  • difficulty/tags

Add validation tests for representative valid and invalid cases. Validation must run offline and should produce actionable field-level errors.

Non-goals

  • No evaluator runtime.
  • No model calls or semantic scoring.
  • No baseline comparison.
  • No external dataset import.

Acceptance criteria

  • Public eval cases are validated against a versioned schema.
  • Invalid IDs, missing provenance/license data, malformed locale fields, and unsupported assertion names fail with actionable diagnostics.
  • Validation is provider-independent and network-free.
  • Tests cover valid and invalid examples on Windows and Linux.
  • The schema remains compatible with the contributor fixtures in [M8][contributor] Add synthetic placeholder/protocol eval fixtures and deterministic checks #13 or provides a documented migration path.

Coordination

Coordinate with #13 and #14. This task is useful only if it converges on the same schema used by the parent roadmap #9 rather than creating an alternative format.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions