Skip to content

Keep paired confidence statistics JSON serializable - #312

Open
kocean9-freedom wants to merge 1 commit into
mjpost:masterfrom
kocean9-freedom:fix/paired-chrf-json
Open

kocean9-freedom wants to merge 1 commit into
mjpost:masterfrom
kocean9-freedom:fix/paired-chrf-json

Conversation

@kocean9-freedom

Copy link
Copy Markdown

Paired bootstrap results for chrF and TER cannot be printed with --format json: print_results_table raises TypeError: Object of type float32 is not JSON serializable. The same failure occurs through PairedTest(..., test_type='ar', n_ar_confidence=...) followed by the shared JSON formatter.

The statistics are accumulated as float32 arrays, and estimate_ci returns NumPy scalars. These are stored unchanged in Result.mean and Result.ci, then passed to json.dumps through Result.__dict__. Return Python floats from estimate_ci, as its annotation already specifies. The conversion happens after the existing calculation and preserves its numeric values.

The regression runs the real paired tests and shared output formatter for BLEU, chrF and TER, under both AR with confidence intervals and BS. Before the fix, the four chrF/TER cases raise the serialization error and both BLEU controls pass. Afterward all six pass. This is independent of #311 (seed forwarding) and #309 (single-score formatting of zero intervals).

Validation on macOS, Python 3.12.14, NumPy 2.5.4:

  • python -m pytest --ignore=test/test_api.py --ignore=test/test_dataset.py: 183 passed, including the existing WMT17 significance comparisons.
  • mypy sacrebleu scripts test: passes all 45 files; Ruff passes on both changed files.
  • CLI JSON for BLEU, chrF and TER passes for paired AR/BS with one and two workers. BS includes numeric confidence fields. AR confidence intervals are covered through the Python API; the CLI does not expose that option.
  • Wheel and sdist builds pass. A hand-checked 40-value fixture retains mean 19.5 and CI half-width 18.5 for both float32 and float64 input arrays.

Dataset-download/API network tests and the network-heavy shell suite were not completed locally. Whole-repository Ruff reports the existing circular-import E402 at sacrebleu/utils.py:650, which is unchanged.

AI assistance: an automated AI assistant performed the investigation, implementation, tests, and description. No independent human review has occurred.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant