Skip to content

Clear stale bootstrap metadata when reusing metrics - #313

Open
kocean9-freedom wants to merge 1 commit into
mjpost:masterfrom
kocean9-freedom:fix/reset-bootstrap-signature
Open

kocean9-freedom wants to merge 1 commit into
mjpost:masterfrom
kocean9-freedom:fix/reset-bootstrap-signature

Conversation

@kocean9-freedom

Copy link
Copy Markdown

After metric.corpus_score(..., n_bootstrap=10), reusing the same BLEU, chrF or TER object for ordinary corpus or sentence scoring leaves bs:10|seed:12345 in get_signature(), even though the new score has no confidence interval. A subsequent paired AR test without confidence estimation also inherits bs:10. The reported reproducibility settings therefore depend on earlier calls.

Reset the single-score bootstrap metadata after computing a new score, and clear inherited bs metadata before PairedTest adds its own settings. Scoring, resampling and p-value calculations are unchanged. Previously returned signature objects retain their original settings.

The regression tests compare reused and fresh metric instances across all three metrics, explicit/cached references, sentence scores, AR with and without confidence intervals, and paired BS. They also check that a later bootstrap call records its new count and seed. On unchanged master c596d9d, 12 new cases fail and six controls pass; all pass with the fix.

Validation on macOS with Python 3.12.14:

  • 195 tests pass with python -m pytest --ignore=test/test_api.py --ignore=test/test_dataset.py; three additional offline API tests pass. This includes the existing WMT17 significance comparisons.
  • mypy sacrebleu scripts test passes for 44 files. Ruff passes on the four changed files.
  • Offline source-distribution and wheel builds pass. Their changed source/test files match the tested checkout.
  • 33 public API scenarios pass from the unpacked wheel, including JSON score output and paired tests with one and two workers.

Dataset-download tests, the network-heavy shell suite and other platforms were not run. This does not include the independent fixes in #311 or #312.

AI assistance: OpenAI Codex performed the investigation, implementation, tests, self-review and description. No independent human review has occurred.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant