You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add reproducible Weighted Mean calibration benchmark - #50
Add a reproducible calibration baseline for the Weighted Mean detector across effective token lengths. The benchmark uses deterministic synthetic g-value fixtures, calibrates thresholds at requested false-positive rates, and reports ROC-AUC plus TPR with bootstrap confidence intervals in a versioned JSON result.
The synthetic fixture is explicitly documented as a calibration/reporting baseline rather than a production-performance dataset, so a public text corpus and benign transformations can be added later without changing the metric helpers or result shape.
Refreshed onto current main; the only conflict was README wrapping around the Weighted Mean detector guidance. I retained the current upstream wording and added the benchmark link.
The branch is now 0 behind, mergeable, and GitHub Verified. The detector calibration benchmark tests pass 4/4 on a CPU JAX environment, and git diff --check is clean. Google CI is currently awaiting maintainer approval to run.
Could a maintainer review the refreshed head when convenient?
Refreshed onto current main. The branch is now 0 behind, mergeable, and GitHub Verified. The benchmark remains a single focused commit; GitHub CI and the Google workflow are currently at the maintainer-approval gate. Could a maintainer re-review/approve the refreshed head when convenient?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #48.
Summary
Add a reproducible calibration baseline for the Weighted Mean detector across effective token lengths. The benchmark uses deterministic synthetic g-value fixtures, calibrates thresholds at requested false-positive rates, and reports ROC-AUC plus TPR with bootstrap confidence intervals in a versioned JSON result.
The synthetic fixture is explicitly documented as a calibration/reporting baseline rather than a production-performance dataset, so a public text corpus and benign transformations can be added later without changing the metric helpers or result shape.
Testing
pytest -q— 20 passedpyink --target-version py39 --check benchmarks/detector_eval/benchmark.py benchmarks/detector_eval/benchmark_test.pygit diff --check