Skip to content

Add reproducible Weighted Mean calibration benchmark - #50

Open
sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:mainfrom
sylvesterkaczmarek:issue-48-detector-benchmark
Open

sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:mainfrom
sylvesterkaczmarek:issue-48-detector-benchmark

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Fixes #48.

Summary

Add a reproducible calibration baseline for the Weighted Mean detector across effective token lengths. The benchmark uses deterministic synthetic g-value fixtures, calibrates thresholds at requested false-positive rates, and reports ROC-AUC plus TPR with bootstrap confidence intervals in a versioned JSON result.

The synthetic fixture is explicitly documented as a calibration/reporting baseline rather than a production-performance dataset, so a public text corpus and benign transformations can be added later without changing the metric helpers or result shape.

Testing

  • pytest -q — 20 passed
  • pyink --target-version py39 --check benchmarks/detector_eval/benchmark.py benchmarks/detector_eval/benchmark_test.py
  • benchmark CLI smoke run with two length buckets and JSON output
  • git diff --check

@sylvesterkaczmarek
sylvesterkaczmarek marked this pull request as ready for review September 11, 2026 21:43
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the issue-48-detector-benchmark branch from da79654 to cb535d2 Compare October 5, 2026 22:56
@sylvesterkaczmarek

Copy link
Copy Markdown
Author

Refreshed onto current main; the only conflict was README wrapping around the Weighted Mean detector guidance. I retained the current upstream wording and added the benchmark link.

The branch is now 0 behind, mergeable, and GitHub Verified. The detector calibration benchmark tests pass 4/4 on a CPU JAX environment, and git diff --check is clean. Google CI is currently awaiting maintainer approval to run.

Could a maintainer review the refreshed head when convenient?

@sylvesterkaczmarek

Copy link
Copy Markdown
Author

Refreshed onto current main. The branch is now 0 behind, mergeable, and GitHub Verified. The benchmark remains a single focused commit; GitHub CI and the Google workflow are currently at the maintainer-approval gate. Could a maintainer re-review/approve the refreshed head when convenient?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: reproducible detector calibration and robustness benchmark

1 participant