Skip to content

Weight validation minibatch losses by their example counts - #53

Open
sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:mainfrom
sylvesterkaczmarek:fix/validation-loss-example-weighting-20261006
Open

sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:mainfrom
sylvesterkaczmarek:fix/validation-loss-example-weighting-20261006

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Fixes #52.

Aggregate validation cross entropy by example count instead of giving every minibatch equal weight. A partial final minibatch previously had disproportionate influence on validation scores and best-checkpoint selection. Training updates, training-loss reporting, regularization and the TPR-at-FPR branch are unchanged.

Validation

python -m pytest -q src/synthid_text/logits_processing_test.py src/synthid_text/detector_validation_test.py

23 tests passed. All existing logits-processing tests and eight new validation cases pass. Five new cases fail on unchanged upstream; three partition controls pass. The tests execute the actual Flax detector and train loop, compare every epoch with direct full-set cross entropy, test validation-order invariance, and verify checkpoint selection after a real two-epoch parameter update. The existing suite emitted five PyTorch batching-performance warnings. No language-model checkpoint, tokenizer download, generated corpus, GPU or production detector calibration was used. This does not change single-epoch/no-validation edge cases or claim CPU training quality at production scale.

Tested locally on macOS CPU using real module imports. New-test formatting, scoped static checks, syntax checks and git diff --check pass. No dependency or workflow changes. The full repository suite was not run.

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Validation cross entropy overweights the partial final minibatch

1 participant