Repository navigation
fix: reject empty \boxed{} answer and empty ground truths in RewardMathFn - #13
Open
pkcontinue wants to merge 1 commit into
Open
pkcontinue wants to merge 1 commit into
pkcontinue wants to merge 1 commit into
Conversation
…thFn
Fixes #929. Previously, an empty \boxed{} model answer would extract as
an empty string which passed the `is None` check, and when matched
against an empty ground truth, both graders (mathd and sympy) returned
True — awarding full reward for a non-answer.
Changes:
- Use `not model_answer` instead of `model_answer is None` to also
reject empty strings from extract_answer (line 60)
- Filter out empty ground truths (both from \boxed{} extraction and
bare empty strings) so they don't reach the graders (lines 73-82)
- Add test case for empty \boxed{} with both empty and valid ground
truths to prevent regression
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #929.
RewardMathFngives full reward to an empty\boxed{}answer when the ground truth is also empty. This happens because:extract_answerreturns""(empty string) for\boxed{}, which passes themodel_answer is Nonecheckgrade_answer_mathd("", "")returnsTrueChanges
math_reward.pyline 60: Usenot model_answerinstead ofmodel_answer is Noneto also reject empty strings fromextract_answermath_reward.pylines 73-82: Filter out empty ground truths (both from\boxed{}extraction and bare empty strings) so they don't reach the graderstest_math_reward.py: Add test casetest_empty_boxed_answer_not_rewardedcovering empty\boxed{}with both empty and valid ground truthsImpact
\boxed{}model answers now correctly returnformat_error_rewardinstead ofcorrect_rewardunk_error_rewardis returned🤖 Generated with Claude Code