Skip to content

fix: reject empty \boxed{} answer and empty ground truths in RewardMathFn - #13

Open
pkcontinue wants to merge 1 commit into
agentica-project:mainfrom
pkcontinue:fix/929-empty-boxed-reward
Open

pkcontinue wants to merge 1 commit into
agentica-project:mainfrom
pkcontinue:fix/929-empty-boxed-reward

Conversation

@pkcontinue

Copy link
Copy Markdown

Problem

Fixes #929.

RewardMathFn gives full reward to an empty \boxed{} answer when the ground truth is also empty. This happens because:

  1. extract_answer returns "" (empty string) for \boxed{}, which passes the model_answer is None check
  2. Empty ground truths are passed through to graders, where grade_answer_mathd("", "") returns True

Changes

  • math_reward.py line 60: Use not model_answer instead of model_answer is None to also reject empty strings from extract_answer
  • math_reward.py lines 73-82: Filter out empty ground truths (both from \boxed{} extraction and bare empty strings) so they don't reach the graders
  • test_math_reward.py: Add test case test_empty_boxed_answer_not_rewarded covering empty \boxed{} with both empty and valid ground truths

Impact

  • Empty \boxed{} model answers now correctly return format_error_reward instead of correct_reward
  • Empty ground truths are filtered before grading; if all ground truths are empty, unk_error_reward is returned
  • No impact on valid answers — any real answer is a non-empty string

🤖 Generated with Claude Code

…thFn

Fixes #929. Previously, an empty \boxed{} model answer would extract as
an empty string which passed the `is None` check, and when matched
against an empty ground truth, both graders (mathd and sympy) returned
True — awarding full reward for a non-answer.

Changes:
- Use `not model_answer` instead of `model_answer is None` to also
  reject empty strings from extract_answer (line 60)
- Filter out empty ground truths (both from \boxed{} extraction and
  bare empty strings) so they don't reach the graders (lines 73-82)
- Add test case for empty \boxed{} with both empty and valid ground
  truths to prevent regression

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant