Skip to content

fix: skip batch-level whitening in reinforce_plus_plus_baseline when rollout.n=1 - #14

Open
pkcontinue wants to merge 1 commit into
agentica-project:mainfrom
pkcontinue:fix/916-reinforce-pp-zero-advantage
Open

pkcontinue wants to merge 1 commit into
agentica-project:mainfrom
pkcontinue:fix/916-reinforce-pp-zero-advantage

Conversation

@pkcontinue

Copy link
Copy Markdown

Problem

Fixes #916.

compute_reinforce_plus_plus_baseline_outcome_advantage silently produces zero advantages when rollout.n = 1. When all groups have a single sample, masked_whiten computes near-zero variance, causing all advantages to collapse to ~0. This silently makes the policy gradient update a no-op.

Root Cause

The function unconditionally applies masked_whiten (batch-level whitening) at line 321. When rollout.n=1:

  • Each group has exactly 1 sample, so baseline subtraction sets id2mean = 0.0 (no subtraction)
  • masked_whiten then computes (values - batch_mean) / sqrt(batch_var + 1e-8)
  • When all rewards are identical (e.g., all correct or all incorrect), variance ≈ 0, so advantages ≈ 0
  • Even with mixed rewards, the batch-global whitening distorts per-prompt signals

Fix

Track max_group_size and skip masked_whiten when all groups are single-sample. This is consistent with how compute_grpo_outcome_advantage handles the n=1 case (setting mean=0, std=1 and avoiding batch-level normalization).

When multi-sample groups exist, behavior is unchanged — masked_whiten is still applied as before.

Note

This fix is in the verl submodule (verl/verl/trainer/ppo/core_algos.py). The upstream verl repo may need a corresponding PR.

🤖 Generated with Claude Code

…#916)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant