Repository navigation
fix: skip batch-level whitening in reinforce_plus_plus_baseline when rollout.n=1 - #14
Open
pkcontinue wants to merge 1 commit into
Open
pkcontinue wants to merge 1 commit into
pkcontinue wants to merge 1 commit into
Conversation
…#916) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #916.
compute_reinforce_plus_plus_baseline_outcome_advantagesilently produces zero advantages whenrollout.n = 1. When all groups have a single sample,masked_whitencomputes near-zero variance, causing all advantages to collapse to ~0. This silently makes the policy gradient update a no-op.Root Cause
The function unconditionally applies
masked_whiten(batch-level whitening) at line 321. Whenrollout.n=1:id2mean = 0.0(no subtraction)masked_whitenthen computes(values - batch_mean) / sqrt(batch_var + 1e-8)Fix
Track
max_group_sizeand skipmasked_whitenwhen all groups are single-sample. This is consistent with howcompute_grpo_outcome_advantagehandles then=1case (settingmean=0, std=1and avoiding batch-level normalization).When multi-sample groups exist, behavior is unchanged —
masked_whitenis still applied as before.Note
This fix is in the verl submodule (
verl/verl/trainer/ppo/core_algos.py). The upstream verl repo may need a corresponding PR.🤖 Generated with Claude Code