fix(nemotron_3.5_super): make P/D router topology and policy configurable - #3193
Merged
Merged
Conversation
…able --intra-node-data-parallel-size was hard-coded to 1, so every node had to serve tensor-parallel only; data-parallel serving measured roughly twice the prefill throughput on Qwen3.8-Flash-Next. --decode-policy was hard-coded to cache_aware, which sends every decode request to the first node when requests share a prefix. Additional decode nodes stayed idle and adding them changed nothing. It now defaults to round_robin; set ROUTER_DECODE_POLICY to restore the old value. Both settings, and the prefill policy, are read from the environment with defaults that keep existing behaviour, apart from the decode policy noted above. Startup now also fails when vllm-router exits, which previously left every worker running behind an unreachable endpoint. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
yaoyu-33
added
area:environment
yaoyu-33
reviewed
Sep 10, 2026
yaoyu-33
added
waiting-on-customer
The first version of this change shipped round_robin as the new default, which silently retuned every existing P/D deployment. The point is that the policy is settable, not that the shipped value moves: every default in this file now reproduces the hard-coded behaviour it replaced. The Qwen3.8-Flash-Next SWE-bench Pro deployment that motivated round_robin sets ROUTER_DECODE_POLICY itself, on the caller side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Contributor
Author
|
/ok to test d435e8b |
yaoyu-33
added
needs-review
gchlebus
approved these changes
Sep 10, 2026
Contributor
Author
|
/ok to test a993017 |
laszkiewiczp
enabled auto-merge (squash)
September 10, 2026 14:05
laszkiewiczp
deleted the
plaszkiewicz/pd-router-dp-and-decode-policy
branch
September 10, 2026 14:07
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
…able (#3193) Two settings in `sbatch_external_vllm.sh` were hard-coded in a way that limited P/D deployment flexibility. Both showed up while tuning a prefill/decode deployment for Qwen3.8-Flash-Next on SWE-bench Pro. --------- Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
5 tasks
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two settings in
sbatch_external_vllm.shwere hard-coded in a way that limited P/D deployment flexibility. Both showed up while tuning a prefill/decode deployment for Qwen3.8-Flash-Next on SWE-bench Pro.