Skip to content

[Bugfix][Router] Update average decoding duration stats - #1054

Open
dsxyy wants to merge 2 commits into
vllm-project:mainfrom
dsxyy:codex/c-002-decoding-length
Open

[Bugfix][Router] Update average decoding duration stats#1054
dsxyy wants to merge 2 commits into
vllm-project:mainfrom
dsxyy:codex/c-002-decoding-length

Conversation

@dsxyy

@dsxyy dsxyy commented Aug 25, 2026

Copy link
Copy Markdown

Summary

  • update decoding_length_monitors when a request completes
  • measure decoding duration as completion time minus first-token time, matching the existing RequestStats.avg_decoding_length definition
  • expire decoding-duration samples outside the configured sliding window
  • add a regression test covering pre-completion, completed, and expired samples

Problem

decoding_length_monitors was initialized and read by get_request_stats(), but never updated, so avg_decoding_length remained -1 after successfully completed requests. The existing source definition describes this value as the time from first token to completion, and first_token_time was already recorded but otherwise unused.

Testing

  • pytest -q src/tests/test_request_stats.py src/tests/test_singleton.py src/tests/test_stale_metrics.py src/tests/test_metrics.py src/tests/test_instance_failover.py src/tests/test_request_validation.py src/tests/test_transcription_streaming.py (32 passed)
  • Router regression selection (68 passed)
  • black --check
  • isort --check-only
  • ruff check src/tests/test_request_stats.py
  • codespell
  • git diff --check

Signed-off-by: 姚远 <asmile.yy@gmail.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements tracking for average decoding length statistics in RequestStatsMonitor and adds corresponding unit tests. The reviewer suggested using pop instead of get when retrieving first_token_time upon request completion to prevent potential memory leaks.

Comment thread src/vllm_router/stats/request_stats.py Outdated
Signed-off-by: 姚远 <asmile.yy@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant