[wzhao18]

Overview

Adds --scheduling-policy shortest_prefill_first. Before each scheduling step, requests whose remaining prefill length is known are ordered shortest first within each scheduler queue. Nothing else in the scheduler changes: token budget, chunking, KV connector lookups and loads, block allocation and preemption all run as under fcfs.

The target is prefill instances serving multi-turn, prefix-heavy traffic (e.g. agentic coding with an external KV cache tier such as Mooncake), where uncached tokens per request are heavy-tailed and a few long prefills ahead in the queue set the TTFT of many short ones.

Ordering rules

Key: remaining prefill tokens, smallest first; ties keep their current order.

queue	requests that are ordered	key
running	prefill chunks	prompt tokens not yet computed
kv_holding_waiting	all (their external hit is final, e.g. a KV connector load in flight or done)	tokens not yet computed
waiting, async_kv_waiting (first 64)	requests whose KV connector lookup has finished, including a hit of 0	tokens not in that hit

A request whose remaining length is not known keeps its exact position: a decode, a waiting request with no or an unfinished external lookup, or one that skips external KV reads. Known requests are permuted only among the positions they already hold, so nothing moves past an unordered request. Without a KV connector, the waiting queues keep arrival order and only the running and KV-holding queues are ordered.

Requests with an unknown external hit are deliberately not ranked by their local hit: on an external-cache-heavy workload the local hit is a small part of the true hit, so they would sort to the back, and since their lookup is submitted only when the admission loop reaches them, they could stay unknown.

External hits are read with a new optional connector hook, get_known_num_matched_tokens, which returns the prefix the external cache is already known to hold, or None. It must not start a lookup or change state. MooncakeStoreConnector returns the hit of an already finished lookup (LookupKeyClient.peek), MultiConnector the largest of its connectors', and other connectors None.

Costs and side effects

• Per step: one subtraction or one dict read per request, and a sort of the running list, the KV-holding queue and the first 64 waiting requests. No prefix-cache lookups and no connector lookups are added.
• Preemption still takes the last running request; after ordering, that is the one with the most prefill left rather than the newest admission.
• Ordering can change which waiting request the admission loop reaches first, and so the order in which connector lookups and loads start; what triggers them is unchanged.
• No starvation bound: a long prefill can be passed over while shorter ones keep arriving.
• fcfs and priority are unchanged. Cache-aware admission ([Core] Add cache-aware admission ordering #54625) stays fcfs-only.

Validation

• tests/v1/core/test_shortest_prefill_first.py: running prefills ordered and decodes in place; KV-holding queue ordered; waiting requests with known hits (including 0) ordered among their own positions while unknown and skip_reading_external_kv requests keep theirs; a full hit still counts its last token; stable ties; no connector leaves waiting untouched; the 64-request window.
• tests/v1/core/test_scheduler.py::test_shortest_prefill_first_policy_runs_shortest_prefill_first: with two running prefills and a budget that cannot cover both chunks, shortest_prefill_first gives the budget to the one with fewer tokens left first, fcfs to the earlier one.
• tests/v1/kv_connector/unit/test_mooncake_store_lookup.py::test_peek_reads_finished_lookup_without_side_effects: peek returns None before and during a lookup and the hit once it finishes, without submitting, consuming or expiring it.

pytest -v tests/v1/core/test_shortest_prefill_first.py tests/v1/core/test_scheduler.py tests/v1/kv_connector/unit/test_mooncake_store_lookup.py

On a Linux CPU node (VLLM_TARGET_DEVICE=cpu): the SPF, Mooncake lookup and the two policy tests in test_scheduler.py pass (21 tests). The full test_scheduler.py gives 230 passed / 67 failed both with and without this change, with the same 67 failures (environment-dependent tests, e.g. diffusion and CPU-encoder paths).

E2E (Kimi K3.1, GB300, disaggregated P/D with Mooncake, AgentX agentic trace, 60 min): pending.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[mergify]

Documentation preview: https://vllm--66705.org.readthedocs.build/en/66705/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wzhao18.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork