Skip to content

feat(memory): factor final KV capacity planning and record committed bytes - #193

Open
kunkunblueberry wants to merge 1 commit into
InferMatrixLab:mainfrom
kunkunblueberry:feat/rfc192-kv-capacity-planning
Open

kunkunblueberry wants to merge 1 commit into
InferMatrixLab:mainfrom
kunkunblueberry:feat/rfc192-kv-capacity-planning

Conversation

@kunkunblueberry

Copy link
Copy Markdown
Contributor

Required human contributor answers

Answered and confirmed by the human contributor (@kunkunblueberry) before submission.

1. What outcome are you trying to achieve?

Size the KV pool from the memory actually available after runtime initialization, warmup and CUDA Graph capture, instead of relying on fixed per-pool estimates. This is the direction of RFC #192.

2. What is the highest-leverage variable for that outcome?

The number of committed KV blocks that remains once the measured free memory has been reduced by the utilization cap, the existing reservation and every non-KV overhead (probe peak, concurrent/async peak, graph reserve, static buffers, metadata).

3. How will this work move that variable rather than create busywork?

The startup budget arithmetic in plan_cache() becomes a single pure function, kv_capacity(), over measured free memory, the utilization cap, the reservation and an explicit overhead map. A later measured-sizing change then has one place to change and a CPU-testable contract, and the plan now reports kv_committed_bytes and metadata_bytes, making committed capacity observable. This PR is a preparatory step for RFC #192 step 1; it deliberately does not change the number of allocated blocks.

4. Who is the customer or user, and what do they actually need?

Operators and maintainers deploying vllm-rlt on GPUs. They need capacity to be predictable and not to leave usable VRAM unused, or to run out of it, because of estimates that vary with the enabled execution paths.

PR necessity and bugfix evidence

A PR is needed: RFC #192 is an open, direction-defining RFC, the human contributor claimed the runtime KV capacity planning part in #192, and the outcome requires a code change. This PR is not a bugfix; there is no observed failure to link. It is the first, preparatory slice of the claimed scope.

Purpose

Part of RFC #192 (runtime KV cache capacity planning), split into several PRs. This PR covers capacity computation only:

  • Extract the KV budget arithmetic from plan_cache() into kv_capacity(), a pure function taking free_bytes, cap_bytes, reserved_bytes, per_block, overheads and useful_blocks.
  • Keep the arithmetic identical: min(free_bytes, cap - reserved) - sum(overheads), floored to whole blocks and capped by useful_blocks. No capacity value changes for any existing configuration.
  • Make each non-KV overhead explicit in one place instead of an anonymous sum in plan_cache().
  • Report metadata_bytes and kv_committed_bytes in memory_plan, so the committed capacity (not only the budget before block rounding) is observable.

Explicitly out of scope

  • Warmup / CUDA Graph capture stage measurement and the startup-order change from RFC [RFC]: Size KV storage after runtime warmup and CUDA Graph capture #192 section 2 (the rest of rollout step 1).
  • Stable-address GPU KV storage via VMM, and committing physical mappings after capture (rollout step 2).
  • Enabling measured sizing and retiring the fixed per-pool graph reserve (rollout step 3).
  • Any change to gpu_memory_utilization, num_blocks / kv_cache_memory_bytes override semantics, or graph_memory_reserve().

These remain for follow-up PRs under the same RFC claim.

Test Plan

New tests/test_kv_capacity.py (CPU-only, no GPU required):

  • free memory is the binding bound and every overhead is subtracted;
  • the utilization cap minus the reservation is used when it is tighter;
  • useful_blocks caps the block count without altering the reported budget;
  • the existing "no memory remains for KV blocks" error is raised when no block fits;
  • budget_blocks() floors partial blocks.

Pre-existing suites are unchanged; all call sites still go through plan_cache() with the same numbers.

vllm-rlt Version / Commit: 0.2.0 / 0b1daac on top of upstream/main c338aaa

Test Result

Environment: Windows, CPU-only torch 2.12.1+cpu. No CUDA device was available, so the cuda_profile branch of plan_cache() was not executed. The extracted function is pure and fully covered on CPU; the CUDA path is a call-site refactor that passes the same values.

  • python -m pytest tests/test_kv_capacity.py -q -> 5 passed
  • python -m pre_commit run ruff-check --all-files -> Passed
  • python -m pre_commit run ruff-format --all-files -> Passed
  • python -m pytest tests -q -> 1037 passed, 744 skipped, 20 failed

The 20 failures are pre-existing in this environment and unrelated to this change: the same 20 tests fail on a clean upstream/main (c338aaa) checkout on the same host, and they are HTTP serving / control-channel tests affected by the Windows + CPU-only environment, not by memory planning. The pre-commit pytest hook therefore fails here for the same environmental reason; the two ruff hooks pass.

GPU validation of the end-to-end cuda_profile path (auto-sizing under a real CUDA device, and the existing GPU-marked tests) is still pending and will be done before this path is extended with measured post-capture sizing.

Move the startup KV budget arithmetic in plan_cache into kv_capacity(),
a pure function over measured free memory, the utilization cap, and each
non-KV overhead. The numbers are unchanged; the plan now also reports
metadata_bytes and kv_committed_bytes so the committed capacity is
observable. Covered by CPU unit tests.

Part of RFC InferMatrixLab#192 (capacity planning). Storage and warmup/capture ordering
are not changed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant