Repository navigation
feat(memory): factor final KV capacity planning and record committed bytes - #193
Open
kunkunblueberry wants to merge 1 commit into
Open
kunkunblueberry wants to merge 1 commit into
kunkunblueberry wants to merge 1 commit into
Conversation
Move the startup KV budget arithmetic in plan_cache into kv_capacity(), a pure function over measured free memory, the utilization cap, and each non-KV overhead. The numbers are unchanged; the plan now also reports metadata_bytes and kv_committed_bytes so the committed capacity is observable. Covered by CPU unit tests. Part of RFC InferMatrixLab#192 (capacity planning). Storage and warmup/capture ordering are not changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
Answered and confirmed by the human contributor (@kunkunblueberry) before submission.
1. What outcome are you trying to achieve?
Size the KV pool from the memory actually available after runtime initialization, warmup and CUDA Graph capture, instead of relying on fixed per-pool estimates. This is the direction of RFC #192.
2. What is the highest-leverage variable for that outcome?
The number of committed KV blocks that remains once the measured free memory has been reduced by the utilization cap, the existing reservation and every non-KV overhead (probe peak, concurrent/async peak, graph reserve, static buffers, metadata).
3. How will this work move that variable rather than create busywork?
The startup budget arithmetic in
plan_cache()becomes a single pure function,kv_capacity(), over measured free memory, the utilization cap, the reservation and an explicit overhead map. A later measured-sizing change then has one place to change and a CPU-testable contract, and the plan now reportskv_committed_bytesandmetadata_bytes, making committed capacity observable. This PR is a preparatory step for RFC #192 step 1; it deliberately does not change the number of allocated blocks.4. Who is the customer or user, and what do they actually need?
Operators and maintainers deploying vllm-rlt on GPUs. They need capacity to be predictable and not to leave usable VRAM unused, or to run out of it, because of estimates that vary with the enabled execution paths.
PR necessity and bugfix evidence
A PR is needed: RFC #192 is an open, direction-defining RFC, the human contributor claimed the runtime KV capacity planning part in #192, and the outcome requires a code change. This PR is not a bugfix; there is no observed failure to link. It is the first, preparatory slice of the claimed scope.
Purpose
Part of RFC #192 (runtime KV cache capacity planning), split into several PRs. This PR covers capacity computation only:
plan_cache()intokv_capacity(), a pure function takingfree_bytes,cap_bytes,reserved_bytes,per_block,overheadsanduseful_blocks.min(free_bytes, cap - reserved) - sum(overheads), floored to whole blocks and capped byuseful_blocks. No capacity value changes for any existing configuration.plan_cache().metadata_bytesandkv_committed_bytesinmemory_plan, so the committed capacity (not only the budget before block rounding) is observable.Explicitly out of scope
gpu_memory_utilization,num_blocks/kv_cache_memory_bytesoverride semantics, orgraph_memory_reserve().These remain for follow-up PRs under the same RFC claim.
Test Plan
New
tests/test_kv_capacity.py(CPU-only, no GPU required):useful_blockscaps the block count without altering the reported budget;budget_blocks()floors partial blocks.Pre-existing suites are unchanged; all call sites still go through
plan_cache()with the same numbers.vllm-rlt Version / Commit: 0.2.0 /
0b1daacon top ofupstream/mainc338aaaTest Result
Environment: Windows, CPU-only
torch 2.12.1+cpu. No CUDA device was available, so thecuda_profilebranch ofplan_cache()was not executed. The extracted function is pure and fully covered on CPU; the CUDA path is a call-site refactor that passes the same values.python -m pytest tests/test_kv_capacity.py -q-> 5 passedpython -m pre_commit run ruff-check --all-files-> Passedpython -m pre_commit run ruff-format --all-files-> Passedpython -m pytest tests -q-> 1037 passed, 744 skipped, 20 failedThe 20 failures are pre-existing in this environment and unrelated to this change: the same 20 tests fail on a clean
upstream/main(c338aaa) checkout on the same host, and they are HTTP serving / control-channel tests affected by the Windows + CPU-only environment, not by memory planning. Thepre-commitpytesthook therefore fails here for the same environmental reason; the two ruff hooks pass.GPU validation of the end-to-end
cuda_profilepath (auto-sizing under a real CUDA device, and the existing GPU-marked tests) is still pending and will be done before this path is extended with measured post-capture sizing.