Repository navigation
PrefillDecode composes vLLM's disaggregation onto any engine, so SGLang P/D silently never transfers KV #502
Description
What happened?
A ModelReplica with serving.mode: PrefillDecode composes vLLM's disaggregation plumbing onto whatever engine it is given, including SGLang. The objects apply cleanly, both pods start, and no condition reports a problem — but no KV block is ever transferred from prefill to decode, because SGLang is not configured for disaggregation and nothing pairs the two halves.
Rendering a PrefillDecode replica whose engine is lmsysorg/sglang, on 4616c91 (v0.5.0):
p pod containers: ['engine']
engine env: ['VLLM_NIXL_SIDE_CHANNEL_HOST', 'VLLM_NIXL_SIDE_CHANNEL_PORT']
d pod containers: ['engine', 'pd-sidecar']
engine env: ['VLLM_NIXL_SIDE_CHANNEL_HOST', 'VLLM_NIXL_SIDE_CHANNEL_PORT']
pd-sidecar args: ['--secure-proxy=false', '--kv-connector=nixlv2', '--vllm-port=8000']
Three things there are vLLM's, not the engine's:
VLLM_NIXL_SIDE_CHANNEL_HOST/_PORTare injected into both engines (routing.py#L457). SGLang reads neither, so the side channel peers never find each other.- The decode sidecar is told
--vllm-portand--kv-connector=nixlv2(routing.py#L419), which is vLLM's NixlConnector protocol. - Nothing emits SGLang's own disaggregation configuration.
grep -rn "disaggregation" functions/ --include="*.py"returns nothing: no--disaggregation-mode, no--disaggregation-transfer-backend, no bootstrap server.
The architectures differ in a way the composition can't paper over. vLLM pairs prefill and decode through the decode-side sidecar, which is why the llm-d endpoint picker is enough. SGLang pairs them through a router that owns the handshake — sglang-router, or mini_lb in the reference setups. Modelplane composes no such component, and a user can't add one: spec.template.spec.containers is maxItems: 1 and CEL-constrained to a container named engine (definition.yaml#L456), per #108.
The code already knows SGLang's P/D exists — routing.py#L305 notes that "vLLM's NixlConnector (and SGLang's PD transfer) import the nixl package" — but only vLLM's plumbing is injected.
The failure is silent, which is the part worth fixing first. PrefillDecode is documented without an engine restriction, so this reads as supported. Either Modelplane should compose SGLang's disaggregation, or PrefillDecode should reject an engine it cannot wire up, rather than composing something that cannot work.
Found by @pluna while checking rack-scale disaggregated SGLang on GB300 NVL72 against the published inferenceX configurations. Related: #500 (DP-attention routing) and #108 (single container) share the root cause — the router component SGLang needs is one Modelplane neither composes nor permits.
How can we reproduce it?
No GPU required; the gap is visible at composition time.
-
Apply a
ModelDeploymentwithserving.mode: PrefillDecodewhose engine image islmsysorg/sglang:v0.5.10.post1-runtime, with a Prefill and a Decode engine. -
Inspect the composed workloads:
kubectl get object model-serving-<engine> -o jsonpath='{.spec.forProvider.manifest.spec.template.spec}' | jq -
Both engine containers carry
VLLM_NIXL_SIDE_CHANNEL_HOST, the decode pod carries apd-sidecarwith--vllm-port, and neither carries any SGLang disaggregation flag. Everything reports Ready.
There is no workaround within the API: supplying a router needs a second container, which the schema rejects.
What environment did it happen in?
Modelplane version: v0.5.0 (4616c91)
Engine: SGLang (lmsysorg/sglang); affects any engine that is not vLLM.
Reproduced by rendering the composition functions directly; no cluster or GPU involved.
Activity
dennis-upbound commentedon Oct 8, 2026
Rendered the shape @pluna needs for GLM, to see how much of this needs a GB300. None of it does, except the NVLink placement in #503.
The shape is expressible. Three prefill instances of 4 GPUs and one 8-GPU multi-node decode gang has to be written as
copies: 3on one Prefill engine, not three Prefill engines — CEL enforcessize(self.engines) == 2for PrefillDecode, so the alternative is refused at admission rather than half-composed.select_backendreads member roles and the stack and never readsphase, so a phase can be a gang:On that shape with
lmsysorg/sglang, this issue reproduces as filed: both engines getVLLM_NIXL_SIDE_CHANNEL_HOSTand_PORT, and decode gets apd-sidecarwith--kv-connector=nixlv2.The plumbing also stops at the leader, which is a second fault and not the one this issue describes:
A worker in a phased gang gets no side-channel env and no sidecar. That is independent of the engine: it bites a vLLM multi-node decode gang too, which is the GLM configuration. Whether a TP worker needs its own side channel is a question for someone who knows vLLM's NIXL connector, but nothing in the code reads as a decision either way — the disaggregation layer appears to assume a phase is one pod.
Related, on the same render: #424 is sharper than its text on a multi-node shape. On
Dynamo, prefill composes a plainDeploymentand never reaches Grove or KAI at all, so the pair is one gang and one thing the gang scheduler cannot see. For a 20-GPU GLM pair, KAI can place the 8-GPU decode gang and leave 12 GPUs of prefill to the default scheduler.So there are three faults behind "PrefillDecode does not work for SGLang at rack scale", and they are separable: the layer assumes vLLM (this issue), assumes a phase is a single pod (above), and assumes the pair needs no co-scheduling (#424). Each reproduces on a two-phase render with no GPU.
dennis-upbound commentedon Oct 8, 2026
Answering the open question from the comment above: leader-only is wrong, and the value we inject is already right.
VLLM_NIXL_SIDE_CHANNEL_HOSTdefaults tolocalhost(vllm/envs.py:231). vLLM's connector doc is explicit that the side channel is per worker process and that its address travels to the remote peer:So a worker pod with no
VLLM_NIXL_SIDE_CHANNEL_HOSTadvertiseslocalhostto the peer, which cannot route to it. A multi-node phase has every worker in that state.The fix is narrow. routing.py:457 already sets the host from
fieldRef: status.podIP, which is correct per pod. What is wrong is which pods it reaches: the env is applied by walking_serving_pod_templates, which selects pods carrying the serving label. That is leaders only, and it is the right set for the InferencePool, which must never route to a worker. It is the wrong set for KV transfer, which needs every pod running a vLLM worker process.Two sets, one walk. Separating them is the change.
Propose KV transfer protocols for prefill/decode serving
Propose KV transfer formats for prefill/decode serving
Propose KV transfer formats for prefill/decode serving
Propose KV transfer formats for prefill/decode serving
Propose KV transfer formats for prefill/decode serving
Propose KV transfer formats for prefill/decode serving
Propose KV transfer formats for prefill/decode serving
Propose KV transfer formats for prefill/decode serving
Propose KV transfer formats for prefill/decode serving
Propose KV transfer formats for prefill/decode serving
Require the router for PrefillDecode, and keep a default only for Uni…