[dennis-upbound]

Description of your changes

Modelplane writes the prefill/decode router itself, and it writes only one: the llm-d sidecar with --kv-connector=nixlv2, VLLM_NIXL_* environment on each engine of the pair, and a picker configuration whose decider skips prefill for short prompts. An SGLang prefill/decode deployment gets that router anyway. It reports Ready, and its decode engine rejects every request (#502):

Invalid request: Disaggregated request received without bootstrap room id

Each new KV connector would need a Modelplane release, and each llm-d router release can break the router Modelplane writes.

This PR adds design/user-authored-routing.md. It proposes that a PrefillDecode deployment writes its own router: the endpoint picker's container, command line and configuration, and the llm-d sidecar as a container beside engine in the decode member (#108). Modelplane passes the picker what only it knows as environment, and fixes only its ports, so any picker that speaks the protocol fits, not just llm-d's.

serving:
  mode: PrefillDecode
  router: EndpointPicker      # required under PrefillDecode
  endpointPicker:             # required with router
    template:
      spec:
        containers:
        - name: epp
          image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
          args:
          - --pool-name=$(MODELPLANE_POOL_NAME)
          - --config-file=$(MODELPLANE_PICKER_CONFIG)
          # ...
    config: { ... }

Unified keeps a default. Setting router replaces it with the user's picker. Prefill/decode doesn't get one. A wrong Unified default costs some load-balancing quality, but a wrong prefill/decode default gives a pair that reports Ready and rejects requests, and a prefill/decode deployment's engines are tied to one serving stack anyway.

This breaks existing PrefillDecode deployments. Their replicas stop reconciling and keep serving as they are, until the deployment adds its routing.

The calls that need review are no default for prefill/decode, the picker as a pod template whose command line the user writes, and the breaking change.

#529 builds it, with the API change, tests and user docs. It ran on GKE with SGLang and vLLM prefill/decode pairs and a vLLM decode gang, and each one disaggregated.

Refs #502, #108, #179, #418, #500

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes.
• Signed off every commit with git commit -s.

🤖 Generated with Claude Code

[negz]

Have you thought about how this works on an InferenceCluster that doesn't route with llm-d? I'm thinking of the composed DynamoGraphDeployment in serving-stacks.md. There, Dynamo's operator composes the picker itself, the picker reads DYN_* env vars rather than an EndpointPickerConfig, and P/D runs through Dynamo's own frontend sidecar, not llm-d's.

As written, endpointPicker reads as router-neutral, but its config only means something to llm-d. A P/D deployment also has to write llm-d's sidecar into its decode template even when it omits endpointPicker, so it looks portable but only runs where llm-d routes. And existing P/D deployments have to be rewritten before an upgrade.

One option is to name the router with a discriminator, and make omitting it mean "whatever this cluster routes with":

serving:
  mode: PrefillDecode
  # Optional, with no default. Omitted, the cluster picks the routing. On today's
  # clusters that's what Modelplane composes now, sidecar and NIXL env included,
  # so the Kimi-K2 recipe works unchanged.
  router: EndpointPicker
  endpointPicker:
    image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
    config: {...}

With router set, the author writes the routing, including the sidecar in the decode template as the doc has it now, and the scheduler only places the deployment on clusters that can compose that router. A Dynamo variant can come later without breaking anything.

I haven't thought this through properly (for one, the default path keeps the flag peeking you're removing), so please poke holes in it rather than taking it as-is.

[dennis-upbound]

You're right, this was too llm-d focused. The picker config and the sidecar only mean something where llm-d routes, so every P/D deployment would have been pinned to llm-d and needed a rewrite. I've taken your direction: serving.router is optional with no default. Omitted, the cluster routes as today, flag peeking included, so nothing breaks and a future DGD cluster can route the same deployment its own way. router: EndpointPicker is the user-written path the doc already had. I've also added placement by the routers a cluster advertises, and the resolved router in replica status. We can keep the flag peeking on the default path for now and revisit it separately.

[negz]

Thinking about this more, I wonder whether PrefillDecode should have a default at all. What if it required router and the picker config, and only Unified kept a default?

My reasoning is that a wrong default under Unified costs some load-balancing quality, but under PrefillDecode it gives a pair that reports Ready and rejects requests, like #502. A default that suits both vLLM and SGLang, such as always-disagg-pd-decider, looks tempting, but it has Modelplane asserting which engines its config works for again, and I'd expect the next engine that doesn't fit to fail the same way.

serving:
  mode: PrefillDecode
  router: EndpointPicker      # required under PrefillDecode?
  endpointPicker:
    image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
    config: {...}             # required under PrefillDecode?

If that holds up, it might also remove the catch from my comment above. With no P/D default, Modelplane's P/D code would shrink to wiring, and the vLLM-specific injection and flag reading would go. Existing P/D deployments like the Kimi-K2 recipe would have to add their routing, which seems OK to me before v1.

I haven't thought this through properly either, so I'm curious whether it matches what you've seen.

[dennis-upbound]

Agreed, no default for P/D is better, and it matches what I found. A P/D deployment's engines are already specific to one serving stack (vllm serve behind llm-d's sidecar vs. Dynamo's wrapper), so a default bought no portability. It only hid a vLLM-with-NIXL assumption, and I'd ended up adding a check that warned when the engines didn't match it, which is exactly Modelplane asserting which engines its config works for.

So PrefillDecode now requires router: EndpointPicker and endpointPicker.config, and only Unified keeps the cluster's routing. The P/D code drops the sidecar and NIXL injection, the --port peek and that check, and the port rule loses its exceptions. The Kimi-K2 recipe writes its own routing, and the docs publish the default P/D config as a vLLM starting point to copy.

[dennis-upbound]

@negz @haarchri ready for another look. The design now matches #529 after Chris's review, and #529 ran on GKE at ea2ad28 with SGLang and vLLM prefill/decode pairs and a decode gang.

[negz]

The design looks good to me apart from the picker command line. I lean toward the pod template, but I'm happy to approve as is if you think it doesn't hold up. The other comments are small and shouldn't block.

[negz]

I wonder whether Modelplane should write this command line at all. It's llm-d's spelling, so router: EndpointPicker only fits pickers that accept it. As far as I can tell that's only llm-d's. Dynamo's picker reads DYN_* env vars, and the only picker in the Inference Extension's 1.6 releases is the lightweight one, where I couldn't find --config-file.

What if we treated the picker the way the base design treats the engine? Modelplane passes what only it knows, such as the pool's name and where it mounts the config, as MODELPLANE_ env vars. The user writes the args, and the fixed contract is just the ports. Shaping it as the same curated pod template subset the engines use means later fields like resources land in both places:

serving:
  mode: PrefillDecode
  router: EndpointPicker
  endpointPicker:
    template:
      spec:
        containers:
        - name: picker
          image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
          args:
          - --pool-name=$(MODELPLANE_POOL_NAME)
          - --pool-namespace=$(MODELPLANE_POOL_NAMESPACE)
          - --pool-group=inference.networking.k8s.io
          - --config-file=$(MODELPLANE_PICKER_CONFIG)
          - --grpc-port=9002
          - --grpc-health-port=9003
          - --metrics-port=9090
          - --metrics-endpoint-auth=false
    config: {...}   # Modelplane mounts it at $(MODELPLANE_PICKER_CONFIG)

That would make the name EndpointPicker accurate, since any picker that serves the gateway on 9002 and health on 9003 fits. The alternative is renaming it LLMD to match the contract Modelplane hard-codes today.

Curious whether this matches what you saw building #529.

[dennis-upbound]

It does match. Building #529, the command line was the part that kept changing with llm-d: --grpc-health-port arrived in v0.9.0, and --metrics-endpoint-auth=false only showed up when every scrape failed. You're right about the Inference Extension too: its 1.6 picker, lwepp, has no --config-file or --metrics-endpoint-auth, so it would exit on the command line we write.

So the design now takes your shape. endpointPicker.template is a curated pod template with one container, epp, whose image and args the user writes. Modelplane sets MODELPLANE_POOL_NAME, MODELPLANE_POOL_NAMESPACE and, with a config, MODELPLANE_PICKER_CONFIG. The fixed contract is the ports: 9002, 9003 for the readiness probe, and 9090 for metrics. config is optional, since lwepp takes none. A Unified deployment without router still gets llm-d's picker, now written the same way. The old shape and router: LLMD are under Alternatives. #529 implements it, with resources from the template.

[negz]

Could we file the issue for this before merging, so the #TBD points somewhere?

[dennis-upbound]

Filed #552 and linked it.

[negz]

Nit, not blocking: since aligning with #529, some sections read more like a spec of the code than a design. The CEL rules here, rendering the config as JSON with integers, and the labels the upgrade drops are examples. I'd lean toward keeping the rules a reader needs to write a deployment, and leaving the rest to the code and user docs, but I don't feel strongly.

[dennis-upbound]

Agreed, trimmed. The CEL rules are now one sentence on what Modelplane rejects, and the JSON rendering, the dropped labels and the container and port limits are gone. Those stay in the code and the user docs.

[negz]

I'd make these ports env vars too, even though they're pretty static. It helps clarify they're part of the contract and that Modelplane has expectations about what they are. (It also lets us change port for metrics if we want to in future.)

[dennis-upbound]

Agreed, doing it in #529 (40b87aa). Modelplane sets MODELPLANE_GRPC_PORT, MODELPLANE_GRPC_HEALTH_PORT and MODELPLANE_METRICS_PORT beside the pool and config path, the default picker and the examples pass them as --grpc-port=$(MODELPLANE_GRPC_PORT) and so on, and the design table lists them.

[negz]

Nit: The design would be easier to read if these issue refs were clickable links.

[dennis-upbound]

Done in #529 too: every issue reference in the design is a link now.