Skip to content

Some shipped recipes and cloud stacks leave built-in metrics empty #524

Closed

Description

@dennis-upbound

What happened?

Some configurations Modelplane ships leave built-in modelplane_* metrics empty, with no error. The collector drops everything it didn't rename, so when a source metric isn't emitted, the modelplane_* name just never appears at the sink. In each case below Modelplane's own manifests or stacks produce the gap.

Engine recipes that don't emit the mapped names

Cloud stacks that install no GPU exporter

  • Civo turns the exporter off: civo.py#L280 sets dcgmExporter.enabled: False because its ServiceMonitor needs the Prometheus CRDs (#L268-L272). Every modelplane_gpu_* series and modelplane_energy_joules_total is empty on Civo.

#519 covers the same GPU gap for source: Existing clusters, where Modelplane installs nothing. This is about clusters Modelplane builds.

How can we reproduce it?

  1. Deploy the llama-3.1-8b recipe, or either SGLang recipe, with a TelemetryDestination configured.
  2. Send it some requests.
  3. Query the sink for modelplane_kv_cache_utilization_ratio (llama) or modelplane_requests_running (SGLang). No series exist for that deployment.

For GPUs, provision an inference cluster on Civo, deploy any model, and query modelplane_gpu_memory_used_bytes.

What environment did it happen in?

Modelplane version: main at 0ee6ab4 (also v0.5.0)
Engines: vLLM v0.7.3, SGLang v0.5.10.post1 and v0.5.12.post1, as pinned in the recipes
Clouds: Civo (from the stack code)
Found by reading the code and upstream source at the pinned versions, not on a live cluster

Activity

  1. dennis-upbound commented on Oct 20, 2026

    @dennis-upbound
    CollaboratorAuthor

    #526 fixed the Civo half: the stack now runs the DCGM exporter. The recipes remain, and I'd fix them for v0.6:

    1. Move llama-3.1-8b off vLLM v0.7.3. That also gives it the same latency buckets as every other vLLM deployment (Built-in metric mappings merge histograms whose buckets don't match #523).
    2. Add --enable-metrics to both SGLang recipes. Add --collect-tokens-histogram too while qwen3-coder stays on v0.5.10.post1. v0.5.12 always collects the histograms and accepts the flag as a no-op.
    3. Correct telemetry.md#L248. "vLLM needs nothing" holds from v0.9.2.
    4. Add a test over docs/manifests/recipes/ that fails on an SGLang engine without --enable-metrics, or on a vLLM image older than v0.9.2. Nothing checks recipes against the mappings today, which is how these shipped. The e2e test only checks two names, from a mock engine.
  2. moved this from Backlog to In progress in Modelplane Roadmapon Oct 20, 2026
  3. added 12 commits that reference this issue on Oct 20, 2026
    4100131
    9213d61
    62b44bb
    d867e6c
    a49c4fc
    ada68a2
    0d20097
    6aafd50
    3ebedd9
    bef5e2d
    83cbbb9
    1591000
  4. moved this from In progress to In Review in Modelplane Roadmapon Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions