Skip to content

Enable the DCGM exporter on Civo, document it for existing clusters - #526

Merged
dennis-upbound merged 2 commits into
modelplaneai:mainfrom
haarchri:fix/ss-civo
Oct 20, 2026
Merged

dennis-upbound merged 2 commits into
modelplaneai:mainfrom
haarchri:fix/ss-civo

Conversation

@haarchri

@haarchri haarchri commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description of your changes

On a Civo cluster Modelplane provisions, every modelplane_gpu_* series and modelplane_energy_joules_total was empty. The hand-written Civo stack disabled the GPU Operator's DCGM exporter - not catched as other handwritten installing the GPU Operator as pre-req.
The telemetry collector discovers a GPU exporter by pod label, so with no exporter DaemonSet there was nothing to scrape.

This enables the exporter on Civo with the same dcgm/dcgmExporter values the generated clouds pin, so one review moves both halves; the metrics CSV covers every field metrics.py maps. The gpu-operator chart gains a kube-prometheus-stack edge: unlike the generated clouds, Civo's kube-prometheus-stack keeps its bundled CRDs enabled, so that release is what provides the ServiceMonitor CRD, and it now sets wait=True as a depended-on chart must.

On source: Existing clusters Modelplane doesn't install an exporter, on purpose: it can't know whether the cluster already runs one, and the collector scrapes every pod matching .*dcgm.*, so a second exporter would collect each GPU twice. The docs now say so. A new GPUs section in the telemetry page documents the discovery contract, the seven DCGM fields with their unit conversions, and how to check what an exporter publishes; the existing-cluster requirements gain a DCGM exporter bullet pointing at it.

Fixes #519.
Towards #524.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

Validated

on a live Civo NYC1 cluster: all seven renamed series arrive at the collector's debug sink.

    kubectl logs deploy/modelplane-collector -n modelplane-system | grep -E 'modelplane_gpu_|modelplane_energy' | sort -u
    -> Name: modelplane_energy_joules_total
    -> Name: modelplane_gpu_compute_active_ratio
    -> Name: modelplane_gpu_memory_bandwidth_ratio
    -> Name: modelplane_gpu_memory_used_bytes
    -> Name: modelplane_gpu_power_watts
    -> Name: modelplane_gpu_temperature_celsius
    -> Name: modelplane_gpu_tensor_active_ratio

@github-actions

Copy link
Copy Markdown

Docs preview: https://modelplane-docs-pr-526.vercel.app (ready once the site's Content workflow finishes)

@dennis-upbound dennis-upbound left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree on not installing an exporter on Existing. It's the same call as the GPU driver: a cluster running the GPU Operator already has one, and a second double-counts every GPU, which is worse than empty series. Documenting the contract plus the curl check is right.

On the Civo half:

  1. Is the ServiceMonitor needed? The collector scrapes the exporter pods directly, and Civo's Prometheus only opens up PodMonitor selection, so as far as I can tell nothing reads it. With dcgmExporter.serviceMonitor.enabled: false, the new kube-prometheus-stack edge, its wait=True, and the Usage all go away, and the GPU Operator and DRA driver don't wait on a healthy Prometheus. The cost is Civo's values no longer matching the generated clouds verbatim. Your call on which matters more.
  2. The CSV is a copy. It matches GKE/EKS/AKS today, but nothing enforces it. A test asserting Civo's CSV equals the generated one would make "one review moves both" true.
  3. Nit: the E501 ignore covers all of civo.py; a # noqa: E501 on the CSV string would keep it scoped.
  4. Nit: the new inference-cluster.md bullet repeats most of the telemetry GPUs section. A couple of lines and a link would do.

…for existing

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
…le serviceMonitor

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
@haarchri

Copy link
Copy Markdown
Collaborator Author

updated the PR:

  • ServiceMonitor: dropped. Verified your read
  • CSV: went one further than the test, the aicr generator now emits a shared generated/aicr/dcgm.py (fails closed if the clouds CSVs ever diverge) and civo.py imports it. A test pins the generated inline copies to the shared module.
  • E501: with the literal gone, civo.py has no long lines, ignore deleted entirely.
  • Docs bullet: trimmed to the contract plus a link to the telemetry GPUs section.

@dennis-upbound dennis-upbound left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, both points are handled, and the generated dcgm.py with its fail-closed checks is better than the test I suggested. Two nits before merge: the commit subjects carry typed prefixes (fix(servingstack):, feat(refactor):), which CONTRIBUTING.md asks us to drop, and the new paragraph in serving-stack-generation.md could lose its em dashes and split its long middle sentence.

@haarchri haarchri moved this from Backlog to In Review in Modelplane Roadmap Oct 19, 2026
@dennis-upbound
dennis-upbound merged commit 563a3f2 into modelplaneai:main Oct 20, 2026
6 checks passed
@negz negz moved this from In Review to Done in Modelplane Roadmap Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Docs don't say how to get GPU metrics on a source: Existing cluster

3 participants