Enable the DCGM exporter on Civo, document it for existing clusters

design/serving-stack-generation.md

@@ -202,6 +202,7 @@ functions/compose-serving-stack/function/stacks/
     eks.py             # Written by nix run .#stacks. Never edited.
     aks.py
     gke.py             # Needs Modelplane's gke-cos profile fork; see below.
+    dcgm.py            # The DCGM exporter metrics CSV the clouds share; see below.
   common.py            # Hand-written. Modelplane's own, on every stack.
   standard.py          # Hand-written. Modelplane's own, where the stacks differ.
   dynamo.py
@@ -306,6 +307,17 @@ comes from.
 The DRA driver targets the contract and is still AICR's, because its version
 depends on the hardware.
 
+One generated value flows into the hand-written files rather than being
+restated: the DCGM exporter's metrics CSV. A hand-written cloud that installs
+the exporter runs the same one the generated clouds do, and a ~50-line CSV
+copy per file would be enforced only by review discipline. So the generator
+also writes `generated/aicr/dcgm.py` — the CSV, emitted once every regenerated
+cloud carried it byte-identical, failing closed when the clouds diverge or
+when a partial regeneration would move the shared module under clouds that
+didn't regenerate — and the hand-written files import it. The generated cloud
+files keep their inline copies and stay self-contained; a test asserts the
+copies on disk still match the shared module.
+
 The allowlist drops kai-scheduler for a different reason. AICR carries it in
 its base for every cluster, and taking that placement would even be harmless at
 runtime: a scheduler is opt-in, so on a Standard cluster it would only be dead

docs/content/platform/inference-cluster.md

@@ -82,6 +82,13 @@ An existing cluster must meet what Modelplane would otherwise set up for you:
 - **Any multi-node fabric you need.** For multi-node serving you provide and
   configure the RDMA or InfiniBand fabric and its drivers. Modelplane installs
   those only on the clouds it provisions.
+- **A DCGM exporter, if you want GPU metrics.** Modelplane doesn't install
+  one here because your cluster may already run one - a GPU Operator install
+  does by default - and a second exporter would collect every GPU twice.
+  Without one, engine and gateway series arrive and every `modelplane_gpu_*`
+  series is empty. [GPUs]({{< ref "/platform/telemetry.md#gpus" >}})
+  describes how the collector discovers the exporter, the DCGM fields it
+  needs, and how to check what yours publishes.
 
 ## Serving stack
 

docs/content/platform/telemetry.md

@@ -10,8 +10,11 @@ description: Collect normalized metrics across the fleet and send them to any ba
 
 Modelplane runs an OpenTelemetry collector on every inference cluster. It
 collects from every component Modelplane installs. This includes the inference
-server engine, inference gateway and Envoy proxy, router, and the GPU exporter
-your cloud provides. It renames each component's series to a common
+server engine, inference gateway and Envoy proxy, router, and the cluster's
+GPU exporter - on a provisioned cluster Modelplane or the cloud installs one,
+and on an existing cluster you provide it, as
+[Requirements for an existing cluster]({{< ref "/platform/inference-cluster.md#requirements-for-an-existing-cluster" >}})
+describes. It renames each component's series to a common
 `modelplane_*` vocabulary and exports them to wherever you say - any backend the
 collector has an exporter for, not only OTLP.
 
@@ -360,6 +363,57 @@ spec:
         max_tokens: length
 ```
 
+## GPUs
+
+The GPU series come from a [DCGM
+exporter](https://github.com/NVIDIA/dcgm-exporter), not from the engine: a GPU
+belongs to a node, and its series have a `node` label rather than a
+deployment. The collector discovers the exporter by its pod - an `app` or
+`app.kubernetes.io/name` label containing `dcgm`, with a container port named
+`metrics` - and scrapes it directly, not through Prometheus.
+
+Where the exporter comes from depends on the cluster:
+
+- **A cluster Modelplane provisions** runs one already: the serving stack
+  installs an exporter wherever the cloud doesn't provide its own.
+- **An existing cluster provides its own**, as [Requirements for an existing
+  cluster]({{< ref "/platform/inference-cluster.md#requirements-for-an-existing-cluster" >}})
+  describes. A GPU Operator install runs one by default; a cluster without
+  one needs NVIDIA's dcgm-exporter chart, tolerating your GPU taint. The
+  chart's pod already has the label and port the collector matches on.
+
+Don't run two exporters: the collector scrapes every matching pod and would
+collect each GPU twice.
+
+The exporter's metrics ConfigMap decides which DCGM fields it publishes.
+Modelplane maps seven to the `modelplane_*` surface and converts units where
+DCGM reports in different ones:
+
+| DCGM field | Becomes | Unit |
+| --- | --- | --- |
+| `DCGM_FI_DEV_FB_USED` | `modelplane_gpu_memory_used_bytes` | MiB, converted to bytes |
+| `DCGM_FI_PROF_GR_ENGINE_ACTIVE` | `modelplane_gpu_compute_active_ratio` | ratio |
+| `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` | `modelplane_gpu_tensor_active_ratio` | ratio |
+| `DCGM_FI_PROF_DRAM_ACTIVE` | `modelplane_gpu_memory_bandwidth_ratio` | ratio |
+| `DCGM_FI_DEV_GPU_TEMP` | `modelplane_gpu_temperature_celsius` | Celsius |
+| `DCGM_FI_DEV_POWER_USAGE` | `modelplane_gpu_power_watts` | watts |
+| `DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION` | `modelplane_energy_joules_total` | millijoules, converted to joules |
+
+Recent default exporter configurations publish all seven, but older ones left
+the `DCGM_FI_PROF_*` profiling fields out. Check what yours publishes - a
+missing field means its `modelplane_*` series is silently empty:
+
+```bash
+kubectl port-forward -n <exporter-namespace> ds/<exporter> 9400:9400 &
+curl -s localhost:9400/metrics | grep -c -E \
+  'DCGM_FI_DEV_FB_USED|DCGM_FI_PROF_GR_ENGINE_ACTIVE|DCGM_FI_PROF_PIPE_TENSOR_ACTIVE|DCGM_FI_PROF_DRAM_ACTIVE|DCGM_FI_DEV_GPU_TEMP|DCGM_FI_DEV_POWER_USAGE|DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION'
+```
+
+If fields are missing, extend the exporter's metrics ConfigMap - on a GPU
+Operator install that's the `dcgmExporter.config` chart value. The
+configuration Modelplane installs on the clouds it provisions publishes each
+of these fields.
+
 ## Why engine latency and gateway latency differ
 
 `modelplane_request_ttft_seconds` comes from the engine, and engines bucket their

docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt

@@ -246,6 +246,9 @@ GPU
 GPUs
 CDI
 CSI
+DCGM
+dcgm-exporter
+millijoules
 DRA
 NIC
 NFS

functions/compose-serving-stack/function/stacks/clouds/civo.py

@@ -17,7 +17,10 @@
 No generator covers Civo, so Modelplane pins the cloud half by hand,
 in the same shape a generator emits. Where a component also appears on
 a generated cloud, this file states the same pin, so one review moves
-both halves.
+both halves. The DCGM exporter's metrics CSV goes one step further:
+rather than restating it, this file imports it from the shared
+generated/aicr/dcgm.py, which the generator writes only after proving
+every generated cloud carries the same CSV.
 
 Civo pre-installs no GPU stack: its GPU node images ship the NVIDIA
 container toolkit but no driver, and there is no managed GPU Operator
@@ -37,6 +40,7 @@
 import copy
 from dataclasses import replace
 
+from function.stacks.clouds.generated.aicr.dcgm import DCGM_METRICS_CSV
 from function.stacks.components import Chart, Component, Manifests
 
 # The pool label compose-civo-cluster writes on every node pool's nodes.
@@ -267,17 +271,30 @@ def transformed(c: Component) -> Component:
         version="v26.3.3",
         wait=True,
         # No kube-prometheus-stack edge, unlike the generated clouds': the
-        # DCGM exporter (whose ServiceMonitor needs the Prometheus CRDs)
-        # is disabled below.
+        # DCGM exporter's ServiceMonitor, the one piece that would need
+        # the Prometheus CRDs, stays off below. The exporter itself must
+        # run: without it every modelplane_gpu_* series and
+        # modelplane_energy_joules_total is empty on Civo - the
+        # collector's modelplane-gpu job would find no pod to scrape.
         depends_on=["node-feature-discovery", "cert-manager"],
         values={
             "ccManager": {"enabled": False},
             "cdi": {"default": True, "enabled": True},
             # The operator's DaemonSets must run on the GPU nodes, which
             # carry the nvidia.com/gpu taint from compose-civo-cluster.
             "daemonsets": {"tolerations": [{"key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule"}]},
-            "dcgm": {"enabled": False},
-            "dcgmExporter": {"enabled": False},
+            # The CSV is the generator's, imported from the shared
+            # generated/aicr/dcgm.py. The ServiceMonitor stays off, unlike
+            # the generated clouds': the collector scrapes the exporter
+            # pods directly and Civo's Prometheus selects only
+            # PodMonitors, so nothing would read it - and off explicitly,
+            # so a chart default flip can't render one whose CRD this
+            # cluster doesn't carry.
+            "dcgm": {"enabled": True},
+            "dcgmExporter": {
+                "config": {"create": True, "data": DCGM_METRICS_CSV, "name": "dcgm-exporter"},
+                "serviceMonitor": {"enabled": False},
+            },
             # The DRA driver is the sole GPU allocator: the device
             # plugin's nvidia.com/gpu ledger would book the same physical
             # devices the DRA ResourceSlices allocate.

functions/compose-serving-stack/function/stacks/clouds/generated/aicr/__init__.py

@@ -14,7 +14,9 @@
 
 """AICR-derived cloud halves, written by `nix run .#stacks`.
 
-The modules beside this file (eks.py, aks.py, gke.py) are generated by
+The modules beside this file (eks.py, aks.py, gke.py, and dcgm.py, the
+DCGM exporter metrics CSV those three share and the hand-written clouds
+that run the same exporter import) are generated by
 functions/compose-serving-stack/generate.py from AICR recipes and are
 never edited by hand.
 """

functions/compose-serving-stack/function/stacks/clouds/generated/aicr/dcgm.py

@@ -0,0 +1,87 @@
+# Copyright 2026 The Modelplane Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Generated by generate.py. Do not edit.
+# Regenerate with `nix run .#stacks`.
+#
+# The DCGM exporter's metrics CSV, identical on every cloud AICR
+# covers - emit_dcgm fails the run when the clouds diverge. The
+# generated files beside this one carry the CSV inline and stay
+# self-contained; the hand-written clouds that run the same
+# exporter import it from here, so an aicr bump moves them in the
+# same review as the generated clouds.
+
+DCGM_METRICS_CSV = """# Clocks,,
+DCGM_FI_DEV_SM_CLOCK,     gauge, SM clock frequency (in MHz).
+DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).
+# Temperature,,
+DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
+DCGM_FI_DEV_GPU_TEMP,    gauge, GPU temperature (in C).
+# Power,,
+DCGM_FI_DEV_POWER_USAGE,  gauge, Power draw (in W).
+DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).
+# PCIe,,
+DCGM_FI_PROF_PCIE_TX_BYTES,  counter, Total number of bytes transmitted through PCIe TX (in KB) via NVML.
+DCGM_FI_PROF_PCIE_RX_BYTES,  counter, Total number of bytes received through PCIe RX (in KB) via NVML.
+DCGM_FI_DEV_PCIE_REPLAY_COUNTER, counter, Total number of PCIe retries.
+# Utilization (the sample period varies depending on the product),,
+DCGM_FI_DEV_GPU_UTIL,      gauge, GPU utilization (in %).
+DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %).
+DCGM_FI_DEV_ENC_UTIL,      gauge, Encoder utilization (in %).
+DCGM_FI_DEV_DEC_UTIL,      gauge, Decoder utilization (in %).
+# Errors and violations,,
+DCGM_FI_DEV_XID_ERRORS,            gauge, Value of the last XID error encountered.
+DCGM_FI_DEV_POWER_VIOLATION,       counter, Throttling duration due to power constraints (in us).
+DCGM_FI_DEV_THERMAL_VIOLATION,     counter, Throttling duration due to thermal constraints (in us).
+DCGM_FI_DEV_SYNC_BOOST_VIOLATION,  counter, Throttling duration due to sync-boost constraints (in us).
+DCGM_FI_DEV_BOARD_LIMIT_VIOLATION, counter, Throttling duration due to board limit constraints (in us).
+DCGM_FI_DEV_LOW_UTIL_VIOLATION,    counter, Throttling duration due to low utilization (in us).
+DCGM_FI_DEV_RELIABILITY_VIOLATION, counter, Throttling duration due to reliability constraints (in us).
+# Memory usage,,
+DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB).
+DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
+# Retired pages,,
+DCGM_FI_DEV_RETIRED_SBE,     counter, Total number of retired pages due to single-bit errors.
+DCGM_FI_DEV_RETIRED_DBE,     counter, Total number of retired pages due to double-bit errors.
+DCGM_FI_DEV_RETIRED_PENDING, counter, Total number of pages pending retirement.
+# NVLink,,
+DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL, counter, Total number of NVLink bandwidth counters for all lanes
+DCGM_FI_PROF_NVLINK_TX_BYTES,       counter, The rate of data transmitted over NVLink not including protocol headers in bytes per second.
+DCGM_FI_PROF_NVLINK_RX_BYTES,       counter, The rate of data received over NVLink not including protocol headers in bytes per second.
+# Add DCP metrics,,
+DCGM_FI_PROF_GR_ENGINE_ACTIVE,   gauge, Ratio of time the graphics engine is active (in %).
+DCGM_FI_PROF_SM_ACTIVE,          gauge, The ratio of cycles an SM has at least 1 warp assigned (in %).
+DCGM_FI_PROF_SM_OCCUPANCY,       gauge, The ratio of number of warps resident on an SM (in %).
+DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active (in %).
+DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data (in %).
+DCGM_FI_PROF_PCIE_TX_BYTES,      counter, The number of bytes of active pcie tx data including both header and payload.
+DCGM_FI_PROF_PCIE_RX_BYTES,      counter, The number of bytes of active pcie rx data including both header and payload.
+# BCP Additional metrics
+DCGM_FI_DEV_GPU_NVLINK_ERRORS,      gauge, Identifies a GPU NVLink error type returned by DCGM_FI_DEV_GPU_NVLINK_ERRORS.
+# Added RunAI Additional metrics https://docs.run.ai/latest/developer/metrics/metrics-api/#advanced-metrics
+## NVLink
+DCGM_FI_DEV_NVLINK_BANDWIDTH_L0, counter, The number of bytes of active NVLink rx or tx data including both header and payload.
+## VGPU License status
+DCGM_FI_DEV_VGPU_LICENSE_STATUS, gauge, vGPU License status
+## Remapped rows
+DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for uncorrectable errors
+DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for correctable errors
+DCGM_FI_DEV_ROW_REMAP_FAILURE, gauge, Whether remapping of rows has failed
+## Static configuration information. These appear as labels on the other metrics
+DCGM_FI_DRIVER_VERSION, label, Driver Version
+## Profiling metrics
+DCGM_FI_PROF_PIPE_FP64_ACTIVE, gauge, Ratio of cycles the fp64 pipes are active (in %).
+DCGM_FI_PROF_PIPE_FP32_ACTIVE, gauge, Ratio of cycles the fp32 pipes are active (in %).
+DCGM_FI_PROF_PIPE_FP16_ACTIVE, gauge, Ratio of cycles the fp16 pipes are active (in %).
+"""

functions/compose-serving-stack/generate.py

@@ -19,6 +19,9 @@
 function/stacks/clouds/generated/aicr/, each a COMPONENTS list of Chart
 and Manifests entries that the function joins with the hand-written
 common.py and the stack's own file (see function/stacks/__init__.py).
+Beside them it writes dcgm.py, the DCGM exporter metrics CSV proven
+identical across the clouds, for the hand-written clouds that run the
+same exporter (see emit_dcgm).
 
 Per cloud, one `aicr recipe` at Modelplane's coordinate (service +
 intent, os only where a recipe covers Modelplane's node image - see the
@@ -1169,7 +1172,58 @@ def emit(
     (OUT / f"{cloud}.py").write_text(text)
 
 
-def regenerate(cloud: str, workdir: pathlib.Path, catalog: pathlib.Path) -> None:
+def exporter_csv(cloud: str, components: list[Values]) -> str:
+    """The DCGM exporter metrics CSV in the cloud's gpu-operator values."""
+    for entry in components:
+        if entry.get("key") == "gpu-operator":
+            data = lookup(entry["values"], ("dcgmExporter", "config", "data"))
+            if isinstance(data, str):
+                return data
+    sys.exit(f"{cloud}: no dcgm-exporter metrics CSV in the gpu-operator values - fail closed")
+
+
+def render_dcgm(csv: str) -> str:
+    lines = [
+        LICENSE_HEADER,
+        "",
+        "# Generated by generate.py. Do not edit.",
+        "# Regenerate with `nix run .#stacks`.",
+        "#",
+        "# The DCGM exporter's metrics CSV, identical on every cloud AICR",
+        "# covers - emit_dcgm fails the run when the clouds diverge. The",
+        "# generated files beside this one carry the CSV inline and stay",
+        "# self-contained; the hand-written clouds that run the same",
+        "# exporter import it from here, so an aicr bump moves them in the",
+        "# same review as the generated clouds.",
+        "",
+        f"DCGM_METRICS_CSV = {pyfmt(csv)}",
+        "",
+    ]
+    text = "\n".join(lines)
+    ast.parse(text)  # a generated file that doesn't parse fails the build
+    return text
+
+
+def emit_dcgm(csvs: dict[str, str]) -> None:
+    """Write the shared metrics CSV module the hand-written clouds import."""
+    if len(set(csvs.values())) > 1:
+        sys.exit(
+            f"dcgm-exporter metrics CSV differs between {', '.join(sorted(csvs))}; "
+            "the hand-written clouds import one shared copy - decide which they follow"
+        )
+    text = render_dcgm(next(iter(csvs.values())))
+    out = OUT / "dcgm.py"
+    if set(csvs) != set(CLOUDS) and out.exists() and out.read_text() != text:
+        sys.exit(
+            f"the dcgm-exporter metrics CSV moved but only {', '.join(sorted(csvs))} "
+            "regenerated; regenerate every cloud (`nix run .#stacks` with no "
+            "arguments) so dcgm.py moves with all of them"
+        )
+    out.write_text(text)
+    print(f"generated {out.relative_to(ROOT)}")
+
+
+def regenerate(cloud: str, workdir: pathlib.Path, catalog: pathlib.Path) -> str:
     spec = CLOUDS[cloud]
     flags = ["--service", cloud, "--intent", "inference"]
     if spec["os"]:
@@ -1200,6 +1254,7 @@ def regenerate(cloud: str, workdir: pathlib.Path, catalog: pathlib.Path) -> None
 
     digest = hashlib.sha256(recipe_file.read_bytes()).hexdigest()
     emit(cloud, spec, components, recipe, digest, floor)
+    return exporter_csv(cloud, components)
 
 
 def main(clouds: list[str]) -> None:
@@ -1214,9 +1269,11 @@ def main(clouds: list[str]) -> None:
     with tempfile.TemporaryDirectory(prefix="serving-stack-gen-") as tmp:
         workdir = pathlib.Path(tmp)
         catalog = write_catalog(workdir)
+        csvs = {}
         for cloud in clouds or CLOUDS:
-            regenerate(cloud, workdir, catalog)
+            csvs[cloud] = regenerate(cloud, workdir, catalog)
             print(f"generated {OUT.relative_to(ROOT)}/{cloud}.py")
+        emit_dcgm(csvs)
 
 
 if __name__ == "__main__":