From 0219e3ce18e3c59b5a82053c5b939150dafc1934 Mon Sep 17 00:00:00 2001 From: Rae Sharp Date: Thu, 22 Oct 2026 12:57:37 -0400 Subject: [PATCH] Document gateway metric limits in telemetry docs The telemetry docs didn't say that the gateway records modelplane_frontend_ttft_seconds and modelplane_frontend_tpot_seconds only for streaming responses, so a deployment without streaming traffic has no series for either. They also didn't say that the gateway's histograms stop at fixed upper buckets: 81.92 seconds for request duration, 60 for time to first token, and 2.5 for time per output token in AI Gateway v1.1.0. A quantile above the top bucket reads as that bound no matter how long the request took. The metrics table and the p99 example now say both TTFT and TPOT need streaming traffic, and a note points non-streaming deployments at modelplane_frontend_request_duration_seconds. The bucket bounds are listed, and long requests are pointed at the engine's modelplane_request_duration_seconds, whose vLLM buckets reach 7680 seconds. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Rae Sharp --- docs/content/platform/telemetry.md | 25 ++++++++++++++++--- .../config/vocabularies/Modelplane/accept.txt | 1 + 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/docs/content/platform/telemetry.md b/docs/content/platform/telemetry.md index 99f565e15..30fc325d1 100644 --- a/docs/content/platform/telemetry.md +++ b/docs/content/platform/telemetry.md @@ -79,8 +79,8 @@ Some examples of the available metrics: | Metric | Means | | --- | --- | -| `modelplane_frontend_ttft_seconds` | Time to the first token, measured at the gateway | -| `modelplane_frontend_tpot_seconds` | Time per output token, measured at the gateway | +| `modelplane_frontend_ttft_seconds` | Time to the first token, measured at the gateway. Streaming requests only. | +| `modelplane_frontend_tpot_seconds` | Time per output token, measured at the gateway. Streaming requests only. | | `modelplane_frontend_request_duration_seconds` | What the caller waited, end to end | | `modelplane_frontend_requests_active` | Requests open at the gateway, per `ModelService` | | `modelplane_request_queue_seconds` | How long a request waited before the engine started | @@ -96,6 +96,13 @@ measured at the gateway. The engine's own series are what the engine spent. For example, if the frontend metric is slow and the engine isn't, you can troubleshoot routing, queueing, or networking issues instead of the model. +{{< hint "note" >}} +The gateway records time to first token (TTFT) and time per output token (TPOT) +only for streaming responses. A deployment that doesn't serve streaming requests +has no series for either, so use `modelplane_frontend_request_duration_seconds` +for it. +{{}} + Saturation gauges are per replica, so how you combine them decides what you see. Average across a deployment to plan capacity, and take the maximum to alert: three replicas at 0.3 and one at 0.99 average to something comfortable while the fourth evicts and recomputes. A @@ -263,7 +270,7 @@ credential is held on each of them. A collector transforms each measurement as it passes it on. It doesn't store past measurements, so it can't compute rates or quantiles. Your backend does that. A -fleet-wide p99: +fleet-wide p99 of time to first token, which requires streaming traffic: ```promql histogram_quantile(0.99, sum by (le) ( @@ -274,6 +281,18 @@ Modelplane has no dashboards of its own. What it exports is counters and histogr your backend derives the rates and quantiles at query time. To precompute them instead, export to Prometheus and write recording rules there. +The gateway's histograms have fixed upper buckets (AI Gateway v1.1.0): + +- Request duration: 81.92 seconds +- Time to first token: 60 seconds +- Time per output token: 2.5 seconds + +A quantile above the top bucket reads as that bound, no matter how long the request +took, so a p99 of `modelplane_frontend_request_duration_seconds` on long agentic +requests reads 81.92 seconds. For long requests, query the engine's +`modelplane_request_duration_seconds` instead. vLLM's buckets reach 7680 +seconds. + ## Engines Modelplane already knows vLLM's and SGLang's metric names and renames them for you. diff --git a/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt b/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt index 6ea9f82b9..7ace911a0 100644 --- a/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt +++ b/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt @@ -551,6 +551,7 @@ pipeline-parallel Qwen2.5-14B agent +agentic requirement PrefillDecode