Repository navigation
feat(develop-docs): Add timestamp to Metrics API - #17791
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
giortzisg
left a comment
There was a problem hiding this comment.
Nice change @philprime. Though of adding a batch metric API, but this seems like the simplest way to solve the problem currently.
There was a problem hiding this comment.
Good motivation — the Prometheus scraper use case is real and this is the right minimal API surface for it. A few things I think need attention before SDK teams pick this up:
Blocking
1. sentry.timestamp.sequence interaction is broken for non-monotonic timestamps
The existing v2.6.0 ordering spec resets the sequence counter "when the current metric's integer millisecond differs from the previous metric's integer millisecond." This assumes roughly monotonic capture-time timestamps. With user-supplied timestamps, users will interleave scraped metrics (all sharing timestamp T) with normal inline metrics (using Date.now()). This creates duplicate (timestamp_ms, sequence) pairs:
A: timestamp=T (scraped) → seq=0
B: timestamp=T (scraped) → seq=1
C: timestamp=now (inline) → seq=0 (reset)
D: timestamp=T (scraped) → seq=0 (reset — collides with A)
Suggestion: replace the simple "previous differs" reset with per-integer-ms counters, bounded to buffered timestamps. Something like:
Sequence values MUST be unique for metrics captured by the same SDK instance with the same integer millisecond timestamp while those metrics may coexist in the SDK's buffer. SDKs SHOULD implement this with per-integer-ms counters and MAY discard counters once no buffered metric for that timestamp remains.
2. No timestamp validation requirements
The spec doesn't define behavior for invalid timestamps (NaN, Infinity, negative, far-future, naive datetimes). This will cause cross-SDK divergence. Specific risks:
- JS users will pass
Date.now()(milliseconds) as the numeric fallback — it's 1000x too large - Python
datetimewithouttzinfois ambiguous - No guidance on whether SDKs should throw, drop, or coerce
3. Relay/ingest acceptance window
Before SDKs implement this, we need confirmation on what timestamp range Relay accepts for trace_metric items. If scrape delay exceeds the allowed window, metrics silently vanish — exactly the failure mode this feature is trying to prevent.
Should-fix
4. Gauge semantics at same timestamp
Gauges need last-value semantics. If sequence is duplicated (per issue 1), which gauge value wins is nondeterministic. Worth an explicit note on expected behavior.
5. Span/replay association
A metric with a user-supplied timestamp from the past will get the current span's span_id. Should the spec state that association follows capture-call context, not the supplied timestamp?
6. Missing languages in type table
Go (time.Time), Rust (SystemTime/DateTime<Utc>), .NET (DateTimeOffset), Ruby (Time), Dart (DateTime) are missing.
7. Examples only show timestamp on distribution
Since the Prometheus use case is strongest for gauges and counters, consider showing timestamp on all three methods or adding a batch-scrape example that uses the same recordedAt across count/gauge/distribution.
coolguyzone
left a comment
There was a problem hiding this comment.
Looks good, thanks for updating.
|
From internal discussion with @logaretm, the following was proposed:
Note:
Based on internal feedback, ordering is not as important for Metrics as they are for Logs. Therefore I am going to update the PR to incorporate proposals 1-3 but proposal 4 is out-of-scope due to YAGNI. If necessary it can be added at a later stage. |
…quence The sequence counter no longer resets when the timestamp changes. Instead it increments as an Int64 from SDK init, matching the behavior agreed on for user-supplied timestamps. Also allow RFC 3339 strings as an alternative timestamp wire format.
|
Waiting for final approval of @logaretm to confirm wording and details based on the proposed changes. |
| Multiple metrics can share the same timestamp in certain cases. This happens when the runtime freezes timer APIs during synchronous execution (notably Cloudflare Workers freeze `Date.now()` and `performance.now()`), or when the caller supplies the same [user-provided timestamp](#metrics-module) for a batch of metrics. SDKs **MUST** attach a `sentry.timestamp.sequence` integer attribute to every metric to allow deterministic ordering within a single SDK instance. | ||
|
|
||
| When sent, the sequence integer **MUST**: | ||
| - Be a 64-bit signed integer (`Int64`). | ||
| - Start at `0` when the SDK initializes. | ||
| - Increment by `1` for each metric that is captured. | ||
| - Reset to `0` when: | ||
| - The SDK is re-initialized. | ||
| - The current metric's integer millisecond differs from the previous metric's integer millisecond (i.e., `floor(timestamp_seconds * 1000)` changes). | ||
| - Never reset during the SDK's lifetime. The counter **MUST** increment continuously across all metrics regardless of their timestamps. | ||
| - Reset to `0` only when the SDK is re-initialized, or when the counter reaches the maximum `Int64` value (`2^63 - 1`). Overflow to `0` is acceptable because the probability of reaching this limit within a single SDK lifetime is negligible. |
There was a problem hiding this comment.
Bug: The specification for sentry.timestamp.sequence in metrics.mdx now conflicts with logs.mdx regarding reset behavior and whether its inclusion is mandatory.
Severity: MEDIUM
Suggested Fix
Unify the specification for sentry.timestamp.sequence across all telemetry types. Decide on a single, consistent rule for both its attachment requirement (e.g., MUST vs MAY) and its reset behavior, and apply this rule to both metrics.mdx and logs.mdx.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: develop-docs/sdk/telemetry/metrics.mdx#L208-L215
Potential issue: The PR introduces conflicting specifications for the SDK-wide
`sentry.timestamp.sequence` attribute between metrics and logs. The metrics spec
(`metrics.mdx`) is updated to mandate that the sequence must be attached to every metric
and must never reset. However, the logs spec (`logs.mdx`) still states that the sequence
may be omitted for runtimes with reliable sub-millisecond timestamps and must reset to 0
when the timestamp's millisecond value changes. This creates ambiguity for SDKs, as a
single shared counter cannot comply with both specifications, leading to inconsistent
implementations.
DESCRIBE YOUR PR
Problem
The current Metrics API (
Sentry.metrics.count,.gauge,.distribution) always timestamps each metric at the moment the SDK method is called.There is no way for the caller to supply a timestamp.
This is fine when application code emits metrics inline (e.g. incrementing a counter inside a request handler), but it breaks down for an important class of use cases: scraper-style collection, where an external process reads a batch of metrics from a source and then forwards them to Sentry.
Concrete example: Prometheus scraping
A Prometheus-compatible application exposes a
/metricsendpoint that returns all current metric values in a single response:A scraper fetches this snapshot at time T, then iterates through the metrics and calls
Sentry.metrics.gauge(...),Sentry.metrics.count(...), etc. for each one.Without a user-supplied timestamp, each SDK call captures
Date.now()independently.Even on a fast machine, iterating through dozens of metrics introduces sub-millisecond to low-millisecond drift between calls.
This means metrics that were observed at the exact same instant end up with slightly different timestamps.
This change is necessary if we want to extend our sentry-kubernetes integration to support Prometheus-style scraping of metrics in a Kubernetes cluster.
Why this matters: bucket boundaries
Sentry aggregates metrics into time buckets.
When two metrics that logically belong to the same observation are recorded with timestamps close to the bucket boundary, they land in different buckets.
This causes:
Metrics from the same scrape can be close to a bucket boundary, splitting a single observation into two partial data points and causing artificial jitter or gaps on dashboards and correlations.
The probability of hitting a bucket boundary increases with the number of metrics per scrape and the scrape frequency.
Solution
Add an optional
timestampparameter to theoptionsobject of all three metrics methods (count,gauge,distribution).When provided, the SDK uses the caller's timestamp for the metric payload instead of
Date.now().When omitted, behavior is unchanged (current time).
When a timestamp is set, we need to ensure the ordering of metrics as described in Metric Ordering.
Type
SDKs should accept the language's native date/time type for SDK user ergonomics:
DatedatetimeInstantDateDateTimeInterfaceThe SDK converts to seconds-since-epoch for the wire format, which already has a required
timestampfield.This change should be aligned with the current implementation of the SDKs.
Alternative Approaches
Batching
The wire format already supports batching (the
itemsarray), but the public API is per-metric (count,gauge,distribution).A batch API would be a much larger surface area change than necessary to resolve the issue and raises a couple of concerns:
SentrySDK.metrics.capture(items: MetricItem[])would introduce an object-based public API. At this point the users only need to call the API with the values and options, so this is a conceptual change.SentrySDK.metrics.gaugeBatch(values: number[], options)would capture all of the values with the same options, which is not always the right choice.SentrySDK.metrics.gaugeBatch(items: { value: number, options }[])is the same as if we introduce an external item type.All of these changes bring in additional complexity compared to just allowing the same timestamp to be set.
Use
beforeSendMetricto override the timestampThis callback receives the metric object before sending, so it could theoretically mutate the timestamp.
But it runs at flush time (potentially seconds later), not at capture time, and it doesn't have access to the caller's intended timestamp without out-of-band state.
IS YOUR CHANGE URGENT?
Help us prioritize incoming PRs by letting us know when the change needs to go live.
SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST
Make sure you've checked the following before merging your changes: