Skip to content

feat(develop-docs): Add timestamp to Metrics API - #17791

Merged
philprime merged 6 commits into
masterfrom
philprime/metrics-api-timestamps
Jun 9, 2026
Merged

philprime merged 6 commits into
masterfrom
philprime/metrics-api-timestamps

Conversation

@philprime

@philprime philprime commented May 18, 2026 •

Copy link
Copy Markdown
Member

DESCRIBE YOUR PR

Problem

The current Metrics API (Sentry.metrics.count, .gauge, .distribution) always timestamps each metric at the moment the SDK method is called.
There is no way for the caller to supply a timestamp.

This is fine when application code emits metrics inline (e.g. incrementing a counter inside a request handler), but it breaks down for an important class of use cases: scraper-style collection, where an external process reads a batch of metrics from a source and then forwards them to Sentry.

Concrete example: Prometheus scraping

A Prometheus-compatible application exposes a /metrics endpoint that returns all current metric values in a single response:

# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 7.261032e+06

# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 43

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 11.32

# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1

A scraper fetches this snapshot at time T, then iterates through the metrics and calls Sentry.metrics.gauge(...), Sentry.metrics.count(...), etc. for each one.

Without a user-supplied timestamp, each SDK call captures Date.now() independently.
Even on a fast machine, iterating through dozens of metrics introduces sub-millisecond to low-millisecond drift between calls.
This means metrics that were observed at the exact same instant end up with slightly different timestamps.

This change is necessary if we want to extend our sentry-kubernetes integration to support Prometheus-style scraping of metrics in a Kubernetes cluster.

Why this matters: bucket boundaries

Sentry aggregates metrics into time buckets.
When two metrics that logically belong to the same observation are recorded with timestamps close to the bucket boundary, they land in different buckets.
This causes:

Metrics from the same scrape can be close to a bucket boundary, splitting a single observation into two partial data points and causing artificial jitter or gaps on dashboards and correlations.

The probability of hitting a bucket boundary increases with the number of metrics per scrape and the scrape frequency.

Solution

Add an optional timestamp parameter to the options object of all three metrics methods (count, gauge, distribution).

const scrapedAt = new Date();
const metrics = scrapePrometheusEndpoint(url);

for (const m of metrics) {
  Sentry.metrics.gauge(m.name, m.value, {
    timestamp: scrapedAt,  // all metrics share the same observation time
    attributes: m.labels,
  });
}

When provided, the SDK uses the caller's timestamp for the metric payload instead of Date.now().
When omitted, behavior is unchanged (current time).

When a timestamp is set, we need to ensure the ordering of metrics as described in Metric Ordering.

Type

SDKs should accept the language's native date/time type for SDK user ergonomics:

Language Native type
JavaScript Date
Python datetime
Java Instant
Swift Date
PHP DateTimeInterface

The SDK converts to seconds-since-epoch for the wire format, which already has a required timestamp field.
This change should be aligned with the current implementation of the SDKs.

Alternative Approaches

Batching

The wire format already supports batching (the items array), but the public API is per-metric (count, gauge,distribution).
A batch API would be a much larger surface area change than necessary to resolve the issue and raises a couple of concerns:

  • A generic SentrySDK.metrics.capture(items: MetricItem[]) would introduce an object-based public API. At this point the users only need to call the API with the values and options, so this is a conceptual change.
  • Allowing multiple values, e.g. SentrySDK.metrics.gaugeBatch(values: number[], options) would capture all of the values with the same options, which is not always the right choice.
  • Allowing multiple values, e.g. SentrySDK.metrics.gaugeBatch(items: { value: number, options }[]) is the same as if we introduce an external item type.

All of these changes bring in additional complexity compared to just allowing the same timestamp to be set.

Use beforeSendMetric to override the timestamp

This callback receives the metric object before sending, so it could theoretically mutate the timestamp.
But it runs at flush time (potentially seconds later), not at capture time, and it doesn't have access to the caller's intended timestamp without out-of-band state.

IS YOUR CHANGE URGENT?

Help us prioritize incoming PRs by letting us know when the change needs to go live.

  • Urgent deadline (GA date, etc.):
  • Other deadline:
  • None: Not urgent, can wait up to 1 week+

SLA

  • Teamwork makes the dream work, so please add a reviewer to your PRs.
  • Please give the docs team up to 1 week to review your PR unless you've added an urgent due date to it.
    Thanks in advance for your help!

PRE-MERGE CHECKLIST

Make sure you've checked the following before merging your changes:

  • Checked Vercel preview for correctness, including links
  • PR was reviewed and approved by any necessary SMEs (subject matter experts)
  • PR was reviewed and approved by a member of the Sentry docs team

@vercel

vercel Bot commented May 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
develop-docs Ready Ready Preview, Comment Jun 9, 2026 8:26am
sentry-docs Ready Ready Preview, Comment Jun 9, 2026 8:26am

Request Review

@github-actions github-actions Bot added the sdk-develop-docs PRs touching develop-docs/sdk label May 18, 2026
@philprime
philprime requested a review from giortzisg May 18, 2026 14:09

@giortzisg giortzisg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change @philprime. Though of adding a batch metric API, but this seems like the simplest way to solve the problem currently.

Comment thread develop-docs/sdk/telemetry/metrics.mdx Outdated
@philprime
philprime marked this pull request as ready for review May 20, 2026 07:47
@dingsdax
dingsdax self-requested a review May 21, 2026 08:25
@philprime philprime self-assigned this May 21, 2026

@sentry-junior sentry-junior Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good motivation — the Prometheus scraper use case is real and this is the right minimal API surface for it. A few things I think need attention before SDK teams pick this up:

Blocking

1. sentry.timestamp.sequence interaction is broken for non-monotonic timestamps

The existing v2.6.0 ordering spec resets the sequence counter "when the current metric's integer millisecond differs from the previous metric's integer millisecond." This assumes roughly monotonic capture-time timestamps. With user-supplied timestamps, users will interleave scraped metrics (all sharing timestamp T) with normal inline metrics (using Date.now()). This creates duplicate (timestamp_ms, sequence) pairs:

A: timestamp=T (scraped)   → seq=0
B: timestamp=T (scraped)   → seq=1  
C: timestamp=now (inline)  → seq=0 (reset)
D: timestamp=T (scraped)   → seq=0 (reset — collides with A)

Suggestion: replace the simple "previous differs" reset with per-integer-ms counters, bounded to buffered timestamps. Something like:

Sequence values MUST be unique for metrics captured by the same SDK instance with the same integer millisecond timestamp while those metrics may coexist in the SDK's buffer. SDKs SHOULD implement this with per-integer-ms counters and MAY discard counters once no buffered metric for that timestamp remains.

2. No timestamp validation requirements

The spec doesn't define behavior for invalid timestamps (NaN, Infinity, negative, far-future, naive datetimes). This will cause cross-SDK divergence. Specific risks:

  • JS users will pass Date.now() (milliseconds) as the numeric fallback — it's 1000x too large
  • Python datetime without tzinfo is ambiguous
  • No guidance on whether SDKs should throw, drop, or coerce

3. Relay/ingest acceptance window

Before SDKs implement this, we need confirmation on what timestamp range Relay accepts for trace_metric items. If scrape delay exceeds the allowed window, metrics silently vanish — exactly the failure mode this feature is trying to prevent.

Should-fix

4. Gauge semantics at same timestamp

Gauges need last-value semantics. If sequence is duplicated (per issue 1), which gauge value wins is nondeterministic. Worth an explicit note on expected behavior.

5. Span/replay association

A metric with a user-supplied timestamp from the past will get the current span's span_id. Should the spec state that association follows capture-call context, not the supplied timestamp?

6. Missing languages in type table

Go (time.Time), Rust (SystemTime/DateTime<Utc>), .NET (DateTimeOffset), Ruby (Time), Dart (DateTime) are missing.

7. Examples only show timestamp on distribution

Since the Prometheus use case is strongest for gauges and counters, consider showing timestamp on all three methods or adding a batch-scrape example that uses the same recordedAt across count/gauge/distribution.

Comment thread develop-docs/sdk/telemetry/metrics.mdx Outdated
Comment thread develop-docs/sdk/telemetry/metrics.mdx

@coolguyzone coolguyzone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for updating.

@philprime

Copy link
Copy Markdown
Member Author

From internal discussion with @logaretm, the following was proposed:


  1. Multiple metric items can use the same timestamp.
    Proposal: we keep using different sequence numbers to disambiguate
  2. Right now the sequence counter in the SDK is resetting when the last used timestamp changes, this breaks when user collects a metric with timestamp A, then one with timestamp B and then one with A again.
    Proposal: the sequence counter is an Int64, continuously incrementing from SDK start, never resets. Will be sent with every metric item
  3. When log sequence counter hits Int64 max it would overflow.
    Proposal: We do a reset to 0 and accept the blip, but probability of this happening is super very low (requires a ton of metrics in one SDK lifetime)
  4. There is a chance multiple SDK instances on multiple devices/replicas emit metrics at the same timestamp with the same sequence number. There is no shared sequence logic available.
    Proposal: At SDK init we store a timestamp, and this one is included as an attribute too. Chances of collision of timestamp + sequence number + exact same SDK start time is super low and can be considered out-of-scope.
    Alternative: maybe we have another value sent with envelopes that can be used, must be an monotonic growing number to allow ordering

Note:

  • Combining the timestamp in milliseconds with the sequence number as nanoseconds to synthesize a "virtual" timestamp in the backend, limits the amount of metrics per timestamp to 1,000. This could be established as a known limitation and the upper limit of metrics per timestamp in our docs. No further handling in the SDK.

Based on internal feedback, ordering is not as important for Metrics as they are for Logs. Therefore I am going to update the PR to incorporate proposals 1-3 but proposal 4 is out-of-scope due to YAGNI. If necessary it can be added at a later stage.

philprime added 2 commits June 8, 2026 09:27
…quence

The sequence counter no longer resets when the timestamp changes.
Instead it increments as an Int64 from SDK init, matching the
behavior agreed on for user-supplied timestamps. Also allow
RFC 3339 strings as an alternative timestamp wire format.
@philprime
philprime requested a review from logaretm June 8, 2026 07:52
@philprime

Copy link
Copy Markdown
Member Author

Waiting for final approval of @logaretm to confirm wording and details based on the proposed changes.

Comment thread develop-docs/sdk/telemetry/metrics.mdx Outdated
Comment thread develop-docs/sdk/telemetry/metrics.mdx Outdated
@philprime
philprime enabled auto-merge (squash) June 9, 2026 08:16
Comment on lines +208 to +215
Multiple metrics can share the same timestamp in certain cases. This happens when the runtime freezes timer APIs during synchronous execution (notably Cloudflare Workers freeze `Date.now()` and `performance.now()`), or when the caller supplies the same [user-provided timestamp](#metrics-module) for a batch of metrics. SDKs **MUST** attach a `sentry.timestamp.sequence` integer attribute to every metric to allow deterministic ordering within a single SDK instance.

When sent, the sequence integer **MUST**:
- Be a 64-bit signed integer (`Int64`).
- Start at `0` when the SDK initializes.
- Increment by `1` for each metric that is captured.
- Reset to `0` when:
- The SDK is re-initialized.
- The current metric's integer millisecond differs from the previous metric's integer millisecond (i.e., `floor(timestamp_seconds * 1000)` changes).
- Never reset during the SDK's lifetime. The counter **MUST** increment continuously across all metrics regardless of their timestamps.
- Reset to `0` only when the SDK is re-initialized, or when the counter reaches the maximum `Int64` value (`2^63 - 1`). Overflow to `0` is acceptable because the probability of reaching this limit within a single SDK lifetime is negligible.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The specification for sentry.timestamp.sequence in metrics.mdx now conflicts with logs.mdx regarding reset behavior and whether its inclusion is mandatory.
Severity: MEDIUM

Suggested Fix

Unify the specification for sentry.timestamp.sequence across all telemetry types. Decide on a single, consistent rule for both its attachment requirement (e.g., MUST vs MAY) and its reset behavior, and apply this rule to both metrics.mdx and logs.mdx.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: develop-docs/sdk/telemetry/metrics.mdx#L208-L215

Potential issue: The PR introduces conflicting specifications for the SDK-wide
`sentry.timestamp.sequence` attribute between metrics and logs. The metrics spec
(`metrics.mdx`) is updated to mandate that the sequence must be attached to every metric
and must never reset. However, the logs spec (`logs.mdx`) still states that the sequence
may be omitted for runtimes with reliable sub-millisecond timestamps and must reset to 0
when the timestamp's millisecond value changes. This creates ambiguity for SDKs, as a
single shared counter cannot comply with both specifications, leading to inconsistent
implementations.

@philprime
philprime merged commit a0f07a4 into master Jun 9, 2026
21 checks passed
@philprime
philprime deleted the philprime/metrics-api-timestamps branch June 9, 2026 08:26

This branch was successfully deployed

2 active deployments
Preview – sentry-docs — 30c0f617 Deployed Jun 9, 2026 by vercel[bot]
Preview – develop-docs — 30c0f617 Deployed Jun 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sdk-develop-docs PRs touching develop-docs/sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants