Skip to content

feat(hotblocks): expose RocksDB stall metrics - #90

Merged
mo4islona merged 1 commit into
masterfrom
feat/hotblocks-rocksdb-observability
Jul 14, 2026
Merged

feat(hotblocks): expose RocksDB stall metrics#90
mo4islona merged 1 commit into
masterfrom
feat/hotblocks-rocksdb-observability

Conversation

@mo4islona

Copy link
Copy Markdown
Contributor

Summary

  • expose bounded RocksDB write-stall, compaction, flush, memtable, snapshot, and background-error metrics
  • add low-volume startup phase logs around DB open, dataset controller initialization, and disk recovery
  • verify selected properties remain available with RocksDB statistics disabled
  • include an ignored reproducible benchmark for scrape overhead on reads and writes

Why

When all datasets stop advancing together, these signals distinguish a RocksDB hard or soft write stall from blocking-pool or upstream pipeline starvation. Per-column-family backlog metrics make the responsible LSM area visible without enabling expensive RocksDB statistics.

Performance

The collector reads fixed-cardinality in-memory integer properties only during Prometheus scrapes. Expensive properties that iterate SST files or versions are excluded.

Release benchmark under continuous scraping, much harsher than a production scrape interval:

  • full scrape p50: 11.166 us
  • write p50: 9.125 us baseline, 8.667 us with scraping
  • read p50: 1.166 us baseline, 1.250 us with scraping

Verification

  • cargo test -p sqd-hotblocks
  • cargo test -p sqd-storage
  • cargo +nightly fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D clippy::correctness
  • git diff --check

Export bounded pull-time RocksDB pressure metrics, add startup phase logs, and cover property availability and scrape overhead.

Co-Authored-By: Codex <codex@openai.com>
@mo4islona
mo4islona merged commit 8352711 into master Jul 14, 2026
1 of 3 checks passed
dzhelezov added a commit to dzhelezov/data that referenced this pull request Jul 15, 2026
Broader-instrumentation delta rebased on top of upstream/master (tx/block
hash indexes subsquid#93, RocksDB stall metrics subsquid#90, conformance harness).

Reconciled against upstream's stall-metric collector (canonical):
- upstream RocksDbCollector (intrinsic per-CF/DB-wide property gauges) kept
  as-is; our duplicate collector + ROCKSDB_CF/DB_PROPERTIES constants and the
  redundant db.rs int-property/column_families accessors removed.
- our unique delta preserved and wired into upstream's collect_rocksdb_metrics:
  cumulative RocksDB statistics tickers/histograms (block-cache, stalls, bytes,
  compaction) via encode_rocksdb_statistics + db.rocksdb_statistics().
- slow-response detector (SLOW_RESPONSES/SlowResponseConfig), long_poll TTFB
  split, per-dataset labels retained as the additive, upstream-PR-able remainder.

cargo build + test green (metrics: parses_ticker/histogram + upstream
write_duration/per-cf collector tests all pass).
dzhelezov added a commit to dzhelezov/data that referenced this pull request Jul 15, 2026
Broader-instrumentation delta rebased on top of upstream/master (tx/block
hash indexes subsquid#93, RocksDB stall metrics subsquid#90, conformance harness).

Reconciled against upstream's stall-metric collector (canonical):
- upstream RocksDbCollector (intrinsic per-CF/DB-wide property gauges) kept
  as-is; our duplicate collector, property constants, and duplicate storage
  accessors removed (db.rs is unchanged vs upstream).
- our unique delta preserved and wired into upstream's collect_rocksdb_metrics:
  cumulative RocksDB statistics tickers/histograms (block-cache, stalls, bytes,
  compaction) via encode_rocksdb_statistics + upstream's db.get_statistics().
- slow-response detector (SLOW_RESPONSES/SlowResponseConfig), long_poll TTFB
  split, per-dataset labels, write_duration stage timing = additive remainder.

nightly rustfmt-clean, clippy correctness-clean, cargo test green (metrics:
parses_ticker/histogram + upstream write_duration/per-cf collector tests).
dzhelezov added a commit to dzhelezov/data that referenced this pull request Jul 15, 2026
…on (#1)

Broader-instrumentation delta rebased on top of upstream/master (tx/block
hash indexes subsquid#93, RocksDB stall metrics subsquid#90, conformance harness).

Reconciled against upstream's stall-metric collector (canonical):
- upstream RocksDbCollector (intrinsic per-CF/DB-wide property gauges) kept
  as-is; our duplicate collector, property constants, and duplicate storage
  accessors removed (db.rs is unchanged vs upstream).
- our unique delta preserved and wired into upstream's collect_rocksdb_metrics:
  cumulative RocksDB statistics tickers/histograms (block-cache, stalls, bytes,
  compaction) via encode_rocksdb_statistics + upstream's db.get_statistics().
- slow-response detector (SLOW_RESPONSES/SlowResponseConfig), long_poll TTFB
  split, per-dataset labels, write_duration stage timing = additive remainder.

nightly rustfmt-clean, clippy correctness-clean, cargo test green (metrics:
parses_ticker/histogram + upstream write_duration/per-cf collector tests).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant