feat(hotblocks): expose RocksDB stall metrics - #90
Merged
Conversation
Export bounded pull-time RocksDB pressure metrics, add startup phase logs, and cover property availability and scrape overhead. Co-Authored-By: Codex <codex@openai.com>
dzhelezov
added a commit
to dzhelezov/data
that referenced
this pull request
Jul 15, 2026
Broader-instrumentation delta rebased on top of upstream/master (tx/block hash indexes subsquid#93, RocksDB stall metrics subsquid#90, conformance harness). Reconciled against upstream's stall-metric collector (canonical): - upstream RocksDbCollector (intrinsic per-CF/DB-wide property gauges) kept as-is; our duplicate collector + ROCKSDB_CF/DB_PROPERTIES constants and the redundant db.rs int-property/column_families accessors removed. - our unique delta preserved and wired into upstream's collect_rocksdb_metrics: cumulative RocksDB statistics tickers/histograms (block-cache, stalls, bytes, compaction) via encode_rocksdb_statistics + db.rocksdb_statistics(). - slow-response detector (SLOW_RESPONSES/SlowResponseConfig), long_poll TTFB split, per-dataset labels retained as the additive, upstream-PR-able remainder. cargo build + test green (metrics: parses_ticker/histogram + upstream write_duration/per-cf collector tests all pass).
dzhelezov
added a commit
to dzhelezov/data
that referenced
this pull request
Jul 15, 2026
Broader-instrumentation delta rebased on top of upstream/master (tx/block hash indexes subsquid#93, RocksDB stall metrics subsquid#90, conformance harness). Reconciled against upstream's stall-metric collector (canonical): - upstream RocksDbCollector (intrinsic per-CF/DB-wide property gauges) kept as-is; our duplicate collector, property constants, and duplicate storage accessors removed (db.rs is unchanged vs upstream). - our unique delta preserved and wired into upstream's collect_rocksdb_metrics: cumulative RocksDB statistics tickers/histograms (block-cache, stalls, bytes, compaction) via encode_rocksdb_statistics + upstream's db.get_statistics(). - slow-response detector (SLOW_RESPONSES/SlowResponseConfig), long_poll TTFB split, per-dataset labels, write_duration stage timing = additive remainder. nightly rustfmt-clean, clippy correctness-clean, cargo test green (metrics: parses_ticker/histogram + upstream write_duration/per-cf collector tests).
dzhelezov
added a commit
to dzhelezov/data
that referenced
this pull request
Jul 15, 2026
…on (#1) Broader-instrumentation delta rebased on top of upstream/master (tx/block hash indexes subsquid#93, RocksDB stall metrics subsquid#90, conformance harness). Reconciled against upstream's stall-metric collector (canonical): - upstream RocksDbCollector (intrinsic per-CF/DB-wide property gauges) kept as-is; our duplicate collector, property constants, and duplicate storage accessors removed (db.rs is unchanged vs upstream). - our unique delta preserved and wired into upstream's collect_rocksdb_metrics: cumulative RocksDB statistics tickers/histograms (block-cache, stalls, bytes, compaction) via encode_rocksdb_statistics + upstream's db.get_statistics(). - slow-response detector (SLOW_RESPONSES/SlowResponseConfig), long_poll TTFB split, per-dataset labels, write_duration stage timing = additive remainder. nightly rustfmt-clean, clippy correctness-clean, cargo test green (metrics: parses_ticker/histogram + upstream write_duration/per-cf collector tests).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
When all datasets stop advancing together, these signals distinguish a RocksDB hard or soft write stall from blocking-pool or upstream pipeline starvation. Per-column-family backlog metrics make the responsible LSM area visible without enabling expensive RocksDB statistics.
Performance
The collector reads fixed-cardinality in-memory integer properties only during Prometheus scrapes. Expensive properties that iterate SST files or versions are excluded.
Release benchmark under continuous scraping, much harsher than a production scrape interval:
Verification