Skip to content

add fs ephemeral and txoutset build logging - #1457

Merged
AloeareV merged 3 commits into
rc/0.8.0from
fix/release_0_8_0/slow_accumulator_build_and_fs_switchover
Aug 20, 2026
Merged

add fs ephemeral and txoutset build logging#1457
AloeareV merged 3 commits into
rc/0.8.0from
fix/release_0_8_0/slow_accumulator_build_and_fs_switchover

Conversation

@idky137

@idky137 idky137 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Finalised-state observability

Additive observability only — no control flow, status semantics, or routing behaviour changes.

Why

Two blind spots hit while syncing a mainnet node:

  1. The accumulator rebuild is silent. rebuild_tx_out_set_accumulator logs two lines up front and nothing again until it commits. In between it runs a full spent count pass and one full-chain block scan per shard — ~40 minutes on a mainnet DB — with no output. "Working" and "hung" look identical.

  2. chain_state: Ready doesn't mean the persistent DB is serving. During a long sync the router installs an ephemeral passthrough; it tracks the validator and reports Ready exactly like a synced on-disk index. Nothing logged the install or release, and nothing exposed routing state. Containerised tests waiting on Ready have been silently asserting against the passthrough.

What

Rebuild progress — spent-entry count pass, per-shard start/spent-set-size/completion, and intra-shard height progress. Throttled to one line per 10s (reusing the existing write_core idiom, now a shared PROGRESS_LOG_INTERVAL) so output stays bounded regardless of shard count — a memory-constrained container can reach 256 shards. Also covers the startup spent integrity check and the incremental update path.

A shard bisect logs at WARN, naming accumulator_rebuild_memory_size — it means the budget under-provisioned and each split adds another full-chain pass.

Routing mode — new FinalisedStateMode (EphemeralConfigured / EphemeralRouted / Persistent), reported on every transition plus a one-shot "finalised state online" line:

INFO router: finalised state switched to the ephemeral passthrough: reads are served from the
backing validator, NOT the persistent database ... mode=ephemeral(syncing)
INFO router: finalised state switched back to the persistent database mode=persistent
INFO router: finalised state online: reads are now served by the persistent database

The one-shot latch fires from both the ephemeral-release edge and the status poll — a restart against an already-current DB never installs a passthrough, so the release edge alone would never fire.

ephemeral_finalised_state = true now warns at startup; that branch previously returned completely silently.

Status line gains fs_mode, so CI can gate on fs_mode: persistent instead of Ready.

Metrics (prometheus feature, off by default): zaino.db.finalised_ephemeral, zaino.db.accumulator_built_height, zaino.db.accumulator_rebuild_active.

Notes for review

  • fs_height was dropped from the status line: db_height() is async, log_status is a sync pub fn, and making it async is a breaking change. The height is available via the gauges.
  • One private-fn signature change (shard_ordinal/shard_total for logging), one pub accessor chain mirroring the existing status() chain, one AtomicBool latch on Router.
  • Drive-by fix: zaino.chain.tip_height was emitted twice per sync iteration, once via a hard-coded string literal and once via CHAIN_TIP_HEIGHT. Removed the literal. Happy to split this out.
  • makers lint fails on this branch before this PR — --all-features enables transparent_address_history_experimental, which doesn't compile (write_core.rs:404, :1541). Verified against a clean tree. Not addressed here.

Verification

cargo fmt --check, cargo clippy --workspace --all-targets and --features prometheus all clean (no new warnings); makers lint-boundary-conversions clean; 586 tests pass. Two new tests cover the configured-ephemeral mode and the install→release transition.

14 files, +584/−19.

Out of scope and worth separate issues: checkpointing the rebuild so restarts resume, relaxing NO_READAHEAD for the rebuild's sequential scans, and whether Router::status() should report Syncing while ephemeral-backed.

@idky137
idky137 force-pushed the fix/release_0_8_0/slow_accumulator_build_and_fs_switchover branch from 344c8d3 to f751366 Compare August 18, 2026 15:41
@idky137
idky137 requested a review from nachog00 August 18, 2026 15:41
@zancas
zancas requested review from AloeareV and removed request for nachog00 August 19, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants