feat: R2-first witnesses on the validator, and a chain-anchored frontier band in both binaries

[flyq]

Summary

Turns on the R2 witness route for the validator, and makes configuring an R2 target the only switch that turns it on. With a target configured every witness fetch tries the bucket before the --witness-endpoint RPC chain, and any R2 failure hands that one block to the chain instead of stalling it. With no --r2-* flag set, witnesses come from RPC alone, exactly as today. Bulk history then streams from R2 at object-storage parallelism over HTTP/2 while the RPC gateway only ever sees the blocks R2 could not serve.

Root cause

The custom-domain / HTTP/2 / --r2-connections transport already reached the validator in #187, but only behind --witness-source r2, which had no fallback: a pre-upload frontier miss, a throttle or a corrupt object was re-enqueued until R2 served it. The trace server pairs the same transport with the RPC chain (#182), so R2 speeds up the common case there without becoming a single point of failure. The validator had no equivalent, so turning R2 on for speed meant turning the RPC safety net off.

Fix

• The R2 flags are the switch; --witness-source is gone. Every rule the mode flag gated was already a rule about the flags themselves, so build_r2_transport (bin/stateless-validator/src/app.rs:431) builds a transport when validate_r2_flags selects a target and nothing when it does not. --witness-endpoint is now always required (bin/stateless-validator/src/app.rs:415), being either the only witness path or the fallback behind R2.
• Any R2 failure falls back (bin/stateless-validator/src/chain_sync.rs:62). The R2 client records and logs the failure, so the witness fetch stays as infallible to the pipeline as the RPC-only path always was.
• The fast path is bounded per block. A failure surfaces at once on a 3-attempt budget (bin/stateless-validator/src/r2_witness.rs:50), and the whole stage — concurrency-permit wait plus every attempt — shares one --rpc-per-attempt-timeout-ms (bin/stateless-validator/src/r2_witness.rs:71, wired at bin/stateless-validator/src/app.rs:325). The attempt count alone did not bound it: an endpoint that accepts connections and then stalls spent a full per-attempt timeout on every try, about a minute per block at the default, and blocks queued behind the cap waited through several such holders. The decode stays outside that budget, since abandoning it would only re-fetch and re-decode the same witness over RPC.
• Removed with the R2-only mode: the R2FailurePolicy enum, the deterministic-failure and exhausted-retry pauses that paced the pipeline's blind re-enqueue, and the pre-split concurrency-cap guard whose whole premise was the missing fallback.
• A missing is split by band, through one shared rule anchored on the chain. R2Band and r2_band live in stateless-common (crates/stateless-common/src/r2_witness.rs:49, classified by crates/stateless-common/src/r2_witness.rs:66) rather than once per binary, and the validator asks them whether a failure is a routine near-tip miss (bin/stateless-validator/src/r2_witness.rs:92). Those misses are counted on their own r2_witness_frontier_misses_total (bin/stateless-validator/src/metrics.rs:111), so r2_witness_errors_total stays an error rate and its kind="missing" means a hole in objects that must exist — which matters because a tip-following validator produces essentially nothing but frontier misses.
• Two diagnostics the validator gains by validating its R2 flags on every startup: an orphaned tuning flag, and a blank env value. Accepted silently, either would run the RPC-only path while the operator believed R2 was on.
• Carrying only the pre-split --witness-max-concurrent-requests warns (bin/stateless-validator/src/app.rs:459). It sizes the RPC witness path alone, so that configuration leaves R2 uncapped — and the fetcher cannot flag that itself: with no cap there is no per-connection share to compare against the edge's stream limit, so the queueing happens inside the HTTP/2 connection where it is invisible and still spends the per-attempt budget.

Shared with the trace server

• validate_r2_flags returns the values its rules proved rather than only naming the target (crates/stateless-common/src/r2_args.rs:106), and each arm binds them in the same let that proves them, so nothing is asserted twice. The in-flight cap rides on that verdict too: the rules check it against the connection count, so the transport has to be built with the cap they checked, and they name an orphaned cap themselves rather than each binary listing it as a tuning flag.
• R2WitnessTransport::from_config (crates/stateless-common/src/r2_witness.rs:263) is the single place either binary turns that verdict into a transport, publishing what it built through the new R2Metrics trait (crates/stateless-common/src/r2_witness.rs:178), which mirrors how RpcMetrics already works for the RPC client.
• The frontier band's arithmetic is shared too, not just its width. The two binaries had already drifted on it: at exactly tip - 32 the trace server said "hole" and the validator said "frontier", each pinned by a test whose comment contradicted the other's. r2_band is now one rule with the trace server's shipped convention, so the block that flips is the validator's, whose label this PR introduces and has never shipped.
• And they now anchor it on the same thing: the chain head. Both serve witnesses out of the same bucket, filled by the same uploader, so the question — has the uploader had time to reach this block — is about the chain, not about how far a given reader has ingested. The trace server bands against the higher of DataProvider::tip_hint (bin/debug-trace-server/src/data_provider.rs:970) and its local DB tip (bin/debug-trace-server/src/data_provider.rs:1314), so no new upstream call is needed. Neither is the chain tip alone — tip_hint is raised only by by-number and tag resolutions, the DB tip only by what sync has ingested — and each leads the other in a different mode; both are bounded by the real chain, so the maximum is the better estimate and still bands a miss on the safe side.
• And the rule that reads the band is shared now too, which the band consolidation had stopped one line short of: R2WitnessError::is_frontier_miss (crates/stateless-common/src/r2_witness.rs:133) replaces the conjunction each binary used to spell for itself. Same shape that had already drifted once on the edge — the trace server's spelling is covered by a shared test for the first time, and the validator's duplicate test is gone rather than trimmed.
• That retired the third band, and with it a real blind spot. AboveTip existed only because the DB tip could lag the chain without bound, and it was suppressing holes: through a catch-up, every genuine gap between the DB tip and the chain head went to kind="missing_above_tip" instead of the alarm. A block 500 seconds below the head is overdue whatever this process has ingested. R2Band is two variants now and missing_above_tip is gone.
• Together that retires eight expect()s restating checks made elsewhere and two copies of a rule about which flags each target requires. The S3 secret on the verdict is a RedactedSecret, so its Debug is derived rather than hand-written. What stays per binary is what genuinely differs: the trace server's startup wording, its per-band budget share — a speculative eighth of the stage in the frontier against half past it, where R2 is the primary source — and how each keeps frontier misses off the error counter, the trace server on the per-source series it already labels by band and the validator on a counter of its own.
• The --r2-* flag declarations stay per binary. A macro over the env prefix does work (clap accepts env = concat!(…)), but the doc comments are the --help text and several genuinely differ; flattening them to save the declarations was not worth it.

Testing

• cargo test workspace-wide: 486 passed, 0 failed. cargo fmt --check, cargo clippy --workspace --all-targets --all-features, cargo sort --check, and cargo test -p stateless-core --no-default-features --lib --no-run are all clean.
• Startup wiring (bin/stateless-validator/src/app.rs:600): a configured target is the only switch, on both target arms and with none; the R2 cap rather than the RPC one reaches the transport; the two new diagnostics; and that the witness chain is required with R2 configured.
• Fetch behaviour, against a mock RPC plus a scripted R2 (bin/stateless-validator/tests/integration.rs:507): a bucket hit never touches the RPC witness path, and a miss falls back with one GET and one RPC call (bin/stateless-validator/tests/integration.rs:523). A corrupt object takes the identical branch — fetch_witness never inspects the error kind — so it is pinned where the difference lives, on the decode path in r2_witness.rs.
• The stage budget (bin/stateless-validator/src/r2_witness.rs:307) drives a held connection with a per-attempt timeout an order of magnitude above the budget. Reverting the deadline to None makes it fail after the full three per-attempt timeouts, 15.0s against 0.2s at the test's scale.
• The band's edge and the conjunction that reads it are pinned once each, beside the classifier (crates/stateless-common/src/r2_witness.rs:358, crates/stateless-common/src/r2_witness.rs:379): an absent object near the polled head is a frontier miss, the deep edge and below is a hole, and no other kind splits. Both binaries are covered by those two, where each used to carry its own.
• The anchor itself is pinned through the budget share, which the band also selects (bin/debug-trace-server/src/data_provider.rs:2565): with the DB at 4000 and the head known to be 5000, block 4500 takes the historical half of the stage. Reverting the anchor to db_tip cuts it at the speculative eighth and fails the test at 256ms.
• And the other direction, where only the DB knows the chain (bin/debug-trace-server/src/data_provider.rs:2606): with the tip hint never raised — what a server asked only for block hashes and transactions sees for its whole life — a block 1000 below the DB tip still takes the historical half. Reverting to the hint alone fails it while the test above still passes, so the two cover opposite directions.
• The verdict carries its values, the cap included, and redacts the S3 secret in Debug (crates/stateless-common/src/r2_args.rs:487); an orphaned cap is named by the rules alone (crates/stateless-common/src/r2_args.rs:597); the pre-split cap spelling alone builds uncapped rather than being refused (bin/stateless-validator/src/app.rs:572).

Notes

• Breaking for anyone who sets --witness-source. Nothing in mainnet_env does, and the flag is gone rather than kept as a no-op, so a stale command line fails loudly. A stale STATELESS_VALIDATOR_WITNESS_SOURCE env line is simply ignored.
• A blank --r2-* env line now fails startup on an RPC-only validator, where it used to be inert. That is the loud direction, and it is what makes an inferred switch safe: "no R2 configured" can no longer be confused with "R2 configured wrong". The same applies to an inherited STATELESS_VALIDATOR_R2_MAX_CONCURRENT_REQUESTS on a role that configures no target: the orphan rule now names it, where the flag used to be read by nobody.
• Two limits of that claim, both pre-existing and previously undocumented: "blank value" covers the --r2-* flags that travel as text, while the two numeric tuning flags are parsed by clap, so a blank one aborts before the rules run with clap's unnamed "invalid value for one of the arguments"; and --witness-max-concurrent-requests set alone alongside an R2 target warns rather than failing, since R2 is left uncapped but the RPC path it does size is genuinely in use.
• The RPC fallback is a second path to the same bytes, not a second copy of them. The witness gateway reads this same bucket, so what the fallback covers is our client path failing (the CDN edge, an Access token, HTTP/2, credentials, this fetcher). A true bucket hole does not resolve by falling back; it moves the retry onto the shared gateway, which is what kind="missing" is there to catch.
• Metric semantics change on both binaries. debug_trace_r2_witness_errors_total{kind="missing_above_tip"} is retired; those misses now either stay frontier (above the head) or join kind="missing" (below it, where they were always holes). On the validator, a miss within 32 blocks of the polled head is no longer an error at all. It lands on stateless_validator_r2_witness_frontier_misses_total, and ..._r2_witness_errors_total stays an error rate — which it would not, with a tip-following validator producing essentially nothing but frontier misses.
• One statement this PR had made false is corrected. The trace server's --data-dir gate — its comment and its user-facing bail message (bin/debug-trace-server/src/main.rs:703) — still said the R2 route anchors block age on the local DB tip. Once the anchor moved to the chain head that reason no longer held: tip_hint is fed by tag lookups and canonical resolutions and needs no DB. The gate itself is unchanged; what it cites now is the old-block budget clamp and generator routing, which do read the DB tip. The same retired anchor is corrected in three places in README.md, one of which contradicted the line two below it. Worth a follow-up decision: with the band off the DB tip, the gate's remaining justification is weaker than it was, since both DB-tip consumers already fall back to their conservative branch.
• The window is 32 blocks, which on MegaETH is 32 seconds, since the chain produces one block per second. That is the number to reason about when retuning it: an object still absent that long after its block existed means the witness generation pipeline is behind, and the same figure is the alarm's detection latency.
• Which counter is meaningful depends on how far behind the run is. A tip-following validator fetches at head - tip_buffer, and every deployed buffer is far inside a 32-block window, so all of its misses are frontier misses — routine and numerous in practice — and kind="missing" stays at zero by construction; the frontier rate is the signal there. kind="missing" earns its name during catch-up and fixed --end-block backfills, where blocks sit far below the head.
• A hole that first appears near the tip is consequently not detected here: the block is fetched once, falls back, and is never probed again. Left that way on purpose. The fallback already served the block, so this process has nothing to act on, and re-probing purely to keep a counter honest belongs with whatever watches the uploader rather than in a persisted recheck queue inside the validator.
• R2 retries are spaced by the existing --rpc-*-backoff-ms ramp, up to about two seconds per affected block at the defaults, always inside the stage budget.
• R2 GET retries log at debug in the shared fetcher (crates/stateless-r2/src/fetch.rs:885), in both binaries. Each failed fetch still warns once with its final error and every retry is counted, so a throttle brownout no longer turns each block into three warnings; this changes the trace server's logs as well.
• Follow-ups, deliberately left out of this PR:
  • A shorter R2-specific retry pacing, which hands blocks to RPC sooner during a throttle brownout.
  • A breaker that skips R2 for a while after N consecutive stage timeouts. Pacing trims the retries but not an accept-then-stall, which holds every in-flight block for the full budget and floors throughput near fetcher_max_in_flight / --rpc-per-attempt-timeout-ms — fine for tip-following, an order of magnitude slower for catch-up.
  • Skipping the probe for in-band blocks, if r2_witness_frontier_misses_total sits near 100% of the validated-block rate after rollout. That gives back the one edge round trip every tip block pays before its RPC witness call.
• R2WitnessError::is_retryable and R2ObjectFetcher::pacing() are removed; their only caller was the deleted pause. mega-reth pins v2.0.18 and references neither.

🤖 Generated with Claude Code

[mega-maxwell]

Claude review status

Living comment — rewritten in place. The review workflow keeps this single comment up to date instead of posting a new one each round, so it always describes the latest reviewed commit and the earlier text is intentionally gone. No reply is needed here; reply to a finding in its own review thread, and answer an open question in a reply on this PR. The next review round reconciles your answer.

🛠️ Review did not finish

Attempted dd0cbc4c..6e614975 · updated 2026-09-20T05:17:55+00:00

This round did not publish: MODEL_ACTION_FAILED in phase review_retry. Anything listed below is from the last round that did. Re-run the workflow or push a new commit to try again.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 973463125e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

stateless-validator/bin/stateless-validator/src/r2_witness.rs

Line 147 in b0d7f00

.get_block_object(number, hash, MAX_ATTEMPTS, None, metrics::on_r2_witness_retry)

Put a total deadline on the R2 fast path

When R2 accepts a connection but stalls, passing None leaves both the concurrency-permit wait and all three GET attempts without an aggregate deadline. With the default 20-second per-attempt timeout, one block waits roughly a minute before reaching RPC; with --r2-max-concurrent-requests, queued blocks can wait through multiple such holders, effectively stalling the validator during an R2 brownout despite the fallback. Give the R2 stage a short total deadline that also bounds permit waiting before handing the block to RPC.

AGENTS.md reference: AGENTS.md:L154-L154

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[flyq]

Confirmed and fixed in d732b51.

The stage now carries a total deadline covering the permit wait and every attempt together, and the wiring site passes one --rpc-per-attempt-timeout-ms for it: R2 sits in front of a path that retries forever, so it may never cost a block more wall clock than a single upstream hop. A healthy fetch is sub-second and every fast failure mode still fits the full retry count, so the budget only bites on exactly the case you describe.

The decode deliberately stays outside that deadline. Those bytes are already in hand and the work is our own CPU, so abandoning it would only re-fetch and re-decode the same witness over RPC.

a_stalling_endpoint_is_abandoned_on_the_stage_budget in bin/stateless-validator/src/r2_witness.rs drives a held connection with a per-attempt timeout an order of magnitude above the stage budget. Reverting the deadline to None makes it fail after the full three per-attempt timeouts instead of passing on the budget — 15.0s versus 0.2s at the test's scale, which is the ~60s versus 20s you predicted at the 20s default.

[RealiCZ]

Reviewed the full diff and verified the parts the design rests on:

• The stage budget (Codex P1) is implemented end to end: R2ObjectFetcher::get_block_object clamps the permit wait, each request's timeout, and the backoff sleep against the deadline, and stage = per-attempt = --rpc-per-attempt-timeout-ms, so a stall costs exactly one hop.
• "The validator only fetches at or below the head it last polled" holds: block_fetcher polls latest_block_number before spawning any fetch and spawns up to chain_latest - tip_buffer, and remote_head is that same value, so dropping the above-tip band is correct.
• R2Config's derived Debug is safe: RedactedSecret and CfAccessCredentials both redact.
• None of the removed public items (R2Target, R2WitnessError::is_retryable, R2ObjectFetcher::pacing, the R2WitnessClient::new signature, the ValidatorFetcher fields) are referenced by mega-reth (grep over a local checkout).
• Merges cleanly onto main and alongside #197.
• Ran locally: the r2_witness unit tests, the app::tests wiring tests, and the fallback / bucket-hit integration tests all pass; the two timing-bound tests (failures_surface_immediately_for_the_rpc_fallback, a_stalling_endpoint_is_abandoned_on_the_stage_budget) passed 10/10 repeated runs.

The inline notes are all non-blocking. Two more for the record:

• On Codex's STATELESS_VALIDATOR_WITNESS_SOURCE point: dropping it silently is defensible and I won't hold the PR on it, but it is the one stale-configuration case this PR does not name — orphaned tuning flags and blank values all fail by name. A std::env::var_os guard kept for one release would make the rule uniform; your call.
• stateless-common / stateless-r2 lose public items, so this is technically a breaking change on crates mega-reth pins; whether that warrants a minor bump over the usual patch is for the release process.

Approving.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1832b3454

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b26548fc0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Include decoding in the R2 stage deadline

When an R2 GET consumes most of stage_timeout, or a large/pathological payload is slow to decompress and deserialize, passing None here lets decoding continue beyond the advertised per-block budget and delays the RPC fallback indefinitely. Pass the existing deadline to decode_on_blocking_pool; it already skips an expired decode and returns DecodeTimeout, allowing fetch_witness to fall back promptly.

AGENTS.md reference: AGENTS.md:L159-L159

Useful? React with 👍 / 👎.