Add global checkpoint scheduler

[yzang2019]

Describe your changes and provide context

Two main changes in this PR:

1. Move existing SS checkpointer to ss/snapshot
2. Create a new global checkpoint scheduler and put that under controller

This PR introduce the interface of new global scheduler, which is not reachable from production yet, nothing implements it. A follow-up PR will wire it up with existing stores.

Testing performed to validate your change

[cursor]

PR Summary

Medium Risk
Snapshot/checkpoint call paths are reshuffled across packages (behavior should be equivalent), while the new cross-store scheduler is untested in production until wired; mistakes there could misalign checkpoint versions across stores.

Overview
Introduces a global CheckpointScheduler in sei-db/controller that keeps every registered store on the same checkpoint height: a background loop picks the next interval-aligned version above all stores’ committed versions, dispises one shared target at a time, and respects optional minimum wall-clock spacing between completed checkpoints. Stores are expected to implement the new CheckpointableStore interface (ScheduleCheckpoint, LatestVersion, CheckpointInProgress); nothing in production wires this yet—follow-up will connect real stores.

Refactor: Per-engine checkpoint orchestration (ScheduleCheckpoint, SetCheckpointVersion, FanIn, SupportsCheckpoint, ErrCheckpointCanceled) moves from controller into state_db/ss/snapshot as a Checkpointer interface and helpers. SS snapshot Manager config renames Scheduler → Checkpointer; cosmos, EVM, composite, and Pebble tests call the snapshot package instead of controller. Docs in db_engine/types are updated to describe controller vs snapshot responsibilities.

Adds broad unit tests for scheduler cadence, gating, and lifecycle (checkpoint_scheduler_test.go).

^{Reviewed by Cursor Bugbot for commit 15f04a7. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 83.87097% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.34%. Comparing base (7fe5807) to head (15f04a7).

Files with missing lines	Patch %	Lines
sei-db/state_db/ss/snapshot/checkpoint.go	72.50%	7 Missing and 4 partials ⚠️
sei-db/controller/checkpoint_scheduler.go	94.73%	3 Missing and 2 partials ⚠️
sei-db/state_db/ss/evm/store.go	42.85%	3 Missing and 1 partial ⚠️
sei-db/state_db/ss/snapshot/manager.go	42.85%	1 Missing and 3 partials ⚠️
sei-db/state_db/ss/composite/snapshot.go	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4026      +/-   ##
==========================================
- Coverage   61.24%   60.34%   -0.90%     
==========================================
  Files        2153     2068      -85     
  Lines      188379   178213   -10166     
==========================================
- Hits       115371   107542    -7829     
+ Misses      62276    60784    -1492     
+ Partials    10732     9887     -845     

Flag	Coverage Δ
sei-chain-pr	68.91% <94.73%> (?)
sei-db	69.80% <ø> (ø)
sei-db-state-db	?
sei-db-state-db-pr	67.42% <66.66%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/db_engine/types/types.go	50.00% <ø> (+20.00%)	⬆️
sei-db/state_db/ss/cosmos/store.go	82.43% <100.00%> (ø)
sei-db/state_db/ss/composite/snapshot.go	83.43% <50.00%> (ø)
sei-db/state_db/ss/evm/store.go	58.11% <42.85%> (ø)
sei-db/state_db/ss/snapshot/manager.go	56.80% <42.85%> (ø)
sei-db/controller/checkpoint_scheduler.go	94.89% <94.73%> (+19.89%)	⬆️
sei-db/state_db/ss/snapshot/checkpoint.go	72.50% <72.50%> (ø)

... and 158 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Aug 26, 2026, 8:42 PM

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 445e090. Configure here.}

[seidroid]

Clean relocation of the SS checkpoint helpers into ss/snapshot plus a new global CheckpointScheduler; the move is complete and consistent, and the scheduler is thoroughly unit-tested. The scheduler is not yet wired into production, and its store contract leaves checkpoint completion inferred rather than reported, which is worth pinning down before implementations land.

Findings: 0 blocking | 4 non-blocking | 2 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• [suggestion] The new global scheduler is not reachable from production: nothing implements CheckpointableStore (grep for CheckpointInProgress finds no implementation outside sei-db/controller), and NewCheckpointScheduler is called only from checkpoint_scheduler_test.go. SS still constructs the existing per-composite coordinator in sei-db/state_db/ss/composite/snapshot.go, so this PR adds the mechanism without enabling global checkpoint scheduling. Fine as a staged change — worth stating in the description that a follow-up wires it up, so it is not mistaken for a live feature.
• [suggestion] CheckpointableStore gives a store no way to report that a checkpoint failed. A failed checkpoint clears CheckpointInProgress() and the store's version still advances past the target, so the scheduler reads it as a success: it starts the min-time gate and moves to the next boundary. Combined with the fact that only dispatch is logged (checkpoint scheduled), a node can silently stop producing usable checkpoints while the scheduler reports healthy activity. Consider a completion/error signal on the interface, or at minimum a failure counter the stores increment.
• 2 suggestion(s)/nit(s) flagged inline on specific lines.

[yzang2019]

@seidroid review

[seidroid]

The relocation of the SS checkpoint helpers into ss/snapshot and the Scheduler→Checkpointer rename are complete and consistent (no stale references to the removed controller symbols remain), and the new global CheckpointScheduler is well tested. Two non-blocking issues remain in the scheduler: a dispatch that can be silently dropped by a store yet counted as a completed checkpoint, and a first cycle that runs before shutdown/cancellation is observed.

Findings: 0 blocking | 3 non-blocking | 2 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• [suggestion] checkpointPollInterval is a hard-coded 10s while the rest of the cadence is configurable, and CheckpointConfig.Validate does not relate the two. The scheduler can therefore dispatch at most one boundary per 10s, so a small CheckpointInterval (or fast blocks) silently yields a longer effective cadence than configured. Worth either validating/warning when the configured interval is short relative to the poll rate, or deriving the poll interval from the config.
• 2 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[suggestion] A dispatch a store drops is indistinguishable here from one it accepted. nextCheckpointVersion samples LatestVersion() and can return a target only one version above it (whenever latest ≡ interval-1 (mod interval)); if a store commits that version between the sample and this call, the CheckpointableStore contract says it silently ignores the request. The scheduler still sets scheduledVersion = targetVersion, and the next poll finds !CheckpointInProgress() and allStoresCommitted(target+1), so it records a completed checkpoint, starts the min-time gate, and moves to the next boundary. The result is stores holding checkpoints at different versions — the exact invariant this scheduler exists to enforce — with nothing in the logs distinguishing it from a healthy cycle. The window is narrow, but since nothing implements the interface yet the fix is cheap: have ScheduleCheckpoint report whether it accepted the target (bool or error) and have the scheduler log and re-target on rejection. This also covers the more general case the earlier review raised, where a store's checkpoint fails outright and completion is still inferred.

[seidroid]

[suggestion] The cycle runs before the loop ever observes stopCh or ctx.Done(), so Start() immediately followed by Close() — or Start() on an already-cancelled context — still dispatches one target to every store, contradicting the Start godoc ("begins dispatching targets until Close is called or ctx is cancelled"). Close then blocks in wg.Wait() until that dispatch completes. Dispatching ahead of the first wait is the right call for latency; a non-blocking select on stopCh/ctx.Done() before scheduleNextCheckpoint keeps that while making shutdown authoritative.

[cody-littley]

It seems odd to specify a time between checkpoints instead of specifying the target rate of checkpoints. By specifying time between checkpoints, nodes will have different checkpoint frequencies depending on how fast they can checkpoint.

An alternate strategy might be to do the following:

• specify checkpoint period, e.g. "once per 10 minutes"
• if a checkpoint takes longer than 10 minutes, then do not start a new checkpoint until the old one is finished
• Two possible protocols for handling long checkpoints:
  • Option 1: if a checkpoint takes longer than the checkpoint period, start the next checkpoint right away when the previous one finishes
  • Option 2: if a checkpoint takes longer than the checkpoint period, don't write another checkpoint until we hit the next time that is a clean multiple of the checkpoint period

[cody-littley]

Optional suggestion, feel free to ignore if you don't agree.

In many other places we split configs into their own files. Does it make sense to do so here as well?

[cody-littley]

Are we calling the on disk snapshots "checkpoints" now? Changed the name of the in-memory stuff to View so not to conflict with the name "snapshot". I don't mind calling them one thing or the other, I just want to make sure we settle on a uniform name in the code.

[cody-littley]

One nice to have property would be if this utility ensured that different checkpoint stores checkpoint the same block number. Not required for correctness of course, we're intentionally designing a system that doesn't break when checkpoints don't align (since a crash can cause us to be missing some checkpoints, even if we intend to checkpoint the same block). But its still nice to align them when we can.

If we wanted this property, we'd have to flip the model a little. Instead of the scheduler saying "it's time to checkpoint your current block", we'd have each of the stores ask the scheduler "should I checkpoint block X?". Via this mechanism we could ensure that we don't have an off-by-one block height difference when we snapshot our different stores.