feat(rollout): firmware rollout engine - #1017
Conversation
github-actions
Bot
added
server
review-policy: needs-review
🔐 Codex Security Review
Review SummaryOverall Risk: HIGH Findings[HIGH] Automated review incomplete
NotesHuman review is required because the bounded automated review was incomplete. Generated by Codex Security Review | |
fb38236 to
aa7cb94
Compare
aa7cb94 to
8c2457f
Compare
8c2457f to
54c4ded
Compare
b00602d to
eb35247
Compare
eb35247 to
2de6a7f
Compare
1fd4704 to
c1976f5
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3ded33a343
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Apply, rollback, cancel, retry, pause, resume and continue, batching and pilots with review gates, auto-continue, and the enforcement tick, aligned with the merged RolloutService contract (#1014): - assignments are keyed by canonical (manufacturer, model) pairs and identified by payload checksum; a file is resolved to its artifact at apply time, the file's target metadata is snapshotted, and dispatch waits while no uploaded file carries the checksum; - every assignment change opens a generation; rollback and retry of a finished rollout require the rollout to be current, and reconciliation rollouts inherit the generation's lineage; - members halted in the most recent rollout of the generation that holds them are suppressed until retried, and the retry set is exactly that suppression set, whichever rollout produced it; - managed-deployment provenance is recorded when a dispatched target reports the version, and DONE, on-target counts and the mismatch rule all require it; - every mutation carries an actor, an optional expected revision checked against the row under lock (STALE_REVISION carries the current value) and a note; failures carry machine-readable reasons; - the offline budget is the channel's live value shared across its active rollouts, counting offline targets and outstanding commands as slots; - telemetry gates fail closed under the coverage rule, with aggregates over verified miners carrying their sample counts and change fields absent when not derivable; - PreviewFirmware plans an apply, with an optional behavior override that cannot carry an offline budget.
A member with a pending or processing FirmwareUpdate for another artifact is mismatched under the contract's mismatch rule: the engine targets it, holds verification and provenance until the command drains, and tells a missing file (ARTIFACT_MISSING) from one whose metadata cannot back an assignment (ARTIFACT_MISMATCH).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0049e0143a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c800370ef7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 792d74a848
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7336fe2991
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f6e638db87
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d722bf91a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
2 participants
Reviewable diff: +3097/-58 across 17 files (excludes generated, test, and story files).
Summary
This PR turns release-channel firmware assignments into paced updates and corrects drift. It adds assignment workflows, a persistent rollout engine, operator controls, health evidence and progress reads. Dispatch coordinates with operator actions and commits queued commands together with their rollout bookkeeping; a durable channel-wide budget limits concurrent disruption.
Stack and scope. This is the first remaining PR, with its diff against
main. Merged #1015 supplies storage, verified delivery and concurrency primitives; merged #1016 supplies channel configuration, membership views and the cached availability API. The full chain is:#1014 contract, merged → #1015 storage and dispatch, merged → #1016 channel operations, merged → #1017 engine → #1018 API/runtime → #1019 client data → #1020 channel UI → #1021 active updates → #1022 header/activity → #1023 E2E.
Domain decisions live here. #1018 exposes/schedules them, #1019–#1022 add client workflows, and #1023 adds E2E. Delegated execution/controller lifecycle and the rollout events feed remain deferred beyond this stack.
How it works
Assign firmware and snapshot work
Preview reports affected miners and batches, including no-op clears for never-assigned or already-cleared pairs. Target and on-target counts share one database snapshot; queued commands for another artifact prevent a miner from counting as on-target, while generation suppression applies only to proposed targets. Apply locks the channel before reading assignments, then saves artifact checksum and target/version metadata per manufacturer/model pair. Changed assignments advance their generation and atomically persist their predecessor on the assignment, including when every member already matches and no rollout is needed. They supersede active work and snapshot behavior, target order, batch/health baselines and rollback lineage. Later reconciliation inherits that generation’s persisted predecessor. Unchanged assignments preserve active work; clearing stops enforcement.
Apply, rollback, reconciliation and retry use the same channel-before-rollout lock order. Reconciliation treats its initial scan as candidate discovery, then reloads the current assignment and checks for active work under the lock. A finished retry checks the current generation after waiting for that lock. Concurrent clears or replacements therefore cannot create obsolete work. Reconciliation start events include the current channel name. Rollback restores the preceding artifact or clears a first assignment; rollback and drift correction run all at once.
Reconcile membership and dispatch atomically
Before starting new work, each tick adopts eligible successful commands sent by finished rollouts. This updates device provenance without resuming the rollout, rewriting target phases or clearing suppression. New evidence may advance its revision for polling; retry also performs this check before creating replacement work. Active preparation locks the channel and rollout, reloads active state, then reconciles membership, provenance and verification. A canceled or superseded rollout is skipped without changing its targets or retained history. Returners retain their batch/baseline but verify again; late joiners have no baseline and must hash. Enrolled targets remain tied to their paired device even if firmware changes its reported hardware identity. Live compatibility still gates new dispatches.
Dispatch uses the current channel budget. With the channel and rollout locked, the engine reloads active/unpaused state and targets and excludes miners that left since preparation. It holds those locks through command enqueue and attempt recording, so pause, cancel and assignment changes serialize with dispatch. The command batch, queue messages, attempts and command reservations commit together. Workers only see commands whose attempts and reservations committed; rollback removes the batch, queue entries and bookkeeping together. Command activity and batch tracking start only after commit. The dispatch transaction does not retry its callback, and nested dispatch transactions are rejected.
FirmwareUpdateArtifactuses the saved artifact with strict payload checks. Command batches retain the assignment's owning user independently of system/API audit actors; reconciliation and retry preserve that owner, while rollback creates a new assignment. Command activity identifies background enforcement as a system actor without a synthetic user identity. AnyPENDINGorPROCESSINGfirmware command makes the miner wait before another send or attempt exhaustion, even after the resend interval. Queued checksums take precedence over legacy file IDs. Terminal firmware-command results atomically clear trusted provenance, including failures that may occur after installation. The retained empty row advances the provenance timestamp and records the completed batch UUID, so stale observations cannot restore an older artifact. This per-device identity follows database writes rather than worker clocks and remains present after provenance is adopted.Share the offline budget across rollout history
Migration 149 adds durable assignment predecessors, a per-device completion witness and
firmware_rollout_reservation, keyed by channel, device and dispatched batch. The budget counts each device once across all hardware groups: historical rollout targets that are currently offline, plus outstanding dispatched commands whose targets have not yet been observed offline. Exclusion, cancellation, retry, completion and elapsed retry time do not silently free a slot.The engine records offline observations every tick, including when rollouts are paused, gated or terminal. An observed offline-to-online recovery releases the command reservation; command termination also removes it, but an offline historical target still consumes capacity until recovery. Fleet deletion releases the device. Existing dispatches are backfilled conservatively; assignment predecessors are recovered from same-generation history where available. Each newly created rollout snapshots the current channel limit, including reconciliation, retry and rollback. The merged migration 148 stays unchanged.
Verify progress and advance stages
Verification requires the target version, matching provenance, no foreign firmware command, online status and baseline-appropriate hashing. Guarded writes and a reload before DONE protect concurrent deployments. Missing provenance, changed hardware identity or replacing provenance with another artifact reporting the same version additionally requires a durable successful result for the exact batch, device and checksum. The retained current command must resolve to a successful result for the same immutable checksum; a different, failed, unknown or missing result blocks older evidence. Historical legacy file IDs cannot establish immutable artifact identity. Queued/failed commands and preflight skips cannot establish successful evidence; incompatible targets fail without another incompatible send.
Batches and pilots stop at review gates or waits. Auto-continue evaluates stabilization, paired health samples, coverage and new errors. Timers subtract paused time and reject stale advancement decisions. Stage advancement and completion lock the rollout and reload targets before checking settlement. A committed pause or newly retried target therefore prevents a stale tick from advancing or completing work. Drift reopens updating and stabilization; later health changes alone do not rewrite persisted DONE or finished history.
Evidence reports neutral excluded counts without using their health samples or blocking readiness. Its scope is the current batch, every target for an unbatched rollout, or unbatched rest targets after batching; operational completion ignores departed targets.
Control and read work
Actions check optional
expected_revisionand preserve actors/notes. Continue requires resume while paused. Cancel preserves completed work and suppresses unfinished targets for the generation, including targets that have left the channel; retry resets/appends active targets or starts eligible replacement work. Missing organization-scoped resources returnNOT_FOUND; database failures retain their causes instead of becoming business-state errors.List/detail and mutation responses include counts, evidence, paged devices and cached advisory file availability. Each response shares checksum lookups without rehashing payloads; assignment, enforcement and delivery keep strict checks. Incremental polling captures a transaction boundary, drains all pages, then inclusively rechecks that boundary on the next cycle so late commits remain visible. Callers merge replayed rows by ID/revision; status must remain unset to observe every status transition. The UI continues full-list polling.
Diagrams
flowchart TD A["Apply, rollback or retry"] --> B["Lock channel and read current assignment"] C["Enforcement tick"] --> P["Adopt eligible completed dispatches without resuming history"] P --> B B --> D["Create eligible rollout and snapshot targets"] C --> O["Observe offline and recovered reservations"] D --> E["Lock channel then rollout; recheck active state and reconcile"] E --> F["Lock channel then rollout; refresh budget and dispatch state"] F --> G["Wait for pending commands; check artifact and compatibility"] G --> H["One commit: command batch, queue, attempts and reservations"] H --> I["Start tracking; workers execute firmware update"] I --> J["Terminal command results and independently arriving miner telemetry"] J --> C O --> F E --> K["Lock rollout; reload targets and evaluate progress"] K --> L["Rollout list and detail views"] M["Cached advisory file availability"] --> LsequenceDiagram participant C as Polling caller participant D as Rollout domain participant S as SQL store C->>D: First page with previous polling cursor D->>S: Capture next-cycle transaction boundary loop Drain pages with unchanged filters D->>S: Read eligible rows D-->>C: Rows and same next-cycle polling cursor opt Another page remains C->>D: Request next page end end C->>D: Start next cycle with returned polling cursor Note over C,S: Late changes remain eligible and rows may repeatAreas of the code involved
service.go;behavior.gorollout.goenforce.go; terminal provenance reconciliationdb/commit_hooks.go; queue serviceproto/rollout/v1/rollout.protoserver/generated/**; client generated protobuf bindingsKey technical decisions & trade-offs
Testing & validation
This review round is validated with canonical
cd server && just gen, full rollout, command, SQL-store and migration suites, and server lint. Focused provenance, terminal-adoption and completion-retry cases pass under-race. The top of the stack also passes rollout/API handler and migration tests plus lint. New cases cover later failed/foreign commands, backward/equal worker clocks, missing audit witnesses, stale insert/update races, real database deadlocks, and retry without repeating device execution. Migration tests exercise downgrade/upgrade and preserve existing provenance. Client and protobuf sources are unchanged in this round.Coverage also includes assignment lineage and no-op clears; foreign-command preview counts; scope changes between preparation and dispatch; cancellation, exclusion and reentry; pause/cancel and overlapping dispatches; exact provenance and excluded evidence; long-running queues, historical offline targets and paused recovery; atomic queue/bookkeeping rollback and deferred-commit failures. Migration tests cover ordered downgrade/upgrade and reservation backfill from existing dispatches.
Most domain tests substitute miner dispatch; command transaction tests use the production command/queue path with Postgres. Hardware delivery integrity remains covered upstream, and browser workflows land in #1023. Final CI status is in the PR checks.