feat(seinode): add a freeze sub-spec for fullNode and archive nodes

[bdchatham]

Makes a node held at a block height a declared configuration. Spec: specs/freeze-node/spec.md.

Stacked on #507 (the sei-config v0.0.27 bump). Review that one first; this PR retargets to main when it merges.

What

A freeze sub-spec on the fullNode and archive modes:

spec:
  archive:
    freeze:
      height: 5000000

The node executes to that height, stops, and serves query RPC there indefinitely.

Why not spec.overrides

The key is reachable through spec.overrides since v0.0.27, and that route is wrong for two reasons.

The controller cannot derive its own behaviour from an opaque string map. A frozen node needs a different readiness probe. Making readinessProbeForNode parse spec.overrides would be fragile and semantically wrong.

mergeOverrides copies user overrides last, so a user key silently outranks a controller-derived one. Only one source can stay writable.

Peer review

Three specialists reviewed this independently: kubernetes-specialist, systems-engineer, platform-engineer. Two reproduced the top finding against a live 1.34 API server. The second commit is the response.

The defect they found

The field-level self == oldSelf on height was insufficient. CEL skips a rule reading oldSelf when the path is absent from the stored object, so adding freeze to an existing node and removing it from a frozen one both passed admission.

Adding it was the dangerous half:

1. Admission accepts.
2. reconcileStatefulSet runs an unconditional SSA on every reconcile, so the pod template gets the frozen probe immediately.
3. chain.freeze_height never arrives — only TaskConfigApply carries a ConfigIntent, and it appears only in bootstrap programs. A Running node's plan writes config.toml [p2p] keys alone.
4. UpdateStrategy: OnDelete leaves the live pod untouched, so nothing looks wrong.
5. Weeks later an image bump or a Karpenter consolidation replaces the pod. It returns as an ordinary chain-following RPC node with lag-based readiness removed, and reports Ready however far behind it falls.

Removing freeze is the mirror failure: app.toml keeps the height while the probe reverts to /lag_status, so the node goes NotReady permanently at the next pod replacement. The operator's obvious remedy for a mistake is the action that breaks it.

FN-3a — has(self.freeze) == has(oldSelf.freeze) on the mode sub-spec, where has() is observable on both sides — makes both states unreachable.

A model I had wrong

I originally wrote that a frozen node's lag "grows without bound". It does not. Lag is a constant: block sync hands off at the freeze height and cancels the peer-update routines, but the pool is never cleared, so it keeps returning the tip it last observed. For a historical freeze height that constant sits far above the default threshold of 300, so /lag_status returns HTTP 417 permanently and the pod never becomes Ready.

FN-8 is therefore a fix for a real first-bootstrap outage, not the hardening I first called it. Freeze and Autobahn also cannot coexist (sei-tendermint/node/node.go:164), so my earlier giga reasoning was unreachable. The wrong model has been corrected in the code, the tests, and the spec.

Validation, all at admission

ID	Rule
FN-3	freeze.height is immutable
FN-3a	freeze is create-only: no add, no remove
FN-5	chain.freeze_height is rejected in spec.overrides
FN-6	freeze excludes snapshotGeneration
FN-10	freeze excludes chain.halt_height / chain.halt_time — sei-config refuses to resolve the combination
FN-11	snapshot.s3.targetHeight must be below freeze.height — seid refuses to start once a store has reached it
FN-12	freeze excludes snapshot.stateSync — seid silently falls back to block sync from genesis

FN-10 through FN-12 came out of review. Each one turns a failure that happened on the node into a rejection at kubectl apply.

Controller changes

• controllerOverrides in both planners emits chain.freeze_height (FN-7).
• readinessProbeForNode uses HTTP /status for a frozen node (FN-8). A bare TCP connect succeeds while the listener is saturated, because seid wraps it in a LimitListener that stops accepting rather than refusing.
• ResourceLabels stamps sei.io/frozen: "true" (FN-13). NodeFellBehind selects on sei.io/role and would fire forever without resolving once the chain advanced past its threshold beyond the freeze height.

Merge preconditions

Run this first. FN-5 and FN-10 narrow validation on an existing served field, which the spec's own NFR-1 forbids. A reviewer reproduced the consequence: a stored SeiNode already carrying the key becomes un-updatable for any spec edit, and SeiNetworkReconciler performs full updates on children, so such an object would error every loop.

kubectl get seinodes -A -o json | jq -r '.items[]
  | select(.spec.overrides["chain.freeze_height"] != null
        or .spec.overrides["chain.halt_height"]  != null
        or .spec.overrides["chain.halt_time"]    != null)
  | "\(.metadata.namespace)/\(.metadata.name)"'

kubectl get seinetworks -A -o json | jq -r '.items[]
  | select(.spec.configOverrides["chain.freeze_height"] != null)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Empty output on every cell retires the concern. Non-empty output is a blocker.

A frozen node's manifest MUST omit kustomize.toolkit.fluxcd.io/force: enabled. Every prod SeiNode carries it today. Flux's IsImmutableError returns true for any 422 Invalid — what every CEL rejection returns — and the force annotation then makes it delete and recreate. Nothing enforces sei.io/deletion-protected: no policy, no webhook, no controller reads it. So a git edit to freeze.height triggers a delete, and the controller removes the data PVC unless the operator imported the volume. The frozen state is the product.

The spec's Platform preconditions section carries this and three more, all found by reading the platform repo.

Testing

Check	Result
go test ./... all three modules	pass
envtest — 13 freeze cases	pass
envtest — full suite (node, seinetwork, nodetask)	pass
go vet ./...	clean
gofmt -l .	clean
staticcheck ./...	13 findings, same 13 as main — none introduced
make manifests generate	idempotent
vale specs/freeze-node/spec.md	0 errors
Rendered StatefulSet for 7 unfrozen shapes	byte-identical to main (FN-9 / SC-5)

Every requirement in the spec's traceability table has a test that runs. The one criterion left open is SC-3 — a frozen node holding Ready on a real cluster past its freeze height.

Deliberately deferred

• A ValidateFrozenProbes render guard and a FrozenReady condition. All three reviewers ranked these cuttable, and FN-3a removes the silent-failure state that made the condition urgent.
• An exec probe comparing /status's height against freeze.height - 1. /status does not prove the node reached its height, so it reads Ready during the initial block sync. /status is still strictly better than the TCP probe it replaced, at the same cost. Un-defer condition is in the spec (Q-2).
• A mirror guard on SeiNetwork.spec.configOverrides (D-5), and a Freeze field on sdk/sei. Both recorded in the spec.

🤖 Generated with Claude Code

[cursor]

PR Summary

Medium Risk
Changes CRD validation and pod readiness/labels for a new operational mode; existing SeiNodes with forbidden override keys may become un-updatable until cleaned up.

Overview
Adds a declared freeze sub-spec on fullNode and archive so operators pin a node at a block height (execute through height-1, then serve query RPC there) instead of using spec.overrides for chain.freeze_height.

API / admission: Introduces FreezeSpec with immutable height, SeiNodeSpec.Freeze(), and extensive CEL on the CRD—freeze is create-only (including across mode switches), mutually exclusive with snapshot generation, state sync, and halt overrides, and blocks chain.freeze_height in overrides.

Controller: Full/archive planners emit chain.freeze_height via freezeOverrides on bootstrap; frozen pods use HTTP /status readiness (not /lag_status, whose lag never shrinks) and get sei.io/frozen: "true so height-based alerts can skip them. Bumps sei-config to v0.0.27.

Tests: envtest admission suite for freeze rules, planner override/key pinning tests, and readiness/label unit tests. Unfrozen node manifests stay unchanged.

^{Reviewed by Cursor Bugbot for commit d74d5ff. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bca4c31. Configure here.}