Ephemeral deploys can seed from stale/wrong golden snapshots (recency-blind resolver + no refresh after golden bumps)

[nachog00]

Summary

Ephemeral/preview deploys can be seeded from a stale or wrong golden snapshot,
because of two independent gaps in the snapshot pipeline:

1. The snapshot resolver is recency-blind — it picks the alphabetically last
   snapshot name, not the newest by creation time.
2. Golden snapshots are not refreshed after golden advances or crosses a network
   upgrade, so the newest snapshot can be badly out of date.

Together these caused a preview deployment (preview-080-rc1) to be seeded from a
pre-Ironwood golden snapshot and hard-stall for ~62 hours, never syncing a single
block.

Update — 2026-08-17 (post-filing corrections)

Two statements in the original writeup need correcting/updating:

• golden-mainnet runs zebra 6.3.0, not 6.0.0. Both
  clusters/production/values/golden-mainnet.yaml and the live zebra-0 pod
  (zfnd/zebra:6.3.0) confirm it. 6.0.0 was the version golden ran when the stale Aug-4
  snapshot was cut; golden has since been bumped to 6.3.0. So the zebra version is not
  the reason golden is healthy and the preview stalled — both run 6.3.0. The real
  differentiator is that golden crossed the fork gradually while live whereas the preview
  had to bulk-fetch across the boundary from a cold pre-fork snapshot.
• Fresh post-Ironwood golden snapshots have now been cut (manually). The current
  latest=true zebra snapshot is from 2026-08-17 (zebra 6.3.0), taken from golden's live
  tip (~3,451,331, well past Ironwood at 3,428,143). So the "13 days stale / never refreshed
  since Aug 4" framing below no longer holds as of today.

The two underlying gaps this issue is about — the recency-blind resolver and the
missing snapshot automation (nothing cuts snapshots on golden bumps or on a schedule) —
are unchanged and still need fixing; today's snapshot was a manual one-off. The
root-cause analysis below is left intact as the incident record.

Impact — what we hit this session

• preview-080-rc1 (zebra 6.3.0, fully Ironwood-capable) restored a golden zebra
  snapshot at height 3,428,006, which is pre-Ironwood (NU6.3 activation =
  3,428,143).
• From a cold pre-fork snapshot it has to bulk-download across the NU boundary. Those
  block-download requests time out (ValidationRequestError: Elapsed), extra_hashes=0,
  CPU idle at ~45m (not verification-bound) → zero blocks committed, stalled ~62h
  (≈ its entire pod lifetime).
• golden-mainnet (zebra 6.3.0, same build as the preview) is fine at tip 3,451,331
  only because it crossed Ironwood gradually while live and never has to bulk-fetch across
  the fork. The differentiator is how it crossed the fork, not the zebra version — both
  golden and the preview run zebra 6.3.0. (The stale Aug-4 snapshot was cut when golden still
  ran 6.0.0, but golden's binary has since been bumped to 6.3.0.)

Root cause 1 — recency-blind resolver

platform/argo-workflows/workflows/deploy-ephemeral.yaml (resolve-params step):

ZEBRA_SNAP=$(kubectl get volumesnapshot -n "$GOLDEN_NS" \
  -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep '^zebra-' | tail -1)

kubectl get lists alphabetically by name, and snapshot names carry random suffixes
(...-4n8fc, ...-tq54f). tail -1 therefore selects the lexically-last name, which is
unrelated to creation time. "Latest" is effectively random w.r.t. recency. Same bug
for the data-zaino- resolution a few lines down.

The infrastructure to do this correctly already exists and is being ignored. The
snapshot-golden workflow already labels every snapshot it cuts and maintains a
single-winner latest marker:

• label-volumesnapshots sets zcash-stack/app, zcash-stack/network,
  zcash-stack/source=golden, zcash-stack/latest=true, zcash-stack/height,
  zcash-stack/zebra-version, zcash-stack/timestamp.
• unset-latest-label removes latest=true from the previous snapshots first, so exactly
  one snapshot per (source, network, app) is ever latest=true.

The resolver simply doesn't consume this label — it name-sorts instead.

Fix (preferred): select by the label the workflow already maintains:

ZEBRA_SNAP=$(kubectl get volumesnapshot -n "$GOLDEN_NS" \
  -l zcash-stack/app=zebra,zcash-stack/network={{workflow.parameters.network}},zcash-stack/source=golden,zcash-stack/latest=true \
  -o jsonpath='{.items[0].metadata.name}')

(Fallback if we ever can't rely on the label: --sort-by=.metadata.creationTimestamp | tail -1.
Either way, also filter on status.readyToUse==true so we never clone a half-baked snapshot.)

Secondary bug — the height label is not being populated. The current latest=true
zebra snapshot carries zebra-version=6.0.0 but an empty zcash-stack/height label.
query-zebra-metrics extracts height from state_checkpoint_finalized_block_height off the
zebra :8080 metrics endpoint; that grep is coming back empty (metric renamed/missing, or a
scrape-timing miss). Non-fatal (the snapshot still works), but it makes the labels useless for
picking a snapshot at/above a target height and hides staleness. Worth fixing alongside.

Root cause 2 — no snapshot refresh after golden changes

At the time of the incident, the newest golden zebra snapshot was 13 days old and
pre-fork. It was cut ~Aug 4, in the brief window after golden was bumped to Ironwood-capable
zebra 6.0.0 but before it had crossed the fork. No snapshot had been taken since golden
crossed Ironwood, so every ephemeral cut between Aug 4 and Aug 17 seeded pre-Ironwood.
Snapshots are only created by manually running the snapshot-golden workflow — which is
exactly why this went unnoticed for 13 days. (Fresh post-Ironwood snapshots have since been
cut manually — see the Update at the top — but the automation gap that caused the drift
remains.)

Proposed fixes

• Fix the resolver to select by the existing zcash-stack/latest=true label
  (+ require readyToUse); creationTimestamp as fallback. Low-risk, do this regardless
  of the freshness strategy below.
• Fix the empty height label in snapshot-golden (query-zebra-metrics height
  extraction returns blank).
• Establish a snapshot-freshness strategy (see open question).
• Cut fresh post-Ironwood golden snapshots — mainnet done (new latest=true
  zebra 6.3.0 snapshot cut 2026-08-17). Still pending: testnet, and re-seed
  preview-080-rc1. Note this was a manual cut; the resolver + automation fixes above
  are still required so it doesn't silently regress.

Open question — how do we keep golden snapshots fresh?

Option A — per-deploy fresh snapshot: each new ephemeral cuts a fresh golden snapshot
at deploy time.

• Always current; zero staleness.
• But couples snapshot cost + crash-consistency risk into the hot deploy path, on every
  deploy; concurrent ephemerals each snapshot golden's live PVC → storage churn.

Option B — nightly CronWorkflow snapshot (recommended): a cron cuts fresh golden
snapshots nightly (mainnet + testnet); ephemerals resolve the latest (with the sort fix).

• Bounded staleness (≤24h), decoupled from the deploy path (deploys stay fast), one
  snapshot shared by all that day, predictable storage, crash-consistency risk taken once
  per night rather than per deploy.
• Up to 24h stale — acceptable for testing; the only real edge is an NU crossing within the
  day, which Option C covers.

Option C — event-driven (nightly floor + snapshot-on-golden-bump): trigger a fresh
snapshot whenever golden changes version or is known to have crossed an NU, with nightly as
a floor.

• Best correctness for the two discrete events that actually matter (version bump, NU
  crossing) — both of which caused this incident — while keeping the deploy path fast.

Recommendation: B + C — nightly cron as the baseline, plus a snapshot triggered on
golden version bumps. Skip A: the failure here was never "24h stale," it was "13 days stale
and never refreshed after a bump/NU." A nightly floor + on-bump snapshot would have fully
prevented it, without paying snapshot cost/risk on every ephemeral deploy. The resolver
sort fix is required either way.

Related

• There is already a nightly CronWorkflow alerting on new zebra releases (commit ec65798).
  The missing automation is: fresh snapshots (on-bump + nightly) and a recency-correct
  resolver — and, separately, keeping golden itself tracking the latest zebra more closely.

---

Update — 2026-08-17 ~16:23Z (state changed shortly after filing)

The analysis above was accurate as of filing (16:17Z). Within ~6 minutes, golden was
remediated, so two point-in-time facts in the body are now stale — recorded here rather
than rewritten, to keep the incident record intact:

• golden-mainnet zebra bumped 6.0.0 → 6.3.0. zebra-0 restarted at 16:23:35Z onto
  zfnd/zebra:6.3.0 (0 restarts since). So the body's "golden on older zebra 6.0.0" / "crossed
  Ironwood gradually while live on 6.0.0" describes golden at incident time; it now matches
  preview's 6.3.0. (No commit for this bump is in platform/ git history yet — appears applied
  live; GitOps reconciliation still owed.)
• Fresh post-Ironwood golden snapshot cut. zebra-data-zebra-0-snapshot-jb5q8 created
  16:23Z, zebra-version=6.3.0, zcash-stack/latest=true (previous snapshots correctly
  demoted). This supersedes the "newest snapshot is 13 days old / pre-fork" statement in Root
  cause 2 → ✅ checks off "Cut fresh post-Ironwood golden snapshots now."

Everything else stands and is verified against the live cluster:

• Resolver is still recency-blind — deploy-ephemeral.yaml:161-162 (zebra) and :172-173
  (zaino) still grep '^zebra-' | tail -1. Not yet fixed.
• zcash-stack/height label still empty on all golden zebra snapshots (incl. the new jb5q8).
  Not yet fixed.
• preview-080-rc1 is still hard-stalled at height 3,428,006 (pre-Ironwood; NU6.3 =
  3,428,143), network_upgrade=Nu6_2, zero blocks in ~63h. Re-seed from jb5q8 still owed.

Net: the two snapshot-freshness symptoms were manually remediated for this one incident; the
two code defects (recency-blind resolver, empty height label) and the automation strategy
(nightly + on-bump) remain open, and preview-080-rc1 still needs re-seeding.