Fastly chunked-config GC: reclaim orphaned chunk entries on re-push (last-writer-wins)#314
Fastly chunked-config GC: reclaim orphaned chunk entries on re-push (last-writer-wins)#314aram356 wants to merge 27 commits into
Conversation
…te+scope prior_chunk_keys, offline cloud dry-run, local root inference, warning semantics, invert stale no-GC test
…ence unsound); Value-first prior_chunk_keys so invalid pointer-kind warns; drop 'atomic' overclaim; define local dry-run degrade semantics; state cloud GC runs only after full commit
Value-first prior_chunk_keys (pointer-kind-but-malformed warns), thread logical roots into write_fastly_local_config_store via roots: &[&str] (no infix inference, since --key is free-form), best-effort local dry-run counts, post-commit-only cloud sweep. Task-by-task with TDD steps for subagent-driven-development.
…entical re-push counts 0 (was over-counting); enumerate all 10 writer call sites + roots args; forbid --all on delete; reword failed-delete warnings as informational (inert, future config gc); note sequential-spawn latency + approximate line numbers
…t root read-back guard (invariant 5); build keep-sets from per-root expand_root instead of prefix-scanning flattened entries; make reserved-infix --key rejection mandatory at the Fastly adapter boundary + flip the infix test to expect rejection; add local suspicious-pointer real-push test and cloud concurrency-guard test
… (drop 'race-safe' overclaim, add Concurrency model section + plan precondition gate); correct cost note for the post-commit read-back describe; add dry-run suspicious-pointer test
Concurrent cloud pushes are SUPPORTED: root pointer is upsert so the last write wins on the value. GC obeys LWW via the post-commit read-back guard — a push reclaims prior chunks only while it is still the last writer of the root, else it yields (never deletes the winner's live chunks). Removes the single-writer assumption and the blocking 'do not implement' gate; keeps the honest best-effort residual-window note. No code.
…inter test (seed real chunk keys, assert they survive); expand_root errors on empty instead of silent default; reserved-key error wording drops --key assumption
…dated, prefix-scoped) + unit tests
…k_keys, reject_reserved_root_keys, FastlyConfigGcPlan) + unit tests Wired into push paths in the following commits; transient dead-code warnings until then.
write_fastly_local_config_store takes exact per-root keep-sets (gc_roots) and prunes orphaned chunk keys in the same in-memory rewrite; suspicious prior pointers warn and delete nothing. push_config_entries_local rejects reserved keys, threads per-root expansion, and reports best-effort orphan counts in dry-run. Inverts the stale no-GC test; adds shrink-to-direct, suspicious-pointer, reserved-key, and dry-run count/identical/unknown tests.
…riter-wins) push_config_entries rejects reserved keys, reads each root's prior value before commit, and after the commit sweeps orphaned chunks guarded by a post-commit root read-back (deletes only while still the last writer; yields otherwise). Adds delete_config_store_entry (--key --auto-yes, never --all) and an offline dry-run GC-intent line. Failed deletes and suspicious/absent priors degrade to warnings; the push still succeeds. Adds a command-aware fake fastly harness and 7 cloud GC tests.
aram356
changed the title
…astly via handlebars Moves FastlyConfigGcPlan to the struct group with alphabetical fields; renames single-char closure idents; replaces bare arithmetic with saturating_add; fixes map_err/shadow/assert-on-result-state/absolute-path lints; relocates GC helper unit tests after the test-module structs. Adds handlebars dev-dependency and rewrites the cloud fake-fastly test shim to render its shell script from a handlebars template.
…ms_after_statements)
aram356
changed the title
- local: GC of a chunked root leaves a chunked sibling's chunks intact (prefix-scoping vs shared string prefix app_config / app_config_staging) - cloud: identical-bytes re-push deletes nothing (read-back returns our own value, so the assertion is non-vacuous) - cloud: prior-read failure warns and deletes nothing (extends the fake fastly with a describe_hard_error mode)
…t delete argv + cloud shrink-to-direct
- local dry-run: distinguish absent (0) from present-but-non-table
("unknown: could not read prior state") via local_contents_table, so
--local --dry-run no longer reports 0 orphans for state the real writer
would reject; + non-table-contents test
- cloud: assert every delete argv passes --key + --auto-yes and NEVER
--all (blast radius); fake now logs the full delete argv
- cloud: add the shrink-to-direct test (prior chunked -> new direct
deletes all prior chunks, root upserted not deleted)
aram356
changed the title
prk-Jr
left a comment
prk-Jr
left a comment
There was a problem hiding this comment.
🔧 Summary
The local reclamation path and delete-set validation are well implemented, but the cloud sweep needs a safe reclamation boundary before prior chunks can be deleted. Immediate deletion can break readers under eventual propagation or a concurrent push.
📝 CI Status
- fmt: PASS
- clippy: PASS
- tests: PASS
ChristianPavilonis
left a comment
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Summary
Reviewed the Fastly chunk-GC implementation and supporting tests/specification. Approving with two follow-up findings included as inline comments.
…uplicate root keys (P2) P1 (security): fetch/read error paths interpolated the raw `describe` stdout — i.e. the stored config envelope, which may hold credentials — into errors the CLI logs verbatim. The GC prior-read runs on EVERY cloud push, so this exposed the previous config on any schema drift. Now report only size + top-level field-name shape via redact_describe_response; adds sentinel-secret tests for both the read and push paths. P2 (correctness): a duplicate logical root in one batch built two GC plans against the same prior; the earlier plan reclaimed the chunks the LAST tuple had just installed, leaving the final pointer dangling. Reject duplicate root keys in both push paths, before any expansion or I/O.
…nder eventual consistency) Fastly Config Store is eventually consistent: after the control plane accepts pointer N, POPs may still serve pointer N-1 and need N-1's chunks. The post-commit read-back only observes the CONTROL PLANE, so eager deletion could strip chunks from a POP mid-propagation -- breaking readers on every re-push, not merely under concurrent writers. (Reported on PR #314.) Cloud GC is now DEFERRED: - a push never deletes what it just superseded; it RECORDS that generation in a per-root pending record at <root>.__edgezero_gc.pending - a later push reclaims a recorded generation only once it has aged past a grace window (EDGEZERO_FASTLY_GC_GRACE_SECS, default 24h) AND nothing live references it (neither the new generation nor the just-superseded one) - the record stores (sha, count, superseded_at) per generation, not raw keys: keys are derivable, and the raw list would blow the 8 000-char entry limit - it is a LIST, so pushing faster than the grace window cannot drop an un-reclaimed generation and leak it untracked - failed deletes keep the generation pending so a later push retries - the LWW read-back guard still gates all GC state changes The LOCAL path keeps eager pruning: fastly.toml is a single file Viceroy reads at startup -- no propagation window, no POPs. Extracts reclaim_and_record; adds pending-record round-trip and derive_chunk_keys tests, plus reclaim-after-grace / retain-within-grace / defer-instead-of-delete cloud tests.
…and duplicate-root invariants Records the design correction from the PR #314 review: eager cloud deletion is unsafe under Fastly's eventual consistency (the read-back only sees the control plane, not POP propagation), so the superseded generation is recorded and reclaimed by a later push after a grace window. Documents the pending-record shape (compact sha+count, list-not-slot), the 24h configurable grace, and the two further invariants (never echo config payloads; reject duplicate roots).
|
Thanks both — all three findings were real and are now addressed. @prk-Jr — eager cloud deletion (blocking) ✅You're right, and it's worse than a concurrency edge case: the read-back only observes the control plane, so it says nothing about POP propagation. Eager deletion could strip chunks from a POP still serving the previous pointer on every re-push, not just under concurrent writers. Knowing what to delete was never the problem — knowing when it's safe is, and no control-plane read answers that. Cloud GC is now deferred. A push never deletes what it just superseded; it records that generation, and a later push reclaims it once it has aged out:
The local path keeps eager pruning — New tests: @ChristianPavilonis — P1, prior config in push logs ✅Confirmed and fixed. @ChristianPavilonis — P2, duplicate roots in one batch ✅Confirmed exactly as described: two plans against the same prior, and A's plan reclaims the chunks B just installed. Duplicate logical root keys are now a hard error in both push paths, before any expansion or I/O — rejecting rather than coalescing, since a duplicated key is a caller bug and picking a winner would hide it. Regression test uses your scenario (prior B, batch Spec updated with the design correction and both invariants. Green: 123 fastly tests, |
… too The stdout fix was half a fix. The describe and update --stdin paths carry the stored config value, so a Fastly failure that quotes the payload back in stderr put credentials into logs via the FAILURE branch — the same exposure, different door. Redact stderr on those three value-bearing sites (not-found classification still inspects it internally; only the user-facing string is suppressed). delete/create/list stderr carry no value and are left verbatim. Adds sentinel-secret tests for the stderr path on both the read and push paths, complementing the existing successful-but-drifted stdout tests.
…the metadata sidecar The pending-record protocol was unsound and unfixable in place (PR #314 review): Fastly has no CAS, so a failed sidecar write / failed read-back / concurrent push PERMANENTLY lost a generation (and the 'will retry' warnings were false); and the record itself overflowed the 8000-char entry limit at ~71 generations, after which every later generation was silently lost. Verified against the live Fastly API and rebuilt with NO metadata at all -- the store IS the state: - `config-store-entry list --json` returns item_key + created_at per entry - a generation is superseded exactly when the NEXT one is written, so superseded_at(G) = created_at(successor(G)), read straight off the listing - reclaim G iff it is unreferenced by the live pointer AND now - superseded_at(G) >= grace (EDGEZERO_FASTLY_GC_GRACE_SECS, default 24h) - the generation this push just superseded is ALWAYS protected, so POPs that have not yet seen the new pointer keep their chunks IMPORTANT: the root entry's own updated_at is NOT usable -- Fastly does not bump it on `update --upsert`. Verified live: a root last 'updated' 2026-07-07 pointed at chunks created 2026-07-13. Using it as the supersession clock would have reclaimed the previous generation immediately. This dissolves four review findings structurally rather than patching them: no record to be malformed or to overflow, no lost updates, and deletes now target the store's ACTUAL keys (never keys re-derived from a content address, which could retarget onto keys the pointer never referenced). chunk_key_generation validates the key shape (hex sha + numeric index) so a hand-edited or foreign key can never become a delete target.
…an superseded Resolves the contradiction the review flagged: the spec declared deferred reclamation while its algorithm/cloud sections still specified eager deletion, and the plan was entirely eager-delete based. There is now ONE authoritative contract. The spec records both rejected designs (eager delete; metadata sidecar) and why, so the mistakes are not repeated, and documents the live-API findings that drove the final design -- including that the root entry's updated_at is NOT bumped by `update --upsert` and therefore cannot be used as a supersession clock. The plan is retained as a historical build guide with a prominent SUPERSEDED banner pointing at the spec.
|
All six findings addressed. The first four were fatal to the metadata design, not patchable — so I removed the metadata entirely. The sidecar is gone. The store IS the state.I probed the live Fastly API and found two things: (a) Chunk entries' own Post-commit, per root: read-back (LWW) → list → group this root's actual keys into generations → This dissolves four findings structurally rather than patching them:
The just-superseded generation is unconditionally protected — there's a test asserting it survives even with Remaining twoPayload redaction — completed. You were right that I'd only closed half the door: Docs — reconciled. There is now one authoritative contract: the spec describes only the shipped design and records both rejected designs (eager delete; sidecar) with the reasons. The plan carries a prominent SUPERSEDED banner pointing at the spec. Gates: 121 fastly tests, workspace tests, Please re-review the cloud path in full; it is a different design, not a patched one. |
…oked `config gc` A cloud push no longer deletes anything. Safe automatic reclamation is not achievable with Fastly's primitives, and three designs failed review proving it: 1. eager delete -> unsafe (eventual consistency: POPs may still serve the previous pointer, and the read-back only sees the control plane) 2. metadata sidecar -> unsound (no CAS => lost updates; 8000-char overflow) 3. store-derived clock -> unsound (chunk created_at is NOT pointer-supersession time: a chunked -> direct -> direct transition leaves the old generation with no 'successor' at all, so it ages from its own creation and gets deleted while POPs still serve it) The missing fact -- 'the pointer that referenced these chunks has stopped being served everywhere' -- is one Fastly does not record (updated_at is not bumped by an upsert; verified live) and that we cannot safely synthesise. Only the OPERATOR knows their deploy history. So reclamation becomes an explicit `config gc`: - new Adapter::gc_config_entries (default: unimplemented) + Fastly impl - one `config-store-entry list`; roots are classified via their pointers, so live chunks are known exactly - deletes only unreferenced chunk entries older than the operator's --older-than - FAILS CLOSED: an unclassifiable root or an unreadable created_at aborts with nothing deleted - a dry-run prints every key and age it would delete, so the operator's assertion is reviewable The local path keeps eager pruning (one file, no POPs) and is unchanged.
…ossible Resolves the spec's self-contradiction (it claimed pre-existing leaks were both reclaimed and never reclaimed) and makes it the single authoritative contract. Records all three rejected automatic designs (eager delete; metadata sidecar; store-derived clock) with the concrete counterexample that killed each, and the live-API findings behind them -- notably that Fastly does not bump updated_at on an upsert, and that chunk created_at is not a pointer-supersession time. States the conclusion plainly: the fact needed to delete safely is one the platform does not record and we cannot synthesise, so the operator supplies it via `config gc --older-than`. Plan banner updated to match.
|
You were right a fourth time, and this one settles it. I've stopped trying to make cloud reclamation automatic. Your P1 #1 kills the premise
The conclusion I should have reached three rounds agoTo delete safely you must know the pointer that referenced these chunks has stopped being served everywhere. Fastly:
Even embedding a Safe automatic cloud reclamation is not achievable with Fastly's primitives. So: What shipped
New Spec/plan (your P2 #6): the self-contradiction is fixed; the spec is now the single authoritative contract and records all three rejected designs with the counterexample that killed each, so a fourth isn't attempted. Gates: 115 fastly tests, workspace tests, fmt, |
# Conflicts: # crates/edgezero-adapter-fastly/src/cli.rs
Adds `edgezero config gc --adapter <name> [--older-than <dur>] [--yes]`, dispatching to Adapter::gc_config_entries. Runs IN-BAND in the bundled binary (and app-demo) -- unlike push/diff it needs no typed app-config, only the store's own contents -- via manifest-only resolution (no <app>.toml required). Safe by default: without --yes it is a dry-run that lists every key + age it would delete. --older-than accepts 7d/24h/90m/30s or bare seconds (default 7d) and is the operator's safety assertion; parse_duration_secs rejects garbage so a destructive command never guesses its threshold. Verified against the live store: correctly identifies the 3 live chunks from the root pointer and, at --older-than 1h, would reclaim exactly the 33 orphaned entries (11 dead generations) while deleting nothing without --yes.
Update: merged
|
3 participants
Summary
Fastly
config pushchunked storage was upsert-only, so re-pushing changed app config leaked the previous generation of chunk entries: chunk keys are content-addressed by the envelope SHA, so a config change rewrites every chunk key and orphans the old set. This affected both the cloud path (remote Config Store) and the local path (fastly.toml[local_server.config_stores.<name>.contents]).This PR adds best-effort chunk garbage collection on re-push — reclaiming the prior generation the moment a new pointer supersedes it — plus the design spec and TDD implementation plan.
Concurrency: last-writer-wins. Concurrent cloud pushes are supported (the last root-pointer write wins on the value). A push reclaims prior chunks only while a post-commit read-back confirms it is still the last writer of that root; otherwise it yields and deletes nothing, so a superseded push never removes the winner's live chunks. Best-effort, not transactional (Fastly has no compare-and-delete) — the residual is documented, bounded to one store's chunk data, and surfaces as a read-time integrity error rather than wrong data. See the spec's "Concurrency model: last-writer-wins".
Changes
docs/superpowers/specs/2026-07-07-fastly-chunk-gc.mddocs/superpowers/plans/2026-07-07-fastly-chunk-gc.mdcrates/edgezero-adapter-fastly/src/chunked_config.rsprior_chunk_keys(root_key, raw)— Value-first, v1-validated, prefix-scoped extraction of a prior pointer's chunk keys + unit testscrates/edgezero-adapter-fastly/src/cli.rsFastlyConfigGcPlan,expand_root,orphan_chunk_keys,reject_reserved_root_keys,local_contents_table); local prune in the samefastly.tomlrewrite + best-effort dry-run counts; cloud post-commit sweep with last-writer-wins read-back guard +delete_config_store_entry(--key --auto-yes, never--all) + offline dry-run intent; reserved-key rejection at the adapter boundary; ~30 new testscrates/edgezero-adapter-fastly/Cargo.toml,Cargo.lockhandlebarsdev-dependency (renders the cloud fake-fastlytest shim)Behavior details:
.__edgezero_chunks.) are a hard error at the adapter boundary — they'd collide with the chunk namespace.--dry-runstays offline (cloud reports GC intent without a count; local reports an exact count, and classifies malformed prior state asunknown: could not read prior staterather than0).config gc.Closes
Closes #313
Test plan
cargo test -p edgezero-adapter-fastly --features cli— 112 passedcargo test --workspace --all-targets— all passcargo clippy --workspace --all-targets --all-features -- -D warnings— cleancargo fmt --all -- --check— cleancargo check --workspace --all-targets --features "fastly cloudflare spin"— cleancargo check -p edgezero-adapter-spin --target wasm32-wasip2 --features spin— cleanCoverage highlights: prior-pointer validation (valid/direct/garbage/wrong-kind/bad-version/foreign-prefix); reserved-key rejection (local + cloud); local prune / shrink-to-direct / suspicious-pointer-skip / sibling-chunk preservation; local dry-run counts (exact / identical-repush-zero / non-table-unknown / suspicious-unknown); cloud deletes-prior-keeps-new / read-back concurrency skip / identical-repush-no-deletes / no-prior / delete-failure-warns / prior-read-failure-warns / shrink-to-direct / delete argv asserts
--key+--auto-yesand never--all.Checklist
edgezero_core(n/a — adapter-internal code)