ADR-009 stages 9.3-9.6: secret scrubbing, .graphlink archive, asset store, flat edge format - #304
Merged
Merged
Conversation
Adds the two stages that make data portable without making it leaky, plus the content-addressed asset store both of them need. 9.3 - backend/secret_scrub.py is the single chokepoint every surface that lets data leave this machine goes through. It scrubs on two axes, not one: the name of the field a value sits in, AND the shape of the value itself. A name-only filter is one refactor away from useless - the day a key is copied into a differently-named field, interpolated into an error string stored on a node, or pasted by the user into their own chat text, it waves it straight through. Absolute paths are redacted too: they are not credentials but they carry the operator's account name and private folder and file names. 9.4 - backend/workspace_archive.py reads and writes .graphlink files: a plain zip of manifest.json, one JSON file per chat, and assets as real files. Deliberately readable without this app, since the point is portability - someone should be able to unzip it, read their own conversations in a text editor, and recover their images without running anything. Import treats the archive as untrusted even when the user authored it. Every member name is validated before use, because ../../ in a zip entry is honoured by a naive extract and Python's zipfile will not stop you. Oversized members are refused rather than written, so a few KB of zip cannot fill the disk. Assets are verified against their own content hash on the way in and dropped if the bytes do not match the name. 9.5 (partial) - backend/asset_store.py, the content-addressed store. The ref is the SHA-256 of the bytes, so it doubles as an integrity check, identical images deduplicate, and re-storing unchanged bytes is a no-op rather than a rewrite. Writes are atomic and idempotent; nothing is ever deleted, since an asset is cheap to keep and catastrophic to remove while a chat still points at it. Test plan: full pytest from repo root - 2142 passed, 16 skipped. 28 new tests. The scrub suite is written as attempts to sneak a secret past it rather than as a happy path, and ends with a backstop that plants every fixture at once so narrowing any single rule fails there. The archive suite covers zip-slip, absolute-path entries, a newer format version, a non-zip file, a tampered asset, and a round-trip into a DIFFERENT empty asset store - the second-machine case, which a same-store test would pass while failing on the machine that actually matters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two on-disk format changes, both write-new/read-both so no stored row has to be rewritten and no chat saved by an earlier build stops loading. 9.5 - image bytes move out of the chat row. build_chat_data now takes an optional asset store; when one is present an image node serializes to a content-addressed ref instead of inline base64, and session_load reads either shape. Wired into all four production paths (autosave, save, load's digest baseline, eviction flush) via a single store_for(db_path) helper, so autosave stops rewriting megabytes for pictures that have not changed. A ref the store cannot resolve costs the picture, never the conversation. The store is published to the serializer through a contextvar rather than threading a parameter through ~20 same-signature dispatch lambdas; the save path is synchronous throughout, and the wrapper resets in a finally so a failed save cannot leak a stale store into the next one. 9.6 - a flat "edges" list becomes the authoritative edge record, and session_load prefers it over the 14-bucket legacy reconstruction. Those buckets encode a distinction this backend does not have (one connect() primitive, not 14 visual edge types), so save-side classification was a guess and load-side reconstruction had to unpick it through per-bucket index-vs-id fallbacks. Endpoints resolve against a map spanning nodes, notes and charts, since an edge may legitimately end on any of them. The legacy buckets are still WRITTEN, so files this build produces still open in older builds; only the read path is retired. Test plan: - New backend/tests/test_session_format_adr009.py (14 tests): asset round-trip through the store, dedupe across repeated saves, inline fallback for pre-9.5 rows, missing-ref degradation, contextvar reset after a failed save; flat-edge round-trip including note and chart endpoints, fallback when "edges" is absent, unresolvable and duplicate entries. - Full suite from repo root: 2156 passed, 16 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes ADR-009. Stages 9.1-9.2 (migrations, backups, corrupt-DB
recovery) landed in #303; this is the remaining four.
Problem
Nothing decided in one place what counts as a secret, so anything leaving
the app had to be trusted not to carry credentials or local paths. There
was no way to get data out at all. Chat rows carried image bytes inline as
base64, so every 30-second autosave rewrote megabytes for pictures that had
not changed. The on-disk edge format was a 14-bucket classification
inherited from a deleted Qt app, encoding distinctions this backend cannot
make — one
connect()primitive, not fourteen visual edge types.Change
9.3 — secret scrubbing. One chokepoint,
backend/secret_scrub.py,matching both on key name and on credential-shaped values, plus absolute
paths in all three forms (Windows, UNC, POSIX). Value-based matching is the
point: a credential pasted into a note under an unremarkable field name is
the realistic leak, not a tidily-named
api_key.9.4 — the
.graphlinkarchive. A plain zip: chats as JSON, assets asreal files, so it is readable without this app. Every payload is scrubbed
on the way out. Import validates before writing anything, refuses zip-slip
and absolute member names, caps member size against decompression bombs,
refuses newer format versions rather than guessing, and drops assets whose
bytes do not hash to their own ref.
9.5 — asset externalization.
build_chat_datatakes an optional assetstore; with one present an image node serializes to a content-addressed ref
instead of inline base64, and
session_loadreads either shape. Wired intoall four production paths (autosave, save, the load-time digest baseline,
eviction flush) through one
store_for(db_path)helper — all four, becausea mismatch between them would make the first tick after a load rewrite a
row that is already correct. A ref the store cannot resolve costs the
picture, never the conversation.
The store reaches the serializer through a contextvar rather than threading
a parameter through ~20 same-signature dispatch lambdas; the save path is
synchronous throughout, and the wrapper resets in a
finallyso a failedsave cannot leak a stale store into the next one.
9.6 — flat edge format. A flat
edgeslist becomes the authoritativeedge record and
session_loadprefers it, retiring the bucketreconstruction on the read path. Endpoints resolve against a map spanning
nodes, notes and charts, since an edge may legitimately end on any of them.
Both format changes are write-new/read-both: nothing already on disk is
rewritten, and files this build writes still open in older builds — the
legacy edge buckets are still emitted, only reading them is retired.
Test plan
backend/tests/test_secret_scrub.py(12) — written as attempts to sneaka secret past the scrubber rather than a happy-path walk: an unexpected
field name, a credential in prose, a path inside an error message, a
secret three containers deep, plus a backstop planting every fixture at
once. Fixtures are assembled from concatenated parts so no
credential-shaped literal exists in the source.
backend/tests/test_workspace_archive.py(16) — round-trip into adifferent asset store (the second-machine case, which a same-store test
would pass while failing on the machine that matters), zero secrets in
both the raw zip bytes and the parsed form, and the hostile-input set.
backend/tests/test_session_format_adr009.py(14) — asset round-trip anddedupe across repeated saves, inline fallback for pre-9.5 rows,
missing-ref degradation, contextvar reset after a failed save; flat-edge
round-trip including note and chart endpoints, fallback when
edgesisabsent, unresolvable and duplicate entries.