Skip to content

ADR-009 stages 9.3-9.6: secret scrubbing, .graphlink archive, asset store, flat edge format - #304

Merged
dovvnloading merged 2 commits into
mainfrom
adr009b-asset-store-flat-edges
Aug 9, 2026
Merged

ADR-009 stages 9.3-9.6: secret scrubbing, .graphlink archive, asset store, flat edge format#304
dovvnloading merged 2 commits into
mainfrom
adr009b-asset-store-flat-edges

Conversation

@dovvnloading

Copy link
Copy Markdown
Owner

Completes ADR-009. Stages 9.1-9.2 (migrations, backups, corrupt-DB
recovery) landed in #303; this is the remaining four.

Problem

Nothing decided in one place what counts as a secret, so anything leaving
the app had to be trusted not to carry credentials or local paths. There
was no way to get data out at all. Chat rows carried image bytes inline as
base64, so every 30-second autosave rewrote megabytes for pictures that had
not changed. The on-disk edge format was a 14-bucket classification
inherited from a deleted Qt app, encoding distinctions this backend cannot
make — one connect() primitive, not fourteen visual edge types.

Change

9.3 — secret scrubbing. One chokepoint, backend/secret_scrub.py,
matching both on key name and on credential-shaped values, plus absolute
paths in all three forms (Windows, UNC, POSIX). Value-based matching is the
point: a credential pasted into a note under an unremarkable field name is
the realistic leak, not a tidily-named api_key.

9.4 — the .graphlink archive. A plain zip: chats as JSON, assets as
real files, so it is readable without this app. Every payload is scrubbed
on the way out. Import validates before writing anything, refuses zip-slip
and absolute member names, caps member size against decompression bombs,
refuses newer format versions rather than guessing, and drops assets whose
bytes do not hash to their own ref.

9.5 — asset externalization. build_chat_data takes an optional asset
store; with one present an image node serializes to a content-addressed ref
instead of inline base64, and session_load reads either shape. Wired into
all four production paths (autosave, save, the load-time digest baseline,
eviction flush) through one store_for(db_path) helper — all four, because
a mismatch between them would make the first tick after a load rewrite a
row that is already correct. A ref the store cannot resolve costs the
picture, never the conversation.

The store reaches the serializer through a contextvar rather than threading
a parameter through ~20 same-signature dispatch lambdas; the save path is
synchronous throughout, and the wrapper resets in a finally so a failed
save cannot leak a stale store into the next one.

9.6 — flat edge format. A flat edges list becomes the authoritative
edge record and session_load prefers it, retiring the bucket
reconstruction on the read path. Endpoints resolve against a map spanning
nodes, notes and charts, since an edge may legitimately end on any of them.

Both format changes are write-new/read-both: nothing already on disk is
rewritten, and files this build writes still open in older builds — the
legacy edge buckets are still emitted, only reading them is retired.

Test plan

  • backend/tests/test_secret_scrub.py (12) — written as attempts to sneak
    a secret past the scrubber rather than a happy-path walk: an unexpected
    field name, a credential in prose, a path inside an error message, a
    secret three containers deep, plus a backstop planting every fixture at
    once. Fixtures are assembled from concatenated parts so no
    credential-shaped literal exists in the source.
  • backend/tests/test_workspace_archive.py (16) — round-trip into a
    different asset store (the second-machine case, which a same-store test
    would pass while failing on the machine that matters), zero secrets in
    both the raw zip bytes and the parsed form, and the hostile-input set.
  • backend/tests/test_session_format_adr009.py (14) — asset round-trip and
    dedupe across repeated saves, inline fallback for pre-9.5 rows,
    missing-ref degradation, contextvar reset after a failed save; flat-edge
    round-trip including note and chart endpoints, fallback when edges is
    absent, unresolvable and duplicate entries.
  • Full suite from repo root: 2156 passed, 16 skipped.

dovvnloading and others added 2 commits August 9, 2026 11:09
Adds the two stages that make data portable without making it leaky, plus
the content-addressed asset store both of them need.

9.3 - backend/secret_scrub.py is the single chokepoint every surface that
lets data leave this machine goes through. It scrubs on two axes, not one:
the name of the field a value sits in, AND the shape of the value itself.
A name-only filter is one refactor away from useless - the day a key is
copied into a differently-named field, interpolated into an error string
stored on a node, or pasted by the user into their own chat text, it waves
it straight through. Absolute paths are redacted too: they are not
credentials but they carry the operator's account name and private folder
and file names.

9.4 - backend/workspace_archive.py reads and writes .graphlink files: a
plain zip of manifest.json, one JSON file per chat, and assets as real
files. Deliberately readable without this app, since the point is
portability - someone should be able to unzip it, read their own
conversations in a text editor, and recover their images without running
anything.

Import treats the archive as untrusted even when the user authored it.
Every member name is validated before use, because ../../ in a zip entry
is honoured by a naive extract and Python's zipfile will not stop you.
Oversized members are refused rather than written, so a few KB of zip
cannot fill the disk. Assets are verified against their own content hash
on the way in and dropped if the bytes do not match the name.

9.5 (partial) - backend/asset_store.py, the content-addressed store. The
ref is the SHA-256 of the bytes, so it doubles as an integrity check,
identical images deduplicate, and re-storing unchanged bytes is a no-op
rather than a rewrite. Writes are atomic and idempotent; nothing is ever
deleted, since an asset is cheap to keep and catastrophic to remove while
a chat still points at it.

Test plan: full pytest from repo root - 2142 passed, 16 skipped. 28 new
tests. The scrub suite is written as attempts to sneak a secret past it
rather than as a happy path, and ends with a backstop that plants every
fixture at once so narrowing any single rule fails there. The archive
suite covers zip-slip, absolute-path entries, a newer format version, a
non-zip file, a tampered asset, and a round-trip into a DIFFERENT empty
asset store - the second-machine case, which a same-store test would pass
while failing on the machine that actually matters.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two on-disk format changes, both write-new/read-both so no stored row has
to be rewritten and no chat saved by an earlier build stops loading.

9.5 - image bytes move out of the chat row. build_chat_data now takes an
optional asset store; when one is present an image node serializes to a
content-addressed ref instead of inline base64, and session_load reads
either shape. Wired into all four production paths (autosave, save,
load's digest baseline, eviction flush) via a single store_for(db_path)
helper, so autosave stops rewriting megabytes for pictures that have not
changed. A ref the store cannot resolve costs the picture, never the
conversation.

The store is published to the serializer through a contextvar rather than
threading a parameter through ~20 same-signature dispatch lambdas; the
save path is synchronous throughout, and the wrapper resets in a finally
so a failed save cannot leak a stale store into the next one.

9.6 - a flat "edges" list becomes the authoritative edge record, and
session_load prefers it over the 14-bucket legacy reconstruction. Those
buckets encode a distinction this backend does not have (one connect()
primitive, not 14 visual edge types), so save-side classification was a
guess and load-side reconstruction had to unpick it through per-bucket
index-vs-id fallbacks. Endpoints resolve against a map spanning nodes,
notes and charts, since an edge may legitimately end on any of them. The
legacy buckets are still WRITTEN, so files this build produces still open
in older builds; only the read path is retired.

Test plan:
- New backend/tests/test_session_format_adr009.py (14 tests): asset
  round-trip through the store, dedupe across repeated saves, inline
  fallback for pre-9.5 rows, missing-ref degradation, contextvar reset
  after a failed save; flat-edge round-trip including note and chart
  endpoints, fallback when "edges" is absent, unresolvable and duplicate
  entries.
- Full suite from repo root: 2156 passed, 16 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dovvnloading
dovvnloading merged commit 5d1ac50 into main Aug 9, 2026
3 checks passed
@dovvnloading
dovvnloading deleted the adr009b-asset-store-flat-edges branch August 9, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant