feat: derived-view cache for large-doc agent reads (8-40x) - #433
Open
an1va wants to merge 4 commits into
Open
Conversation
…c reads The measured problem (local prototype, real content): on a 1-4MB HTML artifact, every search recomputes sectionMarkers (up to ~83ms) and every markdown/text read reconverts the whole doc (up to ~41ms) — 42-215ms of derived-view CPU per agent session, repeated every session, while docs under ~150K chars cost 1-12ms total and need nothing. The fix is the lazy read-through cache this codebase's own conventions make nearly free: one derived_view table (source_sha PK -> blob_key, auto-generated DDL in all three dialects), one helper (lib/derived-cache.ts) that computes all five views (elided markdown, text, outline, landmarks, section markers) ONCE on the first read of a large HTML doc and stores them as a single content-addressed blob. Content addressing IS the invalidation: a new version is a new sha, so there is no TTL, no staleness state, no write-path hook, and no eviction logic. Small docs and non-HTML skip the cache entirely and take the byte-identical direct path. Wired into every hot read surface, lazily (a format:"html" window or raw byte-exact read never pays a cache fill): the MCP read tool's full-doc present/outline/landmark-map paths, lib/search.ts's section markers + text-scope conversion (single-file branch; bundle pages stay direct), and the REST /content route's outline + full-doc format reads. Cached markers substitute ONLY where the direct path computed source markers — an HTML text-scope search keeps its empty marker list, preserving the search->read coordinate alignment exactly. Every cache interaction is best-effort: a failed store read falls back to computing, a failed write logs and still returns views, a corrupt cached blob falls back to computing. Substitution transparency is the tested contract: cached and direct results must be byte-identical. Tests: store-contract block (round-trip, last-write-wins on sha race, distinct-sha isolation) on sqlite + d1 + pg; helper behavior suite (size/content-type gate, poisoned-blob proof that hits really come from the cache, both best-effort directions, corrupt-blob fallback); an end-to-end MCP test (first search persists the row, second is byte-identical, outline read reuses it, an edit creates a fresh sha). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bove/below Production-readiness pass on the derived-view cache, from a full review of the branch. Two hardenings, no behavior change on the hit path: - Persist off the read response. `derivedViewsFor` now takes an optional `background` dep (waitUntil on the Workers edge, inline-await on Node), so a cache MISS no longer holds a GET open for the R2 put + D1 write. Wired through ctx.derived; unit tests cover both the routed and inline paths and prove the persist promise never rejects (safe to fire-and-forget on the edge). - Cache only the two EXPENSIVE views (markdown + source markers) and gate on an upper size bound. Reads/searches fetch just the view they need — the cheap views (text, outline, landmarks, ~6ms even at 4MB) are computed inline, so a `?outline=1` read no longer drags in the whole multi-MB blob. Above DERIVED_CACHE_MAX_CHARS (8M) the cache is skipped entirely: a miss computes + JSON-serializes both views (a few-x peak-memory multiple), workspace search runs up to 4 concurrently, and uploads reach 100MB — an unbounded gate could exhaust a 128MB edge isolate. Giant docs take the direct single-view path, exactly as before this cache existed. Callers updated: mcp read, workspace + in-artifact search, REST content route. All gates green (typecheck all tiers, 958 api + 576 package tests, Biome). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e, shape validation Findings from a second adversarial review of the cache at 329839c: - Version the cache key (the real catch). Content addressing invalidates on CONTENT change but not on CODE change — and the cached shape already mutated once inside this branch. Rows are now keyed `dv1:<sha>`; DERIVED_CACHE_GEN must be bumped whenever toMarkdown/sectionMarkers/elideDataUris or the DerivedViews shape changes semantically, so a shipped view-code fix can never silently keep serving the previous algorithm's output (there is no TTL to age it out). Old-generation rows simply orphan. - Gate the upper bound in BYTES, not UTF-16 chars. The memory the gate protects is spent in bytes; a chars gate undercounts CJK 3x. The gate now reads byteLength off the encode the sha already needs, and the ceiling drops to 5MB so 4 concurrent workspace-search misses stay well inside a 128MB isolate (worst-case concurrent delta ~60-80MB, previously ~96-160MB+ at 8M chars). - Validate the blob shape on read instead of casting: a corrupt-but-valid-JSON payload reads as a miss (recompute + self-repairing overwrite), never flows a number into .split() or a non-array into annotateSections. Plus the review's smaller items: stale five-view comments fixed on the two cross-package contract surfaces (ports.ts, schema.ts — comments only, no DDL change), vestigial awaits dropped from the now-sync outlineDoc/landmarksDoc, and new tests pinning the generation key, the bytes-vs-chars gate, wrong-shape fallback + repair, and the REST format=markdown cache branch on a >150K doc. All gates green: typecheck all tiers, 962 api + 560 package tests, Biome. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new content-params cache test counted derived_view rows with raw SQL against the shared app's sqlite file — but the pg CI job backs makeStore with Postgres, so better-sqlite3 opened a fresh empty db and threw "no such table: derived_view" (the local sqlite run passed; pnpm test:pg reproduced it exactly). Assert through the shared meta store itself instead, via the exact generation-prefixed cache key — which runs on both dialects AND pins the `dv1:<sha>` key format on the REST path. Also reordered for strength: raw + outline reads are checked FIRST (they must never fill the cache), then the markdown miss persists, then the hit is byte-identical. Verified: pnpm test:pg content-params.test.ts (real Postgres container) and the sqlite suite both green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #430 (1 of 3), carrying all its review hardenings.
Content-addressed, lazy cache for the two expensive derived views of large HTML docs (markdown conversion + search section markers). Agents re-reading the same big artifact via MCP/REST get ~16x faster reads (4.9ms vs 80ms on a 4MB doc, measured); small docs and misses take the identical direct path, byte for byte.
dv1:<sha>): bumpDERIVED_CACHE_GENwhen view code changes so rows can never silently serve a stale algorithm.waitUntilon the edge (inline on Node); the persist promise never rejects.Two adversarial reviews passed over this (one on the initial shape, one on the final); all findings addressed. Tests pin substitution transparency, both gates, generation keys, wrong-shape repair, background routing, and the REST cache branch, on both sqlite and Postgres (
pnpm test:pg).🤖 Generated with Claude Code