feat(wiki): cap hub-page write amplification during ingest - #2489
Open
oscarlius wants to merge 2 commits into
Open
feat(wiki): cap hub-page write amplification during ingest#2489oscarlius wants to merge 2 commits into
oscarlius wants to merge 2 commits into
Conversation
Highly-referenced hub pages (entity/concept pages cited by many documents) get re-synthesized by the LLM on every ingest that touches them, and each rewrite re-tokenizes the whole body into the wiki_pages fulltext GIN index (title + content) — the dominant ingest write-amplification cost as a knowledge base grows. Add two opt-in WikiConfig knobs (0 = disabled, historical behaviour), resolved once per batch into WikiBatchContext: - max_page_content_bytes: when an existing page is already at/over the cap and the batch only ADDS information (no retractions), skip the LLM re-synthesis and persist only the bookkeeping columns (source/chunk refs) through the content-preserving UpdateMeta path, so the fulltext GIN is never touched. Retractions always regenerate the page — they shrink it. - max_refs: trim chunk_refs to the most-recent N entries, bounding the per-row JSONB/TOAST write size on hub pages whose chunk citations grow into the thousands.
Add max_page_content_bytes / max_refs to the WikiConfig definition in the generated swagger docs (surgical update matching swag v1.16.6 output; the committed docs carry unrelated drift from other features and are left untouched).
6 tasks
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
During wiki ingest, every batch that cites a page triggers an LLM re-synthesis of that page plus a content rewrite. For hub pages cited by many documents, the cost grows with every new upload: each add-only batch rewrites the full page body and forces the
fulltextGIN index (to_tsvectorovertitle || content, see migration000037) to rebuild, even when the existing synthesis already covers the topic.This PR adds two opt-in caps to
WikiConfig(both default to0= disabled, preserving current behavior):max_page_content_bytes— when a page’s existing content already meets/exceeds the cap and the current batch is add-only (no retractions), the reduce phase skips the LLM re-synthesis and keeps the existing body.UpdatePagethen detects “content unchanged” and routes to the metadata-onlyUpdateMetapath, so the fulltext GIN entry is never rebuilt. The batch’s bookkeeping (freshly-cited source/chunk refs) still lands for retrieval grounding and delete reconciliation. Retractions are never capped — they shrink the page and always regenerate it.max_refs— trimschunk_refsto the most recent N entries, stopping hub pages from accumulating thousands of refs and bloating the per-row JSON/TOAST write on every ingest.Both values are resolved once per batch in
newWikiBatchContextand exposed through the existingwiki_configJSON on the KB create/update APIs (swagger docs updated; the generated files were edited surgically forWikiConfigonly, since a fullmake docsregeneration currently produces unrelated drift — happy to run it if maintainers prefer).Type of Change
Related Issue
N/A
Testing
New
wiki_ingest_caps_test.gocovers: cap reached → re-synthesis skipped and content preserved; below cap → synthesis still runs; retraction bypasses the cap;max_refstrims to the tail;0disables both guards;capRecentStringArrayunit cases.go test ./internal/application/service/ -run "TestReduceSlugUpdates|TestCapRecentStringArray" -count=1— 6/6 passgo test ./internal/types/ -count=1— passesgo test ./internal/application/service/— only 3 pre-existing Windows SQLite temp-file-lock flakes fail, reproduced identically on an unmodifiedmaincheckoutgofmt -l— clean;git diff --check— passesgolangci-lint run --new-from-rev=upstream/main ./internal/...— 0 issuesChecklist
git diff --check origin/main...HEADpasses