Skip to content

feat(wiki): cap hub-page write amplification during ingest - #2489

Open
oscarlius wants to merge 2 commits into
Tencent:mainfrom
oscarlius:pr/wiki-write-amplification-caps
Open

feat(wiki): cap hub-page write amplification during ingest#2489
oscarlius wants to merge 2 commits into
Tencent:mainfrom
oscarlius:pr/wiki-write-amplification-caps

Conversation

@oscarlius

Copy link
Copy Markdown
Contributor

Description

During wiki ingest, every batch that cites a page triggers an LLM re-synthesis of that page plus a content rewrite. For hub pages cited by many documents, the cost grows with every new upload: each add-only batch rewrites the full page body and forces the fulltext GIN index (to_tsvector over title || content, see migration 000037) to rebuild, even when the existing synthesis already covers the topic.

This PR adds two opt-in caps to WikiConfig (both default to 0 = disabled, preserving current behavior):

  • max_page_content_bytes — when a page’s existing content already meets/exceeds the cap and the current batch is add-only (no retractions), the reduce phase skips the LLM re-synthesis and keeps the existing body. UpdatePage then detects “content unchanged” and routes to the metadata-only UpdateMeta path, so the fulltext GIN entry is never rebuilt. The batch’s bookkeeping (freshly-cited source/chunk refs) still lands for retrieval grounding and delete reconciliation. Retractions are never capped — they shrink the page and always regenerate it.
  • max_refs — trims chunk_refs to the most recent N entries, stopping hub pages from accumulating thousands of refs and bloating the per-row JSON/TOAST write on every ingest.

Both values are resolved once per batch in newWikiBatchContext and exposed through the existing wiki_config JSON on the KB create/update APIs (swagger docs updated; the generated files were edited surgically for WikiConfig only, since a full make docs regeneration currently produces unrelated drift — happy to run it if maintainers prefer).

Type of Change

  • ✨ New feature
  • ⚡ Performance improvement

Related Issue

N/A

Testing

New wiki_ingest_caps_test.go covers: cap reached → re-synthesis skipped and content preserved; below cap → synthesis still runs; retraction bypasses the cap; max_refs trims to the tail; 0 disables both guards; capRecentStringArray unit cases.

  • go test ./internal/application/service/ -run "TestReduceSlugUpdates|TestCapRecentStringArray" -count=1 — 6/6 pass
  • go test ./internal/types/ -count=1 — passes
  • Full go test ./internal/application/service/ — only 3 pre-existing Windows SQLite temp-file-lock flakes fail, reproduced identically on an unmodified main checkout
  • gofmt -l — clean; git diff --check — passes
  • golangci-lint run --new-from-rev=upstream/main ./internal/... — 0 issues

Checklist

  • git diff --check origin/main...HEAD passes
  • Changed source files are formatted
  • Targeted tests for the changed packages/components pass
  • Diff-scoped lint passes where applicable
  • Full-repository checks were run, or any unrelated/environment-dependent failures are documented above
  • Self-reviewed the code
  • Added/updated tests covering the change
  • Updated related documentation (swagger; field comments document the semantics)
  • Breaking changes are clearly called out in the description above

Highly-referenced hub pages (entity/concept pages cited by many
documents) get re-synthesized by the LLM on every ingest that touches
them, and each rewrite re-tokenizes the whole body into the wiki_pages
fulltext GIN index (title + content) — the dominant ingest
write-amplification cost as a knowledge base grows.

Add two opt-in WikiConfig knobs (0 = disabled, historical behaviour),
resolved once per batch into WikiBatchContext:

- max_page_content_bytes: when an existing page is already at/over the
  cap and the batch only ADDS information (no retractions), skip the
  LLM re-synthesis and persist only the bookkeeping columns
  (source/chunk refs) through the content-preserving UpdateMeta path,
  so the fulltext GIN is never touched. Retractions always regenerate
  the page — they shrink it.
- max_refs: trim chunk_refs to the most-recent N entries, bounding the
  per-row JSONB/TOAST write size on hub pages whose chunk citations
  grow into the thousands.
Add max_page_content_bytes / max_refs to the WikiConfig definition in
the generated swagger docs (surgical update matching swag v1.16.6
output; the committed docs carry unrelated drift from other features
and are left untouched).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant