LLM Wiki Framework

A filesystem-native framework for building LLM-maintained wikis from curated source material.

This repository instantiates the pattern from Andrej Karpathy's LLM Wiki idea: raw sources remain preserved, while an LLM incrementally maintains a structured, interlinked wiki that compounds over time.

Repository Shape

.
├── AGENTS.md
├── core/
│   ├── workflows/
│   ├── skills/
│   ├── schemas/
│   └── templates/
├── domains/
│   └── _template/
├── raw/
│   ├── inbox/
│   └── sources/
└── wiki/
    ├── index.md
    ├── log.md
    ├── authors/
    ├── entities/
    ├── concepts/
    ├── places/
    ├── dates/
    ├── synthesis/
    ├── sources/
    ├── claims/
    └── queries/

Layer Contract

raw/inbox/ is immutable intake. Put original material here by type, for example webpages/, words/, pdfs/, and spreadsheets/. Assets captured with a main material live under raw/inbox/assets/<main-material-name>/.

raw/sources/ is the faithful promoted source layer. Files here are rewritten from inbox material into publishable Markdown or HTML. They are not raw dumps, extraction logs, or agent work notes. They must be human-readable, information-only, publish-like wiki/reference pages that preserve all information, add no invented information, use UK English, correct spelling and grammar except inside quotes, fix mathematical notation, verify calculations, preserve tables accurately, and use verified links.

raw/sources/assets/ contains promoted assets with meaningful names derived from nearby text and usage context.

Every promoted Markdown or HTML source starts with metadata, then an assets inventory, then the main content. It ends with workflow verification so ingestion quality is auditable.

wiki/ is the LLM-maintained knowledge layer. It contains indexed synthesis, source notes, authors, entities, concepts, places, dates, claims, queries, contradictions, and cross-references.

domains/ contains optional domain packs. Each domain can extend vocabulary, workflows, page types, quality gates, and naming rules without changing the core framework.

core/ contains the reusable framework: workflows, skills, schemas, and templates.

Wiki Modes

The framework supports these ingestion modes:

• semantic-heavy: the default mode for concepts, research, analysis, narrative material, and synthesis.
• identifier-heavy: for procedural documentation, runbooks, API references, configuration guides, policy/legal text, commands, IDs, error codes, endpoint names, product labels, and other token-sensitive material.
• temporal-heavy: for timelines, incidents, release histories, regulations, project history, events, and supersession-sensitive material.
• evidence-heavy: for legal, compliance, finance, medicine, due diligence, audit, and other claim-by-claim provenance material.
• decision-heavy: for architecture decisions, business decisions, product trade-offs, and policy choices.
• procedural-heavy: for solutions, runbooks, ordered steps, troubleshooting flows, recipes, checklists, and operational instructions.
• reference-heavy: for APIs, schemas, taxonomies, catalogues, configuration keys, compatibility matrices, and lookup material.
• narrative-heavy: for books, interviews, transcripts, histories, meetings, and interpretation-sensitive sources.
• quantitative-heavy: for spreadsheets, KPIs, scientific data, financial reports, tables, charts, formulas, and calculations.
• visual-heavy: for diagrams, screenshots, slide decks, maps, architecture drawings, and figure-led sources.

For procedural material, ingestion must preserve source step order, prerequisites, warnings, commands, validation, rollback, and expected outcomes. If multiple sources conflict, the conflict must be surfaced for review instead of silently merged into one procedure.

Every wiki page includes cross-cutting retrieval metadata: freshness, confidence, source priority, applicability, supersession links, review status, retrieval terms, semantic summary, and whether exact source evidence is required.

Canonical Flow

1. Acquire or extract original material into raw/inbox/<type>/ using a core intake skill when useful.
2. Add related captured assets to raw/inbox/assets/<main-material-name>/.
3. Run core/skills/source-faithful-rewrite to promote the material into raw/sources/.
4. Run core/skills/wiki-layer-ingest to update the wiki layers from the promoted source, with retrieval quality as an explicit target.
5. Update wiki/index.md and append to wiki/log.md.
6. Complete ingestion verification and record the result in the promoted source and wiki/log.md.
7. Periodically run core/skills/wiki-lint to catch drift, stale claims, weak citations, missing pages, and orphaned knowledge.

Core intake skills include:

• fetch-webpage for fetching a webpage URL and all usable local assets.
• extract-word for extracting Word documents.
• extract-pdf for extracting PDFs.
• extract-spreadsheet for extracting Excel or other spreadsheet files.
• extract-presentation for extracting PowerPoint or other slide decks.

App Integration Model

This repository is not tied to VS Code, Codex, or any editor. It is a portable filesystem and Git contract.

The recommended split is:

• authoring agents with tools build and maintain the repo;
• a Java + PostgreSQL web app syncs, indexes, retrieves, and serves the repo;
• simple no-tool AI models receive prepared retrieval context from the Java app.

The retrieval stack is designed to run locally, meaning on the local machine or an internal/on-premises network rather than through a cloud agent provider. The embedding model, reranker, and answer model are reached over localhost or internal-network endpoints, and no content, query, or embedding leaves the network.

For larger repositories, the Java app should use QMD-style retrieval: domain routing, query expansion into lexical and semantic subqueries, parallel full-text and vector search, Reciprocal Rank Fusion, reranking, linked-page expansion, and final context packing before calling the model API.

The ingestion workflow should optimise the repository for that retrieval layer. Every promoted source, wiki page, domain rule, link, claim, and query page should make it easier for Java + PostgreSQL to retrieve precise context with embeddings and full-text search before calling a simple model.

Because the retrieval layer may not have tools, ingestion must maximise retrieval quality up front: clear snippets, aliases, exact terms, semantic descriptions, source-evidence links, chunkable headings, domain routing rules, and reusable wiki/queries.

The retrieval app should store searchable source and wiki content in PostgreSQL, while keeping display assets addressable in the filesystem or object storage. Asset paths and wiki links must be parsed and indexed so interlinks render correctly in the web app.

Any retrieval-to-agent optimisation loop must be finite. It should have bounded attempts, clear stop states, and a NO_RESULT outcome when the app cannot find enough evidence or cannot improve the result.

The retrieval app can also optimise future ingestion by finding duplicate sources, related wiki pages, relevant domain rules, stale claims, and candidate wiki/queries updates before an authoring agent edits the repository.

See docs/integrations/java-postgresql-retrieval.md.

Important Naming Note

This framework uses raw/inbox/ as the canonical intake path. If a note, conversation, or external tool says raw/income/, treat that as an alias or typo and move the material into raw/inbox/.