Perf: Index improvements - #89
Merged
Merged
Conversation
The cold index is bound by the one SQLite writer, not by parsing: on a 70k-file monorepo the parse workers use ~12s of CPU across 15 cores while the writer needs ~4.6s, so the pipeline runs at the speed of one core. These four changes remove work from the writer, or delete it outright. None of them changes a query result. - Dedupe refs in the parser. Refs are line-granular, so a @SPEC repeating `String.t()` writes one identical row per occurrence — 59,792 of them on tiger. Identical rows are invisible to every query (nothing counts refs, and the References handler dedupes by file+line), and dropping them in the parse workers costs CPU that phase already has spare. - Batch the bulk-path inserts into multi-row INSERTs of 900 bound parameters, under even the legacy SQLITE_MAX_VARIABLE_NUMBER of 999. Incremental reindex keeps the row-at-a-time path, where a file's DELETE must stay ordered ahead of its INSERTs. - Retire idx_refs_function_kind. No query leads with `function`; the two that filter on function/kind both lead with file_path and use idx_refs_file_path. It cost 80 MB and a share of every index rebuild. - Replace `module LIKE 'Prefix.%'` with a range in the two prefix lookups. LIKE is case-insensitive by default, so SQLite could not use the index and scanned all 3.9M refs. Elixir module names are case-sensitive, so the range is also the stricter, more correct comparison. Stat the walk's files in parallel too: the traversal is cheap but DirEntry.Info() is one lstat per file, ~70k serialized syscalls. Measured on a large codebase (70k files, 480k definitions): cold index 12.153s -> 10.013s walk 1.758s -> 1.351s write 4.569s -> 4.073s create indices 4.972s -> 3.878s database 1595 MB -> 1497 MB prefix lookup 5.360s -> 0.099s (54x, module rename path) Definitions are identical and refs fall by exactly the number of duplicates (60k). The parse phase costs ~2s more CPU for the dedupe, which is free: it still finishes in ~1s of wall time against a 4.2s pipeline.
The chunk boundary is an exact-multiple test, so an Exec that returned an error left the buffer full at 900 entries and the check never matched again: the rest of the batch grew the buffer without bound and wrote its rows one at a time through flushPending at commit. Nothing reachable triggers this today. definitions and refs carry no UNIQUE constraint, every bound value comes from a Go struct field so the NOT NULL columns cannot fire, and the foreign key to files(path) is satisfied by construction - the parser sets FilePath to the walked path, and indexFile writes the files row before it buffers anything. Only a SQLite-level failure (SQLITE_FULL, IOERR, CORRUPT) can fail a row, and those doom the run anyway. The reset is here because the assumption is invisible in the code and one schema change from being wrong: a UNIQUE index on refs would make row-level conflicts reachable. Also from review: - statFilesParallel's comment claimed order is not preserved. The implementation writes to out[i] and compacts in index order, so it is. - docs/architecture.md said the prefix range is 54x faster, which compared a cold LIKE against a warm range. Measured on tiger's 3.9M refs: 11-14x with a warm page cache, ~190x cold. - Delete cmd/benchprefix, which its own comment marked temporary.
The index carried the absolute file path on every row: 122 characters repeated across 3.9M refs on a 70k-file monorepo, plus a second copy inside idx_refs_file_path. Together that was over half the database, and since both the single writer and CREATE INDEX are bound by bytes moved, it was also most of the cold index. - Intern paths. files gains an INTEGER PRIMARY KEY; definitions and refs carry file_id. The bulk path allocates ids from a counter and writes them explicitly, so a cold index never pays a round trip per file. The incremental path upserts, keeping the id that a file's rows already point at — INSERT OR REPLACE would hand out a new one and detach them. file_path never left store.go, so no caller changes. - Cover idx_refs_module_function with (module, function, file_id, line, kind). References is answered from the index alone; before, every hit cost a random read into the 653 MB refs table — 7,749 of them for one hot function. Ordering moved to Go, which drops the temp B-tree SQLite built on every call. - Drop the foreign key on refs. With _foreign_keys=ON it cost a parent-key lookup on each of 3.9M inserts, and every path that removes a file already deletes its refs explicitly. - Add a partial index for the __using__ rows. LookupUsingModules runs on the References slow path and was scanning all 481,563 definition entries, because idx_definitions_module_function leads with module and cannot be seeked by function. The warm slow path drops from ~30-65ms to ~1-2ms. - Walk directories in parallel and unsorted. The traversal was still single-threaded after the stat work was fanned out. Measured on a 70,785-file monorepo (481,563 definitions, 3,870,314 refs): cold index 10.013s -> 7.900s walk 1.351s -> 0.630s write 4.073s -> 3.303s create indices 3.878s -> 3.672s database 1497 MiB -> 516 MiB refs 653 -> 181 MB, and the 507 MB idx_refs_file_path becomes a 42 MB idx_refs_file_id. Verified against the previous build with 246 probes across 90 files of a real monorepo — 472,715 reference locations — byte-identical for references, definition and hover. IndexVersion 13: the schema change makes an existing index unreadable, so migrate drops the pre-file_id tables and the server rebuilds.
Assertions about references, definition and hover only mean something end to end: the handlers reach the store, the tokenizer, tree-sitter and the __using__ cache, and a unit test on any one of those misses how they combine. Until now nothing drove the server over the wire, and checking a change against a real codebase automatically meant a throwaway script. internal/lsptest is the client. It returns errors and does not import testing, so the same code serves both callers; the T wrapper turns those errors into t.Fatal for tests. It handles server-initiated traffic - notifications dropped, requests answered - so a chatty server cannot wedge a caller, and every request is bounded by a timeout. cmd/lspprobe drives any project on disk, for on-the-spot checks and for diffing two builds against each other: lspprobe -root ~/code/app -method references 'lib/app/accounts.ex#get_user' lspprobe -binary ./a -root ~/p -json @probes.txt > a.json Cursors are file:line:col or file#needle, so a probe can name what to point at rather than a line number. It is a development tool and stays out of the shipped binary: make build compiles ./cmd only. Nothing about any particular codebase lives here. The integration tests cover what unit tests could not: references and definition through plain, as: and grouped aliases and through an import, and three use-chain cases - a function injected by __using__, the use sites of a __using__ macro, and the use sites of the module itself.
findModulesWhoseUsingImports, on the References slow path, consults every module in the codebase that defines defmacro __using__. Those parsed bodies are cached, but nothing filled the cache, so the first request that needed it paid for all of them at once: 472 file reads and parses, ~440ms, in front of a user waiting on a reference lookup. Every request after that cost 1-2ms. Warm the cache at the end of the background reindex instead, where the index is known to exist and nothing is waiting on it. The work is unchanged — it only stops landing on a request. Bounded to NumCPU readers. The existing scan spawns one goroutine per module, which is fine for work a user is waiting on, but a speculative warm-up should not storm the filesystem while the editor is starting. Entries are validated against file mtime when read, so a warmed entry that goes stale is re-parsed, and an unreadable file is skipped exactly as on the request path. If the reindex lock is already held the warm-up is skipped, and whichever reindex holds it warms instead. Measured on a 70k-file monorepo, first References needing the scan: before 374ms after 21ms (warm-up: ~500 modules in 383ms, in the background) lspprobe grows a -settle flag, without which a probe fires before background work finishes and cannot measure this honestly.
…ughput # Conflicts: # cmd/main.go # internal/parser/parser_test.go # internal/store/store.go # internal/store/store_test.go
JesseHerrick
force-pushed
the
perf/index-query-throughput
branch
from
September 7, 2026 01:44
06d82e5 to
e72317f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improves cold indexing and reference-query throughput, while adding an end-to-end LSP test harness and probe command for real-project validation.
What changed
files, referencing them via integerfile_id.__using__definitions.__using__cache after background indexing.usechains.lspprobefor testing and comparing LSP behavior on real projects.IndexVersionto 13 for the schema change.