Skip to content

Refactor/converge cold build - #90

Merged
JesseHerrick merged 3 commits into
mainfrom
refactor/converge-cold-build
Sep 7, 2026
Merged

Refactor/converge cold build#90
JesseHerrick merged 3 commits into
mainfrom
refactor/converge-cold-build

Conversation

@JesseHerrick

Copy link
Copy Markdown
Member

dexter init and the LSP server both build an index from nothing, and each had its own implementation. init parses on every core and writes one bulk transaction with the indexes dropped; the server walked serially, parsed one file at a time, and committed a transaction per file against live indexes. On a 10k-file corpus the server's path took 4.441s against the shared pipeline's 1.088s, for identical rows.

Move init's pipeline into internal/indexer and call it from both. The server reaches it only under IsEmpty(), which is where its own build already lived.

Three things do not transfer from init --force, and this is why the change is a shared pipeline rather than a shared function call:

  • Deleting the database files and reopening. The server holds the store and handlers read that pointer unsynchronised. It is also pointless: an empty index has nothing stale to throw away.
  • SetBulkPragmas. Leaving WAL needs exclusive access, so journal_mode fails whenever another connection is open. Options.InProcess suppresses all four. They were the smallest part of the win; the parse pool and the dropped indexes are the rest of it.
  • Insert-only mode's assumptions. It skips the per-file DELETE and allocates file ids from a counter seeded when the batch opens, so a save landing mid-build would duplicate rows or collide on a primary key. The server now holds indexWrites for writing across a full build, and every single-file write — save, watched file, rename — holds it for reading. Incremental sweeps are unaffected: they use the per-file path and take no lock.

Two fixes fall out of it:

  • The prune step deletes every stored path the walk did not see. A full build populates no seen set, so running the prune after one would delete the index that was just written. It is now inside the incremental branch only.
  • SetBulkPragmas applied synchronous before journal_mode, so a locked database failed after disabling fsync and left it disabled for the life of the process. Harmless in a CLI that exits; not harmless in a server. journal_mode goes first, so a failure changes nothing.

The LSP path also records IndexVersion now, which only init used to do.

`dexter init` and the LSP server both build an index from nothing, and each
had its own implementation. init parses on every core and writes one bulk
transaction with the indexes dropped; the server walked serially, parsed one
file at a time, and committed a transaction per file against live indexes.
On a 10k-file corpus the server's path took 4.441s against the shared
pipeline's 1.088s, for identical rows.

Move init's pipeline into internal/indexer and call it from both. The server
reaches it only under IsEmpty(), which is where its own build already lived.

Three things do not transfer from `init --force`, and this is why the change
is a shared pipeline rather than a shared function call:

- Deleting the database files and reopening. The server holds the store and
  handlers read that pointer unsynchronised. It is also pointless: an empty
  index has nothing stale to throw away.
- SetBulkPragmas. Leaving WAL needs exclusive access, so journal_mode fails
  whenever another connection is open. Options.InProcess suppresses all four.
  They were the smallest part of the win; the parse pool and the dropped
  indexes are the rest of it.
- Insert-only mode's assumptions. It skips the per-file DELETE and allocates
  file ids from a counter seeded when the batch opens, so a save landing
  mid-build would duplicate rows or collide on a primary key. The server now
  holds indexWrites for writing across a full build, and every single-file
  write — save, watched file, rename — holds it for reading. Incremental
  sweeps are unaffected: they use the per-file path and take no lock.

Two fixes fall out of it:

- The prune step deletes every stored path the walk did not see. A full build
  populates no `seen` set, so running the prune after one would delete the
  index that was just written. It is now inside the incremental branch only.
- SetBulkPragmas applied synchronous before journal_mode, so a locked database
  failed after disabling fsync and left it disabled for the life of the
  process. Harmless in a CLI that exits; not harmless in a server. journal_mode
  goes first, so a failure changes nothing.

The LSP path also records IndexVersion now, which only init used to do.
@JesseHerrick
JesseHerrick merged commit 0cff0ed into main Sep 7, 2026
4 checks passed
@JesseHerrick
JesseHerrick deleted the refactor/converge-cold-build branch September 7, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant