Refactor/converge cold build - #90
Merged
Merged
Conversation
`dexter init` and the LSP server both build an index from nothing, and each had its own implementation. init parses on every core and writes one bulk transaction with the indexes dropped; the server walked serially, parsed one file at a time, and committed a transaction per file against live indexes. On a 10k-file corpus the server's path took 4.441s against the shared pipeline's 1.088s, for identical rows. Move init's pipeline into internal/indexer and call it from both. The server reaches it only under IsEmpty(), which is where its own build already lived. Three things do not transfer from `init --force`, and this is why the change is a shared pipeline rather than a shared function call: - Deleting the database files and reopening. The server holds the store and handlers read that pointer unsynchronised. It is also pointless: an empty index has nothing stale to throw away. - SetBulkPragmas. Leaving WAL needs exclusive access, so journal_mode fails whenever another connection is open. Options.InProcess suppresses all four. They were the smallest part of the win; the parse pool and the dropped indexes are the rest of it. - Insert-only mode's assumptions. It skips the per-file DELETE and allocates file ids from a counter seeded when the batch opens, so a save landing mid-build would duplicate rows or collide on a primary key. The server now holds indexWrites for writing across a full build, and every single-file write — save, watched file, rename — holds it for reading. Incremental sweeps are unaffected: they use the per-file path and take no lock. Two fixes fall out of it: - The prune step deletes every stored path the walk did not see. A full build populates no `seen` set, so running the prune after one would delete the index that was just written. It is now inside the incremental branch only. - SetBulkPragmas applied synchronous before journal_mode, so a locked database failed after disabling fsync and left it disabled for the life of the process. Harmless in a CLI that exits; not harmless in a server. journal_mode goes first, so a failure changes nothing. The LSP path also records IndexVersion now, which only init used to do.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dexter initand the LSP server both build an index from nothing, and each had its own implementation. init parses on every core and writes one bulk transaction with the indexes dropped; the server walked serially, parsed one file at a time, and committed a transaction per file against live indexes. On a 10k-file corpus the server's path took 4.441s against the shared pipeline's 1.088s, for identical rows.Move init's pipeline into internal/indexer and call it from both. The server reaches it only under IsEmpty(), which is where its own build already lived.
Three things do not transfer from
init --force, and this is why the change is a shared pipeline rather than a shared function call:Two fixes fall out of it:
seenset, so running the prune after one would delete the index that was just written. It is now inside the incremental branch only.The LSP path also records IndexVersion now, which only init used to do.