From 8df255d862330efda50c1d2809520c7902426247 Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Mon, 28 Sep 2026 17:41:55 -0400 Subject: [PATCH 01/16] Send generated functions to the line that declared them A function a macro generated has no source definition, so go-to-definition fell back to the top of its module. The compiled module's debug info records where each def came from: its :line, and a `file: {path, line}` entry left by `@file` or `quote location: :keep`. When that path is the module's own source, the line is where the code asked for the function; when it names the generator's own file, :line is used instead. beam.ReadDefinitionLines reads those locations from the Dbgi chunk and steps over clause bodies without allocating. generatedDefinitionResultsFor uses a recorded line only when the debug info was compiled from the file being opened, the BEAM is not older than it, and the line falls after the module's own line. Every other case keeps the module result. Bare-call definition and call-hierarchy preparation call it directly. Qualified definition, the references declaration, and `dexter lookup` reach it through LookupName, which takes a recorded line even for a strict lookup, since that line is the function's own definition. No framework is recognized by name. Ash code interfaces (#108) resolve to their `define` line once Ash includes ash-project/ash#2971, which expands each generated def at that line; with released Ash they resolve to line 1 as before. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 + docs/architecture.md | 12 +- internal/beam/debug_info.go | 336 ++++++++++++++++++++ internal/beam/debug_info_test.go | 369 ++++++++++++++++++++++ internal/beam/docs_test.go | 45 +-- internal/beam/etf.go | 10 +- internal/lsp/beam_path_test.go | 16 +- internal/lsp/generated_completion.go | 95 ++++++ internal/lsp/generated_definition_test.go | 315 ++++++++++++++++++ internal/lsp/name_navigation.go | 10 +- internal/lsp/server.go | 14 +- 11 files changed, 1190 insertions(+), 34 deletions(-) create mode 100644 internal/beam/debug_info.go create mode 100644 internal/beam/debug_info_test.go create mode 100644 internal/lsp/generated_definition_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 038f46a..92993a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ ### Added +- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when the BEAM is not older than the source and was compiled from the file being opened; otherwise the module line remains the answer. Ash code interfaces go to their `define` line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) + - **`--root`/`-C` names the workspace on every command** — `dexter init --root ~/project`, `dexter lookup --root ~/project MyApp.Repo`, and `dexter lsp --root ~/project` all run as if they had been started in that directory, so an agent, a script, or an editor wrapper can index or query a project from anywhere. Relative paths, a `reindex` target included, resolve from the named root - **Watching and runtime locations recover without periodic reindexes** — a directory the kernel refuses to watch (an inotify watch limit on a large tree) is tracked instead of disabling the whole native watcher. Dexter reconciles once when coverage is lost, retries only failed registrations, and reconciles once when coverage returns; it does not run recurring full-tree passes that cause CPU spikes. Runtime files use the environment-independent `/tmp/dexter-` directory so a GUI editor and a shell cannot derive different ownership locks. Elixir and mix detection also searches the standard mise, asdf, and Homebrew locations and falls back to a login shell, so an editor that starts the daemon with a stripped PATH no longer disables stdlib indexing or formatting diff --git a/docs/architecture.md b/docs/architecture.md index e503e73..9c1286f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,7 +7,7 @@ Dexter is a fast Elixir LSP server. It indexes module and function definitions f - `cmd/main.go` — CLI entrypoint: `init`, `reindex`, `lookup`, `references`, `lsp`, `version`, and the hidden `daemon` subcommand. `lsp`, `lookup`, `references`, and `reindex` are frontends to the workspace daemon: `lsp` is a raw stdio proxy onto it. Only `init` still opens the store itself, under the same ownership lock the daemon takes, because it needs the workspace to itself. - `internal/indexer/` — the cold build: walk and stat on all cores, parse on all cores, then one bulk transaction with the indexes dropped. `dexter init` and the LSP server (when it finds an empty index) both call `FullBuild`. `Options.InProcess` marks the server, which shares the database with live readers and so cannot use the connection-wide bulk pragmas. - `internal/parser/` — Elixir parser backed by a hand-rolled tokenizer (`tokenizer.go`). The tokenizer produces a flat token stream (handling heredocs, sigils, strings, comments as opaque tokens; the code inside a `#{}` interpolation is tokenized as well, into the separate `TokenResult.Interp` stream — see below) and `parser_tokenized.go` walks it to extract defmodule, def, defp, defmacro, defdelegate, defguard, defprotocol, defimpl, @type, @callback, alias, import, use, and Module.function references. Handles module nesting, alias resolution for defdelegate targets, and multi-line expressions natively via bracket depth tracking. -- `internal/beam/` — bounded, bounds-checked readers for the BEAM container, export table, Elixir Docs chunk, and persisted module attributes. These readers extract generated callables and DSL-provider metadata without starting the Erlang VM. +- `internal/beam/` — bounded, bounds-checked readers for the BEAM container, export table, Elixir Docs chunk, Elixir debug info, and persisted module attributes. These readers extract generated callables, their definition lines, and DSL-provider metadata without starting the Erlang VM. - `internal/store/` — SQLite layer. Tables: `files` (path + mtime), `definitions` (module, function, kind, line, file_path, delegate_to, delegate_as), `refs` (module, function, line, file_path, kind). - `internal/lsp/` — LSP server. `server.go` handles all LSP methods. `elixir.go` contains pure functions for cursor expression extraction, alias/import/use extraction (tokenizer-based), and use-chain parsing. `rename.go` has rename helpers. `hover.go` has hover formatting. `documents.go` is an in-memory open-buffer store. - `internal/workspace/` — the protocol-independent owner of one workspace: the store, the shared `lsp.IndexCoordinator`, the headless language-service instance, stdlib discovery, the native and Git watchers, and the single mutation queue every index change enters. `runtime.go` is the lifecycle and the queue; `watch.go` defines the recursive watcher abstraction, with an FSEvents backend on macOS and an fsnotify backend elsewhere or as fallback. @@ -137,6 +137,16 @@ The block walk is passed as a thunk and runs only after the compiled consumer re Generated-symbol resolution is shared by completion, hover, definition, signature help, references, and call-hierarchy preparation. Definition and call hierarchy cannot point at a source definition for a source-less provider, so they walk the provider's lexical module parents and use the closest module that has an indexed source location. Completion resolve and signature help read the same lazily cached BEAM documentation used by hover. +### Definition lines for generated functions + +A generated function has no source definition, but the compiled module's `Dbgi` chunk still records where each def came from. `beam.ReadDefinitionLines` walks `{:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}}` for the `file`, `relative_file`, and `definitions` keys, and steps over clause bodies without allocating. Clause ASTs nest far deeper than a Docs term, so this reader raises the ETF depth guard to `maxDebugInfoDepth`. + +Each def carries two locations. Its `:line` is the line in the module's own file that was being compiled when the def was produced: the macro call, or wherever a before-compile hook ran. A `file: {path, line}` entry, left by `@file` or `quote location: :keep`, is preferred when `path` is the module's own source, because that is where the code asked for the function. When `path` is another file it is the generator's implementation, and `:line` is used instead. Ash's code interfaces take the second shape: each generated def is expanded at its `define` line, so `:line` is that line and `file` still names `code_interface.ex`, which keeps stacktraces unchanged. No framework is recognized by name. + +`generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line can be trusted in the file being opened: the debug info must have been compiled from that file (absolute path, or relative path as a suffix for a moved or symlinked project), the BEAM must not be older than the file, and the line must fall after the module's own line. Any other case keeps the module result. Unlike generated *names*, which a stale BEAM may still contribute, a stale BEAM's *lines* may have moved, so here source-vs-BEAM mtime is a gate. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. + +The debug info is read on the first definition request that needs it and memoized on the module's generated-function cache entry, so the BEAM stamp invalidates it with everything else. A module compiled without debug info, or an Erlang module, is memoized as empty. + For references, the parser records bare injected calls under the direct `use` module because the generated provider is unavailable while source is indexed. At lookup time, generated-symbol resolution queries those injector rows and validates every candidate against the compiled provider active at that candidate's block path. This keeps same-named macros from another DSL or another section out of the result. Statement-level injected calls with arguments are indexed even when they omit both parentheses and a `do` block, as in `authorize_if always()`. ## References — injector scan diff --git a/internal/beam/debug_info.go b/internal/beam/debug_info.go new file mode 100644 index 0000000..ce08614 --- /dev/null +++ b/internal/beam/debug_info.go @@ -0,0 +1,336 @@ +package beam + +import ( + "errors" + "fmt" +) + +// maxDebugInfoDepth bounds recursion while stepping over clause bodies. Each +// level of Elixir AST costs two ETF levels (the call tuple and its argument +// list), so this allows a thousand nested expressions: far past any real code, +// and still a bounded Go stack on a corrupt file. +const maxDebugInfoDepth = 2048 + +// FunctionKey names one callable by name and arity. +type FunctionKey struct { + Name string + Arity int +} + +// DebugInfo is what ReadDefinitionLines extracts from an Elixir Dbgi chunk. +type DebugInfo struct { + // File is the absolute source path the module was compiled from, and + // RelativeFile the same path relative to the compiler's working directory. + // Either may be empty if the chunk omits it. + File string + RelativeFile string + + // Lines maps each public function and macro to its line in File. Entries + // without a usable line are absent. + Lines map[FunctionKey]int +} + +// ReadDefinitionLines reads the line of every public definition from an Elixir +// BEAM's debug info, for functions that have no source definition to index +// because a macro generated them. +// +// A generated def carries two locations. Its :line is the line in the module's +// own file that was being compiled when the def was produced: the macro call, +// or wherever a before-compile hook ran. A `file: {path, line}` entry, left by +// `@file` or by `quote location: :keep`, names a more specific place. When that +// path is the module's own source it is where the code asked for the function; +// when it is some other file it is the generator's own implementation, which is +// not what a reader of the module is looking for, so :line is used instead. +// +// Everything here is standard compiler output. No framework is recognized by +// name: a generator that wants its functions to navigate to the line that +// declared them either expands each def at that line, as Ash's code interfaces +// do, or stamps it with `@file {file, line}`. +func ReadDefinitionLines(path string) (DebugInfo, error) { + chunk, err := readChunk(path, "Dbgi") + if err != nil { + return DebugInfo{}, err + } + inflated, err := inflateDocsTerm(chunk) + if err != nil { + return DebugInfo{}, fmt.Errorf("decode Dbgi chunk: %w", err) + } + return parseDebugInfo(inflated) +} + +var errNoElixirDebugInfo = errors.New("no Elixir debug info") + +// definitionSite is one definition's raw locations, kept until the module's +// source path is known: small maps put definitions ahead of file and +// relative_file, so they cannot be resolved while walking. +type definitionSite struct { + key FunctionKey + line int + keepFile string + keepLine int +} + +// parseDebugInfo walks {:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}}. +// Only three keys of the map are read; clause bodies, which are nearly all of +// the chunk, are stepped over without allocating. +func parseDebugInfo(buf []byte) (info DebugInfo, err error) { + // Bounds-checked throughout, but as with Docs a reader mistake must not take + // down the server over a corrupt build artifact. + defer func() { + if recovered := recover(); recovered != nil { + info = DebugInfo{} + err = fmt.Errorf("invalid Dbgi term: %v", recovered) + } + }() + + r := &etfReader{buf: buf, maxDepth: maxDebugInfoDepth} + if arity, err := r.enterTuple(); err != nil { + return DebugInfo{}, err + } else if arity != 3 { + return DebugInfo{}, fmt.Errorf("debug_info tuple arity %d", arity) + } + if version, err := r.readAtom(); err != nil { + return DebugInfo{}, err + } else if version != "debug_info_v1" { + return DebugInfo{}, fmt.Errorf("unsupported debug info version %q", version) + } + if backend, err := r.readAtom(); err != nil { + return DebugInfo{}, err + } else if backend != "elixir_erl" { + return DebugInfo{}, fmt.Errorf("%w: backend %q", errNoElixirDebugInfo, backend) + } + // Compiling with `debug_info: false` leaves :none here. + if tag, err := r.peekTag(); err != nil { + return DebugInfo{}, err + } else if tag != tagSmallTuple && tag != tagLargeTuple { + return DebugInfo{}, errNoElixirDebugInfo + } + if arity, err := r.enterTuple(); err != nil { + return DebugInfo{}, err + } else if arity != 3 { + return DebugInfo{}, fmt.Errorf("elixir debug info tuple arity %d", arity) + } + if format, err := r.readAtom(); err != nil { + return DebugInfo{}, err + } else if format != "elixir_v1" { + return DebugInfo{}, fmt.Errorf("unsupported Elixir debug info %q", format) + } + + pairs, err := r.enterMap() + if err != nil { + return DebugInfo{}, err + } + var sites []definitionSite + for i := int64(0); i < pairs; i++ { + var key string + if tag, err := r.peekTag(); err != nil { + return DebugInfo{}, err + } else if isAtomTag(tag) { + if key, err = r.readAtom(); err != nil { + return DebugInfo{}, err + } + } else if err := r.skip(); err != nil { + return DebugInfo{}, err + } + switch key { + case "definitions": + if sites, err = readDefinitionSites(r); err != nil { + return DebugInfo{}, err + } + case "file": + if info.File, err = readOptionalBinary(r); err != nil { + return DebugInfo{}, err + } + case "relative_file": + if info.RelativeFile, err = readOptionalBinary(r); err != nil { + return DebugInfo{}, err + } + default: + if err := r.skip(); err != nil { + return DebugInfo{}, err + } + } + } + // The specs are not needed, but consuming them proves the term is whole. + if err := r.skip(); err != nil { + return DebugInfo{}, err + } + + info.Lines = make(map[FunctionKey]int, len(sites)) + for _, site := range sites { + line := site.line + if site.keepFile != "" && site.keepLine > 0 && + (site.keepFile == info.RelativeFile || site.keepFile == info.File) { + line = site.keepLine + } + if line > 0 { + info.Lines[site.key] = line + } + } + return info, nil +} + +// readDefinitionSites reads the definitions list, whose entries are +// {{name, arity}, kind, meta, clauses}. Private definitions are skipped: +// generated functions are found through the export table, so only public ones +// can be asked about. +func readDefinitionSites(r *etfReader) ([]definitionSite, error) { + count, hasTail, err := r.enterList() + if err != nil { + return nil, err + } + sites := make([]definitionSite, 0, count) + for i := int64(0); i < count; i++ { + arity, err := r.enterTuple() + if err != nil { + return nil, err + } + if arity != 4 { + if err := r.skipTerms(int64(arity)); err != nil { + return nil, err + } + continue + } + if keyArity, err := r.enterTuple(); err != nil { + return nil, err + } else if keyArity != 2 { + return nil, fmt.Errorf("definition key arity %d", keyArity) + } + name, err := r.readAtom() + if err != nil { + return nil, err + } + functionArity, err := r.readInt() + if err != nil { + return nil, err + } + kind, err := r.readAtom() + if err != nil { + return nil, err + } + site := definitionSite{key: FunctionKey{Name: name, Arity: functionArity}} + if err := readDefinitionMeta(r, &site); err != nil { + return nil, err + } + if err := r.skip(); err != nil { // clauses + return nil, err + } + if kind == "def" || kind == "defmacro" { + sites = append(sites, site) + } + } + if hasTail { + if err := r.skip(); err != nil { + return nil, err + } + } + return sites, nil +} + +// readDefinitionMeta reads :line and a `file: {path, line}` entry from a +// definition's keyword metadata. The first of each wins, as Keyword.get would. +func readDefinitionMeta(r *etfReader, site *definitionSite) error { + count, hasTail, err := r.enterList() + if err != nil { + return err + } + seenLine, seenFile := false, false + for i := int64(0); i < count; i++ { + if tag, err := r.peekTag(); err != nil { + return err + } else if tag != tagSmallTuple { + if err := r.skip(); err != nil { + return err + } + continue + } + arity, err := r.enterTuple() + if err != nil { + return err + } + if arity != 2 { + if err := r.skipTerms(int64(arity)); err != nil { + return err + } + continue + } + var key string + if tag, err := r.peekTag(); err != nil { + return err + } else if isAtomTag(tag) { + if key, err = r.readAtom(); err != nil { + return err + } + } else if err := r.skip(); err != nil { + return err + } + switch { + case key == "line" && !seenLine: + seenLine = true + if site.line, err = readOptionalInt(r); err != nil { + return err + } + case key == "file" && !seenFile: + seenFile = true + if site.keepFile, site.keepLine, err = readFileLocation(r); err != nil { + return err + } + default: + if err := r.skip(); err != nil { + return err + } + } + } + if hasTail { + return r.skip() + } + return nil +} + +// readFileLocation reads a {path, line} tuple. Any other shape is skipped and +// reported as no location. +func readFileLocation(r *etfReader) (string, int, error) { + if tag, err := r.peekTag(); err != nil { + return "", 0, err + } else if tag != tagSmallTuple { + return "", 0, r.skip() + } + arity, err := r.enterTuple() + if err != nil { + return "", 0, err + } + if arity != 2 { + return "", 0, r.skipTerms(int64(arity)) + } + file, err := readOptionalBinary(r) + if err != nil { + return "", 0, err + } + line, err := readOptionalInt(r) + if err != nil { + return "", 0, err + } + return file, line, nil +} + +// readOptionalBinary reads a binary, or skips a term of any other type and +// returns "". +func readOptionalBinary(r *etfReader) (string, error) { + if tag, err := r.peekTag(); err != nil { + return "", err + } else if tag != tagBinary { + return "", r.skip() + } + return r.readBinary() +} + +// readOptionalInt reads a small or 32-bit integer, or skips a term of any other +// type and returns 0. +func readOptionalInt(r *etfReader) (int, error) { + if tag, err := r.peekTag(); err != nil { + return 0, err + } else if tag != tagSmallInteger && tag != tagInteger { + return 0, r.skip() + } + return r.readInt() +} diff --git a/internal/beam/debug_info_test.go b/internal/beam/debug_info_test.go new file mode 100644 index 0000000..b070e37 --- /dev/null +++ b/internal/beam/debug_info_test.go @@ -0,0 +1,369 @@ +package beam + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// testDefinition is one entry of the definitions list in a synthetic Dbgi chunk. +type testDefinition struct { + name string + arity int + kind string // def, defp, defmacro, defmacrop + line int // the :line meta; zero omits it + + // keepFile/keepLine write a `file: {path, line}` meta entry, which is what + // both `@file {path, line}` and `quote location: :keep` leave on a def. + keepFile string + keepLine int +} + +// buildDebugInfoTerm encodes {:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}} +// the way the Elixir compiler writes a Dbgi chunk. Each definition carries a +// clause body so the reader has to step over real AST shapes to reach the next. +func buildDebugInfoTerm(file, relativeFile string, definitions ...testDefinition) []byte { + var w etfTestWriter + w.smallTuple(3) + w.atom("debug_info_v1") + w.atom("elixir_erl") + w.smallTuple(3) + w.atom("elixir_v1") + + // Keys in the order ERTS sorts a small map's atom keys, which puts + // definitions before file and relative_file: the reader must not depend on + // having seen the file first. + w.mapHeader(4) + w.atom("attributes") + w.nil() + w.atom("definitions") + w.listHeader(len(definitions)) + for _, definition := range definitions { + w.smallTuple(4) + w.smallTuple(2) + w.atom(definition.name) + w.smallInt(definition.arity) + w.atom(definition.kind) + + pairs := 1 // generated: true + if definition.line > 0 { + pairs++ + } + if definition.keepFile != "" { + pairs++ + } + w.listHeader(pairs) + if definition.line > 0 { + w.smallTuple(2) + w.atom("line") + w.smallInt(definition.line) + } + if definition.keepFile != "" { + w.smallTuple(2) + w.atom("file") + w.smallTuple(2) + w.binary(definition.keepFile) + w.smallInt(definition.keepLine) + } + w.smallTuple(2) + w.atom("generated") + w.atom("true") + w.nil() + + // One clause, {meta, args, guards, body}, with a body shaped like a + // remote call: {{:., [], [Mod, :fun]}, [line: 1], [arg]}. + w.listHeader(1) + w.smallTuple(4) + w.nil() + w.listHeader(1) + w.smallTuple(3) + w.atom("value") + w.nil() + w.atom("nil") + w.nil() + w.nil() + w.smallTuple(3) + w.smallTuple(3) + w.atom(".") + w.nil() + w.listHeader(2) + w.atom("Elixir.Enum") + w.atom("map") + w.nil() + w.nil() + w.listHeader(1) + w.binary("a string literal in the body") + w.nil() + w.nil() + } + w.nil() + w.atom("file") + w.binary(file) + w.atom("relative_file") + w.binary(relativeFile) + + w.nil() // specs + return w.buf +} + +func writeDebugInfoBEAM(t *testing.T, path string, dbgi []byte) { + t.Helper() + writeTestBEAMOpts(t, path, testBEAMOptions{ + atomNames: defaultTestAtoms, + exports: defaultTestExports, + dbgi: dbgi, + }) +} + +// A generator that stamps `@file {path, line}` on the functions it defines +// points each one at the source line that asked for it, such as an Ash +// `define :get_by_slug` inside a domain. When that path is the module's own +// source, that line is the definition. +func TestReadDefinitionLinesPrefersOwnFileLocation(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Accounts.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTerm( + "/src/my_app/lib/my_app/accounts.ex", "lib/my_app/accounts.ex", + testDefinition{name: "get_user!", arity: 1, kind: "def", line: 1, keepFile: "lib/my_app/accounts.ex", keepLine: 7}, + testDefinition{name: "get_user!", arity: 2, kind: "def", line: 1, keepFile: "lib/my_app/accounts.ex", keepLine: 7}, + )) + + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + if info.File != "/src/my_app/lib/my_app/accounts.ex" || info.RelativeFile != "lib/my_app/accounts.ex" { + t.Fatalf("source = %q / %q", info.File, info.RelativeFile) + } + for _, arity := range []int{1, 2} { + if got := info.Lines[FunctionKey{Name: "get_user!", Arity: arity}]; got != 7 { + t.Errorf("get_user!/%d line = %d, want 7", arity, got) + } + } +} + +// `quote location: :keep` inside a library records the library's own file. +// That line belongs to the generator, not to the module, so the module-side +// :line is used instead. +func TestReadDefinitionLinesIgnoresForeignFileLocation(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Accounts.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTerm( + "/src/my_app/lib/my_app/accounts.ex", "lib/my_app/accounts.ex", + testDefinition{name: "list_users", arity: 0, kind: "def", line: 12, keepFile: "deps/shared_lib/lib/shared_lib/interface.ex", keepLine: 1112}, + testDefinition{name: "count_users", arity: 0, kind: "def", line: 1, keepFile: "deps/shared_lib/lib/shared_lib/interface.ex", keepLine: 1112}, + )) + + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + if got := info.Lines[FunctionKey{Name: "list_users", Arity: 0}]; got != 12 { + t.Errorf("list_users/0 line = %d, want the module-side line 12", got) + } + if got := info.Lines[FunctionKey{Name: "count_users", Arity: 0}]; got != 1 { + t.Errorf("count_users/0 line = %d, want 1", got) + } +} + +func TestReadDefinitionLinesPlainLineAndMacros(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Worker.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTerm( + "/src/my_app/lib/my_app/worker.ex", "lib/my_app/worker.ex", + testDefinition{name: "new", arity: 1, kind: "def", line: 2}, + testDefinition{name: "build", arity: 1, kind: "defmacro", line: 9}, + testDefinition{name: "helper", arity: 0, kind: "defp", line: 4}, + testDefinition{name: "no_line", arity: 0, kind: "def"}, + )) + + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + want := map[FunctionKey]int{ + {Name: "new", Arity: 1}: 2, + {Name: "build", Arity: 1}: 9, + } + if len(info.Lines) != len(want) { + t.Fatalf("lines = %v, want only public definitions with a line %v", info.Lines, want) + } + for key, line := range want { + if info.Lines[key] != line { + t.Errorf("%v line = %d, want %d", key, info.Lines[key], line) + } + } +} + +func TestReadDefinitionLinesRejectsMissingOrForeignDebugInfo(t *testing.T) { + dir := t.TempDir() + + noChunk := filepath.Join(dir, "Elixir.NoChunk.beam") + writeTestBEAM(t, noChunk, buildDocsTerm()) + if _, err := ReadDefinitionLines(noChunk); err == nil { + t.Error("expected an error for a BEAM without a Dbgi chunk") + } + + // `debug_info: false` still writes the chunk, with :none as the payload. + var none etfTestWriter + none.smallTuple(3) + none.atom("debug_info_v1") + none.atom("elixir_erl") + none.atom("none") + stripped := filepath.Join(dir, "Elixir.Stripped.beam") + writeDebugInfoBEAM(t, stripped, none.buf) + if _, err := ReadDefinitionLines(stripped); err == nil { + t.Error("expected an error for stripped debug info") + } + + // An Erlang module's abstract code is a different backend entirely. + var erlang etfTestWriter + erlang.smallTuple(3) + erlang.atom("debug_info_v1") + erlang.atom("erl_abstract_code") + erlang.smallTuple(2) + erlang.atom("none") + erlang.nil() + erlangBeam := filepath.Join(dir, "erlang_mod.beam") + writeDebugInfoBEAM(t, erlangBeam, erlang.buf) + if _, err := ReadDefinitionLines(erlangBeam); err == nil { + t.Error("expected an error for a non-Elixir debug info backend") + } +} + +// A corrupt build artifact must fail the read, never the server. +func TestReadDefinitionLinesTruncated(t *testing.T) { + full := buildDebugInfoTerm("/src/a.ex", "a.ex", + testDefinition{name: "run", arity: 0, kind: "def", line: 3, keepFile: "a.ex", keepLine: 5}) + dir := t.TempDir() + for cut := 1; cut < len(full); cut += 7 { + path := filepath.Join(dir, "Elixir.Cut.beam") + writeDebugInfoBEAM(t, path, full[:cut]) + if _, err := ReadDefinitionLines(path); err == nil { + t.Fatalf("expected an error when the term is cut at %d of %d bytes", cut, len(full)) + } + } +} + +// Clause bodies are arbitrary AST and nest far deeper than a Docs chunk. A long +// pipeline or nested case must not trip the depth guard and lose the module. +func TestReadDefinitionLinesDeepClauseBody(t *testing.T) { + var w etfTestWriter + w.smallTuple(3) + w.atom("debug_info_v1") + w.atom("elixir_erl") + w.smallTuple(3) + w.atom("elixir_v1") + w.mapHeader(2) + w.atom("definitions") + w.listHeader(2) + for i, name := range []string{"deep", "after_deep"} { + w.smallTuple(4) + w.smallTuple(2) + w.atom(name) + w.smallInt(0) + w.atom("def") + w.listHeader(1) + w.smallTuple(2) + w.atom("line") + w.smallInt(10 + i) + w.nil() + // 300 nested {:|>, [], [left, ...]} levels: 600 ETF levels. + const depth = 300 + for range depth { + w.smallTuple(3) + w.atom("|>") + w.nil() + w.listHeader(1) + } + w.atom("nil") + for range depth { + w.nil() + } + } + w.nil() + w.atom("relative_file") + w.binary("lib/deep.ex") + w.nil() + + path := filepath.Join(t.TempDir(), "Elixir.Deep.beam") + writeDebugInfoBEAM(t, path, w.buf) + info, err := ReadDefinitionLines(path) + if err != nil { + t.Fatal(err) + } + if info.Lines[FunctionKey{Name: "after_deep", Arity: 0}] != 11 { + t.Fatalf("definition after a deep body = %v, want line 11", info.Lines) + } +} + +// The synthetic fixtures encode what the compiler is believed to write. This +// compiles real modules so a change in Elixir's debug info layout fails here +// instead of silently sending every definition back to the module line. +func TestReadDefinitionLinesFromElixirCompiler(t *testing.T) { + elixirc, err := exec.LookPath("elixirc") + if err != nil { + t.Skip("elixirc not installed") + } + dir := t.TempDir() + write := func(name, source string) { + t.Helper() + if err := os.WriteFile(filepath.Join(dir, name), []byte(source), 0o644); err != nil { + t.Fatal(err) + } + } + // SharedLib.Interface generates functions three ways: with `@file` pointing + // at the line of the call that asked for it (as a DSL records it), from a + // `location: :keep` quote, and plainly. + write("interface.ex", `defmodule SharedLib.Interface do + defmacro define(name) do + line = __CALLER__.line + + quote bind_quoted: [name: name, line: line], location: :keep do + @file {__ENV__.file, line} + def unquote(name)(), do: :stamped + end + end + + defmacro define_kept(name) do + quote bind_quoted: [name: name], location: :keep do + def unquote(name)(), do: :kept + end + end + + defmacro define_plain(name) do + quote do + def unquote(name)(), do: :plain + end + end +end +`) + write("accounts.ex", `defmodule MyApp.Accounts do + require SharedLib.Interface + + SharedLib.Interface.define(:stamped) + + SharedLib.Interface.define_kept(:kept) + + SharedLib.Interface.define_plain(:plain) +end +`) + cmd := exec.Command(elixirc, "-o", dir, "interface.ex", "accounts.ex") + cmd.Dir = dir + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("elixirc: %v\n%s", err, out) + } + + info, err := ReadDefinitionLines(filepath.Join(dir, "Elixir.MyApp.Accounts.beam")) + if err != nil { + t.Fatal(err) + } + if !strings.HasSuffix(filepath.ToSlash(info.File), "/accounts.ex") { + t.Errorf("File = %q, want the compiled source", info.File) + } + want := map[string]int{"stamped": 4, "kept": 6, "plain": 8} + for name, line := range want { + if got := info.Lines[FunctionKey{Name: name, Arity: 0}]; got != line { + t.Errorf("%s/0 line = %d, want %d (all: %v)", name, got, line, info.Lines) + } + } +} diff --git a/internal/beam/docs_test.go b/internal/beam/docs_test.go index 9233d92..ae1e6a7 100644 --- a/internal/beam/docs_test.go +++ b/internal/beam/docs_test.go @@ -374,6 +374,7 @@ type testBEAMOptions struct { exports [][3]uint32 docs []byte attrs []byte + dbgi []byte // written compressed, as the compiler writes it } func writeTestBEAM(t *testing.T, path string, docs []byte) { @@ -389,25 +390,11 @@ func writeTestBEAMOpts(t *testing.T, path string, opts testBEAMOptions) { t.Helper() var size [4]byte - var raw bytes.Buffer - raw.WriteByte(etfVersion) + var raw []byte if len(opts.docs) > 0 { - raw.WriteByte(tagCompressed) - var uncompressed bytes.Buffer - uncompressed.Write(opts.docs) - binary.BigEndian.PutUint32(size[:], uint32(uncompressed.Len())) - raw.Write(size[:]) - var zw bytes.Buffer - compressor := zlib.NewWriter(&zw) - if _, err := compressor.Write(uncompressed.Bytes()); err != nil { - t.Fatal(err) - } - if err := compressor.Close(); err != nil { - t.Fatal(err) - } - raw.Write(zw.Bytes()) + raw = compressedTestTerm(t, opts.docs) } else { - raw.WriteByte(tagNil) + raw = []byte{etfVersion, tagNil} } var atoms bytes.Buffer @@ -443,7 +430,10 @@ func writeTestBEAMOpts(t *testing.T, path string, opts testBEAMOptions) { if len(opts.attrs) > 0 { writeTestChunk(&chunks, "Attr", opts.attrs) } - writeTestChunk(&chunks, "Docs", raw.Bytes()) + writeTestChunk(&chunks, "Docs", raw) + if len(opts.dbgi) > 0 { + writeTestChunk(&chunks, "Dbgi", compressedTestTerm(t, opts.dbgi)) + } var file bytes.Buffer file.WriteString("FOR1") @@ -456,6 +446,25 @@ func writeTestBEAMOpts(t *testing.T, path string, opts testBEAMOptions) { } } +// compressedTestTerm wraps an encoded term as a versioned COMPRESSED_TERM, the +// form Elixir writes its Docs and Dbgi chunks in. +func compressedTestTerm(t *testing.T, term []byte) []byte { + t.Helper() + var size [4]byte + raw := []byte{etfVersion, tagCompressed} + binary.BigEndian.PutUint32(size[:], uint32(len(term))) + raw = append(raw, size[:]...) + var zw bytes.Buffer + compressor := zlib.NewWriter(&zw) + if _, err := compressor.Write(term); err != nil { + t.Fatal(err) + } + if err := compressor.Close(); err != nil { + t.Fatal(err) + } + return append(raw, zw.Bytes()...) +} + func writeTestChunk(chunks *bytes.Buffer, name string, data []byte) { chunks.WriteString(name) var size [4]byte diff --git a/internal/beam/etf.go b/internal/beam/etf.go index 58e5de2..661e410 100644 --- a/internal/beam/etf.go +++ b/internal/beam/etf.go @@ -66,6 +66,10 @@ type etfReader struct { buf []byte pos int depth int + + // maxDepth overrides maxETFDepth for terms that legitimately nest deeper, + // such as the clause ASTs in a Dbgi chunk. Zero keeps the default. + maxDepth int } func (r *etfReader) remaining() int { return len(r.buf) - r.pos } @@ -298,7 +302,11 @@ func (r *etfReader) binarySpan() (start, length int, err error) { // skip advances past one term of any shape without allocating for it. func (r *etfReader) skip() error { - if r.depth >= maxETFDepth { + limit := r.maxDepth + if limit == 0 { + limit = maxETFDepth + } + if r.depth >= limit { return errTooDeep } tag, err := r.u8() diff --git a/internal/lsp/beam_path_test.go b/internal/lsp/beam_path_test.go index ad35ce0..cd8ec41 100644 --- a/internal/lsp/beam_path_test.go +++ b/internal/lsp/beam_path_test.go @@ -157,17 +157,23 @@ type testAttr struct { // makes a compiled module record facts its source does not state, which is the // only way a framework's macro provider can be discovered. func minimalBeamWithAttrs(attrs []testAttr, docs string, exports ...beamExport) []byte { - return buildTestBeam(attrs, docs, exports...) + return buildTestBeam(attrs, docs, nil, exports...) } // minimalBeamWithDocs is minimalBeam plus a Docs chunk whose inflated payload is // docs. ReadDocBody inflates that payload and slices it, so a test can put one // function's prose at a known offset without encoding a real docs_v1 term. func minimalBeamWithDocs(docs string, exports ...beamExport) []byte { - return buildTestBeam(nil, docs, exports...) + return buildTestBeam(nil, docs, nil, exports...) } -func buildTestBeam(attrs []testAttr, docs string, exports ...beamExport) []byte { +// minimalBeamWithDbgi adds a Dbgi chunk holding the encoded debug info term, +// which is where a compiled module records the line of each definition. +func minimalBeamWithDbgi(dbgi []byte, exports ...beamExport) []byte { + return buildTestBeam(nil, "", dbgi, exports...) +} + +func buildTestBeam(attrs []testAttr, docs string, dbgi []byte, exports ...beamExport) []byte { names := make([]string, 0, len(exports)+1) names = append(names, "Elixir.Minimal") for _, export := range exports { @@ -198,6 +204,10 @@ func buildTestBeam(attrs []testAttr, docs string, exports ...beamExport) []byte if len(attrs) > 0 { writeBeamChunk(&chunks, "Attr", attrChunk(attrs)) } + if len(dbgi) > 0 { + // Uncompressed, like docsChunk; the reader accepts both forms. + writeBeamChunk(&chunks, "Dbgi", append([]byte{131}, dbgi...)) + } var file bytes.Buffer file.WriteString("FOR1") diff --git a/internal/lsp/generated_completion.go b/internal/lsp/generated_completion.go index af48e8c..6b951d0 100644 --- a/internal/lsp/generated_completion.go +++ b/internal/lsp/generated_completion.go @@ -3,6 +3,7 @@ package lsp import ( "container/list" "path/filepath" + "slices" "sort" "strings" "sync" @@ -43,6 +44,12 @@ type generatedFunctionCacheEntry struct { providers []string providersResolved bool + // debugInfo holds the definition lines from the BEAM's Dbgi chunk, read on + // the first definition request that needs them. Memoized on the entry for + // the same reason as providers: the BEAM stamp invalidates both. + debugInfo beam.DebugInfo + debugInfoResolved bool + // docs memoizes rendered documentation prose for functions a hover has asked // about, so sweeping the mouse does not re-inflate the Docs chunk each time. // It is a pointer because get hands out a copy of the entry: the copy has to @@ -707,6 +714,94 @@ func (s *Server) generatedDefinitionResults(module string) []store.LookupResult return nil } +// generatedDebugInfo returns a compiled module's debug info and the stamp of the +// BEAM it came from. It is read once per BEAM, and a failed read is memoized as +// empty: a module compiled without debug info will not grow it until rebuilt. +func (s *Server) generatedDebugInfo(module, beamPath string) (beam.DebugInfo, fileStamp, bool) { + // Resolving first validates the entry and yields the BEAM path. + _ = s.generatedFunctionsFor(module, beamPath) + + entry, ok := s.generatedCache.get(module) + if !ok || entry.beamPath == "" { + return beam.DebugInfo{}, fileStamp{}, false + } + if !entry.debugInfoResolved { + started := s.debugNow() + info, err := beam.ReadDefinitionLines(entry.beamPath) + if !started.IsZero() { + s.debugf("Generated BEAM debug info: module=%s definitions=%d error=%v total=%s", module, len(info.Lines), err, time.Since(started).Round(time.Microsecond)) + } + entry.debugInfo = info + entry.debugInfoResolved = true + s.generatedCache.put(module, entry) + } + return entry.debugInfo, entry.beamStamp, len(entry.debugInfo.Lines) > 0 +} + +// generatedDefinitionResultsFor is where navigation sends a generated function. +// It starts from generatedDefinitionResults, the closest source-backed module, +// and moves to the function's own line when the compiled module records one in +// that module's source: for a function a DSL declared, that is the line that +// declared it. precise reports whether it did. +// +// The recorded line is used only when it can be trusted in the file being +// opened. The debug info must come from that file, the BEAM must not be older +// than it (lines move when the source is edited), and the line must fall after +// the module's own line, since a line at or before it says nothing the module +// result does not. Anything else keeps the module result. +func (s *Server) generatedDefinitionResultsFor(module, beamPath string, functions []beam.Function) (results []store.LookupResult, precise bool) { + results = s.generatedDefinitionResults(module) + if len(results) == 0 || len(functions) == 0 { + return results, false + } + info, beamStamp, ok := s.generatedDebugInfo(module, beamPath) + if !ok { + return results, false + } + for _, result := range results { + if !debugInfoDescribesFile(info, result.FilePath) { + continue + } + if source := statFileStamp(result.FilePath); !source.exists || source.mtime > beamStamp.mtime { + s.debugf("Definition: generated %s debug info is older than %s; keeping module line", module, result.FilePath) + return results, false + } + var lines []store.LookupResult + for _, function := range functions { + line := info.Lines[beam.FunctionKey{Name: function.Name, Arity: function.Arity}] + if line <= result.Line || slices.ContainsFunc(lines, func(r store.LookupResult) bool { return r.Line == line }) { + continue + } + lines = append(lines, store.LookupResult{ + Module: module, + FilePath: result.FilePath, + Line: line, + Kind: function.Kind, + Arity: function.Arity, + }) + } + if len(lines) > 0 { + return lines, true + } + return results, false + } + return results, false +} + +// debugInfoDescribesFile reports whether debug info was compiled from path. +// The absolute path is compared first; the relative one covers a project that +// has moved or is reached through a symlink since it was compiled. +func debugInfoDescribesFile(info beam.DebugInfo, path string) bool { + if info.File != "" && info.File == path { + return true + } + relative := filepath.ToSlash(info.RelativeFile) + if relative == "" || filepath.IsAbs(info.RelativeFile) { + return false + } + return strings.HasSuffix(filepath.ToSlash(path), "/"+relative) +} + // filterGeneratedProviderReferences removes the conservative false positives // produced by bare-call indexing. The parser attributes an injected call to // every use module in the file because source alone cannot know which macro won; diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go new file mode 100644 index 0000000..12a475f --- /dev/null +++ b/internal/lsp/generated_definition_test.go @@ -0,0 +1,315 @@ +package lsp + +import ( + "context" + "encoding/binary" + "os" + "path/filepath" + "testing" + "time" + + "go.lsp.dev/protocol" + "go.lsp.dev/uri" +) + +// A DSL that declares functions inside a module, the way an Ash domain's +// `define` calls do. Nothing in the source is a def, so the index has no row for +// get_room_by_slug!; only the compiled module knows it exists. +const generatedDomainSource = `# The chat domain. +defmodule MyApp.Chat do + use SharedLib.Domain + + resources do + resource MyApp.Chat.Room do + define :list_rooms, action: :read + define :get_room_by_slug, action: :read, get_by: [:slug] + end + end + + def run_local, do: get_room_by_slug!("lounge") +end +` + +const generatedCallerSource = `defmodule MyApp.Caller do + alias MyApp.Chat + + def run do + Chat.get_room_by_slug!("lounge") + end +end +` + +const ( + generatedDomainRel = "lib/my_app/chat.ex" + generatedCallerRel = "lib/my_app/caller.ex" + // generatedDefineLine is the 1-based line of `define :get_room_by_slug`. + generatedDefineLine = 8 + // generatedModuleLine is the 1-based line of `defmodule MyApp.Chat`. + generatedModuleLine = 2 +) + +// dbgiDefinition is one definition in a synthetic Dbgi chunk. +type dbgiDefinition struct { + name string + arity int + line int + keepFile string + keepLine int +} + +// dbgiTerm encodes the {:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}} +// term the Elixir compiler writes, with empty clause lists. +func dbgiTerm(file, relativeFile string, definitions ...dbgiDefinition) []byte { + var out []byte + tuple := func(arity int) { out = append(out, 104, byte(arity)) } + atom := func(name string) { out = append(out, etfAtom(name)...) } + list := func(count int) { out = append(out, etfListHeader(count)...) } + nilTerm := func() { out = append(out, 106) } + integer := func(value int) { + out = append(out, 98) + out = binary.BigEndian.AppendUint32(out, uint32(value)) + } + bin := func(text string) { + out = append(out, 109) + out = binary.BigEndian.AppendUint32(out, uint32(len(text))) + out = append(out, text...) + } + + tuple(3) + atom("debug_info_v1") + atom("elixir_erl") + tuple(3) + atom("elixir_v1") + out = append(out, 116, 0, 0, 0, 3) // map with three pairs + atom("definitions") + list(len(definitions)) + for _, definition := range definitions { + tuple(4) + tuple(2) + atom(definition.name) + integer(definition.arity) + atom("def") + pairs := 1 + if definition.keepFile != "" { + pairs++ + } + list(pairs) + tuple(2) + atom("line") + integer(definition.line) + if definition.keepFile != "" { + tuple(2) + atom("file") + tuple(2) + bin(definition.keepFile) + integer(definition.keepLine) + } + nilTerm() + nilTerm() // clauses + } + nilTerm() + atom("file") + bin(file) + atom("relative_file") + bin(relativeFile) + nilTerm() // specs + return out +} + +// newGeneratedDefinitionFixture indexes the domain and its caller, and writes +// the domain's BEAM with the given debug info. The source is backdated so the +// BEAM is the newer of the two, as it is right after `mix compile`. +func newGeneratedDefinitionFixture(t *testing.T, relativeFile string, definitions ...dbgiDefinition) (*Server, string) { + t.Helper() + server, cleanup := setupTestServer(t) + t.Cleanup(cleanup) + + indexFile(t, server.store, server.projectRoot, generatedDomainRel, generatedDomainSource) + indexFile(t, server.store, server.projectRoot, generatedCallerRel, generatedCallerSource) + domainPath := filepath.Join(server.projectRoot, generatedDomainRel) + past := time.Now().Add(-time.Hour) + if err := os.Chtimes(domainPath, past, past); err != nil { + t.Fatal(err) + } + + ebin := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin") + if err := os.MkdirAll(ebin, 0o755); err != nil { + t.Fatal(err) + } + beam := minimalBeamWithDbgi(dbgiTerm(domainPath, relativeFile, definitions...), + beamExport{"run_local", 0}, + beamExport{"list_rooms", 0}, + beamExport{"get_room_by_slug!", 1}, + beamExport{"get_room_by_slug!", 2}, + ) + if err := os.WriteFile(filepath.Join(ebin, "Elixir.MyApp.Chat.beam"), beam, 0o644); err != nil { + t.Fatal(err) + } + return server, domainPath +} + +// stampedDefinitions is what a generator writes when it stamps each function +// with `@file {file, line}` for the DSL call that declared it: :line is where +// the before-compile hook ran, and the file entry names the declaring line. +func stampedDefinitions() []dbgiDefinition { + return []dbgiDefinition{ + {name: "list_rooms", arity: 0, line: 1, keepFile: generatedDomainRel, keepLine: 7}, + {name: "get_room_by_slug!", arity: 1, line: 1, keepFile: generatedDomainRel, keepLine: generatedDefineLine}, + {name: "get_room_by_slug!", arity: 2, line: 1, keepFile: generatedDomainRel, keepLine: generatedDefineLine}, + } +} + +// lineDefinitions is what a generator writes when it expands each function at +// the line of the DSL call that declared it, as Ash's code interfaces do: :line +// is the declaring line, and the file entry still names the generator. +func lineDefinitions() []dbgiDefinition { + const generator = "deps/ash/lib/ash/code_interface.ex" + return []dbgiDefinition{ + {name: "list_rooms", arity: 0, line: 7, keepFile: generator, keepLine: 1112}, + {name: "get_room_by_slug!", arity: 1, line: generatedDefineLine, keepFile: generator, keepLine: 1112}, + {name: "get_room_by_slug!", arity: 2, line: generatedDefineLine, keepFile: generator, keepLine: 1112}, + } +} + +func generatedDefinitionAt(t *testing.T, server *Server, rel string, source string, line, character int) []protocol.Location { + t.Helper() + docURI := string(uri.File(filepath.Join(server.projectRoot, rel))) + server.docs.Set(docURI, source) + locations, err := server.Definition(context.Background(), &protocol.DefinitionParams{ + TextDocumentPositionParams: protocol.TextDocumentPositionParams{ + TextDocument: protocol.TextDocumentIdentifier{URI: protocol.DocumentURI(docURI)}, + Position: protocol.Position{Line: uint32(line), Character: uint32(character)}, + }, + }) + if err != nil { + t.Fatal(err) + } + return locations +} + +func expectSingleLocation(t *testing.T, locations []protocol.Location, path string, line int) { + t.Helper() + if len(locations) != 1 { + t.Fatalf("expected one location at %s:%d, got %#v", path, line, locations) + } + if got := uriToPath(locations[0].URI); got != path || int(locations[0].Range.Start.Line) != line-1 { + t.Fatalf("definition = %s:%d, want %s:%d", got, locations[0].Range.Start.Line+1, path, line) + } +} + +// Issue #108: a function a DSL generated resolved to the top of its module +// even when the compiled module records the line that declared it. +func TestDefinitionGeneratedFunctionUsesDebugInfoLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +func TestDefinitionBareGeneratedFunctionUsesDebugInfoLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + locations := generatedDefinitionAt(t, server, generatedDomainRel, generatedDomainSource, 11, 24) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +func TestDefinitionGeneratedFunctionUsesLineWhenFileIsGenerator(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// `dexter lookup` resolves through LookupName, and a recorded line is the +// function's own definition, so a strict lookup takes it too. +func TestLookupNameGeneratedFunctionUsesDebugInfoLine(t *testing.T) { + for _, tc := range []struct { + name string + opts NameLookupOptions + }{ + {"fallback", NameLookupOptions{FallbackToModule: true}}, + {"strict", NameLookupOptions{ExactModule: true}}, + } { + t.Run(tc.name, func(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + locations, err := server.LookupName("MyApp.Chat", "get_room_by_slug!", tc.opts) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != domainPath || locations[0].Line != generatedDefineLine { + t.Fatalf("expected %s:%d, got %#v", domainPath, generatedDefineLine, locations) + } + }) + } +} + +// Without a recorded line, a strict lookup still answers with the module. +func TestLookupNameStrictGeneratedFunctionWithoutLineKeepsModule(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, + ) + locations, err := server.LookupName("MyApp.Chat", "get_room_by_slug!", NameLookupOptions{ExactModule: true}) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != domainPath || locations[0].Line != generatedModuleLine { + t.Fatalf("expected %s:%d, got %#v", domainPath, generatedModuleLine, locations) + } +} + +// A library's `quote location: :keep` names the library's own file. That is +// the generator, not the declaration, and :line only says where the +// before-compile hook ran, which is no better than the module itself. +func TestDefinitionGeneratedFunctionForeignLocationKeepsModuleLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, + ) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedModuleLine) +} + +// A BEAM older than its source describes lines that may have moved. The +// function still exists, so the module is still the answer, but not the line. +func TestDefinitionGeneratedFunctionStaleBEAMKeepsModuleLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + future := time.Now().Add(time.Hour) + if err := os.Chtimes(domainPath, future, future); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedModuleLine) +} + +// The debug info's line belongs to the file it was compiled from. If that is +// not the file the index has for the module, the line means nothing there. +func TestDefinitionGeneratedFunctionOtherSourceKeepsModuleLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, "lib/my_app/other_chat.ex", + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "lib/my_app/other_chat.ex", keepLine: generatedDefineLine}, + ) + // The absolute path must not match either. + beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") + beam := minimalBeamWithDbgi(dbgiTerm(filepath.Join(server.projectRoot, "lib/my_app/other_chat.ex"), "lib/my_app/other_chat.ex", + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "lib/my_app/other_chat.ex", keepLine: generatedDefineLine}), + beamExport{"get_room_by_slug!", 1}) + if err := os.WriteFile(beamPath, beam, 0o644); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedModuleLine) +} + +// Call hierarchy names the same place definition goes to. +func TestPrepareCallHierarchyGeneratedFunctionUsesDebugInfoLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + docURI := string(uri.File(filepath.Join(server.projectRoot, generatedCallerRel))) + server.docs.Set(docURI, generatedCallerSource) + items, err := server.PrepareCallHierarchy(context.Background(), &protocol.CallHierarchyPrepareParams{ + TextDocumentPositionParams: protocol.TextDocumentPositionParams{ + TextDocument: protocol.TextDocumentIdentifier{URI: protocol.DocumentURI(docURI)}, + Position: protocol.Position{Line: 4, Character: 10}, + }, + }) + if err != nil { + t.Fatal(err) + } + if len(items) != 1 || uriToPath(items[0].URI) != domainPath || int(items[0].Range.Start.Line) != generatedDefineLine-1 { + t.Fatalf("expected the call hierarchy item at %s:%d, got %#v", domainPath, generatedDefineLine, items) + } +} diff --git a/internal/lsp/name_navigation.go b/internal/lsp/name_navigation.go index 4e380f0..15ed5c5 100644 --- a/internal/lsp/name_navigation.go +++ b/internal/lsp/name_navigation.go @@ -69,14 +69,16 @@ func (s *Server) LookupName(module, function string, opts NameLookupOptions) ([] } if len(results) == 0 && opts.Kind != NameKindType { if generated, found := s.generatedSymbol(module, "", function); found && len(generated) > 0 { - if opts.ExactModule { + // A line the compiled module records for the function is its own + // definition, so even an exact lookup takes it. + var precise bool + results, precise = s.generatedDefinitionResultsFor(module, "", generated) + if !precise && opts.ExactModule { if results, err = s.store.LookupModule(module); err != nil { return nil, err } - } else { - results = s.generatedDefinitionResults(module) } - if len(results) > 0 { + if !precise && len(results) > 0 { results[0].Arity = generated[0].Arity results[0].Kind = generated[0].Kind } diff --git a/internal/lsp/server.go b/internal/lsp/server.go index afc3241..335aecf 100644 --- a/internal/lsp/server.go +++ b/internal/lsp/server.go @@ -1159,11 +1159,11 @@ func (s *Server) Definition(ctx context.Context, params *protocol.DefinitionPara s.debugf("Definition: resolved bare %q -> %q", functionName, fullModule) if fullModule == "" { currentModule := s.store.LookupEnclosingModule(uriToPath(protocol.DocumentURI(docURI)), lineNum+1) - if provider, _, found := s.generatedSymbolInScope(currentModule, func() []string { + if provider, functions, found := s.generatedSymbolInScope(currentModule, func() []string { return s.enclosingBlockPath(docURI, lineNum, col) }, functionName); found { - if results := s.generatedDefinitionResults(provider.module); len(results) > 0 { - s.debugf("Definition: generated bare %q provider=%s", functionName, provider.module) + if results, precise := s.generatedDefinitionResultsFor(provider.module, provider.beamPath, functions); len(results) > 0 { + s.debugf("Definition: generated bare %q provider=%s precise=%t", functionName, provider.module, precise) return storeResultsToLocations(results), nil } } @@ -1207,11 +1207,11 @@ func (s *Server) Definition(ctx context.Context, params *protocol.DefinitionPara } currentModule = s.store.LookupEnclosingModule(uriToPath(protocol.DocumentURI(docURI)), lineNum+1) - if provider, _, found := s.generatedSymbolInScope(currentModule, func() []string { + if provider, functions, found := s.generatedSymbolInScope(currentModule, func() []string { return s.enclosingBlockPath(docURI, lineNum, col) }, functionName); found { - if results := s.generatedDefinitionResults(provider.module); len(results) > 0 { - s.debugf("Definition: generated fallback for bare %q provider=%s", functionName, provider.module) + if results, precise := s.generatedDefinitionResultsFor(provider.module, provider.beamPath, functions); len(results) > 0 { + s.debugf("Definition: generated fallback for bare %q provider=%s precise=%t", functionName, provider.module, precise) return storeResultsToLocations(results), nil } } @@ -7102,7 +7102,7 @@ func (s *Server) PrepareCallHierarchy(ctx context.Context, params *protocol.Call if len(generatedFunctions) == 0 { return nil, nil } - defResults = s.generatedDefinitionResults(fullModule) + defResults, _ = s.generatedDefinitionResultsFor(fullModule, "", generatedFunctions) if len(defResults) == 0 { return nil, nil } From c6563fecad9d3c5c12424359efdce9eb00dc47c2 Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Tue, 29 Sep 2026 17:55:52 -0400 Subject: [PATCH 02/16] Combine generated definition lines with compile info from #102 Drop the stale-BEAM guard. Dexter cannot compile the project, so a BEAM older than its source is the usual state while editing, and its line is closer than the module line. From #102: - Keep the Docs chunk anno as beam.Function.Line, and use it when a module has no debug info. The CInf chunk's :source (beam.ReadSourcePath) says which file that line is in. - Send a generated module with no source row, such as a Spark entity module, to the file it was compiled from, rebased onto the project's lib or deps when it was built elsewhere. Move the definition-line code into generated_definition.go. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- docs/architecture.md | 6 +- internal/beam/compile_info.go | 152 ++++++++++++++++ internal/beam/compile_info_test.go | 115 ++++++++++++ internal/beam/debug_info_test.go | 19 ++ internal/beam/docs.go | 32 +++- internal/beam/docs_test.go | 95 +++++++++- internal/lsp/beam_path_test.go | 56 ++++-- internal/lsp/generated_completion.go | 100 +---------- internal/lsp/generated_definition.go | 207 ++++++++++++++++++++++ internal/lsp/generated_definition_test.go | 181 +++++++++++++++++-- 11 files changed, 832 insertions(+), 133 deletions(-) create mode 100644 internal/beam/compile_info.go create mode 100644 internal/beam/compile_info_test.go create mode 100644 internal/lsp/generated_definition.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 92993a5..8ac400a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when the BEAM is not older than the source and was compiled from the file being opened; otherwise the module line remains the answer. Ash code interfaces go to their `define` line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) +- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line. A module compiled without debug info falls back to its Docs chunk annotation. A generated module with no source of its own, such as a Spark DSL entity, goes to the file it was compiled from, rebased onto the project's `deps` when it was built elsewhere. Ash code interfaces go to their `define` line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) - **`--root`/`-C` names the workspace on every command** — `dexter init --root ~/project`, `dexter lookup --root ~/project MyApp.Repo`, and `dexter lsp --root ~/project` all run as if they had been started in that directory, so an agent, a script, or an editor wrapper can index or query a project from anywhere. Relative paths, a `reindex` target included, resolve from the named root diff --git a/docs/architecture.md b/docs/architecture.md index 9c1286f..0dd0d9f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -143,9 +143,11 @@ A generated function has no source definition, but the compiled module's `Dbgi` Each def carries two locations. Its `:line` is the line in the module's own file that was being compiled when the def was produced: the macro call, or wherever a before-compile hook ran. A `file: {path, line}` entry, left by `@file` or `quote location: :keep`, is preferred when `path` is the module's own source, because that is where the code asked for the function. When `path` is another file it is the generator's implementation, and `:line` is used instead. Ash's code interfaces take the second shape: each generated def is expanded at its `define` line, so `:line` is that line and `file` still names `code_interface.ex`, which keeps stacktraces unchanged. No framework is recognized by name. -`generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line can be trusted in the file being opened: the debug info must have been compiled from that file (absolute path, or relative path as a suffix for a moved or symlinked project), the BEAM must not be older than the file, and the line must fall after the module's own line. Any other case keeps the module result. Unlike generated *names*, which a stale BEAM may still contribute, a stale BEAM's *lines* may have moved, so here source-vs-BEAM mtime is a gate. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. +`generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (absolute path, or relative path as a suffix for a moved or symlinked project), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is at worst a few lines off, which is still closer than the module line. -The debug info is read on the first definition request that needs it and memoized on the module's generated-function cache entry, so the BEAM stamp invalidates it with everything else. A module compiled without debug info, or an Erlang module, is memoized as empty. +A module compiled without debug info still has its Docs chunk, whose per-entry anno is the same `:line` (`beam.Function.Line`). It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually a framework file, so when that path does not exist here it is rebased onto the project by the tail below the application's `lib` directory (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. If no file is found, the lexical parent stays the answer. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. + +The debug info and compile info are read on the first definition request that needs them and memoized on the module's generated-function cache entry, so the BEAM stamp invalidates them with everything else. A module compiled without debug info, or an Erlang module, is memoized as empty. For references, the parser records bare injected calls under the direct `use` module because the generated provider is unavailable while source is indexed. At lookup time, generated-symbol resolution queries those injector rows and validates every candidate against the compiled provider active at that candidate's block path. This keeps same-named macros from another DSL or another section out of the result. Statement-level injected calls with arguments are indexed even when they omit both parentheses and a `do` block, as in `authorize_if always()`. diff --git a/internal/beam/compile_info.go b/internal/beam/compile_info.go new file mode 100644 index 0000000..d5a3603 --- /dev/null +++ b/internal/beam/compile_info.go @@ -0,0 +1,152 @@ +package beam + +import ( + "errors" +) + +// ReadSourcePath returns the source file recorded in a BEAM's compile info +// (the CInf chunk). +// +// The chunk is a short keyword list — `[version: ..., options: [...], source: +// ...]` — so this is a handful of terms, far cheaper than the Docs chunk it is +// usually read alongside. +// +// The path is the one the artifact was compiled in, which is not necessarily +// where the artifact lives now: a `_build` copied between checkouts, or any +// Docker/CI build, records a directory that no longer exists. Callers must treat +// the result as a hint and verify it before returning it to an editor. +// +// This matters for modules that have no source file of their own. Spark creates +// entity modules such as `Ash.Resource.Dsl.CodeInterface.Define` with an explicit +// location, so the module's recorded source is the framework file that generated +// it, and the source index has no row to consult. +func ReadSourcePath(path string) (string, bool) { + raw, err := readChunk(path, "CInf") + if err != nil { + return "", false + } + return parseCompileSource(raw) +} + +// parseCompileSource reads `{version, options, source}` pairs and returns the +// source entry. Any other shape is skipped rather than failing the chunk: a +// module without usable compile info must still yield its generated functions. +func parseCompileSource(raw []byte) (string, bool) { + if len(raw) < 1 || raw[0] != etfVersion { + return "", false + } + r := &etfReader{buf: raw[1:]} + + count, hasTail, err := r.enterList() + if err != nil { + return "", false + } + + for i := int64(0); i < count; i++ { + arity, err := r.enterTuple() + if err != nil { + return "", false + } + if arity != 2 { + if err := r.skipTerms(int64(arity)); err != nil { + return "", false + } + continue + } + + key, err := r.readAtom() + if err != nil { + return "", false + } + if key != "source" { + if err := r.skip(); err != nil { + return "", false + } + continue + } + + value, err := r.readStringTerm() + if err != nil || value == "" { + return "", false + } + return value, true + } + + // enterList reports a tail for every LIST_EXT, not only improper ones, so the + // terminator still has to be consumed when the loop runs to completion. + if hasTail { + if err := r.skip(); err != nil { + return "", false + } + } + return "", false +} + +// readStringTerm reads a path-shaped term: a binary, a string, or a charlist of +// small integers. The compiler records :source as a charlist, which is stored as +// a compact string term; other producers of the same chunk use a binary. +func (r *etfReader) readStringTerm() (string, error) { + tag, err := r.peekTag() + if err != nil { + return "", err + } + + switch tag { + case tagNil: + _, err := r.u8() + return "", err + + case tagBinary: + return r.readBinary() + + case tagString: + if _, err := r.u8(); err != nil { + return "", err + } + n, err := r.u16() + if err != nil { + return "", err + } + if err := r.need(n); err != nil { + return "", err + } + s := string(r.buf[r.pos : r.pos+n]) + r.pos += n + return s, nil + + case tagList: + count, hasTail, err := r.enterList() + if err != nil { + return "", err + } + // The chunk size already bounds count; this only keeps a corrupt length + // from preallocating an absurd buffer. + if count > maxSourcePathLen { + return "", errors.New("charlist too long") + } + out := make([]byte, 0, count) + for i := int64(0); i < count; i++ { + n, err := r.readInt() + if err != nil { + return "", err + } + if n < 0 || n > 255 { + return "", errors.New("charlist element out of range") + } + out = append(out, byte(n)) + } + if hasTail { + if err := r.skip(); err != nil { + return "", err + } + } + return string(out), nil + + default: + return "", errors.New("unexpected source path encoding") + } +} + +// maxSourcePathLen bounds a charlist path. Real paths are far shorter; the limit +// exists so a corrupt length cannot drive a large allocation. +const maxSourcePathLen = 4096 diff --git a/internal/beam/compile_info_test.go b/internal/beam/compile_info_test.go new file mode 100644 index 0000000..125c43b --- /dev/null +++ b/internal/beam/compile_info_test.go @@ -0,0 +1,115 @@ +package beam + +import ( + "path/filepath" + "testing" +) + +// compileInfoTerm builds a CInf payload: a keyword list shaped like the one the +// compiler writes, with the source entry encoded in the requested form. +func compileInfoTerm(t *testing.T, source func(*etfTestWriter, string)) []byte { + t.Helper() + // parseCompileSource takes the chunk payload, which begins with the ETF + // version byte, exactly as writeTestBEAM lays it out. + var w etfTestWriter + w.version() + + w.listHeader(2) + + w.smallTuple(2) + w.atom("version") + w.string("10.0.4") + + w.smallTuple(2) + w.atom("source") + source(&w, "/build/agent/deps/shared_lib/lib/shared_lib/worker.ex") + + w.nil() + return w.buf +} + +func TestParseCompileSource(t *testing.T) { + cases := []struct { + name string + source func(*etfTestWriter, string) + }{ + { + // What the compiler writes: a printable charlist stored as a string. + name: "string term", + source: func(w *etfTestWriter, path string) { + w.string(path) + }, + }, + { + name: "binary term", + source: func(w *etfTestWriter, path string) { + w.binary(path) + }, + }, + { + name: "charlist term", + source: func(w *etfTestWriter, path string) { + w.listHeader(len(path)) + for i := 0; i < len(path); i++ { + w.smallInt(int(path[i])) + } + w.nil() + }, + }, + } + + const want = "/build/agent/deps/shared_lib/lib/shared_lib/worker.ex" + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, ok := parseCompileSource(compileInfoTerm(t, tc.source)) + if !ok { + t.Fatalf("parseCompileSource reported no source") + } + if got != want { + t.Errorf("source = %q, want %q", got, want) + } + }) + } +} + +func TestParseCompileSourceMissing(t *testing.T) { + // A compile info list without a source entry, and payloads that are not a + // list at all, must all report "no source" rather than inventing one. + var noSource etfTestWriter + noSource.smallTuple(0) + + var notAList etfTestWriter + notAList.smallInt(1) + + cases := map[string][]byte{ + "empty": nil, + "no entry": noSource.buf, + "not a list": notAList.buf, + } + for name, raw := range cases { + if got, ok := parseCompileSource(raw); ok { + t.Errorf("%s: got %q, want no source", name, got) + } + } +} + +func TestReadSourcePath(t *testing.T) { + path := filepath.Join(t.TempDir(), "Elixir.LibFixture.beam") + writeTestBEAMOpts(t, path, testBEAMOptions{ + atomNames: defaultTestAtoms, + cinf: compileInfoTerm(t, func(w *etfTestWriter, path string) { + w.string(path) + }), + }) + + got, ok := ReadSourcePath(path) + if !ok || got != "/build/agent/deps/shared_lib/lib/shared_lib/worker.ex" { + t.Errorf("ReadSourcePath = %q, %v; want the recorded source", got, ok) + } + + missing := filepath.Join(t.TempDir(), "Elixir.NoCompileInfo.beam") + writeTestBEAM(t, missing, buildDocsTerm()) + if got, ok := ReadSourcePath(missing); ok { + t.Errorf("ReadSourcePath without CInf = %q, want no source", got) + } +} diff --git a/internal/beam/debug_info_test.go b/internal/beam/debug_info_test.go index b070e37..e5dfabb 100644 --- a/internal/beam/debug_info_test.go +++ b/internal/beam/debug_info_test.go @@ -366,4 +366,23 @@ end t.Errorf("%s/0 line = %d, want %d (all: %v)", name, got, line, info.Lines) } } + + // The Docs anno is the fallback for a module without debug info, and the + // compile info says which file it is in. + source, ok := ReadSourcePath(filepath.Join(dir, "Elixir.MyApp.Accounts.beam")) + if !ok || !strings.HasSuffix(filepath.ToSlash(source), "/accounts.ex") { + t.Errorf("ReadSourcePath = %q, %v; want the compiled source", source, ok) + } + documented, err := ReadDocumentedFunctions(filepath.Join(dir, "Elixir.MyApp.Accounts.beam")) + if err != nil { + t.Fatal(err) + } + for _, function := range documented { + if line, ok := want[function.Name]; ok && function.Line != line { + t.Errorf("%s/0 Docs line = %d, want %d", function.Name, function.Line, line) + } + } + if len(documented) != len(want) { + t.Errorf("documented = %+v, want %d functions", documented, len(want)) + } } diff --git a/internal/beam/docs.go b/internal/beam/docs.go index f2e2df4..5a34c34 100644 --- a/internal/beam/docs.go +++ b/internal/beam/docs.go @@ -29,6 +29,18 @@ type Function struct { Kind string Hidden bool + // Line is the source annotation the compiler recorded for this callable, as + // emitted in the Docs chunk. It is zero when the callable came from the export + // table alone, which has no annotation. + // + // The annotation names the site the code was created at, not the module it + // ended up in: a symbol created while expanding `use X` is attributed to the + // `use` line, while one created by a transformer or a `@before_compile` hook is + // attributed to the module line. Combined with the owning module's source file + // (see ReadSourcePath) that is a usable definition target for code that exists + // only in compiled form. + Line int + // DocOffset and DocLen locate the documentation prose inside the inflated // Docs chunk, so a hover can read one function's docs without decoding the // chunk again. They are zero when the function has no prose. zlib output is @@ -456,7 +468,8 @@ func parseDocsEntry(r *etfReader) ([]Function, bool, error) { return nil, false, err } - if err := r.skip(); err != nil { // anno + line, err := readAnnoLine(r) + if err != nil { return nil, false, err } params, err := readSignatureParams(r) @@ -502,6 +515,7 @@ func parseDocsEntry(r *etfReader) ([]Function, bool, error) { Params: strings.Join(callParams, ","), Kind: definitionKind, Hidden: hidden, + Line: line, DocOffset: docOffset, DocLen: docLen, }) @@ -509,6 +523,22 @@ func parseDocsEntry(r *etfReader) ([]Function, bool, error) { return out, true, nil } +// readAnnoLine reads the per-entry source annotation. Elixir writes it as a bare +// line number, but the field is an erl_anno and may hold other shapes (a tuple, +// or `none`), so anything that is not an integer is consumed and reported as line +// 0 rather than failing the whole entry. A dropped annotation must never cost the +// module its generated functions. +func readAnnoLine(r *etfReader) (int, error) { + tag, err := r.peekTag() + if err != nil { + return 0, err + } + if tag == tagSmallInteger || tag == tagInteger { + return r.readInt() + } + return 0, r.skip() +} + // readSignatureParams consumes the signature list and parses the first signature // into argument names. Signatures are short, so unlike documentation prose they // are worth copying. diff --git a/internal/beam/docs_test.go b/internal/beam/docs_test.go index ae1e6a7..9f35a4a 100644 --- a/internal/beam/docs_test.go +++ b/internal/beam/docs_test.go @@ -375,6 +375,7 @@ type testBEAMOptions struct { docs []byte attrs []byte dbgi []byte // written compressed, as the compiler writes it + cinf []byte // written as given } func writeTestBEAM(t *testing.T, path string, docs []byte) { @@ -434,6 +435,9 @@ func writeTestBEAMOpts(t *testing.T, path string, opts testBEAMOptions) { if len(opts.dbgi) > 0 { writeTestChunk(&chunks, "Dbgi", compressedTestTerm(t, opts.dbgi)) } + if len(opts.cinf) > 0 { + writeTestChunk(&chunks, "CInf", opts.cinf) + } var file bytes.Buffer file.WriteString("FOR1") @@ -497,6 +501,10 @@ type docEntry struct { doc string // "none", "hidden", or the documentation prose defaults int + // anno is the source annotation the compiler records for the entry: the line + // the code was created at. Zero means the usual module line. + anno int + // extraMetadata writes additional pairs into the entry's metadata map, the // way a framework's `@doc spark_opts` lands alongside `defaults`. extraMetadata func(*etfTestWriter) @@ -532,7 +540,11 @@ func buildDocsTerm(entries ...docEntry) []byte { w.atom(entry.name) w.smallInt(entry.arity) - w.smallInt(1) // anno + anno := entry.anno + if anno == 0 { + anno = 1 + } + w.smallInt(anno) w.listHeader(1) w.binary(entry.signature) @@ -608,6 +620,14 @@ func (w *etfTestWriter) binary(text string) { w.buf = append(w.buf, text...) } +// string writes a STRING_EXT term, the compact encoding Erlang uses for the +// printable charlists it records in compile info. +func (w *etfTestWriter) string(text string) { + w.byte(tagString) + w.be16(uint16(len(text))) + w.buf = append(w.buf, text...) +} + func (w *etfTestWriter) smallInt(value int) { if value >= 0 && value < 256 { w.byte(tagSmallInteger) @@ -674,3 +694,76 @@ func TestParseDocsMetadataWithCapturedFunction(t *testing.T) { t.Errorf("entry after the captured function = %+v, want list/0", got[1]) } } + +// The compile-time annotation is what makes a generated callable navigable: it +// names the line the code was created at, which for a macro-emitted symbol is a +// specific site rather than the module the symbol ended up in. +func TestReadDocumentedFunctionsRecordsSourceLine(t *testing.T) { + docs := buildDocsTerm( + docEntry{kind: "macro", name: "code_interface", arity: 1, signature: "code_interface(body)", doc: "none", anno: 1811}, + docEntry{kind: "function", name: "generated_at_module_line", arity: 0, signature: "generated_at_module_line()", doc: "none", anno: 1}, + ) + path := filepath.Join(t.TempDir(), "Elixir.LibFixture.beam") + writeTestBEAM(t, path, docs) + + functions, err := ReadDocumentedFunctions(path) + if err != nil { + t.Fatalf("ReadDocumentedFunctions: %v", err) + } + + got := make(map[string]int, len(functions)) + for _, f := range functions { + got[f.Name] = f.Line + } + for name, want := range map[string]int{ + "code_interface": 1811, + "generated_at_module_line": 1, + } { + if got[name] != want { + t.Errorf("%s: Line = %d, want %d", name, got[name], want) + } + } +} + +// An annotation the reader does not understand must cost only the line, never +// the entry: dropping the whole Docs chunk would take every generated function +// with it. +func TestReadDocumentedFunctionsToleratesNonIntegerAnno(t *testing.T) { + // writeTestBEAM prepends the ETF version byte, so the payload starts at the + // term itself — same as buildDocsTerm. + var w etfTestWriter + w.smallTuple(7) + w.atom("docs_v1") + w.smallInt(1) + w.atom("elixir") + w.binary("text/markdown") + w.atom("none") + w.mapHeader(0) + w.listHeader(1) + w.smallTuple(5) + w.smallTuple(3) + w.atom("function") + w.atom("kept") + w.smallInt(0) + w.atom("none") // anno as an atom instead of a line + w.listHeader(1) + w.binary("kept()") + w.nil() + w.atom("none") + w.mapHeader(0) + w.nil() + + path := filepath.Join(t.TempDir(), "Elixir.LibFixture.beam") + writeTestBEAM(t, path, w.buf) + + functions, err := ReadDocumentedFunctions(path) + if err != nil { + t.Fatalf("ReadDocumentedFunctions: %v", err) + } + if len(functions) != 1 || functions[0].Name != "kept" { + t.Fatalf("functions = %+v, want the single kept/0 entry", functions) + } + if functions[0].Line != 0 { + t.Errorf("Line = %d, want 0 for an unreadable anno", functions[0].Line) + } +} diff --git a/internal/lsp/beam_path_test.go b/internal/lsp/beam_path_test.go index cd8ec41..57886f3 100644 --- a/internal/lsp/beam_path_test.go +++ b/internal/lsp/beam_path_test.go @@ -157,23 +157,33 @@ type testAttr struct { // makes a compiled module record facts its source does not state, which is the // only way a framework's macro provider can be discovered. func minimalBeamWithAttrs(attrs []testAttr, docs string, exports ...beamExport) []byte { - return buildTestBeam(attrs, docs, nil, exports...) + return buildTestBeam(testBeamChunks{attrs: attrs, docs: docs}, exports...) } // minimalBeamWithDocs is minimalBeam plus a Docs chunk whose inflated payload is // docs. ReadDocBody inflates that payload and slices it, so a test can put one // function's prose at a known offset without encoding a real docs_v1 term. func minimalBeamWithDocs(docs string, exports ...beamExport) []byte { - return buildTestBeam(nil, docs, nil, exports...) + return buildTestBeam(testBeamChunks{docs: docs}, exports...) } // minimalBeamWithDbgi adds a Dbgi chunk holding the encoded debug info term, // which is where a compiled module records the line of each definition. func minimalBeamWithDbgi(dbgi []byte, exports ...beamExport) []byte { - return buildTestBeam(nil, "", dbgi, exports...) + return buildTestBeam(testBeamChunks{dbgi: dbgi}, exports...) } -func buildTestBeam(attrs []testAttr, docs string, dbgi []byte, exports ...beamExport) []byte { +// testBeamChunks are the optional chunks of a synthetic BEAM. +type testBeamChunks struct { + attrs []testAttr + docs string + dbgi []byte + // source is written as the CInf chunk's :source, the file the module was + // compiled from. + source string +} + +func buildTestBeam(chunks testBeamChunks, exports ...beamExport) []byte { names := make([]string, 0, len(exports)+1) names = append(names, "Elixir.Minimal") for _, export := range exports { @@ -195,25 +205,28 @@ func buildTestBeam(attrs []testAttr, docs string, dbgi []byte, exports ...beamEx writeBE32(&table, uint32(index+1)) // label } - var chunks bytes.Buffer - writeBeamChunk(&chunks, "AtU8", atoms.Bytes()) - writeBeamChunk(&chunks, "ExpT", table.Bytes()) - if docs != "" { - writeBeamChunk(&chunks, "Docs", docsChunk(docs)) + var body bytes.Buffer + writeBeamChunk(&body, "AtU8", atoms.Bytes()) + writeBeamChunk(&body, "ExpT", table.Bytes()) + if chunks.docs != "" { + writeBeamChunk(&body, "Docs", docsChunk(chunks.docs)) } - if len(attrs) > 0 { - writeBeamChunk(&chunks, "Attr", attrChunk(attrs)) + if len(chunks.attrs) > 0 { + writeBeamChunk(&body, "Attr", attrChunk(chunks.attrs)) } - if len(dbgi) > 0 { + if len(chunks.dbgi) > 0 { // Uncompressed, like docsChunk; the reader accepts both forms. - writeBeamChunk(&chunks, "Dbgi", append([]byte{131}, dbgi...)) + writeBeamChunk(&body, "Dbgi", append([]byte{131}, chunks.dbgi...)) + } + if chunks.source != "" { + writeBeamChunk(&body, "CInf", compileInfoChunk(chunks.source)) } var file bytes.Buffer file.WriteString("FOR1") - writeBE32(&file, uint32(chunks.Len()+4)) + writeBE32(&file, uint32(body.Len()+4)) file.WriteString("BEAM") - file.Write(chunks.Bytes()) + file.Write(body.Bytes()) return file.Bytes() } @@ -225,6 +238,19 @@ func docsChunk(docs string) []byte { return append(chunk, docs...) } +// compileInfoChunk encodes the CInf chunk ReadSourcePath reads, with :source +// as the compact string term the compiler writes for a charlist. +func compileInfoChunk(source string) []byte { + out := []byte{131} // ETF version byte + out = append(out, etfListHeader(1)...) + out = append(out, 104, 2) // small tuple of arity 2 + out = append(out, etfAtom("source")...) + out = append(out, 107) // STRING_EXT + out = binary.BigEndian.AppendUint16(out, uint16(len(source))) + out = append(out, source...) + return append(out, 106) // NIL_EXT +} + // attrChunk encodes the Attr chunk ReadModuleAttributes reads: an ETF list of // {name, [values]} pairs, with the values as atoms because that is how the // compiler persists module names. diff --git a/internal/lsp/generated_completion.go b/internal/lsp/generated_completion.go index 6b951d0..2ef2396 100644 --- a/internal/lsp/generated_completion.go +++ b/internal/lsp/generated_completion.go @@ -3,7 +3,6 @@ package lsp import ( "container/list" "path/filepath" - "slices" "sort" "strings" "sync" @@ -44,11 +43,12 @@ type generatedFunctionCacheEntry struct { providers []string providersResolved bool - // debugInfo holds the definition lines from the BEAM's Dbgi chunk, read on - // the first definition request that needs them. Memoized on the entry for - // the same reason as providers: the BEAM stamp invalidates both. - debugInfo beam.DebugInfo - debugInfoResolved bool + // definitionSources holds what the BEAM records about where its functions + // were defined, read on the first definition request that needs it. + // Memoized on the entry for the same reason as providers: the BEAM stamp + // invalidates both. + definitionSources generatedDefinitionSources + definitionSourcesResolved bool // docs memoizes rendered documentation prose for functions a hover has asked // about, so sweeping the mouse does not re-inflate the Docs chunk each time. @@ -714,94 +714,6 @@ func (s *Server) generatedDefinitionResults(module string) []store.LookupResult return nil } -// generatedDebugInfo returns a compiled module's debug info and the stamp of the -// BEAM it came from. It is read once per BEAM, and a failed read is memoized as -// empty: a module compiled without debug info will not grow it until rebuilt. -func (s *Server) generatedDebugInfo(module, beamPath string) (beam.DebugInfo, fileStamp, bool) { - // Resolving first validates the entry and yields the BEAM path. - _ = s.generatedFunctionsFor(module, beamPath) - - entry, ok := s.generatedCache.get(module) - if !ok || entry.beamPath == "" { - return beam.DebugInfo{}, fileStamp{}, false - } - if !entry.debugInfoResolved { - started := s.debugNow() - info, err := beam.ReadDefinitionLines(entry.beamPath) - if !started.IsZero() { - s.debugf("Generated BEAM debug info: module=%s definitions=%d error=%v total=%s", module, len(info.Lines), err, time.Since(started).Round(time.Microsecond)) - } - entry.debugInfo = info - entry.debugInfoResolved = true - s.generatedCache.put(module, entry) - } - return entry.debugInfo, entry.beamStamp, len(entry.debugInfo.Lines) > 0 -} - -// generatedDefinitionResultsFor is where navigation sends a generated function. -// It starts from generatedDefinitionResults, the closest source-backed module, -// and moves to the function's own line when the compiled module records one in -// that module's source: for a function a DSL declared, that is the line that -// declared it. precise reports whether it did. -// -// The recorded line is used only when it can be trusted in the file being -// opened. The debug info must come from that file, the BEAM must not be older -// than it (lines move when the source is edited), and the line must fall after -// the module's own line, since a line at or before it says nothing the module -// result does not. Anything else keeps the module result. -func (s *Server) generatedDefinitionResultsFor(module, beamPath string, functions []beam.Function) (results []store.LookupResult, precise bool) { - results = s.generatedDefinitionResults(module) - if len(results) == 0 || len(functions) == 0 { - return results, false - } - info, beamStamp, ok := s.generatedDebugInfo(module, beamPath) - if !ok { - return results, false - } - for _, result := range results { - if !debugInfoDescribesFile(info, result.FilePath) { - continue - } - if source := statFileStamp(result.FilePath); !source.exists || source.mtime > beamStamp.mtime { - s.debugf("Definition: generated %s debug info is older than %s; keeping module line", module, result.FilePath) - return results, false - } - var lines []store.LookupResult - for _, function := range functions { - line := info.Lines[beam.FunctionKey{Name: function.Name, Arity: function.Arity}] - if line <= result.Line || slices.ContainsFunc(lines, func(r store.LookupResult) bool { return r.Line == line }) { - continue - } - lines = append(lines, store.LookupResult{ - Module: module, - FilePath: result.FilePath, - Line: line, - Kind: function.Kind, - Arity: function.Arity, - }) - } - if len(lines) > 0 { - return lines, true - } - return results, false - } - return results, false -} - -// debugInfoDescribesFile reports whether debug info was compiled from path. -// The absolute path is compared first; the relative one covers a project that -// has moved or is reached through a symlink since it was compiled. -func debugInfoDescribesFile(info beam.DebugInfo, path string) bool { - if info.File != "" && info.File == path { - return true - } - relative := filepath.ToSlash(info.RelativeFile) - if relative == "" || filepath.IsAbs(info.RelativeFile) { - return false - } - return strings.HasSuffix(filepath.ToSlash(path), "/"+relative) -} - // filterGeneratedProviderReferences removes the conservative false positives // produced by bare-call indexing. The parser attributes an injected call to // every use module in the file because source alone cannot know which macro won; diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go new file mode 100644 index 0000000..49c535e --- /dev/null +++ b/internal/lsp/generated_definition.go @@ -0,0 +1,207 @@ +package lsp + +import ( + "os" + "path/filepath" + "slices" + "strings" + "time" + + "github.com/remoteoss/dexter/internal/beam" + "github.com/remoteoss/dexter/internal/store" +) + +// generatedDefinitionSources is what a compiled module records about where its +// functions were defined. +type generatedDefinitionSources struct { + // debugInfo holds the definition lines from the Dbgi chunk. It is empty for + // a module compiled without debug info, and for Erlang modules. + debugInfo beam.DebugInfo + + // compileSource is the source path from the CInf chunk: the file the module + // was compiled from, as it was spelled on the machine that compiled it. + compileSource string +} + +// generatedDefinitionSourcesFor returns a compiled module's definition sources. +// They are read once per BEAM, and a failed read is memoized as empty: a module +// compiled without debug info will not grow it until rebuilt. +func (s *Server) generatedDefinitionSourcesFor(module, beamPath string) (generatedDefinitionSources, bool) { + // Resolving first validates the entry and yields the BEAM path. + _ = s.generatedFunctionsFor(module, beamPath) + + entry, ok := s.generatedCache.get(module) + if !ok || entry.beamPath == "" { + return generatedDefinitionSources{}, false + } + if !entry.definitionSourcesResolved { + started := s.debugNow() + info, err := beam.ReadDefinitionLines(entry.beamPath) + source, _ := beam.ReadSourcePath(entry.beamPath) + if !started.IsZero() { + s.debugf("Generated BEAM definition sources: module=%s definitions=%d error=%v source=%q total=%s", module, len(info.Lines), err, source, time.Since(started).Round(time.Microsecond)) + } + entry.definitionSources = generatedDefinitionSources{debugInfo: info, compileSource: source} + entry.definitionSourcesResolved = true + s.generatedCache.put(module, entry) + } + return entry.definitionSources, true +} + +// generatedDefinitionResultsFor is where navigation sends a generated function. +// It starts from generatedDefinitionResults, the closest source-backed module, +// and moves to the function's own line when the compiled module records one in +// that module's source: for a function a DSL declared, that is the line that +// declared it. precise reports whether it did. +// +// A module with no source row of its own, such as a Spark entity module, may +// still record a line in the file it was compiled from, which is usually a +// framework file under deps. That file is tried after the indexed ones. +// +// A recorded line is used only when it belongs to the file being opened and +// falls after the module's own line, since a line at or before it says nothing +// the module result does not. The BEAM may be older than the source. Dexter +// cannot compile the project, and a line from the last compile is usually +// right and at worst a few lines off, which is still closer than the module +// line. Anything else keeps the module result. +func (s *Server) generatedDefinitionResultsFor(module, beamPath string, functions []beam.Function) (results []store.LookupResult, precise bool) { + results = s.generatedDefinitionResults(module) + if len(functions) == 0 { + return results, false + } + sources, ok := s.generatedDefinitionSourcesFor(module, beamPath) + if !ok { + return results, false + } + for _, result := range results { + if !sources.describe(result.FilePath, s.projectRoot) { + continue + } + if lines := generatedFunctionLines(module, result.FilePath, result.Line, sources, functions); len(lines) > 0 { + return lines, true + } + return results, false + } + if own, err := s.store.LookupModule(module); err == nil && len(own) == 0 { + if source := s.compiledSourceFile(sources.compileSource); source != "" { + if lines := generatedFunctionLines(module, source, 1, sources, functions); len(lines) > 0 { + return lines, true + } + } + } + return results, false +} + +// generatedFunctionLines returns a result for each distinct line after +// afterLine that the compiled module records for functions in file. +// +// The Dbgi line is preferred because it also honors `@file`. The Docs chunk +// annotation is the fallback: it is the same line for a def a macro expanded, +// and a module compiled without debug info still has it. +func generatedFunctionLines(module, file string, afterLine int, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { + useDebugInfo := len(sources.debugInfo.Lines) > 0 + var lines []store.LookupResult + for _, function := range functions { + line := function.Line + if useDebugInfo { + line = sources.debugInfo.Lines[beam.FunctionKey{Name: function.Name, Arity: function.Arity}] + } + if line <= afterLine || slices.ContainsFunc(lines, func(r store.LookupResult) bool { return r.Line == line }) { + continue + } + lines = append(lines, store.LookupResult{ + Module: module, + FilePath: file, + Line: line, + Kind: function.Kind, + Arity: function.Arity, + }) + } + return lines +} + +// describe reports whether the recorded lines belong to path. When the module +// has debug info, its file names are the test; otherwise the compile info's. +func (sources generatedDefinitionSources) describe(path, projectRoot string) bool { + if len(sources.debugInfo.Lines) > 0 { + return debugInfoDescribesFile(sources.debugInfo, path) + } + if sources.compileSource == "" { + return false + } + return sources.compileSource == path || slices.Contains(sourceRebaseCandidates(sources.compileSource, projectRoot), path) +} + +// debugInfoDescribesFile reports whether debug info was compiled from path. +// The absolute path is compared first; the relative one covers a project that +// has moved or is reached through a symlink since it was compiled. +func debugInfoDescribesFile(info beam.DebugInfo, path string) bool { + if info.File != "" && info.File == path { + return true + } + relative := filepath.ToSlash(info.RelativeFile) + if relative == "" || filepath.IsAbs(info.RelativeFile) { + return false + } + return strings.HasSuffix(filepath.ToSlash(path), "/"+relative) +} + +// compiledSourceFile maps a source path from compile info onto this checkout. +// +// The path was recorded wherever the artifact was built, which for a dependency +// is often a different directory: a `_build` copied between checkouts, a Docker +// build, a vendored artifact. When it does not exist here, only the tail below +// the application's own lib directory is stable. +// +// The application is read from the recorded path, not from the BEAM path. A +// generated module can live in one application while its source lives in +// another: Spark creates Ash's entity modules, so the artifact sits in ash's +// ebin while the file that generated it is under spark. +func (s *Server) compiledSourceFile(recorded string) string { + if recorded == "" { + return "" + } + if regularFileExists(recorded) { + return recorded + } + for _, candidate := range sourceRebaseCandidates(recorded, s.projectRoot) { + if regularFileExists(candidate) { + return candidate + } + } + return "" +} + +// sourceRebaseCandidates lists the plausible on-disk locations for a recorded +// source path. Every Mix layout is offered, because the artifact does not record +// which one it was: `lib//...` is the application's own repository, +// `deps//lib//...` is a dependency, and `deps//...` is the older +// vendored shape. +func sourceRebaseCandidates(recorded, projectRoot string) []string { + if projectRoot == "" { + return nil + } + slash := filepath.ToSlash(recorded) + idx := strings.LastIndex(slash, "/lib/") + if idx < 0 { + return nil + } + parts := strings.SplitN(slash[idx+len("/lib/"):], "/", 2) + if len(parts) != 2 || parts[0] == "" || parts[1] == "" { + return nil + } + app, tail := parts[0], filepath.FromSlash(parts[1]) + return []string{ + filepath.Join(projectRoot, "lib", app, tail), + filepath.Join(projectRoot, "deps", app, "lib", app, tail), + filepath.Join(projectRoot, "deps", app, tail), + } +} + +// regularFileExists reports whether path names a readable regular file. +// Anything else is treated as absent, so an editor is never sent to a location +// it cannot open. +func regularFileExists(path string) bool { + info, err := os.Stat(path) + return err == nil && info.Mode().IsRegular() +} diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index 12a475f..3d1f56a 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -117,9 +117,17 @@ func dbgiTerm(file, relativeFile string, definitions ...dbgiDefinition) []byte { } // newGeneratedDefinitionFixture indexes the domain and its caller, and writes -// the domain's BEAM with the given debug info. The source is backdated so the -// BEAM is the newer of the two, as it is right after `mix compile`. +// the domain's BEAM with the given debug info. func newGeneratedDefinitionFixture(t *testing.T, relativeFile string, definitions ...dbgiDefinition) (*Server, string) { + t.Helper() + return newGeneratedDefinitionFixtureWith(t, func(domainPath string) testBeamChunks { + return testBeamChunks{dbgi: dbgiTerm(domainPath, relativeFile, definitions...)} + }) +} + +// newGeneratedDefinitionFixtureWith is newGeneratedDefinitionFixture with the +// domain BEAM's chunks chosen by the test. +func newGeneratedDefinitionFixtureWith(t *testing.T, chunks func(domainPath string) testBeamChunks) (*Server, string) { t.Helper() server, cleanup := setupTestServer(t) t.Cleanup(cleanup) @@ -127,25 +135,73 @@ func newGeneratedDefinitionFixture(t *testing.T, relativeFile string, definition indexFile(t, server.store, server.projectRoot, generatedDomainRel, generatedDomainSource) indexFile(t, server.store, server.projectRoot, generatedCallerRel, generatedCallerSource) domainPath := filepath.Join(server.projectRoot, generatedDomainRel) - past := time.Now().Add(-time.Hour) - if err := os.Chtimes(domainPath, past, past); err != nil { - t.Fatal(err) - } - - ebin := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin") - if err := os.MkdirAll(ebin, 0o755); err != nil { - t.Fatal(err) - } - beam := minimalBeamWithDbgi(dbgiTerm(domainPath, relativeFile, definitions...), + writeGeneratedBeam(t, server, "MyApp.Chat", chunks(domainPath), beamExport{"run_local", 0}, beamExport{"list_rooms", 0}, beamExport{"get_room_by_slug!", 1}, beamExport{"get_room_by_slug!", 2}, ) - if err := os.WriteFile(filepath.Join(ebin, "Elixir.MyApp.Chat.beam"), beam, 0o644); err != nil { + return server, domainPath +} + +func writeGeneratedBeam(t *testing.T, server *Server, module string, chunks testBeamChunks, exports ...beamExport) { + t.Helper() + ebin := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin") + if err := os.MkdirAll(ebin, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(ebin, "Elixir."+module+".beam"), buildTestBeam(chunks, exports...), 0o644); err != nil { t.Fatal(err) } - return server, domainPath +} + +// docsEntry is one function in a synthetic docs_v1 term. +type docsEntry struct { + name string + arity int + anno int +} + +// docsTerm encodes the docs_v1 term the compiler writes in the Docs chunk. Each +// entry's anno is the line its def was expanded at. +func docsTerm(entries ...docsEntry) string { + var out []byte + tuple := func(arity int) { out = append(out, 104, byte(arity)) } + atom := func(name string) { out = append(out, etfAtom(name)...) } + emptyMap := func() { out = append(out, 116, 0, 0, 0, 0) } + integer := func(value int) { + out = append(out, 98) + out = binary.BigEndian.AppendUint32(out, uint32(value)) + } + bin := func(text string) { + out = append(out, 109) + out = binary.BigEndian.AppendUint32(out, uint32(len(text))) + out = append(out, text...) + } + + tuple(7) + atom("docs_v1") + integer(1) + atom("elixir") + bin("text/markdown") + atom("none") + emptyMap() + out = append(out, etfListHeader(len(entries))...) + for _, entry := range entries { + tuple(5) + tuple(3) + atom("function") + atom(entry.name) + integer(entry.arity) + integer(entry.anno) + out = append(out, etfListHeader(1)...) + bin(entry.name + "()") + out = append(out, 106) + atom("none") + emptyMap() + } + out = append(out, 106) + return string(out) } // stampedDefinitions is what a generator writes when it stamps each function @@ -265,18 +321,105 @@ func TestDefinitionGeneratedFunctionForeignLocationKeepsModuleLine(t *testing.T) expectSingleLocation(t, locations, domainPath, generatedModuleLine) } -// A BEAM older than its source describes lines that may have moved. The -// function still exists, so the module is still the answer, but not the line. -func TestDefinitionGeneratedFunctionStaleBEAMKeepsModuleLine(t *testing.T) { +// Dexter cannot compile the project, so a BEAM older than its source is the +// usual state while editing. Its line is from the last compile, which is +// still closer than the module line. +func TestDefinitionGeneratedFunctionStaleBEAMUsesRecordedLine(t *testing.T) { server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) future := time.Now().Add(time.Hour) if err := os.Chtimes(domainPath, future, future); err != nil { t.Fatal(err) } locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// A module compiled without debug info still has its Docs chunk, whose anno is +// the line the def was expanded at. The compile info says which file that line +// is in. +func TestDefinitionGeneratedFunctionUsesDocsLineWithoutDebugInfo(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixtureWith(t, func(domainPath string) testBeamChunks { + return testBeamChunks{ + docs: docsTerm(docsEntry{name: "get_room_by_slug!", arity: 1, anno: generatedDefineLine}), + source: domainPath, + } + }) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// Without compile info, a Docs anno does not say which file it is in. +func TestDefinitionGeneratedFunctionDocsLineWithoutSourceKeepsModuleLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixtureWith(t, func(string) testBeamChunks { + return testBeamChunks{docs: docsTerm(docsEntry{name: "get_room_by_slug!", arity: 1, anno: generatedDefineLine})} + }) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) expectSingleLocation(t, locations, domainPath, generatedModuleLine) } +// sourcelessModule is a generated module with no source row, as Spark creates +// for each DSL entity. It was compiled from a framework file on another machine. +const ( + sourcelessModule = "MyApp.Chat.Define" + sourcelessRecorded = "/build/agent/deps/shared_lib/lib/shared_lib/dsl/extension.ex" +) + +// The recorded path does not exist here, so it is rebased onto the project's +// deps, where the same file is. +func TestLookupNameSourcelessGeneratedModuleUsesCompiledSource(t *testing.T) { + for _, tc := range []struct { + name string + chunks testBeamChunks + }{ + {"debug info", testBeamChunks{ + source: sourcelessRecorded, + dbgi: dbgiTerm(sourcelessRecorded, "lib/shared_lib/dsl/extension.ex", dbgiDefinition{name: "build", arity: 1, line: 42}), + }}, + {"docs", testBeamChunks{ + source: sourcelessRecorded, + docs: docsTerm(docsEntry{name: "build", arity: 1, anno: 42}), + }}, + } { + t.Run(tc.name, func(t *testing.T) { + server, _ := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + source := filepath.Join(server.projectRoot, "deps", "shared_lib", "lib", "shared_lib", "dsl", "extension.ex") + if err := os.MkdirAll(filepath.Dir(source), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(source, []byte("defmodule SharedLib.Dsl.Extension do\nend\n"), 0o644); err != nil { + t.Fatal(err) + } + writeGeneratedBeam(t, server, sourcelessModule, tc.chunks, beamExport{"build", 1}) + + locations, err := server.LookupName(sourcelessModule, "build", NameLookupOptions{ExactModule: true}) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != source || locations[0].Line != 42 { + t.Fatalf("expected %s:42, got %#v", source, locations) + } + }) + } +} + +// If the compiled source is not in this checkout, the lexical parent is still +// the answer. +func TestLookupNameSourcelessGeneratedModuleMissingSourceKeepsParent(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + writeGeneratedBeam(t, server, sourcelessModule, testBeamChunks{ + source: sourcelessRecorded, + docs: docsTerm(docsEntry{name: "build", arity: 1, anno: 42}), + }, beamExport{"build", 1}) + + locations, err := server.LookupName(sourcelessModule, "build", NameLookupOptions{FallbackToModule: true}) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != domainPath || locations[0].Line != generatedModuleLine { + t.Fatalf("expected %s:%d, got %#v", domainPath, generatedModuleLine, locations) + } +} + // The debug info's line belongs to the file it was compiled from. If that is // not the file the index has for the module, the line means nothing there. func TestDefinitionGeneratedFunctionOtherSourceKeepsModuleLine(t *testing.T) { @@ -285,10 +428,10 @@ func TestDefinitionGeneratedFunctionOtherSourceKeepsModuleLine(t *testing.T) { ) // The absolute path must not match either. beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") - beam := minimalBeamWithDbgi(dbgiTerm(filepath.Join(server.projectRoot, "lib/my_app/other_chat.ex"), "lib/my_app/other_chat.ex", + beamFile := minimalBeamWithDbgi(dbgiTerm(filepath.Join(server.projectRoot, "lib/my_app/other_chat.ex"), "lib/my_app/other_chat.ex", dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "lib/my_app/other_chat.ex", keepLine: generatedDefineLine}), beamExport{"get_room_by_slug!", 1}) - if err := os.WriteFile(beamPath, beam, 0o644); err != nil { + if err := os.WriteFile(beamPath, beamFile, 0o644); err != nil { t.Fatal(err) } locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) From 8ada8d5b8ffd3f62d7351b71313d07295821286e Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Tue, 29 Sep 2026 18:16:40 -0400 Subject: [PATCH 03/16] Follow edits made since the compile to a generated function's line When the source, or an unsaved buffer, has changed since the BEAM was written, move each recorded line to the nearest call whose first argument is the function's name as an atom (`define :list_rooms`). That is the generic shape of a macro call that declares a name. Keep the recorded line when it still declares the function, when no line does, or when two matches are equally near. A current BEAM is never corrected. Also say that the Docs anno is often, but not always, the Dbgi line: Ash's code interfaces give the def its define line and the docs line 1. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- docs/architecture.md | 6 +- internal/lsp/generated_definition.go | 140 +++++++++++++++++++++- internal/lsp/generated_definition_test.go | 105 ++++++++++++++++ 4 files changed, 244 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ac400a..bf791a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line. A module compiled without debug info falls back to its Docs chunk annotation. A generated module with no source of its own, such as a Spark DSL entity, goes to the file it was compiled from, rebased onto the project's `deps` when it was built elsewhere. Ash code interfaces go to their `define` line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) +- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line. When the source or an unsaved buffer has changed since the compile, the line follows the edit to the nearest call that declares the function by name as its first atom argument (`define :list_rooms`); if there is no such call, or two are equally near, the recorded line stays. A module compiled without debug info falls back to its Docs chunk annotation, which is often, but not always, the same line. A generated module with no source of its own, such as a Spark DSL entity, goes to the file it was compiled from, rebased onto the project's `deps` when it was built elsewhere. Ash code interfaces go to their `define` line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) - **`--root`/`-C` names the workspace on every command** — `dexter init --root ~/project`, `dexter lookup --root ~/project MyApp.Repo`, and `dexter lsp --root ~/project` all run as if they had been started in that directory, so an agent, a script, or an editor wrapper can index or query a project from anywhere. Relative paths, a `reindex` target included, resolve from the named root diff --git a/docs/architecture.md b/docs/architecture.md index 0dd0d9f..5bb9271 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -143,9 +143,11 @@ A generated function has no source definition, but the compiled module's `Dbgi` Each def carries two locations. Its `:line` is the line in the module's own file that was being compiled when the def was produced: the macro call, or wherever a before-compile hook ran. A `file: {path, line}` entry, left by `@file` or `quote location: :keep`, is preferred when `path` is the module's own source, because that is where the code asked for the function. When `path` is another file it is the generator's implementation, and `:line` is used instead. Ash's code interfaces take the second shape: each generated def is expanded at its `define` line, so `:line` is that line and `file` still names `code_interface.ex`, which keeps stacktraces unchanged. No framework is recognized by name. -`generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (absolute path, or relative path as a suffix for a moved or symlinked project), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is at worst a few lines off, which is still closer than the module line. +`generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (absolute path, or relative path as a suffix for a moved or symlinked project), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is closer than the module line. -A module compiled without debug info still has its Docs chunk, whose per-entry anno is the same `:line` (`beam.Function.Line`). It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually a framework file, so when that path does not exist here it is rebased onto the project by the tail below the application's `lib` directory (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. If no file is found, the lexical parent stays the answer. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. +A stale line is corrected where that can be done reliably. `correctLineDrift` runs only when the text may differ from the compiled text: the file is newer than the BEAM, or the open buffer differs from the file. It then looks for the nearest line that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, or a later argument does not count. The recorded line is kept when it still declares the function, when no line does, or when two matches are equally near. A current BEAM is never corrected, because its lines are exact even where the declaring call does not spell the name, as with a route helper. + +A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually a framework file, so when that path does not exist here it is rebased onto the project by the tail below the application's `lib` directory (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. If no file is found, the lexical parent stays the answer. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. The debug info and compile info are read on the first definition request that needs them and memoized on the module's generated-function cache entry, so the BEAM stamp invalidates them with everything else. A module compiled without debug info, or an Erlang module, is memoized as empty. diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index 49c535e..e1b4313 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -21,6 +21,10 @@ type generatedDefinitionSources struct { // compileSource is the source path from the CInf chunk: the file the module // was compiled from, as it was spelled on the machine that compiled it. compileSource string + + // beamMtime is when the BEAM was written, to tell whether its lines can + // have drifted from the source. + beamMtime int64 } // generatedDefinitionSourcesFor returns a compiled module's definition sources. @@ -41,7 +45,7 @@ func (s *Server) generatedDefinitionSourcesFor(module, beamPath string) (generat if !started.IsZero() { s.debugf("Generated BEAM definition sources: module=%s definitions=%d error=%v source=%q total=%s", module, len(info.Lines), err, source, time.Since(started).Round(time.Microsecond)) } - entry.definitionSources = generatedDefinitionSources{debugInfo: info, compileSource: source} + entry.definitionSources = generatedDefinitionSources{debugInfo: info, compileSource: source, beamMtime: entry.beamStamp.mtime} entry.definitionSourcesResolved = true s.generatedCache.put(module, entry) } @@ -63,7 +67,9 @@ func (s *Server) generatedDefinitionSourcesFor(module, beamPath string) (generat // the module result does not. The BEAM may be older than the source. Dexter // cannot compile the project, and a line from the last compile is usually // right and at worst a few lines off, which is still closer than the module -// line. Anything else keeps the module result. +// line. When the source has changed since the compile, correctLineDrift moves +// each line to follow the edit where it can. Anything else keeps the module +// result. func (s *Server) generatedDefinitionResultsFor(module, beamPath string, functions []beam.Function) (results []store.LookupResult, precise bool) { results = s.generatedDefinitionResults(module) if len(functions) == 0 { @@ -78,14 +84,14 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function continue } if lines := generatedFunctionLines(module, result.FilePath, result.Line, sources, functions); len(lines) > 0 { - return lines, true + return s.correctLineDrift(result.FilePath, sources.beamMtime, lines, functions), true } return results, false } if own, err := s.store.LookupModule(module); err == nil && len(own) == 0 { if source := s.compiledSourceFile(sources.compileSource); source != "" { if lines := generatedFunctionLines(module, source, 1, sources, functions); len(lines) > 0 { - return lines, true + return s.correctLineDrift(source, sources.beamMtime, lines, functions), true } } } @@ -96,8 +102,9 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function // afterLine that the compiled module records for functions in file. // // The Dbgi line is preferred because it also honors `@file`. The Docs chunk -// annotation is the fallback: it is the same line for a def a macro expanded, -// and a module compiled without debug info still has it. +// annotation is the fallback, because a module compiled without debug info +// still has it. It is often the line the def was expanded at, but not always: +// a generator can give the def one line and its docs another. func generatedFunctionLines(module, file string, afterLine int, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { useDebugInfo := len(sources.debugInfo.Lines) > 0 var lines []store.LookupResult @@ -120,6 +127,127 @@ func generatedFunctionLines(module, file string, afterLine int, sources generate return lines } +// correctLineDrift moves recorded lines to follow edits made since the compile. +// +// A BEAM older than its source, or a buffer with unsaved changes, can describe +// lines that have moved. Dexter cannot compile the project, so it looks for +// the declaration in the current text instead: a call whose first argument is +// the function's name as an atom, which is how a macro call names what it +// declares (`define :list_rooms`, `field :email`). The nearest match wins. +// The recorded line is kept when it still declares the function, when no line +// does, or when two matches are equally near. A BEAM that is current is never +// corrected, because its lines are exact even where the declaration does not +// spell the name. +func (s *Server) correctLineDrift(path string, beamMtime int64, results []store.LookupResult, functions []beam.Function) []store.LookupResult { + text, ok := s.driftedSourceText(path, beamMtime) + if !ok { + return results + } + lines := strings.Split(text, "\n") + names := make(map[int]string, len(functions)) + for _, function := range functions { + names[function.Arity] = function.Name + } + out := results[:0] + for _, result := range results { + if line := nearestDeclaringLine(lines, result.Line, names[result.Arity]); line != result.Line { + if slices.ContainsFunc(out, func(r store.LookupResult) bool { return r.Line == line }) { + continue + } + result.Line = line + } + out = append(out, result) + } + return out +} + +// driftedSourceText returns the current text of path when it may differ from +// the text the BEAM was compiled from: the open buffer if it has unsaved +// changes, or the file if it was written after the BEAM. +func (s *Server) driftedSourceText(path string, beamMtime int64) (string, bool) { + open, isOpen := s.docs.Get(string(pathToURI(path))) + source := statFileStamp(path) + stale := source.exists && source.mtime > beamMtime + if !isOpen && !stale { + return "", false + } + disk, err := os.ReadFile(path) + switch { + case isOpen && (err != nil || open != string(disk)): + return open, true + case err == nil && stale: + return string(disk), true + } + return "", false +} + +// nearestDeclaringLine returns the 1-based line nearest to recorded that +// declares function by name, or recorded when that is ambiguous or not found. +func nearestDeclaringLine(lines []string, recorded int, function string) int { + if function == "" || recorded < 1 { + return recorded + } + names := []string{function} + if base := strings.TrimRight(function, "!?"); base != function && base != "" { + names = append(names, base) + } + declares := func(line int) bool { + return line >= 1 && line <= len(lines) && lineDeclaresAtom(lines[line-1], names) + } + if declares(recorded) { + return recorded + } + for distance := 1; distance < len(lines); distance++ { + above, below := declares(recorded-distance), declares(recorded+distance) + switch { + case above && below: + return recorded + case above: + return recorded - distance + case below: + return recorded + distance + } + } + return recorded +} + +// lineDeclaresAtom reports whether text is a call whose first argument is one +// of names as an atom: `define :list_rooms`, `field :email, :string`, or +// `Lib.define(:name)`. That is the shape of a macro call that declares a +// name. An atom anywhere else, such as a keyword value or a typespec, is not. +func lineDeclaresAtom(text string, names []string) bool { + rest := strings.TrimLeft(text, " \t") + callee := 0 + for callee < len(rest) && (isIdentifierByte(rest[callee]) || rest[callee] == '.') { + callee++ + } + if callee == 0 || rest[0] == ':' || (rest[0] >= '0' && rest[0] <= '9') { + return false + } + args := strings.TrimLeft(rest[callee:], " \t") + if len(args) == len(rest[callee:]) { + // No space after the callee: only a parenthesized call qualifies. + if !strings.HasPrefix(args, "(") { + return false + } + } + args = strings.TrimLeft(strings.TrimPrefix(args, "("), " \t") + for _, name := range names { + if strings.HasPrefix(args, ":"+name) { + end := 1 + len(name) + if end == len(args) || !isIdentifierByte(args[end]) { + return true + } + } + } + return false +} + +func isIdentifierByte(b byte) bool { + return b == '_' || b == '!' || b == '?' || b == '@' || + (b >= 'a' && b <= 'z') || (b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9') +} + // describe reports whether the recorded lines belong to path. When the module // has debug info, its file names are the test; otherwise the compile info's. func (sources generatedDefinitionSources) describe(path, projectRoot string) bool { diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index 3d1f56a..c356ecc 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -5,6 +5,7 @@ import ( "encoding/binary" "os" "path/filepath" + "strings" "testing" "time" @@ -456,3 +457,107 @@ func TestPrepareCallHierarchyGeneratedFunctionUsesDebugInfoLine(t *testing.T) { t.Fatalf("expected the call hierarchy item at %s:%d, got %#v", domainPath, generatedDefineLine, items) } } + +func TestNearestDeclaringLine(t *testing.T) { + lines := strings.Split(`defmodule MyApp.Chat do + resources do + define :list_rooms, action: :read + define :get_room_by_slug, action: :read, get_by: [:slug] + end + + def run, do: get_room_by_slug!("lounge") + def typed(x :: :get_room_by_slug), do: x + def other, do: [key:get_room_by_slug] +end`, "\n") + for _, tc := range []struct { + name string + recorded int + function string + want int + }{ + {"recorded line still declares it", 4, "get_room_by_slug!", 4}, + {"declaration moved down", 2, "get_room_by_slug!", 4}, + {"declaration moved up", 7, "get_room_by_slug!", 4}, + {"full name as atom", 1, "list_rooms", 3}, + {"name nowhere keeps the line", 5, "create_room", 5}, + {"equally near matches keep the line", 3, "tie", 3}, + } { + t.Run(tc.name, func(t *testing.T) { + text := lines + if tc.function == "tie" { + text = []string{"define :tie", "", "define :tie"} + tc.recorded = 2 + tc.want = 2 + } + if got := nearestDeclaringLine(text, tc.recorded, tc.function); got != tc.want { + t.Errorf("nearestDeclaringLine(%d, %q) = %d, want %d", tc.recorded, tc.function, got, tc.want) + } + }) + } + + names := []string{"get_room_by_slug!", "get_room_by_slug"} + for _, text := range []string{ + ` define :get_room_by_slug, action: :read`, + ` SharedLib.Interface.define(:get_room_by_slug)`, + `define :get_room_by_slug`, + } { + if !lineDeclaresAtom(text, names) { + t.Errorf("lineDeclaresAtom(%q) = false, want true", text) + } + } + // A call, a typespec, a keyword key or value, a longer atom, and an atom + // that is not the first argument are not declarations. + for _, text := range []string{ + `get_room_by_slug!("x")`, + `def typed(x :: :get_room_by_slug), do: x`, + `[key:get_room_by_slug]`, + `read action: :get_room_by_slug`, + `define :get_room_by_slug_extra`, + `define :other, as: :get_room_by_slug`, + `:get_room_by_slug`, + `define:get_room_by_slug`, + } { + if lineDeclaresAtom(text, names) { + t.Errorf("lineDeclaresAtom(%q) = true, want false", text) + } + } +} + +// Lines added above the declaration after the compile: the recorded line has +// drifted, and the current text still says where the declaration is. +func TestDefinitionGeneratedFunctionFollowsDriftInStaleSource(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + past := time.Now().Add(-time.Hour) + beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") + if err := os.Chtimes(beamPath, past, past); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(domainPath, []byte("# one\n# two\n"+generatedDomainSource), 0o644); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine+2) +} + +// Unsaved edits move lines too, with nothing on disk newer than the BEAM. +func TestDefinitionGeneratedFunctionFollowsDriftInOpenBuffer(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + server.docs.Set(string(uri.File(domainPath)), "# one\n"+generatedDomainSource) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine+1) +} + +// A current BEAM's lines are exact, even when the declaring line does not +// spell the name and another line does. +func TestDefinitionGeneratedFunctionCurrentBEAMKeepsRecordedLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 6, keepFile: "deps/ash/lib/ash/code_interface.ex", keepLine: 1112}, + ) + future := time.Now().Add(time.Hour) + beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") + if err := os.Chtimes(beamPath, future, future); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, 6) +} From 2fab38b5fed0ff5f27f16a91b2e242828c9f27da Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Tue, 29 Sep 2026 18:28:23 -0400 Subject: [PATCH 04/16] Fix edge cases in generated definition lines found by testing - Follow drift before the "after the module line" check, so lines added above defmodule no longer drop the recorded line. - Search to both ends of a file that got shorter than the recorded line, and drop a line past the end with nothing to move to. - Treat only open buffers as unsaved edits; a file cached by another request is not one. - Skip heredoc lines and module attributes when looking for a declaration. - Prefer this checkout over the recorded path when a _build was copied from another checkout or worktree, and try the recorded path's tails under the root first, which covers umbrella apps and moved projects. - Decode CInf charlists as Unicode codepoints, so non-ASCII paths work. - Bound the Dbgi definitions preallocation against a corrupt count. - Do not overwrite a newer BEAM's cache entry with an older one. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/beam/compile_info.go | 19 ++- internal/beam/compile_info_test.go | 30 +++++ internal/beam/debug_info.go | 8 +- internal/beam/debug_info_test.go | 16 +++ internal/lsp/generated_definition.go | 146 +++++++++++++++------ internal/lsp/generated_definition_test.go | 150 ++++++++++++++++++++++ 6 files changed, 324 insertions(+), 45 deletions(-) diff --git a/internal/beam/compile_info.go b/internal/beam/compile_info.go index d5a3603..ab19e28 100644 --- a/internal/beam/compile_info.go +++ b/internal/beam/compile_info.go @@ -2,6 +2,7 @@ package beam import ( "errors" + "unicode/utf8" ) // ReadSourcePath returns the source file recorded in a BEAM's compile info @@ -83,8 +84,9 @@ func parseCompileSource(raw []byte) (string, bool) { } // readStringTerm reads a path-shaped term: a binary, a string, or a charlist of -// small integers. The compiler records :source as a charlist, which is stored as -// a compact string term; other producers of the same chunk use a binary. +// codepoints, returned as UTF-8. The compiler records :source as a charlist, +// stored as a compact string term when every codepoint is below 256 and as a +// list otherwise; other producers of the same chunk use a binary. func (r *etfReader) readStringTerm() (string, error) { tag, err := r.peekTag() if err != nil { @@ -110,9 +112,14 @@ func (r *etfReader) readStringTerm() (string, error) { if err := r.need(n); err != nil { return "", err } - s := string(r.buf[r.pos : r.pos+n]) + // Each byte is one codepoint below 256, not a UTF-8 byte: a path with + // `é` in it stores 0xE9. + out := make([]byte, 0, n) + for _, b := range r.buf[r.pos : r.pos+n] { + out = utf8.AppendRune(out, rune(b)) + } r.pos += n - return s, nil + return string(out), nil case tagList: count, hasTail, err := r.enterList() @@ -130,10 +137,10 @@ func (r *etfReader) readStringTerm() (string, error) { if err != nil { return "", err } - if n < 0 || n > 255 { + if n < 0 || n > utf8.MaxRune || !utf8.ValidRune(rune(n)) { return "", errors.New("charlist element out of range") } - out = append(out, byte(n)) + out = utf8.AppendRune(out, rune(n)) } if hasTail { if err := r.skip(); err != nil { diff --git a/internal/beam/compile_info_test.go b/internal/beam/compile_info_test.go index 125c43b..5903a69 100644 --- a/internal/beam/compile_info_test.go +++ b/internal/beam/compile_info_test.go @@ -113,3 +113,33 @@ func TestReadSourcePath(t *testing.T) { t.Errorf("ReadSourcePath without CInf = %q, want no source", got) } } + +// Erlang records :source as codepoints. A path with `é` is stored as a string +// term whose byte is 0xE9, and one with `日本` as a list, because 26085 does not +// fit in a byte. Both must come back as UTF-8. +func TestParseCompileSourceUnicode(t *testing.T) { + const want = "/home/zoë/日本/lib/app.ex" + codepoints := []rune(want) + list := compileInfoTerm(t, func(w *etfTestWriter, _ string) { + w.listHeader(len(codepoints)) + for _, c := range codepoints { + w.smallInt(int(c)) + } + w.nil() + }) + if got, ok := parseCompileSource(list); !ok || got != want { + t.Errorf("list: source = %q, %v; want %q", got, ok, want) + } + + const latin = "/home/zoë/lib/app.ex" + short := compileInfoTerm(t, func(w *etfTestWriter, _ string) { + var bytes []byte + for _, c := range latin { + bytes = append(bytes, byte(c)) + } + w.string(string(bytes)) + }) + if got, ok := parseCompileSource(short); !ok || got != latin { + t.Errorf("string term: source = %q, %v; want %q", got, ok, latin) + } +} diff --git a/internal/beam/debug_info.go b/internal/beam/debug_info.go index ce08614..2972b48 100644 --- a/internal/beam/debug_info.go +++ b/internal/beam/debug_info.go @@ -11,6 +11,9 @@ import ( // and still a bounded Go stack on a corrupt file. const maxDebugInfoDepth = 2048 +// maxPreallocatedSites bounds the definitions slice allocated up front. +const maxPreallocatedSites = 1024 + // FunctionKey names one callable by name and arity. type FunctionKey struct { Name string @@ -179,7 +182,10 @@ func readDefinitionSites(r *etfReader) ([]definitionSite, error) { if err != nil { return nil, err } - sites := make([]definitionSite, 0, count) + // A corrupt count is bounded only by the chunk size, which can still ask + // for gigabytes of sites before the walk fails. Real modules have far + // fewer definitions, so the slice grows past this if it must. + sites := make([]definitionSite, 0, min(count, maxPreallocatedSites)) for i := int64(0); i < count; i++ { arity, err := r.enterTuple() if err != nil { diff --git a/internal/beam/debug_info_test.go b/internal/beam/debug_info_test.go index e5dfabb..e660c22 100644 --- a/internal/beam/debug_info_test.go +++ b/internal/beam/debug_info_test.go @@ -386,3 +386,19 @@ end t.Errorf("documented = %+v, want %d functions", documented, len(want)) } } + +// A corrupt definitions count must fail the parse, not allocate for it first. +func TestParseDebugInfoHugeDefinitionCount(t *testing.T) { + var w etfTestWriter + w.smallTuple(3) + w.atom("debug_info_v1") + w.atom("elixir_erl") + w.smallTuple(3) + w.atom("elixir_v1") + w.mapHeader(1) + w.atom("definitions") + w.listHeader(1 << 30) + if _, err := parseDebugInfo(w.buf); err == nil { + t.Fatal("parseDebugInfo accepted a truncated definitions list") + } +} diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index e1b4313..b5081c5 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -47,7 +47,11 @@ func (s *Server) generatedDefinitionSourcesFor(module, beamPath string) (generat } entry.definitionSources = generatedDefinitionSources{debugInfo: info, compileSource: source, beamMtime: entry.beamStamp.mtime} entry.definitionSourcesResolved = true - s.generatedCache.put(module, entry) + // Another request may have loaded a newer BEAM while this one read the + // chunks. Its entry must not be replaced with this older one. + if current, ok := s.generatedCache.get(module); ok && current.beamPath == entry.beamPath && current.beamStamp == entry.beamStamp { + s.generatedCache.put(module, entry) + } } return entry.definitionSources, true } @@ -83,29 +87,38 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function if !sources.describe(result.FilePath, s.projectRoot) { continue } - if lines := generatedFunctionLines(module, result.FilePath, result.Line, sources, functions); len(lines) > 0 { - return s.correctLineDrift(result.FilePath, sources.beamMtime, lines, functions), true + if lines := s.recordedLinesIn(module, result.FilePath, result.Line, sources, functions); len(lines) > 0 { + return lines, true } return results, false } if own, err := s.store.LookupModule(module); err == nil && len(own) == 0 { if source := s.compiledSourceFile(sources.compileSource); source != "" { - if lines := generatedFunctionLines(module, source, 1, sources, functions); len(lines) > 0 { - return s.correctLineDrift(source, sources.beamMtime, lines, functions), true + if lines := s.recordedLinesIn(module, source, 1, sources, functions); len(lines) > 0 { + return lines, true } } } return results, false } -// generatedFunctionLines returns a result for each distinct line after -// afterLine that the compiled module records for functions in file. +// recordedLinesIn returns a result for each distinct line after afterLine that +// the compiled module records for functions in file, corrected for edits made +// since the compile. afterLine is the module's line in the current text, so it +// is compared after the correction: lines added above the module move both. +func (s *Server) recordedLinesIn(module, file string, afterLine int, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { + lines := s.correctLineDrift(file, sources.beamMtime, generatedFunctionLines(module, file, sources, functions), functions) + return slices.DeleteFunc(lines, func(r store.LookupResult) bool { return r.Line <= afterLine }) +} + +// generatedFunctionLines returns a result for each distinct line that the +// compiled module records for functions in file. // // The Dbgi line is preferred because it also honors `@file`. The Docs chunk // annotation is the fallback, because a module compiled without debug info // still has it. It is often the line the def was expanded at, but not always: // a generator can give the def one line and its docs another. -func generatedFunctionLines(module, file string, afterLine int, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { +func generatedFunctionLines(module, file string, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { useDebugInfo := len(sources.debugInfo.Lines) > 0 var lines []store.LookupResult for _, function := range functions { @@ -113,7 +126,7 @@ func generatedFunctionLines(module, file string, afterLine int, sources generate if useDebugInfo { line = sources.debugInfo.Lines[beam.FunctionKey{Name: function.Name, Arity: function.Arity}] } - if line <= afterLine || slices.ContainsFunc(lines, func(r store.LookupResult) bool { return r.Line == line }) { + if line <= 0 || slices.ContainsFunc(lines, func(r store.LookupResult) bool { return r.Line == line }) { continue } lines = append(lines, store.LookupResult{ @@ -135,27 +148,27 @@ func generatedFunctionLines(module, file string, afterLine int, sources generate // the function's name as an atom, which is how a macro call names what it // declares (`define :list_rooms`, `field :email`). The nearest match wins. // The recorded line is kept when it still declares the function, when no line -// does, or when two matches are equally near. A BEAM that is current is never -// corrected, because its lines are exact even where the declaration does not -// spell the name. +// does, or when two matches are equally near. A line past the end of the +// current text, with no declaration to move to, is dropped. A BEAM that is +// current is never corrected, because its lines are exact even where the +// declaration does not spell the name. func (s *Server) correctLineDrift(path string, beamMtime int64, results []store.LookupResult, functions []beam.Function) []store.LookupResult { text, ok := s.driftedSourceText(path, beamMtime) if !ok { return results } - lines := strings.Split(text, "\n") + lines := blankHeredocs(strings.Split(text, "\n")) names := make(map[int]string, len(functions)) for _, function := range functions { names[function.Arity] = function.Name } out := results[:0] for _, result := range results { - if line := nearestDeclaringLine(lines, result.Line, names[result.Arity]); line != result.Line { - if slices.ContainsFunc(out, func(r store.LookupResult) bool { return r.Line == line }) { - continue - } - result.Line = line + line := nearestDeclaringLine(lines, result.Line, names[result.Arity]) + if line > len(lines) || slices.ContainsFunc(out, func(r store.LookupResult) bool { return r.Line == line }) { + continue } + result.Line = line out = append(out, result) } return out @@ -165,7 +178,7 @@ func (s *Server) correctLineDrift(path string, beamMtime int64, results []store. // the text the BEAM was compiled from: the open buffer if it has unsaved // changes, or the file if it was written after the BEAM. func (s *Server) driftedSourceText(path string, beamMtime int64) (string, bool) { - open, isOpen := s.docs.Get(string(pathToURI(path))) + open, isOpen := s.docs.GetIfOpen(string(pathToURI(path))) source := statFileStamp(path) stale := source.exists && source.mtime > beamMtime if !isOpen && !stale { @@ -181,6 +194,33 @@ func (s *Server) driftedSourceText(path string, beamMtime int64) (string, bool) return "", false } +// blankHeredocs replaces the lines inside heredocs with empty lines, so that +// an example in a @moduledoc or @doc, such as `define :foo`, is not taken for +// a declaration. Line numbers do not change. +func blankHeredocs(lines []string) []string { + out := make([]string, len(lines)) + var open string + for i, line := range lines { + inside := open != "" + for _, delimiter := range []string{`"""`, "'''"} { + if open != "" && open != delimiter { + continue + } + if strings.Count(line, delimiter)%2 == 1 { + if open == "" { + open = delimiter + } else { + open = "" + } + } + } + if !inside && open == "" { + out[i] = line + } + } + return out +} + // nearestDeclaringLine returns the 1-based line nearest to recorded that // declares function by name, or recorded when that is ambiguous or not found. func nearestDeclaringLine(lines []string, recorded int, function string) int { @@ -197,7 +237,7 @@ func nearestDeclaringLine(lines []string, recorded int, function string) int { if declares(recorded) { return recorded } - for distance := 1; distance < len(lines); distance++ { + for distance := 1; recorded-distance >= 1 || recorded+distance <= len(lines); distance++ { above, below := declares(recorded-distance), declares(recorded+distance) switch { case above && below: @@ -221,7 +261,8 @@ func lineDeclaresAtom(text string, names []string) bool { for callee < len(rest) && (isIdentifierByte(rest[callee]) || rest[callee] == '.') { callee++ } - if callee == 0 || rest[0] == ':' || (rest[0] >= '0' && rest[0] <= '9') { + // A module attribute such as `@tag :slow` sets a value; it declares nothing. + if callee == 0 || rest[0] == ':' || rest[0] == '@' || (rest[0] >= '0' && rest[0] <= '9') { return false } args := strings.TrimLeft(rest[callee:], " \t") @@ -285,11 +326,16 @@ func debugInfoDescribesFile(info beam.DebugInfo, path string) bool { // generated module can live in one application while its source lives in // another: Spark creates Ash's entity modules, so the artifact sits in ash's // ebin while the file that generated it is under spark. +// +// A recorded path outside the project is tried last. It is right for a `path:` +// dependency, but a `_build` copied from another checkout or worktree records +// that checkout, which usually still exists: returning it would open the file +// in the wrong tree. func (s *Server) compiledSourceFile(recorded string) string { if recorded == "" { return "" } - if regularFileExists(recorded) { + if pathWithin(recorded, s.projectRoot) && regularFileExists(recorded) { return recorded } for _, candidate := range sourceRebaseCandidates(recorded, s.projectRoot) { @@ -297,33 +343,57 @@ func (s *Server) compiledSourceFile(recorded string) string { return candidate } } + if regularFileExists(recorded) { + return recorded + } return "" } +// pathWithin reports whether path is root or below it. +func pathWithin(path, root string) bool { + if root == "" { + return false + } + rel, err := filepath.Rel(root, path) + return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) +} + // sourceRebaseCandidates lists the plausible on-disk locations for a recorded -// source path. Every Mix layout is offered, because the artifact does not record -// which one it was: `lib//...` is the application's own repository, -// `deps//lib//...` is a dependency, and `deps//...` is the older -// vendored shape. +// source path, most specific first. +// +// First come the recorded path's own tails under the project root, longest +// first: a checkout that moved or was copied keeps its layout below the root, +// whether that is `lib/`, an umbrella's `apps//lib/`, or `deps/`. A tail +// needs at least one directory, so a bare file name never matches a file at +// the root. +// +// Then every Mix layout for the application named below the last `lib/`, +// because an artifact built elsewhere does not record which one applies here: +// `lib//...` is the application's own repository, `deps//lib//...` +// is a dependency, and `deps//...` is the older vendored shape. func sourceRebaseCandidates(recorded, projectRoot string) []string { if projectRoot == "" { return nil } slash := filepath.ToSlash(recorded) - idx := strings.LastIndex(slash, "/lib/") - if idx < 0 { - return nil + var candidates []string + parts := strings.Split(strings.TrimPrefix(slash, "/"), "/") + for i := 0; i+2 <= len(parts); i++ { + candidates = append(candidates, filepath.Join(projectRoot, filepath.FromSlash(strings.Join(parts[i:], "/")))) } - parts := strings.SplitN(slash[idx+len("/lib/"):], "/", 2) - if len(parts) != 2 || parts[0] == "" || parts[1] == "" { - return nil - } - app, tail := parts[0], filepath.FromSlash(parts[1]) - return []string{ - filepath.Join(projectRoot, "lib", app, tail), - filepath.Join(projectRoot, "deps", app, "lib", app, tail), - filepath.Join(projectRoot, "deps", app, tail), + + if idx := strings.LastIndex(slash, "/lib/"); idx >= 0 { + appTail := strings.SplitN(slash[idx+len("/lib/"):], "/", 2) + if len(appTail) == 2 && appTail[0] != "" && appTail[1] != "" { + app, tail := appTail[0], filepath.FromSlash(appTail[1]) + candidates = append(candidates, + filepath.Join(projectRoot, "lib", app, tail), + filepath.Join(projectRoot, "deps", app, "lib", app, tail), + filepath.Join(projectRoot, "deps", app, tail), + ) + } } + return slices.Compact(candidates) } // regularFileExists reports whether path names a readable regular file. diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index c356ecc..62f0a29 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -5,6 +5,7 @@ import ( "encoding/binary" "os" "path/filepath" + "slices" "strings" "testing" "time" @@ -516,6 +517,7 @@ end`, "\n") `define :other, as: :get_room_by_slug`, `:get_room_by_slug`, `define:get_room_by_slug`, + `@tag :get_room_by_slug`, } { if lineDeclaresAtom(text, names) { t.Errorf("lineDeclaresAtom(%q) = true, want false", text) @@ -561,3 +563,151 @@ func TestDefinitionGeneratedFunctionCurrentBEAMKeepsRecordedLine(t *testing.T) { locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) expectSingleLocation(t, locations, domainPath, 6) } + +// Lines added above the module move the module line too. The recorded line is +// compared with it only after following the edit. +func TestDefinitionGeneratedFunctionFollowsDriftAboveModule(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + header := strings.Repeat("# header\n", 10) + indexFile(t, server.store, server.projectRoot, generatedDomainRel, header+generatedDomainSource) + server.docs.Set(string(uri.File(domainPath)), header+generatedDomainSource) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine+10) +} + +// A file that became shorter than the recorded line is still searched from +// its end, and a line with nothing left to move to is not returned. +func TestNearestDeclaringLinePastEndOfFile(t *testing.T) { + lines := []string{"defmodule A do", " define :rename", "end"} + if got := nearestDeclaringLine(lines, 19, "rename"); got != 2 { + t.Errorf("nearestDeclaringLine past the end = %d, want 2", got) + } + if got := nearestDeclaringLine(lines, 19, "missing"); got != 19 { + t.Errorf("nearestDeclaringLine without a match = %d, want the recorded 19", got) + } +} + +func TestDefinitionGeneratedFunctionPastEndOfFileKeepsModuleLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + short := "defmodule MyApp.Chat do\nend\n" + indexFile(t, server.store, server.projectRoot, generatedDomainRel, short) + server.docs.Set(string(uri.File(domainPath)), short) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, 1) +} + +// A `_build` copied from another checkout records that checkout's path, which +// usually still exists. The copy in this project wins; a path outside the +// project is used only when the project has no copy, as for a `path:` +// dependency. +func TestLookupNameSourcelessGeneratedModulePrefersThisCheckout(t *testing.T) { + for _, tc := range []struct { + name string + localCopy bool + }{ + {"other checkout", true}, + {"path dependency", false}, + } { + t.Run(tc.name, func(t *testing.T) { + server, _ := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + outside := filepath.Join(t.TempDir(), "other_checkout", "lib", "shared_lib", "dsl.ex") + local := filepath.Join(server.projectRoot, "lib", "shared_lib", "dsl.ex") + paths := []string{outside} + if tc.localCopy { + paths = append(paths, local) + } + for _, path := range paths { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte("defmodule SharedLib.Dsl do\nend\n"), 0o644); err != nil { + t.Fatal(err) + } + } + writeGeneratedBeam(t, server, sourcelessModule, testBeamChunks{ + source: outside, + docs: docsTerm(docsEntry{name: "build", arity: 1, anno: 42}), + }, beamExport{"build", 1}) + + want := outside + if tc.localCopy { + want = local + } + locations, err := server.LookupName(sourcelessModule, "build", NameLookupOptions{ExactModule: true}) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != want || locations[0].Line != 42 { + t.Fatalf("expected %s:42, got %#v", want, locations) + } + }) + } +} + +func TestSourceRebaseCandidates(t *testing.T) { + root := filepath.FromSlash("/work/copy") + for _, tc := range []struct { + name, recorded, want string + }{ + {"moved umbrella app", "/work/orig/apps/gen_lib/lib/gen_lib.ex", "/work/copy/apps/gen_lib/lib/gen_lib.ex"}, + {"moved project", "/work/orig/lib/my_app/chat.ex", "/work/copy/lib/my_app/chat.ex"}, + {"dependency built elsewhere", "/build/agent/deps/spark/lib/spark/dsl/extension.ex", "/work/copy/deps/spark/lib/spark/dsl/extension.ex"}, + {"dependency built in its own repository", "/build/spark/lib/spark/dsl/extension.ex", "/work/copy/deps/spark/lib/spark/dsl/extension.ex"}, + } { + t.Run(tc.name, func(t *testing.T) { + got := sourceRebaseCandidates(filepath.FromSlash(tc.recorded), root) + if !slices.Contains(got, filepath.FromSlash(tc.want)) { + t.Errorf("candidates = %v, want %s among them", got, tc.want) + } + }) + } + // A bare file name must not match a file at the project root. + if got := sourceRebaseCandidates(filepath.FromSlash("/elsewhere/mix.exs"), root); slices.Contains(got, filepath.Join(root, "mix.exs")) { + t.Errorf("candidates = %v, must not include the root's mix.exs", got) + } +} + +// An example in a @moduledoc is not a declaration, even when it is nearer. +func TestNearestDeclaringLineSkipsHeredocs(t *testing.T) { + lines := blankHeredocs(strings.Split(`defmodule MyApp.Chat do + @moduledoc """ + Example: + + define :get_room_by_slug + """ + + resources do + define :get_room_by_slug + end +end`, "\n")) + if got := nearestDeclaringLine(lines, 4, "get_room_by_slug!"); got != 9 { + t.Errorf("nearestDeclaringLine = %d, want 9, past the @moduledoc example", got) + } +} + +// A file that a request read from disk, without opening it, is only a cache. +// If the file then changes and is compiled again, the cache is old text, not +// unsaved edits, and a current BEAM's line must not be corrected against it. +func TestDefinitionGeneratedFunctionIgnoresCachedFile(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + if _, ok := server.docs.GetOrLoad(string(uri.File(domainPath))); !ok { + t.Fatal("GetOrLoad did not cache the file") + } + edited := strings.Replace(generatedDomainSource, " resources do\n", " resources do\n # added\n", 1) + if err := os.WriteFile(domainPath, []byte(edited), 0o644); err != nil { + t.Fatal(err) + } + indexFile(t, server.store, server.projectRoot, generatedDomainRel, edited) + // Compiled after the edit: the define is now on line 9, and the BEAM says so. + const generator = "deps/ash/lib/ash/code_interface.ex" + writeGeneratedBeam(t, server, "MyApp.Chat", testBeamChunks{dbgi: dbgiTerm(domainPath, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: generatedDefineLine + 1, keepFile: generator, keepLine: 1112}, + )}, beamExport{"get_room_by_slug!", 1}) + future := time.Now().Add(time.Hour) + beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") + if err := os.Chtimes(beamPath, future, future); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine+1) +} From a162c0321031243b657432450c63e7ab4e8a519c Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Tue, 29 Sep 2026 18:28:38 -0400 Subject: [PATCH 05/16] Describe the rebase order and drift rules in the architecture notes Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/architecture.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 5bb9271..cfe5345 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -145,9 +145,9 @@ Each def carries two locations. Its `:line` is the line in the module's own file `generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (absolute path, or relative path as a suffix for a moved or symlinked project), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is closer than the module line. -A stale line is corrected where that can be done reliably. `correctLineDrift` runs only when the text may differ from the compiled text: the file is newer than the BEAM, or the open buffer differs from the file. It then looks for the nearest line that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, or a later argument does not count. The recorded line is kept when it still declares the function, when no line does, or when two matches are equally near. A current BEAM is never corrected, because its lines are exact even where the declaring call does not spell the name, as with a route helper. +A stale line is corrected where that can be done reliably. `correctLineDrift` runs only when the text may differ from the compiled text: the file is newer than the BEAM, or a buffer open in the editor differs from the file. A file that another request only cached does not count. It runs before the check against the module's line, because lines added above `defmodule` move both. It then looks for the nearest line, outside heredocs, that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, or a later argument does not count. The recorded line is kept when it still declares the function, when no line does, or when two matches are equally near. A current BEAM is never corrected, because its lines are exact even where the declaring call does not spell the name, as with a route helper. -A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually a framework file, so when that path does not exist here it is rebased onto the project by the tail below the application's `lib` directory (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. If no file is found, the lexical parent stays the answer. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. +A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually the generator's own file. A recorded path inside the project is used as it is. Otherwise it is rebased: first each tail of the recorded path under the project root, longest first, which covers a moved or copied checkout and an umbrella's `apps//lib`, then the Mix layouts for the application named below the last `lib/` (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. A recorded path outside the project is the last choice: it is right for a `path:` dependency, but a `_build` copied from another worktree records that worktree, which usually still exists. If no file is found, the lexical parent stays the answer. `CInf` records the path as codepoints, which are decoded to UTF-8. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. The debug info and compile info are read on the first definition request that needs them and memoized on the module's generated-function cache entry, so the BEAM stamp invalidates them with everything else. A module compiled without debug info, or an Erlang module, is memoized as empty. From 3d29ed4a3d79108cb47e4621002d11cc57596135 Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Tue, 29 Sep 2026 18:36:46 -0400 Subject: [PATCH 06/16] Keep line drift inside the owning module, and pick the best-matching file - Follow a stale line only to a declaration in the body of the module that owns it, found with the tokenizer, so a sibling module in the same file that declares the same name is never the answer. A generated module with no source row has no owner in the generator's file, so its line is not corrected there. - When a module has rows in more than one file, as two umbrella apps can, choose the row that shares the most trailing path components with the recorded source. If two rows match equally, keep the module line. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/architecture.md | 4 +- internal/lsp/generated_completion.go | 9 +- internal/lsp/generated_definition.go | 186 ++++++++++++++++++---- internal/lsp/generated_definition_test.go | 102 +++++++++++- 4 files changed, 257 insertions(+), 44 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index cfe5345..59fc8ef 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -143,9 +143,9 @@ A generated function has no source definition, but the compiled module's `Dbgi` Each def carries two locations. Its `:line` is the line in the module's own file that was being compiled when the def was produced: the macro call, or wherever a before-compile hook ran. A `file: {path, line}` entry, left by `@file` or `quote location: :keep`, is preferred when `path` is the module's own source, because that is where the code asked for the function. When `path` is another file it is the generator's implementation, and `:line` is used instead. Ash's code interfaces take the second shape: each generated def is expanded at its `define` line, so `:line` is that line and `file` still names `code_interface.ex`, which keeps stacktraces unchanged. No framework is recognized by name. -`generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (absolute path, or relative path as a suffix for a moved or symlinked project), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is closer than the module line. +`generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (the same absolute path, or else the most path components shared from the end, which covers a moved, copied, or symlinked project; when a module has rows in two files, as two umbrella apps can, the most specific match wins, and an equal match trusts neither), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is closer than the module line. -A stale line is corrected where that can be done reliably. `correctLineDrift` runs only when the text may differ from the compiled text: the file is newer than the BEAM, or a buffer open in the editor differs from the file. A file that another request only cached does not count. It runs before the check against the module's line, because lines added above `defmodule` move both. It then looks for the nearest line, outside heredocs, that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, or a later argument does not count. The recorded line is kept when it still declares the function, when no line does, or when two matches are equally near. A current BEAM is never corrected, because its lines are exact even where the declaring call does not spell the name, as with a route helper. +A stale line is corrected where that can be done reliably. `correctLineDrift` runs only when the text may differ from the compiled text: the file is newer than the BEAM, or a buffer open in the editor differs from the file. A file that another request only cached does not count. It runs before the check against the module's line, because lines added above `defmodule` move both. It then looks for the nearest line, outside heredocs and inside the body of the module that owns the declarations (found with the tokenizer, so a sibling module in the same file never matches), that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, or a later argument does not count. The recorded line is kept when it still declares the function, when no line does, or when two matches are equally near. A current BEAM is never corrected, because its lines are exact even where the declaring call does not spell the name, as with a route helper. A generated module with no source row has no owner in its generator's file, so its line is not corrected. A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually the generator's own file. A recorded path inside the project is used as it is. Otherwise it is rebased: first each tail of the recorded path under the project root, longest first, which covers a moved or copied checkout and an umbrella's `apps//lib`, then the Mix layouts for the application named below the last `lib/` (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. A recorded path outside the project is the last choice: it is right for a `path:` dependency, but a `_build` copied from another worktree records that worktree, which usually still exists. If no file is found, the lexical parent stays the answer. `CInf` records the path as codepoints, which are decoded to UTF-8. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. diff --git a/internal/lsp/generated_completion.go b/internal/lsp/generated_completion.go index 2ef2396..d89140b 100644 --- a/internal/lsp/generated_completion.go +++ b/internal/lsp/generated_completion.go @@ -699,11 +699,12 @@ func generatedFunctionsNamed(functions []beam.Function, name string) []beam.Func // generatedDefinitionResults returns the closest source-backed module for a // generated provider. Generated nested modules have no source row, so walking // their lexical parents yields a stable artifact-level destination without -// knowing which framework created them. -func (s *Server) generatedDefinitionResults(module string) []store.LookupResult { +// knowing which framework created them. owner is the module those results +// belong to: module itself, or the lexical parent that was found. +func (s *Server) generatedDefinitionResults(module string) (results []store.LookupResult, owner string) { for candidate := module; candidate != ""; { if results, err := s.store.LookupModule(candidate); err == nil && len(results) > 0 { - return results + return results, candidate } dot := strings.LastIndexByte(candidate, '.') if dot < 0 { @@ -711,7 +712,7 @@ func (s *Server) generatedDefinitionResults(module string) []store.LookupResult } candidate = candidate[:dot] } - return nil + return nil, "" } // filterGeneratedProviderReferences removes the conservative false positives diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index b5081c5..5d6db58 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -8,6 +8,7 @@ import ( "time" "github.com/remoteoss/dexter/internal/beam" + "github.com/remoteoss/dexter/internal/parser" "github.com/remoteoss/dexter/internal/store" ) @@ -75,7 +76,7 @@ func (s *Server) generatedDefinitionSourcesFor(module, beamPath string) (generat // each line to follow the edit where it can. Anything else keeps the module // result. func (s *Server) generatedDefinitionResultsFor(module, beamPath string, functions []beam.Function) (results []store.LookupResult, precise bool) { - results = s.generatedDefinitionResults(module) + results, owner := s.generatedDefinitionResults(module) if len(functions) == 0 { return results, false } @@ -83,18 +84,19 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function if !ok { return results, false } - for _, result := range results { - if !sources.describe(result.FilePath, s.projectRoot) { - continue - } - if lines := s.recordedLinesIn(module, result.FilePath, result.Line, sources, functions); len(lines) > 0 { + if result, found, ambiguous := sources.bestDescribedResult(results); found { + if lines := s.recordedLinesIn(module, owner, result.FilePath, result.Line, sources, functions); len(lines) > 0 { return lines, true } return results, false + } else if ambiguous { + return results, false } - if own, err := s.store.LookupModule(module); err == nil && len(own) == 0 { + if owner != module { + // The generator's file has no module that owns the declaration, so + // its lines are not followed through edits there. if source := s.compiledSourceFile(sources.compileSource); source != "" { - if lines := s.recordedLinesIn(module, source, 1, sources, functions); len(lines) > 0 { + if lines := s.recordedLinesIn(module, "", source, 1, sources, functions); len(lines) > 0 { return lines, true } } @@ -106,8 +108,8 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function // the compiled module records for functions in file, corrected for edits made // since the compile. afterLine is the module's line in the current text, so it // is compared after the correction: lines added above the module move both. -func (s *Server) recordedLinesIn(module, file string, afterLine int, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { - lines := s.correctLineDrift(file, sources.beamMtime, generatedFunctionLines(module, file, sources, functions), functions) +func (s *Server) recordedLinesIn(module, owner, file string, afterLine int, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { + lines := s.correctLineDrift(file, owner, sources.beamMtime, generatedFunctionLines(module, file, sources, functions), functions) return slices.DeleteFunc(lines, func(r store.LookupResult) bool { return r.Line <= afterLine }) } @@ -146,25 +148,33 @@ func generatedFunctionLines(module, file string, sources generatedDefinitionSour // lines that have moved. Dexter cannot compile the project, so it looks for // the declaration in the current text instead: a call whose first argument is // the function's name as an atom, which is how a macro call names what it -// declares (`define :list_rooms`, `field :email`). The nearest match wins. +// declares (`define :list_rooms`, `field :email`), in the body of owner, the +// module whose source holds the declarations. A match in another module of the +// same file is never taken, and without an owner no line is moved. The nearest +// match wins. // The recorded line is kept when it still declares the function, when no line // does, or when two matches are equally near. A line past the end of the // current text, with no declaration to move to, is dropped. A BEAM that is // current is never corrected, because its lines are exact even where the // declaration does not spell the name. -func (s *Server) correctLineDrift(path string, beamMtime int64, results []store.LookupResult, functions []beam.Function) []store.LookupResult { +func (s *Server) correctLineDrift(path, owner string, beamMtime int64, results []store.LookupResult, functions []beam.Function) []store.LookupResult { text, ok := s.driftedSourceText(path, beamMtime) if !ok { return results } lines := blankHeredocs(strings.Split(text, "\n")) + inOwner := func(int) bool { return false } + if owner != "" { + owners := moduleOwnersByLine([]byte(text), len(lines)) + inOwner = func(line int) bool { return owners[line] == owner } + } names := make(map[int]string, len(functions)) for _, function := range functions { names[function.Arity] = function.Name } out := results[:0] for _, result := range results { - line := nearestDeclaringLine(lines, result.Line, names[result.Arity]) + line := nearestDeclaringLine(lines, result.Line, names[result.Arity], inOwner) if line > len(lines) || slices.ContainsFunc(out, func(r store.LookupResult) bool { return r.Line == line }) { continue } @@ -222,8 +232,9 @@ func blankHeredocs(lines []string) []string { } // nearestDeclaringLine returns the 1-based line nearest to recorded that -// declares function by name, or recorded when that is ambiguous or not found. -func nearestDeclaringLine(lines []string, recorded int, function string) int { +// declares function by name and that accept allows, or recorded when that is +// ambiguous or not found. A nil accept allows every line. +func nearestDeclaringLine(lines []string, recorded int, function string, accept func(line int) bool) int { if function == "" || recorded < 1 { return recorded } @@ -232,7 +243,7 @@ func nearestDeclaringLine(lines []string, recorded int, function string) int { names = append(names, base) } declares := func(line int) bool { - return line >= 1 && line <= len(lines) && lineDeclaresAtom(lines[line-1], names) + return line >= 1 && line <= len(lines) && (accept == nil || accept(line)) && lineDeclaresAtom(lines[line-1], names) } if declares(recorded) { return recorded @@ -289,30 +300,137 @@ func isIdentifierByte(b byte) bool { (b >= 'a' && b <= 'z') || (b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9') } -// describe reports whether the recorded lines belong to path. When the module -// has debug info, its file names are the test; otherwise the compile info's. -func (sources generatedDefinitionSources) describe(path, projectRoot string) bool { +// bestDescribedResult returns the result whose file the recorded lines belong +// to. Usually there is one row, but a module name defined in two files (two +// umbrella apps, say) has two, and a moved project matches neither exactly. +// The row that matches the recorded path most specifically wins. If two match +// equally, ambiguous is set, because a line from one file means nothing in the +// other. +func (sources generatedDefinitionSources) bestDescribedResult(results []store.LookupResult) (best store.LookupResult, found, ambiguous bool) { + bestScore := 0 + for _, result := range results { + score := sources.describe(result.FilePath) + switch { + case score == 0: + case score > bestScore: + best, bestScore, ambiguous = result, score, false + case score == bestScore && result.FilePath != best.FilePath: + ambiguous = true + } + } + if ambiguous { + return store.LookupResult{}, false, true + } + return best, bestScore > 0, false +} + +// describe scores how specifically the recorded source names path: zero when +// it does not, highest for the same absolute path. When the module has debug +// info, its file names are the test; otherwise the compile info's. +func (sources generatedDefinitionSources) describe(path string) int { if len(sources.debugInfo.Lines) > 0 { - return debugInfoDescribesFile(sources.debugInfo, path) + return recordedPathMatch(sources.debugInfo.File, sources.debugInfo.RelativeFile, path) } - if sources.compileSource == "" { - return false + return recordedPathMatch(sources.compileSource, "", path) +} + +// exactPathMatch is the score for a recorded absolute path equal to the path. +const exactPathMatch = 1 << 20 + +// recordedPathMatch scores how specifically a recorded source path names path. +// The same absolute path is best. Otherwise the two are compared from their +// ends, which covers a project that has moved, was copied, or is reached +// through a symlink since it was compiled: each shared trailing component +// counts, and at least a directory and the file name must be shared. relative, +// the path relative to the compiler's working directory, counts its own +// components when it is a suffix of path. +func recordedPathMatch(recorded, relative, path string) int { + if recorded != "" && recorded == path { + return exactPathMatch + } + score := 0 + if recorded != "" { + if shared := sharedTrailingComponents(recorded, path); shared >= 2 { + score = shared + } } - return sources.compileSource == path || slices.Contains(sourceRebaseCandidates(sources.compileSource, projectRoot), path) + if relative != "" && !filepath.IsAbs(relative) { + slash := filepath.ToSlash(relative) + if strings.HasSuffix(filepath.ToSlash(path), "/"+slash) { + score = max(score, strings.Count(slash, "/")+1) + } + } + return score } -// debugInfoDescribesFile reports whether debug info was compiled from path. -// The absolute path is compared first; the relative one covers a project that -// has moved or is reached through a symlink since it was compiled. -func debugInfoDescribesFile(info beam.DebugInfo, path string) bool { - if info.File != "" && info.File == path { - return true - } - relative := filepath.ToSlash(info.RelativeFile) - if relative == "" || filepath.IsAbs(info.RelativeFile) { - return false +// sharedTrailingComponents counts the path components a and b share at their +// ends. +func sharedTrailingComponents(a, b string) int { + as := strings.Split(filepath.ToSlash(a), "/") + bs := strings.Split(filepath.ToSlash(b), "/") + shared := 0 + for shared < len(as) && shared < len(bs) { + component := as[len(as)-1-shared] + if component == "" || component != bs[len(bs)-1-shared] { + break + } + shared++ + } + return shared +} + +// moduleOwnersByLine returns, for each 1-based line of source, the innermost +// module whose body holds it, found with the tokenizer so that strings, +// heredocs and comments do not count. Index 0 is unused. +func moduleOwnersByLine(source []byte, lineCount int) []string { + owners := make([]string, lineCount+1) + tokens := parser.Tokenize(source) + type frame struct { + name string + depth int + } + var stack []frame + depth := 0 + current := func() string { + if len(stack) == 0 { + return "" + } + return stack[len(stack)-1].name + } + filled := 0 + fill := func(through int) { + for filled < through && filled < lineCount { + filled++ + owners[filled] = current() + } + } + for i := 0; i < len(tokens); i++ { + token := tokens[i] + // A line belongs to the module that is open at its first token. + fill(token.Line) + switch token.Kind { + case parser.TokDo, parser.TokFn: + parser.TrackBlockDepth(token.Kind, &depth) + case parser.TokEnd: + before := depth + parser.TrackBlockDepth(token.Kind, &depth) + if len(stack) > 0 && stack[len(stack)-1].depth == before { + stack = stack[:len(stack)-1] + } + case parser.TokDefmodule, parser.TokDefprotocol, parser.TokDefimpl: + name, next, hasDo := tokParseModuleDef(source, tokens, i+1, current()) + if name == "" { + continue + } + if hasDo { + depth++ + stack = append(stack, frame{name: name, depth: depth}) + } + i = next - 1 + } } - return strings.HasSuffix(filepath.ToSlash(path), "/"+relative) + fill(lineCount) + return owners } // compiledSourceFile maps a source path from compile info onto this checkout. diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index 62f0a29..245dbe0 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -10,6 +10,8 @@ import ( "testing" "time" + "github.com/remoteoss/dexter/internal/beam" + "github.com/remoteoss/dexter/internal/store" "go.lsp.dev/protocol" "go.lsp.dev/uri" ) @@ -490,7 +492,7 @@ end`, "\n") tc.recorded = 2 tc.want = 2 } - if got := nearestDeclaringLine(text, tc.recorded, tc.function); got != tc.want { + if got := nearestDeclaringLine(text, tc.recorded, tc.function, nil); got != tc.want { t.Errorf("nearestDeclaringLine(%d, %q) = %d, want %d", tc.recorded, tc.function, got, tc.want) } }) @@ -579,10 +581,10 @@ func TestDefinitionGeneratedFunctionFollowsDriftAboveModule(t *testing.T) { // its end, and a line with nothing left to move to is not returned. func TestNearestDeclaringLinePastEndOfFile(t *testing.T) { lines := []string{"defmodule A do", " define :rename", "end"} - if got := nearestDeclaringLine(lines, 19, "rename"); got != 2 { + if got := nearestDeclaringLine(lines, 19, "rename", nil); got != 2 { t.Errorf("nearestDeclaringLine past the end = %d, want 2", got) } - if got := nearestDeclaringLine(lines, 19, "missing"); got != 19 { + if got := nearestDeclaringLine(lines, 19, "missing", nil); got != 19 { t.Errorf("nearestDeclaringLine without a match = %d, want the recorded 19", got) } } @@ -680,7 +682,7 @@ func TestNearestDeclaringLineSkipsHeredocs(t *testing.T) { define :get_room_by_slug end end`, "\n")) - if got := nearestDeclaringLine(lines, 4, "get_room_by_slug!"); got != 9 { + if got := nearestDeclaringLine(lines, 4, "get_room_by_slug!", nil); got != 9 { t.Errorf("nearestDeclaringLine = %d, want 9, past the @moduledoc example", got) } } @@ -711,3 +713,95 @@ func TestDefinitionGeneratedFunctionIgnoresCachedFile(t *testing.T) { locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) expectSingleLocation(t, locations, domainPath, generatedDefineLine+1) } + +// Another module in the same file can declare the same name. After an edit it +// can be nearer to the recorded line than the real declaration, but a line in +// another module is never the answer. +func TestDefinitionGeneratedFunctionDriftStaysInItsModule(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + sibling := `defmodule MyApp.Archive do + use SharedLib.Domain + + resources do + resource MyApp.Archive.Room do + define :get_room_by_slug, action: :read + end + end +end + +` + // The recorded line is 8. The sibling's define is now line 6, two away; + // this module's define is line 18, ten away. + edited := sibling + generatedDomainSource + indexFile(t, server.store, server.projectRoot, generatedDomainRel, edited) + server.docs.Set(string(uri.File(domainPath)), edited) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine+10) +} + +func TestModuleOwnersByLine(t *testing.T) { + source := `defmodule A do + @moduledoc """ + defmodule NotAModule do + """ + defmodule Inner do + def x, do: :ok + end + + def y, do: "end" +end + +defmodule B do +end +` + owners := moduleOwnersByLine([]byte(source), strings.Count(source, "\n")+1) + for line, want := range map[int]string{1: "", 2: "A", 3: "A", 5: "A", 6: "A.Inner", 7: "A.Inner", 9: "A", 10: "A", 11: "", 13: "B"} { + if owners[line] != want { + t.Errorf("line %d owner = %q, want %q", line, owners[line], want) + } + } +} + +// A module defined in two files, as two umbrella apps can, has two rows. The +// recorded path decides between them; when it cannot, no line is trusted. +func TestBestDescribedResult(t *testing.T) { + a := store.LookupResult{FilePath: "/new/apps/a/lib/dup.ex", Line: 1} + b := store.LookupResult{FilePath: "/new/apps/b/lib/dup.ex", Line: 1} + lines := map[beam.FunctionKey]int{{Name: "f", Arity: 0}: 5} + for _, tc := range []struct { + name string + info beam.DebugInfo + want string + wantAmbiguous bool + }{ + {"exact path", beam.DebugInfo{File: b.FilePath, RelativeFile: "lib/dup.ex", Lines: lines}, b.FilePath, false}, + {"moved umbrella", beam.DebugInfo{File: "/old/apps/b/lib/dup.ex", RelativeFile: "lib/dup.ex", Lines: lines}, b.FilePath, false}, + {"only the relative path", beam.DebugInfo{RelativeFile: "lib/dup.ex", Lines: lines}, "", true}, + } { + t.Run(tc.name, func(t *testing.T) { + sources := generatedDefinitionSources{debugInfo: tc.info} + got, found, ambiguous := sources.bestDescribedResult([]store.LookupResult{a, b}) + if ambiguous != tc.wantAmbiguous || found != (tc.want != "") || (found && got.FilePath != tc.want) { + t.Errorf("bestDescribedResult = %s, found=%v, ambiguous=%v; want %q, ambiguous=%v", got.FilePath, found, ambiguous, tc.want, tc.wantAmbiguous) + } + }) + } +} + +// The declaration was deleted, and a module below declares the same name. The +// only declaration left is in another module, so the recorded line stays. +func TestDefinitionGeneratedFunctionDriftIgnoresModuleBelow(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + withoutDefine := strings.Replace(generatedDomainSource, " define :get_room_by_slug, action: :read, get_by: [:slug]\n", "", 1) + edited := withoutDefine + ` +defmodule MyApp.Archive do + resources do + define :get_room_by_slug, action: :read + end +end +` + indexFile(t, server.store, server.projectRoot, generatedDomainRel, edited) + server.docs.Set(string(uri.File(domainPath)), edited) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} From e9b09357e8464c1688d1a8d8c231c621d83899ff Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Tue, 29 Sep 2026 18:49:30 -0400 Subject: [PATCH 07/16] Drop recorded lines past the end of the file A generator can give a def any line, as `quote line: 99` does, so even a current BEAM can record a line the file does not have. Check each line against the current text, open buffer or disk, and fall back to the module line when it is past the end. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/lsp/generated_definition.go | 20 +++++++++++++++++++- internal/lsp/generated_definition_test.go | 19 +++++++++++++++++-- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index 5d6db58..f1a4494 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -1,6 +1,7 @@ package lsp import ( + "bytes" "os" "path/filepath" "slices" @@ -108,9 +109,26 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function // the compiled module records for functions in file, corrected for edits made // since the compile. afterLine is the module's line in the current text, so it // is compared after the correction: lines added above the module move both. +// +// A generator can give a def any line (`quote line: 99`), so a line past the +// end of the current text is dropped too: it is not a place to send an editor. func (s *Server) recordedLinesIn(module, owner, file string, afterLine int, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { lines := s.correctLineDrift(file, owner, sources.beamMtime, generatedFunctionLines(module, file, sources, functions), functions) - return slices.DeleteFunc(lines, func(r store.LookupResult) bool { return r.Line <= afterLine }) + lastLine := s.currentLineCount(file) + return slices.DeleteFunc(lines, func(r store.LookupResult) bool { return r.Line <= afterLine || r.Line > lastLine }) +} + +// currentLineCount returns how many lines path has now: in the open buffer if +// there is one, otherwise on disk. It is zero when the file cannot be read. +func (s *Server) currentLineCount(path string) int { + if text, ok := s.docs.GetIfOpen(string(pathToURI(path))); ok { + return strings.Count(text, "\n") + 1 + } + data, err := os.ReadFile(path) + if err != nil { + return 0 + } + return bytes.Count(data, []byte("\n")) + 1 } // generatedFunctionLines returns a result for each distinct line that the diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index 245dbe0..a74b1ee 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -390,7 +390,7 @@ func TestLookupNameSourcelessGeneratedModuleUsesCompiledSource(t *testing.T) { if err := os.MkdirAll(filepath.Dir(source), 0o755); err != nil { t.Fatal(err) } - if err := os.WriteFile(source, []byte("defmodule SharedLib.Dsl.Extension do\nend\n"), 0o644); err != nil { + if err := os.WriteFile(source, []byte("defmodule SharedLib.Dsl.Extension do\n"+strings.Repeat("\n", 50)+"end\n"), 0o644); err != nil { t.Fatal(err) } writeGeneratedBeam(t, server, sourcelessModule, tc.chunks, beamExport{"build", 1}) @@ -622,7 +622,7 @@ func TestLookupNameSourcelessGeneratedModulePrefersThisCheckout(t *testing.T) { if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { t.Fatal(err) } - if err := os.WriteFile(path, []byte("defmodule SharedLib.Dsl do\nend\n"), 0o644); err != nil { + if err := os.WriteFile(path, []byte("defmodule SharedLib.Dsl do\n"+strings.Repeat("\n", 50)+"end\n"), 0o644); err != nil { t.Fatal(err) } } @@ -805,3 +805,18 @@ end locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) expectSingleLocation(t, locations, domainPath, generatedDefineLine) } + +// A generator can give a def any line, such as `quote line: 99`. Even from a +// current BEAM, a line past the end of the file is not a place to go. +func TestDefinitionGeneratedFunctionLinePastEndOfFileKeepsModuleLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 99, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, + ) + future := time.Now().Add(time.Hour) + beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") + if err := os.Chtimes(beamPath, future, future); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedModuleLine) +} From f436f4db1d24b55d00d56d20ca1a5ebffb586911 Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Wed, 30 Sep 2026 15:28:14 -0400 Subject: [PATCH 08/16] Place more generated definitions, and test them with the real compiler - A function whose only recorded line is the module line, such as one a @before_compile hook made, goes to the one call in its module that declares it by name. This also places released Ash code interfaces. - A function with a clause per DSL call goes to every clause. - A stale line with no name to search for is placed by the source definitions around it, from every module in the file; when an edit is between them, only a one-to-one match to `def unquote` lines counts. - A BEAM's mtime has whole seconds, so compare staleness by the second; a save in the same second as the compile no longer looks stale. - While a buffer has unsaved edits, the module line comes from the buffer. - Go-to-definition on a module that exists only as a BEAM goes to the file and line the compiler recorded for it. - A bare call to a generated function of an imported module resolves. - Read {line, column} annotations. - Add integration tests that compile a DSL fixture of each shape with mix, and run them in the integration job. Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/workflows/ci.yml | 10 +- CHANGELOG.md | 2 +- docs/architecture.md | 13 +- internal/beam/debug_info.go | 118 ++++- internal/beam/docs.go | 23 +- internal/lsp/generated_definition.go | 459 ++++++++++++++---- .../generated_definition_integration_test.go | 348 +++++++++++++ internal/lsp/generated_definition_test.go | 126 ++++- internal/lsp/name_navigation.go | 5 + internal/lsp/server.go | 8 + 10 files changed, 983 insertions(+), 129 deletions(-) create mode 100644 internal/lsp/generated_definition_integration_test.go diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ea2de7f..a691067 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -73,12 +73,12 @@ jobs: cd internal/lsp/testdata/monorepo/apps/app_with_styler && mix deps.get && mix deps.compile && cd - cd internal/lsp/testdata/monorepo/apps/app_with_ecto_migration && mix deps.get && mix deps.compile && cd - cd internal/lsp/testdata/monorepo/apps/app_basic && mix deps.get && cd - - # Every formatter test and the generated-BEAM completion test need mix on - # PATH and skip themselves without it, so this job is the only place they - # run — the test/test-race jobs have no Elixir. Match the formatter family - # and explicitly include the self-contained BEAM regression fixture. + # Every formatter test and the generated-BEAM completion and definition + # tests need mix on PATH and skip themselves without it, so this job is the + # only place they run — the test/test-race jobs have no Elixir. Match the + # formatter family and explicitly include the self-contained BEAM fixtures. - name: Run integration tests - run: go test ./internal/lsp/ -run 'TestFormatter|TestDidSave_Formatter|TestCompletion_GeneratedFunctionsFromConsumerBEAM' -v -timeout 600s + run: go test ./internal/lsp/ -run 'TestFormatter|TestDidSave_Formatter|TestCompletion_GeneratedFunctionsFromConsumerBEAM|TestDefinition_GeneratedFunctionsFromCompiler' -v -timeout 600s lint: runs-on: ubuntu-latest diff --git a/CHANGELOG.md b/CHANGELOG.md index bf791a8..3c4cc43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line. When the source or an unsaved buffer has changed since the compile, the line follows the edit to the nearest call that declares the function by name as its first atom argument (`define :list_rooms`); if there is no such call, or two are equally near, the recorded line stays. A module compiled without debug info falls back to its Docs chunk annotation, which is often, but not always, the same line. A generated module with no source of its own, such as a Spark DSL entity, goes to the file it was compiled from, rebased onto the project's `deps` when it was built elsewhere. Ash code interfaces go to their `define` line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) +- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line. When the source or an unsaved buffer has changed since the compile, the line follows the edit: to the nearest call in the same module that declares the function by name as its first atom argument (`define :list_rooms`), or else by how far the source definitions around it moved, and a line that cannot be placed reliably stays as recorded. A function whose only recorded line is the module line, such as one a `@before_compile` hook made, goes to the one call in its module that declares it by name, if there is exactly one. A function with a clause per DSL call goes to every clause. A line past the end of the file is never returned. A module compiled without debug info falls back to its Docs chunk annotation, which is often, but not always, the same line. A generated module with no source of its own, such as one `Module.create` made or a Spark DSL entity, goes to the file it was compiled from, rebased onto the project when it was built elsewhere, and so does go-to-definition on its name. A bare call to a generated function of an imported module now resolves. Ash code interfaces go to their `define` line with released Ash, and from the recorded line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) - **`--root`/`-C` names the workspace on every command** — `dexter init --root ~/project`, `dexter lookup --root ~/project MyApp.Repo`, and `dexter lsp --root ~/project` all run as if they had been started in that directory, so an agent, a script, or an editor wrapper can index or query a project from anywhere. Relative paths, a `reindex` target included, resolve from the named root diff --git a/docs/architecture.md b/docs/architecture.md index 59fc8ef..e35b2fb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -145,9 +145,18 @@ Each def carries two locations. Its `:line` is the line in the module's own file `generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (the same absolute path, or else the most path components shared from the end, which covers a moved, copied, or symlinked project; when a module has rows in two files, as two umbrella apps can, the most specific match wins, and an equal match trusts neither), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is closer than the module line. -A stale line is corrected where that can be done reliably. `correctLineDrift` runs only when the text may differ from the compiled text: the file is newer than the BEAM, or a buffer open in the editor differs from the file. A file that another request only cached does not count. It runs before the check against the module's line, because lines added above `defmodule` move both. It then looks for the nearest line, outside heredocs and inside the body of the module that owns the declarations (found with the tokenizer, so a sibling module in the same file never matches), that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, or a later argument does not count. The recorded line is kept when it still declares the function, when no line does, or when two matches are equally near. A current BEAM is never corrected, because its lines are exact even where the declaring call does not spell the name, as with a route helper. A generated module with no source row has no owner in its generator's file, so its line is not corrected. +`beam.ReadDefinitionLines` also reads the module's own line (the map's `anno`, a line or `{line, column}`), the line of each clause when a def's clauses were made at different lines (`DebugInfo.Clauses`: a DSL that adds a clause per call, such as `route :get, "/a"`, gets a location per clause), and the `:line` of every def, private ones included, as `DebugInfo.Anchors`. -A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually the generator's own file. A recorded path inside the project is used as it is. Otherwise it is rebased: first each tail of the recorded path under the project root, longest first, which covers a moved or copied checkout and an umbrella's `apps//lib`, then the Mix layouts for the application named below the last `lib/` (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. A recorded path outside the project is the last choice: it is right for a `path:` dependency, but a `_build` copied from another worktree records that worktree, which usually still exists. If no file is found, the lexical parent stays the answer. `CInf` records the path as codepoints, which are decoded to UTF-8. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. +Lines are checked against the current text, which is the buffer open in the editor if there is one and the file on disk otherwise (`currentSource`). A line at or before the module's line as compiled carries no information and is dropped first. A line past the end of the current text is dropped too, because a generator can give a def any line (`quote line: 99`). When nothing is left, which is the case for a def a `@before_compile` hook made at the module line (and for released Ash's code interfaces), the owning module's body is searched for the declaring call: a call whose first argument is the function's name as an atom. It is used only when exactly one line in the module matches. + +A stale line is corrected where that can be done reliably. The correction runs only when the text may differ from the compiled text: the file was written in a later second than the BEAM, or a buffer open in the editor differs from the file. A BEAM's mtime has whole seconds, so a file saved in the same second as the compile counts as compiled; a file that another request only cached does not count as a buffer. The module's line is then taken from the current text, and the correction runs before the check against it, because lines added above `defmodule` move both. Every signal is limited to the body of the module that owns the declarations, found with the tokenizer, so a sibling module in the same file never matches. The signals, most exact first: + +- The nearest line, outside heredocs, that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count. The recorded line is kept when it still declares the function, and two equally near matches count as none. +- Anchors: the nearest definitions above and below the recorded line that a compiled module and the current text both have. Lines move for the whole file, so the defs and module lines of every module in the file whose BEAM was compiled from it count. When both anchors moved by the same amount, the line moved with them. When they did not, the edit was between them, and the line is placed only if the compiled module's generated defs between the anchors match the current text's `def unquote(...)` lines there one to one, in order: the shape a comprehension leaves. + +A line with no signal is kept as recorded. A current BEAM is never corrected, because its lines are exact even where the declaring call does not spell the name, as with a route helper. A generated module with no source row has no owner in its generator's file, so its line is not corrected. + +A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually the generator's own file. A recorded path inside the project is used as it is. Otherwise it is rebased: first each tail of the recorded path under the project root, longest first, which covers a moved or copied checkout and an umbrella's `apps//lib`, then the Mix layouts for the application named below the last `lib/` (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. A recorded path outside the project is the last choice: it is right for a `path:` dependency, but a `_build` copied from another worktree records that worktree, which usually still exists. If no file is found, the lexical parent stays the answer. `CInf` records the path as codepoints, which are decoded to UTF-8. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. `LookupName` on the name of a module that has no source row returns the file from `CInf` at the module's recorded line, so go-to-definition on a module that `Module.create` made works too. A bare call that no source resolves also checks the BEAMs of the file's imported modules for a generated function of that name. The debug info and compile info are read on the first definition request that needs them and memoized on the module's generated-function cache entry, so the BEAM stamp invalidates them with everything else. A module compiled without debug info, or an Erlang module, is memoized as empty. diff --git a/internal/beam/debug_info.go b/internal/beam/debug_info.go index 2972b48..a8e7a9c 100644 --- a/internal/beam/debug_info.go +++ b/internal/beam/debug_info.go @@ -3,6 +3,7 @@ package beam import ( "errors" "fmt" + "slices" ) // maxDebugInfoDepth bounds recursion while stepping over clause bodies. Each @@ -28,9 +29,24 @@ type DebugInfo struct { File string RelativeFile string + // ModuleLine is the line of the module itself in File: its defmodule, or + // wherever Module.create was called. Zero if the chunk has none. + ModuleLine int + // Lines maps each public function and macro to its line in File. Entries // without a usable line are absent. Lines map[FunctionKey]int + + // Clauses holds the line of each clause for a public definition whose + // clauses were made at more than one line, in clause order: a DSL that + // adds one clause per call, such as `route :get, "/a"`. Definitions with + // one line are absent; Lines has it. + Clauses map[FunctionKey][]int + + // Anchors maps every definition, private ones included, to its :line as + // compiled. Comparing these with the same definitions in the current + // source shows how far the source has moved since the compile. + Anchors map[FunctionKey]int } // ReadDefinitionLines reads the line of every public definition from an Elixir @@ -67,10 +83,12 @@ var errNoElixirDebugInfo = errors.New("no Elixir debug info") // source path is known: small maps put definitions ahead of file and // relative_file, so they cannot be resolved while walking. type definitionSite struct { - key FunctionKey - line int - keepFile string - keepLine int + key FunctionKey + public bool + line int + keepFile string + keepLine int + clauseLines []int } // parseDebugInfo walks {:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}}. @@ -148,6 +166,10 @@ func parseDebugInfo(buf []byte) (info DebugInfo, err error) { if info.RelativeFile, err = readOptionalBinary(r); err != nil { return DebugInfo{}, err } + case "anno": + if info.ModuleLine, err = readAnnoLine(r); err != nil { + return DebugInfo{}, err + } default: if err := r.skip(); err != nil { return DebugInfo{}, err @@ -160,23 +182,41 @@ func parseDebugInfo(buf []byte) (info DebugInfo, err error) { } info.Lines = make(map[FunctionKey]int, len(sites)) + info.Anchors = make(map[FunctionKey]int, len(sites)) for _, site := range sites { + if site.line > 0 { + if _, seen := info.Anchors[site.key]; !seen { + info.Anchors[site.key] = site.line + } + } + if !site.public { + continue + } line := site.line - if site.keepFile != "" && site.keepLine > 0 && - (site.keepFile == info.RelativeFile || site.keepFile == info.File) { + stamped := site.keepFile != "" && site.keepLine > 0 && + (site.keepFile == info.RelativeFile || site.keepFile == info.File) + if stamped { line = site.keepLine } if line > 0 { info.Lines[site.key] = line } + // A stamped def names its own line; its clauses' lines are the + // generator's, so only an unstamped def's clauses are kept. + if !stamped && len(site.clauseLines) > 1 { + if info.Clauses == nil { + info.Clauses = make(map[FunctionKey][]int) + } + info.Clauses[site.key] = site.clauseLines + } } return info, nil } // readDefinitionSites reads the definitions list, whose entries are -// {{name, arity}, kind, meta, clauses}. Private definitions are skipped: -// generated functions are found through the export table, so only public ones -// can be asked about. +// {{name, arity}, kind, meta, clauses}. Private definitions are kept only as +// anchors: generated functions are found through the export table, so only +// public ones can be asked about. func readDefinitionSites(r *etfReader) ([]definitionSite, error) { count, hasTail, err := r.enterList() if err != nil { @@ -214,16 +254,14 @@ func readDefinitionSites(r *etfReader) ([]definitionSite, error) { if err != nil { return nil, err } - site := definitionSite{key: FunctionKey{Name: name, Arity: functionArity}} + site := definitionSite{key: FunctionKey{Name: name, Arity: functionArity}, public: kind == "def" || kind == "defmacro"} if err := readDefinitionMeta(r, &site); err != nil { return nil, err } - if err := r.skip(); err != nil { // clauses + if site.clauseLines, err = readClauseLines(r); err != nil { return nil, err } - if kind == "def" || kind == "defmacro" { - sites = append(sites, site) - } + sites = append(sites, site) } if hasTail { if err := r.skip(); err != nil { @@ -233,6 +271,58 @@ func readDefinitionSites(r *etfReader) ([]definitionSite, error) { return sites, nil } +// readClauseLines reads the distinct :line of each {meta, args, guards, body} +// clause, in order, and steps over everything else without allocating. A value +// that is not a list is skipped. +func readClauseLines(r *etfReader) ([]int, error) { + if tag, err := r.peekTag(); err != nil { + return nil, err + } else if tag != tagList && tag != tagNil { + return nil, r.skip() + } + count, hasTail, err := r.enterList() + if err != nil { + return nil, err + } + var lines []int + for i := int64(0); i < count; i++ { + if tag, err := r.peekTag(); err != nil { + return nil, err + } else if tag != tagSmallTuple { + if err := r.skip(); err != nil { + return nil, err + } + continue + } + arity, err := r.enterTuple() + if err != nil { + return nil, err + } + if arity != 4 { + if err := r.skipTerms(int64(arity)); err != nil { + return nil, err + } + continue + } + var clause definitionSite + if err := readDefinitionMeta(r, &clause); err != nil { + return nil, err + } + if err := r.skipTerms(3); err != nil { // args, guards, body + return nil, err + } + if clause.line > 0 && !slices.Contains(lines, clause.line) { + lines = append(lines, clause.line) + } + } + if hasTail { + if err := r.skip(); err != nil { + return nil, err + } + } + return lines, nil +} + // readDefinitionMeta reads :line and a `file: {path, line}` entry from a // definition's keyword metadata. The first of each wins, as Keyword.get would. func readDefinitionMeta(r *etfReader, site *definitionSite) error { diff --git a/internal/beam/docs.go b/internal/beam/docs.go index 5a34c34..479ef08 100644 --- a/internal/beam/docs.go +++ b/internal/beam/docs.go @@ -524,17 +524,32 @@ func parseDocsEntry(r *etfReader) ([]Function, bool, error) { } // readAnnoLine reads the per-entry source annotation. Elixir writes it as a bare -// line number, but the field is an erl_anno and may hold other shapes (a tuple, -// or `none`), so anything that is not an integer is consumed and reported as line -// 0 rather than failing the whole entry. A dropped annotation must never cost the +// line number, but the field is an erl_anno and may hold other shapes: a +// {line, column} tuple gives its line, and anything else, such as `none`, is +// consumed and reported as line 0 rather than failing the whole entry. A dropped annotation must never cost the // module its generated functions. func readAnnoLine(r *etfReader) (int, error) { tag, err := r.peekTag() if err != nil { return 0, err } - if tag == tagSmallInteger || tag == tagInteger { + switch tag { + case tagSmallInteger, tagInteger: return r.readInt() + case tagSmallTuple: + // {line, column}, as erl_anno writes a location with a column. + arity, err := r.enterTuple() + if err != nil { + return 0, err + } + if arity == 0 { + return 0, nil + } + line, err := readAnnoLine(r) + if err != nil { + return 0, err + } + return line, r.skipTerms(int64(arity) - 1) } return 0, r.skip() } diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index f1a4494..36a4780 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -1,7 +1,6 @@ package lsp import ( - "bytes" "os" "path/filepath" "slices" @@ -86,10 +85,11 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function return results, false } if result, found, ambiguous := sources.bestDescribedResult(results); found { - if lines := s.recordedLinesIn(module, owner, result.FilePath, result.Line, sources, functions); len(lines) > 0 { + src := s.currentSourceOf(result.FilePath, sources.beamMtime) + if lines := recordedLinesIn(module, owner, result, src, sources, functions); len(lines) > 0 { return lines, true } - return results, false + return src.currentModuleResults(results, owner), false } else if ambiguous { return results, false } @@ -97,7 +97,8 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function // The generator's file has no module that owns the declaration, so // its lines are not followed through edits there. if source := s.compiledSourceFile(sources.compileSource); source != "" { - if lines := s.recordedLinesIn(module, "", source, 1, sources, functions); len(lines) > 0 { + src := s.currentSourceOf(source, sources.beamMtime) + if lines := recordedLinesIn(module, "", store.LookupResult{FilePath: source, Line: 1}, src, sources, functions); len(lines) > 0 { return lines, true } } @@ -105,34 +106,74 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function return results, false } -// recordedLinesIn returns a result for each distinct line after afterLine that -// the compiled module records for functions in file, corrected for edits made -// since the compile. afterLine is the module's line in the current text, so it -// is compared after the correction: lines added above the module move both. -// -// A generator can give a def any line (`quote line: 99`), so a line past the -// end of the current text is dropped too: it is not a place to send an editor. -func (s *Server) recordedLinesIn(module, owner, file string, afterLine int, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { - lines := s.correctLineDrift(file, owner, sources.beamMtime, generatedFunctionLines(module, file, sources, functions), functions) - lastLine := s.currentLineCount(file) - return slices.DeleteFunc(lines, func(r store.LookupResult) bool { return r.Line <= afterLine || r.Line > lastLine }) +// generatedModuleLocation is where a module that exists only as a BEAM is +// defined: the file it was compiled from, at the line the compiler recorded +// for the module, such as a Module.create call. It is empty when the BEAM does +// not record both, or the line is not in the file. +func (s *Server) generatedModuleLocation(module string) []store.LookupResult { + if len(s.generatedFunctionsForModule(module)) == 0 { + return nil + } + sources, ok := s.generatedDefinitionSourcesFor(module, "") + if !ok || sources.debugInfo.ModuleLine < 1 { + return nil + } + source := s.compiledSourceFile(sources.compileSource) + if source == "" { + return nil + } + if src := s.currentSourceOf(source, sources.beamMtime); src == nil || sources.debugInfo.ModuleLine > len(src.lines) { + return nil + } + return []store.LookupResult{{Module: module, FilePath: source, Line: sources.debugInfo.ModuleLine, Kind: "module"}} } -// currentLineCount returns how many lines path has now: in the open buffer if -// there is one, otherwise on disk. It is zero when the file cannot be read. -func (s *Server) currentLineCount(path string) int { - if text, ok := s.docs.GetIfOpen(string(pathToURI(path))); ok { - return strings.Count(text, "\n") + 1 +// recordedLinesIn returns a result for each distinct line in result's file +// that the compiled module records for functions, corrected for edits made +// since the compile, and after the module's own line. +// +// The module's line is taken from the current text when it has changed, so +// that it is compared after the correction: lines added above the module move +// both. A generator can give a def any line (`quote line: 99`), so a line past +// the end of the current text is dropped too. +// +// When nothing is left, which is the case for a def a @before_compile hook +// made at the module line, the module's body is searched for the call that +// declared the function: the one call whose first argument is its name as an +// atom (`later :deferred`). Only a single match counts. +func recordedLinesIn(module, owner string, result store.LookupResult, src *currentSource, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { + if src == nil { + return nil + } + moduleLine := result.Line + if line, ok := src.moduleLine(owner); ok && src.changed { + moduleLine = line + } + // A line at or before the module's line as compiled says nothing the + // module result does not, so it is dropped before any correction can move + // it somewhere that only looks meaningful. + compiledModuleLine := result.Line + if owner == module && sources.debugInfo.ModuleLine > 0 { + compiledModuleLine = sources.debugInfo.ModuleLine } - data, err := os.ReadFile(path) - if err != nil { - return 0 + lines := slices.DeleteFunc(generatedFunctionLines(module, result.FilePath, sources, functions), func(r store.LookupResult) bool { return r.Line <= compiledModuleLine }) + if src.changed { + lines = src.correctDrift(module, owner, lines, sources, functions) } - return bytes.Count(data, []byte("\n")) + 1 + lines = slices.DeleteFunc(lines, func(r store.LookupResult) bool { return r.Line <= moduleLine || r.Line > len(src.lines) }) + if len(lines) > 0 || owner == "" { + return lines + } + if line, ok := src.uniqueDeclaration(owner, functions); ok && line > moduleLine { + return []store.LookupResult{{Module: module, FilePath: result.FilePath, Line: line, Kind: functions[0].Kind, Arity: functions[0].Arity}} + } + return nil } // generatedFunctionLines returns a result for each distinct line that the -// compiled module records for functions in file. +// compiled module records for functions in file: one per clause when the +// clauses were made at different lines, as a DSL that adds a clause per call +// does. // // The Dbgi line is preferred because it also honors `@file`. The Docs chunk // annotation is the fallback, because a module compiled without debug info @@ -142,58 +183,175 @@ func generatedFunctionLines(module, file string, sources generatedDefinitionSour useDebugInfo := len(sources.debugInfo.Lines) > 0 var lines []store.LookupResult for _, function := range functions { - line := function.Line + key := beam.FunctionKey{Name: function.Name, Arity: function.Arity} + recorded := []int{function.Line} if useDebugInfo { - line = sources.debugInfo.Lines[beam.FunctionKey{Name: function.Name, Arity: function.Arity}] + recorded = []int{sources.debugInfo.Lines[key]} + if clauses := sources.debugInfo.Clauses[key]; len(clauses) > 0 { + recorded = clauses + } } - if line <= 0 || slices.ContainsFunc(lines, func(r store.LookupResult) bool { return r.Line == line }) { - continue + for _, line := range recorded { + if line <= 0 || slices.ContainsFunc(lines, func(r store.LookupResult) bool { return r.Line == line }) { + continue + } + lines = append(lines, store.LookupResult{ + Module: module, + FilePath: file, + Line: line, + Kind: function.Kind, + Arity: function.Arity, + }) } - lines = append(lines, store.LookupResult{ - Module: module, - FilePath: file, - Line: line, - Kind: function.Kind, - Arity: function.Arity, - }) } return lines } -// correctLineDrift moves recorded lines to follow edits made since the compile. -// -// A BEAM older than its source, or a buffer with unsaved changes, can describe -// lines that have moved. Dexter cannot compile the project, so it looks for -// the declaration in the current text instead: a call whose first argument is -// the function's name as an atom, which is how a macro call names what it -// declares (`define :list_rooms`, `field :email`), in the body of owner, the -// module whose source holds the declarations. A match in another module of the -// same file is never taken, and without an owner no line is moved. The nearest -// match wins. -// The recorded line is kept when it still declares the function, when no line -// does, or when two matches are equally near. A line past the end of the -// current text, with no declaration to move to, is dropped. A BEAM that is -// current is never corrected, because its lines are exact even where the -// declaration does not spell the name. -func (s *Server) correctLineDrift(path, owner string, beamMtime int64, results []store.LookupResult, functions []beam.Function) []store.LookupResult { - text, ok := s.driftedSourceText(path, beamMtime) +// currentSource is a file's text as it is now, in the open buffer if there is +// one and otherwise on disk, with what navigation derives from it computed on +// first use. +type currentSource struct { + path string + text string + // lines is text split into lines, with the lines inside heredocs blanked. + lines []string + // changed reports that the text may differ from the text the BEAM was + // compiled from: the buffer has unsaved changes, or the file was written + // after the BEAM. + changed bool + + owners []string + defs []parser.Definition + parsed bool + + // compiled returns the debug info of another module's BEAM, so that the + // other modules in this file can serve as anchors. It may be nil. + compiled func(module string) (beam.DebugInfo, bool) +} + +// currentSourceOf returns path's current text, or nil if it cannot be read. +// Only a buffer open in the editor counts: a file that another request cached +// is not unsaved edits. +func (s *Server) currentSourceOf(path string, beamMtime int64) *currentSource { + open, isOpen := s.docs.GetIfOpen(string(pathToURI(path))) + disk, err := os.ReadFile(path) + src := ¤tSource{path: path, compiled: func(module string) (beam.DebugInfo, bool) { + sources, ok := s.generatedDefinitionSourcesFor(module, "") + return sources.debugInfo, ok && len(sources.debugInfo.Anchors) > 0 + }} + switch { + case isOpen: + src.text = open + src.changed = err != nil || open != string(disk) + case err == nil: + src.text = string(disk) + default: + return nil + } + if stamp := statFileStamp(path); stamp.exists && newerBySecond(stamp.mtime, beamMtime) { + src.changed = true + } + src.lines = blankHeredocs(strings.Split(src.text, "\n")) + return src +} + +// newerBySecond reports whether mtime is in a later second than beamMtime. A +// compiled BEAM's mtime has whole seconds while a source file's does not, so a +// file saved and compiled within one second would otherwise look newer than +// its BEAM. An edit in the same second as the compile counts as compiled, which +// only means its lines are not corrected. +func newerBySecond(mtime, beamMtime int64) bool { + return mtime/int64(time.Second) > beamMtime/int64(time.Second) +} + +// ownerAt returns the innermost module whose body holds the 1-based line. +func (src *currentSource) ownerAt(line int) string { + if src.owners == nil { + src.owners = moduleOwnersByLine([]byte(src.text), len(src.lines)) + } + if line < 1 || line >= len(src.owners) { + return "" + } + return src.owners[line] +} + +// definitions returns the definitions the parser finds in the current text. +func (src *currentSource) definitions() []parser.Definition { + if !src.parsed { + src.parsed = true + src.defs, _, _ = parser.ParseText(src.path, src.text) + } + return src.defs +} + +// moduleLine returns the line of module's defmodule in the current text. +func (src *currentSource) moduleLine(module string) (int, bool) { + if module == "" { + return 0, false + } + for _, def := range src.definitions() { + if def.Module == module && def.Function == "" { + return def.Line, true + } + } + return 0, false +} + +// currentModuleResults moves the module results in this file to owner's line +// in the current text, which differs from the index while a buffer has unsaved +// changes. +func (src *currentSource) currentModuleResults(results []store.LookupResult, owner string) []store.LookupResult { + if src == nil || !src.changed { + return results + } + line, ok := src.moduleLine(owner) if !ok { return results } - lines := blankHeredocs(strings.Split(text, "\n")) - inOwner := func(int) bool { return false } - if owner != "" { - owners := moduleOwnersByLine([]byte(text), len(lines)) - inOwner = func(line int) bool { return owners[line] == owner } + out := slices.Clone(results) + for i := range out { + if out[i].FilePath == src.path { + out[i].Line = line + } + } + return out +} + +// correctDrift moves recorded lines to follow edits made since the compile. +// +// Dexter cannot compile the project, so it reads the current text instead, +// and only the body of owner, the module whose source holds the declarations: +// a sibling module in the same file never matches, and without an owner no +// line is moved. Two signals are used, most exact first: +// +// - A call whose first argument is the function's name as an atom, which is +// how a macro call names what it declares (`define :list_rooms`, +// `field :email`). The nearest wins; the recorded line is kept when it +// still declares the function, and two equally near matches count as none. +// - Otherwise, the definitions the source still has on either side. The +// compiled module records their old lines and the current text has their +// new ones, so the difference is how far this part of the file moved (see +// anchoredLine). +// +// A line with neither signal is kept as recorded. +func (src *currentSource) correctDrift(module, owner string, results []store.LookupResult, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { + if owner == "" { + return results } names := make(map[int]string, len(functions)) for _, function := range functions { names[function.Arity] = function.Name } + inOwner := func(line int) bool { return src.ownerAt(line) == owner } out := results[:0] for _, result := range results { - line := nearestDeclaringLine(lines, result.Line, names[result.Arity], inOwner) - if line > len(lines) || slices.ContainsFunc(out, func(r store.LookupResult) bool { return r.Line == line }) { + line, found := nearestDeclaringLine(src.lines, result.Line, names[result.Arity], inOwner) + if !found { + if moved, ok := src.anchoredLine(module, owner, result.Line, sources.debugInfo); ok { + line = moved + } + } + if slices.ContainsFunc(out, func(r store.LookupResult) bool { return r.Line == line }) { continue } result.Line = line @@ -202,24 +360,149 @@ func (s *Server) correctLineDrift(path, owner string, beamMtime int64, results [ return out } -// driftedSourceText returns the current text of path when it may differ from -// the text the BEAM was compiled from: the open buffer if it has unsaved -// changes, or the file if it was written after the BEAM. -func (s *Server) driftedSourceText(path string, beamMtime int64) (string, bool) { - open, isOpen := s.docs.GetIfOpen(string(pathToURI(path))) - source := statFileStamp(path) - stale := source.exists && source.mtime > beamMtime - if !isOpen && !stale { - return "", false +// anchoredLine places recorded in the current text by the nearest +// definitions above and below it that a compiled module and the current text +// both have: the defs and the module lines of every module in the file whose +// BEAM was compiled from it. +// +// When both anchors moved by the same amount, the line moved with them. When +// they moved by different amounts, the edit was between them. Then the line is +// placed only when every generated def between the anchors can be matched to a +// line in owner that defines a function with an unquoted name +// (`def unquote(name)()`), one to one and in order: the shape a comprehension +// leaves. Otherwise it is not placed. +func (src *currentSource) anchoredLine(module, owner string, recorded int, info beam.DebugInfo) (int, bool) { + // Lines move for the whole file, so every module defined in it can serve + // as an anchor, not only this one: a nested module's line counts as much + // as a def. Another module's BEAM is used only when it was compiled from + // this file. + type moduleKey struct { + module string + key beam.FunctionKey + } + current := make(map[moduleKey]int) + moduleLines := make(map[string]int) + for _, def := range src.definitions() { + if def.Function == "" { + if _, seen := moduleLines[def.Module]; !seen { + moduleLines[def.Module] = def.Line + } + continue + } + key := moduleKey{def.Module, beam.FunctionKey{Name: def.Function, Arity: def.Arity}} + if line, seen := current[key]; !seen || def.Line < line { + current[key] = def.Line + } } - disk, err := os.ReadFile(path) - switch { - case isOpen && (err != nil || open != string(disk)): - return open, true - case err == nil && stale: - return string(disk), true + compiled := map[string]beam.DebugInfo{module: info} + if src.compiled != nil { + for other := range moduleLines { + if other == module { + continue + } + if otherInfo, ok := src.compiled(other); ok && recordedPathMatch(otherInfo.File, otherInfo.RelativeFile, src.path) > 0 { + compiled[other] = otherInfo + } + } + } + + type anchor struct{ old, now int } + var above, below *anchor + consider := func(old, now int) { + switch { + case old < recorded && (above == nil || old > above.old): + above = &anchor{old, now} + case old > recorded && (below == nil || old < below.old): + below = &anchor{old, now} + } + } + for name, moduleInfo := range compiled { + for key, old := range moduleInfo.Anchors { + if now, ok := current[moduleKey{name, key}]; ok { + consider(old, now) + } + } + if now, ok := moduleLines[name]; ok && moduleInfo.ModuleLine > 0 { + consider(moduleInfo.ModuleLine, now) + } + } + if above != nil && below != nil && above.now-above.old == below.now-below.old { + return recorded + above.now - above.old, true + } + + // The edit was between the anchors. A def a comprehension generates has + // no name to search for, but its line is a `def unquote(...)`: if the + // compiled module's generated defs between the anchors have as many + // distinct lines as the current text has such defs there, they are the + // same defs in the same order. + oldFrom, oldTo, newFrom, newTo := 1, int(^uint(0)>>1), 1, len(src.lines) + if above != nil { + oldFrom, newFrom = above.old+1, above.now+1 + } + if below != nil { + oldTo, newTo = below.old-1, below.now-1 + } + var generated []int + for key, old := range info.Anchors { + if _, inSource := current[moduleKey{module, key}]; !inSource && old >= oldFrom && old <= oldTo && !slices.Contains(generated, old) { + generated = append(generated, old) + } } - return "", false + var unquoted []int + for line := newFrom; line <= newTo; line++ { + if lineDefinesUnquotedName(src.lines[line-1]) && src.ownerAt(line) == owner { + unquoted = append(unquoted, line) + } + } + if len(generated) == 0 || len(generated) != len(unquoted) { + return 0, false + } + slices.Sort(generated) + index := slices.Index(generated, recorded) + if index < 0 { + return 0, false + } + return unquoted[index], true +} + +// lineDefinesUnquotedName reports whether text starts a def, defp, defmacro, +// or defmacrop whose name is unquoted. +func lineDefinesUnquotedName(text string) bool { + rest := strings.TrimLeft(text, " \t") + for _, keyword := range []string{"def ", "defp ", "defmacro ", "defmacrop "} { + if after, ok := strings.CutPrefix(rest, keyword); ok { + return strings.HasPrefix(strings.TrimLeft(after, " \t"), "unquote(") + } + } + return false +} + +// uniqueDeclaration returns the one line in owner's body that declares one of +// functions by name, if exactly one does. +func (src *currentSource) uniqueDeclaration(owner string, functions []beam.Function) (int, bool) { + names := declarationNames(functions[0].Name) + found := 0 + for line := 1; line <= len(src.lines); line++ { + if !lineDeclaresAtom(src.lines[line-1], names) || src.ownerAt(line) != owner { + continue + } + if found != 0 { + return 0, false + } + found = line + } + return found, found != 0 +} + +// declarationNames is the names a declaring call can spell for function: the +// name itself, and without a trailing `!` or `?`, as `define :get` makes +// `get!` and `flag :active` makes `active?`. +func declarationNames(function string) []string { + names := []string{function} + if base := strings.TrimRight(function, "!?"); base != function && base != "" { + names = append(names, base) + } + return names } // blankHeredocs replaces the lines inside heredocs with empty lines, so that @@ -250,34 +533,32 @@ func blankHeredocs(lines []string) []string { } // nearestDeclaringLine returns the 1-based line nearest to recorded that -// declares function by name and that accept allows, or recorded when that is -// ambiguous or not found. A nil accept allows every line. -func nearestDeclaringLine(lines []string, recorded int, function string, accept func(line int) bool) int { +// declares function by name and that accept allows. found is false, and the +// line is recorded, when no line does or two are equally near. A nil accept +// allows every line. +func nearestDeclaringLine(lines []string, recorded int, function string, accept func(line int) bool) (line int, found bool) { if function == "" || recorded < 1 { - return recorded - } - names := []string{function} - if base := strings.TrimRight(function, "!?"); base != function && base != "" { - names = append(names, base) + return recorded, false } + names := declarationNames(function) declares := func(line int) bool { return line >= 1 && line <= len(lines) && (accept == nil || accept(line)) && lineDeclaresAtom(lines[line-1], names) } if declares(recorded) { - return recorded + return recorded, true } for distance := 1; recorded-distance >= 1 || recorded+distance <= len(lines); distance++ { above, below := declares(recorded-distance), declares(recorded+distance) switch { case above && below: - return recorded + return recorded, false case above: - return recorded - distance + return recorded - distance, true case below: - return recorded + distance + return recorded + distance, true } } - return recorded + return recorded, false } // lineDeclaresAtom reports whether text is a call whose first argument is one diff --git a/internal/lsp/generated_definition_integration_test.go b/internal/lsp/generated_definition_integration_test.go new file mode 100644 index 0000000..da3dfb2 --- /dev/null +++ b/internal/lsp/generated_definition_integration_test.go @@ -0,0 +1,348 @@ +package lsp + +import ( + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "testing" + "time" + + "go.lsp.dev/uri" +) + +// The fixture compiles DSLs of every shape navigation has to handle, with the +// real compiler, so a change in what Elixir records fails here. It needs mix, +// so it runs in the integration job. + +const compiledDslSource = `defmodule Weird.Dsl do + defmacro outer(name), do: quote(do: Weird.Dsl.inner(unquote(name))) + defmacro inner(name), do: quote(do: def(unquote(name)(), do: :inner)) + + defmacro flag(name) do + fun = :"#{name}?" + quote do: def(unquote(fun)(map), do: Map.get(map, unquote(name), false)) + end + + defmacro later(name), do: quote(do: @later_names(unquote(name))) + + defmacro __before_compile__(env) do + for name <- Module.get_attribute(env.module, :later_names) do + quote do: def(unquote(name)(), do: :later) + end + end + + defmacro __using__(_) do + quote do + import Weird.Dsl + Module.register_attribute(__MODULE__, :later_names, accumulate: true) + @before_compile Weird.Dsl + end + end + + defmacro pinned(name), do: quote(line: 99, do: def(unquote(name)(), do: :pinned)) + + defmacro route(verb, path) do + quote do: def(match(unquote(verb), unquote(path)), do: {unquote(verb), unquote(path)}) + end + + defmacro with_default(name), do: quote(do: def(unquote(name)(value \\ 1), do: value)) + + defmacro elsewhere(name) do + quote bind_quoted: [name: name] do + Module.create( + Module.concat(Weird.Elsewhere, Macro.camelize(Atom.to_string(name))), + quote(do: def(run, do: :elsewhere)), + __ENV__ + ) + end + end + + defmacro delegate(name, to), do: quote(do: defdelegate(unquote(name)(value), to: unquote(to))) + + defmacro many(prefix, count) do + for i <- 1..count do + fun = :"#{prefix}_#{i}" + quote do: def(unquote(fun)(), do: unquote(i)) + end + end + + defmacro kept_field(name) do + quote bind_quoted: [name: name], location: :keep do + def unquote(name)(), do: unquote(name) + end + end + + defmacro stamped_field(name) do + line = __CALLER__.line + + quote bind_quoted: [name: name, line: line] do + @file {__ENV__.file, line} + def unquote(name)(), do: unquote(name) + end + end +end +` + +const compiledUserSource = `defmodule Weird.User do + use Weird.Dsl + + outer :two_level + + flag :active + + later :deferred + + pinned :pinned_fun + + route :get, "/a" + route :post, "/b" + + with_default :defaulted + + elsewhere :remote + + delegate :double, Weird.Target + + many :gen, 2000 + + kept_field :kept + + stamped_field :stamped + + for name <- [:loop_a, :loop_b] do + def unquote(name)(), do: unquote(name) + end + + defmodule Nested do + use Weird.Dsl + flag :nested + end +end +` + +const compiledAnchoredSource = `defmodule Weird.Anchored do + def before_loop, do: :before + + for name <- [:looped] do + def unquote(name)(), do: unquote(name) + end + + defp after_loop, do: :after + def use_after, do: after_loop() +end +` + +const compiledCallerSource = `defmodule Weird.Caller do + import Weird.User, only: [active?: 1] + + def run do + Weird.User.two_level() + Weird.User.active?(%{}) + Weird.User.deferred() + Weird.User.pinned_fun() + Weird.User.match(:get, "/a") + Weird.User.defaulted(2) + Weird.Elsewhere.Remote.run() + Weird.User.double(2) + Weird.User.gen_1500() + Weird.User.kept() + Weird.User.stamped() + Weird.User.loop_b() + Weird.User.Nested.nested?(%{}) + Weird.Anchored.looped() + active?(%{}) + f = &Weird.User.two_level/0 + f.() + end +end +` + +const ( + compiledDslRel = "lib/weird/dsl.ex" + compiledUserRel = "lib/weird/user.ex" + compiledAnchoredRel = "lib/weird/anchored.ex" + compiledCallerRel = "lib/weird/caller.ex" +) + +// compiledFixture is the compiled DSL project and the server that indexed it. +type compiledFixture struct { + server *Server + t *testing.T +} + +func newCompiledFixture(t *testing.T, elixircOptions string) *compiledFixture { + t.Helper() + if _, err := exec.LookPath("mix"); err != nil { + t.Skip("mix not available") + } + server, cleanup := setupTestServer(t) + t.Cleanup(cleanup) + + mixExs := `defmodule Weird.MixProject do + use Mix.Project + + def project, do: [app: :weird, version: "0.1.0", elixir: "~> 1.18", elixirc_options: [` + elixircOptions + `]] +end +` + if err := os.WriteFile(filepath.Join(server.projectRoot, "mix.exs"), []byte(mixExs), 0o644); err != nil { + t.Fatal(err) + } + for rel, source := range map[string]string{ + compiledDslRel: compiledDslSource, + compiledUserRel: compiledUserSource, + compiledAnchoredRel: compiledAnchoredSource, + compiledCallerRel: compiledCallerSource, + "lib/weird/target.ex": `defmodule Weird.Target do + def double(value), do: value * 2 +end +`, + } { + indexFile(t, server.store, server.projectRoot, rel, source) + } + cmd := exec.Command("mix", "compile") + cmd.Dir = server.projectRoot + if output, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("compile the DSL fixture: %v\n%s", err, output) + } + return &compiledFixture{server: server, t: t} +} + +// saveLater saves and indexes an edit as if it were made a while after the +// compile. A BEAM's mtime has whole seconds, so an edit in the same second as +// the compile counts as compiled. +func (f *compiledFixture) saveLater(rel, source string) { + f.t.Helper() + indexFile(f.t, f.server.store, f.server.projectRoot, rel, source) + later := time.Now().Add(5 * time.Second) + if err := os.Chtimes(filepath.Join(f.server.projectRoot, rel), later, later); err != nil { + f.t.Fatal(err) + } +} + +// lineOf returns the 1-based line of the nth occurrence of needle in source. +func lineOf(t *testing.T, source, needle string, nth int) int { + t.Helper() + for i, line := range strings.Split(source, "\n") { + if strings.Contains(line, needle) { + if nth--; nth == 0 { + return i + 1 + } + } + } + t.Fatalf("%q not found in source", needle) + return 0 +} + +// definitionOf returns the definition lines for the nth occurrence of needle in +// the caller, with the cursor inside the name the needle starts with. +func (f *compiledFixture) definitionOf(needle string, nth int) []string { + f.t.Helper() + line := lineOf(f.t, compiledCallerSource, needle, nth) + text := strings.Split(compiledCallerSource, "\n")[line-1] + col := strings.Index(text, needle) + 1 + locations := generatedDefinitionAt(f.t, f.server, compiledCallerRel, compiledCallerSource, line-1, col) + var got []string + for _, location := range locations { + rel, _ := filepath.Rel(f.server.projectRoot, uriToPath(location.URI)) + got = append(got, rel+":"+strconv.Itoa(int(location.Range.Start.Line)+1)) + } + return got +} + +func at(rel, source, needle string, t *testing.T) string { + t.Helper() + return rel + ":" + strconv.Itoa(lineOf(t, source, needle, 1)) +} + +func expectDefinition(t *testing.T, got []string, want ...string) { + t.Helper() + if strings.Join(got, " ") != strings.Join(want, " ") { + t.Errorf("definition = %v, want %v", got, want) + } +} + +func TestDefinition_GeneratedFunctionsFromCompiler(t *testing.T) { + f := newCompiledFixture(t, "") + user := func(needle string) string { return at(compiledUserRel, compiledUserSource, needle, t) } + + for _, tc := range []struct { + name, needle string + nth int + want []string + }{ + {"macro that calls a macro", "two_level(", 1, []string{user("outer :two_level")}}, + {"predicate from an atom", "active?(%{})", 1, []string{user("flag :active")}}, + {"@before_compile hook", "deferred(", 1, []string{user("later :deferred")}}, + {"line past the end of the file", "pinned_fun(", 1, []string{user("pinned :pinned_fun")}}, + {"one clause per call", "match(", 1, []string{user(`route :get`), user(`route :post`)}}, + {"default arguments", "defaulted(", 1, []string{user("with_default :defaulted")}}, + {"module from Module.create", "run()", 1, []string{user("elsewhere :remote")}}, + {"generated defdelegate", "double(", 1, []string{user("delegate :double")}}, + {"one of many generated", "gen_1500(", 1, []string{user("many :gen")}}, + {"location: :keep", "kept(", 1, []string{user("kept_field :kept")}}, + {"@file stamp", "stamped(", 1, []string{user("stamped_field :stamped")}}, + {"comprehension", "loop_b(", 1, []string{user("def unquote(name)()")}}, + {"nested module", "nested?(", 1, []string{user("flag :nested")}}, + {"bare call through import", "active?(%{})", 2, []string{user("flag :active")}}, + {"capture", "two_level/0", 1, []string{user("outer :two_level")}}, + } { + t.Run(tc.name, func(t *testing.T) { + expectDefinition(t, f.definitionOf(tc.needle, tc.nth), tc.want...) + }) + } + + t.Run("module that exists only as a BEAM", func(t *testing.T) { + locations, err := f.server.LookupName("Weird.Elsewhere.Remote", "", NameLookupOptions{}) + if err != nil { + t.Fatal(err) + } + want := filepath.Join(f.server.projectRoot, compiledUserRel) + if len(locations) != 1 || locations[0].FilePath != want || locations[0].Line != lineOf(t, compiledUserSource, "elsewhere :remote", 1) { + t.Errorf("module location = %#v, want the elsewhere call in %s", locations, want) + } + }) +} + +// Dexter cannot compile, so the source usually has edits the BEAM has not +// seen. Each case edits the source without compiling and checks that the line +// follows the edit. +func TestDefinition_GeneratedFunctionsFromCompilerFollowEdits(t *testing.T) { + f := newCompiledFixture(t, "") + + t.Run("lines added above the module, saved", func(t *testing.T) { + edited := "# one\n# two\n# three\n" + compiledUserSource + f.saveLater(compiledUserRel, edited) + t.Cleanup(func() { indexFile(t, f.server.store, f.server.projectRoot, compiledUserRel, compiledUserSource) }) + expectDefinition(t, f.definitionOf("two_level(", 1), at(compiledUserRel, edited, "outer :two_level", t)) + expectDefinition(t, f.definitionOf("loop_b(", 1), at(compiledUserRel, edited, "def unquote(name)()", t)) + }) + + t.Run("lines added in an unsaved buffer", func(t *testing.T) { + edited := strings.Replace(compiledUserSource, " outer :two_level\n", " # one\n # two\n outer :two_level\n", 1) + userURI := string(uri.File(filepath.Join(f.server.projectRoot, compiledUserRel))) + f.server.docs.Set(userURI, edited) + t.Cleanup(func() { f.server.docs.Close(userURI) }) + expectDefinition(t, f.definitionOf("two_level(", 1), at(compiledUserRel, edited, "outer :two_level", t)) + expectDefinition(t, f.definitionOf("deferred(", 1), at(compiledUserRel, edited, "later :deferred", t)) + }) + + t.Run("lines added between a source def and a comprehension", func(t *testing.T) { + edited := strings.Replace(compiledAnchoredSource, " for name", " # one\n # two\n for name", 1) + f.saveLater(compiledAnchoredRel, edited) + t.Cleanup(func() { + indexFile(t, f.server.store, f.server.projectRoot, compiledAnchoredRel, compiledAnchoredSource) + }) + expectDefinition(t, f.definitionOf("looped(", 1), at(compiledAnchoredRel, edited, "def unquote(name)()", t)) + }) +} + +// A project compiled without debug info still has its Docs and compile info +// chunks, and its source. +func TestDefinition_GeneratedFunctionsFromCompilerWithoutDebugInfo(t *testing.T) { + f := newCompiledFixture(t, "debug_info: false") + user := func(needle string) string { return at(compiledUserRel, compiledUserSource, needle, t) } + expectDefinition(t, f.definitionOf("two_level(", 1), user("outer :two_level")) + expectDefinition(t, f.definitionOf("deferred(", 1), user("later :deferred")) +} diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index a74b1ee..439383d 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -3,6 +3,7 @@ package lsp import ( "context" "encoding/binary" + "maps" "os" "path/filepath" "slices" @@ -300,8 +301,9 @@ func TestLookupNameGeneratedFunctionUsesDebugInfoLine(t *testing.T) { } } -// Without a recorded line, a strict lookup still answers with the module. -func TestLookupNameStrictGeneratedFunctionWithoutLineKeepsModule(t *testing.T) { +// Without a recorded line, the call that declared the function is found in +// the module, and a strict lookup takes it: it is the function's own place. +func TestLookupNameStrictGeneratedFunctionWithoutLineFindsDeclaration(t *testing.T) { server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, ) @@ -309,20 +311,20 @@ func TestLookupNameStrictGeneratedFunctionWithoutLineKeepsModule(t *testing.T) { if err != nil { t.Fatal(err) } - if len(locations) != 1 || locations[0].FilePath != domainPath || locations[0].Line != generatedModuleLine { - t.Fatalf("expected %s:%d, got %#v", domainPath, generatedModuleLine, locations) + if len(locations) != 1 || locations[0].FilePath != domainPath || locations[0].Line != generatedDefineLine { + t.Fatalf("expected %s:%d, got %#v", domainPath, generatedDefineLine, locations) } } // A library's `quote location: :keep` names the library's own file. That is // the generator, not the declaration, and :line only says where the -// before-compile hook ran, which is no better than the module itself. -func TestDefinitionGeneratedFunctionForeignLocationKeepsModuleLine(t *testing.T) { +// before-compile hook ran. The module's body still has the declaring call. +func TestDefinitionGeneratedFunctionForeignLocationFindsDeclaration(t *testing.T) { server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, ) locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) - expectSingleLocation(t, locations, domainPath, generatedModuleLine) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) } // Dexter cannot compile the project, so a BEAM older than its source is the @@ -492,7 +494,7 @@ end`, "\n") tc.recorded = 2 tc.want = 2 } - if got := nearestDeclaringLine(text, tc.recorded, tc.function, nil); got != tc.want { + if got, _ := nearestDeclaringLine(text, tc.recorded, tc.function, nil); got != tc.want { t.Errorf("nearestDeclaringLine(%d, %q) = %d, want %d", tc.recorded, tc.function, got, tc.want) } }) @@ -581,10 +583,10 @@ func TestDefinitionGeneratedFunctionFollowsDriftAboveModule(t *testing.T) { // its end, and a line with nothing left to move to is not returned. func TestNearestDeclaringLinePastEndOfFile(t *testing.T) { lines := []string{"defmodule A do", " define :rename", "end"} - if got := nearestDeclaringLine(lines, 19, "rename", nil); got != 2 { + if got, _ := nearestDeclaringLine(lines, 19, "rename", nil); got != 2 { t.Errorf("nearestDeclaringLine past the end = %d, want 2", got) } - if got := nearestDeclaringLine(lines, 19, "missing", nil); got != 19 { + if got, _ := nearestDeclaringLine(lines, 19, "missing", nil); got != 19 { t.Errorf("nearestDeclaringLine without a match = %d, want the recorded 19", got) } } @@ -682,7 +684,7 @@ func TestNearestDeclaringLineSkipsHeredocs(t *testing.T) { define :get_room_by_slug end end`, "\n")) - if got := nearestDeclaringLine(lines, 4, "get_room_by_slug!", nil); got != 9 { + if got, _ := nearestDeclaringLine(lines, 4, "get_room_by_slug!", nil); got != 9 { t.Errorf("nearestDeclaringLine = %d, want 9, past the @moduledoc example", got) } } @@ -807,8 +809,9 @@ end } // A generator can give a def any line, such as `quote line: 99`. Even from a -// current BEAM, a line past the end of the file is not a place to go. -func TestDefinitionGeneratedFunctionLinePastEndOfFileKeepsModuleLine(t *testing.T) { +// current BEAM, a line past the end of the file is not a place to go; the +// declaring call is. +func TestDefinitionGeneratedFunctionLinePastEndOfFileFindsDeclaration(t *testing.T) { server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 99, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, ) @@ -818,5 +821,100 @@ func TestDefinitionGeneratedFunctionLinePastEndOfFileKeepsModuleLine(t *testing. t.Fatal(err) } locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) - expectSingleLocation(t, locations, domainPath, generatedModuleLine) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// With no line from the BEAM and no call that declares the name, the module +// is still the answer. Two declaring calls are no better than none. +func TestDefinitionGeneratedFunctionWithoutDeclarationKeepsModuleLine(t *testing.T) { + for _, tc := range []struct { + name, source string + }{ + {"no declaring call", strings.Replace(generatedDomainSource, " define :get_room_by_slug, action: :read, get_by: [:slug]\n", "", 1)}, + {"two declaring calls", strings.Replace(generatedDomainSource, " define :list_rooms, action: :read\n", " define :get_room_by_slug, action: :read\n", 1)}, + } { + t.Run(tc.name, func(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, + ) + indexFile(t, server.store, server.projectRoot, generatedDomainRel, tc.source) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedModuleLine) + }) + } +} + +func TestLineDefinesUnquotedName(t *testing.T) { + for text, want := range map[string]bool{ + " def unquote(name)(), do: name": true, + " defp unquote(fun)(x) do": true, + " defmacro unquote(name)(opts)": true, + " def named(), do: :ok": false, + " # def unquote(name)()": false, + " quote do: def(unquote(name)(), do:": false, + } { + if got := lineDefinesUnquotedName(text); got != want { + t.Errorf("lineDefinesUnquotedName(%q) = %v, want %v", text, got, want) + } + } +} + +// The compiled module records where its source defs were; the current text +// says where they are now. Anchors that moved together move the line; anchors +// that did not locate a single `def unquote` between them, or nothing. +func TestAnchoredLine(t *testing.T) { + info := beam.DebugInfo{ + ModuleLine: 1, + Anchors: map[beam.FunctionKey]int{ + {Name: "before_loop", Arity: 0}: 2, + {Name: "looped", Arity: 0}: 5, + {Name: "after_loop", Arity: 0}: 8, + }, + } + compiled := "defmodule A do\n def before_loop, do: 1\n\n for n <- [:looped] do\n def unquote(n)(), do: n\n end\n\n defp after_loop, do: 2\nend\n" + for _, tc := range []struct { + name, text string + want int + wantOK bool + }{ + {"moved together", "# x\n# y\n" + compiled, 7, true}, + {"edit between, one unquoted def", strings.Replace(compiled, " for n", " # x\n for n", 1), 6, true}, + {"edit between, a generated def with no unquoted line", strings.Replace(compiled, " for n", " # x\n for n", 1), 0, false}, + {"edit between, two unquoted defs", strings.Replace(compiled, " for n", " for m <- [:b] do\n def unquote(m)(), do: m\n end\n for n", 1), 0, false}, + } { + t.Run(tc.name, func(t *testing.T) { + src := ¤tSource{path: "a.ex", text: tc.text, lines: blankHeredocs(strings.Split(tc.text, "\n")), changed: true} + compiledInfo := info + if strings.HasPrefix(tc.name, "edit between, a generated def") { + // A DSL call also generated a def between the anchors; the one + // unquoted line cannot stand for both. + compiledInfo.Anchors = maps.Clone(info.Anchors) + compiledInfo.Anchors[beam.FunctionKey{Name: "from_dsl", Arity: 0}] = 3 + } + got, ok := src.anchoredLine("A", "A", 5, compiledInfo) + if got != tc.want || ok != tc.wantOK { + t.Errorf("anchoredLine = %d, %v; want %d, %v", got, ok, tc.want, tc.wantOK) + } + }) + } +} + +// A BEAM's mtime has whole seconds, so a source saved in the same second as the +// compile is not newer than its BEAM. +func TestNewerBySecond(t *testing.T) { + const second = int64(time.Second) + beam := 100 * second + for _, tc := range []struct { + mtime int64 + want bool + }{ + {beam, false}, + {beam + second/2, false}, + {beam + second, true}, + {beam - 1, false}, + } { + if got := newerBySecond(tc.mtime, beam); got != tc.want { + t.Errorf("newerBySecond(%d, %d) = %v, want %v", tc.mtime, beam, got, tc.want) + } + } } diff --git a/internal/lsp/name_navigation.go b/internal/lsp/name_navigation.go index 15ed5c5..3719bcf 100644 --- a/internal/lsp/name_navigation.go +++ b/internal/lsp/name_navigation.go @@ -47,6 +47,11 @@ type NameReferenceOptions struct { func (s *Server) LookupName(module, function string, opts NameLookupOptions) ([]NameLocation, error) { if function == "" { results, err := s.store.LookupModule(module) + if err == nil && len(results) == 0 { + // A module that exists only as a BEAM, such as one Module.create + // made, is defined where the compiler recorded it. + results = s.generatedModuleLocation(module) + } return s.lookupLocations(results, opts.ExcludeStdlib), err } diff --git a/internal/lsp/server.go b/internal/lsp/server.go index 335aecf..94df13a 100644 --- a/internal/lsp/server.go +++ b/internal/lsp/server.go @@ -3639,6 +3639,14 @@ func (s *Server) resolveBareFunctionModuleWithOrigin(filePath, text string, tf * return "Kernel", false } + // An imported module can export a function that a macro generated, which + // only its BEAM knows about. + for _, mod := range imports { + if _, found := s.generatedSymbol(mod, "", functionName); found { + return mod, false + } + } + // Slow fallback: function may be injected into an imported module via its // own use chain (e.g. MyApp.Factory uses ExMachina, which injects `insert`). for _, mod := range imports { From 7426ec26f160b0184f68193b35f011370be687bc Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Wed, 30 Sep 2026 15:40:54 -0400 Subject: [PATCH 09/16] Keep route clauses apart when a stale file is corrected Anchors that moved together are the most exact signal, so they now come before the search for a call that spells the function's name. That search is skipped for a function whose clauses several calls made: those calls do not spell its name, so a call that does, such as `plug :match`, would pull every route clause onto one line. Also test the cases from review: route clauses next to `plug :match` in a stale file, a multi-clause `location: :keep` generator, and lines removed above the module in an unsaved buffer. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/architecture.md | 5 +- internal/lsp/generated_definition.go | 46 +++++++++++++------ .../generated_definition_integration_test.go | 18 ++++++++ internal/lsp/generated_definition_test.go | 22 ++++++++- 4 files changed, 74 insertions(+), 17 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index e35b2fb..00750ed 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -151,8 +151,9 @@ Lines are checked against the current text, which is the buffer open in the edit A stale line is corrected where that can be done reliably. The correction runs only when the text may differ from the compiled text: the file was written in a later second than the BEAM, or a buffer open in the editor differs from the file. A BEAM's mtime has whole seconds, so a file saved in the same second as the compile counts as compiled; a file that another request only cached does not count as a buffer. The module's line is then taken from the current text, and the correction runs before the check against it, because lines added above `defmodule` move both. Every signal is limited to the body of the module that owns the declarations, found with the tokenizer, so a sibling module in the same file never matches. The signals, most exact first: -- The nearest line, outside heredocs, that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count. The recorded line is kept when it still declares the function, and two equally near matches count as none. -- Anchors: the nearest definitions above and below the recorded line that a compiled module and the current text both have. Lines move for the whole file, so the defs and module lines of every module in the file whose BEAM was compiled from it count. When both anchors moved by the same amount, the line moved with them. When they did not, the edit was between them, and the line is placed only if the compiled module's generated defs between the anchors match the current text's `def unquote(...)` lines there one to one, in order: the shape a comprehension leaves. +- Anchors: the nearest definitions above and below the recorded line that a compiled module and the current text both have. Lines move for the whole file, so the defs and module lines of every module in the file whose BEAM was compiled from it count. When both anchors moved by the same amount, the line moved with them. +- The nearest line, outside heredocs, that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count. The recorded line is kept when it still declares the function, and two equally near matches count as none. It is not used for a function whose clauses several calls made: those calls do not spell its name (`get "/a"` makes a `match/2` clause), so a call that does, such as `plug :match`, is not one of them. +- When the anchors disagree, the edit was between them, and the line is placed only if the compiled module's generated defs between the anchors match the current text's `def unquote(...)` lines there one to one, in order: the shape a comprehension leaves. A line with no signal is kept as recorded. A current BEAM is never corrected, because its lines are exact even where the declaring call does not spell the name, as with a route helper. A generated module with no source row has no owner in its generator's file, so its line is not corrected. diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index 36a4780..f0f8660 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -322,16 +322,17 @@ func (src *currentSource) currentModuleResults(results []store.LookupResult, own // Dexter cannot compile the project, so it reads the current text instead, // and only the body of owner, the module whose source holds the declarations: // a sibling module in the same file never matches, and without an owner no -// line is moved. Two signals are used, most exact first: +// line is moved. The signals, most exact first (see anchoredLine): // +// - The definitions the source still has on either side. The compiled +// module records their old lines and the current text has their new ones; +// when both moved by the same amount, so did this line. // - A call whose first argument is the function's name as an atom, which is // how a macro call names what it declares (`define :list_rooms`, // `field :email`). The nearest wins; the recorded line is kept when it // still declares the function, and two equally near matches count as none. -// - Otherwise, the definitions the source still has on either side. The -// compiled module records their old lines and the current text has their -// new ones, so the difference is how far this part of the file moved (see -// anchoredLine). +// It is not used for a function whose clauses several calls made. +// - A one-to-one match of generated defs to `def unquote(...)` lines. // // A line with neither signal is kept as recorded. func (src *currentSource) correctDrift(module, owner string, results []store.LookupResult, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { @@ -343,13 +344,24 @@ func (src *currentSource) correctDrift(module, owner string, results []store.Loo names[function.Arity] = function.Name } inOwner := func(line int) bool { return src.ownerAt(line) == owner } + clauses := make(map[int]int, len(results)) + for _, result := range results { + clauses[result.Arity]++ + } out := results[:0] for _, result := range results { - line, found := nearestDeclaringLine(src.lines, result.Line, names[result.Arity], inOwner) - if !found { - if moved, ok := src.anchoredLine(module, owner, result.Line, sources.debugInfo); ok { - line = moved + byName := func() (int, bool) { + // Clauses made by several calls come from calls that do not spell + // the function's name (`get "/a"` makes a `match/2` clause), so a + // call that does, such as `plug :match`, is not one of them. + if clauses[result.Arity] > 1 { + return 0, false } + return nearestDeclaringLine(src.lines, result.Line, names[result.Arity], inOwner) + } + line := result.Line + if moved, ok := src.anchoredLine(module, owner, result.Line, sources.debugInfo, byName); ok { + line = moved } if slices.ContainsFunc(out, func(r store.LookupResult) bool { return r.Line == line }) { continue @@ -365,13 +377,14 @@ func (src *currentSource) correctDrift(module, owner string, results []store.Loo // both have: the defs and the module lines of every module in the file whose // BEAM was compiled from it. // -// When both anchors moved by the same amount, the line moved with them. When -// they moved by different amounts, the edit was between them. Then the line is -// placed only when every generated def between the anchors can be matched to a -// line in owner that defines a function with an unquoted name +// When both anchors moved by the same amount, the line moved with them. +// Otherwise byName, if given, may place it by a call that declares the +// function. Failing that, the edit was between the anchors, and the line is +// placed only when every generated def between them can be matched to a line +// in owner that defines a function with an unquoted name // (`def unquote(name)()`), one to one and in order: the shape a comprehension // leaves. Otherwise it is not placed. -func (src *currentSource) anchoredLine(module, owner string, recorded int, info beam.DebugInfo) (int, bool) { +func (src *currentSource) anchoredLine(module, owner string, recorded int, info beam.DebugInfo, byName func() (int, bool)) (int, bool) { // Lines move for the whole file, so every module defined in it can serve // as an anchor, not only this one: a nested module's line counts as much // as a def. Another module's BEAM is used only when it was compiled from @@ -429,6 +442,11 @@ func (src *currentSource) anchoredLine(module, owner string, recorded int, info if above != nil && below != nil && above.now-above.old == below.now-below.old { return recorded + above.now - above.old, true } + if byName != nil { + if line, ok := byName(); ok { + return line, true + } + } // The edit was between the anchors. A def a comprehension generates has // no name to search for, but its line is a `def unquote(...)`: if the diff --git a/internal/lsp/generated_definition_integration_test.go b/internal/lsp/generated_definition_integration_test.go index da3dfb2..7f4b482 100644 --- a/internal/lsp/generated_definition_integration_test.go +++ b/internal/lsp/generated_definition_integration_test.go @@ -47,6 +47,14 @@ const compiledDslSource = `defmodule Weird.Dsl do quote do: def(match(unquote(verb), unquote(path)), do: {unquote(verb), unquote(path)}) end + defmacro plug(_name), do: nil + + defmacro keep_route(name, verb) do + quote bind_quoted: [name: name, verb: verb], location: :keep do + def unquote(name)(unquote(verb)), do: unquote(verb) + end + end + defmacro with_default(name), do: quote(do: def(unquote(name)(value \\ 1), do: value)) defmacro elsewhere(name) do @@ -96,9 +104,14 @@ const compiledUserSource = `defmodule Weird.User do pinned :pinned_fun + plug :match + route :get, "/a" route :post, "/b" + keep_route :handle, :get + keep_route :handle, :post + with_default :defaulted elsewhere :remote @@ -143,6 +156,7 @@ const compiledCallerSource = `defmodule Weird.Caller do Weird.User.deferred() Weird.User.pinned_fun() Weird.User.match(:get, "/a") + Weird.User.handle(:get) Weird.User.defaulted(2) Weird.Elsewhere.Remote.run() Weird.User.double(2) @@ -277,6 +291,7 @@ func TestDefinition_GeneratedFunctionsFromCompiler(t *testing.T) { {"@before_compile hook", "deferred(", 1, []string{user("later :deferred")}}, {"line past the end of the file", "pinned_fun(", 1, []string{user("pinned :pinned_fun")}}, {"one clause per call", "match(", 1, []string{user(`route :get`), user(`route :post`)}}, + {"one location: :keep clause per call", "handle(", 1, []string{user("keep_route :handle, :get"), user("keep_route :handle, :post")}}, {"default arguments", "defaulted(", 1, []string{user("with_default :defaulted")}}, {"module from Module.create", "run()", 1, []string{user("elsewhere :remote")}}, {"generated defdelegate", "double(", 1, []string{user("delegate :double")}}, @@ -317,6 +332,9 @@ func TestDefinition_GeneratedFunctionsFromCompilerFollowEdits(t *testing.T) { t.Cleanup(func() { indexFile(t, f.server.store, f.server.projectRoot, compiledUserRel, compiledUserSource) }) expectDefinition(t, f.definitionOf("two_level(", 1), at(compiledUserRel, edited, "outer :two_level", t)) expectDefinition(t, f.definitionOf("loop_b(", 1), at(compiledUserRel, edited, "def unquote(name)()", t)) + // `plug :match` spells the name; the route clauses do not, and follow + // the edit instead of collapsing onto it. + expectDefinition(t, f.definitionOf("match(", 1), at(compiledUserRel, edited, "route :get", t), at(compiledUserRel, edited, "route :post", t)) }) t.Run("lines added in an unsaved buffer", func(t *testing.T) { diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index 439383d..3c91bd0 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -891,7 +891,7 @@ func TestAnchoredLine(t *testing.T) { compiledInfo.Anchors = maps.Clone(info.Anchors) compiledInfo.Anchors[beam.FunctionKey{Name: "from_dsl", Arity: 0}] = 3 } - got, ok := src.anchoredLine("A", "A", 5, compiledInfo) + got, ok := src.anchoredLine("A", "A", 5, compiledInfo, nil) if got != tc.want || ok != tc.wantOK { t.Errorf("anchoredLine = %d, %v; want %d, %v", got, ok, tc.want, tc.wantOK) } @@ -918,3 +918,23 @@ func TestNewerBySecond(t *testing.T) { } } } + +// Lines removed above the module in an unsaved buffer move the declaration +// above the module line the index still has. The module line of the current +// text is the one that counts. +func TestDefinitionGeneratedFunctionLinesRemovedAboveModule(t *testing.T) { + header := strings.Repeat("# header\n", 10) + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: generatedDefineLine + 10, keepFile: "deps/ash/lib/ash/code_interface.ex", keepLine: 1112}, + ) + // The index and the BEAM have the header: the module is at 12, the define at 18. + indexFile(t, server.store, server.projectRoot, generatedDomainRel, header+generatedDomainSource) + past := time.Now().Add(-time.Hour) + if err := os.Chtimes(domainPath, past, past); err != nil { + t.Fatal(err) + } + // The buffer does not: the define is at 8, before the index's module line. + server.docs.Set(string(uri.File(domainPath)), generatedDomainSource) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} From 477f738997a7e8ece763d6fec82f14b3cfde2845 Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Wed, 30 Sep 2026 15:50:07 -0400 Subject: [PATCH 10/16] Remove the correction of stale definition lines A stale BEAM's line is now used as it is. Only edits to the declaring file move it, and those files (DSL modules, schemas, routers, dependencies) change less often than their callers; the next compile makes it exact. Correcting it meant guessing at the text the BEAM was compiled from, and each signal either gave up often or could send the editor to a line that looked meaningful and was not. What stays: a line past the end of the file is never returned, the module line check compares lines from the same compile, and the search for the one declaring call, which reads the current text, still places a def made at the module line. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- docs/architecture.md | 12 +- internal/beam/debug_info.go | 27 +- internal/lsp/generated_definition.go | 374 ++---------------- .../generated_definition_integration_test.go | 72 ++-- internal/lsp/generated_definition_test.go | 227 +---------- 6 files changed, 93 insertions(+), 621 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c4cc43..9effe86 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line. When the source or an unsaved buffer has changed since the compile, the line follows the edit: to the nearest call in the same module that declares the function by name as its first atom argument (`define :list_rooms`), or else by how far the source definitions around it moved, and a line that cannot be placed reliably stays as recorded. A function whose only recorded line is the module line, such as one a `@before_compile` hook made, goes to the one call in its module that declares it by name, if there is exactly one. A function with a clause per DSL call goes to every clause. A line past the end of the file is never returned. A module compiled without debug info falls back to its Docs chunk annotation, which is often, but not always, the same line. A generated module with no source of its own, such as one `Module.create` made or a Spark DSL entity, goes to the file it was compiled from, rebased onto the project when it was built elsewhere, and so does go-to-definition on its name. A bare call to a generated function of an imported module now resolves. Ash code interfaces go to their `define` line with released Ash, and from the recorded line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) +- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line; only edits to the declaring file move it, and the next compile makes it exact. A function whose only recorded line is the module line, such as one a `@before_compile` hook made, goes to the one call in its module that declares it by name, if there is exactly one. A function with a clause per DSL call goes to every clause. A line past the end of the file is never returned. A module compiled without debug info falls back to its Docs chunk annotation, which is often, but not always, the same line. A generated module with no source of its own, such as one `Module.create` made or a Spark DSL entity, goes to the file it was compiled from, rebased onto the project when it was built elsewhere, and so does go-to-definition on its name. A bare call to a generated function of an imported module now resolves. Ash code interfaces go to their `define` line with released Ash, and from the recorded line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) - **`--root`/`-C` names the workspace on every command** — `dexter init --root ~/project`, `dexter lookup --root ~/project MyApp.Repo`, and `dexter lsp --root ~/project` all run as if they had been started in that directory, so an agent, a script, or an editor wrapper can index or query a project from anywhere. Relative paths, a `reindex` target included, resolve from the named root diff --git a/docs/architecture.md b/docs/architecture.md index 00750ed..19f4d9a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -145,17 +145,11 @@ Each def carries two locations. Its `:line` is the line in the module's own file `generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (the same absolute path, or else the most path components shared from the end, which covers a moved, copied, or symlinked project; when a module has rows in two files, as two umbrella apps can, the most specific match wins, and an equal match trusts neither), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is closer than the module line. -`beam.ReadDefinitionLines` also reads the module's own line (the map's `anno`, a line or `{line, column}`), the line of each clause when a def's clauses were made at different lines (`DebugInfo.Clauses`: a DSL that adds a clause per call, such as `route :get, "/a"`, gets a location per clause), and the `:line` of every def, private ones included, as `DebugInfo.Anchors`. +`beam.ReadDefinitionLines` also reads the module's own line (the map's `anno`, a line or `{line, column}`) and the line of each clause when a def's clauses were made at different lines (`DebugInfo.Clauses`: a DSL that adds a clause per call, such as `route :get, "/a"`, gets a location per clause). -Lines are checked against the current text, which is the buffer open in the editor if there is one and the file on disk otherwise (`currentSource`). A line at or before the module's line as compiled carries no information and is dropped first. A line past the end of the current text is dropped too, because a generator can give a def any line (`quote line: 99`). When nothing is left, which is the case for a def a `@before_compile` hook made at the module line (and for released Ash's code interfaces), the owning module's body is searched for the declaring call: a call whose first argument is the function's name as an atom. It is used only when exactly one line in the module matches. +Lines are checked against the current text, which is the buffer open in the editor if there is one and the file on disk otherwise (`currentSource`). A line at or before the module's line carries no information and is dropped; both lines come from the same compile (the module's line is Dbgi's `anno` when it has one), so the comparison holds even when the file has changed since. A line past the end of the current text is dropped too, because a generator can give a def any line (`quote line: 99`) and an edit can remove lines. When nothing is left, which is the case for a def a `@before_compile` hook made at the module line (and for released Ash's code interfaces), the owning module's body is searched for the declaring call: a call, outside heredocs, whose first argument is the function's name as an atom, also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count, and neither does a call in a sibling module of the same file (found with the tokenizer). It is used only when exactly one line matches. -A stale line is corrected where that can be done reliably. The correction runs only when the text may differ from the compiled text: the file was written in a later second than the BEAM, or a buffer open in the editor differs from the file. A BEAM's mtime has whole seconds, so a file saved in the same second as the compile counts as compiled; a file that another request only cached does not count as a buffer. The module's line is then taken from the current text, and the correction runs before the check against it, because lines added above `defmodule` move both. Every signal is limited to the body of the module that owns the declarations, found with the tokenizer, so a sibling module in the same file never matches. The signals, most exact first: - -- Anchors: the nearest definitions above and below the recorded line that a compiled module and the current text both have. Lines move for the whole file, so the defs and module lines of every module in the file whose BEAM was compiled from it count. When both anchors moved by the same amount, the line moved with them. -- The nearest line, outside heredocs, that is a call whose first argument is the function's name as an atom (`define :list_rooms`, `Lib.define(:name)`), also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count. The recorded line is kept when it still declares the function, and two equally near matches count as none. It is not used for a function whose clauses several calls made: those calls do not spell its name (`get "/a"` makes a `match/2` clause), so a call that does, such as `plug :match`, is not one of them. -- When the anchors disagree, the edit was between them, and the line is placed only if the compiled module's generated defs between the anchors match the current text's `def unquote(...)` lines there one to one, in order: the shape a comprehension leaves. - -A line with no signal is kept as recorded. A current BEAM is never corrected, because its lines are exact even where the declaring call does not spell the name, as with a route helper. A generated module with no source row has no owner in its generator's file, so its line is not corrected. +A stale BEAM's lines are not corrected. Only edits to the declaring file move them, by the lines added or removed above, and those files (DSL modules, schemas, routers, dependencies) change less often than their callers. Correcting them meant guessing at text the BEAM was compiled from, and every signal tried either gave up often or could send the editor to a line that looked meaningful and was not. The line from the last compile is near, and the next compile makes it exact. A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually the generator's own file. A recorded path inside the project is used as it is. Otherwise it is rebased: first each tail of the recorded path under the project root, longest first, which covers a moved or copied checkout and an umbrella's `apps//lib`, then the Mix layouts for the application named below the last `lib/` (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. A recorded path outside the project is the last choice: it is right for a `path:` dependency, but a `_build` copied from another worktree records that worktree, which usually still exists. If no file is found, the lexical parent stays the answer. `CInf` records the path as codepoints, which are decoded to UTF-8. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. `LookupName` on the name of a module that has no source row returns the file from `CInf` at the module's recorded line, so go-to-definition on a module that `Module.create` made works too. A bare call that no source resolves also checks the BEAMs of the file's imported modules for a generated function of that name. diff --git a/internal/beam/debug_info.go b/internal/beam/debug_info.go index a8e7a9c..35a5059 100644 --- a/internal/beam/debug_info.go +++ b/internal/beam/debug_info.go @@ -42,11 +42,6 @@ type DebugInfo struct { // adds one clause per call, such as `route :get, "/a"`. Definitions with // one line are absent; Lines has it. Clauses map[FunctionKey][]int - - // Anchors maps every definition, private ones included, to its :line as - // compiled. Comparing these with the same definitions in the current - // source shows how far the source has moved since the compile. - Anchors map[FunctionKey]int } // ReadDefinitionLines reads the line of every public definition from an Elixir @@ -84,7 +79,6 @@ var errNoElixirDebugInfo = errors.New("no Elixir debug info") // relative_file, so they cannot be resolved while walking. type definitionSite struct { key FunctionKey - public bool line int keepFile string keepLine int @@ -182,16 +176,7 @@ func parseDebugInfo(buf []byte) (info DebugInfo, err error) { } info.Lines = make(map[FunctionKey]int, len(sites)) - info.Anchors = make(map[FunctionKey]int, len(sites)) for _, site := range sites { - if site.line > 0 { - if _, seen := info.Anchors[site.key]; !seen { - info.Anchors[site.key] = site.line - } - } - if !site.public { - continue - } line := site.line stamped := site.keepFile != "" && site.keepLine > 0 && (site.keepFile == info.RelativeFile || site.keepFile == info.File) @@ -214,9 +199,9 @@ func parseDebugInfo(buf []byte) (info DebugInfo, err error) { } // readDefinitionSites reads the definitions list, whose entries are -// {{name, arity}, kind, meta, clauses}. Private definitions are kept only as -// anchors: generated functions are found through the export table, so only -// public ones can be asked about. +// {{name, arity}, kind, meta, clauses}. Private definitions are skipped: +// generated functions are found through the export table, so only public ones +// can be asked about. func readDefinitionSites(r *etfReader) ([]definitionSite, error) { count, hasTail, err := r.enterList() if err != nil { @@ -254,14 +239,16 @@ func readDefinitionSites(r *etfReader) ([]definitionSite, error) { if err != nil { return nil, err } - site := definitionSite{key: FunctionKey{Name: name, Arity: functionArity}, public: kind == "def" || kind == "defmacro"} + site := definitionSite{key: FunctionKey{Name: name, Arity: functionArity}} if err := readDefinitionMeta(r, &site); err != nil { return nil, err } if site.clauseLines, err = readClauseLines(r); err != nil { return nil, err } - sites = append(sites, site) + if kind == "def" || kind == "defmacro" { + sites = append(sites, site) + } } if hasTail { if err := r.skip(); err != nil { diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index f0f8660..7175d07 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -22,10 +22,6 @@ type generatedDefinitionSources struct { // compileSource is the source path from the CInf chunk: the file the module // was compiled from, as it was spelled on the machine that compiled it. compileSource string - - // beamMtime is when the BEAM was written, to tell whether its lines can - // have drifted from the source. - beamMtime int64 } // generatedDefinitionSourcesFor returns a compiled module's definition sources. @@ -46,7 +42,7 @@ func (s *Server) generatedDefinitionSourcesFor(module, beamPath string) (generat if !started.IsZero() { s.debugf("Generated BEAM definition sources: module=%s definitions=%d error=%v source=%q total=%s", module, len(info.Lines), err, source, time.Since(started).Round(time.Microsecond)) } - entry.definitionSources = generatedDefinitionSources{debugInfo: info, compileSource: source, beamMtime: entry.beamStamp.mtime} + entry.definitionSources = generatedDefinitionSources{debugInfo: info, compileSource: source} entry.definitionSourcesResolved = true // Another request may have loaded a newer BEAM while this one read the // chunks. Its entry must not be replaced with this older one. @@ -67,14 +63,11 @@ func (s *Server) generatedDefinitionSourcesFor(module, beamPath string) (generat // still record a line in the file it was compiled from, which is usually a // framework file under deps. That file is tried after the indexed ones. // -// A recorded line is used only when it belongs to the file being opened and -// falls after the module's own line, since a line at or before it says nothing -// the module result does not. The BEAM may be older than the source. Dexter -// cannot compile the project, and a line from the last compile is usually -// right and at worst a few lines off, which is still closer than the module -// line. When the source has changed since the compile, correctLineDrift moves -// each line to follow the edit where it can. Anything else keeps the module -// result. +// The BEAM may be older than the source. Dexter cannot compile the project, +// and the line from the last compile is still the best one it has: an edit +// moves it by the lines added or removed above it, and the next compile makes +// it exact again. Only the declaring file's own edits move it; edits to the +// callers, which are most edits, do not. func (s *Server) generatedDefinitionResultsFor(module, beamPath string, functions []beam.Function) (results []store.LookupResult, precise bool) { results, owner := s.generatedDefinitionResults(module) if len(functions) == 0 { @@ -85,20 +78,16 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function return results, false } if result, found, ambiguous := sources.bestDescribedResult(results); found { - src := s.currentSourceOf(result.FilePath, sources.beamMtime) - if lines := recordedLinesIn(module, owner, result, src, sources, functions); len(lines) > 0 { + if lines := recordedLinesIn(module, owner, result, s.currentSourceOf(result.FilePath), sources, functions); len(lines) > 0 { return lines, true } - return src.currentModuleResults(results, owner), false + return results, false } else if ambiguous { return results, false } if owner != module { - // The generator's file has no module that owns the declaration, so - // its lines are not followed through edits there. if source := s.compiledSourceFile(sources.compileSource); source != "" { - src := s.currentSourceOf(source, sources.beamMtime) - if lines := recordedLinesIn(module, "", store.LookupResult{FilePath: source, Line: 1}, src, sources, functions); len(lines) > 0 { + if lines := recordedLinesIn(module, "", store.LookupResult{FilePath: source, Line: 1}, s.currentSourceOf(source), sources, functions); len(lines) > 0 { return lines, true } } @@ -122,20 +111,21 @@ func (s *Server) generatedModuleLocation(module string) []store.LookupResult { if source == "" { return nil } - if src := s.currentSourceOf(source, sources.beamMtime); src == nil || sources.debugInfo.ModuleLine > len(src.lines) { + if src := s.currentSourceOf(source); src == nil || sources.debugInfo.ModuleLine > len(src.lines) { return nil } return []store.LookupResult{{Module: module, FilePath: source, Line: sources.debugInfo.ModuleLine, Kind: "module"}} } // recordedLinesIn returns a result for each distinct line in result's file -// that the compiled module records for functions, corrected for edits made -// since the compile, and after the module's own line. +// that the compiled module records for functions, after the module's own line. // -// The module's line is taken from the current text when it has changed, so -// that it is compared after the correction: lines added above the module move -// both. A generator can give a def any line (`quote line: 99`), so a line past -// the end of the current text is dropped too. +// Both lines come from the same compile: the module's line is the one the BEAM +// records when it has one, and the index's otherwise. So the comparison holds +// even when the file has changed since. A line past the end of the current +// text is dropped, because a generator can give a def any line +// (`quote line: 99`) and an edit can remove lines: an editor is never sent to +// a line the file does not have. // // When nothing is left, which is the case for a def a @before_compile hook // made at the module line, the module's body is searched for the call that @@ -146,25 +136,16 @@ func recordedLinesIn(module, owner string, result store.LookupResult, src *curre return nil } moduleLine := result.Line - if line, ok := src.moduleLine(owner); ok && src.changed { - moduleLine = line - } - // A line at or before the module's line as compiled says nothing the - // module result does not, so it is dropped before any correction can move - // it somewhere that only looks meaningful. - compiledModuleLine := result.Line if owner == module && sources.debugInfo.ModuleLine > 0 { - compiledModuleLine = sources.debugInfo.ModuleLine + moduleLine = sources.debugInfo.ModuleLine } - lines := slices.DeleteFunc(generatedFunctionLines(module, result.FilePath, sources, functions), func(r store.LookupResult) bool { return r.Line <= compiledModuleLine }) - if src.changed { - lines = src.correctDrift(module, owner, lines, sources, functions) - } - lines = slices.DeleteFunc(lines, func(r store.LookupResult) bool { return r.Line <= moduleLine || r.Line > len(src.lines) }) + lines := slices.DeleteFunc(generatedFunctionLines(module, result.FilePath, sources, functions), func(r store.LookupResult) bool { + return r.Line <= moduleLine || r.Line > len(src.lines) + }) if len(lines) > 0 || owner == "" { return lines } - if line, ok := src.uniqueDeclaration(owner, functions); ok && line > moduleLine { + if line, ok := src.uniqueDeclaration(owner, functions); ok { return []store.LookupResult{{Module: module, FilePath: result.FilePath, Line: line, Kind: functions[0].Kind, Arity: functions[0].Arity}} } return nil @@ -207,61 +188,28 @@ func generatedFunctionLines(module, file string, sources generatedDefinitionSour return lines } -// currentSource is a file's text as it is now, in the open buffer if there is -// one and otherwise on disk, with what navigation derives from it computed on -// first use. +// currentSource is a file's text as it is now: in the open buffer if there is +// one, otherwise on disk. type currentSource struct { - path string text string // lines is text split into lines, with the lines inside heredocs blanked. - lines []string - // changed reports that the text may differ from the text the BEAM was - // compiled from: the buffer has unsaved changes, or the file was written - // after the BEAM. - changed bool - + lines []string owners []string - defs []parser.Definition - parsed bool - - // compiled returns the debug info of another module's BEAM, so that the - // other modules in this file can serve as anchors. It may be nil. - compiled func(module string) (beam.DebugInfo, bool) } // currentSourceOf returns path's current text, or nil if it cannot be read. // Only a buffer open in the editor counts: a file that another request cached -// is not unsaved edits. -func (s *Server) currentSourceOf(path string, beamMtime int64) *currentSource { - open, isOpen := s.docs.GetIfOpen(string(pathToURI(path))) - disk, err := os.ReadFile(path) - src := ¤tSource{path: path, compiled: func(module string) (beam.DebugInfo, bool) { - sources, ok := s.generatedDefinitionSourcesFor(module, "") - return sources.debugInfo, ok && len(sources.debugInfo.Anchors) > 0 - }} - switch { - case isOpen: - src.text = open - src.changed = err != nil || open != string(disk) - case err == nil: - src.text = string(disk) - default: - return nil - } - if stamp := statFileStamp(path); stamp.exists && newerBySecond(stamp.mtime, beamMtime) { - src.changed = true +// may be older than the file on disk. +func (s *Server) currentSourceOf(path string) *currentSource { + text, ok := s.docs.GetIfOpen(string(pathToURI(path))) + if !ok { + data, err := os.ReadFile(path) + if err != nil { + return nil + } + text = string(data) } - src.lines = blankHeredocs(strings.Split(src.text, "\n")) - return src -} - -// newerBySecond reports whether mtime is in a later second than beamMtime. A -// compiled BEAM's mtime has whole seconds while a source file's does not, so a -// file saved and compiled within one second would otherwise look newer than -// its BEAM. An edit in the same second as the compile counts as compiled, which -// only means its lines are not corrected. -func newerBySecond(mtime, beamMtime int64) bool { - return mtime/int64(time.Second) > beamMtime/int64(time.Second) + return ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} } // ownerAt returns the innermost module whose body holds the 1-based line. @@ -275,228 +223,9 @@ func (src *currentSource) ownerAt(line int) string { return src.owners[line] } -// definitions returns the definitions the parser finds in the current text. -func (src *currentSource) definitions() []parser.Definition { - if !src.parsed { - src.parsed = true - src.defs, _, _ = parser.ParseText(src.path, src.text) - } - return src.defs -} - -// moduleLine returns the line of module's defmodule in the current text. -func (src *currentSource) moduleLine(module string) (int, bool) { - if module == "" { - return 0, false - } - for _, def := range src.definitions() { - if def.Module == module && def.Function == "" { - return def.Line, true - } - } - return 0, false -} - -// currentModuleResults moves the module results in this file to owner's line -// in the current text, which differs from the index while a buffer has unsaved -// changes. -func (src *currentSource) currentModuleResults(results []store.LookupResult, owner string) []store.LookupResult { - if src == nil || !src.changed { - return results - } - line, ok := src.moduleLine(owner) - if !ok { - return results - } - out := slices.Clone(results) - for i := range out { - if out[i].FilePath == src.path { - out[i].Line = line - } - } - return out -} - -// correctDrift moves recorded lines to follow edits made since the compile. -// -// Dexter cannot compile the project, so it reads the current text instead, -// and only the body of owner, the module whose source holds the declarations: -// a sibling module in the same file never matches, and without an owner no -// line is moved. The signals, most exact first (see anchoredLine): -// -// - The definitions the source still has on either side. The compiled -// module records their old lines and the current text has their new ones; -// when both moved by the same amount, so did this line. -// - A call whose first argument is the function's name as an atom, which is -// how a macro call names what it declares (`define :list_rooms`, -// `field :email`). The nearest wins; the recorded line is kept when it -// still declares the function, and two equally near matches count as none. -// It is not used for a function whose clauses several calls made. -// - A one-to-one match of generated defs to `def unquote(...)` lines. -// -// A line with neither signal is kept as recorded. -func (src *currentSource) correctDrift(module, owner string, results []store.LookupResult, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { - if owner == "" { - return results - } - names := make(map[int]string, len(functions)) - for _, function := range functions { - names[function.Arity] = function.Name - } - inOwner := func(line int) bool { return src.ownerAt(line) == owner } - clauses := make(map[int]int, len(results)) - for _, result := range results { - clauses[result.Arity]++ - } - out := results[:0] - for _, result := range results { - byName := func() (int, bool) { - // Clauses made by several calls come from calls that do not spell - // the function's name (`get "/a"` makes a `match/2` clause), so a - // call that does, such as `plug :match`, is not one of them. - if clauses[result.Arity] > 1 { - return 0, false - } - return nearestDeclaringLine(src.lines, result.Line, names[result.Arity], inOwner) - } - line := result.Line - if moved, ok := src.anchoredLine(module, owner, result.Line, sources.debugInfo, byName); ok { - line = moved - } - if slices.ContainsFunc(out, func(r store.LookupResult) bool { return r.Line == line }) { - continue - } - result.Line = line - out = append(out, result) - } - return out -} - -// anchoredLine places recorded in the current text by the nearest -// definitions above and below it that a compiled module and the current text -// both have: the defs and the module lines of every module in the file whose -// BEAM was compiled from it. -// -// When both anchors moved by the same amount, the line moved with them. -// Otherwise byName, if given, may place it by a call that declares the -// function. Failing that, the edit was between the anchors, and the line is -// placed only when every generated def between them can be matched to a line -// in owner that defines a function with an unquoted name -// (`def unquote(name)()`), one to one and in order: the shape a comprehension -// leaves. Otherwise it is not placed. -func (src *currentSource) anchoredLine(module, owner string, recorded int, info beam.DebugInfo, byName func() (int, bool)) (int, bool) { - // Lines move for the whole file, so every module defined in it can serve - // as an anchor, not only this one: a nested module's line counts as much - // as a def. Another module's BEAM is used only when it was compiled from - // this file. - type moduleKey struct { - module string - key beam.FunctionKey - } - current := make(map[moduleKey]int) - moduleLines := make(map[string]int) - for _, def := range src.definitions() { - if def.Function == "" { - if _, seen := moduleLines[def.Module]; !seen { - moduleLines[def.Module] = def.Line - } - continue - } - key := moduleKey{def.Module, beam.FunctionKey{Name: def.Function, Arity: def.Arity}} - if line, seen := current[key]; !seen || def.Line < line { - current[key] = def.Line - } - } - compiled := map[string]beam.DebugInfo{module: info} - if src.compiled != nil { - for other := range moduleLines { - if other == module { - continue - } - if otherInfo, ok := src.compiled(other); ok && recordedPathMatch(otherInfo.File, otherInfo.RelativeFile, src.path) > 0 { - compiled[other] = otherInfo - } - } - } - - type anchor struct{ old, now int } - var above, below *anchor - consider := func(old, now int) { - switch { - case old < recorded && (above == nil || old > above.old): - above = &anchor{old, now} - case old > recorded && (below == nil || old < below.old): - below = &anchor{old, now} - } - } - for name, moduleInfo := range compiled { - for key, old := range moduleInfo.Anchors { - if now, ok := current[moduleKey{name, key}]; ok { - consider(old, now) - } - } - if now, ok := moduleLines[name]; ok && moduleInfo.ModuleLine > 0 { - consider(moduleInfo.ModuleLine, now) - } - } - if above != nil && below != nil && above.now-above.old == below.now-below.old { - return recorded + above.now - above.old, true - } - if byName != nil { - if line, ok := byName(); ok { - return line, true - } - } - - // The edit was between the anchors. A def a comprehension generates has - // no name to search for, but its line is a `def unquote(...)`: if the - // compiled module's generated defs between the anchors have as many - // distinct lines as the current text has such defs there, they are the - // same defs in the same order. - oldFrom, oldTo, newFrom, newTo := 1, int(^uint(0)>>1), 1, len(src.lines) - if above != nil { - oldFrom, newFrom = above.old+1, above.now+1 - } - if below != nil { - oldTo, newTo = below.old-1, below.now-1 - } - var generated []int - for key, old := range info.Anchors { - if _, inSource := current[moduleKey{module, key}]; !inSource && old >= oldFrom && old <= oldTo && !slices.Contains(generated, old) { - generated = append(generated, old) - } - } - var unquoted []int - for line := newFrom; line <= newTo; line++ { - if lineDefinesUnquotedName(src.lines[line-1]) && src.ownerAt(line) == owner { - unquoted = append(unquoted, line) - } - } - if len(generated) == 0 || len(generated) != len(unquoted) { - return 0, false - } - slices.Sort(generated) - index := slices.Index(generated, recorded) - if index < 0 { - return 0, false - } - return unquoted[index], true -} - -// lineDefinesUnquotedName reports whether text starts a def, defp, defmacro, -// or defmacrop whose name is unquoted. -func lineDefinesUnquotedName(text string) bool { - rest := strings.TrimLeft(text, " \t") - for _, keyword := range []string{"def ", "defp ", "defmacro ", "defmacrop "} { - if after, ok := strings.CutPrefix(rest, keyword); ok { - return strings.HasPrefix(strings.TrimLeft(after, " \t"), "unquote(") - } - } - return false -} - // uniqueDeclaration returns the one line in owner's body that declares one of -// functions by name, if exactly one does. +// functions by name, if exactly one does. A sibling module in the same file +// that declares the same name does not count. func (src *currentSource) uniqueDeclaration(owner string, functions []beam.Function) (int, bool) { names := declarationNames(functions[0].Name) found := 0 @@ -550,35 +279,6 @@ func blankHeredocs(lines []string) []string { return out } -// nearestDeclaringLine returns the 1-based line nearest to recorded that -// declares function by name and that accept allows. found is false, and the -// line is recorded, when no line does or two are equally near. A nil accept -// allows every line. -func nearestDeclaringLine(lines []string, recorded int, function string, accept func(line int) bool) (line int, found bool) { - if function == "" || recorded < 1 { - return recorded, false - } - names := declarationNames(function) - declares := func(line int) bool { - return line >= 1 && line <= len(lines) && (accept == nil || accept(line)) && lineDeclaresAtom(lines[line-1], names) - } - if declares(recorded) { - return recorded, true - } - for distance := 1; recorded-distance >= 1 || recorded+distance <= len(lines); distance++ { - above, below := declares(recorded-distance), declares(recorded+distance) - switch { - case above && below: - return recorded, false - case above: - return recorded - distance, true - case below: - return recorded + distance, true - } - } - return recorded, false -} - // lineDeclaresAtom reports whether text is a call whose first argument is one // of names as an atom: `define :list_rooms`, `field :email, :string`, or // `Lib.define(:name)`. That is the shape of a macro call that declares a diff --git a/internal/lsp/generated_definition_integration_test.go b/internal/lsp/generated_definition_integration_test.go index 7f4b482..52ba234 100644 --- a/internal/lsp/generated_definition_integration_test.go +++ b/internal/lsp/generated_definition_integration_test.go @@ -7,7 +7,6 @@ import ( "strconv" "strings" "testing" - "time" "go.lsp.dev/uri" ) @@ -135,18 +134,6 @@ const compiledUserSource = `defmodule Weird.User do end ` -const compiledAnchoredSource = `defmodule Weird.Anchored do - def before_loop, do: :before - - for name <- [:looped] do - def unquote(name)(), do: unquote(name) - end - - defp after_loop, do: :after - def use_after, do: after_loop() -end -` - const compiledCallerSource = `defmodule Weird.Caller do import Weird.User, only: [active?: 1] @@ -165,7 +152,6 @@ const compiledCallerSource = `defmodule Weird.Caller do Weird.User.stamped() Weird.User.loop_b() Weird.User.Nested.nested?(%{}) - Weird.Anchored.looped() active?(%{}) f = &Weird.User.two_level/0 f.() @@ -174,10 +160,9 @@ end ` const ( - compiledDslRel = "lib/weird/dsl.ex" - compiledUserRel = "lib/weird/user.ex" - compiledAnchoredRel = "lib/weird/anchored.ex" - compiledCallerRel = "lib/weird/caller.ex" + compiledDslRel = "lib/weird/dsl.ex" + compiledUserRel = "lib/weird/user.ex" + compiledCallerRel = "lib/weird/caller.ex" ) // compiledFixture is the compiled DSL project and the server that indexed it. @@ -204,10 +189,9 @@ end t.Fatal(err) } for rel, source := range map[string]string{ - compiledDslRel: compiledDslSource, - compiledUserRel: compiledUserSource, - compiledAnchoredRel: compiledAnchoredSource, - compiledCallerRel: compiledCallerSource, + compiledDslRel: compiledDslSource, + compiledUserRel: compiledUserSource, + compiledCallerRel: compiledCallerSource, "lib/weird/target.ex": `defmodule Weird.Target do def double(value), do: value * 2 end @@ -223,16 +207,10 @@ end return &compiledFixture{server: server, t: t} } -// saveLater saves and indexes an edit as if it were made a while after the -// compile. A BEAM's mtime has whole seconds, so an edit in the same second as -// the compile counts as compiled. +// saveLater saves and indexes an edit made after the compile. func (f *compiledFixture) saveLater(rel, source string) { f.t.Helper() indexFile(f.t, f.server.store, f.server.projectRoot, rel, source) - later := time.Now().Add(5 * time.Second) - if err := os.Chtimes(filepath.Join(f.server.projectRoot, rel), later, later); err != nil { - f.t.Fatal(err) - } } // lineOf returns the 1-based line of the nth occurrence of needle in source. @@ -320,21 +298,20 @@ func TestDefinition_GeneratedFunctionsFromCompiler(t *testing.T) { }) } -// Dexter cannot compile, so the source usually has edits the BEAM has not -// seen. Each case edits the source without compiling and checks that the line -// follows the edit. -func TestDefinition_GeneratedFunctionsFromCompilerFollowEdits(t *testing.T) { +// Dexter cannot compile, so the declaring file can have edits the BEAM has not +// seen. The line from the last compile is still the answer: it is near, and +// the next compile makes it exact. A line the file no longer has is never +// returned, and the search for a declaring call reads the current text. +func TestDefinition_GeneratedFunctionsFromCompilerWithStaleSource(t *testing.T) { f := newCompiledFixture(t, "") + compiled := func(needle string) string { return at(compiledUserRel, compiledUserSource, needle, t) } - t.Run("lines added above the module, saved", func(t *testing.T) { + t.Run("lines added above, saved", func(t *testing.T) { edited := "# one\n# two\n# three\n" + compiledUserSource f.saveLater(compiledUserRel, edited) t.Cleanup(func() { indexFile(t, f.server.store, f.server.projectRoot, compiledUserRel, compiledUserSource) }) - expectDefinition(t, f.definitionOf("two_level(", 1), at(compiledUserRel, edited, "outer :two_level", t)) - expectDefinition(t, f.definitionOf("loop_b(", 1), at(compiledUserRel, edited, "def unquote(name)()", t)) - // `plug :match` spells the name; the route clauses do not, and follow - // the edit instead of collapsing onto it. - expectDefinition(t, f.definitionOf("match(", 1), at(compiledUserRel, edited, "route :get", t), at(compiledUserRel, edited, "route :post", t)) + expectDefinition(t, f.definitionOf("two_level(", 1), compiled("outer :two_level")) + expectDefinition(t, f.definitionOf("match(", 1), compiled("route :get"), compiled("route :post")) }) t.Run("lines added in an unsaved buffer", func(t *testing.T) { @@ -342,17 +319,18 @@ func TestDefinition_GeneratedFunctionsFromCompilerFollowEdits(t *testing.T) { userURI := string(uri.File(filepath.Join(f.server.projectRoot, compiledUserRel))) f.server.docs.Set(userURI, edited) t.Cleanup(func() { f.server.docs.Close(userURI) }) - expectDefinition(t, f.definitionOf("two_level(", 1), at(compiledUserRel, edited, "outer :two_level", t)) + expectDefinition(t, f.definitionOf("two_level(", 1), compiled("outer :two_level")) + // The BEAM records only the module line here, so the declaring call + // is searched for, in the text as it is now. expectDefinition(t, f.definitionOf("deferred(", 1), at(compiledUserRel, edited, "later :deferred", t)) }) - t.Run("lines added between a source def and a comprehension", func(t *testing.T) { - edited := strings.Replace(compiledAnchoredSource, " for name", " # one\n # two\n for name", 1) - f.saveLater(compiledAnchoredRel, edited) - t.Cleanup(func() { - indexFile(t, f.server.store, f.server.projectRoot, compiledAnchoredRel, compiledAnchoredSource) - }) - expectDefinition(t, f.definitionOf("looped(", 1), at(compiledAnchoredRel, edited, "def unquote(name)()", t)) + t.Run("file shorter than the recorded line", func(t *testing.T) { + edited := "defmodule Weird.User do\n use Weird.Dsl\n outer :two_level\nend\n" + f.saveLater(compiledUserRel, edited) + t.Cleanup(func() { indexFile(t, f.server.store, f.server.projectRoot, compiledUserRel, compiledUserSource) }) + expectDefinition(t, f.definitionOf("gen_1500(", 1), compiledUserRel+":1") + expectDefinition(t, f.definitionOf("two_level(", 1), compiled("outer :two_level")) }) } diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index 3c91bd0..878355f 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -3,7 +3,6 @@ package lsp import ( "context" "encoding/binary" - "maps" "os" "path/filepath" "slices" @@ -463,43 +462,7 @@ func TestPrepareCallHierarchyGeneratedFunctionUsesDebugInfoLine(t *testing.T) { } } -func TestNearestDeclaringLine(t *testing.T) { - lines := strings.Split(`defmodule MyApp.Chat do - resources do - define :list_rooms, action: :read - define :get_room_by_slug, action: :read, get_by: [:slug] - end - - def run, do: get_room_by_slug!("lounge") - def typed(x :: :get_room_by_slug), do: x - def other, do: [key:get_room_by_slug] -end`, "\n") - for _, tc := range []struct { - name string - recorded int - function string - want int - }{ - {"recorded line still declares it", 4, "get_room_by_slug!", 4}, - {"declaration moved down", 2, "get_room_by_slug!", 4}, - {"declaration moved up", 7, "get_room_by_slug!", 4}, - {"full name as atom", 1, "list_rooms", 3}, - {"name nowhere keeps the line", 5, "create_room", 5}, - {"equally near matches keep the line", 3, "tie", 3}, - } { - t.Run(tc.name, func(t *testing.T) { - text := lines - if tc.function == "tie" { - text = []string{"define :tie", "", "define :tie"} - tc.recorded = 2 - tc.want = 2 - } - if got, _ := nearestDeclaringLine(text, tc.recorded, tc.function, nil); got != tc.want { - t.Errorf("nearestDeclaringLine(%d, %q) = %d, want %d", tc.recorded, tc.function, got, tc.want) - } - }) - } - +func TestLineDeclaresAtom(t *testing.T) { names := []string{"get_room_by_slug!", "get_room_by_slug"} for _, text := range []string{ ` define :get_room_by_slug, action: :read`, @@ -529,28 +492,13 @@ end`, "\n") } } -// Lines added above the declaration after the compile: the recorded line has -// drifted, and the current text still says where the declaration is. -func TestDefinitionGeneratedFunctionFollowsDriftInStaleSource(t *testing.T) { - server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) - past := time.Now().Add(-time.Hour) - beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") - if err := os.Chtimes(beamPath, past, past); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(domainPath, []byte("# one\n# two\n"+generatedDomainSource), 0o644); err != nil { - t.Fatal(err) - } - locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) - expectSingleLocation(t, locations, domainPath, generatedDefineLine+2) -} - -// Unsaved edits move lines too, with nothing on disk newer than the BEAM. -func TestDefinitionGeneratedFunctionFollowsDriftInOpenBuffer(t *testing.T) { +// Lines added above the declaration after the compile move it, but Dexter +// cannot compile: the line from the last compile is the answer until the next. +func TestDefinitionGeneratedFunctionStaleSourceKeepsCompiledLine(t *testing.T) { server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) - server.docs.Set(string(uri.File(domainPath)), "# one\n"+generatedDomainSource) + indexFile(t, server.store, server.projectRoot, generatedDomainRel, "# one\n# two\n"+generatedDomainSource) locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) - expectSingleLocation(t, locations, domainPath, generatedDefineLine+1) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) } // A current BEAM's lines are exact, even when the declaring line does not @@ -568,29 +516,6 @@ func TestDefinitionGeneratedFunctionCurrentBEAMKeepsRecordedLine(t *testing.T) { expectSingleLocation(t, locations, domainPath, 6) } -// Lines added above the module move the module line too. The recorded line is -// compared with it only after following the edit. -func TestDefinitionGeneratedFunctionFollowsDriftAboveModule(t *testing.T) { - server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) - header := strings.Repeat("# header\n", 10) - indexFile(t, server.store, server.projectRoot, generatedDomainRel, header+generatedDomainSource) - server.docs.Set(string(uri.File(domainPath)), header+generatedDomainSource) - locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) - expectSingleLocation(t, locations, domainPath, generatedDefineLine+10) -} - -// A file that became shorter than the recorded line is still searched from -// its end, and a line with nothing left to move to is not returned. -func TestNearestDeclaringLinePastEndOfFile(t *testing.T) { - lines := []string{"defmodule A do", " define :rename", "end"} - if got, _ := nearestDeclaringLine(lines, 19, "rename", nil); got != 2 { - t.Errorf("nearestDeclaringLine past the end = %d, want 2", got) - } - if got, _ := nearestDeclaringLine(lines, 19, "missing", nil); got != 19 { - t.Errorf("nearestDeclaringLine without a match = %d, want the recorded 19", got) - } -} - func TestDefinitionGeneratedFunctionPastEndOfFileKeepsModuleLine(t *testing.T) { server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) short := "defmodule MyApp.Chat do\nend\n" @@ -671,9 +596,10 @@ func TestSourceRebaseCandidates(t *testing.T) { } } -// An example in a @moduledoc is not a declaration, even when it is nearer. -func TestNearestDeclaringLineSkipsHeredocs(t *testing.T) { - lines := blankHeredocs(strings.Split(`defmodule MyApp.Chat do +// An example in a @moduledoc is not a declaration, and neither is the same +// call in a sibling module of the same file. +func TestUniqueDeclarationSkipsHeredocsAndSiblings(t *testing.T) { + text := `defmodule MyApp.Chat do @moduledoc """ Example: @@ -683,9 +609,16 @@ func TestNearestDeclaringLineSkipsHeredocs(t *testing.T) { resources do define :get_room_by_slug end -end`, "\n")) - if got, _ := nearestDeclaringLine(lines, 4, "get_room_by_slug!", nil); got != 9 { - t.Errorf("nearestDeclaringLine = %d, want 9, past the @moduledoc example", got) +end + +defmodule MyApp.Archive do + define :get_room_by_slug +end +` + src := ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} + functions := []beam.Function{{Name: "get_room_by_slug!", Arity: 1}} + if got, ok := src.uniqueDeclaration("MyApp.Chat", functions); !ok || got != 9 { + t.Errorf("uniqueDeclaration = %d, %v; want 9, the one call in MyApp.Chat's body", got, ok) } } @@ -716,31 +649,6 @@ func TestDefinitionGeneratedFunctionIgnoresCachedFile(t *testing.T) { expectSingleLocation(t, locations, domainPath, generatedDefineLine+1) } -// Another module in the same file can declare the same name. After an edit it -// can be nearer to the recorded line than the real declaration, but a line in -// another module is never the answer. -func TestDefinitionGeneratedFunctionDriftStaysInItsModule(t *testing.T) { - server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) - sibling := `defmodule MyApp.Archive do - use SharedLib.Domain - - resources do - resource MyApp.Archive.Room do - define :get_room_by_slug, action: :read - end - end -end - -` - // The recorded line is 8. The sibling's define is now line 6, two away; - // this module's define is line 18, ten away. - edited := sibling + generatedDomainSource - indexFile(t, server.store, server.projectRoot, generatedDomainRel, edited) - server.docs.Set(string(uri.File(domainPath)), edited) - locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) - expectSingleLocation(t, locations, domainPath, generatedDefineLine+10) -} - func TestModuleOwnersByLine(t *testing.T) { source := `defmodule A do @moduledoc """ @@ -843,98 +751,3 @@ func TestDefinitionGeneratedFunctionWithoutDeclarationKeepsModuleLine(t *testing }) } } - -func TestLineDefinesUnquotedName(t *testing.T) { - for text, want := range map[string]bool{ - " def unquote(name)(), do: name": true, - " defp unquote(fun)(x) do": true, - " defmacro unquote(name)(opts)": true, - " def named(), do: :ok": false, - " # def unquote(name)()": false, - " quote do: def(unquote(name)(), do:": false, - } { - if got := lineDefinesUnquotedName(text); got != want { - t.Errorf("lineDefinesUnquotedName(%q) = %v, want %v", text, got, want) - } - } -} - -// The compiled module records where its source defs were; the current text -// says where they are now. Anchors that moved together move the line; anchors -// that did not locate a single `def unquote` between them, or nothing. -func TestAnchoredLine(t *testing.T) { - info := beam.DebugInfo{ - ModuleLine: 1, - Anchors: map[beam.FunctionKey]int{ - {Name: "before_loop", Arity: 0}: 2, - {Name: "looped", Arity: 0}: 5, - {Name: "after_loop", Arity: 0}: 8, - }, - } - compiled := "defmodule A do\n def before_loop, do: 1\n\n for n <- [:looped] do\n def unquote(n)(), do: n\n end\n\n defp after_loop, do: 2\nend\n" - for _, tc := range []struct { - name, text string - want int - wantOK bool - }{ - {"moved together", "# x\n# y\n" + compiled, 7, true}, - {"edit between, one unquoted def", strings.Replace(compiled, " for n", " # x\n for n", 1), 6, true}, - {"edit between, a generated def with no unquoted line", strings.Replace(compiled, " for n", " # x\n for n", 1), 0, false}, - {"edit between, two unquoted defs", strings.Replace(compiled, " for n", " for m <- [:b] do\n def unquote(m)(), do: m\n end\n for n", 1), 0, false}, - } { - t.Run(tc.name, func(t *testing.T) { - src := ¤tSource{path: "a.ex", text: tc.text, lines: blankHeredocs(strings.Split(tc.text, "\n")), changed: true} - compiledInfo := info - if strings.HasPrefix(tc.name, "edit between, a generated def") { - // A DSL call also generated a def between the anchors; the one - // unquoted line cannot stand for both. - compiledInfo.Anchors = maps.Clone(info.Anchors) - compiledInfo.Anchors[beam.FunctionKey{Name: "from_dsl", Arity: 0}] = 3 - } - got, ok := src.anchoredLine("A", "A", 5, compiledInfo, nil) - if got != tc.want || ok != tc.wantOK { - t.Errorf("anchoredLine = %d, %v; want %d, %v", got, ok, tc.want, tc.wantOK) - } - }) - } -} - -// A BEAM's mtime has whole seconds, so a source saved in the same second as the -// compile is not newer than its BEAM. -func TestNewerBySecond(t *testing.T) { - const second = int64(time.Second) - beam := 100 * second - for _, tc := range []struct { - mtime int64 - want bool - }{ - {beam, false}, - {beam + second/2, false}, - {beam + second, true}, - {beam - 1, false}, - } { - if got := newerBySecond(tc.mtime, beam); got != tc.want { - t.Errorf("newerBySecond(%d, %d) = %v, want %v", tc.mtime, beam, got, tc.want) - } - } -} - -// Lines removed above the module in an unsaved buffer move the declaration -// above the module line the index still has. The module line of the current -// text is the one that counts. -func TestDefinitionGeneratedFunctionLinesRemovedAboveModule(t *testing.T) { - header := strings.Repeat("# header\n", 10) - server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, - dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: generatedDefineLine + 10, keepFile: "deps/ash/lib/ash/code_interface.ex", keepLine: 1112}, - ) - // The index and the BEAM have the header: the module is at 12, the define at 18. - indexFile(t, server.store, server.projectRoot, generatedDomainRel, header+generatedDomainSource) - past := time.Now().Add(-time.Hour) - if err := os.Chtimes(domainPath, past, past); err != nil { - t.Fatal(err) - } - // The buffer does not: the define is at 8, before the index's module line. - server.docs.Set(string(uri.File(domainPath)), generatedDomainSource) - locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) - expectSingleLocation(t, locations, domainPath, generatedDefineLine) -} From d70acb97680110c00fce8cbcd051687257569acf Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Wed, 30 Sep 2026 23:28:00 -0400 Subject: [PATCH 11/16] Compare recorded lines with the module line from the same compile The module line came from the index when the BEAM had none for the module, as for a module a macro nested in its parent. The index has the current text, so after lines were added above defmodule it could fall after a recorded line from the compile and drop it. Take the parent's line from the parent's BEAM, and use the index only when no BEAM has one. Do not search for a declaring call for a function whose clauses were made at several lines: those calls do not spell its name, so a call that does, such as `plug :match`, is not where they were declared. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/architecture.md | 2 +- internal/lsp/generated_definition.go | 41 ++++++++++++++----- .../generated_definition_integration_test.go | 29 ++++++++++++- 3 files changed, 60 insertions(+), 12 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 19f4d9a..89da62e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -147,7 +147,7 @@ Each def carries two locations. Its `:line` is the line in the module's own file `beam.ReadDefinitionLines` also reads the module's own line (the map's `anno`, a line or `{line, column}`) and the line of each clause when a def's clauses were made at different lines (`DebugInfo.Clauses`: a DSL that adds a clause per call, such as `route :get, "/a"`, gets a location per clause). -Lines are checked against the current text, which is the buffer open in the editor if there is one and the file on disk otherwise (`currentSource`). A line at or before the module's line carries no information and is dropped; both lines come from the same compile (the module's line is Dbgi's `anno` when it has one), so the comparison holds even when the file has changed since. A line past the end of the current text is dropped too, because a generator can give a def any line (`quote line: 99`) and an edit can remove lines. When nothing is left, which is the case for a def a `@before_compile` hook made at the module line (and for released Ash's code interfaces), the owning module's body is searched for the declaring call: a call, outside heredocs, whose first argument is the function's name as an atom, also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count, and neither does a call in a sibling module of the same file (found with the tokenizer). It is used only when exactly one line matches. +Lines are checked against the current text, which is the buffer open in the editor if there is one and the file on disk otherwise (`currentSource`). A line at or before the module's line carries no information and is dropped; both lines come from the same compile (the module's line is Dbgi's `anno` of the BEAM that holds the module, which is the parent's for a module a macro nested in it, and the index's only when no BEAM records one), so the comparison holds even when the file has changed since. A line past the end of the current text is dropped too, because a generator can give a def any line (`quote line: 99`) and an edit can remove lines. When nothing is left, which is the case for a def a `@before_compile` hook made at the module line (and for released Ash's code interfaces), the owning module's body is searched for the declaring call: a call, outside heredocs, whose first argument is the function's name as an atom, also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count, and neither does a call in a sibling module of the same file (found with the tokenizer). It is used only when exactly one line matches, and not for a function whose clauses were made at several lines: those came from calls that do not spell its name (`get "/a"` makes a `match/2` clause), so a call that does, such as `plug :match`, is not where they were declared. A stale BEAM's lines are not corrected. Only edits to the declaring file move them, by the lines added or removed above, and those files (DSL modules, schemas, routers, dependencies) change less often than their callers. Correcting them meant guessing at text the BEAM was compiled from, and every signal tried either gave up often or could send the editor to a line that looked meaningful and was not. The line from the last compile is near, and the next compile makes it exact. diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index 7175d07..cbb3529 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -78,6 +78,7 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function return results, false } if result, found, ambiguous := sources.bestDescribedResult(results); found { + result.Line = s.compiledModuleLine(module, owner, result.Line, sources) if lines := recordedLinesIn(module, owner, result, s.currentSourceOf(result.FilePath), sources, functions); len(lines) > 0 { return lines, true } @@ -95,6 +96,25 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function return results, false } +// compiledModuleLine returns owner's line as the BEAM that holds it recorded +// it, so that it compares with the recorded function lines, which are from a +// compile too: the index has the line in the current text, which an edit since +// the compile has moved. The BEAM is module's own when owner is module, and +// the parent's when module is a generated module nested in it. The index's +// line is the fallback when neither BEAM records one. +func (s *Server) compiledModuleLine(module, owner string, indexed int, sources generatedDefinitionSources) int { + if owner == module { + if line := sources.debugInfo.ModuleLine; line > 0 { + return line + } + return indexed + } + if ownerSources, ok := s.generatedDefinitionSourcesFor(owner, ""); ok && ownerSources.debugInfo.ModuleLine > 0 { + return ownerSources.debugInfo.ModuleLine + } + return indexed +} + // generatedModuleLocation is where a module that exists only as a BEAM is // defined: the file it was compiled from, at the line the compiler recorded // for the module, such as a Module.create call. It is empty when the BEAM does @@ -120,9 +140,9 @@ func (s *Server) generatedModuleLocation(module string) []store.LookupResult { // recordedLinesIn returns a result for each distinct line in result's file // that the compiled module records for functions, after the module's own line. // -// Both lines come from the same compile: the module's line is the one the BEAM -// records when it has one, and the index's otherwise. So the comparison holds -// even when the file has changed since. A line past the end of the current +// result.Line is the module's line from the same compile as the function lines +// (see compiledModuleLine), so the comparison holds even when the file has +// changed since. A line past the end of the current // text is dropped, because a generator can give a def any line // (`quote line: 99`) and an edit can remove lines: an editor is never sent to // a line the file does not have. @@ -135,14 +155,15 @@ func recordedLinesIn(module, owner string, result store.LookupResult, src *curre if src == nil { return nil } - moduleLine := result.Line - if owner == module && sources.debugInfo.ModuleLine > 0 { - moduleLine = sources.debugInfo.ModuleLine - } - lines := slices.DeleteFunc(generatedFunctionLines(module, result.FilePath, sources, functions), func(r store.LookupResult) bool { - return r.Line <= moduleLine || r.Line > len(src.lines) + recorded := generatedFunctionLines(module, result.FilePath, sources, functions) + // Clauses made at several lines come from calls that do not spell the + // function's name (`get "/a"` makes a `match/2` clause), so a call that + // does, such as `plug :match`, is not where they were declared. + severalClauses := len(recorded) > len(functions) + lines := slices.DeleteFunc(recorded, func(r store.LookupResult) bool { + return r.Line <= result.Line || r.Line > len(src.lines) }) - if len(lines) > 0 || owner == "" { + if len(lines) > 0 || owner == "" || severalClauses { return lines } if line, ok := src.uniqueDeclaration(owner, functions); ok { diff --git a/internal/lsp/generated_definition_integration_test.go b/internal/lsp/generated_definition_integration_test.go index 52ba234..27fdbf2 100644 --- a/internal/lsp/generated_definition_integration_test.go +++ b/internal/lsp/generated_definition_integration_test.go @@ -48,6 +48,14 @@ const compiledDslSource = `defmodule Weird.Dsl do defmacro plug(_name), do: nil + defmacro helper(name) do + quote bind_quoted: [name: name] do + defmodule Module.concat(__MODULE__, Macro.camelize(Atom.to_string(name))) do + def run, do: :helper + end + end + end + defmacro keep_route(name, verb) do quote bind_quoted: [name: name, verb: verb], location: :keep do def unquote(name)(unquote(verb)), do: unquote(verb) @@ -123,6 +131,8 @@ const compiledUserSource = `defmodule Weird.User do stamped_field :stamped + helper :audit + for name <- [:loop_a, :loop_b] do def unquote(name)(), do: unquote(name) end @@ -151,6 +161,7 @@ const compiledCallerSource = `defmodule Weird.Caller do Weird.User.kept() Weird.User.stamped() Weird.User.loop_b() + Weird.User.Audit.run() Weird.User.Nested.nested?(%{}) active?(%{}) f = &Weird.User.two_level/0 @@ -278,6 +289,8 @@ func TestDefinition_GeneratedFunctionsFromCompiler(t *testing.T) { {"@file stamp", "stamped(", 1, []string{user("stamped_field :stamped")}}, {"comprehension", "loop_b(", 1, []string{user("def unquote(name)()")}}, {"nested module", "nested?(", 1, []string{user("flag :nested")}}, + {"function of a module a macro nested", "run()", 2, []string{user("helper :audit")}}, + {"name of a module a macro nested", "Audit.run", 1, []string{user("helper :audit")}}, {"bare call through import", "active?(%{})", 2, []string{user("flag :active")}}, {"capture", "two_level/0", 1, []string{user("outer :two_level")}}, } { @@ -314,6 +327,17 @@ func TestDefinition_GeneratedFunctionsFromCompilerWithStaleSource(t *testing.T) expectDefinition(t, f.definitionOf("match(", 1), compiled("route :get"), compiled("route :post")) }) + // The index then has the module below the recorded lines. The module + // line they are compared with comes from the same compile as they do, + // also for a module a macro nested in it. + t.Run("many lines added above, saved", func(t *testing.T) { + edited := strings.Repeat("# header\n", 40) + compiledUserSource + f.saveLater(compiledUserRel, edited) + t.Cleanup(func() { indexFile(t, f.server.store, f.server.projectRoot, compiledUserRel, compiledUserSource) }) + expectDefinition(t, f.definitionOf("two_level(", 1), compiled("outer :two_level")) + expectDefinition(t, f.definitionOf("run()", 2), compiled("helper :audit")) + }) + t.Run("lines added in an unsaved buffer", func(t *testing.T) { edited := strings.Replace(compiledUserSource, " outer :two_level\n", " # one\n # two\n outer :two_level\n", 1) userURI := string(uri.File(filepath.Join(f.server.projectRoot, compiledUserRel))) @@ -326,10 +350,13 @@ func TestDefinition_GeneratedFunctionsFromCompilerWithStaleSource(t *testing.T) }) t.Run("file shorter than the recorded line", func(t *testing.T) { - edited := "defmodule Weird.User do\n use Weird.Dsl\n outer :two_level\nend\n" + edited := "defmodule Weird.User do\n use Weird.Dsl\n plug :match\n outer :two_level\nend\n" f.saveLater(compiledUserRel, edited) t.Cleanup(func() { indexFile(t, f.server.store, f.server.projectRoot, compiledUserRel, compiledUserSource) }) expectDefinition(t, f.definitionOf("gen_1500(", 1), compiledUserRel+":1") + // The route clauses are gone; `plug :match` spells the name but is not + // where a clause was declared. + expectDefinition(t, f.definitionOf("match(", 1), compiledUserRel+":1") expectDefinition(t, f.definitionOf("two_level(", 1), compiled("outer :two_level")) }) } From d1fa86fc56bac893cddf3eb4023e87039139d55b Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Thu, 1 Oct 2026 01:16:37 -0400 Subject: [PATCH 12/16] Keep a declaring call that opens a heredoc blankHeredocs blanked the whole line that opens a heredoc, so a call such as `define :foo, description: """` was not found by the declaring-call search. The opening line now keeps its text before the delimiter. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/lsp/generated_definition.go | 11 +++++++++-- internal/lsp/generated_definition_test.go | 20 ++++++++++++++++++++ 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index cbb3529..40a26f8 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -275,12 +275,15 @@ func declarationNames(function string) []string { // blankHeredocs replaces the lines inside heredocs with empty lines, so that // an example in a @moduledoc or @doc, such as `define :foo`, is not taken for -// a declaration. Line numbers do not change. +// a declaration. The line that opens a heredoc keeps its text before the +// delimiter, as `define :foo, description: """` still declares foo. Line +// numbers do not change. func blankHeredocs(lines []string) []string { out := make([]string, len(lines)) var open string for i, line := range lines { inside := open != "" + opened := -1 for _, delimiter := range []string{`"""`, "'''"} { if open != "" && open != delimiter { continue @@ -288,13 +291,17 @@ func blankHeredocs(lines []string) []string { if strings.Count(line, delimiter)%2 == 1 { if open == "" { open = delimiter + opened = strings.LastIndex(line, delimiter) } else { open = "" } } } - if !inside && open == "" { + switch { + case !inside && open == "": out[i] = line + case !inside && opened >= 0: + out[i] = line[:opened] } } return out diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index 878355f..ad8b2b7 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -622,6 +622,26 @@ end } } +// A declaring call whose argument opens a heredoc still declares its name; +// only the heredoc's body is blanked. +func TestUniqueDeclarationKeepsCallThatOpensHeredoc(t *testing.T) { + text := `defmodule MyApp.Chat do + resources do + define :get_room_by_slug, description: """ + define :list_rooms + """ + end +end +` + src := ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} + if got, ok := src.uniqueDeclaration("MyApp.Chat", []beam.Function{{Name: "get_room_by_slug!", Arity: 1}}); !ok || got != 3 { + t.Errorf("uniqueDeclaration(get_room_by_slug!) = %d, %v; want 3, the call that opens the heredoc", got, ok) + } + if got, ok := src.uniqueDeclaration("MyApp.Chat", []beam.Function{{Name: "list_rooms", Arity: 0}}); ok { + t.Errorf("uniqueDeclaration(list_rooms) = %d; want none, the only match is inside the heredoc", got) + } +} + // A file that a request read from disk, without opening it, is only a cache. // If the file then changes and is compiled again, the cache is old text, not // unsaved edits, and a current BEAM's line must not be corrected against it. From 191490f6fddafdcd1f80ff6db12a95146a51bec7 Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Sat, 3 Oct 2026 16:01:28 -0400 Subject: [PATCH 13/16] Place more generated definitions on their declaring line A macro that expands `defmodule unquote(name)` with a name it computed leaves the module line at zero in the debug info, so go-to-definition on that module's name found nothing. Its functions still record the line of the call that made them, so the first of those is used instead. When several calls in a module spell a generated function's name, the search for the declaring call gave up. That is the usual shape of an Ash resource, where an action and the code interface that runs it share a name (`update :publish` and `define :publish`). The macro whose calls name the most of the module's generated functions now wins, because it is the one that generates them. A tie still keeps the module line. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- docs/architecture.md | 4 +- internal/lsp/generated_definition.go | 153 +++++++++++++----- .../generated_definition_integration_test.go | 23 +++ internal/lsp/generated_definition_test.go | 73 ++++++++- 5 files changed, 205 insertions(+), 50 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bde6b37..d0e7bc1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line; only edits to the declaring file move it, and the next compile makes it exact. A function whose only recorded line is the module line, such as one a `@before_compile` hook made, goes to the one call in its module that declares it by name, if there is exactly one. A function with a clause per DSL call goes to every clause. A line past the end of the file is never returned. A module compiled without debug info falls back to its Docs chunk annotation, which is often, but not always, the same line. A generated module with no source of its own, such as one `Module.create` made or a Spark DSL entity, goes to the file it was compiled from, rebased onto the project when it was built elsewhere, and so does go-to-definition on its name. A bare call to a generated function of an imported module now resolves. Ash code interfaces go to their `define` line with released Ash, and from the recorded line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) +- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line; only edits to the declaring file move it, and the next compile makes it exact. A function whose only recorded line is the module line, such as one a `@before_compile` hook made, goes to the call in its module that declares it by name. When several calls spell the name, as an Ash action and the code interface that runs it do, the macro whose calls name the most of the module's generated functions wins, and a tie keeps the module line. A function with a clause per DSL call goes to every clause. A line past the end of the file is never returned. A module compiled without debug info falls back to its Docs chunk annotation, which is often, but not always, the same line. A generated module with no source of its own, such as one `Module.create` made or a Spark DSL entity, goes to the file it was compiled from, rebased onto the project when it was built elsewhere, and so does go-to-definition on its name. A module a macro made with `defmodule` and a name it computed records no line of its own, so its name goes to its first function's line. A bare call to a generated function of an imported module now resolves. Ash code interfaces go to their `define` line with released Ash, and from the recorded line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) - **`--root`/`-C` names the workspace on every command** — `dexter init --root ~/project`, `dexter lookup --root ~/project MyApp.Repo`, and `dexter lsp --root ~/project` all run as if they had been started in that directory, so an agent, a script, or an editor wrapper can index or query a project from anywhere. Relative paths, a `reindex` target included, resolve from the named root diff --git a/docs/architecture.md b/docs/architecture.md index 89da62e..9c861c9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -147,11 +147,11 @@ Each def carries two locations. Its `:line` is the line in the module's own file `beam.ReadDefinitionLines` also reads the module's own line (the map's `anno`, a line or `{line, column}`) and the line of each clause when a def's clauses were made at different lines (`DebugInfo.Clauses`: a DSL that adds a clause per call, such as `route :get, "/a"`, gets a location per clause). -Lines are checked against the current text, which is the buffer open in the editor if there is one and the file on disk otherwise (`currentSource`). A line at or before the module's line carries no information and is dropped; both lines come from the same compile (the module's line is Dbgi's `anno` of the BEAM that holds the module, which is the parent's for a module a macro nested in it, and the index's only when no BEAM records one), so the comparison holds even when the file has changed since. A line past the end of the current text is dropped too, because a generator can give a def any line (`quote line: 99`) and an edit can remove lines. When nothing is left, which is the case for a def a `@before_compile` hook made at the module line (and for released Ash's code interfaces), the owning module's body is searched for the declaring call: a call, outside heredocs, whose first argument is the function's name as an atom, also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count, and neither does a call in a sibling module of the same file (found with the tokenizer). It is used only when exactly one line matches, and not for a function whose clauses were made at several lines: those came from calls that do not spell its name (`get "/a"` makes a `match/2` clause), so a call that does, such as `plug :match`, is not where they were declared. +Lines are checked against the current text, which is the buffer open in the editor if there is one and the file on disk otherwise (`currentSource`). A line at or before the module's line carries no information and is dropped; both lines come from the same compile (the module's line is Dbgi's `anno` of the BEAM that holds the module, which is the parent's for a module a macro nested in it, and the index's only when no BEAM records one), so the comparison holds even when the file has changed since. A line past the end of the current text is dropped too, because a generator can give a def any line (`quote line: 99`) and an edit can remove lines. When nothing is left, which is the case for a def a `@before_compile` hook made at the module line (and for released Ash's code interfaces), the owning module's body is searched for the declaring call: a call, outside heredocs, whose first argument is the function's name as an atom, also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count, and neither does a call in a sibling module of the same file (found with the tokenizer). When one line matches, it is used. When several do, as an Ash action (`update :publish`) and the code interface that runs it (`define :publish`) do, the macro that made the function is taken to be the one whose calls name the most of the module's generated functions (`define` names every interface; an action names only itself), and a tie gives no answer. The search is not used for a function whose clauses were made at several lines: those came from calls that do not spell its name (`get "/a"` makes a `match/2` clause), so a call that does, such as `plug :match`, is not where they were declared. A stale BEAM's lines are not corrected. Only edits to the declaring file move them, by the lines added or removed above, and those files (DSL modules, schemas, routers, dependencies) change less often than their callers. Correcting them meant guessing at text the BEAM was compiled from, and every signal tried either gave up often or could send the editor to a line that looked meaningful and was not. The line from the last compile is near, and the next compile makes it exact. -A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually the generator's own file. A recorded path inside the project is used as it is. Otherwise it is rebased: first each tail of the recorded path under the project root, longest first, which covers a moved or copied checkout and an umbrella's `apps//lib`, then the Mix layouts for the application named below the last `lib/` (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. A recorded path outside the project is the last choice: it is right for a `path:` dependency, but a `_build` copied from another worktree records that worktree, which usually still exists. If no file is found, the lexical parent stays the answer. `CInf` records the path as codepoints, which are decoded to UTF-8. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. `LookupName` on the name of a module that has no source row returns the file from `CInf` at the module's recorded line, so go-to-definition on a module that `Module.create` made works too. A bare call that no source resolves also checks the BEAMs of the file's imported modules for a generated function of that name. +A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually the generator's own file. A recorded path inside the project is used as it is. Otherwise it is rebased: first each tail of the recorded path under the project root, longest first, which covers a moved or copied checkout and an umbrella's `apps//lib`, then the Mix layouts for the application named below the last `lib/` (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. A recorded path outside the project is the last choice: it is right for a `path:` dependency, but a `_build` copied from another worktree records that worktree, which usually still exists. If no file is found, the lexical parent stays the answer. `CInf` records the path as codepoints, which are decoded to UTF-8. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. `LookupName` on the name of a module that has no source row returns the file from `CInf` at the module's recorded line, or at its first function's line when the module line is zero, as it is for a module a macro made with `defmodule unquote(name)` and a name it computed, so go-to-definition on a module that `Module.create` made works too. A bare call that no source resolves also checks the BEAMs of the file's imported modules for a generated function of that name. The debug info and compile info are read on the first definition request that needs them and memoized on the module's generated-function cache entry, so the BEAM stamp invalidates them with everything else. A module compiled without debug info, or an Erlang module, is memoized as empty. diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go index 40a26f8..d8c9cbf 100644 --- a/internal/lsp/generated_definition.go +++ b/internal/lsp/generated_definition.go @@ -79,7 +79,8 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function } if result, found, ambiguous := sources.bestDescribedResult(results); found { result.Line = s.compiledModuleLine(module, owner, result.Line, sources) - if lines := recordedLinesIn(module, owner, result, s.currentSourceOf(result.FilePath), sources, functions); len(lines) > 0 { + generated := s.generatedFunctionsFor(module, beamPath) + if lines := recordedLinesIn(module, owner, result, s.currentSourceOf(result.FilePath), sources, functions, generated); len(lines) > 0 { return lines, true } return results, false @@ -88,7 +89,7 @@ func (s *Server) generatedDefinitionResultsFor(module, beamPath string, function } if owner != module { if source := s.compiledSourceFile(sources.compileSource); source != "" { - if lines := recordedLinesIn(module, "", store.LookupResult{FilePath: source, Line: 1}, s.currentSourceOf(source), sources, functions); len(lines) > 0 { + if lines := recordedLinesIn(module, "", store.LookupResult{FilePath: source, Line: 1}, s.currentSourceOf(source), sources, functions, nil); len(lines) > 0 { return lines, true } } @@ -118,23 +119,44 @@ func (s *Server) compiledModuleLine(module, owner string, indexed int, sources g // generatedModuleLocation is where a module that exists only as a BEAM is // defined: the file it was compiled from, at the line the compiler recorded // for the module, such as a Module.create call. It is empty when the BEAM does -// not record both, or the line is not in the file. +// not record a line, or the line is not in the file. func (s *Server) generatedModuleLocation(module string) []store.LookupResult { if len(s.generatedFunctionsForModule(module)) == 0 { return nil } sources, ok := s.generatedDefinitionSourcesFor(module, "") - if !ok || sources.debugInfo.ModuleLine < 1 { + if !ok { + return nil + } + line := recordedModuleLine(sources.debugInfo) + if line < 1 { return nil } source := s.compiledSourceFile(sources.compileSource) if source == "" { return nil } - if src := s.currentSourceOf(source); src == nil || sources.debugInfo.ModuleLine > len(src.lines) { + if src := s.currentSourceOf(source); src == nil || line > len(src.lines) { return nil } - return []store.LookupResult{{Module: module, FilePath: source, Line: sources.debugInfo.ModuleLine, Kind: "module"}} + return []store.LookupResult{{Module: module, FilePath: source, Line: line, Kind: "module"}} +} + +// recordedModuleLine is the module's recorded line or, when the compiler +// recorded none, its first function's. A macro that expands +// `defmodule unquote(name)` with a name it computed leaves the module line at +// zero, but each def in the body still has the line of the call that made it. +func recordedModuleLine(info beam.DebugInfo) int { + if info.ModuleLine > 0 { + return info.ModuleLine + } + first := 0 + for _, line := range info.Lines { + if line > 0 && (first == 0 || line < first) { + first = line + } + } + return first } // recordedLinesIn returns a result for each distinct line in result's file @@ -149,9 +171,10 @@ func (s *Server) generatedModuleLocation(module string) []store.LookupResult { // // When nothing is left, which is the case for a def a @before_compile hook // made at the module line, the module's body is searched for the call that -// declared the function: the one call whose first argument is its name as an -// atom (`later :deferred`). Only a single match counts. -func recordedLinesIn(module, owner string, result store.LookupResult, src *currentSource, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { +// declared the function: a call whose first argument is its name as an atom +// (`later :deferred`). See declaration for how one of several is chosen; +// generated is every function the module's BEAM adds to its source. +func recordedLinesIn(module, owner string, result store.LookupResult, src *currentSource, sources generatedDefinitionSources, functions, generated []beam.Function) []store.LookupResult { if src == nil { return nil } @@ -166,7 +189,7 @@ func recordedLinesIn(module, owner string, result store.LookupResult, src *curre if len(lines) > 0 || owner == "" || severalClauses { return lines } - if line, ok := src.uniqueDeclaration(owner, functions); ok { + if line, ok := src.declaration(owner, functions, generated); ok { return []store.LookupResult{{Module: module, FilePath: result.FilePath, Line: line, Kind: functions[0].Kind, Arity: functions[0].Arity}} } return nil @@ -244,22 +267,63 @@ func (src *currentSource) ownerAt(line int) string { return src.owners[line] } -// uniqueDeclaration returns the one line in owner's body that declares one of -// functions by name, if exactly one does. A sibling module in the same file -// that declares the same name does not count. -func (src *currentSource) uniqueDeclaration(owner string, functions []beam.Function) (int, bool) { +// declaration returns the line in owner's body of the call that declared one +// of functions by name. A sibling module in the same file that declares the +// same name does not count. +// +// When one call spells the name, it is the answer. When several do, as +// `update :publish` and `define :publish` can, the macro that generated the +// function is taken to be the one whose calls name the most of the module's +// generated functions: `define` names every function it made, while an action +// names only its own. A tie gives no answer. +func (src *currentSource) declaration(owner string, functions, generated []beam.Function) (int, bool) { names := declarationNames(functions[0].Name) - found := 0 + type match struct { + line int + callee string + } + var matches []match for line := 1; line <= len(src.lines); line++ { - if !lineDeclaresAtom(src.lines[line-1], names) || src.ownerAt(line) != owner { + callee, atom, ok := declaringCall(src.lines[line-1]) + if !ok || !slices.Contains(names, atom) || src.ownerAt(line) != owner { continue } - if found != 0 { - return 0, false + matches = append(matches, match{line: line, callee: callee}) + } + switch len(matches) { + case 0: + return 0, false + case 1: + return matches[0].line, true + } + + generatedNames := make(map[string]bool) + for _, function := range generated { + for _, name := range declarationNames(function.Name) { + generatedNames[name] = true } - found = line } - return found, found != 0 + named := make(map[string]map[string]bool) + for _, m := range matches { + named[m.callee] = make(map[string]bool) + } + for line := 1; line <= len(src.lines); line++ { + callee, atom, ok := declaringCall(src.lines[line-1]) + if !ok || named[callee] == nil || !generatedNames[atom] || src.ownerAt(line) != owner { + continue + } + named[callee][atom] = true + } + best, bestScore, tied := 0, -1, false + for _, m := range matches { + switch score := len(named[m.callee]); { + case score > bestScore: + best, bestScore, tied = m.line, score, false + case score == bestScore: + tied = true + } + } + return best, !tied } // declarationNames is the names a declaring call can spell for function: the @@ -308,36 +372,47 @@ func blankHeredocs(lines []string) []string { } // lineDeclaresAtom reports whether text is a call whose first argument is one -// of names as an atom: `define :list_rooms`, `field :email, :string`, or -// `Lib.define(:name)`. That is the shape of a macro call that declares a -// name. An atom anywhere else, such as a keyword value or a typespec, is not. +// of names as an atom (see declaringCall). func lineDeclaresAtom(text string, names []string) bool { + _, atom, ok := declaringCall(text) + return ok && slices.Contains(names, atom) +} + +// declaringCall returns the callee and the first argument of text when text +// is a call whose first argument is an atom: `define :list_rooms`, +// `field :email, :string`, or `Lib.define(:name)`. That is the shape of a +// macro call that declares a name. An atom anywhere else, such as a keyword +// value or a typespec, is not. +func declaringCall(text string) (callee, atom string, ok bool) { rest := strings.TrimLeft(text, " \t") - callee := 0 - for callee < len(rest) && (isIdentifierByte(rest[callee]) || rest[callee] == '.') { - callee++ + end := 0 + for end < len(rest) && (isIdentifierByte(rest[end]) || rest[end] == '.') { + end++ } // A module attribute such as `@tag :slow` sets a value; it declares nothing. - if callee == 0 || rest[0] == ':' || rest[0] == '@' || (rest[0] >= '0' && rest[0] <= '9') { - return false + if end == 0 || rest[0] == ':' || rest[0] == '@' || (rest[0] >= '0' && rest[0] <= '9') { + return "", "", false } - args := strings.TrimLeft(rest[callee:], " \t") - if len(args) == len(rest[callee:]) { + args := strings.TrimLeft(rest[end:], " \t") + if len(args) == len(rest[end:]) { // No space after the callee: only a parenthesized call qualifies. if !strings.HasPrefix(args, "(") { - return false + return "", "", false } } args = strings.TrimLeft(strings.TrimPrefix(args, "("), " \t") - for _, name := range names { - if strings.HasPrefix(args, ":"+name) { - end := 1 + len(name) - if end == len(args) || !isIdentifierByte(args[end]) { - return true - } - } + if !strings.HasPrefix(args, ":") { + return "", "", false + } + // Bytes from 0x80 up are parts of a UTF-8 identifier such as `:café`. + atomEnd := 1 + for atomEnd < len(args) && (isIdentifierByte(args[atomEnd]) || args[atomEnd] >= 0x80) { + atomEnd++ + } + if atomEnd == 1 { + return "", "", false } - return false + return rest[:end], args[1:atomEnd], true } func isIdentifierByte(b byte) bool { diff --git a/internal/lsp/generated_definition_integration_test.go b/internal/lsp/generated_definition_integration_test.go index 27fdbf2..93b0212 100644 --- a/internal/lsp/generated_definition_integration_test.go +++ b/internal/lsp/generated_definition_integration_test.go @@ -47,6 +47,7 @@ const compiledDslSource = `defmodule Weird.Dsl do end defmacro plug(_name), do: nil + defmacro action(_name), do: nil defmacro helper(name) do quote bind_quoted: [name: name] do @@ -56,6 +57,18 @@ const compiledDslSource = `defmodule Weird.Dsl do end end + # The module name is computed here, so the compiler records no line for + # the module itself. + defmacro named_helper(name) do + module = Module.concat(__CALLER__.module, Macro.camelize(Atom.to_string(name))) + + quote do + defmodule unquote(module) do + def run, do: :named + end + end + end + defmacro keep_route(name, verb) do quote bind_quoted: [name: name, verb: verb], location: :keep do def unquote(name)(unquote(verb)), do: unquote(verb) @@ -109,6 +122,9 @@ const compiledUserSource = `defmodule Weird.User do later :deferred + action :submit + later :submit + pinned :pinned_fun plug :match @@ -133,6 +149,8 @@ const compiledUserSource = `defmodule Weird.User do helper :audit + named_helper :report + for name <- [:loop_a, :loop_b] do def unquote(name)(), do: unquote(name) end @@ -151,6 +169,7 @@ const compiledCallerSource = `defmodule Weird.Caller do Weird.User.two_level() Weird.User.active?(%{}) Weird.User.deferred() + Weird.User.submit() Weird.User.pinned_fun() Weird.User.match(:get, "/a") Weird.User.handle(:get) @@ -162,6 +181,7 @@ const compiledCallerSource = `defmodule Weird.Caller do Weird.User.stamped() Weird.User.loop_b() Weird.User.Audit.run() + Weird.User.Report.run() Weird.User.Nested.nested?(%{}) active?(%{}) f = &Weird.User.two_level/0 @@ -278,6 +298,7 @@ func TestDefinition_GeneratedFunctionsFromCompiler(t *testing.T) { {"macro that calls a macro", "two_level(", 1, []string{user("outer :two_level")}}, {"predicate from an atom", "active?(%{})", 1, []string{user("flag :active")}}, {"@before_compile hook", "deferred(", 1, []string{user("later :deferred")}}, + {"declaring call next to a same-named call", "submit(", 1, []string{user("later :submit")}}, {"line past the end of the file", "pinned_fun(", 1, []string{user("pinned :pinned_fun")}}, {"one clause per call", "match(", 1, []string{user(`route :get`), user(`route :post`)}}, {"one location: :keep clause per call", "handle(", 1, []string{user("keep_route :handle, :get"), user("keep_route :handle, :post")}}, @@ -291,6 +312,8 @@ func TestDefinition_GeneratedFunctionsFromCompiler(t *testing.T) { {"nested module", "nested?(", 1, []string{user("flag :nested")}}, {"function of a module a macro nested", "run()", 2, []string{user("helper :audit")}}, {"name of a module a macro nested", "Audit.run", 1, []string{user("helper :audit")}}, + {"function of a module a macro named", "run()", 3, []string{user("named_helper :report")}}, + {"name of a module a macro named", "Report.run", 1, []string{user("named_helper :report")}}, {"bare call through import", "active?(%{})", 2, []string{user("flag :active")}}, {"capture", "two_level/0", 1, []string{user("outer :two_level")}}, } { diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go index ad8b2b7..1ac136b 100644 --- a/internal/lsp/generated_definition_test.go +++ b/internal/lsp/generated_definition_test.go @@ -598,7 +598,7 @@ func TestSourceRebaseCandidates(t *testing.T) { // An example in a @moduledoc is not a declaration, and neither is the same // call in a sibling module of the same file. -func TestUniqueDeclarationSkipsHeredocsAndSiblings(t *testing.T) { +func TestDeclarationSkipsHeredocsAndSiblings(t *testing.T) { text := `defmodule MyApp.Chat do @moduledoc """ Example: @@ -617,14 +617,14 @@ end ` src := ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} functions := []beam.Function{{Name: "get_room_by_slug!", Arity: 1}} - if got, ok := src.uniqueDeclaration("MyApp.Chat", functions); !ok || got != 9 { - t.Errorf("uniqueDeclaration = %d, %v; want 9, the one call in MyApp.Chat's body", got, ok) + if got, ok := src.declaration("MyApp.Chat", functions, nil); !ok || got != 9 { + t.Errorf("declaration = %d, %v; want 9, the one call in MyApp.Chat's body", got, ok) } } // A declaring call whose argument opens a heredoc still declares its name; // only the heredoc's body is blanked. -func TestUniqueDeclarationKeepsCallThatOpensHeredoc(t *testing.T) { +func TestDeclarationKeepsCallThatOpensHeredoc(t *testing.T) { text := `defmodule MyApp.Chat do resources do define :get_room_by_slug, description: """ @@ -634,11 +634,68 @@ func TestUniqueDeclarationKeepsCallThatOpensHeredoc(t *testing.T) { end ` src := ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} - if got, ok := src.uniqueDeclaration("MyApp.Chat", []beam.Function{{Name: "get_room_by_slug!", Arity: 1}}); !ok || got != 3 { - t.Errorf("uniqueDeclaration(get_room_by_slug!) = %d, %v; want 3, the call that opens the heredoc", got, ok) + if got, ok := src.declaration("MyApp.Chat", []beam.Function{{Name: "get_room_by_slug!", Arity: 1}}, nil); !ok || got != 3 { + t.Errorf("declaration(get_room_by_slug!) = %d, %v; want 3, the call that opens the heredoc", got, ok) } - if got, ok := src.uniqueDeclaration("MyApp.Chat", []beam.Function{{Name: "list_rooms", Arity: 0}}); ok { - t.Errorf("uniqueDeclaration(list_rooms) = %d; want none, the only match is inside the heredoc", got) + if got, ok := src.declaration("MyApp.Chat", []beam.Function{{Name: "list_rooms", Arity: 0}}, nil); ok { + t.Errorf("declaration(list_rooms) = %d; want none, the only match is inside the heredoc", got) + } +} + +// When several calls spell a function's name, as an action and the code +// interface that runs it do, the macro whose calls name the most of the +// module's generated functions declared it. A tie gives no answer. +func TestDeclarationPrefersMacroThatNamesGeneratedFunctions(t *testing.T) { + text := `defmodule MyApp.Article do + actions do + create :create do + accept [:title] + end + + update :publish do + end + + read :published + end + + code_interface do + define :create, args: [:title] + define :publish + define :published + end + + later :twice + later :twice + + first :tied + second :tied +end +` + src := ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} + var generated []beam.Function + for _, name := range []string{"create", "create!", "publish", "publish!", "published", "published!", "twice", "tied"} { + generated = append(generated, beam.Function{Name: name, Arity: 1}) + } + for _, tc := range []struct { + name string + want int + }{ + {"create", 14}, + {"publish!", 15}, + {"published", 16}, + } { + if got, ok := src.declaration("MyApp.Article", []beam.Function{{Name: tc.name, Arity: 1}}, generated); !ok || got != tc.want { + t.Errorf("declaration(%s) = %d, %v; want %d, the define call", tc.name, got, ok, tc.want) + } + } + for _, name := range []string{"twice", "tied"} { + if got, ok := src.declaration("MyApp.Article", []beam.Function{{Name: name, Arity: 0}}, generated); ok { + t.Errorf("declaration(%s) = %d; want none, two calls are equally likely", name, got) + } + } + // Without the module's generated functions, there is nothing to choose by. + if got, ok := src.declaration("MyApp.Article", []beam.Function{{Name: "publish", Arity: 1}}, nil); ok { + t.Errorf("declaration(publish) without generated functions = %d; want none", got) } } From 904d60ead34c2d976d06475018c60cc060b87415 Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Sat, 3 Oct 2026 16:22:40 -0400 Subject: [PATCH 14/16] Stop indexing defs in macro quotes, and read more compiled modules A def inside a quote in a macro other than __using__ is code the macro generates in its caller, not a function of the macro's module. The parser indexed it as one, so `defmacro route ... quote do def handle` made the index claim that the DSL defines handle/2, and a consumer that imports the DSL resolved `Consumer.handle` into the macro's body before the compiled BEAM was asked. Such defs are now skipped. What __using__ injects and a quote in a helper function are still indexed. This changes the index, so IndexVersion and the daemon contract are bumped. Elixir 1.17 and earlier record a module's line under the debug info map's `line` key, not `anno`, so it was never read and recorded function lines were compared with the index's module line instead. Both keys are now read. A `location: :keep` quote in a macro defined above its caller in the same file records that file and the line inside the macro, which was taken for an @file stamp. A recorded file line at or before the module's own line is now ignored, and the call's line is used. A BEAM compiled with the `compressed` option is a gzip stream around the container, which was rejected as an invalid BEAM. It is now decompressed with the same size limit. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 4 + docs/architecture.md | 4 +- internal/beam/debug_info.go | 28 ++++- internal/beam/debug_info_test.go | 101 +++++++++++++++++- internal/beam/docs.go | 44 +++++++- internal/beam/docs_test.go | 54 ++++++++++ internal/daemon/endpoint.go | 2 +- .../generated_definition_integration_test.go | 11 ++ internal/parser/parser_test.go | 77 +++++++++++++ internal/parser/parser_tokenized.go | 101 ++++++++++++++++++ internal/version/version.go | 2 +- 11 files changed, 416 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d0e7bc1..b69b5a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,10 @@ ### Fixed +- **A def inside a macro's `quote` is no longer indexed as a function of the macro's module** — `defmacro route(...) do quote do def handle(...) end end` made the index say that the DSL module defines `handle/2`, which it does not. A consumer that imports the DSL then resolved `Consumer.handle` into the macro's body. Such a def is now skipped, so the call goes to the line in the consumer that declared it, from the compiled BEAM. What `__using__` injects, and a quote in a helper function, are still indexed as before. The index is rebuilt once after the upgrade + +- **Compressed BEAM files are read** — a module compiled with the `compressed` option, as some Erlang dependencies are, is a gzip stream around the BEAM container. Dexter rejected it as an invalid BEAM, so its exports were missing from completion and generated-function navigation. It is now decompressed, with the same size limit as an uncompressed file + - **Git worktrees nested inside a project were indexed as part of it** — a linked worktree checked out below the project root (Claude Code's `.claude/worktrees/`, or any gitignored worktree folder) is a full copy of the repository, so every go-to-definition returned one result per checkout. The indexer, the incremental sweep, the native watcher and single-file updates now skip any directory that is a linked worktree, including worktrees of bare repositories; submodules and Mix git dependencies stay indexed. Adding, changing, moving or removing such a worktree, including its `mix.exs` and `mix.lock`, no longer reindexes the enclosing project, and an existing index drops the worktree's files at the next start. A worktree that git still records stays skipped after its `.git` file is gone, as during `git worktree remove`, until `git worktree prune`. Starting Dexter inside such a worktree no longer climbs to the enclosing checkout's `.dexter/dexter.db` either: the worktree is its own project - **Branch switches were missed when the project root is a linked worktree or a submodule** — the HEAD poller read `/.git/HEAD`, but in such a checkout `.git` is a file that names the git directory. The poller now follows that file, so a branch switch reconciles the index even when the native file watcher is unavailable - **Find references through an injected alias was slow and memory-hungry on large projects** — a module such as `MyApp.Repo`, aliased by a `__using__` block that most of the project uses, made every references query read and tokenize each file that used the injector: about 20,000 files and 2.4 GB of allocations per query on a large monorepo. Only files that contain a candidate reference are read now, which brought that query from 1–2 s to under 0.3 s and its allocations to about 70 MB diff --git a/docs/architecture.md b/docs/architecture.md index 9c861c9..cb833d0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -141,7 +141,9 @@ Generated-symbol resolution is shared by completion, hover, definition, signatur A generated function has no source definition, but the compiled module's `Dbgi` chunk still records where each def came from. `beam.ReadDefinitionLines` walks `{:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}}` for the `file`, `relative_file`, and `definitions` keys, and steps over clause bodies without allocating. Clause ASTs nest far deeper than a Docs term, so this reader raises the ETF depth guard to `maxDebugInfoDepth`. -Each def carries two locations. Its `:line` is the line in the module's own file that was being compiled when the def was produced: the macro call, or wherever a before-compile hook ran. A `file: {path, line}` entry, left by `@file` or `quote location: :keep`, is preferred when `path` is the module's own source, because that is where the code asked for the function. When `path` is another file it is the generator's implementation, and `:line` is used instead. Ash's code interfaces take the second shape: each generated def is expanded at its `define` line, so `:line` is that line and `file` still names `code_interface.ex`, which keeps stacktraces unchanged. No framework is recognized by name. +Each def carries two locations. Its `:line` is the line in the module's own file that was being compiled when the def was produced: the macro call, or wherever a before-compile hook ran. A `file: {path, line}` entry, left by `@file` or `quote location: :keep`, is preferred when `path` is the module's own source, because that is where the code asked for the function. When `path` is another file it is the generator's implementation, and `:line` is used instead. A `file:` line at or before the module's own line is not used either: it is a `location: :keep` quote in a macro defined above the module in the same file. Elixir 1.17 and earlier record the module's line under the map's `line` key instead of `anno`, and both are read. Ash's code interfaces take the second shape: each generated def is expanded at its `define` line, so `:line` is that line and `file` still names `code_interface.ex`, which keeps stacktraces unchanged. No framework is recognized by name. + +The parser does not index a def inside a `quote` in a macro other than `__using__` (`defmacro route ... do quote do def handle ... end end`) as a function of the macro's module: it is code the macro generates in its caller, and the caller's BEAM records where it was declared. Without that, a consumer that imports the DSL resolved `Consumer.handle` through the import into the macro's body before the BEAM was asked. What `__using__` injects and a quote in a helper function stay indexed, because the use-chain features above resolve them. `generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (the same absolute path, or else the most path components shared from the end, which covers a moved, copied, or symlinked project; when a module has rows in two files, as two umbrella apps can, the most specific match wins, and an equal match trusts neither), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is closer than the module line. diff --git a/internal/beam/debug_info.go b/internal/beam/debug_info.go index 35a5059..e73f012 100644 --- a/internal/beam/debug_info.go +++ b/internal/beam/debug_info.go @@ -30,7 +30,8 @@ type DebugInfo struct { RelativeFile string // ModuleLine is the line of the module itself in File: its defmodule, or - // wherever Module.create was called. Zero if the chunk has none. + // wherever Module.create was called. It is the map's anno, or its line + // before Elixir 1.18. Zero if the chunk has none. ModuleLine int // Lines maps each public function and macro to its line in File. Entries @@ -52,9 +53,10 @@ type DebugInfo struct { // own file that was being compiled when the def was produced: the macro call, // or wherever a before-compile hook ran. A `file: {path, line}` entry, left by // `@file` or by `quote location: :keep`, names a more specific place. When that -// path is the module's own source it is where the code asked for the function; -// when it is some other file it is the generator's own implementation, which is -// not what a reader of the module is looking for, so :line is used instead. +// path is the module's own source and the line is after the module's own, it +// is where the code asked for the function. Otherwise it is the generator's own +// implementation, in another file or in a macro defined above the module, which +// is not what a reader of the module is looking for, so :line is used instead. // // Everything here is standard compiler output. No framework is recognized by // name: a generator that wants its functions to navigate to the line that @@ -136,6 +138,7 @@ func parseDebugInfo(buf []byte) (info DebugInfo, err error) { return DebugInfo{}, err } var sites []definitionSite + var legacyModuleLine int for i := int64(0); i < pairs; i++ { var key string if tag, err := r.peekTag(); err != nil { @@ -164,6 +167,12 @@ func parseDebugInfo(buf []byte) (info DebugInfo, err error) { if info.ModuleLine, err = readAnnoLine(r); err != nil { return DebugInfo{}, err } + case "line": + // Elixir 1.17 and earlier record the module's line here, as a + // bare line, and have no anno. + if legacyModuleLine, err = readAnnoLine(r); err != nil { + return DebugInfo{}, err + } default: if err := r.skip(); err != nil { return DebugInfo{}, err @@ -175,11 +184,20 @@ func parseDebugInfo(buf []byte) (info DebugInfo, err error) { return DebugInfo{}, err } + if info.ModuleLine == 0 { + info.ModuleLine = legacyModuleLine + } + info.Lines = make(map[FunctionKey]int, len(sites)) for _, site := range sites { line := site.line + // A location at or before the module's own line cannot be where the + // module asked for the function. It is a `location: :keep` quote in a + // macro defined above the module in the same file, so :line, the + // call, is used. stamped := site.keepFile != "" && site.keepLine > 0 && - (site.keepFile == info.RelativeFile || site.keepFile == info.File) + (site.keepFile == info.RelativeFile || site.keepFile == info.File) && + (info.ModuleLine == 0 || site.keepLine > info.ModuleLine) if stamped { line = site.keepLine } diff --git a/internal/beam/debug_info_test.go b/internal/beam/debug_info_test.go index e660c22..bb28612 100644 --- a/internal/beam/debug_info_test.go +++ b/internal/beam/debug_info_test.go @@ -25,6 +25,13 @@ type testDefinition struct { // the way the Elixir compiler writes a Dbgi chunk. Each definition carries a // clause body so the reader has to step over real AST shapes to reach the next. func buildDebugInfoTerm(file, relativeFile string, definitions ...testDefinition) []byte { + return buildDebugInfoTermWithModuleLine(file, relativeFile, "", 0, definitions...) +} + +// buildDebugInfoTermWithModuleLine also writes the module's own line under +// moduleKey: "anno" as Elixir 1.18 and later do, or "line" as earlier +// versions do. An empty key writes none. +func buildDebugInfoTermWithModuleLine(file, relativeFile, moduleKey string, moduleLine int, definitions ...testDefinition) []byte { var w etfTestWriter w.smallTuple(3) w.atom("debug_info_v1") @@ -35,7 +42,11 @@ func buildDebugInfoTerm(file, relativeFile string, definitions ...testDefinition // Keys in the order ERTS sorts a small map's atom keys, which puts // definitions before file and relative_file: the reader must not depend on // having seen the file first. - w.mapHeader(4) + if moduleKey != "" { + w.mapHeader(5) + } else { + w.mapHeader(4) + } w.atom("attributes") w.nil() w.atom("definitions") @@ -103,6 +114,10 @@ func buildDebugInfoTerm(file, relativeFile string, definitions ...testDefinition w.binary(file) w.atom("relative_file") w.binary(relativeFile) + if moduleKey != "" { + w.atom(moduleKey) + w.smallInt(moduleLine) + } w.nil() // specs return w.buf @@ -143,6 +158,48 @@ func TestReadDefinitionLinesPrefersOwnFileLocation(t *testing.T) { } } +// Elixir 1.17 and earlier record the module's line under :line, and later +// versions under :anno. +func TestReadDefinitionLinesModuleLineKeys(t *testing.T) { + for _, key := range []string{"anno", "line"} { + t.Run(key, func(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Accounts.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTermWithModuleLine( + "/src/my_app/lib/my_app/accounts.ex", "lib/my_app/accounts.ex", key, 3, + testDefinition{name: "get_user", arity: 1, kind: "def", line: 5}, + )) + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + if info.ModuleLine != 3 { + t.Errorf("ModuleLine = %d, want 3", info.ModuleLine) + } + }) + } +} + +// A location in the module's own file at or before the module's line is a +// `location: :keep` quote in a macro defined above the module, not a stamp +// that names where the module asked for the function, so :line is used. +func TestReadDefinitionLinesOwnFileLocationBeforeModule(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Accounts.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTermWithModuleLine( + "/src/my_app/lib/my_app/accounts.ex", "lib/my_app/accounts.ex", "anno", 9, + testDefinition{name: "kept", arity: 0, kind: "def", line: 12, keepFile: "lib/my_app/accounts.ex", keepLine: 4}, + testDefinition{name: "stamped", arity: 0, kind: "def", line: 9, keepFile: "lib/my_app/accounts.ex", keepLine: 14}, + )) + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + for name, want := range map[string]int{"kept": 12, "stamped": 14} { + if got := info.Lines[FunctionKey{Name: name, Arity: 0}]; got != want { + t.Errorf("%s/0 line = %d, want %d", name, got, want) + } + } +} + // `quote location: :keep` inside a library records the library's own file. // That line belongs to the generator, not to the module, so the module-side // :line is used instead. @@ -387,6 +444,48 @@ end } } +// A macro defined above its caller in the same file, with `location: :keep`, +// records that file and the line inside the macro. The call is the answer. +func TestReadDefinitionLinesSameFileKeepFromElixirCompiler(t *testing.T) { + elixirc, err := exec.LookPath("elixirc") + if err != nil { + t.Skip("elixirc not installed") + } + dir := t.TempDir() + source := `defmodule Same.Dsl do + defmacro kept(name) do + quote location: :keep do + def unquote(name)(), do: :kept + end + end +end + +defmodule Same.User do + require Same.Dsl + + Same.Dsl.kept(:one) +end +` + if err := os.WriteFile(filepath.Join(dir, "same.ex"), []byte(source), 0o644); err != nil { + t.Fatal(err) + } + cmd := exec.Command(elixirc, "-o", dir, "same.ex") + cmd.Dir = dir + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("elixirc: %v\n%s", err, out) + } + info, err := ReadDefinitionLines(filepath.Join(dir, "Elixir.Same.User.beam")) + if err != nil { + t.Fatal(err) + } + if info.ModuleLine != 9 { + t.Errorf("ModuleLine = %d, want 9", info.ModuleLine) + } + if got := info.Lines[FunctionKey{Name: "one", Arity: 0}]; got != 12 { + t.Errorf("one/0 line = %d, want 12, the call (all: %v)", got, info.Lines) + } +} + // A corrupt definitions count must fail the parse, not allocate for it first. func TestParseDebugInfoHugeDefinitionCount(t *testing.T) { var w etfTestWriter diff --git a/internal/beam/docs.go b/internal/beam/docs.go index 479ef08..1dd010c 100644 --- a/internal/beam/docs.go +++ b/internal/beam/docs.go @@ -4,6 +4,7 @@ package beam import ( "bytes" + "compress/gzip" "compress/zlib" "encoding/binary" "errors" @@ -137,11 +138,26 @@ func readChunks(path string, wanted ...string) (map[string][]byte, error) { if _, err := io.ReadFull(f, header[:]); err != nil { return nil, err } + // A module compiled with the `compressed` option, as some Erlang + // dependencies are, is a gzip stream around the usual container. + var r io.ReaderAt = f + size := info.Size() + if header[0] == 0x1f && header[1] == 0x8b { + data, err := gunzipBEAM(f) + if err != nil { + return nil, err + } + if len(data) < 12 { + return nil, fmt.Errorf("invalid BEAM size %d", len(data)) + } + r, size = bytes.NewReader(data), int64(len(data)) + copy(header[:], data) + } if string(header[:4]) != "FOR1" || string(header[8:]) != "BEAM" { return nil, errors.New("invalid BEAM header") } declared := int64(binary.BigEndian.Uint32(header[4:8])) + 8 - if declared > info.Size() { + if declared > size { return nil, errors.New("truncated BEAM container") } @@ -153,7 +169,7 @@ func readChunks(path string, wanted ...string) (map[string][]byte, error) { for offset := int64(12); offset+8 <= declared && len(found) < len(pending); { var chunkHeader [8]byte - if _, err := f.ReadAt(chunkHeader[:], offset); err != nil { + if _, err := r.ReadAt(chunkHeader[:], offset); err != nil { return nil, err } length := int64(binary.BigEndian.Uint32(chunkHeader[4:8])) @@ -169,7 +185,7 @@ func readChunks(path string, wanted ...string) (map[string][]byte, error) { return nil, fmt.Errorf("%s chunk too large: %d", name, length) } data := make([]byte, length) - if _, err := f.ReadAt(data, dataOffset); err != nil { + if _, err := r.ReadAt(data, dataOffset); err != nil { return nil, err } found[name] = data @@ -179,6 +195,28 @@ func readChunks(path string, wanted ...string) (map[string][]byte, error) { return found, nil } +// gunzipBEAM decompresses a compressed BEAM from its start. The output is +// bounded like an uncompressed file, so a small file cannot expand into an +// unbounded allocation. +func gunzipBEAM(f *os.File) ([]byte, error) { + if _, err := f.Seek(0, io.SeekStart); err != nil { + return nil, err + } + zr, err := gzip.NewReader(f) + if err != nil { + return nil, fmt.Errorf("decompress BEAM: %w", err) + } + defer func() { _ = zr.Close() }() + data, err := io.ReadAll(io.LimitReader(zr, maxBEAMSize+1)) + if err != nil { + return nil, fmt.Errorf("decompress BEAM: %w", err) + } + if len(data) > maxBEAMSize { + return nil, errors.New("decompressed BEAM too large") + } + return data, nil +} + func readChunk(path, wanted string) ([]byte, error) { chunks, err := readChunks(path, wanted) if err != nil { diff --git a/internal/beam/docs_test.go b/internal/beam/docs_test.go index 9f35a4a..2eec639 100644 --- a/internal/beam/docs_test.go +++ b/internal/beam/docs_test.go @@ -2,6 +2,7 @@ package beam import ( "bytes" + "compress/gzip" "compress/zlib" "encoding/binary" "os" @@ -197,6 +198,59 @@ func TestReadExports(t *testing.T) { } } +// A module compiled with the `compressed` option is a gzip stream around the +// container. Its exports read the same, and a compressed file that does not +// hold a container is still rejected. +func TestReadExportsCompressedBEAM(t *testing.T) { + dir := t.TempDir() + plain := filepath.Join(dir, "plain.beam") + writeTestBEAM(t, plain, buildDocsTerm()) + data, err := os.ReadFile(plain) + if err != nil { + t.Fatal(err) + } + gzipped := func(data []byte) []byte { + var buf bytes.Buffer + zw := gzip.NewWriter(&buf) + if _, err := zw.Write(data); err != nil { + t.Fatal(err) + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + return buf.Bytes() + } + + compressed := filepath.Join(dir, "Elixir.Example.beam") + if err := os.WriteFile(compressed, gzipped(data), 0o644); err != nil { + t.Fatal(err) + } + want, err := ReadExports(plain) + if err != nil { + t.Fatal(err) + } + got, err := ReadExports(compressed) + if err != nil { + t.Fatal(err) + } + if len(got) != len(want) { + t.Fatalf("compressed exports = %#v, want %#v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("compressed export %d = %#v, want %#v", i, got[i], want[i]) + } + } + + notBEAM := filepath.Join(dir, "not.beam") + if err := os.WriteFile(notBEAM, gzipped([]byte("this is not a BEAM container")), 0o644); err != nil { + t.Fatal(err) + } + if _, err := ReadExports(notBEAM); err == nil { + t.Fatal("ReadExports accepted a compressed file that holds no BEAM container") + } +} + // OTP 28 widened atom names past 255 bytes and re-encoded the AtU8 chunk: the // count is stored negated and every length becomes a tagged varint instead of a // raw byte. The chunk kept its name, so parsers must branch on the sign bit. diff --git a/internal/daemon/endpoint.go b/internal/daemon/endpoint.go index 9c550bd..f5bdb68 100644 --- a/internal/daemon/endpoint.go +++ b/internal/daemon/endpoint.go @@ -18,7 +18,7 @@ import ( // running daemon, whose startup then rebuilds a populated index whose // IndexVersion differs. Routine changes that leave frontends and daemons // compatible do not bump it, so a running daemon is left alone. -const ContractVersion = 2 +const ContractVersion = 3 // maxSocketPath keeps a workspace socket inside sockaddr_un on every supported // platform (about 104 bytes on macOS, 108 on Linux), including the NUL. diff --git a/internal/lsp/generated_definition_integration_test.go b/internal/lsp/generated_definition_integration_test.go index 93b0212..b47cda3 100644 --- a/internal/lsp/generated_definition_integration_test.go +++ b/internal/lsp/generated_definition_integration_test.go @@ -49,6 +49,12 @@ const compiledDslSource = `defmodule Weird.Dsl do defmacro plug(_name), do: nil defmacro action(_name), do: nil + defmacro on(event) do + quote do + def handle_event(unquote(event)), do: unquote(event) + end + end + defmacro helper(name) do quote bind_quoted: [name: name] do defmodule Module.concat(__MODULE__, Macro.camelize(Atom.to_string(name))) do @@ -125,6 +131,9 @@ const compiledUserSource = `defmodule Weird.User do action :submit later :submit + on :click + on :hover + pinned :pinned_fun plug :match @@ -170,6 +179,7 @@ const compiledCallerSource = `defmodule Weird.Caller do Weird.User.active?(%{}) Weird.User.deferred() Weird.User.submit() + Weird.User.handle_event(:click) Weird.User.pinned_fun() Weird.User.match(:get, "/a") Weird.User.handle(:get) @@ -299,6 +309,7 @@ func TestDefinition_GeneratedFunctionsFromCompiler(t *testing.T) { {"predicate from an atom", "active?(%{})", 1, []string{user("flag :active")}}, {"@before_compile hook", "deferred(", 1, []string{user("later :deferred")}}, {"declaring call next to a same-named call", "submit(", 1, []string{user("later :submit")}}, + {"def in a macro's quote", "handle_event(", 1, []string{user("on :click"), user("on :hover")}}, {"line past the end of the file", "pinned_fun(", 1, []string{user("pinned :pinned_fun")}}, {"one clause per call", "match(", 1, []string{user(`route :get`), user(`route :post`)}}, {"one location: :keep clause per call", "handle(", 1, []string{user("keep_route :handle, :get"), user("keep_route :handle, :post")}}, diff --git a/internal/parser/parser_test.go b/internal/parser/parser_test.go index 6ba6441..27c1a2d 100644 --- a/internal/parser/parser_test.go +++ b/internal/parser/parser_test.go @@ -233,6 +233,83 @@ end } } +// A def inside a macro's quote is code the macro generates in its caller, not a +// function of the macro's module. What __using__ injects is still indexed, and +// so is a quote in a helper function, which the use-chain resolves. +func TestParseFile_DoesNotIndexDefsInsideMacroQuotes(t *testing.T) { + path := writeTempFile(t, `defmodule MyApp.Dsl do + defmacro __using__(_opts) do + quote do + import MyApp.Dsl + def injected, do: :injected + end + end + + defmacro route(verb, path) do + quote do + def handle(unquote(verb), unquote(path)), do: :ok + end + end + + defmacro kept(name) do + quote bind_quoted: [ + name: name + ], + location: :keep do + def bound_def, do: name + end + end + + defmacro assigned do + ast = quote do + def assigned_def, do: :ok + end + ast + end + + defmacro inline do + quote do: def inline_def, do: :ok + end + + defmacro __before_compile__(_env) do + quote do + def before_compile_def, do: :ok + end + end + + def controller do + quote do + def helper_quote_def, do: :ok + end + end + + def handle_request(conn), do: conn + defmacro real_macro(x), do: x +end +`) + + defs, _, err := ParseFile(path) + if err != nil { + t.Fatal(err) + } + found := map[string]bool{} + for _, d := range defs { + if d.Function != "" { + found[d.Function] = true + } + } + for _, name := range []string{"__using__", "injected", "route", "kept", "assigned", "inline", "__before_compile__", "controller", "helper_quote_def", "handle_request", "real_macro"} { + if !found[name] { + t.Errorf("%s should be indexed; got %v", name, found) + } + } + for _, name := range []string{"handle", "bound_def", "assigned_def", "inline_def", "before_compile_def"} { + if found[name] { + t.Errorf("%s is generated by a macro's quote and should not be indexed as MyApp.Dsl.%s", name, name) + } + } +} + func TestParseFile_DoesNotIndexUnquoteFragmentsAsDefinitions(t *testing.T) { path := writeTempFile(t, `defmodule MyApp.Generated do for name <- [:foo, :bar] do diff --git a/internal/parser/parser_tokenized.go b/internal/parser/parser_tokenized.go index efaffbf..756e507 100644 --- a/internal/parser/parser_tokenized.go +++ b/internal/parser/parser_tokenized.go @@ -18,6 +18,24 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] var moduleStack []moduleFrame depth := 0 + + // A def inside a macro's quote is code the macro generates in its caller, + // not a function of the macro's module, so it is not indexed as one. The + // compiled caller's BEAM records where it was declared. `__using__` is the + // exception: what it injects is resolved through its body, and kept. + // macroBodies holds the depth inside each open do-block of such a macro, + // innermost last. quoteDepth is the depth inside an open `quote do` block + // in one, or zero. quoteLine is the line of an inline `quote do:`. + var macroBodies []int + quoteDepth, quoteLine := 0, 0 + leaveBlocks := func() { + if quoteDepth > depth { + quoteDepth = 0 + } + for len(macroBodies) > 0 && macroBodies[len(macroBodies)-1] > depth { + macroBodies = macroBodies[:len(macroBodies)-1] + } + } aliases := map[string]string{} injectors := map[string]bool{} @@ -218,6 +236,7 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] aliases = frame.savedAliases injectors = frame.savedInjectors } + leaveBlocks() } } } @@ -254,6 +273,7 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] aliases = frame.savedAliases injectors = frame.savedInjectors } + leaveBlocks() i++ continue @@ -308,6 +328,16 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] delegateTo, delegateAs = scanDelegateOpts(pj) } + if (kind == "defmacro" || kind == "defmacrop") && funcName != "__using__" && opensDoBlock(tokens, n, pj) { + // The do on this line is counted when the line is walked + // below, so the body is one deeper than here. + macroBodies = append(macroBodies, depth+1) + } + if (quoteDepth > 0 && depth >= quoteDepth) || quoteLine == defLine { + i = j + goto extractRefsForLine + } + minArity := maxArity - defaultCount for arity := minArity; arity <= maxArity; arity++ { params := JoinParams(paramNames, arity) @@ -636,6 +666,15 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] continue case TokIdent: + if len(macroBodies) > 0 && quoteDepth == 0 && tokenText(tok) == "quote" { + switch form, line := quoteForm(source, tokens, n, i+1); form { + case quoteBlock: + // Counted when the walk reaches the do. + quoteDepth = depth + 1 + case quoteInline: + quoteLine = line + } + } cm := currentModule() if cm != "" && len(injectors) > 0 { isStatementStart := i == 0 || tokens[i-1].Kind == TokEOL || tokens[i-1].Kind == TokComment @@ -1014,3 +1053,65 @@ func FixParamNames(names []string) []string { } return names } + +// opensDoBlock reports whether the declaration whose head ends before tokens[from] +// opens a do-block on its line, rather than taking `, do:` or having no body. +func opensDoBlock(tokens []Token, n, from int) bool { + for k := from; k < n; k++ { + switch tokens[k].Kind { + case TokDo: + return true + case TokComma, TokEOL, TokEOF: + return false + } + } + return false +} + +type quoteShape int + +const ( + quoteNone quoteShape = iota + quoteBlock + quoteInline +) + +// quoteForm reads what follows a `quote` identifier: options such as +// `location: :keep` or `bind_quoted: [...]`, which may span lines inside +// brackets, then either a do-block or an inline `do:`. For the inline form it +// also returns the line of the `do:`. +func quoteForm(source []byte, tokens []Token, n, from int) (quoteShape, int) { + balance := 0 + // A trailing comma carries the options on to the next line. + continued := false + for k := from; k < n; k++ { + kind := tokens[k].Kind + switch kind { + case TokOpenParen, TokOpenBracket, TokOpenBrace: + balance++ + case TokCloseParen, TokCloseBracket, TokCloseBrace: + balance-- + if balance < 0 { + return quoteNone, 0 + } + case TokDo: + if balance == 0 { + return quoteBlock, 0 + } + case TokIdent: + if TokenText(source, tokens[k]) == "do" && k+1 < n && tokens[k+1].Kind == TokColon { + return quoteInline, tokens[k].Line + } + case TokEOL: + if balance == 0 && !continued { + return quoteNone, 0 + } + case TokEOF: + return quoteNone, 0 + } + if kind != TokEOL && kind != TokComment { + continued = kind == TokComma + } + } + return quoteNone, 0 +} diff --git a/internal/version/version.go b/internal/version/version.go index 97903fe..b896a42 100644 --- a/internal/version/version.go +++ b/internal/version/version.go @@ -7,4 +7,4 @@ const Version = "0.7.2" // a change that makes existing indexes stale — and bump daemon.ContractVersion // in the same release, because a running daemon from the older build would // otherwise keep serving the new frontend from that stale index. -const IndexVersion = 14 +const IndexVersion = 15 From 3bbd46a365b61a61b2e6abeca5edcc1fbdd88e8d Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Sat, 3 Oct 2026 16:35:54 -0400 Subject: [PATCH 15/16] Close the remaining gaps in quote skipping, worktree moves and deep terms A macro written with `, do:` has no do-block of its own, so a quote in it (`, do: quote do` on one line, or `quote do` on the next) was never seen as inside a macro, and its defs stayed indexed. The macro's head line is now scanned for a quote, and a quote right after its `do:` is recognized too. git worktree move renames the directory, then writes its .git file again in place, so a watcher can read the file empty. The worktree then looked like a plain directory: the fsnotify backend reported every file in it, and the FSEvents backend ran a full reconcile. A .git file that names no git directory yet now makes the directory a pending top, which is checked again once git is done. This was the cause of the flaky TestFSNotifyWatcherSkipsWorktreeAddedWhileRunning on Linux. Stepping over an ETF term recursed, under a depth limit that a deeply nested clause body could exceed, which cost the whole module its debug info. It is now a loop over a count of the terms still to skip, bounded by the bytes left, so nesting costs no stack. It is also about 1.8x faster on large Dbgi chunks. readAnnoLine is a loop for the same reason. Co-Authored-By: Claude Opus 5.5 --- docs/architecture.md | 2 +- internal/beam/debug_info.go | 8 +- internal/beam/debug_info_test.go | 7 +- internal/beam/docs.go | 43 ++-- internal/beam/docs_test.go | 29 ++- internal/beam/etf.go | 231 ++++++++---------- internal/parser/parser.go | 25 ++ internal/parser/parser_test.go | 18 +- internal/parser/parser_tokenized.go | 80 ++++-- internal/workspace/watch_fsnotify.go | 11 +- internal/workspace/watch_platform_darwin.go | 13 +- .../workspace/watch_platform_darwin_test.go | 23 ++ internal/workspace/watch_test.go | 26 ++ 13 files changed, 331 insertions(+), 185 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index cb833d0..780b385 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -139,7 +139,7 @@ Generated-symbol resolution is shared by completion, hover, definition, signatur ### Definition lines for generated functions -A generated function has no source definition, but the compiled module's `Dbgi` chunk still records where each def came from. `beam.ReadDefinitionLines` walks `{:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}}` for the `file`, `relative_file`, and `definitions` keys, and steps over clause bodies without allocating. Clause ASTs nest far deeper than a Docs term, so this reader raises the ETF depth guard to `maxDebugInfoDepth`. +A generated function has no source definition, but the compiled module's `Dbgi` chunk still records where each def came from. `beam.ReadDefinitionLines` walks `{:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}}` for the `file`, `relative_file`, and `definitions` keys, and steps over clause bodies without allocating. Clause ASTs can nest very deep, so the ETF reader steps over a term with a count of the terms still to skip instead of recursion: nesting costs no stack, and a count larger than the bytes left is rejected as corrupt. Each def carries two locations. Its `:line` is the line in the module's own file that was being compiled when the def was produced: the macro call, or wherever a before-compile hook ran. A `file: {path, line}` entry, left by `@file` or `quote location: :keep`, is preferred when `path` is the module's own source, because that is where the code asked for the function. When `path` is another file it is the generator's implementation, and `:line` is used instead. A `file:` line at or before the module's own line is not used either: it is a `location: :keep` quote in a macro defined above the module in the same file. Elixir 1.17 and earlier record the module's line under the map's `line` key instead of `anno`, and both are read. Ash's code interfaces take the second shape: each generated def is expanded at its `define` line, so `:line` is that line and `file` still names `code_interface.ex`, which keeps stacktraces unchanged. No framework is recognized by name. diff --git a/internal/beam/debug_info.go b/internal/beam/debug_info.go index e73f012..27b4435 100644 --- a/internal/beam/debug_info.go +++ b/internal/beam/debug_info.go @@ -6,12 +6,6 @@ import ( "slices" ) -// maxDebugInfoDepth bounds recursion while stepping over clause bodies. Each -// level of Elixir AST costs two ETF levels (the call tuple and its argument -// list), so this allows a thousand nested expressions: far past any real code, -// and still a bounded Go stack on a corrupt file. -const maxDebugInfoDepth = 2048 - // maxPreallocatedSites bounds the definitions slice allocated up front. const maxPreallocatedSites = 1024 @@ -100,7 +94,7 @@ func parseDebugInfo(buf []byte) (info DebugInfo, err error) { } }() - r := &etfReader{buf: buf, maxDepth: maxDebugInfoDepth} + r := &etfReader{buf: buf} if arity, err := r.enterTuple(); err != nil { return DebugInfo{}, err } else if arity != 3 { diff --git a/internal/beam/debug_info_test.go b/internal/beam/debug_info_test.go index bb28612..0643d3b 100644 --- a/internal/beam/debug_info_test.go +++ b/internal/beam/debug_info_test.go @@ -302,7 +302,8 @@ func TestReadDefinitionLinesTruncated(t *testing.T) { } // Clause bodies are arbitrary AST and nest far deeper than a Docs chunk. A long -// pipeline or nested case must not trip the depth guard and lose the module. +// pipeline or nested case is stepped over without recursion, so it costs no +// stack and does not lose the module. func TestReadDefinitionLinesDeepClauseBody(t *testing.T) { var w etfTestWriter w.smallTuple(3) @@ -324,8 +325,8 @@ func TestReadDefinitionLinesDeepClauseBody(t *testing.T) { w.atom("line") w.smallInt(10 + i) w.nil() - // 300 nested {:|>, [], [left, ...]} levels: 600 ETF levels. - const depth = 300 + // 100,000 nested {:|>, [], [left, ...]} levels: 200,000 ETF levels. + const depth = 100_000 for range depth { w.smallTuple(3) w.atom("|>") diff --git a/internal/beam/docs.go b/internal/beam/docs.go index 1dd010c..d4b042a 100644 --- a/internal/beam/docs.go +++ b/internal/beam/docs.go @@ -567,29 +567,36 @@ func parseDocsEntry(r *etfReader) ([]Function, bool, error) { // consumed and reported as line 0 rather than failing the whole entry. A dropped annotation must never cost the // module its generated functions. func readAnnoLine(r *etfReader) (int, error) { - tag, err := r.peekTag() - if err != nil { - return 0, err - } - switch tag { - case tagSmallInteger, tagInteger: - return r.readInt() - case tagSmallTuple: - // {line, column}, as erl_anno writes a location with a column. - arity, err := r.enterTuple() + // rest counts the elements after each tuple's first one, which are stepped + // over once the line is found. A loop rather than recursion, so a corrupt + // chunk of nested tuples costs no stack. + rest := int64(0) + for { + tag, err := r.peekTag() if err != nil { return 0, err } - if arity == 0 { - return 0, nil - } - line, err := readAnnoLine(r) - if err != nil { - return 0, err + switch tag { + case tagSmallInteger, tagInteger: + line, err := r.readInt() + if err != nil { + return 0, err + } + return line, r.skipTerms(rest) + case tagSmallTuple: + // {line, column}, as erl_anno writes a location with a column. + arity, err := r.enterTuple() + if err != nil { + return 0, err + } + if arity == 0 { + return 0, r.skipTerms(rest) + } + rest += int64(arity) - 1 + continue } - return line, r.skipTerms(int64(arity) - 1) + return 0, r.skipTerms(rest + 1) } - return 0, r.skip() } // readSignatureParams consumes the signature list and parses the first signature diff --git a/internal/beam/docs_test.go b/internal/beam/docs_test.go index 2eec639..d01abfd 100644 --- a/internal/beam/docs_test.go +++ b/internal/beam/docs_test.go @@ -136,20 +136,35 @@ func TestParseDocsMalformed(t *testing.T) { } } -// Deeply nested terms must be rejected rather than followed into a stack -// overflow, which a corrupt length field could otherwise cause. -func TestParseDocsRejectsDeepNesting(t *testing.T) { +// Stepping over a deeply nested term costs no stack, and a nested anno is read +// the same way: a million levels, which a corrupt chunk could hold, must not +// overflow the stack. +func TestParseDocsDeepNestingCostsNoStack(t *testing.T) { var w etfTestWriter w.smallTuple(7) w.atom("docs_v1") - // The anno field is skipped, so nesting it past the limit exercises the - // depth guard inside skip rather than the header checks. - for range maxETFDepth + 10 { + for range 1_000_000 { w.smallTuple(1) } w.nil() + // The rest of docs_v1 is missing, so the parse fails, but only after it + // stepped over the nested anno. if _, err := parseDocs(w.buf); err == nil { - t.Error("expected deeply nested input to be rejected") + t.Error("expected the incomplete term to be rejected") + } + + var anno etfTestWriter + for range 1_000_000 { + anno.smallTuple(2) + } + anno.smallInt(7) + for range 1_000_000 { + anno.smallInt(1) + } + r := &etfReader{buf: anno.buf} + line, err := readAnnoLine(r) + if err != nil || line != 7 || r.remaining() != 0 { + t.Errorf("readAnnoLine = %d, %v with %d bytes left; want 7 and the whole term read", line, err, r.remaining()) } } diff --git a/internal/beam/etf.go b/internal/beam/etf.go index 661e410..1d7ab7a 100644 --- a/internal/beam/etf.go +++ b/internal/beam/etf.go @@ -40,15 +40,10 @@ func isAtomTag(tag byte) bool { return tag == tagAtom || tag == tagAtomUTF8 || tag == tagSmallAtom || tag == tagSmallAtomUTF8 } -// maxETFDepth bounds recursion in skip. A Docs chunk nests at most five levels -// deep, so a corrupt length that implied more is rejected rather than followed. -const maxETFDepth = 64 - var ( errUnsupportedTag = errors.New("unsupported ETF tag") errTruncated = errors.New("truncated ETF term") errBadCount = errors.New("ETF count exceeds remaining bytes") - errTooDeep = errors.New("ETF nesting too deep") ) // etfReader walks an ETF term in place. @@ -63,13 +58,8 @@ var ( // Every read is bounds-checked, so malformed input yields an error rather than a // panic or an out-of-range slice. type etfReader struct { - buf []byte - pos int - depth int - - // maxDepth overrides maxETFDepth for terms that legitimately nest deeper, - // such as the clause ASTs in a Dbgi chunk. Zero keeps the default. - maxDepth int + buf []byte + pos int } func (r *etfReader) remaining() int { return len(r.buf) - r.pos } @@ -302,133 +292,116 @@ func (r *etfReader) binarySpan() (start, length int, err error) { // skip advances past one term of any shape without allocating for it. func (r *etfReader) skip() error { - limit := r.maxDepth - if limit == 0 { - limit = maxETFDepth - } - if r.depth >= limit { - return errTooDeep + return r.skipTerms(1) +} + +// skipTerms steps over count terms. It does not recurse: a tuple, list or map +// adds its elements to the terms still to step over, so a deeply nested term, +// as a clause body in a Dbgi chunk can be, costs no stack. Every term takes at +// least one byte, so more terms to step over than bytes left is corrupt input, +// and is rejected before it can drive the loop. +func (r *etfReader) skipTerms(count int64) error { + pending := int64(0) + add := func(n int64) error { + if n < 0 { + return errBadCount + } + pending += n + if pending > int64(r.remaining()) { + return errBadCount + } + return nil } - tag, err := r.u8() - if err != nil { + if err := add(count); err != nil { return err } - switch tag { - case tagNil: - return nil - case tagSmallInteger: - return r.skipBytes(1) - case tagInteger: - return r.skipBytes(4) - case tagNewFloat: - return r.skipBytes(8) - case tagFloat: - return r.skipBytes(31) - case tagAtom, tagAtomUTF8, tagString: - n, err := r.u16() - if err != nil { - return err - } - return r.skipBytes(int64(n)) - case tagSmallAtom, tagSmallAtomUTF8: - n, err := r.u8() - if err != nil { - return err - } - return r.skipBytes(int64(n)) - case tagBinary: - n, err := r.u32() - if err != nil { - return err - } - return r.skipBytes(n) - case tagBitBinary: - n, err := r.u32() - if err != nil { - return err - } - if err := r.skipBytes(1); err != nil { - return err - } - return r.skipBytes(n) - case tagSmallBig: - n, err := r.u8() - if err != nil { - return err - } - return r.skipBytes(int64(n) + 1) - case tagLargeBig: - n, err := r.u32() + for ; pending > 0; pending-- { + tag, err := r.u8() if err != nil { return err } - return r.skipBytes(n + 1) - case tagSmallTuple: - n, err := r.u8() - if err != nil { - return err - } - return r.skipTerms(int64(n)) - case tagLargeTuple: - n, err := r.u32() - if err != nil { - return err - } - return r.skipTerms(n) - case tagList: - n, err := r.u32() - if err != nil { - return err - } - if err := r.checkCount(n, 1); err != nil { - return err - } - if err := r.skipTerms(n); err != nil { - return err - } - return r.skip() // the tail - case tagMap: - n, err := r.u32() - if err != nil { - return err - } - if err := r.checkCount(n, 2); err != nil { - return err + switch tag { + case tagNil: + case tagSmallInteger: + err = r.skipBytes(1) + case tagInteger: + err = r.skipBytes(4) + case tagNewFloat: + err = r.skipBytes(8) + case tagFloat: + err = r.skipBytes(31) + case tagAtom, tagAtomUTF8, tagString: + var n int + if n, err = r.u16(); err == nil { + err = r.skipBytes(int64(n)) + } + case tagSmallAtom, tagSmallAtomUTF8: + var n byte + if n, err = r.u8(); err == nil { + err = r.skipBytes(int64(n)) + } + case tagBinary: + var n int64 + if n, err = r.u32(); err == nil { + err = r.skipBytes(n) + } + case tagBitBinary: + var n int64 + if n, err = r.u32(); err == nil { + err = r.skipBytes(n + 1) + } + case tagSmallBig: + var n byte + if n, err = r.u8(); err == nil { + err = r.skipBytes(int64(n) + 1) + } + case tagLargeBig: + var n int64 + if n, err = r.u32(); err == nil { + err = r.skipBytes(n + 1) + } + case tagSmallTuple: + var n byte + if n, err = r.u8(); err == nil { + err = add(int64(n)) + } + case tagLargeTuple: + var n int64 + if n, err = r.u32(); err == nil { + err = add(n) + } + case tagList: + // The elements, then the tail. + var n int64 + if n, err = r.u32(); err == nil { + err = add(n + 1) + } + case tagMap: + var n int64 + if n, err = r.u32(); err == nil { + err = add(n * 2) + } + case tagNewFun: + // Size counts the whole term including the four size bytes themselves. + var n int64 + if n, err = r.u32(); err == nil { + err = r.skipBytes(n - 4) + } + case tagExport: + // An external fun, `fun M:F/A`. Unlike NEW_FUN_EXT, whose Arity is a + // bare byte, this one encodes the arity as an integer term, so + // skipping a fixed byte would misalign every term that follows it. + if _, err = r.readAtom(); err == nil { + if _, err = r.readAtom(); err == nil { + err = add(1) + } + } + default: + err = fmt.Errorf("%w: %d", errUnsupportedTag, tag) } - return r.skipTerms(n * 2) - case tagNewFun: - // Size counts the whole term including the four size bytes themselves. - n, err := r.u32() if err != nil { return err } - return r.skipBytes(n - 4) - case tagExport: - // An external fun, `fun M:F/A`. Unlike NEW_FUN_EXT, whose Arity is a bare - // byte, this one encodes the arity as an integer term, so skipping a fixed - // byte would misalign every term that follows it. - if _, err := r.readAtom(); err != nil { - return err - } - if _, err := r.readAtom(); err != nil { - return err - } - return r.skip() - default: - return fmt.Errorf("%w: %d", errUnsupportedTag, tag) - } -} - -func (r *etfReader) skipTerms(count int64) error { - if err := r.checkCount(count, 1); err != nil { - return err - } - r.depth++ - defer func() { r.depth-- }() - for i := int64(0); i < count; i++ { - if err := r.skip(); err != nil { - return err - } } return nil } diff --git a/internal/parser/parser.go b/internal/parser/parser.go index fb2714c..e356b5d 100644 --- a/internal/parser/parser.go +++ b/internal/parser/parser.go @@ -237,6 +237,31 @@ func HasLinkedWorktreeGitFile(dir string, entries []fs.DirEntry) bool { return false } +// UnsettledGitFile reports whether dir holds a .git file that names no git +// directory yet. git rewrites a worktree's .git file in place when it moves the +// worktree, so for a moment after the rename the file is empty, and a directory +// that is a worktree looks like a plain one. Such a directory is checked again +// once git is done, rather than indexed. +func UnsettledGitFile(dir string) bool { + info, err := os.Lstat(filepath.Join(dir, ".git")) + if err != nil || !info.Mode().IsRegular() { + return false + } + _, ok := gitdirFromFile(filepath.Join(dir, ".git")) + return !ok +} + +// HasUnsettledGitFile is UnsettledGitFile for a directory whose entries are +// already read, so a directory without a .git file costs no syscall. +func HasUnsettledGitFile(dir string, entries []fs.DirEntry) bool { + for _, e := range entries { + if e.Name() == ".git" { + return !e.IsDir() && UnsettledGitFile(dir) + } + } + return false +} + // isLinkedWorktreeGitFile reports whether the .git file at path belongs to a // linked worktree. Submodules also have a .git file, and they stay indexed like // any other directory. Only a directory that has a .git file pays for this check. diff --git a/internal/parser/parser_test.go b/internal/parser/parser_test.go index 27c1a2d..39fafe2 100644 --- a/internal/parser/parser_test.go +++ b/internal/parser/parser_test.go @@ -277,6 +277,20 @@ func TestParseFile_DoesNotIndexDefsInsideMacroQuotes(t *testing.T) { end end + defmacro same_line(name), do: quote do + def same_line_def, do: unquote(name) + end + + defmacro next_line(name), do: + quote do + def next_line_def, do: unquote(name) + end + + defmacro one_line do quote do + def one_line_def, do: :ok + end + end + def controller do quote do def helper_quote_def, do: :ok @@ -298,12 +312,12 @@ end found[d.Function] = true } } - for _, name := range []string{"__using__", "injected", "route", "kept", "assigned", "inline", "__before_compile__", "controller", "helper_quote_def", "handle_request", "real_macro"} { + for _, name := range []string{"__using__", "injected", "route", "kept", "assigned", "inline", "__before_compile__", "same_line", "next_line", "one_line", "controller", "helper_quote_def", "handle_request", "real_macro"} { if !found[name] { t.Errorf("%s should be indexed; got %v", name, found) } } - for _, name := range []string{"handle", "bound_def", "assigned_def", "inline_def", "before_compile_def"} { + for _, name := range []string{"handle", "bound_def", "assigned_def", "inline_def", "before_compile_def", "same_line_def", "next_line_def", "one_line_def"} { if found[name] { t.Errorf("%s is generated by a macro's quote and should not be indexed as MyApp.Dsl.%s", name, name) } diff --git a/internal/parser/parser_tokenized.go b/internal/parser/parser_tokenized.go index 756e507..ef37abc 100644 --- a/internal/parser/parser_tokenized.go +++ b/internal/parser/parser_tokenized.go @@ -19,6 +19,11 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] var moduleStack []moduleFrame depth := 0 + aliases := map[string]string{} + injectors := map[string]bool{} + + n := len(tokens) + // A def inside a macro's quote is code the macro generates in its caller, // not a function of the macro's module, so it is not indexed as one. The // compiled caller's BEAM records where it was declared. `__using__` is the @@ -28,6 +33,18 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] // in one, or zero. quoteLine is the line of an inline `quote do:`. var macroBodies []int quoteDepth, quoteLine := 0, 0 + // inlineMacroLine is the line of the last such macro written with `, do:`, + // whose quote can follow on the next line. + inlineMacroLine := 0 + markQuote := func(at, depthAt int) { + switch form, line := quoteForm(source, tokens, n, at+1); form { + case quoteBlock: + // The do is counted when the walk reaches it. + quoteDepth = depthAt + 1 + case quoteInline: + quoteLine = line + } + } leaveBlocks := func() { if quoteDepth > depth { quoteDepth = 0 @@ -36,10 +53,6 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] macroBodies = macroBodies[:len(macroBodies)-1] } } - aliases := map[string]string{} - injectors := map[string]bool{} - - n := len(tokens) tokenText := func(t Token) string { return TokenText(source, t) @@ -328,10 +341,34 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] delegateTo, delegateAs = scanDelegateOpts(pj) } - if (kind == "defmacro" || kind == "defmacrop") && funcName != "__using__" && opensDoBlock(tokens, n, pj) { - // The do on this line is counted when the line is walked - // below, so the body is one deeper than here. - macroBodies = append(macroBodies, depth+1) + if (kind == "defmacro" || kind == "defmacrop") && funcName != "__using__" { + if opensDoBlock(tokens, n, pj) { + // The do on this line is counted when the line is walked + // below, so the body is one deeper than here. + macroBodies = append(macroBodies, depth+1) + } else { + inlineMacroLine = defLine + } + // The walk below takes the rest of this line, so a quote on + // it (`, do: quote do`) is found here. + if quoteDepth == 0 { + lineDepth := depth + for k := pj; k < n && tokens[k].Kind != TokEOL && tokens[k].Kind != TokEOF; k++ { + switch tokens[k].Kind { + case TokDo, TokFn: + lineDepth++ + case TokEnd: + lineDepth-- + case TokIdent: + if tokenText(tokens[k]) == "quote" { + markQuote(k, lineDepth) + } + } + if quoteDepth != 0 || quoteLine == defLine { + break + } + } + } } if (quoteDepth > 0 && depth >= quoteDepth) || quoteLine == defLine { i = j @@ -666,14 +703,8 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] continue case TokIdent: - if len(macroBodies) > 0 && quoteDepth == 0 && tokenText(tok) == "quote" { - switch form, line := quoteForm(source, tokens, n, i+1); form { - case quoteBlock: - // Counted when the walk reaches the do. - quoteDepth = depth + 1 - case quoteInline: - quoteLine = line - } + if quoteDepth == 0 && tokenText(tok) == "quote" && (len(macroBodies) > 0 || followsInlineMacroHead(source, tokens, i, inlineMacroLine)) { + markQuote(i, depth) } cm := currentModule() if cm != "" && len(injectors) > 0 { @@ -1115,3 +1146,20 @@ func quoteForm(source []byte, tokens []Token, n, from int) (quoteShape, int) { } return quoteNone, 0 } + +// followsInlineMacroHead reports whether the token at i comes right after the +// `do:` that ends the head of a macro on line, as in +// +// defmacro route(verb, path), do: +// quote do +func followsInlineMacroHead(source []byte, tokens []Token, i, line int) bool { + if line == 0 { + return false + } + k := i - 1 + for k >= 0 && (tokens[k].Kind == TokEOL || tokens[k].Kind == TokComment) { + k-- + } + return k >= 1 && tokens[k].Kind == TokColon && tokens[k-1].Kind == TokIdent && + TokenText(source, tokens[k-1]) == "do" && tokens[k-1].Line == line +} diff --git a/internal/workspace/watch_fsnotify.go b/internal/workspace/watch_fsnotify.go index 8c8e29f..f40f670 100644 --- a/internal/workspace/watch_fsnotify.go +++ b/internal/workspace/watch_fsnotify.go @@ -136,6 +136,14 @@ func (w *fsnotifyWatcher) walkDirectories(root string, includeRoot bool) int { w.tops.add(dir) return } + // A worktree that git is still moving has an empty .git file. It is a + // top until the pending check says otherwise, so that its files are + // not reported in the meantime. + if dir != w.root && parser.HasUnsettledGitFile(dir, entries) { + w.tops.add(dir) + w.markPending(dir) + return + } for _, e := range entries { if e.IsDir() && !skipWatchDir(e.Name()) { walk(filepath.Join(dir, e.Name())) @@ -179,7 +187,8 @@ func (w *fsnotifyWatcher) checkPending() { if parser.IsLinkedWorktree(dir) { continue } - if recordedWorktree(w.root, dir) { + // git may still be writing the .git file of a worktree it moved. + if parser.UnsettledGitFile(dir) || recordedWorktree(w.root, dir) { recorded = append(recorded, dir) continue } diff --git a/internal/workspace/watch_platform_darwin.go b/internal/workspace/watch_platform_darwin.go index f07e2b8..ad0565b 100644 --- a/internal/workspace/watch_platform_darwin.go +++ b/internal/workspace/watch_platform_darwin.go @@ -150,6 +150,16 @@ func (w *fseventsWatcher) handle(event fsevents.Event) { } return } + // A worktree that git is still moving has an empty .git file. It is a + // top until it is checked again, so that no full reconcile indexes it + // in the meantime. + if flags&(fsevents.ItemCreated|fsevents.ItemRenamed) != 0 && parser.UnsettledGitFile(path) { + if w.tops.add(path) { + w.callbacks.PathChanged(path) + } + w.checkTopLater(path) + return + } if flags&(fsevents.ItemCreated|fsevents.ItemRemoved|fsevents.ItemRenamed) != 0 { w.callbacks.FullReconcile() } @@ -196,7 +206,8 @@ func (w *fseventsWatcher) checkTopLater(dir string) { if parser.IsLinkedWorktree(dir) { return } - if recordedWorktree(w.root, dir) { + // git may still be writing the .git file of a worktree it moved. + if parser.UnsettledGitFile(dir) || recordedWorktree(w.root, dir) { w.checkTopLater(dir) return } diff --git a/internal/workspace/watch_platform_darwin_test.go b/internal/workspace/watch_platform_darwin_test.go index 50b6052..aef744d 100644 --- a/internal/workspace/watch_platform_darwin_test.go +++ b/internal/workspace/watch_platform_darwin_test.go @@ -350,6 +350,29 @@ func TestFSEventsWorktreeLifecycle(t *testing.T) { git("worktree", "remove", wt) settle() + // git worktree move renames the directory, then writes its .git file again + // in place, so the watcher can find the file empty. + outside := filepath.Join(t.TempDir(), "feature") + git("worktree", "add", "-q", outside) + content, err := os.ReadFile(filepath.Join(outside, ".git")) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(outside, ".git"), nil, 0o644); err != nil { + t.Fatal(err) + } + if err := os.Rename(outside, wt); err != nil { + t.Fatal(err) + } + settle() + if err := os.WriteFile(filepath.Join(wt, ".git"), content, 0o644); err != nil { + t.Fatal(err) + } + git("worktree", "repair", wt) + settle() + git("worktree", "remove", "--force", wt) + settle() + mu.Lock() defer mu.Unlock() if full != 0 { diff --git a/internal/workspace/watch_test.go b/internal/workspace/watch_test.go index a99080d..99776c9 100644 --- a/internal/workspace/watch_test.go +++ b/internal/workspace/watch_test.go @@ -539,6 +539,32 @@ func TestFSNotifyWatcherSkipsWorktreeAddedWhileRunning(t *testing.T) { env.settle(t) env.git("worktree", "move", outside, wt) }}, + // git worktree move renames the directory, then writes its .git file + // again in place, so the watcher can find the file empty. + {"move in while git rewrites its .git file", func(t *testing.T, env *watchedRepo, wt string) { + outside := filepath.Join(t.TempDir(), "feature") + env.git("worktree", "add", "-q", outside) + dotgit := filepath.Join(outside, ".git") + content, err := os.ReadFile(dotgit) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(dotgit, nil, 0o644); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Dir(wt), 0o755); err != nil { + t.Fatal(err) + } + env.settle(t) + if err := os.Rename(outside, wt); err != nil { + t.Fatal(err) + } + env.settle(t) + if err := os.WriteFile(filepath.Join(wt, ".git"), content, 0o644); err != nil { + t.Fatal(err) + } + env.git("worktree", "repair", wt) + }}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { From 7e8f1633258a78a8745df2103070807d9d1a1764 Mon Sep 17 00:00:00 2001 From: Jesse Herrick Date: Sat, 3 Oct 2026 16:42:58 -0400 Subject: [PATCH 16/16] Step over terms that end the input, and keep one-line macros indexed skipTerms counted a term among those still to step over while it read the term's header, so a valid term at the end of the input, such as an empty map or `[[], []]`, looked like more terms than bytes left and was rejected. A term is now counted off before its header is read. A macro written as `defmacro name, do: quote do: def generated` set the quote line to its own line while its head was scanned, and the check that follows then skipped the macro itself. Whether a def is inside a quote is now decided before its head is scanned. Co-Authored-By: Claude Opus 5.5 --- internal/beam/etf.go | 5 ++- internal/beam/etf_test.go | 50 +++++++++++++++++++++++++++++ internal/parser/parser_test.go | 6 ++-- internal/parser/parser_tokenized.go | 5 ++- 4 files changed, 62 insertions(+), 4 deletions(-) diff --git a/internal/beam/etf.go b/internal/beam/etf.go index 1d7ab7a..5db9899 100644 --- a/internal/beam/etf.go +++ b/internal/beam/etf.go @@ -315,7 +315,10 @@ func (r *etfReader) skipTerms(count int64) error { if err := add(count); err != nil { return err } - for ; pending > 0; pending-- { + for pending > 0 { + // Counted off before its header is read, so that its elements are + // compared with the bytes left after it, not with it as well. + pending-- tag, err := r.u8() if err != nil { return err diff --git a/internal/beam/etf_test.go b/internal/beam/etf_test.go index d38de00..05359a1 100644 --- a/internal/beam/etf_test.go +++ b/internal/beam/etf_test.go @@ -45,3 +45,53 @@ func TestSkipExportExtKeepsAlignment(t *testing.T) { }) } } + +// A term that ends exactly at the end of the input is stepped over whole, and +// every strict prefix of it is rejected: the count of terms still to step over +// is compared with the bytes after each header, not with the header as well. +func TestSkipTermsThatEndTheInput(t *testing.T) { + cases := map[string]func(w *etfTestWriter){ + "tuple of an empty list": func(w *etfTestWriter) { + w.smallTuple(1) + w.nil() + }, + "empty map": func(w *etfTestWriter) { + w.mapHeader(0) + }, + "list of empty lists": func(w *etfTestWriter) { + w.listHeader(2) + w.nil() + w.nil() + w.nil() + }, + "map of empty lists": func(w *etfTestWriter) { + w.mapHeader(1) + w.nil() + w.nil() + }, + "nested tuples": func(w *etfTestWriter) { + w.smallTuple(2) + w.smallTuple(1) + w.nil() + w.listHeader(1) + w.smallTuple(0) + w.nil() + }, + } + for name, build := range cases { + t.Run(name, func(t *testing.T) { + var w etfTestWriter + build(&w) + r := &etfReader{buf: w.buf} + if err := r.skip(); err != nil || r.remaining() != 0 { + t.Fatalf("skip = %v with %d bytes left, want the whole term", err, r.remaining()) + } + for i := range len(w.buf) { + r := &etfReader{buf: w.buf[:i]} + if err := r.skip(); err == nil { + t.Errorf("skip accepted a truncation at %d of %d bytes", i, len(w.buf)) + } + } + }) + } +} diff --git a/internal/parser/parser_test.go b/internal/parser/parser_test.go index 39fafe2..b09aa0f 100644 --- a/internal/parser/parser_test.go +++ b/internal/parser/parser_test.go @@ -291,6 +291,8 @@ func TestParseFile_DoesNotIndexDefsInsideMacroQuotes(t *testing.T) { end end + defmacro compact, do: quote do: def compact_def, do: :ok + def controller do quote do def helper_quote_def, do: :ok @@ -312,12 +314,12 @@ end found[d.Function] = true } } - for _, name := range []string{"__using__", "injected", "route", "kept", "assigned", "inline", "__before_compile__", "same_line", "next_line", "one_line", "controller", "helper_quote_def", "handle_request", "real_macro"} { + for _, name := range []string{"__using__", "injected", "route", "kept", "assigned", "inline", "__before_compile__", "same_line", "next_line", "one_line", "compact", "controller", "helper_quote_def", "handle_request", "real_macro"} { if !found[name] { t.Errorf("%s should be indexed; got %v", name, found) } } - for _, name := range []string{"handle", "bound_def", "assigned_def", "inline_def", "before_compile_def", "same_line_def", "next_line_def", "one_line_def"} { + for _, name := range []string{"handle", "bound_def", "assigned_def", "inline_def", "before_compile_def", "same_line_def", "next_line_def", "one_line_def", "compact_def"} { if found[name] { t.Errorf("%s is generated by a macro's quote and should not be indexed as MyApp.Dsl.%s", name, name) } diff --git a/internal/parser/parser_tokenized.go b/internal/parser/parser_tokenized.go index ef37abc..b36e476 100644 --- a/internal/parser/parser_tokenized.go +++ b/internal/parser/parser_tokenized.go @@ -341,6 +341,9 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] delegateTo, delegateAs = scanDelegateOpts(pj) } + // Decided before the head is scanned below, which can mark a + // quote on this same line (`defmacro m, do: quote do: ...`). + insideQuote := (quoteDepth > 0 && depth >= quoteDepth) || quoteLine == defLine if (kind == "defmacro" || kind == "defmacrop") && funcName != "__using__" { if opensDoBlock(tokens, n, pj) { // The do on this line is counted when the line is walked @@ -370,7 +373,7 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] } } } - if (quoteDepth > 0 && depth >= quoteDepth) || quoteLine == defLine { + if insideQuote { i = j goto extractRefsForLine }