Repository navigation
feat(plugins): add extractor plugins, so a language can be added by installing a package - #4875
arvind-iyer-2001 wants to merge 2 commits into
Conversation
Plugins are discovered through the entry-point group graphify.extractors (or GRAPHIFY_PLUGINS=mod:attr). A plugin matches by suffix, filename, glob or shebang and extracts per file or in one batch over all its files. Output is validated; a failing plugin leaves a visible error and never falls back. detect, extract, cache salting, watch and a graphify plugins command use it.
|
Thanks for the pull request, @arvind-iyer-2001. A maintainer will review it soon. Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions. A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic. |
github-actions
Bot
added
the
needs:title
|
Thanks @arvind-iyer-2001! One small thing: our changelog and release notes are built from PR titles, so we use the conventional-commits format, for example: Prefixes we use: |
arvind-iyer-2001
changed the title
github-actions
Bot
removed
the
needs:title
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; no formal proof of the changed code.
Graphify review — findings
Adds an extractor-plugin system: packages register through the graphify.extractors entry-point group (or GRAPHIFY_PLUGINS) and add languages without graphify naming them. detect classifies their files by the plugin's file_type, and extract runs batch plugins once in the main process before the worker pool, then bypasses the per-file cache for batch and cross-file plugins. Built-in extractors keep their files unless a plugin sets override_builtin, and malformed output or a failing plugin becomes a per-file error with no fallback to another extractor. The new graphify plugins [--json] command lists what's installed and any load problems. Matching batch results back to files via resolved paths, and plugin validation and suffix normalization in coerce, are where this can silently drop or misroute files.
Worth a look
- Batch result keys are resolved against cwd, not the project root —
graphify/plugins.py· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- ExtractorPlugin instance suffixes are not normalized, so they silently never match —
graphify/plugins.py:128· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- coerce() skips validation for ExtractorPlugin instances —
graphify/plugins.py:128· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- _call passes ctx into an optional positional parameter —
graphify/plugins.py· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- Plugin file_type is ignored for built-in extensions —
graphify/detect.py:571· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 4767 functions depend on the 824 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
extract()— 1060 callers, 64 callees - worse:
dispatch_command()— 5 callers, 138 callees - worse:
classify_file()— 54 callers, 6 callees - worse:
_shebang_interpreter()— 39 callers, 3 callees - worse:
cache_salt()— 11 callers, 3 callees - new:
coerce()— 7 callers, 4 callees - new:
run_batches()— 2 callers, 8 callees - new:
_builtin_extractor()— 1 callers, 7 callees
Verification — 4767 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 4665 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
470 of 470 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/pi.test.ts— full-run-safetytests/test_affected_cli.py— impact, full-run-safetytests/test_affected_defines.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_affected_overrides.py— impact, full-run-safetytests/test_affected_path_file_node.py— full-run-safetytests/test_agents_platform.py— impact, full-run-safetytests/test_alias_folding.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— impact, full-run-safetytests/test_astro_import_ids.py— impact, full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— impact, full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_backend_opt_in.py— impact, full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_benchmark_real_query.py— full-run-safetytests/test_binary_media_words.py— impact, full-run-safetytests/test_blade_extractor.py— impact, full-run-safetytests/test_build.py— impact, full-run-safetytests/test_build_located_semantic_identity.py— full-run-safetytests/test_build_merge_dedup_scope.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— impact, full-run-safetytests/test_c_include_resolution.py— impact, full-run-safetytests/test_cache.py— impact, full-run-safetytests/test_cache_long_paths.py— full-run-safetytests/test_cache_stale_import_target.py— impact, full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— impact, full-run-safetytests/test_cargo_missing_manifest.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_sensitive_resolution.py— impact, full-run-safetytests/test_charmap_encoding.py— impact, full-run-safetytests/test_chunking.py— impact, full-run-safetytests/test_cjs_module_extension.py— impact, full-run-safetytests/test_claude_cli_backend.py— impact, full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— impact, full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cloud_cta.py— impact, full-run-safety- … and 420 more
non-code file(s) changed (
docs/PLUGINS.md) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
docs/PLUGINS.md) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Docs that may be stale (advisory)
CHANGELOG.md§ 0.9.95 (2026-10-23) (lines 62-87): references changed symbolsmatchCHANGELOG.md§ 0.9.90 (2026-10-13) (lines 176-194): references changed symbolsloadCHANGELOG.md§ 0.9.89 (2026-10-13) (lines 195-208): references changed symbolsmatchCHANGELOG.md§ 0.9.81 (2026-10-08) (lines 337-355): references changed symbolsmatchCHANGELOG.md§ 0.9.73 (2026-09-30) (lines 472-487): references changed symbolsbatchCHANGELOG.md§ 0.9.57 (2026-09-09) (lines 666-673): references changed symbolsmatchCHANGELOG.md§ 0.9.56 (2026-09-07) (lines 674-686): references changed symbolsmatchCHANGELOG.md§ 0.9.52 (2026-08-29) (lines 728-743): references changed symbolsmatchCHANGELOG.md§ 0.9.50 (2026-08-25) (lines 757-768): references changed symbolsmatchCHANGELOG.md§ 0.9.36 (2026-08-07) (lines 942-948): references changed symbolsbatch
…and 10 more.
· 3 grounded finding(s) anchored inline below; 5 more finding(s) on lines outside this diff (see the check run).
| return effective_suffix(path) in _JS_CACHE_BYPASS_SUFFIXES or plugins.is_cross_file(path) | ||
|
|
||
|
|
||
| def _builtin_extractor(path: Path) -> Any | None: |
There was a problem hiding this comment.
_builtin_extractor()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return tuple(value) | ||
|
|
||
|
|
||
| def coerce(obj: Any, source: str = "") -> list[ExtractorPlugin]: |
There was a problem hiding this comment.
coerce()
7 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return _report(res, path) | ||
|
|
||
|
|
||
| def run_batches(paths: list[Path], root: Path) -> dict[str, dict]: |
There was a problem hiding this comment.
run_batches()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
What does this PR do?
Lets a language be added by installing a package, instead of editing
extract.pyanddetect.py.Why I built it. graphify has no AST extractor for q/kdb+, so q files fall through to the LLM pass, which is slow, costs tokens and isn't deterministic. I wrote qlens (not published yet), a q extractor that uses q's own
parseas the parser (no LLM, no tree-sitter grammar) and puts a confidence score on every node and edge. I wanted it to feed graphify's AST step for.qfiles, but not by hard-wiring q intoextract.py. A plugin hook means it ships as its own package, and the next language that doesn't fit the built-in tables can do the same.A package registers an extractor under the entry-point group
graphify.extractors. It says which files it handles (suffix, filename, glob or shebang) and gives either a per-fileextract(path)or a batchextract_batch(paths, ctx)that sees all its files at once. Batch exists because some extractors resolve names across files; I wanted this for q/kdb+, where that matters.GRAPHIFY_PLUGINS=module:attrloads a plugin that isn't installed as a package.graphify pluginslists what's loaded. The contract is indocs/PLUGINS.md.Built-in extractors still win unless a plugin sets
override_builtin. With no plugins installed, nothing changes.If a plugin crashes or returns malformed nodes/edges, the file goes into
failed_sourcesand the reason is printed. There is no fallback to another extractor.Type of change
Verification & Invariants
cache_key, so changing a plugin's output invalidates its cached files. Batch plugins skip the per-file cache, like JS/TS.confidenceis kept as given. If it omits one, it defaults toEXTRACTED, so a plugin that infers should say so.GRAPHIFY_PLUGINS, never from a project's.graphifyrc, because that file belongs to the repo being scanned.Limits:
register()is per-process, so spawn workers don't see it (entry points andGRAPHIFY_PLUGINSwork in workers). Plugins are discovered once per process.How was this tested?
tests/test_plugins.pyadds 31 tests, including a real run through the worker pool.I also installed a real q/kdb+ extractor as a plugin and ran a mixed Python/YAML/q repo: Python and YAML went through the worker pool,
.qthrough the batch plugin.tests/test_plugins.pyandtests/test_detect.pyalso pass on Python 3.10 and 3.13 (361 each), besides 3.14.Not run: the full suite with
--all-extras(lean environment) and any platform besides macOS.Existing flake, unrelated:
test_active_scan_root_defaults_to_noneandtest_active_scan_root_set_get_restore_nestedfail when run aftertest_extract_cache_location.py. They fail the same way on untouchedv8.Graphify-specific checklist