Engine registry unification, Arrow string layouts, DuckDB streaming and out-of-core paths - #89
Open
Haigutus wants to merge 25 commits into
Open
Engine registry unification, Arrow string layouts, DuckDB streaming and out-of-core paths#89Haigutus wants to merge 25 commits into
Haigutus wants to merge 25 commits into
Conversation
added 25 commits
July 26, 2026 09:09
Wrap duckdb.connect(table=, schema=), store defaults per connection, and resolve call kwargs → connection → package defaults into quoted SQL refs. Tools, read_rdf, exports, SHACL, and loaders all use the same path.
Keep all DuckDB engine concerns in one module; delete duckdb_table.py so there is no extra file to hunt for connection defaults and quoting.
Add to_pandas/to_arrow/to_polars/as_frame/match_flavor as the single materialization path for pandas, polars, pyarrow, and DuckDB inputs, and route loaders, SHACL, SPARQL, cgmes, export, and DuckDB export wrappers through them.
… schema parity - filter_triplets / filter_triplets_by_value with regex=True used SQL SIMILAR TO (full match); now regexp_matches() — search semantics anywhere in the value, matching the pandas/polars str.contains engines - filter_triplets_by_type interpolated the type name unescaped into SQL; now quoted via _lit() like every other literal in the engine - parse([], return_type="arrow"/"polars") now carries the same string / dictionary-encoded schema as a non-empty parse, so empties concatenate - new regex parity specs (real IdentifiedObject.name substring), duckdb quote-escaping test, empty-vs-full schema tests
…) override
One EngineRegistry mechanism for every subsystem: availability is probed
eagerly at import with find_spec (microseconds, imports nothing), the chosen
module still imports lazily on first use. New ctor args: requires (extra
probe targets per engine) and policy ("auto" = fastest wins, "input" =
engine bound to the input flavor, not overridable).
- triplets.engines(): what "auto" resolves to per subsystem, plus available
alternatives, aliases and auto order
- triplets.set_engine(parser=..., sparql=..., ...): global override; None or
"auto" restores; loads eagerly to fail fast. Precedence: per-call engine=
> set_engine > auto probe order
- export folds its ad-hoc dispatch into three registries: cimxml (replacing
the hand-rolled module map), nquads (auto: polars first) and csv
(policy="input": each engine is fastest for its own input flavor)
- broken-build fall-through: a probe-available engine whose import fails is
skipped with a warning and dropped from the availability report
tools._get_engine is now flavor-driven and registry-backed (kind "tools", policy="input"): triplets.tools.<fn>(con) reaches the duckdb engine (previously it silently fell through to pandas and raised AttributeError deep inside). Explicit engine= must match the input flavor (TypeError at the boundary), unknown names raise ValueError, and functions an engine does not implement raise NotImplementedError instead of AttributeError. cgmes_tools keeps its deliberate pandas-boundary policy for duckdb/arrow input; its private engine resolution now uses flavor() and rejects unsupported explicit engines early.
Lift the offset/dictionary/large_utf8-aware Arrow column accessor out of
the qlever ingest into a shared header, triplets/_arrow/string_column.h
(header-only, arrow+std includes only), and make both compiled consumers
use it:
- sparql/_qlever_arrow_parser: private Column struct replaced by the
shared triplets_arrow::StringColumn
- export/cimxml_cython_pugixml.pyx: drops the hard arrow::StringArray
cast (32-bit pointer arithmetic) — now accepts utf8, large_utf8 and
dictionary-encoded string columns via zero-copy string_views
On top of that, CIM XML export consumes polars input natively with the
cython engine: per-instance splitting runs in the input's own flavor
(polars partition_by / pandas groupby), polars columns reach the extension
as Arrow large_utf8/dictionary with no pandas hop, and output is
byte-identical across input flavors. Already-string pandas columns
(arrow-backed string, dictionary, categorical-of-strings) pass through
undecoded; non-string columns keep the legacy astype("string[pyarrow]")
text formatting. python_lxml stays pandas-only (auto-picked for
datatypes=True).
max_workers with polars frames uses a spawn multiprocessing context —
polars is incompatible with fork (rayon pool locks held in the child).
Build: triplets/_arrow added to include dirs (setup.py,
setup_cython_parser.py, setup_qlever.py, header in depends= and sdist).
A NOTE(string_view) slot marks where the string_view branch lands next.
…gelog - docs/development.md: the engine model rule (auto-pick fastest engine only for flavor-independent results; frame ops bind to the input flavor), the registry table per subsystem, and the 'dispatch goes through EngineRegistry' contributor rule - docs/exports.md: polars-native cimxml path, accepted arrow layouts - README: engines()/set_engine() usage - API reference: tools.duckdb_engine added - CHANGELOG: Added/Changed/Fixed entries; TODO: accessor item done
The ID and VALUE columns' Arrow layout is now selectable: "utf8" (32-bit offsets, the stable default), "large_utf8" (64-bit), "string_view" (polars'/duckdb's native 16-byte view layout, adopted zero-copy; needs pyarrow >= 16). "auto" picks the layout the return_type adopts zero-copy: string_view for polars, utf8 otherwise. KEY and INSTANCE_ID stay dictionary-encoded regardless — consumers use the indices. - cython engine: StringColBuilder, a layout-selecting wrapper over arrow::StringBuilder / LargeStringBuilder / StringViewBuilder — the layout branch is constant per parse, so the hot loop cost is unchanged; guarded by ARROW_VERSION_MAJOR >= 16 with a clear runtime error below it - python_lxml_arrow: layouts applied by a single cast on the combined table at finalize (noise on the 1.4s lxml path) - shared accessor (triplets/_arrow/string_column.h): string_view branch added, so cimxml export and qlever ingest accept every layout - empty parses carry the requested layout; unknown names raise ValueError - parity tests: every layout x arrow engine against the lxml reference
…nknown kwargs The unused **kwargs on parse() swallowed typos silently — a misspelled option looked like it worked while doing nothing (it also invalidated a benchmark run by ignoring string_type= on an older build). Unknown keyword arguments now raise TypeError. Measured on RealGrid (1.14M rows, interleaved min-of-7): parse→arrow identical across layouts (the StringColBuilder facade costs nothing); parse→polars ~2-4% faster with string_view — the polars import is bounded by the dictionary→Categorical conversion of KEY/INSTANCE_ID (~11 ms/col), noted in docs/parsers.md as the next lever.
…or; honest input-policy report - Registry kinds carry a role prefix when format-specific, so "cimxml" is never ambiguous between the parser and the exporter: parser_cimxml, exporter_cimxml, exporter_nquads, exporter_csv (sparql/validation/tools unchanged). set_engine kwargs and engines() keys follow. The names had not shipped in any release. - is_polars is removed entirely: flavor(data) is the single detection idiom at every boundary (export, sparql engines, cimxml_utils, cgmes_tools) — no per-flavor boolean helpers. The nquads input conversion also generalizes: any flavor mismatching the resolved engine converts (arrow input with the pandas engine previously passed through raw). - policy="input" kinds (tools, exporter_csv) report engine=None, source="input" in engines() instead of pretending an auto pick exists — the input object's flavor decides per call. - docs: engine-model table updated; cgmes_tools documented as deliberately registry-free (pandas boundary for in-place VALUE mutation semantics).
…df(append=) read_rdf previously materialized the ENTIRE dataset as one in-memory Arrow table before CREATE OR REPLACE, and a second load silently replaced the first. Now: - parser.iter_all_xml(): lazy generator behind find_all_xml — zip members are read into memory only when yielded (one at a time), and zip handles this module opens are actually closed (fd-leak fix) - parser.parse_batches(paths, engine=, ...) -> pyarrow.RecordBatchReader: one RecordBatch per XML file, produced as consumed; fixed all-utf8 ID/KEY/VALUE/INSTANCE_ID schema (no dictionary encoding — per-file dictionaries differ, database consumers re-encode internally); requires an arrow engine, no silent pandas fallback - con.read_rdf(paths, append=False): batches stream straight into DuckDB via the registered reader. append=True INSERTs into the existing table (created if missing); default replaces. Empty input creates the standard 4-column table. One transactional statement per load — a mid-stream parse failure leaves the previous table intact. string_type / categorical_columns no longer apply on this path (they were arrow-return knobs; passing them now raises TypeError).
The per-connection table/schema defaults lived only in a process-local WeakKeyDictionary — reopening a persisted file forgot them. Now explicit configuration (connect(table=/schema=), set_triplets_table, read_rdf with table=/schema=) is also written to a tiny main."_triplets_config" key/value table inside the database, and _get_table resolves lazily: in-process config → DB-stored config → package defaults. Cursors and duplicates share the database, so they resolve the same config with no special-casing. - _install_connect uses _UNSET sentinels: a bare duckdb.connect() no longer touches the config at all (and never writes — :memory: stays clean) - read-only connections resolve stored config; set_triplets_table on them updates the in-process config only - ATTACHed extra catalogs are documented as out of scope (config lives in the default catalog)
set_value_at_key(_and_id) → UPDATE; remove_triplets_from_triplets → DELETE WHERE EXISTS; _apply_update → UPDATE ... FROM then INSERT ... WHERE NOT EXISTS (UPDATE first so updated rows are never re-inserted; MERGE rejected as more machinery than two plain statements). Fixes a real bug: the old CREATE OR REPLACE projections hard-coded ID/KEY/VALUE/INSTANCE_ID, silently dropping any extra user columns on the first mutation. Extra columns now survive (INSERTed rows get NULL extras). Also: mutations no longer copy the whole table (works at any size) and rowids/load order stay stable, so the tableview/reference "first value by load order" picks are preserved across mutations instead of re-normalized.
… the schema rowid only exists on base tables — tableviews, references_* and content_hash(order_sensitive=True) raised BinderException whenever table= pointed at a VIEW or a registered frame (the shacl_duckdb pattern). Now: - _ord_expr picks the load-order expression per target: rowid on base tables (probed via duckdb_tables()), row_number() OVER () otherwise — order then follows scan order, affecting only tie-breaks among duplicate (ID, KEY) rows. _table_parts extracted; _resolve_table is a one-liner. - content_hash(order_sensitive=True) on a non-rowid target raises a clear ValueError instead of hashing a meaningless arbitrary order; the default order-invariant hash works on any target. - _create_view qualifies the view with the resolved schema, so tableview views land next to the data (previously always the default schema, even with connect(schema="cim")). Data-derived default view names are kept (view_name= is the escape hatch; collision surface documented).
…olars type/key/id_tableview and triplets_to_tableviews gain multivalue= (a key holding several values renders the literal ['a', 'b'] text, single values stay bare — byte-identical to the polars encoding, element order = load order) and string_to_number= (accepted for signature parity; True raises ValueError — tableview columns are VARCHAR, TRY_CAST left as a follow-up). tableview_to_triplets(multivalue=True) decodes bracketed cells (split + trim per element + UNNEST), mirroring the polars decoder incl. its embedded-comma/quote limitation. Parity harness: duckdb no longer skips string_to_number kwargs; new multivalue encode + round-trip specs run across all three engines on a synthetic multi-valued frame (Svedala has no genuine multivalues). TODO multivalue item resolved; deferred chunked-export design recorded.
…fd leaks 1,593 warnings → 0. The bulk was one rdflib 7.6.0 upstream bug: its own inherited ConjunctiveGraph/Graph code (and pyshacl's graph cloning) calls its own deprecated Dataset.default_context/.identifier on every parse, match and hash — filtered per-module in pyproject with a pointer to drop the ignores when rdflib migrates its internals. always::ResourceWarning is now on so real leaks stay visible: - test_path_handling built ALL parametrized sources eagerly, leaking an open file handle in every run; sources are now lazy and closed - the tools benchmark exercised a legacy alias 119× (pytest-benchmark amplification) — switched to triplets.tools.filter_triplets_by_type; the deliberate shim-net tests keep their aliases under one module-scoped filterwarnings mark stating the intent - CHANGELOG entries for the duckdb phases; README duckdb section shows append= and config persistence
Review of the SHACL constraint IR found four concrete defects, all fixed: - shacl_report._COMPONENT_MAP was missing MinExclusive/MaxExclusive/ LessThanOrEquals — pyshacl reports carried VIOLATION_TYPE "sh:MinExclusiveConstraintComponent" where the vectorized engines emit "sh:minExclusive" (vocabulary drift for 3 of 24 components) - sh:or/sh:and/sh:not recursion threaded the visited set without ever adding to it — mutually referencing shapes recursed to RecursionError (sh:node already had the guard); _shape_rows now guards all four - FALLBACK_COMPONENTS moves to shacl_ir (the shared contract's home) — shacl_duckdb no longer imports shacl_polars for a 5-string constant (engine="duckdb" without polars installed used to fail at import); shacl_polars re-exports the name - new test_component_registries_agree pins the 8 stringly-typed component registries together (pandas complete; polars/duckdb + fallback cover everything; pyshacl vocabulary maps onto the same short names), and a cyclic-sh:or fixture pins the new guard docs/validation.md gains the full IR structural reference: field table, exact params shapes per component (incl. the sh:node/sh:sparql dict keys), compile-time behaviors, and the per-engine component coverage matrix. Also fixes two doc drifts (lexical dedup columns, stale engine signature).
A docs-vs-code audit found 20 findings; all actionable ones fixed: Code-level: - pyproject: the duckdb extra now pulls pyarrow — con.read_rdf streams through parse_batches, which requires an arrow parser engine, so pip install triplets[duckdb] alone could no longer load data - cimxml_pugixml._string_like accepts string_view — the shared accessor and extension already handled it, but the Python gate re-encoded string_view columns through a cast/astype copy - engines() docstring matched the pre-rename behavior (input-policy rows now correctly described as engine=None/source="input"); read_rdf docstring states the sequential/arrow-only contract and the append-path transactionality nuance; two module docstrings still claimed a literal "triplets" table (config resolution shipped weeks ago) Docs: - development.md: set_engine example used the removed 'parser' kind; to_return_type added to the conversion table - parsers.md: parse_batches/iter_all_xml section (the streaming ingest path was absent from the guide) - exports.md: duckdb streaming/append + export memory-spike notes, datatypes in the shared engine signature, DuckDB-native arrow row, string_view in the accessor line, _split_instances in the call sequence - testing.md: the zero-warning policy (always::ResourceWarning, rdflib ignores) and both markers documented - README: parser speed claims aligned with the committed benchmarks (~10x / measured 9.4x, tableview 15ms), unsupported duckdb parse cell replaced with a pointer to the streaming section - API reference: the top-level package page (engines/set_engine/caches) joined the toctree; parser.nquads, export.cimxml_pugixml/cimxml_utils/ nquads_utils autodoc'd - CHANGELOG: read_rdf capability changes (max_workers/non-arrow engine/ string_type no longer apply), cgmes explicit-engine ValueError, string_view in the accessor entry; TODO header/stale items refreshed
…kers restored The streaming ingest was scoped sequential in v1, which dropped the max_workers parallelism con.read_rdf used to have (default None was always sequential; only explicit max_workers callers lost capability). The proper fix keeps both properties: a bounded, in-order prefetch window submits up to max_workers files ahead on a thread pool and yields batches in file order, so multi-file ingest parallelizes while memory stays bounded by max_workers+1 batches. Order preserved by construction; verified by test. con.read_rdf(paths, max_workers=N) works again.
con.export_to_nquads/csv/cimxml/excel used to pull the whole table into a pandas DataFrame before exporting. duckdb's native arrow result is ~4x cheaper (measured on RealGrid 1.14M rows: .df() 383 ms vs to_arrow_table() 101 ms), and the consumers adopt arrow near zero-copy — the nquads polars engine converts arrow in ~9 ms where the old path paid pandas→polars again (17 ms on top of the 383). cimxml gets arrow-backed strings that flow into the compiled extension with no re-encode. Enabler: the exporters now accept pyarrow input directly — _check_columns is flavor-aware (pa.Table.column_names), export_to_csv rides the pandas engine via near-zero-copy ArrowDtype conversion, export_to_cimxml converts arrow input the same way, and export_to_arrow reuses the unified check. Parity tests gained arrow-input variants for nquads/csv/cimxml, plus a connection-vs-frame export equivalence test. Exports remain whole-table in-memory; chunked (larger-than-RAM) export is still the recorded TODO design.
…treaming) con.export_to_nquads now streams: duckdb's record-batch reader produces ~1M-row batches that the polars nquads formatter writes into one open handle, one batch at a time — the whole-table materialization is gone from the export path. N-Quads is the natural first format: every output line is row-local (no ORDER BY needed — a sort would buffer inside duckdb; it is only required for grouped formats like a future streaming cimxml). - nquads_polars: the expression plan extracted into _quads(); new write_nquads_batches(reader, handle, rdf_map) runs the same plan per batch - export_to_nquads accepts a pyarrow.RecordBatchReader directly (polars engine required — clear error otherwise); _check_columns reads .schema.names for any pyarrow input - _accessor: the connection's nquads export streams via a cursor (a streaming result pins its connection) when polars is available; other formats keep the whole-table native-arrow fetch Measured (file-backed DB, duckdb memory_limit=300MB, 100k-row batches): peak-RSS delta stays FLAT at ~350-400MB while the dataset grows 1.1M → 2.3M → 4.6M rows (whole-table path: +1.2GB → +2.1GB → +3.3GB, linear), at identical speed (stream 870ms vs whole 852ms at 2.3M rows, 352MB output). Line order differs from table order (documented; N-Quads consumers are order-independent, parity canonicalizes). Left for this branch's follow-up: streaming cimxml (ORDER BY INSTANCE_ID + per-instance generator into the existing packaging loops).
…n in TODO The explore/duckdb-chunked-export branch is merged: CHANGELOG gains the out-of-core export entry (flat ~350-400MB peak from 1.1M to 4.6M rows vs linear before, same speed), exports.md reflects that nquads streams while the other formats remain whole-table, and TODO.md splits the chunked-export item: nquads done; cimxml streaming (ORDER BY INSTANCE_ID + instance generator into the existing packaging), and an optional per-batch pandas fallback writer, remain open.
- con.export_to_nquads: stream only when the nquads registry resolves to polars, so an explicit engine="pandas" (or set_engine override) falls back to the whole-table arrow path instead of raising; regression test. - con.read_rdf: unregister the _arrow_import reader in a finally block so a mid-stream parse failure cannot leave it registered on the connection. - tools dispatch: report the real input flavor (pyarrow Table) in the engine/input mismatch TypeError instead of calling it a pandas DataFrame. - to_arrow: document that pyarrow input passes through as-is (a RecordBatch is not upgraded to a Table).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
24 commits unifying engine selection and hardening the DuckDB flavor, following the engine model rule (auto-pick the fastest engine only for flavor-independent results; frame-in/frame-out ops always run in the input's engine).
Engine architecture
EngineRegistrymechanism everywhere (parser, sparql, validation, nquads, cimxml, csv, tools dispatch): eagerfind_specavailability probing at import, lazy module import on first use.triplets.engines()report andset_engine()override; role-prefixed registry kinds (parser_cimxml,exporter_nquads, ...).flavor()as the single input-flavor detector; shared conversion choke point in_engine_detect.Parser
parse(string_type=...): selectable Arrow layout for ID/VALUE (utf8/large_utf8/string_view),"auto"picks what the return type adopts zero-copy; cython engine builds the layout natively.parse_batches()with bounded in-order prefetch (max_workers);parse()rejects unknown kwargs.DuckDB
con.read_rdf(..., append=)fed byparse_batches(never materialized in Python); per-connection table/schema defaults persisted in the database (closes Make duckdb table/schema name configurable #85).to_arrow_readerstreaming (flat memory profile 1.1M -> 4.6M rows).Export
triplets/_arrow/string_column.h); polars frames export CIM XML without a pandas hop, byte-identical output across input flavors (closes Investigate possibility to align pandas and polars string arrow data type #34 for the export path).Quality
Closes #85. Addresses #34.