Skip to content

Latest commit

 

History

History
65 lines (53 loc) · 7.79 KB

ROADMAP.md

File metadata and controls

65 lines (53 loc) · 7.79 KB

adbcBridge roadmap

adbcBridge turns legacy database connectivity layers into Arrow-native ADBC drivers. ODBC is the first bridge; the name leaves room for the others.

1. ODBC bridge (libadbc_driver_odbc) — now

Milestone Status
Query, types, NULLs, Unicode, parameters, bulk ingest, metadata, errors done
Compatibility matrix: 53 databases verified on Linux by one workload (docs/COMPATIBILITY.md) done
Clients measured against all 53: Python, Rust, Go, Java, C# (bench/LANGUAGE_BENCHMARKS.md — 261 of 265 Linux cells); R smoke-tested done
Array-bound and multi-row bulk ingest, probed per driver, default on done
Plug-and-play install: install.sh (Linux, macOS), driver manifest and driver="odbc" by name (Linux, macOS, Windows — there cmake --install writes the manifest) done
macOS verified on a real machine: SQLite, PostgreSQL 15, SQL Server 2022 (bench/BENCHMARKS-macos.md) done
Windows: first build 2026-08-24 (ten defects across driver, tests and harnesses, three of them the ANSI code page in three disguises); compat campaign on an 8 GB laptop, then the full 46-entry re-measure on a 32 GB machine on 2026-08-25 (45 pass; MySQL Connector/ODBC 26.7.1 retires the astral class) — bench/BENCHMARKS-windows.md done
macOS: 53 of 53 results (45 pass, 8 driver/server unavailable); a bridge built against iODBC is a supported configuration for vendor drivers that only ship iODBC builds (MySQL Connector/ODBC for macOS) — three width bugs found and fixed on the way (docs/TROUBLESHOOTING.md) done
2026-09-03 batch: SingleStore, SAP HANA Express, Exasol, Altibase, Kinetica, Actian Ingres and IBM Db2 for i (PUB400.COM) verified on Linux — 53 entries; macOS and Windows cells measured the same week (45 and 48 pass; every other cell names its reason in docs/COMPATIBILITY.md) done
Windows: per-connection narrow-text route for ANSI-only drivers whose narrow path is UTF-8 (narrow_sql + narrow_params + wchar_as_utf8, keyed on the driver, no global state) — Apache Ignite passes on Windows, measured on the second Windows machine done
Windows, to verify on the next Windows session (reported by the second Windows campaign, not reproduced on a machine still available): (1) whether cmake --install under MSVC can write the manifest key windows_amd64_mingw (CMake's MINGW is only set for a GNU toolchain; the campaign's own build log recorded windows_amd64); (2) the Go binding's SQLite test leaves the database file open after Close so t.TempDir() cleanup fails — the drivermgr Close calls AdbcConnectionRelease/AdbcDatabaseRelease synchronously, so the open handle is either a driver-side pool or a missing release in the test's reader path next
Connection-level reader options (adbc.odbc.batch_size, sqllen_32bit) set before AdbcConnectionInit are discarded — OdbcConnectionInit copies the database defaults over them; re-apply the recorded pre-options after the copy next
Windows: prefetch pipeline and parallel ingest on Win32 threads (SRWLOCK, CONDITION_VARIABLE, _beginthreadex), the same options and behaviour as Linux and macOS done (0.1.4)
Driver bootstrap: install.sh --drivers installs the open-licence ODBC drivers a first run needs (sqliteodbc, psqlodbc, MariaDB Connector/ODBC through apt, dnf or Homebrew; clickhouse-odbc from its pinned, checksummed release tarball) and prints what landed where with a connection string each; vendor drivers (Oracle, Db2, SQL Server, Snowflake…) stay the user's download — their licences do not allow redistribution, and Windows already ships the SQL Server driver done (0.1.4), Linux and macOS
Driver bootstrap on Windows: a PowerShell counterpart that runs the vendors' MSI installers (psqlodbc, sqliteodbc, MariaDB Connector/ODBC, clickhouse-odbc) next
ADBC Driver Foundry validation suite (tests/validation/RESULTS.md): PostgreSQL 293/327 pass with the other 34 declared backend limitations (2026-09-15; was 265 on 2026-09-06), SQLite 212/327 with the remainder being precision SQLiteODBC does not report; baseline findings D1–D9, D11, D14 and the PostgreSQL findings P1–P4 (TIME/TIMESTAMP precision and zone fidelity), P7 (db_schema option) and P8 (schema catalog fill) fixed; tests/test_postgres.py pins them in CI done — released in 0.1.2 (2026-09-15)
Trust artefacts: security policy with private vulnerability reporting and a stated support window (SECURITY.md); Dependabot on every manifest; SHA256SUMS and its signature (SHA256SUMS.asc) on v0.1.0; every later release signed, attested (GitHub build provenance, gh attestation verify) and shipped with an SPDX SBOM by the release workflow done
Foundry listing (adbc-drivers.org) next
Release workflow: Linux x86_64/aarch64 (manylinux_2_28), macOS arm64 and Windows x64 libraries + Python wheels built on tag, attached to the GitHub Release (.github/workflows/release.yml) done — v0.1.0 on 2026-08-25: 4 libraries, 4 wheels + sdist, crate, nupkg, jar
PyPI via trusted publishing (.github/workflows/publish-pypi.yml, dispatched by hand with the tag once the Release workflow has published it); NuGet the same way (publish-nuget.yml); crates.io from a checkout of the tag done (0.1.0 on PyPI, crates.io and nuget.org, 2026-08-26)
Language packages: Python wheel (python/), Rust crate (rust/), NuGet (csharp/), Maven (java/), Go module (go/) — built and attached to every release (the crate tested there, the others by their own suites under tests/); PyPI, crates.io, nuget.org and Maven Central (org.adbcbridge:adbcbridge, published 2026-09-04 through the Central Portal, java/PUBLISHING.md) carry 0.1.0 done
Databases that need vendor downloads behind a login: Teradata, Snowflake, Databricks/Spark (Simba), Hive help wanted

2. QuestDB bridge (libadbc_driver_questdb) — now

QuestDB 10 moved its preferred wire to QWP, its own columnar protocol over WebSocket, and asked for ADBC to arrive that way rather than over the PostgreSQL wire (questdb/roadmap#126). This driver is an ADBC 1.1.0 shell over QuestDB's official C client: results are the client's Arrow batches passed through, ingest is Arrow batches the other way (docs/getting-started/questdb-qwp.md).

Milestone Status
Driver: connect (basic auth, token, TLS), queries with bind parameters, ExecuteUpdate row counts, bulk ingest in the four ADBC modes with CREATE TABLE from the Arrow schema, catalog (GetObjects, GetTableSchema, GetTableTypes), cancel, per-query timeout; questdb/ (#165) done 0.1.4 (Linux CI against QuestDB 10.1.0)
macOS and Windows builds in CI next
Packaging: PyPI wheel bundling the library, driver manifest next
Measured against the ODBC route and the native PostgreSQL ADBC driver on the same workload, published as a note next

3. JDBC bridge (libadbc_driver_jdbc) — next

Same idea for the JDBC universe: load a JVM in-process (JNI), drive any JDBC driver, and return Arrow batches through the ADBC ABI — so Python, R, Go, Rust and C# can use JDBC-only drivers without a Java application. Apache Arrow already has a Java-side JDBC adapter; this is the C-side one.

4. OLE DB bridge — later

Windows-only; covers the few remaining sources that have OLE DB providers but no ODBC driver.

Non-goals

  • Replacing native ADBC drivers where they exist (PostgreSQL, Snowflake, BigQuery, …): those will always be faster. adbcBridge is for everything else, and for the long tail of enterprise databases that will never get a native driver.