VexDB

English | 中文

VexDB-Lite extends PostgreSQL, DuckDB, and SQLite with a shared graph index, SIMD distance dispatch, and PQ/RaBitQ quantization kernels. The SQLite edition also includes VexFS, so one database can hold regular tables, vector indexes, and database-managed files.

See vexdb_duckdb/README.md for the DuckDB extension docs.
This root README is a project-level overview and build guide.

---

1. Components

1.1 PostgreSQL: vexdb_lite

Current functionality:

• floatvector(N) type
• Distance functions and operators:
  • l2_distance (<->)
  • cosine_distance (<=>)
  • inner_product (use <~> for negative inner product / MIPS)
• Scalar helpers: vector_dims(), vector_norm(), l2_normalize(), vexdb_index_info()
• CREATE INDEX ... USING vexdb_graph
• Index options: m, ef_construction, parallel_workers, quantizer (pq / rabitq), pq_m, and memory_mode
• PQ and RaBitQ code storage; memory_mode='compact' requires quantizer activation rather than silently storing raw index vectors
• Optimizer rewrite into an ANN Index Scan
• Shared-memory vector buffer cache and parallel index build
• Runtime settings: vexdb.ef_search, vexdb.vec_architecture

1.2 DuckDB: vexdb_lite

Current functionality:

• GRAPH_INDEX on FLOAT[N] vector columns
• Distance functions and operators:
  • l2_distance (<->)
  • cosine_distance (<=>)
  • inner_product (use <~> for negative inner product / MIPS)

  Note: <#> is unavailable in DuckDB (# clashes with its comment syntax); use <~> for negative inner product — same meaning as in PG.

• Scalar helpers: vector_dims(), l2_normalize(), vexdb_version(), vexdb_index_info()
• CREATE INDEX ... USING GRAPH_INDEX (vec) with WHERE filtering during queries
• Index options: m, ef_construction, parallel_workers, quantizer (pq / rabitq), and pq_m
• PQ and RaBitQ support memory_mode='compact' for index-side storage
• Optimizer rewrite into VEXDB_INDEX_SCAN
• Vector buffer cache and parallel index build
• Runtime settings: vexdb_ef_search, vexdb_brute_force_threshold, vexdb_pq_search_mode, vexdb_pq_refine_k_factor

1.3 SQLite: unified VexDB-Lite

The unified macOS package installs:

• vexdb, with SQLite 3.45.3 and automatic extension registration;
• GRAPH_INDEX and vector distance functions;
• VexFS files, directories, history, restore, SHA-256, and read-only integrity checks;
• vexdb fs ..., including database-batched grep and an optional FTS5 trigram index, with vexfs ... kept as a compatible shortcut;
• a VexFS FSKit extension for native ls, cat, grep, cp, and mv after enablement.

vexdb agent.db
vexdb agent.db "SELECT vexdb_l2_distance('[1,2]', '[4,6]');"
vexdb fs --db agent.db write /notes/hello.txt
vexdb fs --db agent.db grep -n hello /notes
vexdb fs --db agent.db check
vexdb fs --db agent.db index enable
vexdb fs --db agent.db mount ~/VexDB

The text index is disabled by default, so ordinary writes have no index overhead. Without the index, or when the SQLite host has no FTS5 support, grep falls back to a single-connection database scan.

Build the unified macOS preview with bash agent_files/macos/package_preview.sh.

---

2. Capability Matrix

2.1 PG Extension Comparison (PGVector vs VexDB-Lite vs VexDB)

Category	Feature	Description	PGVector	VexDB-Lite (open-source)	VexDB (commercial)
Graph Index	graph_index	A fully self-developed high-performance graph index that merges the advantages of various graph indexes and works seamlessly across all scenarios.	❌	✅	✅
Distance	Distance function dispatch	Inlined distance functions, compile-time optimized	❌	✅	✅
Cache	vector buffer	General vector cache, all scenarios	❌	✅	✅
Cache	bulk buffer	Full in-memory vector cache for max throughput	❌	❌	✅
Cache	Async I/O cache	Accelerated disk-to-cache reads under memory pressure	❌	❌	✅
Data types	floatvector	Standard float32 vector type	✅	✅	✅
Data types	halfvector	Float16 vector type	✅	🟡	✅
Data types	int8vector	Int8 vector type	❌	🟡	✅
Quantization	PQ quantization	Code compression and optional refinement; speed depends on data, filters, and configuration	❌	✅	✅
Quantization	RaBitQ quantization	Quantized distance estimation and graph search with compact index-side storage	❌	✅	✅
Quantization	Auto quantization	Background auto-enable, supports empty-table index build	❌	❌	✅
Graph index enhancement	Async insert	Fast ingestion for write-heavy workloads	❌	❌	✅
Graph index enhancement	Graph sharding	Large-scale vectors on small-memory machines	❌	❌	✅
Graph index enhancement	Subgraph index build	Continue using memory for index building even in low-memory scenarios to accelerate build speed	❌	❌	✅
HA	Primary-replica HA	Synchronous replication and backup restore	✅	❌	✅
Maintenance	Parallel vacuum	Parallel index cleanup and reclaim	❌	❌	✅

2.2 DuckDB Extension Comparison (DuckDB VSS vs VexDB-Lite)

Category	Feature	Description	DuckDB VSS	VexDB-Lite (vexdb_lite)
Index	Graph index	VSS: HNSW; VexDB: graph_index (self-developed hybrid)	✅	✅
Distance	SIMD dispatch	Inlined distance functions, compile-time optimized	❌	✅
Quantization	PQ	Vector compression for memory-constrained scenarios	❌	✅
Quantization	RaBitQ	Quantized graph search with optional compact index-side storage	❌	✅
Cache	Buffer management	Disk-to-memory vector caching	❌	✅
Maintenance	Index compaction	Reclaim space from soft-deleted entries	✅	❌
Search	Filtered ANN search	Opt-in scalar/graph/PQ integration; exact filtering remains the default	❌	🟡
Persistence	Disk-backed index	Index survives database restart without rebuild	✅†	✅

† VSS persistence is experimental — WAL recovery is not implemented, unexpected shutdowns may cause index corruption. VexDB-Lite persists via DuckDB's standard serialization.

---

✅ Supported · 🟡 Coming soon · ❌ Not included in open-source edition

3. PostgreSQL Syntax Examples

3.1 Install and Create Table

CREATE EXTENSION vexdb_lite;

CREATE TABLE items (
    id  BIGSERIAL PRIMARY KEY,
    vec floatvector(128)
);

INSERT INTO items (vec) VALUES
    ('[0.10, 0.20, 0.30]'),
    ('[0.40, 0.50, 0.60]');

3.2 Build Index

CREATE INDEX idx_items_vec
ON items
USING vexdb_graph (vec floatvector_l2_ops)
WITH (
    m = 16,
    ef_construction = 64
);

3.2.1 PQ Quantized Index (v1)

PQ reduces vector storage by encoding indexed vectors with a trained codebook:

SET maintenance_work_mem = '2GB';   -- required; low memory falls back to a plain graph index

CREATE INDEX idx_pq
ON items
USING vexdb_graph (vec floatvector_l2_ops)
WITH (quantizer = 'pq', pq_m = 4);

Current v1 behavior:

• If there are fewer than 256 training samples, or the memory build budget is too low, PQ falls back to a plain graph index with a NOTICE.
• PQ indexes support post-build INSERT / UPDATE / DELETE; new vectors are encoded with the trained codebook.
• For parallel_workers > 0, PQ training still runs in the leader process; PG parallel workers only participate in the disk-build phase.

Check the active index state:

SELECT indexname, use_pq, pq_m
FROM vexdb_index_info()
WHERE indexname = 'idx_pq';

3.3 ANN Query

SET vexdb.ef_search = 100;
SET enable_seqscan = off;

SELECT id, vec <-> '[0.15, 0.25, 0.35]' AS dist
FROM items
ORDER BY vec <-> '[0.15, 0.25, 0.35]'
LIMIT 10;

3.4 Other Metrics

SELECT id
FROM items
ORDER BY vec <~> '[0.15, 0.25, 0.35]'
LIMIT 10;

SELECT id
FROM items
ORDER BY vec <=> '[0.15, 0.25, 0.35]'
LIMIT 10;

---

4. DuckDB Syntax Examples

4.1 Load Extension

LOAD '/path/to/vexdb_lite.duckdb_extension';
SELECT vexdb_version();

Typical Python usage:

import duckdb

con = duckdb.connect(config={"allow_unsigned_extensions": "true"})
con.execute("LOAD '/path/to/vexdb_lite.duckdb_extension'")

4.2 Create Table and Index

CREATE TABLE items (
    id       INTEGER,
    category VARCHAR,
    vec      FLOAT[128]
);

CREATE INDEX idx_items_vec
ON items
USING GRAPH_INDEX (vec)
WITH (
    metric = 'l2',
    m = 16,
    ef_construction = 64
);

4.3 ANN Query

SET vexdb_ef_search = 100;

SELECT id
FROM items
ORDER BY l2_distance(vec, [0.15, 0.25, 0.35]::FLOAT[3])
LIMIT 10;

4.4 Filtered Index Example

CREATE INDEX idx_items_vec_meta
ON items
USING GRAPH_INDEX (vec);

SELECT id
FROM items
WHERE category = 'book'
ORDER BY l2_distance(vec, [0.15, 0.25, 0.35]::FLOAT[3])
LIMIT 10;

4.5 Other Functions

SELECT inner_product([1.0, 0.0]::FLOAT[2], [0.5, 0.5]::FLOAT[2]);
SELECT cosine_distance([1.0, 0.0]::FLOAT[2], [0.5, 0.5]::FLOAT[2]);
SELECT vector_dims([1.0, 2.0, 3.0]::FLOAT[3]);
SELECT l2_normalize([3.0, 4.0]::FLOAT[2]);
SELECT * FROM vexdb_index_info();

---

5. Build

5.1 Build the PostgreSQL Variant

Dependencies

• PostgreSQL 16 ~ 19 (PG 16/17/18/19 supported; primary validation target is 19devel)
• CMake ≥ 3.14
• C++17 compiler (GCC 9+ or Clang 10+)

Build PostgreSQL (release example)

cd /path/to/postgresql-19-source
./configure \
  --prefix=/opt/postgresql-19rel-install \
  --without-icu \
  --without-readline \
  --without-zlib \
  CFLAGS="-O3 -DNDEBUG"
make -j$(nproc)
make install

Build vexdb_lite

cd /path/to/VexDB-Lite
export PG_CONFIG=/opt/postgresql-19rel-install/bin/pg_config
cmake -S vexdb_pg -B build/pg -DCMAKE_BUILD_TYPE=Release
cmake --build build/pg -j$(nproc)
cmake --install build/pg

PostgreSQL Configuration

At minimum:

shared_preload_libraries = 'vexdb_lite'

Then restart PostgreSQL and run:

CREATE EXTENSION vexdb_lite;

---

5.2 Build the DuckDB Variant

Recommended: use build_duck.sh — it handles DuckDB clone, cmake configuration, compilation, and metadata processing in one command.

bash build_duck.sh setup   # First time: clone DuckDB v1.5.2 and cmake configure
bash build_duck.sh build   # Compile the extension (incremental)

Output: build/duck/v1.5.2/build/extension/vexdb_lite/vexdb_lite.duckdb_extension

Dependencies

• CMake ≥ 3.28, < 4.x
• C++17 compiler (GCC 9+ or Clang 10+)
• Git

Why a shell script and not plain cmake?

DuckDB extensions must be compiled inside DuckDB's source tree — you cannot run cmake -B build vexdb_duckdb/ standalone. build_duck.sh automates:

1. Cloning DuckDB v1.5.2
2. Writing extension_config_local.cmake to register the vexdb_lite extension
3. Running cmake + cmake --build
4. Appending the extension metadata footer (required by DuckDB's release format)

---

6. Running Tests

DuckDB Extension Tests

bash build_duck.sh build          # Build the extension
bash tests/spec/_lib/docker/run_duckdb.sh test  # Run full spec tests locally; Docker is not needed

PostgreSQL Plugin Tests

python3 tests/spec/_lib/render.py --engine pg --out build/spec
BUILDX_EXTRA_ARGS='--build-arg PG_GIT_TAG=REL_19_STABLE' bash tests/spec/_lib/docker/run_pg.sh build
bash tests/spec/_lib/docker/run_pg.sh test      # Requires Docker

Install renderer dependencies with python3 -m pip install -r tests/spec/requirements.txt. SQLite build and tests use bash build_sqlite.sh test. Test files live under tests/spec/. See current status and CI.

---

7. Benchmark Results

Dataset: SIFT-1M 128-dim, m=16, ef_construction=128. Columns: QPS (reads=1) / QPS (reads=16) / Recall@10.

Test environment: Intel Core Ultra 7-265K (20c/20t, 3.9 GHz) / 16 GB DDR5 / x86_64 Linux

7.1 Comparison with pgvector / VSS (x86_64)

ef_search = 50

System	QPS (r=1)	QPS (r=16)	Recall@10
pgvector	507.9	7153.5	96.22%
vexdb_lite (PostgreSQL)	994.7	12084.6	95.97%
vexdb_lite (DuckDB)	717.5	8667.8	95.06%
duckdb-vss	496.1	5360.9	94.07%

ef_search = 100

System	QPS (r=1)	QPS (r=16)	Recall@10
pgvector	313.4	4272.5	98.82%
vexdb_lite (PostgreSQL)	618.5	7883.1	98.62%
vexdb_lite (DuckDB)	547.2	5379.1	98.40%
duckdb-vss	405.2	4433.3	98.04%

ef_search = 200

System	QPS (r=1)	QPS (r=16)	Recall@10
pgvector	193.1	2694.1	99.66%
vexdb_lite (PostgreSQL)	421.3	5038.0	99.58%
vexdb_lite (DuckDB)	383.6	4298.8	99.53%
duckdb-vss	321.9	3809.3	99.42%

---

8. Known Limitations

PostgreSQL

• Supports PostgreSQL 16 ~ 19; primary validation target is PostgreSQL 19

DuckDB

• The compiled extension must match the exact DuckDB runtime version.
• CPU distance dispatch is selected at runtime; inspect vexdb_simd_arch() for the active architecture.

7. Repository Structure

Directory	Description
common/	Shared core: graph index algorithm, SIMD distance dispatch, quantizer (PQ/RaBitQ), template containers
vexdb_pg/	PostgreSQL extension: index AM, build, search, DML, WAL, distance entry
vexdb_duckdb/	DuckDB extension: index lifecycle, optimizer rewrite, distance functions → README
vexdb_sqlite/	SQLite GRAPH_INDEX, distance functions and VexFS SQL
agent_files/	File commands, versions, snapshots and mount adapters
documentation/	Feature docs, build guide and current CI scope
tests/spec/	YAML-based spec tests (shared / pg / duckdb / sqlite / opengauss)
scripts/	Build, release, and packaging scripts
thirdparties/	Vendored dependencies (patched Boost)

---

Community

Channel	Description
GitHub Issues	Bug reports and feature requests
GitHub Discussions	Questions, proposals and general discussion
Discord	Real-time chat and Q&A
WeChat Group	Scan QR code at vexdb.com/community · Chinese community

---

License

MIT License. See LICENSE.