Skip to content

feat: multi-clause Cypher write composition (MERGE chains) via sequential staging#1519

Merged
bplatz merged 9 commits into
mainfrom
feature/cypher-merge-composition
Jul 20, 2026
Merged

feat: multi-clause Cypher write composition (MERGE chains) via sequential staging#1519
bplatz merged 9 commits into
mainfrom
feature/cypher-merge-composition

Conversation

@bplatz

@bplatz bplatz commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Lifts the v1 restriction "MERGE must be the only write clause." A Cypher write statement may now chain write clauses — MERGE chains, CREATE+MERGE mixes, interleaved SET/REMOVE — with each clause observing the writes of the clauses before it and rows piping downstream, publishing as one atomic commit. The standard loading idioms now just work:

UNWIND $rows AS row
MERGE (a:Person {id: row.src})
MERGE (b:Person {id: row.dst})
MERGE (a)-[:KNOWS]->(b)
MATCH (p:Player)
MERGE (t:Team {name: p.team})
MERGE (p)-[:PLAYS_FOR]->(t)

Why a driver, not a rewrite

A single Txn evaluates its whole WHERE — including every MERGE NOT EXISTS guard — against one snapshot horizon, so it cannot express "later clause observes earlier clause" (the second MERGE's guard must see the first's creations). No single-transaction rewriting fixes that. Instead, the statement executes as an ordered sequence of sub-Txns over the same virtual-state machinery behind SPARQL ;-separated updates (D-10), now factored into a reusable SequentialStager: each clause stages against base + prior clauses' flakes, flakes fold last-wins re-stamped to the single final t+1, and one commit publishes — all-or-nothing.

What Cypher adds over SPARQL ;: row threading

Clauses pipe rows, not just graph state. The driver evaluates the read prefix as an ordinary read query into a row table (full read surface: MATCH, UNWIND, WITH incl. aggregation), then seeds each write clause with it — an UnresolvedPattern::Values of new PreBound binding cells, lossless for entities and typed literals. Per MERGE clause:

  1. match probe — seeded VALUES ⋈ pattern against the virtual state;
  2. dedup — absent rows key by their engine-evaluated identity tuple; the first row per distinct key creates (running ON CREATE SET);
  3. re-bind — all rows re-match against the post-create state (Cypher's multi-match row multiplication included);
  4. ON MATCH SET — applies to every non-creating row, including in-batch duplicate keys, which converges exactly on sequential per-row semantics.

CREATE-bound variables thread as reconstructed skolem Sids (VALUES-row ↔ solution order, pinned by test). Computed values in merge keys and SET positions ({name: p.team}, p.age + 1) hoist into engine-evaluated row columns before lowering. The previously-rejected node MERGE under a plain leading MATCH routes here too — one create per distinct key, every row bound, so the old "cartesian footgun" shape now has correct semantics.

Capability notes

  • RETURN on multi-clause writes answers from the final row table with the full read expression surface — matched and created bindings, projections, modifiers — a strict superset of the created-entity-only reconstruction single-clause writes use. Embedded and Bolt return typed rows; HTTP carries the Cypher-JSON envelope back on the receipt. (Raft consensus supports sequential writes but rejects the RETURN-carrying combination pre-submission — its receipt path has no channel for rows.)
  • All three write paths are wired: autocommit, Bolt explicit transactions (statements stage against the private state, so probes see earlier statements in the transaction), and the consensus committers — probes run under the write lock per retry attempt, preserving the existing no-TOCTOU contract, and probe views are policy-wrapped when governance carries policy inputs (same contract as conditional MERGE).
  • In-batch duplicate keys: first row creates + ON CREATE SET, later rows match + ON MATCH SET — matching Neo4j's sequential LOAD CSV behavior. The single-clause UNWIND-batch last-wins dedup is now gated to single-clause statements (it keys on the first node MERGE and would drop legitimate rows of a chain).
  • Deferred with clear errors: DELETE/FOREACH members in a chain, OPTIONAL MATCH / CALL { … } in the read prefix.

Non-impact

  • JSON-LD / Turtle / SPARQL transaction paths untouched; stage() in fluree-db-transact unmodified.
  • stage_transaction_from_txns keeps its single-op fast path; only its N≥2 loop body moved into SequentialStager (its 5 multi-op semantics tests pass unchanged).
  • Single-clause Cypher writes keep their exact plans — detection fires only on statements that previously returned "unsupported" errors.

Testing

13 new API integration tests (it_cypher_seq_writes.rs) + 3 HTTP tests: chain idempotence, endpoint reuse, batch relationship upsert, MATCH…MERGE distinct-key creation, CREATE threading row-exactness, ON CREATE/ON MATCH branch firing across runs and within batches, interleaved SET, RETURN (embedded/Bolt/HTTP), zero-row no-ops, failing-clause atomicity, DELETE rejection, Bolt pipelining with read-your-writes. Full sweep of the six touched crates: 5,244 tests, 0 failures. Clippy all-features clean.

Design doc: docs/design/cypher-sequential-writes.md; Cypher guide updated.

Stacked on #1518 (base branch fix/cypher-kb-feedback).

bplatz added 4 commits July 18, 2026 14:42
…alStager

The SPARQL ;-separated update loop (D-10: N ordered operations, each
observing the previous through a virtual state, one atomic commit) moves
from an inline loop in stage_transaction_from_txns into a reusable
SequentialStager: stage-per-op with last-wins fact folding, simulated
sequential graph-id assignment, namespace-delta union, and a finish()
that layers the merged fold over the original base. The single-op fast
path and the skip-last-apply optimization are unchanged; finish() now
also tolerates a zero-op run (empty staged view) for callers whose
operation count is data-dependent.
Three small mechanisms the sequential write driver threads rows through:

- UnresolvedValue::PreBound(Binding): an already-resolved VALUES cell,
  passed through lower_values_cell verbatim. Never parser-produced —
  bindings extracted from a prior query result re-enter a Txn's WHERE
  losslessly (entities as Sids, typed literals as-is).
- CypherLowerOpts::seeded_vars: columns the caller binds by appending a
  Values row block to the lowered Txn post-hoc. Lowering registers them
  bound (SET/REMOVE target checks pass, templates may reference them)
  and stages the statement as an update. InlineRows vars now register as
  bound too.
- The UNWIND $batch last-wins row dedup now applies only to
  single-write-clause statements: it keys on the first node MERGE, which
  would drop legitimate rows of a multi-clause chain (two rows sharing
  src but differing in dst are distinct work). Multi-clause statements
  own duplicate-key semantics in the driver (first row per key creates,
  later rows match).

Materializer's JoinKeyMode is re-exported for callers that materialize
encoded bindings at extraction boundaries.
Lifts the 'MERGE must be the only write clause' restriction. A statement
may now chain write clauses — MERGE chains, CREATE+MERGE mixes,
interleaved SET/REMOVE — with each clause observing the writes before it
and rows piping downstream, publishing as ONE atomic commit:

  UNWIND $rows AS row
  MERGE (a:Person {id: row.src})
  MERGE (b:Person {id: row.dst})
  MERGE (a)-[:KNOWS]->(b)

The driver (cypher_seq.rs) executes over a SequentialStager: the read
prefix evaluates as a read query into a row table; each write clause
stages one or two seeded sub-Txns against the accumulated virtual state.
A MERGE clause partitions rows by probing (match probe over a seeded
VALUES join), dedups absent rows by their engine-evaluated identity-key
tuple (first row per key creates, running ON CREATE SET), re-binds every
row against the post-create state (Cypher's multi-match multiplication
included), and applies ON MATCH SET to the non-creating rows — exactly
sequential per-row semantics, vectorized. CREATE-bound variables thread
as reconstructed skolem Sids; computed values in merge keys and SET
positions ({name: p.team}, p.age + 1) hoist into engine-evaluated row
columns before lowering.

The previously-rejected node MERGE under a plain leading MATCH routes
here too — one create per distinct absent key, every row bound.

A trailing RETURN is answered from the final row table with the full
read expression surface (matched AND created bindings, modifiers), not
just created-entity reconstruction.

Wiring: WritePlan::Sequential dispatches on all three write paths —
autocommit (transact_cypher*), Bolt explicit transactions (staging
against the private state, so probes see earlier statements), and the
consensus committers via RefTransactBuilder::cypher_sequential (probes
run under the write lock, preserving the no-TOCTOU contract; the retry
loop re-resolves per attempt). The local committer carries the RETURN
envelope on TransactionReceipt::cypher_return; Raft supports sequential
writes but rejects a RETURN-carrying one at the route (its receipt path
has no channel for the rows). Existing single-clause plans — including
the conditional MergeSet/MergePerRowSet shapes — keep their exact paths;
detection fires only on statements that previously errored, and the
JSON-LD / SPARQL transaction paths are untouched.
…n model

Integration matrix (it_cypher_seq_writes + HTTP suite): MERGE chains
create-once and no-op on re-run; chains reuse pre-existing nodes; the
UNWIND batch relationship-upsert idiom dedups shared endpoints and is
idempotent; MATCH…MERGE creates once per distinct computed key
({name: p.team}) and wires every row; CREATE-bound variables thread
row-exactly into later MERGEs (pins VALUES-row ↔ solution order); ON
CREATE/ON MATCH fire per branch across runs and within a batch (first
duplicate-key row creates, the second matches); interleaved SET applies
per clause; trailing RETURN reads matched bindings post-write (embedded,
Bolt typed rows, and HTTP receipt channel); zero-row statements are
clean no-ops; a failing clause commits nothing; DELETE-in-chain errors
clearly; Bolt transactions pipeline sequential statements with
read-your-writes.

Docs: design note (docs/design/cypher-sequential-writes.md), the Cypher
guide's write-surface section (multi-clause composition, per-row MERGE
semantics, RETURN surface), SUMMARY entry.
@bplatz
bplatz requested review from aaj3f, anh-h and zonotope July 18, 2026 19:05
The UNWIND-batch last-wins dedup was gated on write_clauses.len() == 1,
but the upsert idiom MERGE (n {id: row.id}) SET n.name = row.name is two
write clauses ([MERGE, SET]) and stays on the conditional path — the
gate disabled its in-batch dedup (caught by
load_csv_dedups_within_batch_merge_key). Gate on the driver's actual
routing instead: dedup applies to the [MERGE, SET…] family unless a
plain read besides the desugared UNWIND routes it to the driver.

Also drop unnecessary raw-string hashes in the new HTTP tests.

@anh-h anh-h left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! This fixes the limitation I had worked around by splitting complex writes into sequential calls

Base automatically changed from fix/cypher-kb-feedback to main July 20, 2026 19:38
bplatz added 3 commits July 20, 2026 15:59
Preserve anonymous-prefix row multiplicity and WITH * scope, round-trip
path bindings, and evaluate computed MERGE branch updates against the
evolving virtual state.

Add regression coverage and update the sequential-write design notes.
@bplatz
bplatz merged commit 47ce0c8 into main Jul 20, 2026
12 of 13 checks passed
@bplatz
bplatz deleted the feature/cypher-merge-composition branch July 20, 2026 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants