feat: multi-clause Cypher write composition (MERGE chains) via sequential staging#1519
Merged
Conversation
…alStager The SPARQL ;-separated update loop (D-10: N ordered operations, each observing the previous through a virtual state, one atomic commit) moves from an inline loop in stage_transaction_from_txns into a reusable SequentialStager: stage-per-op with last-wins fact folding, simulated sequential graph-id assignment, namespace-delta union, and a finish() that layers the merged fold over the original base. The single-op fast path and the skip-last-apply optimization are unchanged; finish() now also tolerates a zero-op run (empty staged view) for callers whose operation count is data-dependent.
Three small mechanisms the sequential write driver threads rows through: - UnresolvedValue::PreBound(Binding): an already-resolved VALUES cell, passed through lower_values_cell verbatim. Never parser-produced — bindings extracted from a prior query result re-enter a Txn's WHERE losslessly (entities as Sids, typed literals as-is). - CypherLowerOpts::seeded_vars: columns the caller binds by appending a Values row block to the lowered Txn post-hoc. Lowering registers them bound (SET/REMOVE target checks pass, templates may reference them) and stages the statement as an update. InlineRows vars now register as bound too. - The UNWIND $batch last-wins row dedup now applies only to single-write-clause statements: it keys on the first node MERGE, which would drop legitimate rows of a multi-clause chain (two rows sharing src but differing in dst are distinct work). Multi-clause statements own duplicate-key semantics in the driver (first row per key creates, later rows match). Materializer's JoinKeyMode is re-exported for callers that materialize encoded bindings at extraction boundaries.
Lifts the 'MERGE must be the only write clause' restriction. A statement
may now chain write clauses — MERGE chains, CREATE+MERGE mixes,
interleaved SET/REMOVE — with each clause observing the writes before it
and rows piping downstream, publishing as ONE atomic commit:
UNWIND $rows AS row
MERGE (a:Person {id: row.src})
MERGE (b:Person {id: row.dst})
MERGE (a)-[:KNOWS]->(b)
The driver (cypher_seq.rs) executes over a SequentialStager: the read
prefix evaluates as a read query into a row table; each write clause
stages one or two seeded sub-Txns against the accumulated virtual state.
A MERGE clause partitions rows by probing (match probe over a seeded
VALUES join), dedups absent rows by their engine-evaluated identity-key
tuple (first row per key creates, running ON CREATE SET), re-binds every
row against the post-create state (Cypher's multi-match multiplication
included), and applies ON MATCH SET to the non-creating rows — exactly
sequential per-row semantics, vectorized. CREATE-bound variables thread
as reconstructed skolem Sids; computed values in merge keys and SET
positions ({name: p.team}, p.age + 1) hoist into engine-evaluated row
columns before lowering.
The previously-rejected node MERGE under a plain leading MATCH routes
here too — one create per distinct absent key, every row bound.
A trailing RETURN is answered from the final row table with the full
read expression surface (matched AND created bindings, modifiers), not
just created-entity reconstruction.
Wiring: WritePlan::Sequential dispatches on all three write paths —
autocommit (transact_cypher*), Bolt explicit transactions (staging
against the private state, so probes see earlier statements), and the
consensus committers via RefTransactBuilder::cypher_sequential (probes
run under the write lock, preserving the no-TOCTOU contract; the retry
loop re-resolves per attempt). The local committer carries the RETURN
envelope on TransactionReceipt::cypher_return; Raft supports sequential
writes but rejects a RETURN-carrying one at the route (its receipt path
has no channel for the rows). Existing single-clause plans — including
the conditional MergeSet/MergePerRowSet shapes — keep their exact paths;
detection fires only on statements that previously errored, and the
JSON-LD / SPARQL transaction paths are untouched.
…n model
Integration matrix (it_cypher_seq_writes + HTTP suite): MERGE chains
create-once and no-op on re-run; chains reuse pre-existing nodes; the
UNWIND batch relationship-upsert idiom dedups shared endpoints and is
idempotent; MATCH…MERGE creates once per distinct computed key
({name: p.team}) and wires every row; CREATE-bound variables thread
row-exactly into later MERGEs (pins VALUES-row ↔ solution order); ON
CREATE/ON MATCH fire per branch across runs and within a batch (first
duplicate-key row creates, the second matches); interleaved SET applies
per clause; trailing RETURN reads matched bindings post-write (embedded,
Bolt typed rows, and HTTP receipt channel); zero-row statements are
clean no-ops; a failing clause commits nothing; DELETE-in-chain errors
clearly; Bolt transactions pipeline sequential statements with
read-your-writes.
Docs: design note (docs/design/cypher-sequential-writes.md), the Cypher
guide's write-surface section (multi-clause composition, per-row MERGE
semantics, RETURN surface), SUMMARY entry.
The UNWIND-batch last-wins dedup was gated on write_clauses.len() == 1,
but the upsert idiom MERGE (n {id: row.id}) SET n.name = row.name is two
write clauses ([MERGE, SET]) and stays on the conditional path — the
gate disabled its in-batch dedup (caught by
load_csv_dedups_within_batch_merge_key). Gate on the driver's actual
routing instead: dedup applies to the [MERGE, SET…] family unless a
plain read besides the desugared UNWIND routes it to the driver.
Also drop unnecessary raw-string hashes in the new HTTP tests.
anh-h
approved these changes
Jul 19, 2026
anh-h
left a comment
There was a problem hiding this comment.
Great! This fixes the limitation I had worked around by splitting complex writes into sequential calls
Preserve anonymous-prefix row multiplicity and WITH * scope, round-trip path bindings, and evaluate computed MERGE branch updates against the evolving virtual state. Add regression coverage and update the sequential-write design notes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lifts the v1 restriction "MERGE must be the only write clause." A Cypher write statement may now chain write clauses — MERGE chains, CREATE+MERGE mixes, interleaved SET/REMOVE — with each clause observing the writes of the clauses before it and rows piping downstream, publishing as one atomic commit. The standard loading idioms now just work:
Why a driver, not a rewrite
A single
Txnevaluates its whole WHERE — including every MERGENOT EXISTSguard — against one snapshot horizon, so it cannot express "later clause observes earlier clause" (the second MERGE's guard must see the first's creations). No single-transaction rewriting fixes that. Instead, the statement executes as an ordered sequence of sub-Txns over the same virtual-state machinery behind SPARQL;-separated updates (D-10), now factored into a reusableSequentialStager: each clause stages against base + prior clauses' flakes, flakes fold last-wins re-stamped to the single finalt+1, and one commit publishes — all-or-nothing.What Cypher adds over SPARQL
;: row threadingClauses pipe rows, not just graph state. The driver evaluates the read prefix as an ordinary read query into a row table (full read surface:
MATCH,UNWIND,WITHincl. aggregation), then seeds each write clause with it — anUnresolvedPattern::Valuesof newPreBoundbinding cells, lossless for entities and typed literals. Per MERGE clause:VALUES ⋈ patternagainst the virtual state;ON CREATE SET);ON MATCH SET— applies to every non-creating row, including in-batch duplicate keys, which converges exactly on sequential per-row semantics.CREATE-bound variables thread as reconstructed skolem Sids (VALUES-row ↔ solution order, pinned by test). Computed values in merge keys and SET positions (
{name: p.team},p.age + 1) hoist into engine-evaluated row columns before lowering. The previously-rejected node MERGE under a plain leading MATCH routes here too — one create per distinct key, every row bound, so the old "cartesian footgun" shape now has correct semantics.Capability notes
ON CREATE SET, later rows match +ON MATCH SET— matching Neo4j's sequential LOAD CSV behavior. The single-clause UNWIND-batch last-wins dedup is now gated to single-clause statements (it keys on the first node MERGE and would drop legitimate rows of a chain).DELETE/FOREACHmembers in a chain,OPTIONAL MATCH/CALL { … }in the read prefix.Non-impact
stage()in fluree-db-transact unmodified.stage_transaction_from_txnskeeps its single-op fast path; only its N≥2 loop body moved intoSequentialStager(its 5 multi-op semantics tests pass unchanged).Testing
13 new API integration tests (
it_cypher_seq_writes.rs) + 3 HTTP tests: chain idempotence, endpoint reuse, batch relationship upsert, MATCH…MERGE distinct-key creation, CREATE threading row-exactness, ON CREATE/ON MATCH branch firing across runs and within batches, interleaved SET, RETURN (embedded/Bolt/HTTP), zero-row no-ops, failing-clause atomicity, DELETE rejection, Bolt pipelining with read-your-writes. Full sweep of the six touched crates: 5,244 tests, 0 failures. Clippy all-features clean.Design doc:
docs/design/cypher-sequential-writes.md; Cypher guide updated.Stacked on #1518 (base branch
fix/cypher-kb-feedback).