[SLOP(claude-opus-4-8-high)] perf(universaldb): batch leader apply and fold follower commit round-trips

[MasterPtato]

No description provided.

[MasterPtato]

Stack for rivet-dev/rivet

Current stack:

• [SLOP(claude-opus-4-8-high)] perf(universaldb): batch leader apply and fold follower commit round-trips #5342 vtrtqpyp · 2026-06-26 02:20:21 👈
• [slopfix] docs(self-hosting): promote postgres from experimental to recommended OSS multi-node backend #5341 zsmysquk · 2026-06-26 02:20:19
• [slopfix] fix(self-host): repair dev compose generation, kitchen-sink serverful runner, and postgres/udb bootstrap #5340 wpvkrymu · 2026-06-25 22:53:05
• [SLOP(claude-opus-4-8-high)] feat(universaldb): graceful postgres leader handoff on shutdown #5337 mvxpzxsl · 2026-06-25 20:34:04
• [slopfix] test(universaldb): postgres leader failover + abort resolver task on driver drop #5336 tvmntvqo · 2026-06-25 20:34:03
• [SLOP(claude-opus-4-8-high)] feat(ups): table-backed postgres transport with coalesced doorbell #5335 xutxrzrv · 2026-06-25 20:34:01
• [SLOP(claude-opus-4-8-high)] feat(universaldb): postgres leader-resolver driver overhaul #5329 xrrmomwz · 2026-06-25 00:55:07
• [SLOP(claude-opus-4-8)] feat(pegboard-envoy): rate-limit envoy ws ingress and cap get_pages, trim unused metrics #5327 tnzyzqrl · 2026-06-24 20:57:36
• [SLOP(claude-opus-4-8)] feat(util): add rate limiter primitive and ingress throttles for actor create and gateway websocket #5326 pnyswwtm · 2026-06-24 20:57:35
• [SLOP(claude-opus-4-8)] feat(rivetkit-core): add serde duration and size metrics #5322 nqkrytvr · 2026-06-23 22:49:13
• Add observation for serde_bare #5321 oskoutmw · 2026-06-23 20:23:10
• Add observation and ser/de metrics for serde calls #5320 ztnuwvts · 2026-06-23 20:23:09

Dependencies:

• [SLOP(claude-opus-4-8-medium)] fix(rivetkit-napi): reap completed tasks from event-loop JoinSet (#5315) #5314 xxqwtrzp · 2026-06-22 22:42:01
• main

Get stack: forklift get 5342
Push local edits: forklift submit
Merge when ready: forklift merge 5342

[claude]

Review

This PR bundles several well-designed performance improvements to the UniversalDB Postgres driver: combining the follower's INSERT + pg_notify into one autocommit round-trip, resolving commits directly from the NOTIFY payload instead of always polling the DB, splitting lease renewal into its own task so it can't be starved by a long drain, folding all winners' operations in memory and writing them in O(1) bulk SQL statements per batch, and fixing the recovery floor bug (max(durable, seq_high) → durable_version only). The architecture is substantially cleaner after this change.

---

1. Debug log level baked into compose templates (medium)

Files: self-host/compose/template/src/docker-compose.ts:295, self-host/dev-multinode/docker-compose.yml:182, 222, 262

RUST_LOG=universaldb::driver::postgres=debug is added to the shipped templates for all three node services. Under commit load, tracing::debug! fires on every await_result resolution (committed / conflict / backstop) and every batch drain — potentially hundreds of times per second. This creates significant log I/O and CPU overhead for production deployments and appears to be a temporary debugging artifact. Either remove it or gate it on a separate env var before merging.

---

2. .get(1) column index is fragile after combined INSERT+NOTIFY query (low)

File: engine/packages/universaldb/src/driver/postgres/commit.rs

The new query:

SELECT pg_notify($5, id::text), id FROM ins

returns (void, id), and the code calls .get(1) to extract id. This works today because tokio-postgres lazily decodes columns, but the index is non-obvious and silently breaks if the SELECT order is ever changed.

A safer form that puts id at column 0 — so .get(0) is unambiguous — is to move pg_notify into a side-effect CTE:

WITH ins AS (
    INSERT INTO udb_commit_requests (...) VALUES (...) RETURNING id
),
_notify AS (SELECT pg_notify($5, id::text) FROM ins)
SELECT id FROM ins, _notify

or simply reorder to SELECT id, pg_notify($5, id::text) FROM ins and change the call to .get(0).

---

3. atomic_read_keys emits duplicate keys without deduplication (low, efficiency)

File: engine/packages/universaldb/src/driver/postgres/resolver/apply.rs:atomic_read_keys

If multiple winners in a batch have read-modify-write atomic ops on the same key (e.g., two sequential Add ops on a shared counter — a common hot-key pattern), that key is pushed into keys once per winner. WHERE key = ANY($1::bytea[]) handles duplicates correctly, but sends redundant bytes over the wire and may do extra Postgres scan work for large keys vectors.

A simple fix before returning:

keys.sort_unstable();
keys.dedup();

Or collect into a HashSet and convert back. Under a hot-key batch this matters; under typical workloads it's minor.

---

Other observations (no action needed)

• The recovery_floor fix (using only durable_version rather than max(durable, seq_high)) is well-reasoned. The long comment explaining why seq_high is wrong (loser/cold-reject versions race far ahead, causing mass-reject storms) is exactly the kind of non-obvious invariant CLAUDE.md asks you to document.
• The Overlay fold correctly handles all edge cases: set-after-clear-range re-inserts a point write, point writes win over range tombstones on read-through, and the final SQL execution order (range deletes → point deletes → upserts) matches the fold semantics.
• The sibling-task architecture (AbortOnDropHandle for both renew_loop and drain_loop) correctly ensures a hard abort of lead() tears down both tasks, and a crashed drain rolls back cleanly via the pool connection drop.
• The parse_reply function safely returns None for payloads from other waiters on the shared reply channel, with no ID collision risk across i64 values.

Generated with Claude Code