Skip to content

fix(storage): close the sqlite connection without racing the consumer thread - #480

Merged
gemone merged 1 commit into
mainfrom
fix/storage-close-consumer-race
Aug 9, 2026
Merged

gemone merged 1 commit into
mainfrom
fix/storage-close-consumer-race

Conversation

@gemone

@gemone gemone commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes the native access violation in test_storage_does_not_deadlock_under_threaded_contention seen on the Windows CI runner (PR #479, re-run passed but the crash is a real race, not a one-off).

Root cause. Session.close() joins the consumer thread with a bounded timeout (_CLOSE_JOIN_TIMEOUT + grace) then closes storage unconditionally — the comment at the close site explicitly says so ("storage is closed below regardless of whether the consumer finished"). When the consumer is slow (WAL fsync, write_atomic retry backoffs of 0.1/0.5/2.0s on a loaded runner), it is still executing write_atomic on the shared sqlite3.Connection when the caller thread runs self._conn.close(). Closing a connection under an in-flight statement is a use-after-close: native access violation on Windows (the #479 trace — consumer thread faulting inside write_atomic), SIGSEGV on Linux (reproduced: EXIT=139).

The old design contract — "temporal partitioning, no mutex" (module docstring) — assumed close joins before touching the connection again, which is exactly what the bounded-timeout close stopped guaranteeing.

Fix. Storage now serialises consumer-facing DML and close() behind a threading.Lock + _closed flag:

  • write_atomic / archive_skill_blocks acquire the lock; if _closed is set (close won the race), the residual batch is dropped — the documented "loss ceiling = kill -9" path in Session.close, made crash-free.
  • close() acquires the lock, so it waits out the in-flight write before closing; idempotent, safe to call twice (existing contract preserved).
  • Reads stay temporally partitioned as before (main thread during load, before the consumer starts) — unchanged contract; the lock is uncontended in the single-writer steady state and only contends at the close boundary.

promote_session / archive helpers use their own connections (verified) — no other self._conn user can race close.

Tests / CI

New regression test test_close_without_waiting_for_consumer_drops_residual_without_crashing: zeroes the join timeouts + enqueues 1000 writes so Session.close() runs while the consumer is deterministically mid-drain. Without the fix this segfaults the whole pytest process (verified EXIT=139); with the fix it passes (consumer drops residuals, exits cleanly). Also covers the exact production timeout path that crashed on Windows.

Local gates: uv run ruff check ✓, uv run ty check ✓, uv run pytest tests/test_session.py ✓ (49 pass). Full suite ✓ except the pre-existing environmental test_startup_latency.py flake (fails on clean main too — machine-load, unrelated).

… thread

Session.close() joins the consumer with a bounded timeout and closes
storage unconditionally afterwards, so a slow consumer (WAL fsync,
write-retry backoffs) can still be inside write_atomic when the caller
thread closes the shared connection. That is a use-after-close: native
access violation on the Windows CI runner (reproduced as SIGSEGV on
Linux). Storage now serialises consumer-facing DML and close() behind a
threading.Lock + _closed flag: close() waits out the in-flight write,
then any residual batch the consumer drains afterwards is dropped
instead of executing on the closed connection.
@gemone
gemone merged commit 7a38eea into main Aug 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant