Skip to content

Fix crash on server shutdown racing HTTP workers - #681

Merged
adamant-pwn merged 2 commits into
masterfrom
fix/server-shutdown-race
Oct 28, 2026
Merged

adamant-pwn merged 2 commits into
masterfrom
fix/server-shutdown-race

Conversation

@adamant-pwn

@adamant-pwn adamant-pwn commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Fixes the intermittent test_shutdown_1_term_multiple_threads failure (AssertionError: -11 != 0) on master: master run 40606635554 (b7f51b0f, Protein Debug) and 40110252632 (98291431, Protein Release static noAVX). It is not caused by #671, which only changes tests/test_all.cpp.

Root cause

The SIGTERM/SIGINT handler added in #675 calls server.stop() on the main thread while the HTTP worker threads are still running io_service->run(). stop() closes the acceptor and shuts down every open connection socket. Asio sockets must not be used concurrently, so:

  • an accept handler on a worker re-arms acceptor->async_accept (server_http.hpp:859) while the main thread runs acceptor->close(). epoll_reactor::start_op then dereferences the released descriptor state, which is the SIGSEGV;
  • Connection::close() (socket shutdown/cancel) races with the accept operation assigning that connection's socket (do_assign).

The test sends an incomplete request right before the signal, so a just-accepted connection's handlers are often in flight at that moment.

Fix

The signal handler now only calls io_service->stop(), which is thread-safe. The acceptor and the connections are closed with server.stop() after server_thread.join(), when no worker is left. Shutdown behaviour is otherwise unchanged: in-flight handlers still finish, and an incomplete request still does not keep a worker alive.

A new term_while_connecting case keeps clients connecting (RST on close, capped, so it leaves no TIME_WAIT) while the signal arrives, so accept handlers are in flight during shutdown.

Evidence

  • Standalone harness: the same Simple-Web-Server/asio pins and the same shutdown sequence, with a client connecting during shutdown.
    • Old sequence: ThreadSanitizer reports 5 data races (acceptor close vs async_accept, shutdown vs do_assign, …). AddressSanitizer reports a SEGV in epoll_reactor::start_op ← async_accept ← the accept handler, with both -p 4 and -p 1.
    • New sequence: 0 TSan races; ASan 2000/2000 clean with -p 4 and 2000/2000 with -p 1.
  • End to end, master Release binary at b7f51b0f:
    • The plain test scenario was clean in 300 + 4×150 (pinned to 2 cores) + 400 runs.
    • With clients connecting during the signal it reproduced the CI failure once (rc=-11) in about 250 runs.
    • The crash is too rare on a 32-core workstation for the end-to-end loops to separate the two builds: after switching the storm to RST-closed sockets, both master and the fix were 400/400 clean.
  • Fixed build (Release): repro loop 400/400 plain and 400/400 with connecting clients. test_server_shutdown.py passed 6/6 runs, 7/7 tests each.

Not run locally: the unit tests and the rest of the integration suite (left to CI). No unit test covers server.cpp.

🤖 Generated with Claude Code


Summary by cubic

Fixes the intermittent crash on server shutdown where SIGTERM/SIGINT raced with running HTTP workers. The signal handler previously called server.stop(), which closed the acceptor and all connection sockets from the main thread while accept and read handlers were still running on worker threads, occasionally crashing in asio.

  • The signal handler now only stops the io_service; server.stop() is called after the worker threads have been joined.
  • Adds a term_while_connecting shutdown test that keeps clients connecting while the signal arrives and only sends the signal after the first connection is accepted, so accept handlers are in flight during shutdown.

Shutdown behavior is unchanged otherwise: in-flight handlers still finish and an incomplete request still does not keep a worker alive.

Written for commit f52876b. Summary will update on new commits.

View guided diff Turn on auto-fix

The SIGTERM/SIGINT handler called server.stop() from the main thread
while the HTTP worker threads were still running. stop() closes the
acceptor and every open connection socket, and asio objects must not be
used concurrently: an accept handler re-arming async_accept on a worker
thread could see the acceptor closed underneath it and crash in
epoll_reactor::start_op, and Connection::close() could race with the
accept operation assigning that connection's socket. This surfaced as
an intermittent "-11 != 0" in test_shutdown_1_term_multiple_threads.

Stop only the io_service from the signal handler, which is thread-safe,
and close the acceptor and the connections after the workers have been
joined.

Add a shutdown case that keeps clients connecting while the signal
arrives, so accept handlers are in flight during shutdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread metagraph/integration_tests/test_server_shutdown.py Outdated
Connect until stopped instead of a fixed count, and send the signal only
after a connector has been accepted, so clients are still connecting
when shutdown starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adamant-pwn
adamant-pwn merged commit 490cb66 into master Oct 28, 2026
52 checks passed
@adamant-pwn
adamant-pwn deleted the fix/server-shutdown-race branch October 28, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant