fix: prevent connection sends from racing close - #1630
Conversation
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #1630 +/- ##
==========================================
+ Coverage 12.54% 12.98% +0.44%
==========================================
Files 96 96
Lines 14526 14560 +34
==========================================
+ Hits 1822 1891 +69
+ Misses 12526 12492 -34
+ Partials 178 177 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
✅ Looks Good - Code looks good
Reviewed the connection-close race fix across the core lifecycle gate and all five endpoint connection routines. The concurrency design is sound and the change is well-scoped.
Correctness of the lifecycle gate (nhp/core/udpconn.go)
- Send-admission vs. close is properly serialized.
channelSendMuguards bothbeginChannelSend'schannelSendWg.Add(1)andClose'sclosed.CompareAndSwap(false, true). OnceClosesetsclosed=trueunder the lock, no furtherAddcan happen, sochannelSendWg.Wait()(invoked after the unlock) never races a positiveAdd— this is correctsync.WaitGroupusage and avoids the "Add called concurrently with Wait" panic. - No deadlock.
close(c.StopSignal)runs beforeWait(), so any already-admitted sender blocked in itsselectunblocks via the<-c.StopSignalbranch, releases its packet, and callsDone().TestConnectionDataCloseUnblocksPendingSendscovers exactly this. - Keeping channels non-nil after close is the right call. Removing the old
c.SendQueue = nil(etc.) assignments is what makes the new_, ok := <-chreceive checks fire (ok=false) so the routines exit cleanly. Nil-ing them would have left thoseokbranches dead (a nil channel case never becomes ready). I confirmed no other code depends on the fields being nil after close.
Receive side
All readers that could observe a closed channel now detect it: BlockSignal in all five endpoints, SetTimeoutSignal in the four that read it (relay intentionally has no SetTimeoutSignal case — it uses time.After(TimeoutMs)), and SendQueue/RecvQueue already checked ok. This closes the real bug where a closed BlockSignal was interpreted as a genuine block request (e.g. s.AddBlockAddr(...)).
Other checks
- Routing the server's initial inbound packet through
ForwardInboundPacket(was a rawRecvQueue <- pkt) means every send now goes through the gate — I confirmed no ungated raw sends to these channels remain. - The race-stress tests are focused and meaningful; the zero-value
&Device{}harness is safe becauseReleasePoolPacketno-ops on non-PoolAllocatedpackets.
Minor (non-blocking) observations
- Every packet forward now briefly acquires the per-connection
channelSendMu. The critical section is tiny (a singleAdd/Load) and per-connection, so contention is negligible — just noting it for the hot path. - The
c.TimeoutMs = mswrite inSetTimeoutis still read from the routine'stime.After(...)case without synchronization, but that's pre-existing and out of scope for this PR.
Nice, careful fix with good test coverage.
What changed
ConnectionData.Closeand waited for in-flight senders before closing queues and signal channels.Closecalls atomic and kept channel fields stable after closure.ForwardInboundPacketinstead of a raw queue send.Why
The previous
IsClosedcheck was separate from the channel send.Closecould run between them, close and nil the channel, and make the sender panic or race on the channel field. ConcurrentClosecalls could also both pass the check. In addition, a closedBlockSignalcould be interpreted as a genuine request to block the peer.The lifecycle gate prevents new sends once closure starts, and
Closewaits for already admitted sends to leave their select before it drains and closes channels. This removes process-killing teardown races without recovery or retry behavior and improves endpoint availability under connection churn and UDP flood pressure. It ports the repository-agnostic core of layervai/nhp#2691 to current OpenNHP.Validation
go test -race -run 'TestConnectionData' -count=10 ./core(nhpmodule)go test -race -count=1 ./...(nhpmodule)go test -race -count=1 ./ac ./agent ./db(endpointsmodule)go test -race -c ./serverandgo test -race -c ./relay(endpointsmodule)go vet ./...in both modulesThe full endpoint test sweep also reaches two unrelated current-main failures: relay expects a 301/400 redirect but receives 307, and server test initialization attempts to create
/opt/confidential-containers. The changed endpoint packages pass and the server/relay race builds succeed.