fix(core): isolate asynchronous packet ownership - #1649
Conversation
Give transaction sends an independent sender-owned packet so transaction teardown cannot recycle bytes during a socket write. Preserve response-side EncryptedPktCh delivery when deriving from PrevParserData and cover both handoff contracts under the race detector.
|
@claude /simplify |
There was a problem hiding this comment.
✅ Looks Good - Code looks good
Thorough review of the packet-ownership fix. This is a well-constructed, correct concurrency fix with strong regression coverage. No blocking issues found.
What the change does (and why it's correct)
The bug being fixed (a genuine use-after-free / data race):
In the base code, a transaction request's mad.BasePacket was marked KeepAfterSend = true and handed to the asynchronous physical sender via ForwardOutboundPacket → SendQueue → SendPacket, which does netConn.Write(pkt.Content). That same BasePacket is retained by the LocalTransaction. On connection close / device stop / timeout, LocalTransaction.Run's deferred t.mad.Destroy() calls ReleasePoolPacket(BasePacket) — niling Content and returning the buffer to the pool. If teardown lands while the socket write is in flight, the write races on / reads a recycled buffer (empty or corrupted datagram). The new endpoints/agent/packet_lifetime_test.go reproduces exactly this window deterministically.
The fix — independent sender-owned packet (nhp/core/packet.go, device.go):
clonePacketForSend allocates a fresh pool packet, copies HeaderType + Content, and leaves KeepAfterSend = false. The sender now owns and releases the clone after the write, while the transaction keeps exclusive ownership of BasePacket. I verified:
- The transaction never re-sends
BasePacketover the wire (transaction.goonly usest.madasPrevAssemblerDatafor response crypto), so droppingKeepAfterSend = trueis safe —BasePacketis released solely byt.mad.Destroy(). Destroy()(initiator.go:443) releases onlyBasePacket; the clone has a separateBuf, so there is no double-free between the transaction and the sender.- The
outboundPacketOwnedflag + deferred release correctly plugs the panic window betweenclonePacketForSendandForwardOutboundPacket(AddLocalTransaction).outboundPacketOwned == trueimpliesmad != nil; it's cleared after handoff, andForwardOutboundPackettakes ownership (enqueue → drained bySendPacket/Closeflush, or released on closed conn), so no leak and no double-release on any path. clonePacketForSendbounds-checkslen(Content)againstPacketBufferSizebefore the copy.
Response-channel fix (initiator.go):
Moving mad.encryptedPktCh = md.EncryptedPktCh out of the PrevParserData == nil branch is safe and backward-compatible: deriveMsgAssemblerData never sets encryptedPktCh, so this doesn't overwrite anything, and no existing production caller sets both PrevParserData and EncryptedPktCh (the only EncryptedPktCh user, agent/knock.go, is a new-initiator path). For all current PrevParserData callers md.EncryptedPktCh is nil, so behavior is unchanged; the change only enables response-side callers to divert encrypted bytes instead of silently falling through to SendQueue. Covered by encrypted_channel_test.go's exact-once assertion.
Minor, non-blocking notes
- Each transaction request now incurs an extra pool allocation +
Contentmemcpy. This is negligible and scoped to transaction requests only — a reasonable price for eliminating the shared-ownership race. packet_lifetime_test.gouses alog.Infofunction-field override as a blocking pre-write seam. It's clever and effective; just note it depends on the exact"Send ["log prefix inUdpAgent.SendPacket, so a future log-message change would silently defang the test rather than fail it.
Nice work — the ownership boundaries are now clean and the race-detector regressions document both contracts well.
There was a problem hiding this comment.
✅ Looks Good - Code looks good
Thorough, well-motivated fix for a real data-race in asynchronous packet ownership, backed by deterministic race-detector regressions. I traced the ownership paths end-to-end and it holds up.
What the fix does (and why it's correct)
-
nhp/core/device.go/packet.go— Previously a transaction request setmad.BasePacket.KeepAfterSend = trueand handed the same pooledPacketto the physical sender. Transaction teardown (mad.Destroy()→ReleasePoolPacket) could recycle that buffer whilenetConn.Write(pkt.Content)was still reading it, producing an empty/garbled datagram and racing under load. The newclonePacketForSendgives the sender an independently-owned pool packet (KeepAfterSend=false, released after the write) while the transaction retains and destroysBasePacket. Clean ownership split.- Verified
LocalTransaction.Run(nhp/core/transaction.go) never retransmitsBasePacket— it only awaits response/timeout/stop thenDestroys — so droppingKeepAfterSendon the base packet is safe. - The
outboundPacketOwnedflag correctly gates the deferred release on the error/panic path and is cleared afterForwardOutboundPacket; the clone andBasePacketare distinct buffers, so a concurrent teardown cannot double-free. - The clone copies
HeaderType+ContentandAllocatePoolPacketsuppliesBuf/PoolAllocated— complete for everything the senders (udpagent.go,relay.go) consume.
- Verified
-
nhp/core/initiator.go— Hoistingmad.encryptedPktCh = md.EncryptedPktChout of the new-initiator branch fixes responses derived fromPrevParserData, which previously dropped the channel (never set byderiveMsgAssemblerData) and silently fell through to the socketSendQueue. Backward compatible: only changes behavior when a caller explicitly setsEncryptedPktCh.
Minor (non-blocking) suggestions
endpoints/agent/packet_lifetime_test.goinstalls a custom global loggerInfohook viaSetGlobalLoggerand never restores it, so later tests in the package inherit the silent logger + hook. It's harmless (the closedreleaseSend/enteredSendchannels return immediately for any subsequentSend [log), but consider restoring the prior global logger in at.Cleanupfor hygiene.- Each transaction send now incurs an extra pool allocation + content copy for the clone. Pool-backed and released promptly, so acceptable given the correctness win — just noting it.
No security, correctness, or breaking-change concerns found. Nice work isolating the seam deterministically rather than relying on load to reproduce.
Summary
EncryptedPktChdelivery for response packets derived fromPrevParserDataType of Change
Related Issues
Found while validating NHP REG/RAK response handling through asynchronous packet channels. No wire format or protocol message shape changes.
Testing
Validated locally:
cd nhp && go test -race ./...cd endpoints && go test -race ./agent ./relaygo vet ./...,go build ./..., and golangci-lint 2.7.2 in both modulesgofmt,goimports, andgit diff --checkThe full endpoints suite on unmodified upstream
mainhas two local Go 1.26/environment baselines: a POST redirect assertion expects 301 whilenet/httpreturns 307, and server KBS initialization writes under/opt. The changed agent/relay packages and the complete NHP module pass under-race; upstream CI will remain authoritative for the full matrix.Checklist