fix(core/txpool,eth): prevent a stalled tx subscriber from freezing the pool

[gzliudan]

Proposed changes

Problem

On mainnet a peer lost its transaction broadcaster. AsyncSendTransactions blocked on the stalled peer writer, which froze txBroadcastLoop, filled pm.txsCh, and eventually pinned pool.mu via a txFeed.Send issued while holding the legacy pool lock. The node kept answering RPC but stopped importing blocks: 131 goroutines piled up in Add, 20 in Pending, and runReorg never ran again.

Root cause

LegacyPool.promoteSpecialTx delivered the special-tx event with pool.txFeed.Send(...) while holding pool.mu. event.Feed.Send blocks until every subscriber drains its channel, so a single subscriber that stops reading wedges the entire pool.

Fix

Three layered changes, each independently revertible: (1) stop delivering special tx events under the pool lock — route the event through queueTxEvent (the same async path every other add() branch uses) so runReorg delivers it after releasing the lock; (2) make peer transaction broadcasts non-blocking — buffer txBroadcast/txAnnounce (txBatchBuffer) and drop a batch when the queue is full, matching AsyncSendNewBlock/AsyncSendNewBlockHash, with drop meters to surface stalled peers; (3) unblock shutdown on stalled tx event subscribers — track subscriptions in a SubscriptionScope and Close it before wg.Wait() so a stuck subscriber is removed and the blocked txFeed.Send returns, and unsubscribe listeners in TxPool.Close before terminating subpools.

Scope / follow-up

These commits guarantee the node never freezes, never loses transactions, and shuts down gracefully. A non-peer in-process subscriber that stalls will still freeze the reorg pipeline (stale pending, halted proactive broadcast, growing queued events) until it unsubscribes or the node restarts — this is the deferred L2 hardening (decouple event delivery from the reorg done, bounded drop queue), not done here because no such subscriber exists in the standard deployment.

Types of changes

What types of changes does your code introduce to XDC network?
Put an ✅ in the boxes that apply

• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Changes that don't change source code or tests
• docs: Documentation only changes
• feat: A new feature
• fix: A bug fix
• perf: A code change that improves performance
• refactor: A code change that neither fixes a bug nor adds a feature
• revert: Revert something
• style: Changes that do not affect the meaning of the code
• test: Adding missing tests or correcting existing tests

Impacted Components

Which parts of the codebase does this PR touch?
Put an ✅ in the boxes that apply

• Consensus
• Account
• Network
• Geth
• Smart Contract
• External components
• Not sure (Please specify below)

Checklist

Put an ✅ in the boxes once you have confirmed below actions (or provide reasons on not doing so) that

• This PR has sufficient test coverage (unit/integration test) OR I have provided reason in the PR description for not having test coverage
• Tested on a private network from the genesis block and monitored the chain operating correctly for multiple epochs.
• Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet.
• Tested the backwards compatibility.
• Tested with XDC nodes running this version co-exist with those running the previous version.
• Relevant documentation has been updated as part of this PR
• N/A

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 02a5104a-4d96-4c58-bde6-57358731fd1d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Moves special-transaction event delivery outside the txpool lock to prevent stalled subscribers from freezing pool operations.

Changes:

• Queues special transaction events through the reorg path.
• Adds announcement limits and revised shutdown behavior.
• Adds regression tests for stalled subscribers and reset handling.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.

File	Description
core/txpool/legacypool/legacypool.go	Revises event delivery, reorg scheduling, and shutdown handling.
core/txpool/legacypool/legacypool_test.go	Adds regression tests for event delivery and queue behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (5)

eth/peer.go:128

• This meter misses the drops caused by the failure mode it is intended to expose. While the network writer is stalled, broadcastTransactions continues draining txBroadcast and truncates its internal hash queue at eth/peer.go:227-230, so the outer channel may never fill and those discarded hashes are not counted. Mark the number removed at that truncation site as well so this metric actually surfaces stalled-peer loss.

	txBroadcastDropMeter = metrics.NewRegisteredMeter("eth/peer/transaction/broadcasts/drop", nil)

eth/peer.go:132

• The announcement meter has the same observability gap: announceTransactions keeps consuming this channel while its writer is stalled, then silently removes excess hashes at eth/peer.go:289-292. Consequently, sustained stalled-writer drops can remain invisible. Increment this meter by the number removed at the internal queue cap too.

	txAnnounceDropMeter = metrics.NewRegisteredMeter("eth/peer/transaction/announces/drop", nil)

eth/peer.go:148

• The buffer is bounded by batch count, but each batch can contain thousands of hashes; this bypasses the existing 4,096-hash queue caps and can retain many times that amount per peer when consumers stop. Bound the buffered work by hash count (for example, cap/split batches before enqueueing) so stalled peers cannot amplify memory use across all connected peers.

		txBroadcast:     make(chan []common.Hash, txBatchBuffer),
		txAnnounce:      make(chan []common.Hash, txBatchBuffer),

eth/peer.go:60

• The central non-blocking guarantee is untested: eth/peer_test.go has no coverage for either asynchronous transaction queue. Add tests that saturate both queues, assert subsequent calls return without blocking, verify dropped batches are not marked known, and check the drop counters; otherwise a blocking-send regression in this production freeze fix will pass CI.

	// announcements) that may be queued for a peer before newer batches are
	// dropped. Together with the non-blocking send in the AsyncSend* methods,
	// it guards the shared broadcast loop against a stalled or exited per-peer
	// writer blocking it indefinitely.
	txBatchBuffer = 16

core/txpool/txpool.go:143

• The production TxPool.Close ordering is not exercised by the added direct LegacyPool.Close test. Add a coordinator-level test with a subpool whose Close waits for its transaction subscription to be cancelled, then assert TxPool.Close completes; this locks in the requirement that p.subs.Close() runs before subpool shutdown.

	// terminating the subpools: a subscriber that stopped draining its channel
	// would otherwise leave a subpool's runReorg blocked in txFeed.Send, and the
	// subpool Close (wg.Wait) would hang indefinitely. Closing the scope removes
	// the stuck subscription and unblocks Send.
	p.subs.Close()