Skip to content

op-batcher: simplify Espresso worker pool to a shared job channel - #524

Open
jjeangal wants to merge 1 commit into
espresso/batcherfrom
espresso/simplify-worker-pool
Open

jjeangal wants to merge 1 commit into
espresso/batcherfrom
espresso/simplify-worker-pool

Conversation

@jjeangal

@jjeangal jjeangal commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Part of #506.

What

The submit/verify worker pool hand-rolled a load balancer: per stage, an extra chan chan … "worker queue" onto which idle workers advertised a private channel, plus a scheduler goroutine that matched each waiting job to an advertised worker.

N workers all receiving from one shared channel already do exactly that — the Go runtime hands each job to whichever worker is ready. So this removes the hand-rolled version:

  • Workers now read the shared submitJobQueue / verifyReceiptJobQueue directly and reply on the shared response queues.
  • Deletes the two scheduler goroutines (scheduleSubmitTransactionJobs, scheduleVerifyReceiptsJobs), the two chan chan worker queues, and the two *JobAttempt wrapper structs.

Net −129 lines (43 insertions, 172 deletions), two fewer goroutines, one hop per job instead of three.

Behavior is unchanged

Same buffered queues and capacities, same in-flight counter and fast-reject (ErrSubmitToEspressoChannelFull), same response handlers and retry/re-submit paths, same SpawnWorkers(4, 4) surface. Only the plumbing shrank. Workers also now exit cleanly if the job channel is ever closed (a ok check), in addition to the existing ctx.Done() path.

This is the safe, behavior-identical subset. The larger errgroup.SetLimit rewrite (pure evaluate*, per-block submitAndConfirm) from the issue is a follow-up, since it changes backpressure semantics.

Testing

  • go build ./op-batcher/...
  • go test ./op-batcher/batcher/... (incl. -race) — passes; the existing submitter progress test drives the pool end-to-end via SpawnWorkers(4, 4).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant