Espresso hotshot doesn't enforce any ordering on batches it simply processes batches in the order it receives them, this can lead to streamer instances getting stuck due to restarts.
Imagine batches are delivered in this order 1 2 3 4 6 5 a streamer instance that did not restart would output batches 1 2 3 4 5 6, a streamer instance that restarts between 6 and 5 (1 2 3 4 6 | 5) would (if using my suggestion here to have the consumer persist the latest batch and hotshot pos between restarts) have persisted 4 as the latest batch and upon restart would process 5 but would never receive 6 and would hence be permanently stuck.
Potential solutions
-
Have streamer instances rescan some arbitrary number of hotshot blocks back before the starting hot shot pos. What determines how many blocks to scan back would be the batcher code that submits those blocks to hotshot, at present it looks like the out of orderness could be unbounded, so there may be no safe re-scan value (apart from rescanning from the absolute beginning).
-
Ensure submission occurs in order, the transaction submission code under https://github.com/celo-org/optimism/blob/af2282204da455a6296d3c25a7e1ecdb23e1d7eb/op-batcher/batcher/espresso.go seems highly complex (many workers and channels, lots of concurrency), I wonder if it is necessary at all or if it would be sufficient to submit blocks in order, given celo produces one block per second a submission time of 0.1 seconds seems acceptable, providing for a 10X catchup speed for the batcher if it fell behind.
Espresso hotshot doesn't enforce any ordering on batches it simply processes batches in the order it receives them, this can lead to streamer instances getting stuck due to restarts.
Imagine batches are delivered in this order
1 2 3 4 6 5a streamer instance that did not restart would output batches1 2 3 4 5 6, a streamer instance that restarts between 6 and 5 (1 2 3 4 6 | 5) would (if using my suggestion here to have the consumer persist the latest batch and hotshot pos between restarts) have persisted 4 as the latest batch and upon restart would process 5 but would never receive 6 and would hence be permanently stuck.Potential solutions
Have streamer instances rescan some arbitrary number of hotshot blocks back before the starting hot shot pos. What determines how many blocks to scan back would be the batcher code that submits those blocks to hotshot, at present it looks like the out of orderness could be unbounded, so there may be no safe re-scan value (apart from rescanning from the absolute beginning).
Ensure submission occurs in order, the transaction submission code under https://github.com/celo-org/optimism/blob/af2282204da455a6296d3c25a7e1ecdb23e1d7eb/op-batcher/batcher/espresso.go seems highly complex (many workers and channels, lots of concurrency), I wonder if it is necessary at all or if it would be sufficient to submit blocks in order, given celo produces one block per second a submission time of 0.1 seconds seems acceptable, providing for a 10X catchup speed for the batcher if it fell behind.