Skip to content

Backfill the in-memory canonical chain window at startup - #228

Merged
Chengxuan merged 6 commits into
client-side-filtering-feedbackfrom
client-side-filtering-startup-fetch
Sep 9, 2026
Merged

Chengxuan merged 6 commits into
client-side-filtering-feedbackfrom
client-side-filtering-startup-fetch

Conversation

@Chengxuan

@Chengxuan Chengxuan commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

NOTE: this gap affects the full tracking mode + client filtering most due to the usage of steadyStateScanCeiling when deciding the highest block to fetch to.

  • seedMonitoredHead previously fetched only a single anchor block (highestBlock-checkpointBlockGap+1), leaving the rest of the monitored window to fill in one block at a time as the live block filter delivered new blocks - paced by real chain block production rather than polling interval.
  • Right after connector startup this held back steadyStateScanCeiling in full chain-tracking mode, delaying client-side getLogs polling from reaching the chain head. It also affected ReconcileConfirmationsForTransaction (validateChainCaughtUp), which returns FF23062 for any transaction in that gap until the window catches up - this applies to full mode regardless of filterPollingMode (server or client).
  • seedMonitoredHead now fetches and reconciles the whole window up front. All but the last block are reconciled silently (no consumer notification, since they're historical, not new); the last block is still returned for the listen loop's first iteration to reconcile as before, preserving existing notification timing. On any fetch failure it falls back to the original one-block-at-a-time behavior.

Test plan

  • go build ./...
  • go vet ./...
  • go test ./...
  • Updated TestBlockListenerSeedMonitoredHead_BlockFound / _ReconcileAndDispatch for the new return-value contract (top of window, not the anchor), with new coverage asserting the backfilled blocks land in SnapshotMonitoredHeadChain()

@Chengxuan
Chengxuan requested a review from a team as a code owner September 7, 2026 10:11
@Chengxuan
Chengxuan force-pushed the client-side-filtering-startup-fetch branch from 17255ea to 096f21a Compare September 8, 2026 09:33
@Chengxuan
Chengxuan changed the base branch from client-side-filtering to client-side-filtering-feedback September 8, 2026 09:34
Signed-off-by: Chengxuan Xing <chengxuan.xing@kaleido.io>
@Chengxuan
Chengxuan force-pushed the client-side-filtering-startup-fetch branch from 096f21a to 1f87289 Compare September 8, 2026 09:49

@peterbroadhurst peterbroadhurst left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor request for a comment, with approval

func (bl *blockListener) seedMonitoredHead() *ethrpc.BlockInfoJSONRPC {
bl.canonicalChainLock.RLock()
highestBlockSet := bl.highestBlockSet
highestBlock := bl.highestBlock

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the key change is this is still read inside the lock, but we use it outside the lock below as a threshold to preemptively catch up to blocking this function until all those blocks are fetched (or one isn't available)

bl.reconcileCanonicalChain(bi)
}

bi := fetchBlock(highestBlock)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this outside of the loop as a separate item, vs just <=?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like it's because we don't call reconcileCanonicalChain - but why isn't that necessary just for the last block if it is for all the others?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok - found this in the comments on the issue. Maybe add as a comment above this line.

the last block is still returned for the listen loop's first iteration to reconcile as before, preserving existing notification timing

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add the comment.

Signed-off-by: Chengxuan Xing <chengxuan.xing@kaleido.io>
…er-firefly/evmconnect into client-side-filtering-startup-fetch

Signed-off-by: Chengxuan Xing <chengxuan.xing@kaleido.io>
Signed-off-by: Chengxuan Xing <chengxuan.xing@kaleido.io>
…er-firefly/evmconnect into client-side-filtering-startup-fetch
Signed-off-by: Chengxuan Xing <chengxuan.xing@kaleido.io>
startDone: make(chan struct{}),
initialBlockHeightObtained: make(chan struct{}),
newHeadsTap: make(chan struct{}),
newHeadsTap: make(chan struct{}, 1), // buffer 1 so a tap is not dropped if the listen loop is not currently waiting

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Chengxuan
Chengxuan merged commit 8f2ac5d into main Sep 9, 2026
4 checks passed
@Chengxuan
Chengxuan deleted the client-side-filtering-startup-fetch branch September 9, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants