fix(util-internal): don't wedge ingest head-polling on a hung finalized/head fetch

[elina-chertova]

Cause (proven)

Both zora-mainnet EVM hotblocks pods (dwellir and uniblock — two independent providers) went silent at the identical block 49023729 at 17:28:16Z, stopped emitting head metrics (firing Portal_Hotblocks_Head_Metrics_Absent_zora_mainnet), and did not recover even though the chain kept producing blocks (verified block 49024000 exists; providers advanced past the stuck point). No crash/restart — the process was hung.

Root cause is in Throttler (util/util-internal/src/throttler.ts), the shared primitive behind every ingest head-/finalized-head tracker:

async get(): Promise<T> {
    let now = Date.now()
    if (now - this.lastAccess < this.interval) return this.lastValue
    return this.performCall(0)   // <-- returns the in-flight (hung) promise
}

lastAccess is updated only on success (execute()). When the underlying fetch hangs or times out for longer than interval, the next get() falls through and returns the still-pending refresh promise. In evm-rpc/src/data-source/ingest.ts the hot loop does await finalizedHeadTracker.get() before the head poll (poll.next()), so a stalled eth_getBlockByNumber("finalized") (the pod logged HttpTimeoutError: request timed out after 30000 ms on exactly that call) blocks the whole loop. Because the loop runs in a worker, the main-thread DataService for await then blocks forever with no further logging — matching the observed permanent silence. enablePrefetch() is used by every EVM/Solana/Tron/Bitcoin head/finalized tracker, so this can wedge any of them.

Fix

Honor the prefetch contract: once a value exists, get() returns the last known value immediately and lets the (slow/hung) refresh settle in the background, instead of blocking on it. Head polling proceeds; a stale-but-conservative finalized head is safe (only affects the finalized upper bound, which self-corrects on recovery). The first read still awaits the initial fetch, and non-prefetch throttlers are unchanged.

Tests (red → green)

util/util-internal/src/throttler.test.ts (vitest, newly wired into the package's test script):

• the wedge: prefetch on, first fetch succeeds, subsequent refresh hangs → get() must return the cached value promptly. FAILS pre-fix (returns the hung promise → 'HUNG'), PASSES post-fix.
• first get() blocks until the initial value resolves.
• non-prefetch throttler awaits a fresh value when stale (unchanged behavior).

Verified locally: rush build --to @subsquid/util-internal ✅, rush test --only @subsquid/util-internal ✅. Confirmed red→green by reverting get() to the old body (the wedge test fails with expected 'HUNG' to be 1).

Scope / relation to other PRs

This is the transient-hang → permanent-wedge fatality (the process should survive a slow/unresponsive finalized endpoint and keep polling head). Distinct root cause from #513 (retry a transient invalid block height error on the finalized probe) and #523 (portal-client retries transient null heads). Complementary, not a duplicate.

Falsification

If a hotblocks/dumper pod still goes permanently silent (no logs, no restart) after its finalized/head RPC endpoint hangs — i.e. the head loop stops advancing while the chain keeps producing — this fix is insufficient and the block is elsewhere on the head path.

[eldargab]

The contract of the Throttler is - Allow the value to be stale for no longer than .interval,
and it is used in many places.

This PR changes the contract and I can't calculate the consequences out of my head.

Depending on how relevant the AI analysis actually is, the fix should be related to either
the Ingest loop or "Worker thread - Main thread" error propagation.