Add response caching with a bounded in-memory LRU store - #3
Open
wille wants to merge 3 commits into
Open
Conversation
Cache read-only responses (block-by-hash, mined receipts, chainId, ...) so repeat calls are served locally instead of hitting an upstream provider, cutting provider load, latency, and rate-limit exposure. Enabled by default on every endpoint, in memory, on both the HTTP and WebSocket paths. Cacheability is decided in three layers: a per-chain method allowlist (chain.Cacheable), chain-level param-volatility exclusion (EVM latest/pending/ safe/finalized), and a response guard (chain.CacheableResult) that skips error/null results and unconfirmed eth_getTransactionByHash. A per-entry TTL bounds reorg staleness. The new internal/cache package exposes a pluggable Storage interface with a default bounded-LRU memory store: capped by total bytes and entry count, evicting least-recently-used on either limit, and never caching a response larger than 5 MB. Healthchecks bypass the cache (they call providers directly). HTTP caching is per sub-request, so batches are cached element-wise: cached members are served locally, only misses are forwarded, and responses are merged back in order. Adds haprovider_cache_requests_total and cache_ttl / cache_max_size_mb per-endpoint config.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a response cache so repeat reads of immutable data (a block by hash, a mined receipt,
eth_chainId) are served locally instead of hitting an upstream provider. This cuts provider load, latency, and rate-limit (429) exposure.Enabled by default on every endpoint, using an in-memory store, on both the HTTP JSON-RPC and WebSocket paths.
Correctness model
A response is cached only when all three layers agree:
chain.Cacheable(method, params), a per-chain set of read-only methods (EVM list shared by eth + tron; solana and btc have their own). Distinct from coalesceability:eth_getTransactionCount/eth_blockNumberare safe to share between concurrent callers but change too often to cache.latest/pending/safe/finalized).chain.CacheableResult(method, result): never caches error or null results, and skips an unconfirmedeth_getTransactionByHash(nullblockNumber) until it is mined.A per-entry TTL bounds residual reorg staleness.
Storage
New
internal/cachepackage with a pluggableStorageinterface (the seam for a future Redis/etc. backend) and a default in-memory implementation:Integration
X-Cache: HITon fully-cached responses.haprovider_cache_requests_total{endpoint,method,result}metric.Configuration
Per endpoint (all optional; caching is on by default):
cache_ttl— how long entries live (default10s;0disables caching).cache_max_size_mb— total cache byte budget (default128).Performance
Benchmarks included. The cache read is effectively free relative to request handling; the win is skipping the network:
MemoryGet(hit): 84 ns/op, 0 allocsThe one scaling ceiling:
Gettakes the full mutex (LRU reordering is a write), so concurrent reads on a single endpoint serialize at ~1.7M hits/sec. Well above typical throughput; shardable behind theStorageinterface if ever needed.Testing
Cacheable/CacheableResultper chain (incl. pending vs confirmed tx), LRU eviction by bytes and by count, 5 MB cap, byte accounting, TTL expiry, and a concurrent (-race) test asserting the budget holds.latest/null/error/non-allowlisted never cached,includeTransactionsdistinguishes entries, batch fully-cached and partial-forward-with-merge.-race;go vetandgolangci-lintclean.Scope / non-goals
Storagebackends (Redis, etc.).