Skip to content

Add response caching with a bounded in-memory LRU store - #3

Open
wille wants to merge 3 commits into
masterfrom
request-caching
Open

Add response caching with a bounded in-memory LRU store#3
wille wants to merge 3 commits into
masterfrom
request-caching

Conversation

@wille

@wille wille commented Jul 9, 2026

Copy link
Copy Markdown
Owner

What

Adds a response cache so repeat reads of immutable data (a block by hash, a mined receipt, eth_chainId) are served locally instead of hitting an upstream provider. This cuts provider load, latency, and rate-limit (429) exposure.

Enabled by default on every endpoint, using an in-memory store, on both the HTTP JSON-RPC and WebSocket paths.

Correctness model

A response is cached only when all three layers agree:

  1. Method allowlistchain.Cacheable(method, params), a per-chain set of read-only methods (EVM list shared by eth + tron; solana and btc have their own). Distinct from coalesceability: eth_getTransactionCount / eth_blockNumber are safe to share between concurrent callers but change too often to cache.
  2. Param volatility — decided at the chain level: EVM refuses to cache requests referencing a mutable block tag (latest/pending/safe/finalized).
  3. Response guardchain.CacheableResult(method, result): never caches error or null results, and skips an unconfirmed eth_getTransactionByHash (null blockNumber) until it is mined.

A per-entry TTL bounds residual reorg staleness.

Storage

New internal/cache package with a pluggable Storage interface (the seam for a future Redis/etc. backend) and a default in-memory implementation:

  • Bounded LRU capped by both a total byte budget and an entry count; evicts least-recently-used on either limit. O(1) ops.
  • 5 MB per-response cap — larger payloads (e.g. a full block with all transactions on a high-throughput chain) pass through uncached so one payload can't dominate the cache.
  • Lazy expiry on read plus a background janitor bounded by the entry limit.

Integration

  • HTTP — caching is per sub-request, so it works for single requests and batches: cached members are served locally and only the misses are forwarded, then responses are merged back in order. An all-cached batch never contacts a provider. X-Cache: HIT on fully-cached responses.
  • WebSocket — each (split) request is looked up in the cache; hits are answered without forwarding, and misses are stored when their response arrives.
  • Healthchecks are never cached — they call the provider directly, bypassing the handler layer where the cache lives, so they always see fresh block/sync data.
  • New haprovider_cache_requests_total{endpoint,method,result} metric.

Configuration

Per endpoint (all optional; caching is on by default):

  • cache_ttl — how long entries live (default 10s; 0 disables caching).
  • cache_max_size_mb — total cache byte budget (default 128).

Performance

Benchmarks included. The cache read is effectively free relative to request handling; the win is skipping the network:

  • MemoryGet (hit): 84 ns/op, 0 allocs
  • End-to-end cache hit: ~30 us/op vs ~275 us against a localhost upstream (~9x; far more against a real remote provider), with zero upstream load.

The one scaling ceiling: Get takes the full mutex (LRU reordering is a write), so concurrent reads on a single endpoint serialize at ~1.7M hits/sec. Well above typical throughput; shardable behind the Storage interface if ever needed.

Testing

  • Unit: Cacheable/CacheableResult per chain (incl. pending vs confirmed tx), LRU eviction by bytes and by count, 5 MB cap, byte accounting, TTL expiry, and a concurrent (-race) test asserting the budget holds.
  • HTTP integration: hit-after-miss, id echo, latest/null/error/non-allowlisted never cached, includeTransactions distinguishes entries, batch fully-cached and partial-forward-with-merge.
  • WebSocket integration: warm cache served without a second upstream message.
  • Full suite green under -race; go vet and golangci-lint clean.

Scope / non-goals

  • Non-memory Storage backends (Redis, etc.).
  • Per-method / finality-aware TTL tiers.
  • The TRON native HTTP API path (JSON-RPC only).

wille added 3 commits July 10, 2026 01:32
Cache read-only responses (block-by-hash, mined receipts, chainId, ...) so
repeat calls are served locally instead of hitting an upstream provider,
cutting provider load, latency, and rate-limit exposure. Enabled by default
on every endpoint, in memory, on both the HTTP and WebSocket paths.

Cacheability is decided in three layers: a per-chain method allowlist
(chain.Cacheable), chain-level param-volatility exclusion (EVM latest/pending/
safe/finalized), and a response guard (chain.CacheableResult) that skips
error/null results and unconfirmed eth_getTransactionByHash. A per-entry TTL
bounds reorg staleness.

The new internal/cache package exposes a pluggable Storage interface with a
default bounded-LRU memory store: capped by total bytes and entry count,
evicting least-recently-used on either limit, and never caching a response
larger than 5 MB. Healthchecks bypass the cache (they call providers directly).

HTTP caching is per sub-request, so batches are cached element-wise: cached
members are served locally, only misses are forwarded, and responses are merged
back in order. Adds haprovider_cache_requests_total and cache_ttl /
cache_max_size_mb per-endpoint config.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant