Skip to content

perf: serve year-long financial series from monthly caches - #318

Merged
tekwani merged 1 commit into
tetherto:developfrom
paragmore:feature/financials-month-caches
Oct 19, 2026
Merged

tekwani merged 1 commit into
tetherto:developfrom
paragmore:feature/financials-month-caches

Conversation

@paragmore

Copy link
Copy Markdown
Contributor

Project Financials phase tabs take 13-17s to render (Total Site ~6s). Each tab fires three revenue-summary windows plus a metrics/hashrate interval=1d call over 12 months, and almost all of that work was repeated on every request: the 1d hashrate series rebuilt ~8,760 hourly buckets (plus a year of pool samples with pool=true) per call, and a phase revenue-summary re-fetched the whole-range by-meter series and manual consumption log every time by design. The concurrent windows then each re-scanned the same cold months independently.

This reuses the pattern the 1M rollup already has. getHashrate interval=1d caches day rows month by month (ended, fully covered months only; edge slices and the running month always recompute; overwriteCache refreshes). Phase consumption caches its hourly rows per UTC month, and saveEnergyConsumption drops that cache on a confirmed write, so manual edits are never served stale; store-side backfills are bounded by the same 12h TTL the other monthly rollups accept. getDailySeries additionally single-flights per-month rollups so the overlapping windows a tab fires at once compute each cold month once. The phase revenue-summary keeps its always-refresh semantics, so coverage/missing-hours accounting is unchanged - only the fetch underneath got cheap.

One side effect worth calling out: phase consumption now fetches per month, so months at or after energyRollup.sinceTs read the energy-1h rollup even when the request started before it, where previously one pre-rollup start forced the whole span onto the legacy stat path. The two sources are meant to agree; the rollup is the higher-fidelity one.

Bench on the real handler stack against a simulated ork (span-proportional, serialized latency; responses byte-identical before/after): warm tab renders drop from ~12s/31s/8s to ~4s/7s/4s for total/phase1/phase1_5, and cold renders improve ~25% from the single-flight dedup. Full unit and integration suites pass; new tests cover the caches (scope keys, edge months, running month, tombstones, pool rows, invalidation, concurrent dedup, failure retry).

The Project Financials tabs rebuilt a year of hourly data on every render:
metrics/hashrate interval=1d re-scanned ~8,760 hourly buckets (and a year of
pool samples with pool=true) per request, and a phase revenue-summary
re-fetched the full-range by-meter series plus the manual consumption log
every time. With the three windows the page fires at once, each tab cost
several concurrent year-long scans on the ork.

Cache the 1d day rows month by month like getMonthlyHashrate already caches
its calendar months (ended, fully covered months only; partial edges and the
running month recompute), cache the phase consumption hourly rows per UTC
month with an explicit drop from saveEnergyConsumption so manual edits are
never served stale, and let concurrent windows share one in-flight rollup per
month in getDailySeries instead of each re-scanning the same cold months.
Phase revenue-summary keeps its refresh semantics, so its coverage accounting
is unchanged and only the fetch cost moves.

Adjacent cold phase months fetch as one span rather than one query pair per
month, a fetch that straddles a manual write no longer re-stores its pre-write
rows over the invalidation (and a request arriving after the write reads fresh
instead of riding the stale fetch), and a window ending exactly on a month
boundary now answers the same from a warm cache as from a cold one - the
zero-width edge month used to be dropped (getDailySeries) or rejected with
ERR_INVALID_DATE_RANGE (the month rollup) once its neighbours were cached.
@paragmore
paragmore force-pushed the feature/financials-month-caches branch from 9f4d156 to b20e32b Compare October 19, 2026 12:34
@paragmore

Copy link
Copy Markdown
Contributor Author

Pushed a hardening pass after a deeper review and a differential QA run (develop vs this branch driven through ~200 randomized scenarios - timezones, data gaps, phases, concurrent window bursts, manual writes between reads - with every response deep-compared).

Fixes folded in:

  • A window ending exactly on a month boundary leaves a zero-width edge month once its neighbours are cached. The month rollups rejected it with ERR_INVALID_DATE_RANGE on the repeat request, and getDailySeries silently dropped the boundary hour on a warm cache while keeping it on a cold one (that inconsistency predates this PR - the single-flight just made it easier to hit). All four sites now step the degenerate span back one millisecond, which cannot add a bucket, so warm and cold answers are identical.
  • A phase-hourly fetch that straddled a manual consumption write could re-store its pre-write rows after the invalidation and mask the correction for the TTL. Clears now bump a generation; a straddling fetch still answers its own request but no longer stores, and a request arriving after the write reads fresh instead of riding the stale in-flight promise.
  • Adjacent cold phase months now fetch as one span per contiguous run instead of one query pair per month, so a cold 19-month window is two reads, not thirty-eight.

The differential run is clean apart from the boundary-window case above, where develop itself returns different answers for the same query depending on cache warmth and this branch consistently returns the cold (complete) one. Full unit and integration suites pass; new tests cover the boundary windows, the write-vs-fetch race, and the run batching.

@tekwani
tekwani merged commit 22fdd14 into tetherto:develop Oct 19, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants