UN-3973 [DEV] Cut dashboard cron DB time by deriving monthly metrics from the daily tier

[kirtimanmishrazipstack]

What

• event_metrics_monthly is now derived by rolling up event_metrics_daily, instead of re-querying source tables per organisation.
• The per-run source window for the daily tier drops from ~32–62 days to 2 days.
• A once-daily reconciliation pass reruns the same task at a 7-day window to repair gaps left by cron downtime.

Acceptance criteria — all five met

#	Criterion	Met by	Verified
1	Monthly totals derived from event_metrics_daily, not source tables	_rollup_monthly_from_daily reads only EventMetricsDaily; _aggregate_org no longer receives monthly_start	test_monthly_rollup_never_touches_source_tables captures the SQL and asserts no workflow_file_execution / workflow_execution / page_usage
2	Per-run daily source window is 2 days	DASHBOARD_SOURCE_WINDOW_DAYS = 2, the default on both the task and _run_aggregation	test_default_window_bounds_the_daily_query, test_late_terminal_status_does_not_re_enter_the_narrow_window; a live run reported daily.start = today − 2
3	A once-daily 7-day reconciliation pass exists and is scheduled	Migration 0004 creates the beat row — 04:00 UTC daily, queue dashboard_metric_events, kwargs={"source_window_days": 7}, enabled	test_migration_schedules_the_pass_at_0400_with_a_7_day_window, test_migration_is_idempotent_and_reversible
4	Daily and monthly figures match pre-change values for a sample of orgs, incl. across a month boundary	Monthly is a pure sum of the daily tier	Re-run against a live dev DB: 14/14 rows MATCH on value and count, 2 orgs × 2 months (2026-07 and 2026-08), compared against backfill_metrics — which still computes monthly the old way and is untouched by this PR
5	Tests cover the month-boundary case and the reconciliation pass	—	test_month_boundary_keeps_months_separate; test_reconciliation_window_widens_the_daily_query, test_reconciliation_recovers_a_day_the_narrow_window_missed, plus the two schedule tests

The parity check in row 4 is the one that matters most, since it is the only check that asks are the numbers the same as before? rather than is the code self-consistent? Every row agreed exactly:

 org |   month    |        metric         | new | old | verdict
   3 | 2026-07-01 | documents_processed   |  21 |  21 | MATCH
   3 | 2026-07-01 | failed_pages          |   0 |   0 | MATCH
   3 | 2026-07-01 | prompt_executions     |  12 |  12 | MATCH
   4 | 2026-07-01 | documents_processed   |  10 |  10 | MATCH
   4 | 2026-07-01 | failed_pages          |   0 |   0 | MATCH
   4 | 2026-07-01 | prompt_executions     |   6 |   6 | MATCH
   3 | 2026-08-01 | deployed_api_requests |   1 |   1 | MATCH
   3 | 2026-08-01 | documents_processed   |  46 |  46 | MATCH
   3 | 2026-08-01 | failed_pages          |   0 |   0 | MATCH
   3 | 2026-08-01 | pages_processed       |   3 |   3 | MATCH
   3 | 2026-08-01 | prompt_executions     |  27 |  27 | MATCH
   4 | 2026-08-01 | documents_processed   |  22 |  22 | MATCH
   4 | 2026-08-01 | failed_pages          |   0 |   0 | MATCH
   4 | 2026-08-01 | prompt_executions     |  12 |  12 | MATCH

Why

_aggregate_single_metric and _aggregate_llm_combined widened their DAY-granularity query to monthly_start — the first of the previous calendar month — so monthly buckets could be summed in Python from the same rows, saving a third query per metric. The cost was that every run re-read 32–62 days of source data for each of 38 organisations, 96 times a day.

Six of the eight cron queries are index-backed (11–107 ms avg), so their cost is proportional to that range. Per the analysis on UN-3883, ~460 s per 6 h is recoverable by narrowing it, with no schema change.

Two supporting facts make the rollup safe: event_metrics_daily retains 365 days against a 2-month monthly window (11 MB / 25 k rows), and the code already derives monthly by summing DAY buckets — this only moves where the sum happens, not what it sums.

The 2-day window is sized on measurement, not guesswork: over 30 days and 405,951 terminal rows on the read replica, the worst created_at → terminal-status lag was 2 h 16 m, with zero rows beyond 6 h. It is bounded by FILE_PROCESSING_TASK_TIME_LIMIT (2 h) and the 2.5 h stuck-execution reaper, so 2 days leaves ~21× margin.

This will not move the top-line 55-minute figure. It cannot touch get_documents_processed (44% of cron time) or get_failed_pages (43%) — those are UN-3972's. Measure it against §4 rows 5, 6, 7, 10 and 14.

How

All changes are in backend/dashboard_metrics/.

• Constants — DASHBOARD_SOURCE_WINDOW_DAYS = 2, DASHBOARD_RECONCILE_WINDOW_DAYS = 7, DASHBOARD_ACTIVE_ORG_LOOKBACK_DAYS = 7.

• aggregate_metrics_from_sources(source_window_days=2) threads the window through to _run_aggregation. The reconciliation pass is the same task at a different bound, so it needs no separate code path — just a second PeriodicTask row.

• _aggregate_single_metric / _aggregate_llm_combined lost their monthly_start and monthly_agg parameters. The DAY query now binds daily_start, and the if day_ts >= daily_start guard is gone — it was a Python filter over rows the query had already fetched, and the bind value does that work now. Still 2 queries per metric, just a narrow one.

• _bulk_upsert_monthly → _rollup_monthly_from_daily(month_start) — one ORM aggregate (TruncMonth + Sum) over event_metrics_daily, called once after the org loop for all organisations. The monthly tier stops scaling with tenant count.

  Two details worth review attention: the grouping matches unique_monthly_metric — (organization, month, metric_name, project, tag) — exactly, and metric_type is aggregated (Min) rather than grouped, because it is not part of that constraint. Grouping on it would let a metric whose type ever changed mid-month produce two rows on one key, which ON CONFLICT DO UPDATE rejects outright. The Min is aliased mtype because Django refuses an annotation named after a model field.

• The 15-line January special case for monthly_start collapsed to two lines using the existing _truncate_to_month helper, which handles the year rollover on its own.

• The active-org prefilter is decoupled from daily_start and pinned at 7 days — see the breakage section below.

One change outside dashboard_metrics/: backend/settings/base.py moves the find_dotenv() / load_dotenv() call above the Celery block. CELERY_BROKER_BASE_URL, _USER and _PASS were read on the lines above it, so they could not be supplied by an env file at all — only ambiently. Ambient values still take precedence (load_dotenv defaults to override=False) and deployments set them in the chart, so runtime behaviour is unchanged; it only makes a local pytest run work without a four-variable prefix. Happy to split this into its own PR if reviewers prefer.

Chose the ORM over the raw INSERT … SELECT sketched on the ticket: identical arithmetic, no string-built SQL, and it reuses the existing _base_manager convention for Celery context.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why.

Three real behaviour changes, all deliberate:

1. Monthly now inherits gaps in event_metrics_daily. The 62-day source window used to re-derive monthly from source tables, so it incidentally self-healed the monthly tier after cron downtime. Post-change, monthly is only as complete as the daily tier, and the reconciliation pass heals 7 days back. Downtime longer than 7 days needs the backfill_metrics management command. This is the trade the ticket accepts by design.

   One case is worth stating precisely, because "inherits gaps" understates it. Where some daily rows for a key survive the gap, monthly is recomputed low. Where every daily row for that key is missing, the key never enters fresh_keys and _delete_orphan_monthly removes the monthly row outright. Both are bounded to the current and previous month (month__gte=month_start), and both are closed for gaps that already exist by the one-time backfill under Deploy Steps.

2. Source deletions stop self-correcting beyond the window. Deleting a workflow cascades to its executions and file executions. Pre-change, monthly was re-derived from source over 62 days, so that correction propagated. Post-change, monthly sums the daily tier and the daily tier keeps the stale bucket, so only the last 2 days — 7 on the reconciliation pass — self-correct. Older buckets stay as they were until a backfill_metrics run. The durable fix is invalidation when the workflow is deleted, not a wider scan in the cron; raised by Greptile and answered in-thread.

3. Lock contention on the reconciliation pass. Both schedules share AGGREGATION_LOCK_KEY, so a reconciliation pass that fires while a 15-minute run is in flight will skip that day (~10% of days). Nothing is lost — the next day's pass still covers 7 days back — unless a collision and multi-day downtime coincide. Judged not worth a retry branch.

What is not at risk: every dashboard metric filters and buckets on the same column (get_hitl_completions uses approved_at for both, the rest use created_at), so a row can only ever be counted in the bucket of the timestamp it is filtered by. A 2-day window therefore cannot silently drop a late-arriving row into an uncounted bucket.

One near-miss caught during implementation and fixed here: the active-org prefilter reused daily_start, so narrowing that variable would have silently narrowed the prefilter too. Since get_hitl_completions filters approved_at, an org approving today a document processed five days ago would have fallen out of active_org_ids and lost the metric. The prefilter now has its own constant and stays at 7 days — its own 1,849 ms cost is UN-3974's problem, not this PR's.

Database Migrations

• dashboard_metrics/0004_add_reconciliation_task.py — data migration only, no schema change. Creates one django_celery_beat PeriodicTask named dashboard_metrics_reconcile_source_window (04:00 UTC, queue dashboard_metric_events, kwargs={"source_window_days": 7}), chosen to sit clear of the existing 02:00 and 03:00 cleanup tasks. Uses update_or_create, so it is safe to re-run; the reverse deletes the row by name. 0002_setup_periodic_tasks.py is untouched.

Deploy Steps

One-time, after the migration and before the first scheduled rollup:

manage.py backfill_metrics --days 60 --skip-hourly --skip-monthly

Monthly is derived from event_metrics_daily from this PR onward, so the daily tier has to be complete across the rollup window — current + previous month — before the first rollup runs. A gap already sitting in the daily tier, from downtime longer than the old 7-day window, would otherwise propagate straight into monthly on that first run: undercounted where the gap is partial, deleted where a key lost every daily row behind it.

--skip-monthly is deliberate. The point is to repair daily and let the scheduled rollup derive monthly from it, not to write monthly from source here. --skip-hourly keeps the run cheap; hourly has its own 24-hour window and 30-day retention and is untouched by this change.

Needed once. After it, the 04:00 reconciliation pass carries the daily tier forward.

Env Config

• None.

Relevant Docs

• UN-3883 analysis §6.2, §6.3, §7 step 1, §8 (revised 2026-08-12).

Related Issues or PRs

• Parent: UN-3883 — Analyze and optimize dashboard cron queries causing high DB load
• Siblings: UN-3972 (indexes), UN-3974 (schedule split + prefilter)
• Raised against UN-3883-Optimize-DB-cron-queries-causing-high-DB-load, not main.

Dependencies Versions

• None.

Notes on Testing

backend/dashboard_metrics/tests/test_tasks.py — 26 pass (9 pre-existing, 17 new).

cd backend
DJANGO_SETTINGS_MODULE=backend.settings.test_cloud uv run pytest dashboard_metrics/ --reuse-db

TestMonthlyRollup (8) — sums day rows into the right month bucket; month boundary, rows spanning the 1st land in two separate rows with no bleed; rows older than month_start excluded; rerun overwrites rather than doubles (ON CONFLICT); a metric whose metric_type differs across days yields one row rather than crashing; an empty daily tier is a no-op; a monthly row is dropped once its day rows go; months before the window are left alone.

TestRollupQueryShape (1) — the rollup issues no source-table SQL. Without this, a regression to re-counting raw records would leave every other test green and silently undo the saving.

TestSourceWindow (6) — the per-run and reconciliation windows each bound the daily query; the task forwards its kwarg; and three tests that seed real source rows to document the ladder 2 days → 7 days → manual backfill: a 5-day-old row recovered only by the wider pass, a row that finishes after the narrow window moved past its created_at and so never re-enters it, and a 62-day gap that neither scheduled pass reaches.

TestReconciliationSchedule (2) — the beat row's contents, its idempotency and its reverse. These call the migration's function directly: the suite runs with --no-migrations, so data migrations never execute and asserting on the row itself would fail regardless of the migration being correct.

Every new behaviour was mutation-checked rather than assumed: setting the window to 7 and the cron hour to 5 made exactly the three intended tests fail, and no others.

These reach CI automatically — conftest.py auto-marks DB-bound tests integration, so they run under the rig's integration-backend group with no manifest edit.

Also verified manually against a local Postgres: the migration applies and all four PeriodicTask rows read back correct; makemigrations --check reports no model drift; and an end-to-end _run_aggregation over the real source tables returned errors: 0 with monthly.start = 2026-07-01 and daily.start seven days back for the reconciliation pass.

Screenshots

N/A — backend only.

Checklist

I have read and understood the Contribution Guidelines.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KGZBF68CShem3pbUJM2tBc

[greptile-apps]

Greptile Summary

The PR reduces dashboard aggregation database load by limiting routine source queries to two days and deriving current and previous monthly metrics from the persisted daily tier.

• Adds a daily seven-day reconciliation schedule for recovering short aggregation gaps.
• Reworks monthly aggregation into a global transactional daily-tier rollup with stale-key cleanup.
• Adds coverage for rollup boundaries, source windows, recovery behavior, and migration lifecycle.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
backend/dashboard_metrics/migrations/0004_add_reconciliation_task.py	Adds an idempotent and reversible 04:00 UTC reconciliation task using the seven-day source window.
backend/dashboard_metrics/tasks.py	Narrows routine source aggregation, globally rolls monthly data up from the daily tier, and transactionally removes obsolete derived keys.
backend/dashboard_metrics/tests/test_tasks.py	Adds focused coverage for monthly derivation, month boundaries, reconciliation windows, late terminal rows, query shape, and migration behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    S[Source metric tables] -->|Every 15 minutes: 2-day window| D[(EventMetricsDaily)]
    S -->|Daily reconciliation: 7-day window| D
    D -->|Current and previous month rollup| M[(EventMetricsMonthly)]
    B[Manual backfill for older gaps] --> D

Loading

_{Reviews (9): Last reviewed commit: "Merge branch 'UN-3883-Optimize-DB-cron-q..." | Re-trigger Greptile}

[kirtimanmishrazipstack]

@greptile-apps please review

[kirtimanmishrazipstack]

@greptile-apps please review again

[kirtimanmishrazipstack]

@greptile-apps On the summary verdict — the mechanism is right, "not safe to merge" isn't. This is the trade the ticket asks for, and it was in the PR description before the review.

It's documented and accepted. Under "Can this PR break any existing features", item 1:

Monthly now inherits gaps in event_metrics_daily. […] the reconciliation pass heals 7 days back. Downtime longer than 7 days needs the backfill_metrics management command. This is the trade the ticket accepts by design.

Monthly stopped self-healing over 62 days deliberately — that 62-day re-derivation from source tables is the DB load UN-3973 exists to remove. Widening the rollup's source window to make orphan cleanup safe against arbitrarily old gaps would revert the PR.

You did surface one thing the description understated, and I've fixed it. "Inherits gaps" reads as undercounting; the deletion case is worse than that:

• some daily rows for a key survive the gap → monthly recomputed low
• every daily row for that key missing → key never enters fresh_keys → monthly row deleted outright

Confirmed against the test DB (SKIP_DAILY_NO_BACKING -> None). Both are bounded to the current and previous month by month__gte=month_start; a month outside the window is untouched (OLD_MONTH -> 777.0). Item 1 now states the deletion case explicitly rather than folding it into "gaps".

The one-time exposure is real, and it's a deploy step rather than a code change. Gaps already sitting in the daily tier would propagate into monthly on the first rollup after this ships. Added a ## Deploy Steps section:

manage.py backfill_metrics --days 60 --skip-hourly --skip-monthly

That rebuilds daily from source across the rollup window; the scheduled rollup then derives monthly from a complete tier. --skip-monthly is deliberate — repair daily and let the rollup derive monthly, rather than writing monthly from source here. Needed once; the 04:00 reconciliation pass carries it forward after that.

Ongoing gaps beyond 7 days remain the accepted trade, with backfill_metrics as the documented remedy.

[greptile-apps]

You're right. Given that this tradeoff is explicit, ticket-approved, bounded to the current and previous month, and addressed operationally by the one-time daily backfill, it should not be a merge blocker. The orphan-deletion behavior is a real consequence, but it is now documented rather than an unaddressed regression. I withdraw the “not safe to merge” verdict. The remaining operational caveat is to run the documented backfill before the first rollup and use backfill_metrics for outages or source changes older than the 7-day reconciliation window.

[kirtimanmishrazipstack]

@greptile-apps can you review and generate new confidence score?

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud