Fetch dashboard daily stats from Redis with DB as fallback

[whabanks]

Summary | Résumé

This PR tweaks _get_daily_stats to improve performance. Instead of a round trip to the DB to fetch the daily stats, we first attempt to fetch them from Redis using the RedisAnnualLimit client with the DB as a fallback.

Perf timings before (measured on staging directly):

{
  "template_statistics_weekly": 1400.3448486328125,
  "aggregate_template_usage": 5.429983139038086,
  "get_scheduled_jobs": 45.98712921142578,
  "get_immediate_jobs": 32.1347713470459,
  "add_rate_to_job": 0.000476837158203125,
  "_get_daily_stats": 1246.3033199310303, <---
  "weekly_stats_aggregation": 3.3087730407714844,
  "get_bounce_rate_data": 13.918161392211914,
  "get_annual_data": 3.88336181640625
}

Perf timings after (measured with this branch hooked up to staging):

{
  "template_statistics_weekly": 3350.5444526672363,
  "aggregate_template_usage": 4.066944122314453,
  "get_scheduled_jobs": 147.20726013183594,
  "get_immediate_jobs": 182.31868743896484,
  "add_rate_to_job": 0.000476837158203125,
  "_get_daily_stats": 62.27922439575195, <--
  "weekly_stats_aggregation": 2.112865447998047,
  "get_bounce_rate_data": 346.47130966186523,
  "get_annual_data": 93.42670440673828
}

Test instructions | Instructions pour tester la modification

Basic tests

1. Open the review app and staging
2. Visit /services/8dad0c16-6952-4424-be2a-d98b8bfc3c2d in both
3. Note the performance is improved
4. Note that dashboard counts are the same between the two environments.

---

1. Repeat the above with a service that has sent both today and in previous days
2. Check that the dashboard counts remain consistent between staging and the review env.

Perf via logs

1. Repeat the above test against /services/8dad0c16-6952-4424-be2a-d98b8bfc3c2d
2. Check the cloudwatch logs to see the detailed timings with the following query:

fields @timestamp, @message
| filter @message like /TIMING SUMMARY/
| sort @timestamp desc
| limit 100

[github-actions]

🧪 Review environment

https://bd36m4cy7sa6yvunmsdiakbxbe0vukyc.lambda-url.ca-central-1.on.aws/

[smcmurtry]

The performance is way better for that service (4s load time for the large service vs 6s before), however I am seeing different number for the daily/annual limits.

Staging:

Review app for this PR:

Not sure which set of numbers is correct.

[whabanks]

The performance is way better for that service (4s load time for the large service vs 6s before), however I am seeing different number for the daily/annual limits.

Oh yea they are a little bit off... I tried deleting and re-seeding the data in Redis but it seems the counting may be off. I'll dig into it and see if I can find where the seeding differs from what the DAO fetches.

[whabanks]

The performance is way better for that service (4s load time for the large service vs 6s before), however I am seeing different number for the daily/annual limits.

It seems like the difference we're seeing is because of notifications with a non-terminal statuses i.e. inflights - referred to as requested in code.
The annual_limit keys in Redis only count delivered and failed and don't count statuses like pending, pending-virus-check, created etc. Which is likely the source of, or at least contributing to, the discrepancy here.

Some thoughts on options moving forward:

1. Collect the stats from Redis initially to improve page load performance, then on a subsequent polls only use stats from the DB like we're doing now.
   • I have a version of this working currently, but I'm not sure it is a good experience for users. If you open your dashboard and scroll down immediately to the daily/annual sections, you'll notice numbers "correcting" which would be confusing if you're not currently sending.
2. Create a new API endpoint that collects only inflight / requested notifications. Call Redis for the bulk of the notification counts, then combine with the counts from the "inflights" endpoint.
   • Likely to be faster than the current DB call.
3. Ignore daily and annual stat loading on page load initially and only fetch them when some form of activity occurs on the page (e.g. user begins scrolling down) or the first dashboard poll occurs - whichever comes first.