Upstream-freshness dashboard: when was each dataset last checked against its source, and does it still match?

[mmcky]

For every dataset with an upstream source, show when it was last checked against that source and whether it still matches — so upstream drift is something the repo notices, rather than something someone happens to discover while writing a manifest.

The audit dashboard already answers "where does the lecture series read its data from?". This answers the complementary question: "is the data we serve still what the source publishes?" Same generated-not-hand-tended principle.

Why now

The mpd2020.xlsx delta (#39) was found by accident. Nobody was looking for it; it surfaced because writing a manifest happened to involve re-downloading the file, and comparing it took a few minutes. Three header labels had been sitting divergent from upstream since 2023, invisible, and load-bearing for a published lecture.

That is the argument for automating the check. The information needed is already in the manifests — source, integrity.upstream.status, integrity.upstream.date — it is simply never revisited after the day it is written. A manifest's verified: 2026-08-03 is a claim with a shelf life, and nothing currently ages it.

What it would show

One row per dataset with an upstream source:

Column	Source
Dataset	manifest filename
Class	verbatim / constructed / dynamic-snapshot — determines how it is checked
Upstream	source.url, and whether a fetchable source.data_url exists
Last checked	integrity.upstream.date
Age of that check	derived — the number that should drive attention
Status	verified / spot-checked / unverifiable / drift detected
Result of the last automated check	matches / differs / not checkable

With the freshness treatment the audit overview already uses for its build age: green when recent, amber when stale, red when a check has actually failed. "Never checked" and "cannot be checked" must be visually distinct from "checked and fine" — the current catalog's ⚠️ unverifiable conflates unknown with unverifiable, and they are different.

Three check types, because one method does not fit

Grouping the 18 current manifests by what a check would actually mean:

Check	Applies to	Method
Re-fetch and diff	files with a direct upstream file URL	download, compare against integrity.sha256, and on mismatch diff the parsed content so the report says what changed rather than "hash differs"
Re-run the builder	constructed datasets with a committed builder	run it, compare the overlap window with the committed snapshot — this is the validate() stage of the four-stage contract (#14), reused
Not checkable	no source, no builder, or a prose-only provenance	show honestly as such. caron.npy and nom_balances.npy have no recorded source at all; they should read "cannot be checked", never "stale"

The blocker worth knowing before scoping this

Of the 18 manifests today, 16 have an upstream URL but only one has a directly fetchable file URL. The other fifteen point at a landing page, a search interface, or a data portal — https://ec.europa.eu/eurostat/data/database, https://earthquake.usgs.gov/earthquakes/search/, and so on. Those cannot be fetched and compared by a script.

So the first step is not the dashboard, it is the manifests: add a source.data_url (the exact retrievable artifact) wherever one exists, distinct from source.url (the human landing page). mpd2020.xlsx.yml in #38 already carries both and is the model.

For sources with no stable artifact URL — a query interface, an export builder — the honest answer is that the committed builder is the retrieval spec, and the check is "re-run the builder", not "re-fetch the file". Six of the seven verified datasets have committed builders, so that path covers most of what matters.

Current state, for scoping:

	Count
Manifests	18
Have any upstream URL	16
Have a fetchable data_url	1
Have a committed builder (re-runnable)	6
unverifiable — no automated check possible	11
No recorded source at all	2

Where it should live

Extend the existing dashboard rather than building a second one. scripts/build_audit.py already loads every manifest and render_audit.py already has the freshness-badge treatment; this is a section, not a new site. The scheduled audit-dashboard workflow is the natural place to run the checks — weekly is right for this, and the drift-alarm inbox added in #29 already exists to make a failure land somewhere a human sees it.

One design caution: a failed upstream check must not fail the build the way a failed internal-consistency check does. Upstream moving is normal and is not a defect in this repo; it should open or update an issue, not block the Pages deploy. Keep the two failure classes separate, the same way #14 separates a ValidationError from an infrastructure failure.

Relationship to the other threads

• Upstream-delta register: migrate what the lectures use, review data updates after the migration #39 is the human register of known deltas. This issue automates finding them; Upstream-delta register: migrate what the lectures use, review data updates after the migration #39 remains where a delta is reasoned about and resolved. Ideally a detected drift files itself into Upstream-delta register: migrate what the lectures use, review data updates after the migration #39.
• Builder architecture: four-stage template, traceback-clean failures, notebook policy (Phase 5) #14 defines the builder validate() stage that the "re-run the builder" check reuses. Worth building this after that lands, so there is one validation implementation and not two.
• Phase 5's sources-alive canary is adjacent but narrower — it asks "is the source still reachable", this asks "does the source still agree". The canary is the right place to host the fetch machinery.

Part of #8. See PLAN Phase 5 and Phase 7.