Storage policy for large data files: plain git in lectures/, LFS only in sources/ — and the ladder above 100 MiB

[mmcky]

Recording the reasoning behind the storage decisions in #55, #56 and #57, so the next session inherits the measurements rather than the conclusions. Supersedes #1 ("Setup Git LFS for large file support"), which asked the question this answers. Context: QuantEcon/workspace-lectures#23 step 3.

Why now

The high_dim_data fold (#2) takes the published tree from a 1,765,204 B maximum file (mpd2020.xlsx) to a 75,902,999 B maximum (SCF_plus_mini_no_weights.csv) in a single PR — a 43× jump, and 72.4% of GitHub's hard blob limit. Total lectures/ goes from 5,582,772 B across 21 files to roughly 115 MB across 27. Until now the repo has had no stated policy for a file that approaches the limit, and the one file that nearly exceeds it (SCF_plus.dta, 103,934,093 B — 99.12% of the limit) arrives in the same fold.

This is not hypothetical elsewhere in the org either: the largest public plain-git data blob QuantEcon ships anywhere is 86,111,224 B (year_origin_sitc_rev2.csv), serving real bytes at 200 with access-control-allow-origin: *.

The limits, verified 2026-08-10

Limit	Value	Notes
Hard blob limit, plain git push	100 MiB (104,857,600 B)	Two GitHub doc pages disagree on units — one says 100 MiB, one says 100 MB. Treat 100 MiB as the safe reading; the MB reading is 4.86% stricter
Push warning threshold	50 MiB (52,428,800 B)	Warns, does not block
Single push	2 GiB	Independent of per-file size
Web-UI upload	25 MiB
Git LFS per file, team plan	4 GB	The doc's own prose contradicts its table, quoting the 5 GB Enterprise figure
Git LFS allowance, team	250 GiB storage + 250 GiB bandwidth per cycle	Pre-paid data packs no longer exist — replaced by metered billing
GitHub Pages	1 GB published site, 100 GB/month soft bandwidth	No documented per-file limit; the artifact deploy path (which this repo uses) is exempt from the 10-builds/hour limit
Release assets	2 GiB per file, 1000 per release, no bandwidth limit	Irrelevant here — see the alternatives table
Repository size	<1 GB ideal, <5 GB strongly recommended	No hard cap; enforcement is at the push and object level

The decision

The published tree (lectures/) is 100% plain git. LFS is confined to sources/** — builder inputs that no lecture reads and that are never served. This is what #57 encodes in .gitattributes, and it is why both workflow checkouts move to lfs: false.

Why LFS is wrong for anything published

Three arguments, in order of how load-bearing they are.

1. The failure mode is silent, not loud. raw.githubusercontent.com returns HTTP 200 with ~133 bytes of pointer text for an LFS-tracked path. Re-measured through pandas: pd.read_csv on such a URL raises nothing and returns a (2, 1) DataFrame whose single column name is version https://git-lfs.github.com/spec/v1. No HTTP status check, no CI job, and no executing notebook build catches that. Meanwhile media.githubusercontent.com is the inverse — it serves LFS paths and 404s plain-git files, routing per path rather than per repo (that is repoint rule 6, now asserted in #55).

2. There is no browser-safe URL form that is invariant under a storage flip. This is the finding that makes the rule structural rather than a preference. https://github.com/<org>/<repo>/raw/<ref>/<path> — the form AGENTS.md calls "works regardless of storage" — is a 302 whose Location is raw.githubusercontent.com for a plain-git path and media.githubusercontent.com for an LFS path. It is storage-agnostic only because the server resolves storage on the author's behalf, and its access-control-allow-origin header is present but empty, with OPTIONS preflight returning 404 — so a browser rejects it before the redirect is followed. A lecture-wasm consumer must therefore name the terminal host directly, and the terminal host is a function of storage.

Worse: LFS makes the correct host a per-file property, while consumer code binds it per-directory. french_rev.md sets one base_url and derives five filenames from it, in both lecture-python-intro and lecture-wasm. LFS-tracking any one of those five silently breaks exactly that one URL while the other four keep working, and there is no edit to base_url that fixes it — the lecture must abandon the idiom and hardcode a host per file.

3. Per-reader cost, roughly 21×, paid in the reader's browser. Measured with Accept-Encoding: gzip explicitly sent, media.githubusercontent.com returns no content-encoding header at all — it never compresses. raw.githubusercontent.com and GitHub Pages both transport-gzip text at level 6. So a reader of mle.md under Pyodide would download 75,902,999 B where plain git over raw sends about 3.4 MB. Git also stores that file at roughly 3.35 MiB packed, where LFS meters the full uncompressed size — so LFS is worse on storage, wire bytes and latency simultaneously for compressible text, which inverts the usual intuition.

One argument to stop making

"LFS is expensive" is refutable from our own billing. Only four repos org-wide use LFS, and the net LFS charge across all of 2026 is $0.04 on $1.04 gross — inside the included Team allowance, and that is with 17 live media-host lecture reads already in production. Anyone defending the no-LFS rule on cost will lose that argument. The correctness, URL-structure and per-reader-latency arguments carry it alone.

The quota facts that are worth recording are different, and they are about failure rather than cost: anonymous public LFS downloads are billed to the repository owner's bandwidth, with no open-source exemption, forks and pulls count against the parent, and under a $0 budget GitHub blocks LFS usage for the rest of the calendar month rather than billing it. At 250 GiB/month, roughly 3,536 reads of one SCF-sized file would exhaust the entire org allowance — after which the next reader gets a hard failure. That is a reader-facing outage triggered by popularity, which is a different risk class from a bill.

The ladder for a file that approaches the limit

Proposed for AGENTS.md. Thresholds are on the file as committed.

1. Subset first — trigger: any published file over 50 MiB. Justify the size in the PR: does the lecture actually read all of it? The house precedent is the SCF chain (103,934,093 B .dta → 75,902,999 B mini → a no-weights mini). A derived teaching extract is a dataset in its own right and gets its own manifest.
2. Plain git — anything up to ~90 MiB. Served gzipped from raw and Pages, packed compressed in git, zero metered quota.
3. gzip in place — over ~90 MiB. SCF_plus.dta goes 103,934,093 B → 17,854,577 B (5.82×, 17% of the limit) under gzip -9, and pandas infers decompression from the URL suffix, so the only consumer change is the filename. Verified that both raw and Pages return application/gzip with identity encoding, so there is no double-decompression trap.
4. Split into ≤50 MiB parts — if compression is not enough.
5. Publish from outside git via the Pages artifact — the genuine escape hatch. This repo is already build_type: workflow; the blob limit binds only because the workflow does cp -r lectures _site/lectures out of the git checkout. Fetching a file during the workflow and copying it into _site would publish it with no git blob at all. Cost: the published file stops being reproducible from a checkout, and the hash gate needs a source other than the working tree.

Hard nevers: LFS anywhere under lectures/; release assets; OSF. Separate arm: a large file that no lecture reads goes to sources/ under LFS — which is exactly where SCF_plus.dta lands.

Caveat to close before rung 3 goes into AGENTS.md: the .gz read is verified in CPython only. It needs one browser-console pd.read_csv against a .csv.gz on the Pages origin, since Pyodide is the consumer class that matters most.

Alternatives, with the fact that decides each

Option	Verdict	Deciding fact
Plain git + raw / Pages	incumbent, correct	raw served a 104,795,654 B plain-git file complete in 1.3 s with access-control-allow-origin: *, range support, and gzip on text/plain
gzip into plain git	the missing rung	5.82× on the worst case; no client change beyond the suffix; both hosts pass application/gzip through identity
Pages artifact from a non-git source	escape hatch, unused	Site is ~5.8 MB against a 1 GB limit; artifact ceiling is 10 GB
Release assets	never	Zero access-control-allow-origin on any hop — the github.com 302 or the release-assets.githubusercontent.com 200 — on HEAD or GET, with or without an Origin header; OPTIONS preflight 404s. Also 302s to a ~45-minute signed Azure URL forcing Content-Disposition: attachment. Fine for Colab, unusable from a browser
Parquet	no	pyarrow is absent from the Pyodide lecture-wasm actually pins (0.27.0, via thebe-lite 0.5.0) and from current stable 0.28.3 — present only in 0.27.7, so availability is non-monotonic. And on this data gzipped CSV (3,443,988 B) beats the best Parquet variant (4,043,639 B)
Zenodo	archival mirror only	The citable zenodo.org/records/<id>/files/<name> form sends no ACAO and sets a session cookie; only the API form is CORS-clean, and it is rate-limited and version-scoped
OSF	no	Sends access-control-allow-credentials and expose-headers but never allow-origin
Harvard Dataverse	viable if a DOI is ever wanted	/api/access/datafile/<id> is CORS-clean end to end, including the presigned-S3 hop
Cloudflare R2 / S3	no	The public r2.dev URL is documented as non-production and rate-limited; production needs a custom domain, per-bucket CORS config, and a long-lived API token in CI (no OIDC for R2)
Hugging Face Datasets	technically viable, rejected	Adds a second account and identity surface for no capability Pages lacks; free public storage is best-effort

What the schema records about storage today: nothing

The manifest has 14 top-level fields and no storage, lfs, host, size or url_form field at any nesting level. Every URL-bearing field (source.url, source.file_url, license.url) points at the third-party upstream, never at where we serve the bytes. The size the audit reports is not a manifest field — it is synthesized at load time by stat() under the private key _size in both build_audit.py and build_catalog.py, and it never round-trips to YAML. migration.yml's closest field, prior_pattern, records the consumer's old URL pattern, not a storage backend.

So if a dataset in lectures/ were LFS-tracked tomorrow, nothing in its manifest, in migration.yml, or on the audit dashboard would say so. The only storage signal anywhere in the toolchain is lfs_media, computed per reference from the consumer's URL string and — until #55 — rendered as a cosmetic note and asserted on nowhere.

Recommendation: do not add a per-dataset storage field. Three reasons, all measured:

• Actual storage is already exactly queryable, for free. git check-attr filter -- lectures/* must print unspecified for every file; a blanket *.csv filter=lfs rule makes it print lfs. A declared field adds a second thing that can be wrong and disagree with the first.
• The serving URL is a repo-level constant, not per-dataset data. Only 2 of 18 manifests mention data-lectures at all, and both incidentally. Every published URL is generated from one of two hardcoded constants — build_catalog.py and render_audit.py — which currently disagree about the URL form. A per-dataset field would write one string 18 times today and 27 after the fold, and make the data.quantecon.org cutover a 27-manifest rewrite instead of a one-value edit.
• The failure originates producer-side. Audit: assert on the media host, and on data-lectures URL resolvability #55's assertions inspect consumer URL strings, so they cannot see an LFS object landing in lectures/ before any consumer exists. A three-line check-attr guard covers that direction, and covers the three files that have no manifest at all (business_cycle_data.csv, business_cycle_info.md, business_cycle_metadata.md) and therefore no hash gate under any keying.

The honest minimal change is one repo-level canonical base URL that the catalog, the dashboard and classify_url all share — plus the check-attr guard. Not a schema field.

Two follow-ups this surfaced

The audit cannot reach its own terminal state green. classify_url recognises exactly six regexes, all over github.com / raw.githubusercontent.com / media.githubusercontent.com. A read from https://quantecon.github.io/data-lectures/lectures/<file> — live, CORS-clean and already working today — matches none of them, falls through to pattern: external-web, and so migrated evaluates false. migration.yml defines final as "every consumer reads https://data.quantecon.org/lectures/…", which means the migration's terminal status is a state the audit is structurally guaranteed to report as broken. Both assertions added in #55 are gated on pattern == "data-lectures", so they also silently stop applying the moment a consumer adopts the canonical host. Not a blocker for #55 — a reason to add the canonical host as a recognised URL form before Phase 4.

PLAN.md still asserts the position this overturns. The "Where we're going" section reads "Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting." That is contradicted by repoint rule 6 in the same document, by the settled-2026-08-06 note, by AGENTS.md, and by the 302 measurement above showing the redirect target is the storage decision. It is the sentence a future session would cite to put an LFS object back into lectures/, and it should go.

Incidentally: data.quantecon.org is NXDOMAIN at its own authoritative nameserver, with the Pages cname null — so PLAN's note about an old AWS box answering that name is stale, and the Phase 4 cutover is unscheduled rather than imminent (#15).

Related idea: a qeld helper package

Raised separately and worth its own thread — recorded here because it bears directly on the structural argument above.

The idea is a small helper package the lectures import, so a data read becomes something like qeld.load("SCF_plus_mini.csv") instead of a hardcoded URL. What makes it interesting in this context is that it would make the claim in PLAN's "Where we're going" true instead of false: the host would live in one package constant rather than in 21 hardcoded URLs across three repos, and a storage or host change would stop being a cross-repo repoint at all. It would also make the github.com/*/raw/ versus raw.githubusercontent.com distinction (repoint rules 5 and 6) invisible to lecture authors, which is the class of mistake that has now cost three sessions.

Constraints it would have to satisfy, none obviously fatal:

• It must install and run in Pyodide/JupyterLite, not just CPython — a pure-Python wheel installed via piplite/micropip. lecture-wasm already has work in flight on exactly this surface (its open PRs on piplite output and a local PyPI warehouse), so the mechanism is being built anyway.
• It must work in Colab and in a plain notebook with no auth and no config.
• It must not hide provenance. The manifest and sha256 are the audit trail; a fetch layer that silently caches or falls back would break the property that a lecture reads exactly the bytes the manifest records. Verifying the hash on load is the obvious upside here, and it is something no URL can do.
• It would break the audit's consumer detection as currently written. The dashboard finds data reads by regexing URLs and quoted filenames out of code cells. A bare qeld.load("SCF_plus_mini.csv") would match the quoted-filename pattern and be classified local-path — i.e. every migrated read would start reporting as an un-migrated local read. The audit would need to learn the call form before any lecture adopts it, or the dashboard inverts.
• It adds a dependency and an install cell to every lecture that reads data, which is a real pedagogical cost in an introductory series, and a per-page cost in wasm.

Note this is a consumer-side fetch layer, distinct from the builder-side entry-point question in #26.

Status

Item	State
.gitattributes scoping LFS to sources/**, lfs: false on both checkouts	#57, open
Media-host and resolvability assertions (repoint rule 6)	#55, open
Hash gate keyed on a recorded hash rather than on consumers	#56, open
The ladder, written into AGENTS.md	not started
check-attr producer-side guard	not started
Canonical base URL / classify_url reaching final	not started
Remove the contradicted line from PLAN.md	not started
qeld helper package	idea only

Re-verifying any of this

Every measurement above is reproducible read-only. The load-bearing ones: curl -sI -H 'Accept-Encoding: gzip' against the two hosts for a known LFS path and a known plain-git path; curl -s -D - -H 'Origin: https://quantecon.github.io' against the github.com/*/raw/ form to see the empty ACAO header and the storage-dependent Location; pd.read_csv against an LFS path on raw to see the silent 2×1 frame; and gh api /organizations/QuantEcon/settings/billing/usage?year=2026 for the per-repo LFS rows.

[mmcky]

The policy has now been exercised end to end, and its size prediction was exact

The high_dim_data fold landed 2026-08-10 (#62, #63), which is the event this issue was written ahead of. Everything it predicted held.

The size projection was right to the significant figure. This issue said lectures/ would go "from 5,582,772 B across 21 files to roughly 115 MB across 27". Measured after the fold: 115,318,266 B across 27 files, largest SCF_plus_mini_no_weights.csv at 75,902,999 B — 72.4% of the hard blob limit, as forecast.

Plain git cost far less than the working tree suggests. After git gc, the entire .git directory is 13 MB (size-pack 12.36 MiB) for a 119 MB working tree. CSV is what zlib is good at: SCF_plus_mini.csv packs 5.37×, and SCF_plus_mini_no_weights.csv packs 21.63×, since weight-expansion makes it largely repeated rows. Readers get the same benefit on the wire — both hosts gzip.

Rung 1 of the ladder was already taken upstream, which is worth noting as precedent. The SCF chain — a 103,934,093 B .dta subset to a 75,902,999 B mini and then to a 32,853,734 B weighted mini — is exactly "subset it", performed years before this policy existed. The ladder codifies what was already house practice rather than inventing it.

Rungs 3–5 were not needed and remain untested. No published file required gzipping, splitting or the Pages-artifact escape.

The one thing this issue did not anticipate

sources/README.md is now load-bearing rather than documentary. #63 added a CI gate asserting that every sources/ file is LFS-captured and hashes to the sha256 recorded under its ## <filename> heading, and that no README entry names a missing file. Without it, sources/ would have carried no validation of any kind while every file in lectures/ is validated as it migrates — and SCF_plus.dta is the provenance root for two published datasets, so a drift would have made both unreproducible silently.

The gate reads the pointer's oid rather than the object, which is the detail that makes it free: it verifies ~100 MiB under a lfs: false checkout at zero LFS bandwidth. Worth folding into this issue's storage guidance, since it generalises — any LFS file's hash is available without fetching it.

The 0.88% margin, now that it has been used in anger

SCF_plus.dta sits 923,507 B under the hard limit. That margin is what makes git check-attr filter -- sources/<file> a gate rather than a formality: below 100 MiB a mis-scoped rule does not error, so the push silently succeeds as plain git and the blob is in history permanently. AGENTS.md now says CI catches this only after the blob is in the branch's history, so run it yourself first.