CI: gate on model correctness; make vendor-reference actually run - #2
Merged
Conversation
…ally run The vendor-reference job was `if: workflow_dispatch` + `continue-on-error`, so the reference-CSV-based check never ran and could not fail a build. The correctness invariants were only implicitly covered by a bare `pytest -q`. - Add a `correctness-gates` job running the tracking invariant (< 1e-8), cross-validation vs the committed pyfrbus reference (< 1e-6), and the monetary-IRF sanity test as separate, hard-failing steps. - Make `vendor-reference` run on every push/PR without continue-on-error. It now regenerates the reference by actually executing the Fed's pyfrbus from vendor/, gates this implementation against that FRESH solution, and checks the committed anchor has not drifted from the vendor package via the new scripts/check_reference_drift.py. It writes to a scratch path and fails if tests/data/ is modified in place. - Add a weekly `scheduled-validation` job: full suite on Python 3.10-3.12 plus the vendor regeneration gate. - Strengthen the monetary IRF test into an economic-sanity test with explicit sign assertions, a trough-timing assertion, and magnitude bands sourced to Brayton/Laubach/Reifschneider (FEDS Notes 2014) and the FRB/US package documentation, documented in the test docstring. - Document the CI gating table in VALIDATION.md. No tolerance was loosened; all gates pass locally, including a real vendor regeneration (committed reference matches at 8.2e-9 abs / 3.0e-8 rel). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What CI covered before
testjob: ruff + a barepytest -qon Python 3.10/3.12. The tracking and cross-validation invariants existed as tests but were not called out or gated independently.vendor-referencejob that wasif: github.event_name == 'workflow_dispatch'andcontinue-on-error: true— on run 29654867764 it shows as skipped. The reference-CSV path had effectively zero CI coverage and could not fail a build.What this adds
correctness-gatesjob — hard-failing, one step per invariant so a red build names what broke:vendor-referencejob — now runs on every push/PR, nocontinue-on-error. It:vendor/pyfrbus_package(patched copy, throwaway py3.11 venv),test_cross_validation_vs_freshly_generated_vendor, < 1e-6) — so the gate exercises vendor code, not a possibly-stale CSV,scripts/check_reference_drift.pyto confirm the committed anchor still matches the vendor package,tests/data/was modified in place.generate_vendor_reference.shnow takes an optional output path so CI never rewrites the committed anchor. Runtime ~1 min.scheduled-validationjob — weekly (Mondays 05:17 UTC) + on demand: full suite on Python 3.10/3.11/3.12 plus the vendor regeneration gate, to catch transitive-dependency drift without a PR.Economic-sanity test (
test_monetary_irf_economic_sanity, wastest_sanity_signs) — explicit sign assertions, magnitude bands, plus new assertions that the output trough is lagged (quarters 3–15, not on impact) and that output never rises above baseline. The band and its source are documented in the docstring: Brayton, Laubach & Reifschneider, The FRB/US Model: A Tool for Macroeconomic Policy Analysis (FEDS Notes, 2014), and the FRB/US package documentation invendor/frbus_package/documentation/. Bands are deliberately wider than observed values so this gates on economics, not bit-level reproduction.Genuine failures discovered
None. All gates pass locally, including a real vendor regeneration: the freshly generated vendor solution matches the committed reference at 8.2e-9 abs / 3.0e-8 rel (worst variable
wpsn), consistent with the solver-tolerance noise floor documented in VALIDATION.md Test 2b. No tolerance was loosened and no test was skipped.🤖 Generated with Claude Code