Retarget the UK build to FRS 2024-25: re-pin raw vintages, regenerate parity instruments, re-measure gate baselines

[juaristi22]

Closes #723. Workstream E of the UK-migration master epic #665, child of #145. Retargets the UK build to FRS 2024-25 (survey/base 2024, calibration year 2025) per the vintage decision recorded on the issue: the parity bar re-freezes on the incumbent's live artifact enhanced_frs_2024_25.h5 @ the v1.56.14 tag (= incumbent-at-pin ebf733c). Donor vintages (SPI 2022-23, WAS r8, LCFS 2023-24, ETB) are unchanged. The 2023 instruments are preserved in git history at the pre-#723 pins (#687's model).

What moved

1. Release object (ex-E11 discipline) — new uk/frs_release.json + load_uk_frs_release() loader: survey/base/calibration years, time_period, vintage token, SN 9563, DOI 10.5255/UKDA-SN-9563-1, UKDS tab-zip sha, and the HF acquisition pins (repo, revision a2039519… = tag 1.56.14, zip sha 05dd0069… — the HF zip is byte-identical to the UKDS EUL zip). Lockstep tests bind TIME_PERIOD, every frs_table artifact vintage, and take_up_contract.build_year to it.
2. Raw-vintage re-pin — all 21 frs_table artifact entries re-pinned (sha256/size for the 2024-25 tabs, vintage: "2024_25") and upgraded to the SPI-stage convention (tax_year_start: 2024, ukds_study_number: 9563, doi); survey/source prose updated on all FRS stages; typed spec/sources.yaml moved in lockstep (its artifact schema gains the three keys).
3. Build period → "2024" — TIME_PERIOD, HMRC_SPI_BUILD_PERIOD, HMRC_CGT_BUILD_PERIOD; the take-up contract's build_year: 2024 flips two date-keyed rates (tax_free_childcare 0.586→0.88 @2024-04-06, tv_licence_evasion 0.1058→0.1252 @2024-01-01); both drivers stamp source_vintages: {frs: 2024_25} into build sidecars.
4. HMRC published surface re-mapped, not uprated (adjudicated 2026-08-20) — the 2023-24 Tables 3.6/3.7 ODS keeps its bytes, values, and vintage label; mapped_build_period: 2024 with the new explicit token period_mapping: "latest_published_tax_year". The frozen hmrc_income_source_stages.json and its sha pin are untouched; the frozen-copy lockstep test's named-rewrite whitelist carries the re-map; the default HMRC source-contract validator now reconstructs the live canonical payload from source_stages.json (frozen base_candidate retained). Fact-value alignment to the target year stays with Adopt the UK national calibration contract (re-homed from chronicle#164 per the #166 ruling) #707/Consume Ledger UK facts: uk_target_references, mapping module, and a compile-parity gate #622/Adjudicate the UK spine grain at the ingest/calibration seam #688.
5. Parity instruments regenerated from enhanced_frs_2024_25.h5 (sha 97a07f9c…, time_period "2024") — efrs_parity_reference.json (145 columns — surface unchanged; engine 2.89.0) and release_input_coverage_manifest.json (145 required / 0 exclusions; candidate side stays the certified populace_uk_2023 until Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686). Known gaps: none added, none removed. New test pins the record-count identity: household 52,846 = (16,288 raw + 10,000 SPI) × 2 + 270 CGT band donors (30 × 9 HMRC bands; 181 FRS-side + 89 SPI-side). Two vintage findings: no composition-rule drop at 2024-25 (2023-24 dropped 390 raw households) and the donor count moved 780 → 270 (band-file min-lower filter).
6. Gate-side reference pins — the efrs-post-calibration registry descriptor replaced in place with the 2024-25 identity and totals_sha256 e70a4538… (131-column register, width unchanged); mirrors re-pinned in gates.json, weighted_integrity.py, and the data-shard contract. No thresholds move — Retarget the UK build to FRS 2024-25: re-pin raw vintages, regenerate parity instruments, re-measure gate baselines #723 measures, Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686 arms. UK_REFERENCE_DATASET_NAME → enhanced_frs_2024_25_recalibrated.
7. Smoke rung on the ingest driver (adjudicated 2026-08-20) — --sample-fraction (Doctrine: the scale ladder — dev-scale runs before production spend ("boring before billed") #624 rungs) subsamples households post-frs_spine via the generic frame-sampling helpers; sampled rungs refuse checkpoints, record a sampling receipt + rung token, and adopt the named-edge abort. Three full-scale fences gain an explicit receipt-posture on sampled rungs (leaves raw-coverage, SPI stack scaling, ultra-sparse effective-mass restoration); f100 keeps every fence strict.

Signed differences and fixed defects

• HMRC period re-map (above) — signed nearest-available-vintage declaration; no 2024-25 SPI-based HMRC edition exists (~2-year lag). The 208-fact replay honestly becomes "2023-24 published facts replayed against a 2024 build".
• SN 9252 prose defect (fixed and signed) — six FRS stages cited the FRS 2022-23 study number in source prose; all now cite SN 9563 with the DOI, and the release object is the single source.
• WAS bridge-donor pin defect on merged main (fixed and signed) — lcfs_consumption declared the WAS tab with locator: <filename> while was_wealth pins locator: "caller-supplied local input"; _artifact_pins() refused every full-roster licensed run (introduced by the E6 merge-time provenance fixup, after E6's acceptance twins ran; invisible to secrets-free PR CI). Locator aligned; hermetic pin-coherence regression test added.
• Take-up rate flips — semantics of the ported date-keyed rate_at() at build_year 2024; values verified against the committed contract keys.
• Consequence note — the national staging driver derives its build period from the input frame, so staging runs of the certified 2023 line now require the pre-Retarget the UK build to FRS 2024-25: re-pin raw vintages, regenerate parity instruments, re-measure gate baselines #723 checkout (consistent with Legacy 2023-vintage build of the microcosm UK spine (deferred) #687's git-history model); Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686's control build should pin accordingly.

Licensed acceptance (all data local; disclosure-safe evidence only, data/ukds/acceptance/723-frs-2024-25/)

• Raw-vintage audit (raw_vintage_audit_2024_25.json): staged 25-tab set byte-identical to the UKDS zip @ the pinned HF revision; provenance.json written (E6 schema). No consumed-column regressions (benunit's 178-column material-deprivation drop, covesa1-4, month everywhere — all unread). househol.tab confirmed sernum-scrambled (the E2 sort guard now binds for real); SALSAC domain 216/3,743/12,732 of 16,691 (asked-state capture unchanged); EDUCQUAL codes 1–87 both vintages, no map change.
• Twin f100 builds payload-identical (twin_payload_compare.json, exit 0) on the full 21-stage roster.
• Per-stage identity receipts green: e4 (recompute scoped to the survey channel per the post-Create the SPI income channel on the microcosm spine with the FRS-only second stage #717 rule — take-up targets are unweighted int(rate × n_units)), e5, e6.
• Smoke rung f001 end-to-end PASS (spine-smoke-f001.h5 + sampling sidecar + Logbook row) — no abort receipt needed.
• Nonzero shares vs the re-frozen reference: 139/145 columns overlap; 27 beyond ±0.02, all in the already-signed method classes — E6 consumption (donor-faithful vs the incumbent's zero-inflation collapse, e.g. dfe_education_spending ours 0.225 vs ref 0.0003), E5 wealth-QRF, E7 SPI-channel QRF — plus E8-pending salary-sacrifice. Missing columns: capital_gains/student_loan_plan (E8 Port capital gains structure, salary sacrifice, and student loan stages #684), three E9-class derived-benefit columns, household_weight (bookkeeping). Weighted-distributional comparison binds at Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686.
• owned_land stability re-derived at 2024-25 (owned_land_stability_receipt.json): 71.4% national / 112.9% max-region seed swing — the uk-data#448 class persists (37.7% at 2023-24), evidencing the exclusion renewal. Seed0 cross-check annotated as the Carry manifest rewrites into the shared Stage/StageRecord producer graph #721 in-place-rewrite class (in-build determinism = the payload-identical twins).
• Baselines recorded (uk_weighted_integrity_baselines_2024_25.json): 174 input-mass totals + 47 QRF tail grids at spine grain; incumbent-side totals evidence (uk_input_mass_reference_2024_25.json, 131 columns) digest-bound into the gate registry. These are the candidate thresholds Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686 arms.

Adjudications (@juaristi22, 2026-08-20 — confirmed in session)

1. Input-mass exclusion re-signings confirmed: charitable_investment_gifts (expires 2027-02-20) and owned_land (expires 2026-09-20, fresh 2024-25 stability receipt above). The registry scope notes were tightened afterward to state the precise fact (the 2024-25 incumbent carries the SPI-synthetic rows structurally but no admin-restored mass in the channel-exclusive columns); the approved exclusion reasons are verbatim-untouched.
2. UK_REFERENCE_DATASET_NAME = "enhanced_frs_2024_25" — the _recalibrated suffix was dropped per adjudication: the label now names the pinned artifact exactly (no recalibrated variant exists at this vintage; the June report strings keep their own label).
3. Shares-divergence classification confirmed: the 27 columns rest on the existing signed method classes; no new class minted at this seam (the weighted-distributional comparison at Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686 is where one would be, if residuals survive the E8 composition close).

Adversarial review (Codex, 2026-08-20) — dispositions

Two high findings, both dispositioned in Adversarial-review dispositions (last commit):

• Accepted — evidence fields must bind to the bytes their hash names: the family-coverage block rendered the re-mapped period fields from the canonical manifest while hashing only the frozen mirror. Now carries a dual pin (source_manifest = the frozen June identity; canonical_source_manifest = the bytes the re-mapped fields come from), with a test binding each field set to the sha256 of its actual source and asserting the frozen mirror still declares the June mapping.
• Rejected with armor — do not regenerate the June replay report: the committed hmrc_income_replay_report.json deliberately records mapped_build_period: 2023; it is evidence for the grandfathered June release, not the 2024 line, and retires with the frozen manifest after Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686 (Legacy 2023-vintage build of the microcosm UK spine (deferred) #687's disposition). Regenerating it would destroy the June anchor. Instead, a new assertion binds it to the frozen manifest's declared mapping, making the freeze partition self-describing. (This is the one additive touch to test_uk_hmrc_replay_artifacts.py; every pre-existing assertion is unchanged.)

Also in this round: rebased onto main past #729 (uk-national-calibrate) — conflicts resolved by keeping #729's structures with our values applied, and the gate-battery digests re-cut over the union from the live producer payload.

UK-specificity note

UK-specific: the release object and its loader (FRS/UKDS identity), the manifest re-pins, the HMRC period re-map, take-up contract year. General: the ingest-driver rung support (generic frame_sampling helpers), the sampled-rung fence postures (any country's ladder can use the pattern), the pin-coherence regression test shape, and the three added artifact-schema keys.

Implemented by Codex under /codex-implement-plan (4 recorded deviations, all reviewed: uv cache relocation; coverage-manifest regenerated in two commits; HMRC contract validator reconstruction — kept, it is the mechanism for signed re-maps over a frozen original; frs_release.json allowlisted in the immutable-reference naming test). Claude review added the WAS-pin fix, the sampled-rung fence postures, and the e4-receipt scoping — all licensed-run findings PR CI cannot see.

🤖 Generated with Claude Code

[vahid-ahmadi]

Automated review pass (Claude Code, low-effort single diff pass — no verification run, and the bulk pin/sha/nonzero-share data churn was not validated).

1. tools/build_uk_frs_spine.py (~4398, new except ValueError block) — the non-named-edge fallback duplicates the pre-existing except Exception handler verbatim (error receipt → apply_error_verdict → _record_attempt(disposition="failed") → return 1). Any future change to failure handling has to be made twice and the copies will silently diverge. Suggest re-raising when the named-edge signature is absent and letting the generic handler run.

2. tools/verify_uk_identity_stability.py (~4612, _frs_only_frame) — rebuilds the frame via uk_national_frame(...) without passing mass_log, unlike the sibling _engine_safe_frame just below which does. The E4 identity receipt then runs against a frame whose mass log is empty, losing the mass provenance the loaded artifact carried.

3. packages/microcosm-build/src/microcosm/build/uk/cgt_source_stages.json:96,119 — minimum_allocation_people and declared_factor retyped from 1.0 to 1 (JSON int). If anything does an isinstance(..., float) or exact-float check on these it's a behavior change unrelated to the retarget; if it's just canonicalization it's noise in a signed-manifest diff whose shas moved.

4. packages/microcosm-build/src/microcosm/build/uk_runtime/terminal_gates.py:2881 — UK_CANDIDATE_DATASET_NAME stays "populace_uk_2023" while the reference name and build period both move to 2024; the candidate label now names the wrong vintage in gate reports.

[juaristi22]

Thanks @vahid-ahmadi — dispositions of the four findings (fixes cherry-picked from the WS-D worktree where they first landed, commit Address the #733 automated review):

1. Confirmed, fixed — the rung named-edge ValueError path duplicated the generic failure handler verbatim. The two handlers are now one except Exception with the named-edge branch first (a plain re-raise can't reach a sibling except clause of the same try, so the merge is structural rather than a re-raise). The named-edge branch still writes the .rung_abort.json receipt and exits 3; everything else falls through to the single error-receipt path.
2. Confirmed, fixed — _frs_only_frame now passes mass_log=frame.mass_log through, matching its sibling _engine_safe_frame; the scoped E4 instrument frame keeps the artifact's mass provenance. Re-verified against the licensed spine-a: the e4 identity receipt still passes with the mass log carried.
3. Reviewed, no change — the 1.0 → 1 retypes in cgt_source_stages.json sit in descriptive operation blocks with no type-sensitive consumer: the only JSON read coerces through float(entry["declared_factor"]) (national_build.py), minimum_allocation_people has no Python reader, and the operative declared_factor values are minted as Python literals in the runtime. Reverting the canonicalization would re-cut three digests for zero behavioral difference.
4. Intentional — UK_CANDIDATE_DATASET_NAME stays populace_uk_2023 by the Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686 partition: the candidate side of every gate comparison genuinely remains the certified 2023 artifact until the swap decision; only the reference side re-freezes at Retarget the UK build to FRS 2024-25: re-pin raw vintages, regenerate parity instruments, re-measure gate baselines #723. The label moves with the candidate at Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686, alongside the rest of the candidate-side pins.

[juaristi22]

Spine build vs the eFRS incumbent (FRS 2024-25)

Head-to-head of the microcosm-built spine against the re-frozen parity reference extracted from the incumbent's enhanced_frs_2024_25.h5 (uk-data v1.56.14 tag, rev a2039519…, sha 97a07f9c…). Licensed run, local only; disclosure-safe aggregates below (shares, counts — no unit records).

What this instrument is: unweighted per-column nonzero shares from --emit-nonzero-shares on the full 19-stage roster, against the reference's 145-column surface. It is a coverage and mechanism screen, not a distributional one — the weighted comparison binds at #686.

Structural

	spine (ours)	reference (incumbent)
households	26,288	52,846
persons / benunits	56,492 / 30,455	113,617 / 61,223
SPI-synthetic share	0.3804	0.3801
household weight total	28,972,433	— (post-calibration)
zero-weight rows	0	—

Ours is the pre-clone product, the reference the post-clone artifact — the #688-adjudicated grain difference: (16,288 FRS + 10,000 SPI) × 2 clones + 270 CGT band donors = 52,846. Counts therefore aren't comparable directly, but the synthetic composition is (0.3804 vs 0.3801), which is what makes the share screen meaningful: no divergence below is explained by channel-mix dilution. Twin builds payload-identical; prior mass splits exactly in half across channels (14,486,216.5 each).

Share agreement

• 139 / 145 reference columns present on the spine
• 112 within ±0.02 — max |Δ| among them 0.0183
• 27 beyond ±0.02, every one traced to a producing stage below
• 60 spine-only columns (channel/provenance carriers and net-new outputs)

The 27, by producing stage

E6 consumption — 16 (lcfs_consumption 14, etb_services 2). The donor-faithful-vs-incumbent-collapse class established in #730, re-measured at the new vintage; the incumbent's comonotone draws collapse zero-inflated targets. Sharpest: dfe_education_spending +0.2246 (ours 0.2249 vs 0.0003), bus_subsidy_spending +0.2401, restaurants_and_hotels_consumption +0.1589, petrol_spending −0.1445, education_consumption −0.1090. Also electricity/gas/domestic-energy, furnishings, miscellaneous, communication, alcohol-and-tobacco, diesel, transport, health, recreation.

E5 wealth — 6 (was_wealth, 2 rewritten by regional_property_uprating). The correlated-rank-draw class from #714: savings −0.0507, property_wealth −0.0466, corporate_wealth −0.0427, main_residence_value −0.0399, other_residential_property_value −0.0374, student_loan_balance +0.0299 (the incumbent-side allocation anomaly E5 signed).

E7 SPI channel — 3 (rewritten by hmrc_spi_income_spine / hmrc_spi_income): tax_free_savings_income −0.0545, employer_pension_contributions −0.0470, savings_interest_income −0.0305 — the U14 QRF-surface class already signed on #717.

E8-pending — 1: pension_contributions_via_salary_sacrifice −0.0740. Ours is reported-only until #684 ports the asked-subset imputation; the asked-state capture itself is verified at this vintage (216 yes / 3,743 no / 12,732 not-asked of 16,691 job rows).

One divergence not covered by an existing class

water_and_sewerage_charges +0.1009 (ours 0.8779 vs 0.7769). Unlike the other 26 this is produced by frs_spine — a direct raw mapping, not an imputation, so it falls outside every signed class. It is not a composition or channel artifact: the share is uniform across our two channels (FRS-side 0.8784, SPI-side 0.8770) and the synthetic mix matches the reference. Candidate mechanisms (unverified): the incumbent zeroing bundled water charges for Scotland/NI or by tenure, versus our unconditional watsewrt/cwatamtd/csewamt mapping. Proposed disposition: investigate before #686 and either fix-and-sign or register as a new signed difference — flagging rather than folding it into a neighbouring class. @juaristi22 for the call on whether that lands here or as an E10 item.

Missing from the spine — 6, all mechanism-known

capital_gains (0.2338) and student_loan_plan (1.0) → #684 (E8); free_school_meals (0.0342), free_school_fruit_veg (0.0045), healthy_start_vouchers (0.0022) → the E9 derived-benefit class; household_weight (1.0) is shares-emission bookkeeping, not a gap.

What this does and does not establish

Establishes: no degenerate or missing raw mapping at the new vintage; every imputation stage produces signal on the expected surface; all divergence beyond noise is attributable to a named stage. Does not establish distributional agreement — unweighted shares are blind to level and tail, and the reference carries CGT doubling and band donors we don't produce until #684. Weighted totals across the 131-column register, and the composition close, bind at #686.

Evidence: data/ukds/acceptance/723-frs-2024-25/ — shares-a.json / shares-b.json, twin_payload_compare.json, acceptance_receipt.json, raw_vintage_audit_2024_25.json.