Add TY2023 Form 8960 line anchors and Table 1.4 pass-through facts

FINAL_REPORT.md

@@ -0,0 +1,163 @@
+# Final report: chronicle#179 passive pass-through anchors
+
+## Result
+
+Implemented and validated the TY2023 passive pass-through administrative
+anchors requested in chronicle#179. The work adds 20 Form 8960 facts and
+extends the existing Publication 1304 Table 1.4 record set with 160 partnership
+and S corporation facts, for 180 net-new TY2023 facts. Every affected record set
+and fact has `provenance_class: administrative`.
+
+No network access, fetch, push, or pull request was used.
+
+## Record sets and fact counts
+
+Form 8960 package ID: `soi-form-8960-2023`.
+
+Each Form 8960 record set has one all-filers row and two facts: a return count
+and a USD amount. The ten record sets therefore emit 20 facts:
+
+- `irs_soi.ty2023.form_8960.line_1`
+- `irs_soi.ty2023.form_8960.line_2`
+- `irs_soi.ty2023.form_8960.line_3`
+- `irs_soi.ty2023.form_8960.line_4a`
+- `irs_soi.ty2023.form_8960.line_4b`
+- `irs_soi.ty2023.form_8960.line_4c`
+- `irs_soi.ty2023.form_8960.line_5d`
+- `irs_soi.ty2023.form_8960.line_8`
+- `irs_soi.ty2023.form_8960.line_12`
+- `irs_soi.ty2023.form_8960.line_17`
+
+Publication 1304 package ID: `soi-table-1-4`.
+
+- Existing record set extended: `irs_soi.ty2023.table_1_4`.
+- TY2023 total: 20 AGI rows × 37 measures = 740 facts.
+- Net-new partnership and S corporation slice: 20 rows × 8 measures = 160
+  facts.
+- Complete Schedule E entity slice, including the four existing estate/trust
+  measures: 20 rows × 12 measures = 240 facts.
+- TY2020 remains at its original 29 measures and 580 facts because that
+  publication combines partnership and S corporation values. The new separate
+  measures are source-accurately scoped to TY2021–TY2023.
+
+Amounts are normalized from published thousands of dollars to `unit: usd`
+with `value_scale: 1000`; return counts use `unit: count`. Table 1.4 net-loss
+amounts remain the publisher's positive magnitudes.
+
+## Source identity and scope notes
+
+Form 8960 is pinned to:
+
+- URL: `https://www.irs.gov/pub/irs-pdf/p4801.pdf`
+- Vintage: `tax_year_2023_rev_6_2026`
+- Publication identity: Publication 4801 (Rev. 6-2026), pages 214–215
+- Curated source artifact SHA-256:
+  `a831947133806455b971560102bf6d7710865ece481616844bead7c9b81958fd`
+- Supervisor-verified extract SHA-256:
+  `1cafe8a381e83c5365b6a2ef54695ac6b4beca369eb61338169e24f2581ef080`
+- Full Publication 4801 transcript SHA-256:
+  `0c157394542a0b3e2cbfe361d6ac4c744568c6476bada9d7a7647470b54a9364`
+
+Table 1.4 is pinned to:
+
+- URL: `https://www.irs.gov/pub/irs-soi/23in14ar.xls`
+- SHA-256:
+  `b6c1f87fbb5533417e195f6938538e5de09b6a0825a6a54346bf9363a18d96af`
+- Size: 115,712 bytes
+
+The source-package description and manifest document the identified data
+boundary: IRS SOI entity-side partnership tables 23pa01, 23pa04, 23pa06,
+23pa10, and 23pa23 provide no passive/nonpassive split by industry, while Table
+1.4 provides entity income and loss without a passive split. Form 8960 line 4
+is therefore the sole administrative anchor for the passive pass-through NIIT
+base.
+
+## Form 8960 line 4 reconciliation
+
+The publisher's displayed amounts in thousands do not satisfy literal equality:
+
+- Line 4a: `$1,185,607,258,000`
+- Line 4b: `-$1,076,350,273,000`
+- Their sum: `$109,256,985,000`
+- Published line 4c: `$109,256,984,000`
+- Residual: `$1,000`, exactly one published reporting unit
+
+The ingestion preserves all publisher values. The unit-level consistency test
+asserts the exact `$1,000` residual and reconciliation within one reporting
+unit; forcing exact equality would falsify one of the supplied source values.
+
+## Microcosm concept IDs
+
+The primary passive-plus-rental NIIT base is:
+
+- `irs_soi.form_8960.line_4c`
+
+Its associated line 4 inputs and counts are:
+
+- `irs_soi.form_8960.line_4a`
+- `irs_soi.form_8960.line_4a.return_count`
+- `irs_soi.form_8960.line_4b`
+- `irs_soi.form_8960.line_4b.return_count`
+- `irs_soi.form_8960.line_4c.return_count`
+
+The other Form 8960 concepts are:
+
+- `irs_soi.form_8960.line_1`
+- `irs_soi.form_8960.line_1.return_count`
+- `irs_soi.form_8960.line_2`
+- `irs_soi.form_8960.line_2.return_count`
+- `irs_soi.form_8960.line_3`
+- `irs_soi.form_8960.line_3.return_count`
+- `irs_soi.form_8960.line_5d`
+- `irs_soi.form_8960.line_5d.return_count`
+- `irs_soi.form_8960.line_8`
+- `irs_soi.form_8960.line_8.return_count`
+- `irs_soi.form_8960.line_12`
+- `irs_soi.form_8960.line_12.return_count`
+- `irs_soi.form_8960.line_17`
+- `irs_soi.form_8960.line_17.return_count`
+
+The Table 1.4 entity concepts are:
+
+- `irs_soi.partnership_net_income`
+- `irs_soi.returns_with_partnership_net_income`
+- `irs_soi.partnership_net_loss`
+- `irs_soi.returns_with_partnership_net_loss`
+- `irs_soi.s_corporation_net_income`
+- `irs_soi.returns_with_s_corporation_net_income`
+- `irs_soi.s_corporation_net_loss`
+- `irs_soi.returns_with_s_corporation_net_loss`
+- `irs_soi.estate_and_trust_net_income`
+- `irs_soi.returns_with_estate_and_trust_net_income`
+- `irs_soi.estate_and_trust_net_loss`
+- `irs_soi.returns_with_estate_and_trust_net_loss`
+
+## Validation results
+
+- `ruff check chronicle policyengine_chronicle db scripts tests`: passed.
+- Governance, boundary, facts-only, consumer-contract, alias-drift, and Form
+  8960 focused tests: 67 passed.
+- Full `pytest -q`: 650 passed, 1 skipped, 0 failed in 27m22s. The skip and 14
+  dependency warnings are pre-existing/non-blocking.
+- `validate-package soi-form-8960-2023 --year 2023`: valid; 10 record sets, 20
+  measures, 20 source records, 10 regions.
+- `build-suite soi-form-8960-2023 --year 2023`: valid; 20 facts and consumer
+  facts, 66 source cells, 10 source rows, 10 regions, lineage coverage 1.0, no
+  acceptance errors.
+- `validate-package soi-table-1-4 --year 2023`: valid; 1 record set, 37
+  measures, 20 rows, 740 source records, 1 region.
+- `build-suite soi-table-1-4 --year 2023`: valid; 740 facts and consumer facts,
+  8,109 source cells, 1 region, lineage coverage 1.0, no acceptance errors.
+- Direct Table 1.4 fact builds for TY2020–TY2023: valid; 580, 740, 740, and 740
+  facts respectively.
+- Canonical merged-bundle test: passed; 145,402 facts across 121 included source
+  packages, including 33,957 IRS SOI facts.
+
+The repository has no Makefile. All checks used its Python harness and pytest
+suite through the existing offline virtual environment.
+
+## Changelog
+
+The functional change fragment is `changelog.d/179.added.md`, following the
+parent PolicyEngine repository convention. `CHANGELOG.md` was not edited
+manually.

PROGRESS.md

@@ -0,0 +1,51 @@
+# Progress: chronicle#179 passive pass-through anchors
+
+## State
+
+Complete. Both source ingestions, repository-wide snapshots, changelog, and
+final validation are finished on branch `passive-pass-through-anchors` in
+`.claude/worktrees/passive-179`.
+
+## Done
+
+- Read the applicable repository instructions in `AGENTS.md` and the parent
+  `CLAUDE.md`.
+- Confirmed that the existing `irs_soi.table_1_4` package already pins the
+  supplied TY2023 workbook, so this task will extend that package rather than
+  duplicate it.
+- Mapped record-set, fact, provenance, source-identity, test, and changelog
+  conventions; the repository has no Makefile or established changelog file.
+- Added `soi-form-8960-2023`: 10 tax-year record sets and 20 administrative
+  facts (return count plus USD amount for each requested line), pinned to
+  Publication 4801 (Rev. 6-2026) and its supervisor-verified transcripts.
+- Added source-level tests for every Form 8960 value, unit, provenance pin, and
+  the published one-$1,000 line 4 rounding residual.
+- Passed the focused tests (10 tests), source-package validation, and the full
+  Form 8960 source-suite build (20 consumer facts; all reports valid).
+- Extended the existing `irs_soi.ty2023.table_1_4` record set with partnership
+  and S-corporation net-income/net-loss return counts and amounts across all 20
+  AGI classes (160 new facts); the existing estate/trust measures remain intact.
+- Added strict per-measure year scoping so TY2020 retains its combined
+  partnership/S-corporation layout while TY2021 through TY2023 ingest the
+  separately published lines.
+- Passed Table 1.4 source-package compilation for TY2020 and TY2023, focused
+  package/suite tests, Ruff checks, and full fact builds for TY2020 through
+  TY2023 (580 facts in 2020; 740 facts in each later year).
+- Ran the full 651-test suite once: 649 passed, one skipped, and the sole
+  failure was the expected stale merged-bundle count snapshot. Updated that
+  snapshot from the generated bundle's exact coverage report (+180 facts and
+  +1 source package).
+- Rebuilt the canonical merged bundle after the snapshot update; its complete
+  aggregate-contract test passed (1 test in 12m31s).
+- Added the required functional-change fragment at
+  `changelog.d/179.added.md`, following the parent repository convention.
+- Passed the final repository-wide checks: Ruff; 67 targeted governance,
+  boundary, facts-only, consumer-contract, alias, and Form tests; both package
+  validators and source-suite builds; and the full suite with 650 passed, one
+  skipped, and no failures.
+- Wrote the requested handoff, record-set/fact inventory, exact Microcosm
+  concept IDs, and check results to `FINAL_REPORT.md`.
+
+## Next
+
+- No implementation work remains. The branch is ready for supervisor review.

changelog.d/179.added.md

@@ -0,0 +1 @@
+Added TY2023 IRS SOI Form 8960 passive pass-through NIIT anchors and Publication 1304 Table 1.4 partnership and S corporation Schedule E facts by AGI class.

chronicle/source_package.py

@@ -219,6 +219,7 @@
     "soi-filing-season-week47-2024-eitc-total": Path(
         "irs_soi/filing_season_week47_2024"
     ),
+    "soi-form-8960-2023": Path("irs_soi/form_8960_2023"),
     "soi-table-4-3": Path("irs_soi/table_4_3"),
     "soi-state-2022": Path("irs_soi/state_2022"),
     "soi-congressional-district-2022": Path("irs_soi/congressional_district_2022"),
@@ -908,6 +909,7 @@ def to_record_set_spec(self, year: int) -> SourceRecordSetSpec:
         measures = tuple(
             _measure_from_mapping(measure, year=year)
             for measure in _required(self.payload, "measures", "record_set")
+            if _mapping_applies_to_year(measure, year=year, context="measure")
         )
         return SourceRecordSetSpec(
             record_set_id=_render_required_string(
@@ -2114,6 +2116,35 @@ def _required(payload: dict[str, Any], key: str, context: str) -> Any:
     return payload[key]
 
 
+def _mapping_applies_to_year(
+    payload: dict[str, Any],
+    *,
+    year: int,
+    context: str,
+) -> bool:
+    """Return whether a declarative mapping applies to the requested year."""
+    years = payload.get("years")
+    if years is None:
+        return True
+    identifier = payload.get(f"{context}_id")
+    context_label = f"{context} {identifier!r}" if identifier else context
+    if not isinstance(years, list) or not years:
+        raise ValueError(f"{context_label} years must be a non-empty list")
+    if any(isinstance(item, bool) or not isinstance(item, int) for item in years):
+        raise ValueError(f"{context_label} years entries must be integers")
+    if len(set(years)) != len(years):
+        raise ValueError(f"{context_label} years entries must be unique")
+    if "column_by_year" in payload:
+        columns = payload["column_by_year"]
+        for declared_year in years:
+            if declared_year not in columns and str(declared_year) not in columns:
+                raise ValueError(
+                    f"{context_label} column_by_year has no entry for "
+                    f"declared year {declared_year}"
+                )
+    return year in years
+
+
 def _year_mapping(files_by_year: dict[Any, Any], year: int) -> dict[str, str]:
     if year in files_by_year:
         return files_by_year[year]

db/data/irs_soi/form_8960_2023/irs_soi_form_8960_2023.csv

@@ -0,0 +1,11 @@
+form_line,label,return_count,amount_thousands,count_page,amount_page
+line_1,Taxable interest,8200131,209033554,214,215
+line_2,Ordinary dividends,6877553,349823646,214,215
+line_3,Annuities,176061,6970315,214,215
+line_4a,"Rental real estate, royalties, partnerships, S corporations, trusts, trades or businesses, etc.",4988033,1185607258,214,215
+line_4b,Adjustment for net income or loss derived in the ordinary course of a non-section 1411 trade or business,4038235,-1076350273,214,215
+line_4c,Combine lines 4a and 4b,2260296,109256984,214,215
+line_5d,Combine lines 5a through 5c,6432002,556322852,214,215
+line_8,Total investment income,8976709,1222447069,214,215
+line_12,Net investment income,8211516,1197238417,214,215
+line_17,Net investment income tax for individuals,8104840,39330017,214,215

db/data/irs_soi/form_8960_2023/manifest.yaml

@@ -0,0 +1,42 @@
+source_id: irs_soi
+package_id: soi-form-8960-2023
+dataset: irs_soi_publication_4801_form_8960_2023
+source_page: https://www.irs.gov/pub/irs-pdf/p4801.pdf
+publication_revision: Publication 4801 (Rev. 6-2026)
+table: Form 8960, Net Investment Income Tax—Individuals, Estates, and Trusts,
+  pages 214–215
+files:
+  2023:
+    filename: irs_soi_form_8960_2023.csv
+    source_url: https://www.irs.gov/pub/irs-pdf/p4801.pdf
+    sha256: a831947133806455b971560102bf6d7710865ece481616844bead7c9b81958fd
+    size_bytes: 795
+    source_table: >-
+      Publication 4801 (Rev. 6-2026), TY2023 Form 8960 facsimile, number of
+      returns and amounts of selected lines
+    year: 2023
+    notes: >-
+      Curated CSV transcription of the ten requested Form 8960 lines from the
+      supervisor-verified p4801-form8960-extract.txt (SHA-256
+      1cafe8a381e83c5365b6a2ef54695ac6b4beca369eb61338169e24f2581ef080),
+      checked against the full Publication 4801 text extraction p4801.txt
+      (SHA-256
+      0c157394542a0b3e2cbfe361d6ac4c744568c6476bada9d7a7647470b54a9364).
+      The canonical publisher artifact is Publication 4801 (Rev. 6-2026) at
+      source_url. Page 214 reports return counts; page 215 reports amounts in
+      thousands of dollars. The CSV keeps the published integers and signs
+      verbatim. SOI's entity-side partnership tables 23pa01, 23pa04, 23pa06,
+      23pa10, and 23pa23 publish no passive/nonpassive split by industry, while
+      Publication 1304 Table 1.4 reports partnership, S-corporation, and
+      estate/trust income and loss without a passive split. Form 8960 line 4 is
+      therefore the sole administrative anchor for the passive pass-through
+      NIIT base. The three independently reported line 4 values have a
+      one-thousand-dollar rounding residual: line 4a plus line 4b is
+      109,256,985 thousand dollars, while published line 4c is 109,256,984
+      thousand dollars. License: US government work, public domain.
+    storage:
+      r2:
+        provider: r2
+        bucket: ledger-raw
+        key: raw/irs_soi/soi-form-8960-2023/2023/a831947133806455b971560102bf6d7710865ece481616844bead7c9b81958fd/irs_soi_form_8960_2023.csv
+        uri: r2://ledger-raw/raw/irs_soi/soi-form-8960-2023/2023/a831947133806455b971560102bf6d7710865ece481616844bead7c9b81958fd/irs_soi_form_8960_2023.csv

db/data/irs_soi/table_1_4/manifest.yaml

@@ -3,6 +3,14 @@ package_id: soi-table-1-4
 dataset: irs_soi_publication_1304_table_1_4
 source_page: https://www.irs.gov/statistics/soi-tax-stats-individual-income-tax-returns-complete-report-publication-1304-basic-tables-part-1
 table: Table 1.4
+notes: >-
+  Table 1.4 publishes partnership, S corporation, and estate/trust net-income
+  and net-loss return counts and amounts by AGI class, but it does not identify
+  the passive share. IRS SOI entity-side partnership tables 23pa01, 23pa04,
+  23pa06, 23pa10, and 23pa23 likewise publish no passive/nonpassive split by
+  industry. Publication 4801 Form 8960 line 4 is therefore the sole
+  administrative anchor for the passive pass-through NIIT base. Table 1.4 loss
+  amounts are reported and ingested as positive magnitudes.
 files:
   2020:
     filename: 20in14ar.xls