Publish a machine-readable incumbent-comparison scorecard JSON at a stable path

[PavelMakarchuk]

Problem

The US incumbent comparison (benchmarks/us/incumbent-comparison/) defines the promotion metrics and inputs, but there's no published, machine-readable scorecard artifact a consumer can read. The per-target populace-vs-enhanced-CPS comparison currently only survives as sound_ecps_replacement_comparison.json attached to the older policyengine/populace-us releases (9f1260b, 5da5a95); it was correctly dropped from the live populace surface in populace#37, and the new releases (f32c2e5) carry no comparison at all.

The calibration-diagnostics dashboard wants to show the incumbent comparison as an "archived" view alongside the live calibration diagnostics. It currently falls back to a committed snapshot of the 9f1260b scorecard because there's nothing to read live.

Request

Publish each incumbent-comparison run's scorecard as a JSON artifact at a stable, discoverable path — e.g. archive/us/<candidate-build-id>/scorecard.json (committed, small) and/or a pointer like benchmarks/us/incumbent-comparison/latest.json. A latest.json-style pointer (mirroring populace#9) would let the dashboard resolve the current scorecard without hard-coding a path.

Schema the dashboard consumes

Modeled on the existing sound_ecps_replacement_comparison.json so existing runs are already close:

{
  "schema_version": 1,
  "candidate_release_id": "populace-us-2024-...",
  "incumbent_manifest": "pinned-production-ecps-2024",
  "period": 2024,
  "baseline_label": "enhanced_cps",
  "candidate_label": "populace",
  "summary": {
    "candidate_loss": 0.2279, "baseline_loss": 1.4055, "loss_delta": -1.1775,
    "candidate_holdout_loss": 0.0384, "baseline_holdout_loss": 0.3167,
    "candidate_train_loss": 0.1896, "baseline_train_loss": 1.0888,
    "candidate_unweighted_msre": 0.2331, "baseline_unweighted_msre": 1.3210,
    "candidate_wins": 1040, "baseline_wins": 2613, "ties": 51,
    "n_targets": 3704, "holdout_targets": 739, "train_targets": 2965,
    "candidate_beats_baseline": true,
    "matched_household_count": 41314
  },
  "family_breakdown": [
    {"family": "national_irs_other", "n_targets": 1060, "candidate_wins": 405,
     "baseline_wins": 624, "ties": 31, "candidate_loss_contribution": 2.6e-05,
     "baseline_loss_contribution": 2.0e-04, "loss_delta": -1.8e-04}
  ],
  "top_improvements": [{ "target_name": "...", "candidate_relative_error": ..., "baseline_relative_error": ..., "loss_delta": ... }],
  "top_regressions": [ ... ],
  "gates": { "export": "pass", "support": "pass", "lineage": "pass" }
}

The dashboard reads it read-only and renders the head-to-head (full/holdout/train loss, win/loss/tie, per-family breakdown, top movers). Per-target full rows are optional — the summary + family breakdown + top movers are enough for the scorecard view.

Consumer

PolicyEngine/calibration-diagnostics — the populace mode's "Incumbent comparison" view (PR adds it with a committed 9f1260b snapshot and an env-gated live fetch ready for this artifact).

🤖 Generated with Claude Code