llm-reachability: provider-exception batches are never counted in batches_dropped/units_not_reviewed — the step report can read success/0/0 for a pass that reviewed nothing
From the 2026-09-07 monitored e2e run and its verification rounds; re-derived from source at current master (ec6b063); pre-existing on master.
The claim
In core/llm_reachability.py, a batch whose LLM call raises (empty completion, transport error, rate-limit exhaustion) takes the provider-exception branch: except Exception builds the failure message, prints it to stderr, and continues — BEFORE the _count_drop closure is ever defined or passed to parse_response (:457-463 exit; :465-476 the parse path where dropped_batches/units_not_reviewed increment, via on_batch_drop). The counters therefore count PARSE-path drops only. The scanner passes no on_error (core/scanner.py:637-643), so the exception message never reaches ctx.errors either — the step summary at :871-872 can read status: success, errors: [], batches_dropped: 0, units_not_reviewed: 0 while every unit in the exception-failed batches received no reachability review.
This is the exact report shape PR #386's body names as the harm it fixed ("status: success, errors: [], units_reviewed: 10845" while 10 of 434 batches were lost) — but #386's counting is scoped to on_batch_drop, the parse-level path. The provider-exception branch is named nowhere in that PR: not fixed, not deferred, not residual — an uncovered sibling path of the same family.
Evidence
A monitored run's first attempt recorded four provider-exception batches ("returned an empty completion") in its event log — batches 1/216, 2/216, 3/216, 5/216 — before the attempt was killed; the report for that pass was never written, so the report-side consequence is source-derived from the code path, not from a surviving artifact. The same run's final pass, whose report survives, shows the counters working on the parse path (a malformed batch was counted). No surviving report anywhere counts the exception batches.
Direction
Count the exception batch on the exception path: units_not_reviewed += len(batch) plus a batches_failed counter distinct from batches_dropped (different failure class, different remediation), and surface the failure message into the step's errors via the existing on_error hook so it reaches ctx.errors and the step report. Tests: an exception batch increments the failed/unreviewed counters; a malformed batch still increments the dropped counters; a mixed pass reports both.
Related: #294/#386 (the parse-path counting this extends), #538 (the deferred-recovery tracker for the malformed class), #537 (the same raise's cost-accounting sibling), #532 (the resume re-run surface).
llm-reachability: provider-exception batches are never counted in batches_dropped/units_not_reviewed — the step report can read success/0/0 for a pass that reviewed nothing
From the 2026-09-07 monitored e2e run and its verification rounds; re-derived from source at current master (ec6b063); pre-existing on master.
The claim
In
core/llm_reachability.py, a batch whose LLM call raises (empty completion, transport error, rate-limit exhaustion) takes the provider-exception branch:except Exceptionbuilds the failure message, prints it to stderr, andcontinues — BEFORE the_count_dropclosure is ever defined or passed toparse_response(:457-463exit;:465-476the parse path wheredropped_batches/units_not_reviewedincrement, viaon_batch_drop). The counters therefore count PARSE-path drops only. The scanner passes noon_error(core/scanner.py:637-643), so the exception message never reachesctx.errorseither — the step summary at:871-872can readstatus: success, errors: [], batches_dropped: 0, units_not_reviewed: 0while every unit in the exception-failed batches received no reachability review.This is the exact report shape PR #386's body names as the harm it fixed ("
status: success, errors: [], units_reviewed: 10845" while 10 of 434 batches were lost) — but #386's counting is scoped toon_batch_drop, the parse-level path. The provider-exception branch is named nowhere in that PR: not fixed, not deferred, not residual — an uncovered sibling path of the same family.Evidence
A monitored run's first attempt recorded four provider-exception batches ("returned an empty completion") in its event log — batches 1/216, 2/216, 3/216, 5/216 — before the attempt was killed; the report for that pass was never written, so the report-side consequence is source-derived from the code path, not from a surviving artifact. The same run's final pass, whose report survives, shows the counters working on the parse path (a malformed batch was counted). No surviving report anywhere counts the exception batches.
Direction
Count the exception batch on the exception path:
units_not_reviewed += len(batch)plus abatches_failedcounter distinct frombatches_dropped(different failure class, different remediation), and surface the failure message into the step's errors via the existingon_errorhook so it reachesctx.errorsand the step report. Tests: an exception batch increments the failed/unreviewed counters; a malformed batch still increments the dropped counters; a mixed pass reports both.Related: #294/#386 (the parse-path counting this extends), #538 (the deferred-recovery tracker for the malformed class), #537 (the same raise's cost-accounting sibling), #532 (the resume re-run surface).