fix(llm-reach): provider-exception batches are counted — batches_failed, the shared units_not_reviewed, and the step's partial status (#541) - #554
Conversation
…ed, the shared units_not_reviewed, and the step's partial status (#541) The #386 counting covered the parse path only (on_batch_drop / _count_drop): a batch whose LLM call raised (empty completion, transport error) exited through the except-continue BEFORE the counter, so a step report could read status: success, batches_dropped: 0, units_not_reviewed: 0 for a pass that reviewed nothing (the receipt: 4 empty-completion batches on the run's first attempt, invisible to every surviving counter). - a DISTINCT counter for the exception class (batches_failed — different failure class, different remediation than the parse drops) + the same units_not_reviewed denominator so the coverage gap is the visible sum; - the #285/#376 partial-status contract: the LLR step summary now carries error_count = batches_dropped + batches_failed, so all-batches-failed reads PARTIAL, never success/0/0 (the review round's catch); - both counters surface in the scanner's step summary. The #538 truncation subclass is orthogonal and unchanged; the absence-as-retry semantics are unchanged (an exception batch still leaves no checkpoint records); the #293 three-state summary arithmetic unchanged (the exception units stay in the incomplete bucket).
|
Gate-session receipt (proportionate round — astra; the counting-surface class) Re-derived independently: the #532 family 29/29 (the 4 new tests included); the family (294 diagnostics + reachability + the #285 status contract) 48/48; full suite 3809/0 fresh-venv; ruff clean; the diff verified (the except-continue increment, the two stats keys, the scanner's Astra: NOT-REFUTED — (a) the error_count sum is the ONE numeric source (the #285 contract's alternatives-not-additives shape); (b) the LLMAuthError re-raise-before-count correct (the designed abort, not a counted failure; the BaseException class escapes CI green. Merge-ready. |
Summary
The
#386counting covered the parse path only (on_batch_drop/_count_drop): a batch whose LLM call raised (empty completion, transport error) exited through theexcept-continueBEFORE the counter — so a step report could readstatus: success, batches_dropped: 0, units_not_reviewed: 0for a pass that reviewed nothing (the receipt: 4 empty-completion batches on the run's first attempt, invisible to every surviving counter).The fix:
batches_failed— a different failure class with different remediation than the parse drops) + the sameunits_not_revieweddenominator so the coverage gap is the visible sum;#285/#376partial-status contract: the LLR step summary now carrieserror_count = batches_dropped + batches_failed, so all-batches-failed reads partial, never success/0/0 (the adversarial review's catch);Unchanged: the
#538truncation subclass (batches_truncated) is orthogonal; the absence-as-retry semantics (an exception batch still leaves no checkpoint records — it re-runs on resume); the#293three-state summary arithmetic (the exception units stay in the incomplete bucket).Closes #541.
Test plan
4 tests: the exception batch counts (
batches_failed == 1,units_not_reviewed == 2, the parse counter untouched); a parse drop NOT counted as failed; a mixed pass reports both classes withunits_not_reviewedas the sum; the step-status derivation — the counters in the summary makestep_contextread partial (never success/0/0).Verification evidence
pytest tests/test_issue532_llr_resume.py -qpytest tests/ -qerror_count; the #293 arithmetic and the auth-error class verified unchanged