Skip to content

fix(llm-reach): provider-exception batches are counted — batches_failed, the shared units_not_reviewed, and the step's partial status (#541) - #554

Merged
gadievron merged 2 commits into
masterfrom
fix/issue541-exception-counters
Sep 9, 2026
Merged

fix(llm-reach): provider-exception batches are counted — batches_failed, the shared units_not_reviewed, and the step's partial status (#541)#554
gadievron merged 2 commits into
masterfrom
fix/issue541-exception-counters

Conversation

@gadievron

Copy link
Copy Markdown
Collaborator

Summary

The #386 counting covered the parse path only (on_batch_drop / _count_drop): a batch whose LLM call raised (empty completion, transport error) exited through the except-continue BEFORE the counter — so a step report could read status: success, batches_dropped: 0, units_not_reviewed: 0 for a pass that reviewed nothing (the receipt: 4 empty-completion batches on the run's first attempt, invisible to every surviving counter).

The fix:

  • a distinct counter for the exception class (batches_failed — a different failure class with different remediation than the parse drops) + the same units_not_reviewed denominator so the coverage gap is the visible sum;
  • the #285/#376 partial-status contract: the LLR step summary now carries error_count = batches_dropped + batches_failed, so all-batches-failed reads partial, never success/0/0 (the adversarial review's catch);
  • both counters surface in the scanner's step summary.

Unchanged: the #538 truncation subclass (batches_truncated) is orthogonal; the absence-as-retry semantics (an exception batch still leaves no checkpoint records — it re-runs on resume); the #293 three-state summary arithmetic (the exception units stay in the incomplete bucket).

Closes #541.

Test plan

4 tests: the exception batch counts (batches_failed == 1, units_not_reviewed == 2, the parse counter untouched); a parse drop NOT counted as failed; a mixed pass reports both classes with units_not_reviewed as the sum; the step-status derivation — the counters in the summary make step_context read partial (never success/0/0).

Verification evidence

check command result
new + family pytest tests/test_issue532_llr_resume.py -q 29 passed
full suite pytest tests/ -q 3995 passed, 2 failed — both pre-existing (SDK pin drift)
static analysis ruff / Semgrep clean / 0
adversarial review combined wave+refute the refute's missed-item folded: the step-status gap (all-batches-failed reading success) closed via error_count; the #293 arithmetic and the auth-error class verified unchanged

…ed, the shared units_not_reviewed, and the step's partial status (#541)

The #386 counting covered the parse path only (on_batch_drop /
_count_drop): a batch whose LLM call raised (empty completion, transport
error) exited through the except-continue BEFORE the counter, so a step
report could read status: success, batches_dropped: 0,
units_not_reviewed: 0 for a pass that reviewed nothing (the receipt: 4
empty-completion batches on the run's first attempt, invisible to every
surviving counter).

- a DISTINCT counter for the exception class (batches_failed — different
  failure class, different remediation than the parse drops) + the same
  units_not_reviewed denominator so the coverage gap is the visible sum;
- the #285/#376 partial-status contract: the LLR step summary now carries
  error_count = batches_dropped + batches_failed, so all-batches-failed
  reads PARTIAL, never success/0/0 (the review round's catch);
- both counters surface in the scanner's step summary. The #538 truncation
  subclass is orthogonal and unchanged; the absence-as-retry semantics are
  unchanged (an exception batch still leaves no checkpoint records); the
  #293 three-state summary arithmetic unchanged (the exception units stay
  in the incomplete bucket).
@gadievron

Copy link
Copy Markdown
Collaborator Author

Gate-session receipt (proportionate round — astra; the counting-surface class)

Re-derived independently: the #532 family 29/29 (the 4 new tests included); the family (294 diagnostics + reachability + the #285 status contract) 48/48; full suite 3809/0 fresh-venv; ruff clean; the diff verified (the except-continue increment, the two stats keys, the scanner's error_count = batches_dropped + batches_failed wiring).

Astra: NOT-REFUTED — (a) the error_count sum is the ONE numeric source (the #285 contract's alternatives-not-additives shape); (b) the LLMAuthError re-raise-before-count correct (the designed abort, not a counted failure; the BaseException class escapes except Exception by construction); (c) no double-count mechanism established (the truncation continue and the provider-failure path are mutually exclusive per iteration — a raised call produces no completion for the truncation path to process); (d) the #293 three-state arithmetic unchanged (the exception units stay incomplete — the counters change observability, not classification).

CI green. Merge-ready.

@gadievron
gadievron merged commit 0f0b068 into master Sep 9, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant