fix(tests): route-specific command telemetry must not move the score - #3516
Conversation
`pass_threshold: 0` was read as "this criterion is advisory", and
`test_v1_only_authoring_commands_match_the_temporary_allowlist` enforces only
that. It is half the idiom. coder_eval's own field documentation spells out the
other half:
weight=0 excludes from the score but NOT from the pass/fail gate ...
To make a criterion truly non-gating, also set pass_threshold=0.
So a criterion with `pass_threshold: 0` and `weight: 1.5` never fails a task and
always moves its score — it sits in both halves of the weighted mean.
That is not neutral between arms. A `command_executed` grades the SHELL COMMAND
an arm ran, and the two arms run different commands by construction. In the
2026-09-23 same-ground run, 15 such criteria across 10 flow tasks scored 1.0 for
v1 and 0.0 for v2 — `flow node add`, `flow registry get`, `agent init
--conversational`, `is connections list`, `is resources run list` — and dragged
tasks that passed every graded check down with them.
`datafabric_integration_create_get` returned SUCCESS at 0.55: three advisories
at weights 1.5/2.0/1.5 against two graded criteria at 3.0.
The outcome each of those stands in for is already graded by a `run_command`
against the artifact, which is route-blind. `slack_channel_description_simulated`
says so in its own comment: the run_command "Supersedes a static connector-key
grep ... THIS is the authoritative proof". So the telemetry keeps reporting and
stops scoring.
61 criteria across 38 tasks get `weight: 0` — three of them had no `weight` line
at all, which defaults to 1.0. `single_node/outlook_trigger_inbox` already did
this and is the shape the rest now match.
Two exclusions, both deliberate:
- `ixp/routing.yaml` — its two criteria carry `stop_early`, where weight is
load-bearing for the pass-stop floor rather than for the score. Its own
sentinel comment says so. The new gate skips `stop_early` criteria.
- `smoke/registry_discovery.yaml` — allowlisted. It "deliberately produces no
artifact", so all four of its criteria grep the shell; zeroing them leaves a
total weight of 0 and `calculate_weighted_score` reports 0.0 for that. It needs
an outcome-graded criterion over the agent's report, which is a task redesign,
not a reweight.
Replaying the run's recorded criterion results against the new weights: the flow
suite's v2 mean score goes 0.919 -> 0.935 and v1 0.959 -> 0.960, narrowing the
arm delta from -0.041 to -0.025. Pass/fail counts do not move, because weight
never gated. Seven v2 tasks reach a clean 1.00.
Two tasks per arm go DOWN, which is the point working: passing telemetry was
padding a failing grade. `e2e_devcon_expense_approval` drops 0.37 -> 0.23 in v2
and now reads as the HITL outcome-port failure it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @dmetzgar's task in 2m 27s —— View job PR Review: fix(tests): non-gating command telemetry must not move the score either
SummarySets Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | N/A | No skill files changed |
| E2E Tests | OK | Test gate added, all YAMLs still parse |
| Skill Body | N/A | No skill files changed |
| References & Assets | N/A | No references or assets changed |
| Repo Hygiene | OK | All changes scoped to tests/, no secrets, no unrelated changes |
Issues for Manual Review
generate_schema.yaml(uipath-platformtag): has the samepass_threshold: 0+weight: 3.0pattern on acommand_executedcriterion. Not covered by the new gate (different tag). Consider whether it needs the same fix in a follow-up.- The PR description claims 1361 pytest passes — I was unable to run the test suite to independently verify.
Conclusion
Clean, well-scoped fix backed by measured data. The mechanical YAML changes are consistent, the new gate is correctly implemented with appropriate exemptions, and the exclusions are well-documented. No issues found — approve.
|
Claude finished @dmetzgar's task in 2m 34s —— View job Coder-eval task lint (advisory)38 task YAMLs modified; 0 Critical, 0 High, 0 Medium, 0 Low, 38 OK. Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge. Evidence of passing run✅ Author confirms passing run: "pytest tests/tasks/uipath-maestro-flow — 1361 pass." and "New gate verified to actually catch a violation: reverting one Per-task lintEvery change across all 38 files is identical in shape: This change does not trigger any rubric axis — it fixes a pre-existing scoring inconsistency where advisory criteria still carried weight in the score calculation. All 38 files — verdict: OK (click to expand)
Within-PR duplicatesNo duplicate clusters detected. (No new task YAMLs were added.) Conclusion✅ All changed tasks pass the rubric. Evidence of passing run confirmed. The change is a targeted score-correction: advisory |
Review caught this, and the review was right. The first pass keyed on criterion
TYPE — every non-gating `command_executed` — on the reasoning that such a
criterion grades the shell command an arm ran, and the two arms run different
commands. That reasoning does not survive the data.
Of the 54 criteria it swept that the 2026-09-23 run actually observed, 31 were
BOTH PASS: `solution init` (5), `flow validate` (4), `flow debug` (3),
`flow eval ...` (3), `flow init` (2), `--output json`, `is webhooks config`,
`ixp projects create`. Those are lifecycle commands both routes run on the same
artifact, and both routes do run them. Zeroing them removed real, satisfiable
signal and bought no neutrality — and it moved v1's own mean, which a neutrality
correction has no business doing.
So the rule is now about the COMMAND: a criterion is in scope only when its
command has no counterpart in the other route. Four families, each with its
reason recorded next to it:
- `flow node add|configure|remove|update`, `flow edge ...` — v1 mutates the
graph a node at a time; the SDK loop writes `.flow.ts`.
- `agent init|refresh --inline-in-flow|--conversational` — v1 scaffolds the
sidecar with the CLI; `conversationalAgent()` emits `agent.json` itself.
- `is connections list`, `is resources run list`, `is triggers objects|describe`
— `registry prepare` picks the connection, pages the collection and writes
bindings in one call.
- `flow registry pull` — the manifest refresh the SDK loop has no need of.
`flow registry get|search|list` is deliberately NOT a family: the SDK arm passes
those in the IxP tasks, so the node registry is not a v1-only surface. Only the
refresh step is.
23 criteria across 17 files, down from 61 across 38.
Replayed against the run's recorded results: v2's mean goes 0.919 -> 0.933 and
the arm delta -0.041 -> -0.027, within a thousandth of the broad sweep's 0.935.
v1's mean does not move at all and no task in either arm goes down — where the
first pass dropped `e2e_devcon_expense_approval` 0.37 -> 0.23 and
`interactive_customer_escalation_triage` 0.45 -> 0.38 by zeroing validate and
solution-init telemetry those tasks were passing.
`smoke/registry_discovery` no longer needs an allowlist: only its `registry
pull` criterion is in scope, leaving 4.5 of weight behind, so the zero-total
score the broad sweep would have produced cannot arise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dmetzgar
changed the title
2 participants
The bug
pass_threshold: 0was read as "this criterion is advisory", andtest_v1_only_authoring_commands_match_the_temporary_allowlistenforces only that. It is half the idiom — coder_eval's own field docs carry the other half:So
pass_threshold: 0withweight: 1.5never fails a task and always moves its score.That matters where the command itself is route-specific. In the 2026-09-23
same-ground-abrun those criteria scored 1.0 for v1 and 0.0 for v2, dragging tasks that passed every graded check down with them.datafabric_integration_create_getreturned SUCCESS at 0.55 — threeflow node addadvisories weighing 5.0 against two graded criteria at 3.0.The rule
A criterion is in scope only when its command has no counterpart in the other route. Four families, each with its reason recorded beside it in
ROUTE_SPECIFIC_COMMANDS:flow node add|configure|remove|update,flow edge ….flow.tsagent init|refresh --inline-in-flow|--conversationalconversationalAgent()emitsagent.jsonitselfis connections list,is resources run list,is triggers objects|describeregistry preparepicks the connection, pages the collection and writes bindings in one callflow registry pull23 criteria across 17 files. A new gate,
test_route_specific_command_telemetry_is_weightless, keeps it true. Three criteria had noweight:line at all — which defaults to1.0— so silence is treated as a violation, not as compliance.What is deliberately not in scope
solution init,flow init,flow validate,flow debug,flow eval …,--output json,is webhooks config,ixp projects create— shared lifecycle commands both routes run on the same artifact. 31 of the 54 observed criteria were BOTH PASS, these among them.flow registry get|search|listis out for the same measured reason: the SDK arm passes those in the IxP tasks, so the node registry is not a v1-only surface. Only the refresh step is.Where an arm fails one of the survivors —
flow debuginslack_channel_description_simulated,solution initinwebhook_waitfor_parallel— that is now a finding about the arm, which is the point.stop_earlycriteria are exempt: thereweightis load-bearing for the pass-stop floor, not just the score (ixp/routing.yaml's sentinel says so verbatim), so the gate skips them rather than trusting one reading of one installed coder-eval version.Effect, replayed against the run's own recorded results
Arm delta narrows −0.041 → −0.027. Eight v2 tasks improve, five to a clean 1.00. v1's mean does not move at all and no task in either arm goes down — the signature of a correction that is actually neutral.
Pass/fail counts do not move, because weight never gated. This is a score correction, not a pass-rate one.
What changed after review
command_executed+pass_threshold: 0)registry_discoverywould have scored 0.0 with no weight anywhereregistry pullis in scope, leaving 4.5 of weightThe two regressions the broad sweep caused (
e2e_devcon_expense_approval0.37 → 0.23,interactive_customer_escalation_triage0.45 → 0.38) came from zeroingflow validateandsolution inittelemetry those tasks were passing. Both files are untouched now.Verification
pytest tests/tasks/uipath-maestro-flow— 1361 pass.weight: 0fails it with the offending path named.success_criteria_resultsagainst the new weights, not an estimate.🤖 Generated with Claude Code