fix(uipath-planner): runnable gate command + complete cell contract for case SDDs - #3183
abhiram-vad wants to merge 5 commits into
Conversation
…atus rule Three case eval tasks scored 0.000 on 2026-09-08 without a byte of output. skill-case-reject-route burned its whole 3600s turn wall on 56 tool calls totalling 3.07s of execution — 20+ of them greps and seds into scripts/case/audit_sdd.py — and its last message at 55 minutes was still "let me check a couple more details on the template's exact Table of Contents anchors". skill-case-phase-0-finalize-draft did the same in 21 calls. Neither ever reached a Write. The lane guide already forbade this. "NEVER read scripts/case/audit_sdd.py" landed 2026-09-01 (#2929) as the mitigation for the identical hunting seen on 2026-08-31, and finalize_from_draft_loan.yaml already records the diagnosis: "If a run still hits this wall, the cause is the doc/validator hunting seen in that run, not the budget." The prohibition did not hold, because the guide created the incentive it then banned: § Resumption step 4 named the shape contract as "enforced by audit_sdd.py; it is not restated here", and the template footer pointed at the layers guide, which finalization may not read. The agent was told it would be hard-gated by rules deliberately written down nowhere it was allowed to look, so it read the validator. Replace the prohibition with a positive, countable instruction: - Finalization is three reads — draft, template, § Resumption + § Terminal step — and then the seed Write, which is tool call #4 at the latest. The agent can count its own calls; it could not verify a whitelist. - Anything those reads leave open is closed by RUNNING the auditor on the partial file and repairing findings (three rounds already allowed), never by reading more. Drops "it is not restated here" and the layers-guide fallback for the finalize path. Second, skill-case-sla-sdd-response-map scored 0.250 for the opposite reason: it finished in 615s, the user simulator approved it, and the shipped gate said AUDIT OK on an SDD the graded contract rejects. The Triage SLA carried an At-Risk row only, with an HTML comment justifying the omission. The rule was prose in a template comment, phrased at SLA level ("Source states no response -> both statuses notify-only"), so an SLA with a stated at-risk response read as exempt. audit_sdd.py never parsed the § SLA Response Map at all. - New sla_map_status_gaps(): every (Scope, SLA) authors both At-Risk and Breached; a status the source is silent on is notify-only with Target and Interrupting em-dash, never an omitted row. Scope and SLA cells normalize away a trailing (slug) qualifier so one pair cannot split into two. - Template and layers guide restate the rule at status level, unconditionally. - Verified: flags the failing artifact with exactly one finding (the graded defect), and fires on zero of the 18 SDD fixtures under tests/tasks. The two fixtures carrying a response map already author both statuses, including "Breached | notify-only | - | -" rows whose rationale is "No work on breach is described" — the convention this pins. Third, grade the ordering so the next regression is loud. The four finalize tasks now fail a command_not_executed criterion when audit_sdd.py's source is read, grepped, or hunted for, instead of silently exhausting the turn wall and reporting turn-timeout-too-short. Not addressed here: the loan variant's 0.000 is infra, not skill — one assistant message hung 3474s (58 minutes, 17 characters of output) before Bedrock returned "Request timed out", and the retry inherited an already-spent task budget, so no criterion was ever graded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34310755247 proved the two criteria added in 64b8b3e miss real escapes. Artifact evidence, per task, counting every command whose parameters name audit_sdd.py outside a legitimate `python3 .../audit_sdd.py` gate run: - skill-case-reject-route ran `wc -l .../audit_sdd.py` and scored clean. `wc` is not a harmless line count, it is the first rung of the ladder: in both the loan and the sla runs the same `wc` is immediately followed by `sed` range reads of the whole file. Added to the Bash verb list — the dry run over that run's recorded commands takes reject-route from 2 caught to 3. - skill-case-sla-sdd-response-map carried no bans at all, because the previous commit scoped them to the four finalize tasks. That task read the entire 1013-line validator across four sed ranges (1-220, 220-560, 560-900, 900-1013) plus two greps and a wc, and still scored SUCCESS 1.000 — the run's worst offender, ungraded. It gets the same four criteria; the dry run takes it from 0 caught to 8. - The Grep and Glob tools bypassed both criteria entirely: tool_name is a per-criterion filter, and both tools are in allowed_tools. No run has yet reached the validator through them, so these two rows close the tool surface rather than a measured escape — but Grep is this lane's heaviest reader (10 calls in the original finalize-draft failure, all aimed at the guides and template rather than at the validator). Weight drops 1.0 -> 0.25 per row. Four criteria express one concern, so together they now carry one ordinary criterion's weight. At 1.0 each they were 4.0 of 8.0 total weight on sdd_response_map — half a task's score for a hygiene rule, which would have made the re-run's numbers unreadable. Gating is untouched: pass_threshold stays 1.0, so a violation still turns the task red, and the gating-vs-advisory decision stays open on real numbers. Verified by replaying both pattern sets over the 34310755247 artifacts: every previously-escaping archaeology command is now matched, and skill-case-picker- pairing — the clean control, whose only audit_sdd.py touches are two legitimate gate runs — stays at zero matches under the new patterns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
skill-case-reject-route never reached its content check in two runs, so the
defect underneath the timeouts was never graded. It is a doc defect, and a
sharp one: § Resumption step 3 carried ONE repair for two opposite diagnoses.
The two finalize fixtures are a matched pair, and each states the
discriminator in its own authoritative Requirements:
picker (VendorOnboarding): "Compliance Hold is launched by a person, not by
an event ... from the stage picker. Nothing triggers it automatically."
-> user-selected-stage is CORRECT; the missing half is the upstream
wait-for-user exit.
reject (GrantReview): "Rejection is automatic from that decision ... the
decision itself does the routing. Nobody picks that lane by hand."
-> user-selected-stage is THE DEFECT; the lane must be re-keyed on the
decision fact.
Step 3 said only "retain the authored lane" — the picker repair — so an agent
following it verbatim authored `required-tasks-completed | — | exit-only | Yes`
and failed the reject grader. That is exactly what run 34310755247 recorded.
Step 3 now diagnoses before it repairs, with branch (a) unchanged so the picker
task keeps passing, and branch (b) spelling out the four edits a decision route
needs: re-key the lane entry (Interrupting Yes), add the origin's guarded
diverting exit, make the completion exit the complement, and drop a now-orphaned
wait-for-user exposure. Silence in the source defaults to (a), because re-keying
on an unstated fact invents a business rule.
The trap that makes this worth a test: the two gates disagreed. audit_sdd.py's
own finding says to "give the origin the matching Marks Stage Complete: No
diverting exit", but § Lifecycle gates allows `Marks Complete: No` ONLY with
`selected-tasks-completed` / `wait-for-connector`. So the obvious
`required-tasks-completed | No` row SATISFIES check_reject_route.py and
audit_sdd.py rejects it as a schema error — an agent following the prose lands
between them with no way out. Step 3(b) now names the legal WHEN and the
precedence reason (stage exit evaluates before stage completion).
Verified by applying step 3(b) literally to the staged draft:
check_reject_route.py --scope lane -> OK
check_reject_route.py --scope all -> OK (the advisory origin check, which
has failed in every run so far)
audit_sdd.py -> zero findings introduced vs the untouched draft, and two
removed, including the picker-route finding itself.
test_reject_route_repair_shape.py pins all of that, plus the trap: it asserts
the `required-tasks-completed | No` row passes the grader and is rejected by the
gate, so a future edit to either side fails in 0.6s instead of a 20-minute eval.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ns go advisory Two changes, both driven by artifacts from run 34319308685. 1. The template now says its cell rules are complete where they stand. That run's reject-route wrote at tool call #51 of 61 and exhausted max_turns mid-repair, so the decision-route fix landed but the diverting exit never did. The 50 pre-write calls were not aimed at the validator by intent — they were the same cell questions asked over and over: `HITL Implementation` / `Action App` / `JSON Schema` five times, `Priority` six times, plus `Case App`, `Task-output passing`, `Case Identifier source`, `Execution autonomy`. The agent could not tell whether a cell mentioned once was fully specified, so it kept looking for a second source, and the validator's source is where that search ends up. A new header comment states the invariant directly: a cell named ONCE is fully specified by that one mention, nothing outside the template adds or overrides a value, and a value that still looks unstated IS unstated — take the section's default or `—`, record provenance, move on. This is the answer source the write-first counter in case-design-lane-guide.md was missing; a counter with nowhere to resolve a question gets overridden the same way a prohibition with no substitute did. Three cells that genuinely could not be resolved are now closed: - `HITL Implementation` appears exactly once in the whole skill and `JSON Schema` appears ZERO times, yet the run searched five times for a JSON-schema variant. The cell now says `Action App: <title>` is its only form and that the task's fields are the Input Schema table, not a value of this cell. - `Execution autonomy` had a closed set but no default, and finalization never asks. It now carries the draft's value, defaulting to `autonomous` (the option sdd-generation-guide.md marks recommended). - Case-level `Priority` had no default and no platform choiceset to find. It is now explicitly optional (`—` when the source names no scheme) and disambiguated from a task's `**Priority:**`, which is a closed set. 2. The archaeology bans become advisory (20 fields, pass_threshold 1.0 -> 0.0). The measurement they were added for is done, and gating them costs more than it buys: picker finished that run at 0.947 with every content criterion at 1.00 and was still red. The behaviour is bounded and non-fatal, so weight 0.25 keeps both the score dent and the reported hit count while a hygiene rule stops failing a correct document. Same shape as the task's existing "Advisory:" origin criterion, and reversible in one field per row. Verified: 479 passed (the pre-existing test_stage_shared failure is unrelated and fails on a clean tree), and the SLA row-closure sweep still fires on zero of the 18 SDD fixtures under tests/tasks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…of the gate and § Resumption The previous four commits fixed the right defects in the wrong places: a tool-call counter the agent has to self-audit, a new check inside `audit_sdd.py`, and edits to § Resumption. Same three defects, re-landed as prose in the two files the agent must read anyway. Reverted whole: `scripts/case/audit_sdd.py` (the `sla_map_status_gaps` addition), `case-design-lane-guide.md` (the "seed Write is tool call #4" budget and § Resumption steps 1/3/4), and the two tests that pinned them. What the run artifacts showed, and what now answers it: 1. The gate command was not runnable as written. The footer said `run <py> "<skill folder>/scripts/case/audit_sdd.py"` and told the agent `<py>` is "the first of python3, python, py that runs" — so 5 of 7 failing runs opened with `ls scripts/case/; which python3 python py` or `find -iname audit_sdd.py`, and that locate call is what put the source in hand. The footer now carries one literal command anchored on the skill base dir given at invocation, with the interpreter ladder demoted to a command-not-found fallback, and names the locate call as the first rung of the source-reading it forbids. 2. The contract had a dangling end. The footer named nine check families and deferred the rules to "the cells above and in case-design-layers-guide.md" — a file draft finalization may not read — so single-mention cells resolved to nowhere. It now states the cells are COMPLETE for the render contract, with design semantics explicitly left to the layers guide. 3. Cells with no answer. Unchanged from 84f340a: the "cell rules are complete where they stand" header, plus `HITL Implementation` (Action App is its only form), `Execution autonomy` (carry the draft's value, else `autonomous`) and case `Priority` (optional). The SLA both-statuses rule survives as prose at status level in the template and in `case-design-layers-guide.md` § Defaults when the source is silent, without the enforcement claim — with the gate change reverted, the shipped auditor stays blind to the § SLA Response Map and a stated at-risk response with an omitted breach row will still print AUDIT OK. The picker-vs-decision repair moves out of § Resumption into `case-design-layers-guide.md`, beside the rule it belongs to ("`user-selected-stage` is picker exposure … never deterministic routing"), diagnosis first and both branches spelled out. The 20 advisory `command_not_executed` criteria stay: they are eval-side only, never read by the agent, and they are the measurement that says whether the placeholder fix removed the reason to go near the script. Verified: 238 passed / 1 failed in tests/scripts and 89 passed in tests/tasks/uipath-planner. The single failure — test_stage_shared.py:: test_stage_shared_anchors_to_checkout_and_stages_nested_checkers — reproduces on a clean `main` worktree and is unrelated. check-skill-status.py and check-skills-sh.py both OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| `py` that runs (Windows usually has no `python3` alias); only if all three are absent verify manually. RUN it, never open the script source — | ||
| Gate: run this on the on-disk file BEFORE the Status: ready flip — in every mode: | ||
|
|
||
| python3 "<skill base dir>/scripts/case/audit_sdd.py" <sdd path> [--draft <draft path>] |
There was a problem hiding this comment.
I though this should no longer be a python script
| **Diagnose before repairing a `user-selected-stage` lane — the source says which defect you have, and the two repairs are opposites.** | ||
|
|
||
| - **A person launches it** — the source says the lane is pulled aside by hand, chosen from the stage picker, that nothing triggers it automatically. The lane is CORRECT and its other half is missing: give every eligible upstream primary stage a completing `required-tasks-completed` / `wait-for-user` / `Marks Stage Complete: Yes` exit ("any active case" means every primary stage). This REPLACES that stage's existing `required-tasks-completed | exit-only | Yes` row — never a second completion row, never a `Marks Stage Complete: No` row. Add no event, SLA, or decision trigger. | ||
| - **A decision, event, or SLA routes it** — the source says entry is automatic from a decision, that the decision itself does the routing, that nobody picks the lane by hand. The picker rule IS the defect. All four edits, or the branch dual-fires or deadlocks: | ||
| 1. **Lane entry** — REPLACE the `user-selected-stage` row (never keep it alongside) with `selected-stage-completed("<origin stage>")` (or `selected-stage-exited(...)`), `IF` the affirmative guard on the deciding variable — `=js:(vars.<decisionVar> === "<Value>")` — and `Interrupting: Yes`. | ||
| 2. **Origin diverting exit** — ADD a row carrying that same affirmative guard with `Marks Stage Complete: No`, so taking the branch does not also complete the stage. Its WHEN is `selected-tasks-completed("<deciding task>")` or `wait-for-connector`: `Marks Complete: No` pairs with nothing else, and `required-tasks-completed | No` is a schema error (§ Lifecycle gates). Stage exit evaluates before stage completion, so an UNguarded diverting row would fire first and the stage would never complete. | ||
| 3. **Origin completion exit** — its guard becomes the COMPLEMENT, `=js:(vars.<decisionVar> !== "<Value>")`. Unguarded, it fires on the diverted case too; repeating the affirmative guard on both rows fires both. With more than one diverted outcome the complement excludes every diverted value. | ||
| 4. **Orphaned picker exposure** — once no `user-selected-stage` entry remains anywhere, DROP any upstream `wait-for-user` exit that existed only to expose this lane; the pairing above fails a `wait-for-user` with no picker entry. | ||
|
|
||
| A sibling outcome of the same decision already keyed this way is the shape to copy verbatim — the approve branch usually is. Source silent on who launches the lane ⟹ treat it as person-launched and keep what is authored: re-keying on an unstated fact invents a business rule. | ||
|
|
There was a problem hiding this comment.
This section seems more like agent's thoughts rather then instructions. Please rephrase this in a shorter way. I would also like to raise my concerns again that the planner should not know about the case semantics in this depth, that's way the case skill exists. Please take this into consideration for future work.
…in, and close the review comments Additive on top of the existing #3057 commit — no history rewritten. Two commits: a merge of origin/main (the branch was 210 commits behind and showing conflicts), then this one. What this adds to the original proposal: - @AlvinStanescu's review: the 35-item walk now runs in ONE subagent when the Agent tool is present, so a read-only verdict stays out of the authoring context. Requested on this PR and previously approved without being implemented. - @RaduAna-Maria's review on #3183: the user-selected-stage repair is restated as a diagnosis table — what the source says, the defect, the repair — instead of two paragraphs of narration. - Design-only requests seed sdd.md before settling the body. The lane used to state the model "lives in memory — no checkpoint writes", which turned a design-only request into one unbounded generation pass that regularly ran out of turn before writing anything. - Response is a closed enum: `—` is legal in Target and Interrupting only, and a scope with no SLA gets no row rather than a placeholder row. - The Interrupting cell is read off the source's words — "interrupt", "take over", "before the case can close" — rather than judged after choosing the lane. - The checklist moves to references/case/case-sdd-conformance-checklist.md and audit_sdd.py's two unit tests are deleted with it. Every item above closed a specific graded failure in the coder-eval suite; the numbers are in the follow-up test layer's commit message and in the PR body. Layers 2 and 3 stack on this branch: the uipath-maestro-case receipt change, then the eval-suite moves and grading. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Superseded by #3057, which now carries this PR's substance plus the review comments from both threads here. What moved across:
Verification on the superseding stack — three identical replicate runs of the seven planner case evals: mean 0.965 / 0.968 / 1.000, zero timeouts across 21 task-executions. The seven-run failure pattern that opened this PR (five runs at 0.000 with no gradeable artifact) is gone. Closing rather than rebasing because the work now sits in a reviewed stack: #3057 → #3456 → #3457. |
The defect
Seven case eval runs, five of them scoring
0.000without producing a gradeable artifact. Tool executionacross all seven totals 0.3–3.1 seconds each, against walls of 178–7204s — the time went into
generation, not tools. What the runs have in common:
find/lsto locateaudit_sdd.pyreject-routefinalize-draftfinalize-draft-loanprocurement-sla-interruptscase-reasoning-regressionssla-sdd-response-mapcase-reasoning-regressions(terra)Every run that opened the validator's source scored 0.000. Both runs that only executed it produced a
graded artifact. Three doc defects put them there.
1. The gate command was not runnable as written
case-sdd-template.mdsaid:Two unresolved placeholders, and an explicit instruction to probe for the interpreter. Five of seven runs
opened by resolving both at once —
ls scripts/case/ 2>&1; which python3 python py 2>&1,find … -iname "audit_sdd.py" 2>/dev/null; python3 --version— and that locate call is what put the filein hand. Meanwhile
case-design-lane-guide.md:13says "never hunt withfind/ globalls": the firstrung was mandated and forbidden at the same time.
2. The contract had a dangling end
The footer named nine check families and then deferred: "the rules themselves live at the cells above and
in
case-design-layers-guide.md" — a file draft finalization may not read. So a cell like**HITL Implementation:**, which appears once in the whole skill, resolved to nowhere.finalize-draftsaid so at turn 26:Fifteen more Greps followed, and no
Write.3. The SLA both-statuses rule was written at the wrong level
case-design-layers-guide.mdsaid "No stated response → both statusesnotify-only" — phrased per SLA,while the contract is per status. An SLA with a stated at-risk response read as exempt, so
sla-sdd-response-mapauthored the At-Risk row only, with an HTML comment justifying the omission, andfailed the graded contract:
The change
Two files. Prose only — no script change, no lane-guide change.
assets/templates/case/case-sdd-template.mdladder is demoted to a command-not-found fallback, and the text names the locate call as the first rung
of the source-reading it forbids.
the script's source states nothing they do not. Design semantics stay explicitly with the layers guide.
specified; still-unstated means take the default or
—, note provenance, move on.HITL Implementation(Action App: <title>is its only form),Execution autonomy(carry the draft's value, else
autonomous), casePriority(optional, and disambiguated from a task'sclosed-set
**Priority:**).references/case/case-design-layers-guide.mdnever removes the other's row.
user-selected-stageis pickerexposure … never deterministic routing"), diagnosis first: a person launches it keeps the lane and
adds the missing
wait-for-userhalf; a decision routes it re-keys the lane on the decision fact andspells out all four required edits. Source silent defaults to person-launched, because re-keying on an
unstated fact invents a business rule.
Deliberately not done
scripts/case/audit_sdd.pyis untouched. The consequence is stated rather than hidden: the shippedauditor stays blind to § SLA Response Map, so a stated at-risk response with an omitted breach row will
still print
AUDIT OKon a document the grader rejects. The rule is in the template where the agentreads it; nothing enforces it.
command_not_executedcriteria in the five task YAMLs stay. They are eval-side only —never read by the agent — and they are the measurement that says whether removing the placeholder removed
the reason to go near the script.
Evidence
pytest tests/scriptspytest tests/tasks/uipath-plannerpython3 scripts/check-skill-status.pyOK — 27 skills, manifest valid.python3 scripts/check-skills-sh.pyOK — 27 skills grouped across 4 section(s).The single failure is
test_stage_shared.py::test_stage_shared_anchors_to_checkout_and_stages_nested_checkers.It reproduces identically on a clean
mainworktree — verified before claiming it unrelated, not asserted.Coder eval in flight: run 34448866335 —
the five planner case tasks plus the two
uipath-maestro-casedesign-delegation tasks,claude,j=4.No result claimed until it lands. The tell is
commands_by_tooland the index of the firstWrite: iffind/whichstill open the runs and the first Write still lands past call #20, the placeholder fix didnot do the work.
Review notes
assets/skill-status.jsonandskills.sh.jsonneed no edit.SKILL.mdfrontmatter change, soactivation-gate.yml's recall-eval gate is not triggered..sh/.ps1twin obligations.CODEOWNERSalready covers both touched paths (/skills/uipath-planner/assets/templates/case/and/skills/uipath-planner/references/case/).🤖 Generated with Claude Code