fix(tests): migrate HITL task yamls off completed-handle wording

[dushyant-uipath]

Problem

22 task files across uipath-human-in-the-loop and uipath-maestro-flow prompted for or asserted a single completed handle regardless of outcome count, or checked wiring with a file-scoped regex against a guessed literal port id instead of the node-scoped, id-derived check.

Fixed

• Prompts and success-criteria descriptions reworded from completed to outcome-port language in all 22 files.
• smoke_02_completed_port_wired.yaml, quality_04 in both skills, and the 6 e2e_* invoice/escalation/gdpr/multi-hitl/expense tasks now call check_simulated_hitl.py outcome-wiring (node-scoped, id-derived) instead of a file-scoped regex against a guessed port name.
• TEST_PLAN.md's handle inventory and per-test descriptions updated to match.

Test plan

• skill-flow-hitl-smoke-completed-port (smoke_02): green in CI.
• skill-hitl-e2e-invoice-approval-greenfield (e2e_01) run locally: SUCCESS, score 1.000, 7/7 criteria, 482.7s.
• Remaining ~13 affected integration/e2e tasks not run through a full coder-eval pass.

Depends on

#3260 (checker logic). These task files call the outcome-wiring check that PR adds; based on its branch, and its CI must pass first.

Split from #3180

Split out of #3180 along with #3259 (docs) and #3260 (checker logic).

🤖 Generated with Claude Code

[github-actions]

Claude finished @dushyant-uipath's task in 6m 39s —— View job

---

Coder-eval task lint (advisory)

21 task YAMLs changed (all modified, none added); 0 Critical, 0 High, 2 Medium, 7 Low, 12 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

⚠️ High — 2 of 21 modified tasks have passing-run claims; the remaining 19 do not. The PR body transparently acknowledges this ("Remaining ~13 affected integration/e2e tasks not run through a full coder-eval pass"). Please consider running additional changed tasks and editing the PR description to add claims.

Confirmed:

• ✅ skill-flow-hitl-smoke-completed-port (smoke_02) — "green in CI"
• ✅ skill-hitl-e2e-invoice-approval-greenfield (e2e_01) — "run locally: SUCCESS, score 1.000, 7/7 criteria, 482.7s"

Per-task lint

tests/tasks/uipath-human-in-the-loop/quality_09_dev_mistake_wrong_type.yaml — verdict: Medium

Issues:

• [Medium] Meaningful coverage: prompt specifies three field types to infer — boolean (L20), number (L22), and date (L18: "submission date (should be a DATE)") — but only boolean (L45-49) and number (L51-55) have criteria. Date type goes unvalidated.

Suggested fixes:

• Add a criterion: run_command with flow_contains.py --flow-name ExpenseReview '"date"' to verify the agent used the date type for the submission date field.

---

tests/tasks/uipath-human-in-the-loop/quality_05_priority_and_timeout.yaml — verdict: Low

Issues:

• [Low] Meaningful coverage: prompt says "Wire the node's outcome port(s) to a script node" (L17) but no criterion checks outcome wiring. Priority and validate are well-covered; the wiring gap is minor since it's not the test's focus.

Suggested fixes:

• Consider adding a run_command with check_simulated_hitl.py outcome-wiring (same pattern used by sibling tasks).

---

tests/tasks/uipath-human-in-the-loop/e2e_01_invoice_approval_greenfield.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/e2e_02_ai_escalation_brownfield.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/e2e_03_gdpr_compliance_greenfield.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/e2e_04_multi_hitl_brownfield.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/e2e_05_expense_approval_brownfield.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/e2e_06_invoice_approval_greenfield_simple.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/quality_04_all_handles.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/quality_07_runtime_vars.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/quality_08_variable_binding_fieldid.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/quality_10_dev_mistake_binding_direction.yaml — verdict: OK

tests/tasks/uipath-human-in-the-loop/quality_11_inout_field_access.yaml — verdict: OK

---

tests/tasks/uipath-maestro-flow/hitl/smoke_02_completed_port_wired.yaml — verdict: OK

tests/tasks/uipath-maestro-flow/hitl/smoke_03_multi_outcome_routing.yaml — verdict: OK

---

tests/tasks/uipath-maestro-flow/hitl/quality_01_schema_design.yaml — verdict: Low (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-flow/hitl/quality_02_result_downstream.yaml — verdict: Low (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-flow/hitl/quality_03_boolean_decision.yaml — verdict: Low (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-flow/hitl/quality_04_brownfield_insert.yaml — verdict: Low (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-flow/e2e/devcon_expense_approval.yaml — verdict: Low (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-flow/interactive/expense_approval_simulated/expense_approval_simulated.yaml — verdict: Low (theme-captured; see Theme 1)

Within-PR duplicates

No duplicate clusters detected. All 21 files are modifications (no new tasks added).

Themes

Theme 1 — [Medium] Validate-only flow tests (6 maestro-flow tasks): quality_01_schema_design, quality_02_result_downstream, quality_03_boolean_decision, quality_04_brownfield_insert (integration tier), devcon_expense_approval, expense_approval_simulated (e2e tier) — all build and validate flows without flow debug. This is an inherent HITL constraint: inline HITL nodes block on human review and cannot be debugged headlessly. The two e2e tasks document this rationale in their description fields (High → Medium carve-out); the four integration tasks do not (Medium, integration tier default). Suggested fix for the integration tasks: add "Validate-only: inline HITL nodes cannot be flow-debugged headlessly." to their description fields for rubric alignment and severity downgrade. Pre-existing — not introduced by this PR.

Conclusion

⚠️ 2 task(s) have unique issues (max severity Medium), 6 tasks are theme-captured at Low. All issues are pre-existing — this PR's changes (completed-handle → outcome-port wording migration) are clean. The main action item is adding passing-run evidence for the remaining 19 modified tasks.

---

[github-actions]

Claude finished @dushyant-uipath's task in 3m 41s —— View job

---

PR Review: fix(tests): migrate HITL task yamls off completed-handle wording

• Read review criteria and project conventions
• Analyze full diff against base branch (22 files, +127/-88)
• Read all 22 changed files in full context
• Check for consistency and correctness issues
• Post review findings

---

Summary

Migrates 22 HITL task YAMLs across uipath-human-in-the-loop and uipath-maestro-flow from "completed handle" terminology to "outcome port" language. The stronger change: tasks that previously asserted wiring via file_contains: 'completed' (whole-file substring match) or flow_contains.py 'completed' (file-scoped regex) now call check_simulated_hitl.py outcome-wiring — a node-scoped, schema-derived check. The e2e_01 golden scenario was also simplified to mock integration points as Script nodes, reducing timeouts from 2400s to 1200s.

Change-by-Change Review

1. tests/tasks/uipath-human-in-the-loop/TEST_PLAN.md

OK

Handle inventory row and per-test descriptions updated to match the new outcome-port semantics. Correctly documents the QuickForm vs App-based distinction (per-outcome ports vs static completed). e2e_07_apptask_brownfield is correctly left unchanged since AppTask uses completed.

2. tests/tasks/uipath-human-in-the-loop/e2e_01_invoice_approval_greenfield.yaml

OK — Most substantial change in the PR.

• Prompt simplified to mock SharePoint/SAP as Script nodes — well-motivated by the 2400s timeout blowout (119 turns of tangential connector discovery).
• Timeouts reduced from 2400/2400 to 1200/1200 (matches sibling brownfield tasks).
• expected_turns raised from 19 to 34 (matches the confirmed local run).
• max_turns lowered from 60 to 50 — reasonable since the shorter scenario is more predictable.
• Wiring criterion upgraded from flow_contains.py 'completed' to check_simulated_hitl.py outcome-wiring.
• Comment block documents the rationale with run IDs. Thorough.

3–6. e2e_02 through e2e_05 (HITL skill)

OK

All upgraded from file_contains: 'completed' or flow_contains.py 'completed' to check_simulated_hitl.py outcome-wiring. Prompts consistently reworded. Weights preserved.

7. e2e_06_invoice_approval_greenfield_simple.yaml

OK

Description and criterion updated consistently. Already used flow_contains.py; now uses check_simulated_hitl.py outcome-wiring.

8. quality_04_all_handles.yaml

OK

Criterion upgraded to check_simulated_hitl.py outcome-wiring. task_id (skill-hitl-quality-completed-handle-and-result) intentionally kept stable — correct, since changing identifiers would break historical run data.

9–14. quality_05, quality_07–quality_11 (HITL skill, prompt-only changes)

OK

Prompt wording updated from "completed handle" / →|completed| to "outcome port(s)" / explicit outcome routing. No outcome-wiring criterion added — these tests focus on other aspects (priority, runtime vars, field types, binding direction, inOut access) and the validate_flow.py check catches basic wiring errors. The prompt update ensures agents practice the correct wiring pattern even without a dedicated criterion.

15. uipath-maestro-flow/e2e/devcon_expense_approval.yaml

OK

Description updated; the actual criterion still calls check_devcon_expense_approval.py (which has its own assert_outcome_wiring internally). Correct.

16–18. maestro-flow/hitl/quality_01–quality_03 (prompt-only changes)

OK

quality_01 prompt changes from "Wire completed → decision node" to "Wire the outcome-approve and outcome-reject ports to their targets" — more prescriptive and technically accurate. quality_02 and quality_03 follow the same "every outcome needs its own wired port" pattern used consistently across the HITL tasks.

19. maestro-flow/hitl/quality_04_brownfield_insert.yaml

OK

Both prompt and criterion updated. Wiring description changed from HITL →|completed| End to HITL → End, with both outcomes wired to End. Criterion upgraded from flow_contains.py 'completed' to check_simulated_hitl.py outcome-wiring.

20. maestro-flow/hitl/smoke_02_completed_port_wired.yaml

OK

The most conceptually significant change. Description rewritten to explain per-outcome ports vs the generic outcome-completed placeholder. Prompt restructured to show the fork:

HITL →
  Approve: Script → End
  Reject: End

Comment block replaced: old explained v1.0 vs v1.1 port handle serialization; new explains node-scoped vs file-scoped checking. Both old and new criterion have weight: 3.0.

21. maestro-flow/hitl/smoke_03_multi_outcome_routing.yaml

OK

Prompt expanded with the standard "every outcome needs its own wired port" clause plus the Decision-node rationale. No outcome-wiring criterion added — this test's focus is Decision node routing, not port wiring per se, and validate_flow.py covers it.

22. maestro-flow/interactive/expense_approval_simulated.yaml

OK

Comment and criterion description updated; the command still calls check_simulated_hitl.py expense (which includes assert_outcome_wiring internally). Correct.

What's Missing

Nothing identified. The one unchanged file in scope (e2e_07_apptask_brownfield.yaml) correctly keeps completed because it's an AppTask (App-based, not QuickForm), and the TEST_PLAN distinguishes this. Tasks with prompt-only updates (quality_05, quality_07–11, smoke_03, maestro quality_01–03) intentionally omit an outcome-wiring criterion because they test other aspects and validate_flow.py covers basic wiring.

Area Ratings

Area	Status	Notes
Frontmatter	OK	No frontmatter changes (test YAMLs, not skills)
E2E Tests	OK	Upgraded from weak substring checks to node-scoped semantic checker
Skill Body	N/A	No skill files changed
References & Assets	OK	TEST_PLAN.md updated consistently
Repo Hygiene	OK	No secrets, no cross-skill deps, changes scoped to affected task files

Issues for Manual Review

• Remaining ~13 tasks not run through coder-eval: The PR body notes only smoke_02 and e2e_01 have passing runs. The prompt changes (especially the wiring fork diagrams in quality_08–quality_11 and quality_03) are more prescriptive than before, which should help agents, but confirming with a full eval pass would catch any cases where the new phrasing confuses agents.
• e2e_01 scenario simplification: Mocking SharePoint/SAP as Script nodes is well-justified for the timeout issue, but makes this task less representative of the real-world invoice approval flow. The original connector-discovery behavior is now only tested in the simulated interactive task. This is a deliberate tradeoff, not a bug.

Conclusion

Clean, well-scoped migration. The upgrade from file-scoped substring matching to node-scoped semantic checking (check_simulated_hitl.py outcome-wiring) is a meaningful improvement in criterion precision — it eliminates both false positives (unrelated node's completed port matching) and false negatives (wrong port wired but substring still present). Prompt changes are internally consistent and follow the same "every outcome needs its own wired port" pattern throughout.

Approve — no blocking issues. The main risk is the untested tasks, which the PR body already flags.