Repository navigation
PLU-222: Fix evaluator flow: answer refinements when user delegates, surface frozen-task constraint - #9
Conversation
…rozen-task constraint
Two related orchestration issues in the eval flow:
1. When the user delegated task definition ("generate evals for my X agent"),
Claude was forwarding the platform's refinement questions back via
evals_ask_user — to a user who had no context to answer them. The MCP
responses now describe a WHO-answers rule: answer yourself from the agent
codebase when the user delegated, only call evals_ask_user for genuinely
user-only choices (reuse-vs-create, SLM-vs-LLM, integration language).
2. The Plurai platform freezes task_description at evals_start_evaluator —
subsequent evals_send_message calls only refine generated samples, never
the task. Claude was trying to "edit" the task via send_message, silently
leaving it wrong. A new platform_constraint field on both
evals_start_evaluator and the post-commit evals_send_message branch makes
the constraint explicit: to change the task, restart with a fresh
evals_start_evaluator call.
Also addresses review feedback: dropped the now-vestigial action_required
field (decision lives in instructions), restructured _send_message as
if/else, and softened the pre-commit instructions so they don't misfire on
status messages like "Generating examples...". Reconciled the
platform_constraint with the UI experience link the skill surfaces (UI is
preferred path for sample edits). Added presence + load-bearing-substring
assertions in tests/test_tools.py to lock the new fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ompts Four occurrences across skills/evaluator/SKILL.md and commands/eval.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eladlev
changed the title
There was a problem hiding this comment.
also I'd set commit to False just in case (as its persistent in state), check:
plurai-plugins/src/evals_mcp/state.py
Line 30 in 7682efd
| "platform_constraint": ( | ||
| "TASK DEFINITION IS FROZEN. The task_description passed here is " | ||
| "permanent for this evaluator — subsequent evals_send_message calls " | ||
| "only refine the generated samples, never the task itself (labels, " |
There was a problem hiding this comment.
labels is something the user can change (within the same task)
There was a problem hiding this comment.
Good catch — fixed in 2819eb7. Removed "labels" from the frozen-list in both MCP platform_constraint strings and in the skill/command markdown. Each spot now also explicitly says "Labels CAN still be changed within the same task" so Claude doesn't over-restrict.
| **Who answers the refinement questions.** Decide from how the user framed the request: | ||
|
|
||
| - **User defined the task** (supplied concrete label names, judging rules, or a labeled dataset; or invoked `/eval` with a detailed prompt): present the agent's refinement questions to the user by calling `evals_ask_user` with them rephrased as options. | ||
| - **User delegated task definition** (e.g. "generate evals for my <X> agent" without spelling out criteria, or you were invoked from another skill such as `plurai-simulation:generate-evals`): answer the refinement questions yourself using the agent codebase context and call `evals_send_message` with your composed answers. Do NOT bounce questions back to a user who has no context to answer them. |
There was a problem hiding this comment.
is that intentionally that specific? (the generate-evals skill)
There was a problem hiding this comment.
Not intentional — made it generic in 2819eb7. Both SKILL.md and eval.md now say "or you were invoked from another skill" without naming a specific one.
| "The agent returned refinement questions in agent_response. " | ||
| "First decide WHO answers them: " | ||
| "(a) If the user defined the task with concrete criteria — e.g. they " | ||
| "supplied label names, judging rules, or a labeled dataset — present " |
There was a problem hiding this comment.
Thanks — left as-is. "label names" here is listed as a criterion for user-defined tasks (something the user supplies up front to signal they've thought about the task), not as a frozen aspect of the task itself, so the wording stays.
| ) | ||
| result["platform_constraint"] = ( | ||
| "TASK DEFINITION IS FROZEN. If the user reacts to the samples by " | ||
| "asking to change labels, judging criteria, or task scope, do NOT " |
There was a problem hiding this comment.
Fixed in 2819eb7 — same change as the other "labels" thread. The post-commit platform_constraint now says "asking to change the judging criteria or task scope", with no mention of labels.
- Remove "labels" from frozen-task constraint (4 places: 2 in evaluator.py MCP responses, 1 in SKILL.md, 1 in commands/eval.md). Labels can be changed within the same task — only judging criteria and scope are frozen. - Reset state.committed = False in _start_evaluator. ServerState is long-lived, so the previous evaluator's True would leak into the new flow and mis-route the first _send_message follow-up. Added a regression test that pre-seeds committed=True and asserts the reset. - Drop specific "plurai-simulation:generate-evals" mention from the delegation example in SKILL.md and eval.md — "invoked from another skill" stays generic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Re: setting |
Summary
Two related orchestration fixes in the eval flow, both reported by the user:
evals_ask_user— to a user who had no context to answer them. The MCPinstructionsnow describe a WHO-answers rule: answer yourself from the agent codebase when the user delegated; only callevals_ask_userfor genuinely user-only choices (reuse-vs-create, SLM-vs-LLM, integration language).task_descriptionatevals_start_evaluator— subsequentevals_send_messagecalls only refine the generated samples, never the task. Claude was trying to "edit" the task viasend_message, silently leaving it wrong. A newplatform_constraintfield on bothevals_start_evaluatorand the post-commitevals_send_messagebranch makes this explicit: to change the task, restart with a freshevals_start_evaluatorcall.Both behaviors are reinforced in three places — MCP tool response
instructions(strongest signal),skills/evaluator/SKILL.md, andcommands/eval.md.Review polish (from
/pr-review-toolkit:review-pr)action_requiredfield — decision logic lives entirely ininstructions, so the enum-shaped field risked being read as authoritative over the longer text._send_messagerestructured asif state.committed: … else: …instead of the dual-if pattern.instructionsnow opens with "If agent_response contains a refinement question…" and explicitly handles status messages (e.g. "Generating examples…"), so the WHO-answers rule doesn't misfire on non-question responses.platform_constraintsays "for sample edits, prefer the UI experience link surfaced above; only edit viaevals_send_messageif the user explicitly asks" — reconciles with the UI link the skill already surfaces.tests/test_tools.pyfor the newplatform_constraintand pre-commitinstructionsfields, plus a negative assertion pinning the branch asymmetry.Test plan
uv run ruff check .— cleanuv run pyright src/evals_mcp/tools/evaluator.py— 0 errorsuv run pytest tests/ -q— 67/67 pass/evalwith a delegation-style prompt ("generate evals for my X agent") and confirm Claude answers refinement questions itself rather than bouncing them to the user🤖 Generated with Claude Code