PLU-222: Fix evaluator flow: answer refinements when user delegates, surface frozen-task constraint

commands/eval.md

@@ -8,20 +8,22 @@ allowed-tools: ["Bash", "Read", "Edit", "Write", "Glob", "Grep", "Agent"]
 
 **Case A — `Plurai API key not set`** (no key configured yet, e.g. fresh install or after `auth logout`):
 
-1. Ask the user (in chat) to paste their Plurai API key. If they don't have one, point them to https://app.plurai.ai/settings?tab=api-keys → **Create new key**. Warn that the key will appear in this conversation.
+1. Ask the user (in chat) to paste their Plurai API key. If they don't have one, point them to https://app.plurai.ai/settings?tab=api-keys → **Create new key**.
 2. Run `uv run --project ${CLAUDE_PLUGIN_ROOT} python -m evals_mcp auth login --key <KEY>` with the pasted key.
 3. On success (`Saved API key to <path>.`), retry the failed tool call. On failure, relay the stderr message to the user.
 
 **Case B — `Plurai API key invalid or expired`** (server returned 401: the on-disk key was rejected):
 
-The key currently on disk is bad. If you have a key from earlier in this conversation, that IS the rejected key — do NOT call `auth login` with it. You MUST ask the user (in chat, this turn) to paste a NEW, freshly-generated key from https://app.plurai.ai/settings?tab=api-keys → **Create new key**. Warn the key will appear in this conversation. Only after the user supplies a new key in this turn, run `uv run --project ${CLAUDE_PLUGIN_ROOT} python -m evals_mcp auth login --key <NEW_KEY>` and retry. Never silently auto-renew with a remembered key — the user must see the prompt and supply a fresh key.
+The key currently on disk is bad. If you have a key from earlier in this conversation, that IS the rejected key — do NOT call `auth login` with it. You MUST ask the user (in chat, this turn) to paste a NEW, freshly-generated key from https://app.plurai.ai/settings?tab=api-keys → **Create new key**. Only after the user supplies a new key in this turn, run `uv run --project ${CLAUDE_PLUGIN_ROOT} python -m evals_mcp auth login --key <NEW_KEY>` and retry. Never silently auto-renew with a remembered key — the user must see the prompt and supply a fresh key.
 
 Call `evals_search_evaluators` first as an optimization to check whether the user already has an evaluator in their Plurai workspace that fits this task. **If the list is empty, say nothing about it and proceed silently to create a new one** — a fresh user has no evaluators yet and should not be told something is missing. If one or more existing evaluators match, surface the full list to the user and use `evals_ask_user` to ask whether to reuse one or create a new one. If reusing, skip to providing the endpoint URL and API key.
 
 If creating new, call `evals_start_evaluator`.
 
 For `task_description`: 1-2 short sentences. Include the core task and desired label names if the user mentioned them. Do NOT include examples, detailed criteria, or long explanations.
 
+**Platform constraint — the task definition is frozen.** The `task_description` passed to `evals_start_evaluator` is permanent for that evaluator. Subsequent `evals_send_message` calls only refine the *generated samples* (add/remove/edit examples), never the task itself (judging criteria, scope). Labels CAN still be changed within the same task. If at any point — including after the user sees the samples — they want to change the underlying task, you MUST tell them the task can't be edited, confirm they want to restart, then call `evals_start_evaluator` again with a revised `task_description`. Do NOT try to amend the task via `evals_send_message`; it will silently leave the underlying task wrong while mutating samples.
+
 **Important — input template**: If the evaluation involves multiple fields (e.g. context + response for grounding, or a conversation), you MUST specify the input template explicitly in the task description. The evaluator receives a SINGLE text input, so all fields must be combined into one message using a clear template. Examples:
 
 - Grounding: "Input format: '## Context:\n{context}\n\n## Response:\n{response}'"
@@ -30,7 +32,12 @@ For `task_description`: 1-2 short sentences. Include the core task and desired l
 
 **Data path — if the user supplied a labeled data file** (e.g. `/eval --data path/to/file.csv` or they pasted a path): after `evals_start_evaluator`, read the file, parse it into `{sample, label, reasoning?}` records, then call `evals_upload_data` with the `example_set_id` from the start_evaluator response. Do NOT synthesize records. Continue with `evals_ask_user` using the refinement questions from the start_evaluator `agent_response`.
 
-Then follow the `instructions` field in the response — it tells you to call `evals_ask_user`.
+**Who answers the refinement questions.** Decide from how the user framed the request:
+
+- **User defined the task** (supplied concrete label names, judging rules, or a labeled dataset; or invoked `/eval` with a detailed prompt): present the agent's refinement questions to the user by calling `evals_ask_user` with them rephrased as options.
+- **User delegated task definition** (e.g. "generate evals for my <X> agent" without spelling out criteria, or you were invoked from another skill): answer the refinement questions yourself using the agent codebase context and call `evals_send_message` with your composed answers. Do NOT bounce questions back to a user who has no context to answer them.
+
+When in doubt, prefer answering yourself. Reserve `evals_ask_user` for genuinely user-only choices: reuse-vs-create when there's a search match, model choice (SLM vs LLM), and integration-snippet language.
 
 **Surfacing progress to the user.** After every `evals_send_message`, show the user the `agent_response` text verbatim so they see what the platform is doing (e.g. "I've generated 16 synthetic examples..."). Whenever a response includes `url`, you MUST display it to the user as a clickable markdown link in that turn — never silently move on.
 

skills/evaluator/SKILL.md

@@ -12,18 +12,25 @@ The user wants to build an LLM-as-a-judge evaluator. Use the Plurai platform MCP
 
 **Auth.** Handle the two error variants differently.
 
-- **`Plurai API key not set`** (no key configured): ask the user to paste their Plurai API key — if they don't have one, point them to https://app.plurai.ai/settings?tab=api-keys → **Create new key** (warn the key will appear in this conversation). Then run `uv run --project ${CLAUDE_PLUGIN_ROOT} python -m evals_mcp auth login --key <KEY>` and retry.
-- **`Plurai API key invalid or expired`** (server returned 401, on-disk key rejected): if you have a key from earlier in this conversation, that IS the rejected key — do NOT call `auth login` with it. You MUST ask the user (in chat, this turn) to paste a NEW, freshly-generated key from https://app.plurai.ai/settings?tab=api-keys → **Create new key** (warn the key will appear in this conversation). Only after the user supplies a new key in this turn, run `uv run --project ${CLAUDE_PLUGIN_ROOT} python -m evals_mcp auth login --key <NEW_KEY>` and retry. Never silently auto-renew with a remembered key.
+- **`Plurai API key not set`** (no key configured): ask the user to paste their Plurai API key — if they don't have one, point them to https://app.plurai.ai/settings?tab=api-keys → **Create new key**. Then run `uv run --project ${CLAUDE_PLUGIN_ROOT} python -m evals_mcp auth login --key <KEY>` and retry.
+- **`Plurai API key invalid or expired`** (server returned 401, on-disk key rejected): if you have a key from earlier in this conversation, that IS the rejected key — do NOT call `auth login` with it. You MUST ask the user (in chat, this turn) to paste a NEW, freshly-generated key from https://app.plurai.ai/settings?tab=api-keys → **Create new key**. Only after the user supplies a new key in this turn, run `uv run --project ${CLAUDE_PLUGIN_ROOT} python -m evals_mcp auth login --key <NEW_KEY>` and retry. Never silently auto-renew with a remembered key.
 
 Call `evals_search_evaluators` first as an optimization to see whether the user already has an evaluator in their Plurai workspace that fits this task. **If the list is empty, say nothing about it and proceed silently to create a new one** — a new user has no evaluators yet and should not be told something is missing. If one or more existing evaluators match, surface the full list to the user and use `evals_ask_user` to ask whether to reuse one or create a new one. If reusing, provide the endpoint URL and API key.
 
 If creating new, call `evals_start_evaluator`. For `task_description`: 1-2 short sentences. Include task + desired label names.
 
+**Platform constraint — the task definition is frozen.** The `task_description` passed to `evals_start_evaluator` is permanent for that evaluator. Subsequent `evals_send_message` calls only refine the *generated samples* (add/remove/edit examples), never the task itself (judging criteria, scope). Labels CAN still be changed within the same task. If at any point — including after the user sees the samples — they want to change the underlying task, you MUST tell them the task can't be edited, confirm they want to restart, then call `evals_start_evaluator` again with a revised `task_description`. Do NOT try to amend the task via `evals_send_message`; it will silently leave the underlying task wrong while mutating samples.
+
 **Input template**: If evaluation involves multiple fields (context + response, conversation turns, etc.), specify the template in the task description. The evaluator receives ONE text input — all fields must be in a single message. E.g. "Input format: '## Context:\n{context}\n\n## Response:\n{response}'"
 
 If the user provides a labeled data file, after `evals_start_evaluator` call `evals_upload_data` with the parsed records and `example_set_id`. Continue with `evals_ask_user` using the refinement questions from the start_evaluator response.
 
-Then follow the `instructions` field in the response — it tells you to call `evals_ask_user`.
+**Who answers the refinement questions.** Decide from how the user framed the request:
+
+- **User defined the task** (supplied concrete label names, judging rules, or a labeled dataset; or invoked `/eval` with a detailed prompt): present the agent's refinement questions to the user by calling `evals_ask_user` with them rephrased as options.
+- **User delegated task definition** (e.g. "generate evals for my <X> agent" without spelling out criteria, or you were invoked from another skill): answer the refinement questions yourself using the agent codebase context and call `evals_send_message` with your composed answers. Do NOT bounce questions back to a user who has no context to answer them.
+
+When in doubt, prefer answering yourself. Reserve `evals_ask_user` for genuinely user-only choices: reuse-vs-create when there's a search match, model choice (SLM vs LLM), and integration-snippet language.
 
 **Surfacing progress to the user.** After every `evals_send_message`, show the user the `agent_response` text verbatim so they see what the platform is doing (e.g. "I've generated 16 synthetic examples..."). Whenever a response includes `url`, you MUST display it to the user as a clickable markdown link in that turn — never silently move on.
 

src/evals_mcp/tools/evaluator.py

@@ -214,17 +214,41 @@ async def _start_evaluator(args: StartEvaluatorArgs, ctx: Context[Any, Any, Any]
     snapshot = await state.agent.get_state(thread.id)
     agent_response = snapshot.last_assistant_message()
     state.has_questions = True
+    # Reset persistent state — a prior evaluator on this server may have
+    # committed, and the leaked True would mis-route the post-commit branch
+    # of _send_message on the very first follow-up.
+    state.committed = False
 
     return {
         "thread_id": thread.id,
         "example_set_id": thread.example_set_id,
         "url": f"{settings.api_base.rstrip('/')}/thread/{thread.id}",
         "agent_response": agent_response,
-        "action_required": "PRESENT_QUESTIONS_TO_USER",
+        "platform_constraint": (
+            "TASK DEFINITION IS FROZEN. The task_description passed here is "
+            "permanent for this evaluator — subsequent evals_send_message calls "
+            "only refine the generated samples, never the task itself "
+            "(judging criteria, scope). If the user later wants to change the "
+            "task, you MUST start a fresh evaluator by calling "
+            "evals_start_evaluator again with a revised task_description. "
+            "Never try to amend the task via evals_send_message."
+        ),
         "instructions": (
-            "The agent returned refinement questions. "
-            "Call evals_ask_user with the questions rephrased as options. "
-            "Do NOT present the questions as text."
+            "The agent returned refinement questions in agent_response. "
+            "First decide WHO answers them: "
+            "(a) If the user defined the task with concrete criteria — e.g. they "
+            "supplied label names, judging rules, or a labeled dataset — present "
+            "the questions to the user by calling evals_ask_user with them "
+            "rephrased as options. "
+            "(b) If the user DELEGATED task definition — e.g. 'generate evals for "
+            "my <X> agent' without spelling out criteria, or you were invoked "
+            "from another skill — answer the questions yourself using the agent "
+            "codebase context and call evals_send_message with your composed "
+            "answers. Do NOT bounce questions back to a user who has no context "
+            "to answer them. "
+            "When in doubt, prefer answering yourself. Reserve evals_ask_user "
+            "for genuinely user-only choices (reuse-vs-create, SLM-vs-LLM, "
+            "integration language)."
         ),
     }
 
@@ -287,6 +311,28 @@ async def _send_message(args: SendMessageArgs, ctx: Context[Any, Any, Any]) -> d
             "Do NOT add an explicit Other option, and do NOT add any extra "
             "confirmation question before this ask."
         )
+        result["platform_constraint"] = (
+            "TASK DEFINITION IS FROZEN. If the user reacts to the samples by "
+            "asking to change the judging criteria or task scope, do NOT "
+            "send that as a chat message — evals_send_message can only refine "
+            "samples, not the task. Tell the user the task can't be edited, "
+            "confirm they want to restart, then call evals_start_evaluator "
+            "with a revised task_description. For sample edits "
+            "(add/remove/modify individual examples), prefer the UI experience "
+            "link surfaced above; only edit via evals_send_message if the user "
+            "explicitly asks to do it through chat."
+        )
+    else:
+        result["instructions"] = (
+            "If agent_response contains a refinement question, apply the "
+            "WHO-answers rule from evals_start_evaluator: if the user delegated "
+            "task definition, answer yourself via evals_send_message from the "
+            "agent codebase context; otherwise call evals_ask_user. When in "
+            "doubt, answer yourself — do NOT bounce questions back to a user "
+            "who has no context. If agent_response is a status message (e.g. "
+            "'Generating examples...'), just surface it verbatim and wait — "
+            "no follow-up tool call is required."
+        )
     return result
 
 

tests/test_tools.py

@@ -520,15 +520,24 @@ async def test_start_evaluator_happy_path(
         }
     )
 
+    # Seed leftover state from a prior evaluator on the same server — the
+    # post-commit branch of _send_message keys off this flag, so a leaked
+    # True would mis-route the first follow-up of the new evaluator.
+    ctx.request_context.lifespan_context.committed = True
+
     out = await _start_evaluator(
         StartEvaluatorArgs(task_description="Classify outputs as safe or unsafe"),
         ctx,
     )
     assert out["thread_id"] == "thread-1"
     assert out["example_set_id"] == "es-1"
-    assert out["action_required"] == "PRESENT_QUESTIONS_TO_USER"
+    assert "action_required" not in out
     assert out["agent_response"] == "What labels?"
+    assert "platform_constraint" in out
+    assert "FROZEN" in out["platform_constraint"]
+    assert "evals_start_evaluator" in out["platform_constraint"]
     assert ctx.request_context.lifespan_context.has_questions is True
+    assert ctx.request_context.lifespan_context.committed is False
 
 
 # ── ask_user: gating + decline-fallback ──────────────────────────────────
@@ -652,6 +661,12 @@ async def test_send_message_surfaces_url_when_commit_id_present(
     assert "SLM" in instructions and "LLM" in instructions
     assert "Ready to optimize" not in instructions
     assert "review-confirm" not in instructions.lower()
+    # Frozen-task constraint must be surfaced post-commit, so a user who asks
+    # to "change the task" after seeing samples gets a restart, not a silent
+    # sample-only edit.
+    assert "platform_constraint" in out
+    assert "FROZEN" in out["platform_constraint"]
+    assert "evals_start_evaluator" in out["platform_constraint"]
     assert state.committed is True
     # Re-armed so the next ask_user (optimization choice) is allowed through.
     assert state.has_questions is True
@@ -678,6 +693,13 @@ async def test_send_message_no_url_when_no_commit_id(
 
     out = await _send_message(SendMessageArgs(thread_id="thread-1", message="more context"), ctx)
     assert "url" not in out
+    # Frozen-task constraint is only meaningful post-commit; the refinement
+    # branch must not leak it.
+    assert "platform_constraint" not in out
+    # Pre-commit branch now carries WHO-answers guidance — pin the contract
+    # by name (evals_ask_user) without freezing the wording.
+    assert "instructions" in out
+    assert "evals_ask_user" in out["instructions"]
     assert state.committed is False
     # Refinement question detected → has_questions re-armed via the '?' branch.
     assert state.has_questions is True