[benv2k6]

No description provided.

[linear]

PLU-212 evals plugin: Data Canvas link not shown after synthetic data generation

Symptom

After answering the refinement questions in /evals:eval, the orchestrator skipped showing the Data Canvas link entirely and jumped straight to the SLM-vs-LLM elicitation. The user only saw:

16 synthetic examples were generated. How would you like to optimize the evaluator?

…with no link to review/edit the generated data.

Root cause

Two issues stacked:

1. Wrong signal. The MCP gate for surfacing the URL + post-data-generation instructions was state.initial_flow_done, sourced from STATE_SNAPSHOT.snapshot.initial_flow_done. The Plurai agent does not emit that field. The actual signal is commit_id — non-null means the synthetic example set has been committed and the user can review it.
2. Heavy / gated instructions. The post-flow-done instructions string mandated a "Have you reviewed?" confirmation before SLM/LLM. That gate isn't desired, and it bloated the directive enough that the orchestrator dropped the URL display step.

Fix (shipped on docs/login-key-instructions)

Code changes in pluto-judge repo (commit pending):

• src/evals_mcp/tools/judge.py
  • AgentStateSnapshot.initial_flow_done: bool → commit_id: str | None.
  • _parse_run now reads snap["commit_id"] from STATE_SNAPSHOT.
  • _send_message / _start_judge mirror commit_id into state.commit_id.
  • Post-data-gen branch gated on state.commit_id is not None.
  • Rewrote the instructions string: show agent_response verbatim → share URL as clickable markdown link → call evals_ask_user with the SLM/LLM question. No review-confirm gate.
  • Added logger.info("agent_run_parsed", event_types=…, last_snapshot_keys=…, classifier_id=…, commit_id=…) so future regressions are debuggable from the MCP log.
• src/evals_mcp/state.py — initial_flow_done: bool → commit_id: str | None.
• skills/judge-evaluator/SKILL.md and commands/eval.md — collapsed steps 2–3 into one "Surface link + ask SLM/LLM" step; tightened the URL-display rule to mandatory.
• tests/test_tools.py — replaced the initial_flow_done tests with commit_id equivalents; all 11 tool tests pass.

Verification

• Local: uv run ruff check . && uv run ruff format --check . && uv run pyright src/ tests/test_tools.py && uv run pytest tests/test_tools.py — clean.
• End-to-end: run /evals:eval, answer refinement questions; confirm orchestrator emits the Data Canvas markdown link before the SLM/LLM elicitation. New log line in ~/Library/Caches/claude-cli-nodejs/<workspace>/mcp-logs-plugin-evals-evals/<timestamp>.jsonl shows commit_id per call for debugging.

Follow-ups (out of scope for this ticket)

• Confirm with the agent team that commit_id is the canonical "initial flow done" signal across all flows (no other variant we should also recognize).
• Decide whether to also surface commit_id to consumers (currently only used as a flag).

Review in Linear