Skip to content

fix(tools): make screenshots, keyboard targeting and launch_app usable for agents - #6

Draft
synthet wants to merge 2 commits into
mainfrom
fix/agent-ergonomics-from-transcripts
Draft

synthet wants to merge 2 commits into
mainfrom
fix/agent-ergonomics-from-transcripts

Conversation

@synthet

@synthet synthet commented Nov 9, 2026

Copy link
Copy Markdown
Owner

What and why

I parsed the Claude Code, Cursor, Codex and Antigravity transcripts that used win-gui-mcp (1,081 tool calls, mostly from projects other than this one). Three problems came up again and again.

  1. Screenshots couldn't be viewed. 56 of 70 screenshots were too big for the client's tool-result limit, because the PNG was base64 inside the JSON text. Another 10 came back as raw base64 the model couldn't see. Agents in several projects wrote their own decode scripts.
    → screenshot now returns an MCP image content block. The JSON holds only metadata, including origin and scale, so an image pixel maps back to a screen point: x = origin.x + round(px / scale).
  2. Keystrokes went to the wrong window. 164 of 167 type_text calls had no target and typed into whatever window was in front; twice that was the agent's own terminal.
    → type_text accepts handle. When given a handle, type_text, hotkey and press_key check that the window is in front before sending and fail with FocusFailed if it isn't. press_key re-checks before every repeat. Keyboard results report the receiving window as target.
  3. launch_app returned the wrong window or timed out. For a GPUI app it returned a zero-size helper window, and every wt.exe launch timed out.
    → Candidate windows are now ranked: on screen, not hidden (cloaked), not a tool window, real title, new since launch, in the foreground, then largest. When the launcher process exits, a titled window that appeared after the launch is accepted, which covers wt.exe and explorer.exe.

Also included:

  • Server instructions sent at initialize.
  • A project skill, win-gui-mcp-usage, with the operating rules learned from the transcripts. It includes the valid wait_for condition names; agents had guessed window_title_contains and window_idle.

Spec first: docs/tools.md (screenshot output, the new Keyboard input target section, type_text.handle, launch window selection) and docs/requirements/tools/*. I regenerated the golden tool surface: the diff is type_text.handle plus four tool descriptions.

Breaking change

MCP clients that read structuredContent.imageBase64 from screenshot must read the image content block instead. Code that calls ToolService.Screenshot directly still gets imageBase64.

Tests

  • Headless: 84 of 84 pass, including 10 new ones in AgentErgonomicsTests (image block shape, region/origin maths, main-window ranking including the Notepad session-restore case, type_text refusing an invalid handle).
  • Live stdio check against the WinForms sample:
    • the screenshot arrives as an image block with about 100 characters of metadata;
    • typing by handle reaches the text box and reports target;
    • inspect_at at pixels mapped through origin/scale finds the text box and the button;
    • wt.exe returns the new terminal window instead of Timeout.
  • Desktop gates:
    • Before rebasing onto feat: extract WinGuiMcp.Core and add declarative scenario runner #5, all 12 passed.
    • On this branch, 10 of 11 pass. NotepadGate_SaveToFileOnDisk fails, as does NotepadGate_SaveDialogOpensOnCtrlS when the class runs alone.
    • Two causes, both from the machine's state:
      • Notepad session restore: Notepad was set to restore its last session, so it reopened old windows instead of a fresh one. That's now switched off on this machine.
      • Focus-stealing protection: while the desktop is in active use, Windows won't bring a background-launched window to the front, so focus_window fails.
    • origin/main fails the same two tests in the same state with the same FocusFailed.
    • This is a draft until the gates are rerun on an idle desktop.

Known limits

  • When an app restores old windows at launch and none of them takes the foreground, the ranking can still choose a restored window. The usage skill warns not to use clearFirst on a window you didn't create.

🤖 Generated with Claude Code

synthet and others added 2 commits November 9, 2026 14:44
…e for agents

Agent transcripts (Claude Code, Cursor, Codex, Antigravity) showed three
recurring failures when other projects drove apps through win-gui-mcp:

- 56 of 70 screenshots exceeded the client's tool-result limit because the
  PNG travelled as base64 inside the JSON text; agents wrote ad-hoc decoders.
  Screenshots now return an MCP image content block plus metadata with
  `origin` and `scale`, so image pixels map back to screen points.
- 164 of 167 type_text calls had no target and typed into whatever window
  was in front (twice the agent's own terminal). type_text takes `handle`;
  keyboard tools verify the foreground before sending, stop with
  FocusFailed otherwise, and report the receiving window as `target`.
- launch_app returned a zero-size helper window for GPUI apps and timed out
  on every wt.exe launch. It now ranks candidates (on-screen, not cloaked,
  not a tool window, real title, new since launch, foreground, largest) and
  accepts a window that appeared after the launch when the launcher exited.

Also adds server instructions sent at initialize and a project
win-gui-mcp-usage skill with the operating rules learned from the
transcripts. Spec updated first in docs/tools.md and docs/requirements;
golden tool surface regenerated for the new type_text handle and the
updated descriptions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds a research note with the transcript-mined problems, root causes and
fix status (fixed in this PR vs still open), environment lessons for the
desktop gates, and matching entries in LESSONS_LEARNED.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant