Repository navigation
Conversation
…e for agents Agent transcripts (Claude Code, Cursor, Codex, Antigravity) showed three recurring failures when other projects drove apps through win-gui-mcp: - 56 of 70 screenshots exceeded the client's tool-result limit because the PNG travelled as base64 inside the JSON text; agents wrote ad-hoc decoders. Screenshots now return an MCP image content block plus metadata with `origin` and `scale`, so image pixels map back to screen points. - 164 of 167 type_text calls had no target and typed into whatever window was in front (twice the agent's own terminal). type_text takes `handle`; keyboard tools verify the foreground before sending, stop with FocusFailed otherwise, and report the receiving window as `target`. - launch_app returned a zero-size helper window for GPUI apps and timed out on every wt.exe launch. It now ranks candidates (on-screen, not cloaked, not a tool window, real title, new since launch, foreground, largest) and accepts a window that appeared after the launch when the launcher exited. Also adds server instructions sent at initialize and a project win-gui-mcp-usage skill with the operating rules learned from the transcripts. Spec updated first in docs/tools.md and docs/requirements; golden tool surface regenerated for the new type_text handle and the updated descriptions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds a research note with the transcript-mined problems, root causes and fix status (fixed in this PR vs still open), environment lessons for the desktop gates, and matching entries in LESSONS_LEARNED. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
I parsed the Claude Code, Cursor, Codex and Antigravity transcripts that used win-gui-mcp (1,081 tool calls, mostly from projects other than this one). Three problems came up again and again.
→
screenshotnow returns an MCPimagecontent block. The JSON holds only metadata, includingoriginandscale, so an image pixel maps back to a screen point:x = origin.x + round(px / scale).type_textcalls had no target and typed into whatever window was in front; twice that was the agent's own terminal.→
type_textacceptshandle. When given a handle,type_text,hotkeyandpress_keycheck that the window is in front before sending and fail withFocusFailedif it isn't.press_keyre-checks before every repeat. Keyboard results report the receiving window astarget.launch_appreturned the wrong window or timed out. For a GPUI app it returned a zero-size helper window, and everywt.exelaunch timed out.→ Candidate windows are now ranked: on screen, not hidden (cloaked), not a tool window, real title, new since launch, in the foreground, then largest. When the launcher process exits, a titled window that appeared after the launch is accepted, which covers
wt.exeandexplorer.exe.Also included:
initialize.win-gui-mcp-usage, with the operating rules learned from the transcripts. It includes the validwait_forcondition names; agents had guessedwindow_title_containsandwindow_idle.Spec first:
docs/tools.md(screenshot output, the new Keyboard input target section,type_text.handle, launch window selection) anddocs/requirements/tools/*. I regenerated the golden tool surface: the diff istype_text.handleplus four tool descriptions.Breaking change
MCP clients that read
structuredContent.imageBase64fromscreenshotmust read theimagecontent block instead. Code that callsToolService.Screenshotdirectly still getsimageBase64.Tests
AgentErgonomicsTests(image block shape, region/origin maths, main-window ranking including the Notepad session-restore case,type_textrefusing an invalid handle).handlereaches the text box and reportstarget;inspect_atat pixels mapped throughorigin/scalefinds the text box and the button;wt.exereturns the new terminal window instead ofTimeout.NotepadGate_SaveToFileOnDiskfails, as doesNotepadGate_SaveDialogOpensOnCtrlSwhen the class runs alone.focus_windowfails.origin/mainfails the same two tests in the same state with the sameFocusFailed.Known limits
clearFirston a window you didn't create.🤖 Generated with Claude Code