Skip to content

fix(desktop): harden macOS background computer use - #3434

Merged
bobleer merged 2 commits into
GCWing:mainfrom
bobleer:bob/cua-background-reliability
Oct 17, 2026
Merged

bobleer merged 2 commits into
GCWing:mainfrom
bobleer:bob/cua-background-reliability

Conversation

@bobleer

@bobleer bobleer commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Harden macOS background Computer Use when optional AX hit testing times out, the target loses its local focus, modifier chords are cancelled, or Stop races with input cleanup. Visual input keeps one window-directed delivery path, focus preparation is invalidated by a passive target-local loss monitor, and cancellation releases only the original process/window with validated routing.

Add a scoped production-tool JSONL probe, an independent passive foreground witness, and native timeout/focus-loss regressions. Record real-app comparisons with the installed Codex CUA implementation and remove unsupported parity claims. No new application-specific dispatch rules or foreground restoration are introduced.

Type and Areas

Type: Bug fix, tests, documentation.

Areas: Desktop / macOS native Computer Use.

Motivation / Impact

Background control should recover through target-local mechanisms instead of asking to take over the user's foreground. The change preserves explicit errors for AX mutations and invalid targets, prevents duplicate pointer delivery after an optional read failure, fixes modifier release flags, and serializes cleanup before capture teardown.

Verification

  • cargo test -p openbitfun-desktop --features devtools --lib computer_use -- --test-threads=1: 181 passed, 9 opt-in native tests ignored.
  • pnpm run check:repo-hygiene, git diff --check, and Node syntax checking passed.
  • Independent Objective-C witness compiled with -Wall -Wextra -Werror; read-only baseline/signal/output checks passed.
  • Real TextEdit, Edge, VS Code, and Calculator trials used disposable files/profiles and the full production Tool path. Text editing/saving, button activation, Unicode, multi-modifier selection, scroll, and calculation results were verified. Codex completed comparable ASCII/shortcut actions but its tested typeText calls omitted Chinese/emoji; static layout lookup is consistent with that observation.
  • Edge and Calculator OpenBitFun witness intervals retained foreground PID, physical cursor, and relative window order. Other intervals included concurrent human activity and are not unchanged-desktop evidence. No trial observed the target becoming foreground. The Codex Edge click/text interval had one transient target-above-foreground stacking sample; its cause was not isolated, so invariant reference window order is not claimed.
  • cargo build -p openbitfun-desktop: passed.
  • Final committed-head native reruns passed: node scripts/test-macos-input-controls.mjs --orchestration and node scripts/test-macos-input-controls.mjs --busy-ax. They retained the independent foreground/key-window, stacking, observer-input and physical-cursor assertions. The AX timeout case delivered exactly one pointer down/up and one activation; an additional read-only witness recorded 122 samples with an unchanged hardware mouse-movement count. Earlier gated runs and runs with cursor movement were not counted as passes.
  • Upstream CI passed on 80f6889c6cdd2a53ee6e4b3f3683040f8c73f98b: all 11 applicable checks succeeded, including Linux/macOS/Windows Rust and CLI checks, Frontend Build and the Rust / CLI Validation aggregate. The two impact-selected jobs were skipped by the existing workflow policy.

Reviewer Notes

The REA comparison covers shipped macOS capture, routing, focus, input, menus, cancellation and foreground behavior; it does not recover the full closed-source implementation. Source hash, evidence boundaries and reproducible commands are in the existing architecture/module guides.

No dependencies, persisted shapes or cross-version wire contracts change. Windows/Linux native delivery and remote workspace, remote control, Peer Device Mode and Detached Dispatch were not exercised; local checks do not establish remote behavior.

Checklist

  • This PR is focused and does not include secrets, temporary prompts, generated scratch files, or unrelated artifacts.
  • Relevant verification is recorded above, or skipped checks are explained.
  • User-facing strings, docs, and locales are updated where applicable.

@bobleer
bobleer marked this pull request as ready for review October 17, 2026 07:15
@bobleer
bobleer merged commit 7d2b0c3 into GCWing:main Oct 17, 2026
13 checks passed
@bobleer
bobleer deleted the bob/cua-background-reliability branch October 18, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant