Canopy

Turn the operating system's accessibility tree into a compact, agent-readable view of any app's UI.

Canopy reads the same semantic UI description that screen readers (VoiceOver, NVDA, Orca) consume — a tree of elements, each with a role, a name, a value, states, and a bounding box — and serializes it into terse text a reasoning model can read directly.

This is the cheaper, more reliable alternative to vision-based computer-use. Instead of screenshotting pixels, running OCR, and guessing coordinates, Canopy asks the OS what is on screen and gets back structured truth: this is a button named "Save", it is disabled; this is the focused search field, it currently contains "fox". No pixels, no OCR, a fraction of the tokens.

application "TextEdit"  #0
  window "Untitled — Edited"  #0.0
    toolbar  #0.0.0
      button "Bold"  #0.0.0.0
      button "Save" [disabled]  #0.0.0.3
      textfield "Search" [focused] = "fox"  #0.0.0.4
    text = "Hello world"  #0.0.1
    textarea = "The quick brown fox jumps over the lazy dog. Pack my box with…"  #0.0.2
    group "Account"  #0.0.3
      textfield "Username" = "ada"  #0.0.3.0
      textfield "Password" = <redacted>  #0.0.3.1
    list "Recent Documents"  #0.0.4
      listitem "report.txt"  #0.0.4.0
      listitem "notes.txt" [selected]  #0.0.4.1
      … 5 more children elided

That is real output (canopy capture --demo). Note what already happened: layout-group noise collapsed, split text runs merged, the password value withheld, the long document clipped, keyboard focus and disabled-ness surfaced, and an over-long list honestly truncated.

---

Status

• v1 targets macOS (the Accessibility / "AX" API via pyobjc).
• Read-only. Canopy serializes the UI; it does not click or type. Element invocation is a planned v2 — the data model already carries the stable identity needed for it, so it is an additive change, not a rewrite.
• Windows (UI Automation) and Linux (AT-SPI) are present as honest stubs: the architecture has seams for them, and they report exactly which dependency they will need.

The entire platform-independent core (normalization, pruning, serialization, diffing) is covered by tests that run on any OS. The macOS capture layer requires a Mac and Accessibility permission.

---

Install

# from the project directory, on macOS:
pip install -e '.[macos]'

The [macos] extra pulls the two pyobjc framework packages Canopy needs (pyobjc-framework-Cocoa, pyobjc-framework-ApplicationServices). The core itself has zero runtime dependencies, so pip install -e . (no extra) installs a working install everywhere — you just cannot do a live capture without a backend.

Requires Python 3.9+.

---

Quickstart

1. Check your environment and permission:

canopy diagnose

This reports your platform, whether pyobjc is importable, whether this process is trusted for Accessibility, and runs a tiny end-to-end smoke read. If permission is missing it tells you exactly what to grant. To make macOS pop the permission dialog:

canopy diagnose --prompt

The one thing everyone gets stuck on: macOS attaches Accessibility permission to the process that launched Canopy — your terminal, iTerm, or IDE — not to the Python file. If you run from Terminal, you authorize Terminal. See docs/MACOS.md.

2. Capture the frontmost window:

canopy capture

3. Capture a specific app, as JSON:

canopy capture --app Safari --json

4. Watch for changes (prints only deltas):

canopy watch --app Notes --interval 0.5

5. See the output format without a Mac or permission:

canopy capture --demo

---

The output format

Canopy emits one of two formats.

Outline (default)

One line per element, indentation encoding hierarchy:

<role> "<name>" [states] = "<value>"  #<id>

• role — the normalized role (button, textfield, checkbox, …).
• name — the accessible label, in quotes. Omitted if empty.
• states — only notable states in brackets. enabled is the common case and is never printed; absence of [disabled] means enabled. [focused] is printed, because where keyboard focus sits is high-value. Other states: selected, checked/unchecked, expanded/collapsed, required, busy, hidden.
• value — current contents/position, after =, clipped to keep lines short. Secure (password) fields render = <redacted> and never expose their contents.
• #id — the element's address (its path from the root). Use it to refer to an element ("invoke #0.0.3.1"). Turn off with --no-ids.

When a node had children Canopy chose not to capture (budget or depth limit), it prints a line like … 5 more children elided rather than silently pretending the node was a leaf.

JSON (--json)

Full fidelity — every field, including identifier, subrole, role_raw, bbox, actions — for programmatic consumers, for recording fixtures, and for diffing. Empty/default fields are omitted to keep it compact.

---

CLI reference

canopy [--version] <command> [options]

If you omit the command, capture is assumed (canopy == canopy capture).

capture (default)

Option	Meaning
--app NAME	Target app by localized name or bundle id. Default: frontmost.
--pid N	Target app by process id (takes precedence over --app).
--window {auto,focused,main,all}	Which window. auto = focused, else main, else first. all = whole app, including menus. Default auto.
--depth N	Max tree depth (default 40).
--max-nodes N	Max nodes to capture (default 4000) — the safety valve against giant trees.
--timeout SEC	Per-message AX timeout (default 2.0) — bounds how long a wedged app can stall a capture.
--geometry	Capture bounding boxes (in points).
--drop-offscreen	Prune nodes outside the screen (implies --geometry; best-effort on macOS).
--no-enhance	Do not force-enable Electron/Chromium accessibility.
--no-prune	Disable structural pruning (emit the raw tree).
--no-ids / --no-states	Drop #id addresses / [state] tags from the outline.
--value-width N	Clip values to N chars (default 120).
--json	Emit JSON instead of the outline.
--quiet	Suppress the stats line (printed to stderr).
--demo	Render the bundled sample tree (no live capture; works anywhere).

stdout carries the tree (pipe it freely); the stats line goes to stderr.

watch

All capture options, plus --interval SEC (default 1.0). Prints the full tree once, then only the diff on each subsequent tick. Ctrl-C to stop.

diagnose

--prompt pops the macOS Accessibility dialog for the current host process.

version

Prints the version.

---

Library API

Everything the CLI does is available programmatically. import canopy is safe on any platform — backends are imported lazily, so it never drags in pyobjc until you request a capture on macOS.

import canopy

backend = canopy.get_backend()

avail = backend.is_available()
if not avail.ok:
    raise SystemExit(avail.reason + "\n" + "\n".join(avail.fixes))

result = backend.capture(canopy.CaptureOptions(app="Safari", window="focused"))

print(canopy.to_outline(result.root))     # compact text
print(canopy.to_json(result.root))        # full JSON
print(result.stats)                        # node counts, depth, redactions, timing

# Walk it yourself:
for node in result.root.walk():
    if node.role is canopy.Role.BUTTON and "disabled" not in node.states:
        print("clickable:", node.name, node.id)

Diff two captures (what watch uses):

before = backend.capture(canopy.CaptureOptions(app="Notes")).root
# ... user does something ...
after = backend.capture(canopy.CaptureOptions(app="Notes")).root
print(canopy.diff_trees(before, after).to_text())

See the public surface in canopy/__init__.py.

---

How it works (the short version)

Canopy is a ports-and-adapters design. One normalized core does all the thinking and is fully platform-independent; a thin per-OS backend does the OS-specific reading behind a single interface.

              ┌─────────────────────── platform-independent ───────────────────────┐
  AX / UIA /  │                                                                     │
  AT-SPI  ──► │  Backend ──► traverse (BFS + budget) ──► Node tree ──► prune ──►     │ ──► outline / json
  (per OS)    │  (adapter)        ▲ Visitor seam                       serialize     │ ──► diff (watch)
              │                                                                     │
              └─────────────────────────────────────────────────────────────────────┘

The hard real-world constraint is IPC latency: every attribute read is a cross-process round trip, so a naive full-depth walk of a complex window can take seconds or hang. Canopy answers that with per-message timeouts, a breadth-first walk under a node budget (so the budget is spent on shallow, important elements), and honest elision when limits bite. Then a pure pruning pipeline removes the structural noise, and a serializer turns the result into terse text.

Full detail in docs/ARCHITECTURE.md.

---

What's verified, and what needs your machine

• Verified on any OS (via the test suite, pytest): the node model, the traversal budgeting (depth caps, node budgets, elision) via a fake visitor, the entire pruning pipeline, both serializers, and the diff engine. 31 tests.
• Needs a Mac + Accessibility permission: the live AX capture itself. Start with canopy diagnose, which isolates exactly where any problem is (platform / dependency / permission / smoke read).

Honest limitations

• Read-only in v1 (no clicking/typing yet).
• macOS only in v1.
• On macOS, "offscreen" pruning is best-effort via geometry, because AX has no offscreen flag (UIA and AT-SPI do). Enable it with --drop-offscreen.
• Accessible-name computation is deliberately simple (title → description → static-text value); it does not yet follow AXTitleUIElement label links. See docs/MACOS.md.
• This reads everything on screen into a model's context. Secure fields are redacted by default, but treat captured text as sensitive, and be aware that on-screen text is a prompt-injection surface just as web content is.

---

Roadmap

• v2 — invocation. Re-resolve an element by id and act on it (AXPress on macOS; UIA control patterns on Windows). The node model already carries stable identity for this.
• v2 — Windows & Linux backends. UIA (with CacheRequest batching) and AT-SPI over D-Bus. See docs/EXTENDING.md.
• v2 — vision hybrid. Where pruning leaves a hole (a bare canvas, an unlabeled custom widget), drop in a screenshot of just that subtree.
• MCP server. Expose capture_tree (and later invoke_element) as tools so an agent calls Canopy like any other tool.

---

Documentation

• docs/ARCHITECTURE.md — the design, data flow, every module, the traversal and pruning algorithms, the testing strategy.
• docs/MACOS.md — the AX API in depth: permission/TCC, the responsible-process trap, the Electron trick, Retina, the role table, troubleshooting.
• docs/EXTENDING.md — how to add a backend, the Backend contract, role mapping, the invoke seam.

License

MIT.