Debugging is also clicking 🖱️

In the last couple of posts I let agents debug over DAP —
breakpoints, step over, continue. That's real debugging. But it's only half of it.
When I debug something for real, I also click: I press the button and watch what
happens, read the dialog, notice the toggle is greyed out. No backtrace ever tells
you the Save button never enabled. So — can the agent do that half too?

The web is the easy case

Browsers are automatable by design. Most agent tools ship their own browser or
drive an external one; point Playwright at a page and
every element has a stable, queryable handle. The DOM is an accessibility tree
wearing a different hat — roles, labels, structure, all there for the reading. For
the web, this half of debugging is close to solved.

Native apps are another game

There's no DOM. When the agent has nothing to go on, it falls back to the eyeball
approach: take a screenshot, let the model look, maybe run OCR or a pre-analysis
pass to label what's on screen. It works — and sometimes it's the only option — but
it's brittle (a few pixels off and the click misses) and it burns tokens describing
pictures.

I ran into this by accident. I once wrote a tiny skill whose only job was to
screenshot a running 4D form and stitch an animated GIF for a README —
4d-capture-gif.
Then I noticed Claude Code reaching for it to debug: the skill also reports a bit
of the form's structure — where the buttons are — so the agent knows where to
click. For simple cases it genuinely works. But screenshots-plus-coordinates is not
the thing I want to build on.

The cleaner path: read the tree, don't look at pixels

Instead of staring at the screen, read the UI tree directly. On macOS you can
script the Accessibility API from Python (pyobjc), and there are automation
libraries to help. Now you're clicking element #37, the "Save" button instead of
coordinate (412, 260), and hope.

A couple of open-source tools are pushing exactly here:

• agent-desktop — a native CLI that exposes any app's accessibility tree as snapshot / click / type / press with structured JSON. No screenshots, precise element references. Its progressive skeleton mode drills down the hierarchy instead of dumping it whole, cutting prompt tokens dramatically (they cite 78–96% on dense apps). And it ships a C-ABI library, so Python, Swift, Go or Node can call it in-process instead of shelling out per action. (macOS today, more platforms in progress; needs Accessibility and Screen Recording permissions.)
• Open Interface — the other end of the spectrum: an LLM (GPT-4o/4V, Gemini…) reads your request, drives the real mouse and keyboard, and re-screenshots to course-correct. Demos include solving Wordle and editing a Google Doc. End-to-end agentic desktop use, screenshots in the loop.

Faire d'une pierre deux coups

Here's the thing I keep circling back to. If you build a native macOS or iOS app and
you draw your own components, there's nothing underneath them — the agent has no
tree to grab, so it's back to pixels. But if you implement the accessibility tree
for those custom components, you get faire d'une pierre deux coups — kill two birds
with one stone:

1. an AI agent can drive your UI by real element references, and
2. — the one that actually matters — people using VoiceOver, Switch Control or Voice Control can use your app.

The automation win comes for free, riding on the accessibility win. That's a rare
kind of alignment: the boring, decades-old right-thing-to-do turns out to be the
same work that makes your app agent-ready.

And Apple already tells you how, for Cocoa, UIKit and even SwiftUI. The WWDC26
session Refine accessibility for custom controls
walks through labels, values, adjustable actions and custom actions for exactly this
case — their example turns a custom coffee-dispenser control that assistive tech
could only read as a bare "button" into an adjustable one you swipe to change.
There's written guidance too — AppKit's
custom controls documentation —
and a whole library of accessibility sessions
behind it.

A 4D aside, and a note to self: I suspect that in 4D a form window is drawn as
essentially one big view, with no accessibility tree exposed underneath — which
would explain why the agent has to fall back to screenshots. That's something I want
to investigate.

Something will emerge

LLM-driven desktop automation is a moving space — open tooling and academic interest
are both growing, and I'd bet the right abstraction hasn't shown up yet. Something
will. Meanwhile, if you're drawing custom UI, the move is clear enough: give it an
accessibility tree. The people who rely on it need it — and it happens to be exactly
what an agent needs to help you debug your own app. 🖱️