A screenshot is not enough for a screen-aware Mac assistant

A screenshot can show me what is on a Mac screen. It cannot tell me, by itself, what a control means, whether an action succeeded, or how much authority an assistant should have.

That is why I think a screen-aware assistant needs more than vision. It needs a bounded observation and action loop.

Three kinds of screen context

I separate screen context into three sources:

1. Accessibility structure: roles, labels, values, focus, and actions exposed by macOS. This is usually the strongest source for semantic controls.
2. Visual context: pixels, OCR, or detected marks when accessibility information is incomplete.
3. Interaction state: the active window, recent user input, and the visible change after an action.

Each source has gaps. Accessibility trees can be incomplete. Visual models can misread coordinates. A visible change can happen without proving the intended outcome. A reliable assistant cross-checks the sources and fails visibly when confidence is not good enough.

Use an action loop, not a click generator

The loop I want is observe, plan, request approval when required, act, and verify.

For a click, verification might mean confirming that an accessibility value changed or a new window appeared. For a guided workflow, the assistant should advance only after the user's action produces the expected state. A coordinate is an implementation detail, not a durable result.

Keep the observation bounded

Screen access is powerful, so the assistant should collect only what the current task needs. It should not turn temporary context into a permanent archive by default.

The practical rules are straightforward:

• prefer accessibility data before pixels
• capture visual context only when necessary
• keep observations local in local mode
• make off-device planning visible
• let the user disable screen and journal sources
• preserve an audit trail for actions without retaining unrelated screen history

How Pace approaches it

Pace can combine local accessibility context, OCR, and a local vision model. Its action path scores possible targets, verifies the result, and keeps recovery paths for ambiguous controls. It can also draw guidance on screen and wait for the user to complete the expected step.

The important part is not that the assistant can click. It is that every observation and action stays within a visible trust boundary.

I documented the current screen model, privacy boundary, and limitations at https://heypace.app/screen-aware-ai-assistant-mac/.