A useful Mac assistant needs to do more than generate instructions. It also needs to do less than silently control the computer.
The boundary I want is an observable action loop: understand the request, inspect the current state, propose typed steps, request approval when risk requires it, act through the strongest available interface, and verify what changed.
Prefer semantic actions
Application APIs, App Intents, Shortcuts, MCP tools, and the macOS Accessibility API expose meaning and state. They are usually more durable than raw screen coordinates.
Visual grounding and pointer events still matter for applications with incomplete semantic access, but I treat them as a fallback. A coordinate can tell the system where input was sent. It cannot prove what the control meant or whether the outcome was correct.
Keep planning separate from authority
A model can propose an action. Policy should decide whether it may run.
Read-only navigation, reversible edits, external messages, downloads, and destructive mutations require different treatment. Planner output should become typed actions that can be validated before execution, not free-form instructions with ambient authority.
Verify every material step
A click event proves only that input was sent. Verification should inspect the resulting window, control value, file, reminder, message draft, or other expected state.
If the result is ambiguous, the assistant should stop or recover. It should not continue a long workflow on an assumed success.
Pace's current action stack
Pace combines a typed tool registry, a plan-act-observe loop, application integrations, App Intents and deeplinks, accessibility and pointer execution, click scoring and recovery, undo for supported reversible mutations, failure narration, and local action history.
Users can also record literal flows, install recipes, or teach natural-language skills that are re-grounded against the current screen. MCP connections and external services remain optional.
Start with bounded workflows
The best early workflows have outcomes that are easy to inspect:
- open an application
- create a reminder
- draft rather than send an email
- navigate a known interface
- record a repeatable flow
- guide the user through an unfamiliar screen
Authority can expand after the verification path proves reliable.
The current action and privacy model is at https://heypace.app/mac-ai-assistant-actions/.
Top comments (0)