- Book: Ship It — the pragmatic startup tech stack
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
An agent builds the signup screen, opens the iOS Simulator, taps through the flow, looks at what rendered and fixes the code. Cursor announced that on October 23: its agents can now drive desktop apps like the iOS Simulator to test the features they build (Cursor on X).
The feature is called Computer Use. It gives the agent inside the Cursor app the ability to "click, type, and capture screens across GUI apps on a developer's machine" (AlphaSignal). Until now, that kind of control lived in Cursor's cloud agents and self-hosted workers. Now it reaches the machine you work on, with your mail, your password manager and your signed-in browser on the same screen.
What shipped on October 23
You call it by typing @ in an Agent conversation and picking Computer Use from the mention menu, according to Flavio Copes' walkthrough. The agent can also choose it on its own "when a task requires it" (Runtime Wire). It arrives as an update to the desktop app, and AlphaSignal notes that Cursor "announced no accompanying pricing change" (AlphaSignal).
Developers had been asking for this. A forum request from July 17 asked for local desktop control with opt-in permission and an audit of actions. As late as October 6, a Cursor staff reply in that thread said: "This is still something we're exploring, but I can't share a timeline yet."
The cloud version already existed. Cursor's docs say "each cloud agent runs in its own isolated VM with a full desktop environment," and that agents "use a mouse and keyboard to control the desktop and browser" (Cursor docs). Self-hosted workers on your own Macs or Linux boxes got an opt-in --computer-use flag (Cursor docs). The October 23 release brings the same idea to the agent in the IDE.
Your desktop is not a throwaway VM
A cloud agent clicks around a VM that exists for one task. The local agent clicks around your session. AlphaSignal puts the scope plainly: "local" here "identifies the controlled machine and does not establish on-device model execution" (AlphaSignal). Screenshots are how it sees the app, so plan as if whatever sits on screen during the task is model input.
The written record is thin for the IDE path. When this post was written, the newest entry in Cursor's changelog was dated October 16, and the documented setup steps cover self-hosted workers. An ExplainX write-up reported that no permission or approval behaviour for the new tool was documented at publication, and that supported operating systems weren't stated, so most of the detail below comes from the self-hosted docs.
The local agent clicks in the same session as everything else you have open.
The two macOS permissions go to one helper app
On macOS, Computer Use "needs Accessibility to interact with apps and Screen Recording to capture their interfaces" (Flavio Copes). Both go to a helper app named Cursor Computer Use, not to Cursor itself. As that guide puts it: "Granting a permission to your terminal doesn't grant it to a different app."
Cursor's self-hosted docs spell out the same helper in detail (Cursor docs):
- It installs to
~/.cursor/cursor-computer-use/Cursor Computer Use.app, bundle IDco.anysphere.cursor-computer-use. - Grant both permissions to that helper. "Granting Accessibility or Screen Recording to Terminal, iTerm,
agent, Cursor, or Cursor Agent Helper does nothing for computer use." - Screen Recording approvals belong to the macOS user who granted them.
- On macOS 15 and later, the system may ask that user to re-confirm Screen Recording now and then.
- Grants follow the app's code-signing identity, so don't copy, re-sign or rebuild the helper.
If you run a self-hosted worker, these are the documented commands. The last one prints the helper's code requirement, which you need for an MDM profile:
agent update
agent worker --computer-use start
agent worker debug
codesign -d -r - \
~/.cursor/cursor-computer-use/Cursor\ Computer\ Use.app
One catch from the docs: agent worker debug confirms the helper is installed, but it "doesn't report permission state." The documented check is a real task that opens a browser and takes a screenshot. If the screenshot shows up in chat, the grants work.
For teams: a profile can't grant Screen Recording
If you manage Macs with MDM, Cursor doesn't ship a .mobileconfig, so you build your own (Cursor docs). Accessibility can be silently allowed on most macOS versions, but the docs warn that Apple deprecates silent allow starting with macOS 26.2. Screen Recording can't be allowed by a profile at all. The documented workaround is AllowStandardUserToSetSystemService, which lets a standard user approve it without an admin password.
Linux workers are a different setup. They drive an X11 display and need xdotool, ffmpeg and the X11 utilities installed by you. Desktop sharing, where a teammate watches or takes over the agent's desktop, is Linux-only, runs on an isolated desktop the worker creates, and keeps clipboard transfer blocked (Cursor docs).
Accessibility and Screen Recording go to one helper, not to your terminal or the editor.
Before you type @Computer Use
None of this needs to stop you from trying it. A few habits keep the blast radius small:
- Use a separate macOS user for agent work. Cursor's docs tie Screen Recording grants to the user who approves them, and a worker uses the logged-in GUI session of the user running it. A second account keeps your personal session out of the screenshots.
- Close what you wouldn't paste into a prompt. Mail, chat, a password manager, a banking tab. If it's on screen, assume the model can see it.
- Write a narrow task. "Launch the iOS Simulator, open the app, go through signup, and screenshot each screen" is easier to check than "test the app."
- Turn it off between sessions. Untick Cursor Computer Use under System Settings → Privacy & Security when you're done, and grant it again next time.
- Review the diff as usual. The agent clicking through a flow is a test run, not proof the code is right.
I haven't run Computer Use myself. These habits come from the docs, so adjust them once you have.
What's still open
Flavio Copes' guide and Cursor's self-hosted docs answer the macOS plumbing. They don't yet answer the questions a security review will ask about the IDE path: does the agent confirm before it clicks in a new app, can you limit it to a list of apps, are screenshots kept, and does it run on Windows. When Cursor documents those, the checklist above may shrink.
Would you let a coding agent click through your own desktop session, or only a spare user or VM? Tell me in the comments.
If this was useful
An agent that taps through your app before you merge is one more step in the path from localhost to production. Ship It covers the rest of that path for a small team: hosting, deploys, auth, payments, CI/CD and monitoring, so the feature your agent tested actually reaches users.
If you run Claude Code, Codex and other agents side by side, I build Hermes IDE for that: source-available, on macOS, Windows and Linux, and each task can get its own branch and worktree (GitHub).
Top comments (2)
From the Auten team (we build a computer-use MCP, so this is close to home): a spare macOS user for anything beyond a throwaway test, same as your habit 1.
One practical question on that setup: with the macOS 15 Screen Recording re-confirmation, does someone have to log into the agent user now and then to approve it, or does it hold for a worker that runs unattended?
A habit that pairs well with your "close what you wouldn't paste" point: keep passwords out of the model entirely, typed locally by the tool, so the model only sees that the field was filled.
Helper-only access is the right cut: the allowlist should name the app and the action, not the whole desktop session, so a stray click sits outside the grant.