Some of the software you work with has no API: an old admin panel, a desktop app, a dialog that only exists on screen. Atlarix 15.11 adds computer use, so the agent can look at your screen and use the mouse and keyboard when that is the only way in. It works with any model that can see images, not one vendor's, and it is off until you switch it on.
Which models can use it
Models that ship their own computer tool, like Claude, use their own. Any other model that accepts images gets Atlarix's computer tool, which takes a screenshot, clicks, types, scrolls and presses keys. Text-only models get neither, and the model menu says why instead of showing a switch. So the same feature works with your own key, a local vision model, Amazon Bedrock or a gateway; what matters is whether the model can see.
Off by default, switched on per model
You turn it on for a model, not for everything: in Settings → Computer use, in the chat's model menu, with /computer in the CLI, in the VS Code panel's settings, or in the session options of an ACP editor like Zed. There are no approval cards for each click or keystroke. Asking on every click would make it unusable, so the decision is made once, when you switch it on.
It is offered in Build and Debug modes only, and only one session controls the screen at a time.
Stopping it
Moving the mouse stops it, on every surface. The agent's next click or keystroke is refused and the turn ends, so nothing lands while your hand is on the mouse. Esc, and ⌃⌥Esc in the app, stop it at once.
While it runs, the desktop app shows a "Controlling your screen" pill with a Stop button. The pill is hidden from the model's own screenshots, so it never tries to click it. The CLI posts a system notification instead, and in an editor each action is a step in the chat that the editor's Stop button ends.
What it needs from your machine
It runs on macOS, Windows and Linux (X11 and Wayland; Wayland is not yet tested on real hardware). On macOS it needs Screen Recording and Accessibility once. macOS names the app that asks, which is Atlarix when you use the desktop app, and your terminal or editor when the agent runs there, so Atlarix tells you which name to expect before the prompt appears.
On Windows, typed text is pasted rather than sent key by key, and your clipboard is put back afterwards, kept out of clipboard history.
Using it well
It sees whatever is on your screen, so close what you don't want it to touch before you start. Give it a concrete task with a visible end ("add these three rows in the ledger app and save") rather than an open one. And prefer a real interface when there is one: a CLI command or an API call is faster and more reliable than clicking, and the agent already has those. Computer use is for the software that leaves no other way.
Computer use shipped in Atlarix 15.11 for the desktop app, the CLI, VS Code-family editors and ACP editors. The full list of changes is in the changelog.
FAQ
Can an AI coding agent use my mouse and keyboard with a model other than Claude?
Yes. In Atlarix, models with their own computer tool use it, and any other model that accepts images gets Atlarix's computer tool. Text-only models cannot use computer use.
Is computer use on by default in Atlarix?
No. It is off by default and switched on per model, in Settings, the chat's model menu, /computer in the CLI, or the editor's settings.
How do I stop the agent while it is controlling my screen?
Move the mouse. Its next action is refused and the turn ends. Esc, Ctrl+Option+Esc in the app, or the Stop button also stop it.
Originally published at atlarix.dev.
Top comments (0)