Local AI computer use: a model that clicks for you, and the questions it asks first

Locally Uncensored 3.0.6 added computer control: a model uses the mouse and keyboard in apps you approve. It works with local models, so the screenshots and the clicks stay on your machine. It is a beta, and most of the work in it went into the questions it asks before it does anything, so most of this post is about those.

Locally Uncensored is an open source desktop app (AGPL-3.0) for running AI models locally, with 170,000+ downloads. Computer control is one part of its agent.

Off until you switch it on

It stays off until you switch it on under Settings, Agent, Agent Permissions, Computer control. Switching it on is not a blanket yes. After that, LU still asks at these points:

Every new chat        "Allow computer control in this chat?" Allow / Deny
First touch of an app asks before it uses that app
Every action          asked by default (supervised mode)
Sensitive actions     always asked, also in autonomous mode

Until you click Allow on that first card, LU takes no screenshot and does nothing. The yes holds for that one chat. A new chat asks again, and so does every chat after a restart of the app. No setting skips the card.

Sensitive means anything that sends, buys, deletes or changes permissions. Those are asked with the reason in red. So is a click on a button, link or menu item that has no name, because LU cannot tell what it would do.

What it never touches

Some places are off limits at every level:

terminals
password managers
system sign-in and permission dialogs
security settings
the Run dialog
windows with administrator rights

LU never types into a password field, and pasting there is refused too. A double click on a program, an installer, a script or a shortcut asks first.

How you stop it

Ctrl+Alt+Space    pause
Ctrl+Alt+Esc      stop

On a Mac it is Control+Option instead of Ctrl+Alt. In an autonomous session LU also pauses when you move the mouse. The paused view shows what it did so far and asks whether you take over or let it continue, and you can add an instruction for the rest of the task. A screen task ends after at most 25 steps or 8 minutes either way, and the time you take to answer a question counts.

While it works on Windows, the LU window shrinks to a small window in the top right corner and the edge of the screen glows softly in purple. The small window shows what LU is doing and names the two shortcuts. Questions appear there with a Stop button and take mouse clicks only, so the app LU works in keeps the keyboard focus.

Which model looks at the screen

By default the chat model you already picked. LU does not ask how it should run.

On Windows, LU reads the list of buttons and fields of the window it works in. A model that can see images gets the screenshot plus that list. A model that cannot see images works from the list alone, without a picture. That second case is what makes small local text models usable here at all.

We compared the two on two tasks, three runs each. Both finished all six runs. The runs with the picture took 497 seconds in total, the runs with the list alone 381.

For a dedicated screen model, the catalog has Holo 3.1 9B, marked Beta and recommended from 16 GB of video memory. In our test on Windows it finished a simple typing task in 1 of 3 runs, and it does not fit next to a 9B chat model on a 12 GB card. That is not a strong result, and I would rather say so than leave it out.

A real number

Typing a note in Notepad, asked by voice, with a local 9B model working from the list of buttons and fields: 11 to 22 seconds from the end of the spoken command to the spoken result. That was measured before the question in every new chat existed, so in a new chat your click on Allow comes on top.

Platforms

Windows   checked point by point in the real app, on one Windows 10 machine
Linux     not run in the app yet, needs an X11 session
macOS     in the app, not fully tested yet

The list of buttons and fields and the autonomous mode exist on Windows only. On Linux and macOS, LU cannot name what a click would press, so every click there is asked as a sensitive action. On a Mac it also needs two system permissions first, Accessibility and Screen Recording, and the model needs image input because it only has screenshots.

There is no background mode on any platform. It works in the foreground with your real pointer.

The limits I would want to know

• Small local models misjudge when a task is finished. A task can end as stopped although the work is done.
• Text on the screen can try to give the model instructions. LU marks what it reads from the screen as untrusted and asks before sensitive actions. For anything that matters, stay in the supervised mode.
• The words LU reads on a button to tell a sensitive action from an ordinary one come from a list. A wording the list does not know is not recognised.
• Eight corrections and ten security fixes made after the Windows check have run in automated tests only.

Short version

Computer control in Locally Uncensored 3.0.6 lets a local model click and type in apps you approve. It is off by default, asks in every new chat, asks before sensitive actions in every mode, never touches terminals or password fields, and stops on Ctrl+Alt+Esc. It is checked on Windows, not fully tested on a Mac, and not run on Linux yet. Treat it as a beta that wants you watching.

The code is at github.com/PurpleDoubleD/locally-uncensored, COMPUTER-CONTROL.md describes every question and every limit, and the changelog has the full list of what is and is not tested.