DEV Community

Savindu Thathsara
Savindu Thathsara

Posted on

It Typed the Address Itself

#ai #python #opensource #linux

It Typed the Address Itself

“Open the debian website” was the whole instruction. Four seconds later the ThinkPad was showing the Debian homepage, and nothing in the code knows what Debian is.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the same laptop.

One action, then look again

The earlier path had a skill per job. Play a video, open an app, search the web. That works until the next sentence is something nobody wrote a function for. A computer-use agent does the other thing: look at the front window, choose one key or click, do it, and look again.

Hearthlen keeps a 360M model loaded only for that task. The model may open an installed program, press a short list of keys, type, or click a control that is actually on screen. It cannot emit a shell command. If the focused window is a terminal, the typing step is refused.

On this run Chromium was already in front. The steps were Ctrl+L, then https://www.debian.org, then Return. The window title was “Debian -- The Universal Operating System”. End to end: 4613 ms.

Why it did not click the page

Linux already tells a screen reader the name and place of each button. That list is a better eye than a screenshot on a laptop with no discrete GPU, and it is what the current open Linux computer-use tools read first. FeatherPad’s File, Edit, and Search menus showed up as real controls. Chromium showed none. For a browser, the honest move is the address bar, not a guessed pixel.

The grammar is doing a lot of the supervising. A 360M model will wander if you let it type a paragraph into the address bar. The next step is a stronger local model in the same loop, and a timed “hey Hearthlen” from a real voice, which is still the open check in the roadmap. The password-free browser window from the previous note is the window this loop types into.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

Top comments (0)