Giving the agent a real browser, indexed by element — AIClaw 3.5 browser tools

AIClaw is a local-first desktop agent. Until 3.5 there was one capability it plainly did not have: letting the model operate a web page the way browser-use does — not by screen coordinates, but by an indexed list of the page's interactive elements.

3.5 ships that, on top of the approval, sandbox and session model the app already had.

The idea: DOM indexing instead of pixels

Screen coordinates are brittle. A cookie banner shifts the layout by 40px and every click lands somewhere else. browser-use solved this by handing the model a numbered list of interactive elements and letting it act on numbers.

AIClaw does the same. browser_snapshot returns a numbered list where each entry carries:

• the element's role
• its visible text
• its current value (for inputs)
• the link target, if any
• whether it is inside the viewport

The list is capped at 150 entries with the total stated, so the model knows to scroll rather than assume the page ended. browser_screenshot exists for the cases where layout or a captcha actually matters, but it is not the default path.

Ten tools, forwarded like computer use

When the setting is on, the kernel registers ten tools: navigate, snapshot, click, type, select, scroll, back, key, extract, screenshot. Each one is forwarded to the desktop host as a browser/request — the same path computer use already took, so nothing about the tool loop or approval flow is special-cased.

They also fold into exec under code mode, which is where they get interesting: paging through ten pages of results is a loop, and the intermediate pages never enter the context window.

const out = [];
for (let i = 0; i < 5; i++) {
  const list = await tools.browser_extract();
  out.push(list.slice(0, 200));
  await tools.browser_click({ index: 42 }); // "Next"
}
return out.join("\n---\n");

The browser belongs to the host

Electron already ships Chromium, so AIClaw does not need Playwright or a downloaded browser binary. The agent gets a separate BrowserWindow on a persistent partition:

• Logins survive. The session lives in persist:agent-browser, so signing in once is enough for later sessions.
• The window is visible. You can watch what the model is doing and take over mid-task.
• The page is isolated. It can reach neither the preload script nor Node.

Text typed by the model goes through the real input path, so React and Vue controlled inputs actually see it.

Page content is untrusted input

Everything scraped from a page — the numbered list and the extracted body text — is wrapped in the same boundary markers AIClaw uses for external feeds, and the tool descriptions say so again. That does not stop a page that deliberately disguises a button as something else (browser-use cannot either). What it does stop is a sentence on a web page being read as an instruction.

Approval stays where it was

• Opening a URL asks, every time. It leaves the machine and the model picked the destination.
• In-page actions are writes: the default profile lets them through, the strict profile asks about each one.
• Snapshot, extract and screenshot never ask.
• Channel sessions (unattended) cannot reach the browser at all.
• Only http, https and data: URLs load.

The reasoning for asking only once per navigation: a click might mean "next page" or "confirm order", and there is no way to tell from the element. Asking on every click trains users to approve blindly.

Known gaps, in value order

1. File upload — the host needs to take over the file-chooser event and feed it a path inside the workspace.
2. Sensitive-data placeholders — the model sees $password, the host substitutes the real value so it never enters the transcript.
3. Multiple tabs — a tab list, switch and close.
4. Domain allowlist — configured domains skip the navigation prompt.
5. Headless mode — for scenarios and CI.

How it was verified

• Go tests in internal/agent/browser_test.go: the tools register only when enabled, navigation requires approval, in-page clicks do not, the strict profile asks step by step, and a screenshot arrives as an image in the next request.
• A node test for the snapshot text: numbered-list shape, untrusted boundaries, truncation notice, and that page scripts have no network capability.
• Desktop smoke test that really opens a window and runs navigate → snapshot → type → click → extract against an embedded page, plus a refusal of a file: URL.

The item-by-item comparison with browser-use — including the gaps above — is in the repo at docs/design/browser.md.

AIClaw is open source: https://github.com/chowyu12/aiclaw

Comments

[Reid Marlow]

Passing the element index in a tight code loop is clean, but the nasty failure mode with DOM indexing is index invalidation on dynamic pages. An autosuggest dropdown or a skeleton loader inserting elements between snapshot and click can silently remap index 42 to another target. Pairing the numeric index with a lightweight tag or role checksum on invocation catches the drift before the agent fires a click into the wrong node.