Your AI agent can plan anything. It still can't click "Post".

Wesley van der Stoep — Sat, 12 Sep 2026 11:31:03 +0000

Half the posts on my feed this week are some version of "what even is an AI agent." If-statements in a trench coat. Memory vs. RAG. Verification bottlenecks. All good debates — and all about the agent's brain.

Nobody's talking about its hands.

Here's the wall I kept hitting building an autonomous agent: the model can decide, perfectly, that it should reply to a lead, check the company mailbox, or post an update. Then it just... can't. It has no way to actually do it. An LLM emits text; it doesn't click "Post" on a logged-in account.

The usual answer is a scraper or a Playwright script with CSS selectors. Two problems:

Selectors shatter. The site ships a redesign, .btn-primary-v2 becomes .button__cta, and every script breaks silently.
They can't hold a real session. The interesting actions all require being logged in as someone, and a brittle script re-authing every run is a great way to get an account locked.

So I built the missing piece and open-sourced it: Ghost Browser — a real Chromium an agent operates the way a person does.

It sees the page, it doesn't parse it

The core trick is Set-of-Mark. Before any decision, every interactive element on screen gets a numbered box painted over it in the live DOM, the page is screenshotted, the boxes are stripped off again, and the agent gets the annotated picture plus a numbered list. Then it just says click 12.

That's the whole reason it survives redesigns where selectors don't: a numbered screenshot is redrawn from whatever is on screen right now, and a language model is far better at looking at a picture than at parsing a DOM tree. No selector to get subtly wrong.

You log in once. It drives forever.

You sign into a site by hand, once, in a console. That session lives in a profile — isolated cookies and storage — and from then on the agent operates that real, logged-in session. No credentials in scripts, no re-auth loop. One profile per account, each labelled with the site it's signed into.

The line that matters: it asks before it acts

Reading is free. Posting, messaging, following, joining are not — they happen under a real name and the notification has already reached a person. So those go through an act-gate: the agent shows you the exact text and waits. Approve, edit, or reject.

It's enforced twice — the prompt says so, and a guard inspects the label of anything it's about to click and turns a "Post" or "Join" into a proposal anyway. "Let it act without asking" exists, it's off by default, and it should stay off until you've read a few of its drafts.

Why it exists

This is the tool a fully autonomous agent on my own platform uses for anything that needs a real browser — a master agent plans the work, specialised organs handle research/outreach/publishing, and every time one needs to touch the real web as a real account, it calls Ghost Browser. Without it, the whole thing is a planner with no way to act.

A few things fall out of that design for free:

A per-profile Tailscale exit — route a session out through a device you own at home, so sites see a residential IP instead of a datacentre.
An SSRF guard — a browser anyone can point at any URL is a server-side-request-forgery engine; it resolves and re-checks every hop, including redirects.
A workflow engine + route cards — compose deterministic flows, or record a site's own traffic once and replay it (no model calls on the second run).

It's MIT, and I need a designer

The engine is solid and production-tested. The console UI... was built by an engineer, not a designer, and it shows. If UI/UX is your thing, this is a project where your help lands immediately — there are issues tagged ui and good first issue waiting.

Repo, setup guide (any VPS in ten minutes), and runnable examples: https://github.com/Wvdstoep/ghost-browser