Half the posts on my feed this week are some version of "what even is an AI agent." If-statements in a trench coat. Memory vs. RAG. Verification bottlenecks. All good debates — and all about the agent's brain.
Nobody's talking about its hands.
Here's the wall I kept hitting building an autonomous agent: the model can decide, perfectly, that it should reply to a lead, check the company mailbox, or post an update. Then it just... can't. It has no way to actually do it. An LLM emits text; it doesn't click "Post" on a logged-in account.
The usual answer is a scraper or a Playwright script with CSS selectors. Two problems:
-
Selectors shatter. The site ships a redesign,
.btn-primary-v2becomes.button__cta, and every script breaks silently. - They can't hold a real session. The interesting actions all require being logged in as someone, and a brittle script re-authing every run is a great way to get an account locked.
So I built the missing piece and open-sourced it: Ghost Browser — a real Chromium an agent operates the way a person does.
It sees the page, it doesn't parse it
The core trick is Set-of-Mark. Before any decision, every interactive element on screen gets a numbered box painted over it in the live DOM, the page is screenshotted, the boxes are stripped off again, and the agent gets the annotated picture plus a numbered list. Then it just says click 12.
That's the whole reason it survives redesigns where selectors don't: a numbered screenshot is redrawn from whatever is on screen right now, and a language model is far better at looking at a picture than at parsing a DOM tree. No selector to get subtly wrong.
You log in once. It drives forever.
You sign into a site by hand, once, in a console. That session lives in a profile — isolated cookies and storage — and from then on the agent operates that real, logged-in session. No credentials in scripts, no re-auth loop. One profile per account, each labelled with the site it's signed into.
The line that matters: it asks before it acts
Reading is free. Posting, messaging, following, joining are not — they happen under a real name and the notification has already reached a person. So those go through an act-gate: the agent shows you the exact text and waits. Approve, edit, or reject.
It's enforced twice — the prompt says so, and a guard inspects the label of anything it's about to click and turns a "Post" or "Join" into a proposal anyway. "Let it act without asking" exists, it's off by default, and it should stay off until you've read a few of its drafts.
Why it exists
This is the tool a fully autonomous agent on my own platform uses for anything that needs a real browser — a master agent plans the work, specialised organs handle research/outreach/publishing, and every time one needs to touch the real web as a real account, it calls Ghost Browser. Without it, the whole thing is a planner with no way to act.
A few things fall out of that design for free:
- A per-profile Tailscale exit — route a session out through a device you own at home, so sites see a residential IP instead of a datacentre.
- An SSRF guard — a browser anyone can point at any URL is a server-side-request-forgery engine; it resolves and re-checks every hop, including redirects.
- A workflow engine + route cards — compose deterministic flows, or record a site's own traffic once and replay it (no model calls on the second run).
It's MIT, and I need a designer
The engine is solid and production-tested. The console UI... was built by an engineer, not a designer, and it shows. If UI/UX is your thing, this is a project where your help lands immediately — there are issues tagged ui and good first issue waiting.
Repo, setup guide (any VPS in ten minutes), and runnable examples: https://github.com/Wvdstoep/ghost-browser
What's your take — is "hands" the missing layer in the agent stack, or am I solving a problem you'd solve a different way? Genuinely curious.
Top comments (2)
The act-gate is the part I'd defend hardest, and the double enforcement (prompt plus a guard that inspects the click label) is the right instinct -- a policy that only lives in the system prompt is a suggestion. What changed the maths for me on the sessions side is that a real logged-in profile is a long-lived credential: it needs the same custody discipline as an API key, which means the profiles have to survive browser restarts, and that is quietly the hard engineering part, not the clicking.
One cost I'd like your read on: Set-of-Mark buys redesign-resilience but charges a vision round-trip per decision. On a linear flow that's noise, but our multi-step flows are mostly deterministic once the page is known, so the interesting design seems to be caching the mark map per page state and only re-looking when the DOM signature moves. Are you re-screenshotting every single action, or is there a page-hash layer that skips the vision call when nothing changed?
Spot on about profiles being long-lived credentials, that's exactly the framing, and the custody is the quiet hard part. Each profile persists to its own volume with isolated cookies/storage, survives restarts, and never leaves the box; treating it like an API key is right, down to who's allowed to open a session against it.
On the vision cost, you're reading it the way I'd hope. Two layers today:
Within a walk it doesn't re-screenshot every action — the click path reuses the last mark map and only re-analyzes when the page actually moved, which right now is a URL-change + scroll-delta check, not a full DOM signature. Cheap, but coarser than what you're describing.
The real answer to "deterministic multi-step flows shouldn't pay the vision tax" is route cards: run a flow once under vision, it records the actual steps, distils them, and replays with no model or vision call at all. Set-of-Mark is the explorer; the route card is the compiled path. So the round-trip is a first-run cost, not a per-run one.
Where you're pushing is the gap between those two, an action-level page-hash cache that skips the re-look when the DOM signature hasn't moved, even mid-exploration. That's not in there yet, and it's a genuinely good idea; it'd tighten the window between "known page" and "recorded flow." If you want to take a swing at it I'd happily review a PR, the marking lives in src/inspector.js and the staleness check is in the click handler in src/server.js.
Really good comment, this is the exact axis the design lives on.