Read freely, confirm before writing: a safety model for computer-use agents

Every few weeks I hit the same wall. A tool I need to pull data from has no usable API. Seller centers, supplier portals, 3PL dashboards. The data is right there on the screen, but the only way to get it is to log in and click through the UI like a human. So I built Webhands: a computer-use agent that operates those dashboards in a real headless browser, returns clean structured data, and refuses any write action unless I explicitly confirm it.

The problem with agents that click

The moment you hand an agent a real browser session, you have given it the power to do everything the logged-in human can do. That includes the dangerous stuff. Issuing a refund. Confirming a shipment. Canceling an order. Reading a page is safe. Clicking "Issue refund" is not, and the difference between the two is one button.

Most automation frameworks treat every action the same way. A click is a click. That is exactly the design choice I did not want, because a scraping run that quietly mutates production state is not a convenience, it is an incident waiting to happen. I wanted reads to be safe by default and writes to be deliberate, with no way to trip into a mutation by accident.

The core idea: recipes and write-gating

You drive Webhands by POSTing a recipe. A recipe is just an entry URL, an optional list of login and navigation steps, and an extraction spec. Here is the shape of a step in the source:

export type Step =
  | { action: "goto"; url: string }
  | { action: "type"; selector: string; text: string; secret?: boolean }
  | { action: "click"; selector: string; write?: boolean }
  | { action: "waitFor"; selector: string; timeoutMs?: number };

Notice the write?: boolean on click. That flag is the whole safety model. Typing, waiting, navigating, and reading are inherently safe. The only action that can change state is a click, so the only action that can carry write: true is a click. If a step is marked as a write, the request is refused unless it also carries confirm: true.

The gate itself is tiny and lives before any browser even launches:

if (hasWriteStep(recipe) && !confirm) {
  return {
    ok: false,
    mode: env.BROWSER ? "live" : "dry",
    steps: [],
    error: "recipe contains a write step; resend with confirm:true to execute",
  };
}

hasWriteStep returns true if any step is a click marked write: true. When that is the case and confirm is absent, the function returns an error and never opens the browser. To actually run the write, you resend the exact same recipe with "confirm": true. Reads never need confirmation because they can never mutate anything.

I like that this check runs before the browser is provisioned. There is no window where a write step has partially executed and then gets caught. The refusal happens up front, deterministically, based purely on the shape of the recipe.

What comes back

On a successful run you get three things. First, data: structured JSON. You can extract it two ways. Give the recipe a list of fields with CSS selectors and it scrapes them directly with no model involved. Or give it a natural language extract.prompt and it hands the page text to Claude and asks for JSON matching your request. Second, screenshotBase64: a base64 PNG of exactly what the browser saw, so you have proof. Third, steps: the log of actions it actually took, like goto ..., type into #email (secret), click #signin.

The extraction path is honest about degrading. If no Anthropic key is set, the prompt-based extractor returns the raw text slice instead of failing, so the pipeline still runs in development. When the key is present it calls the Messages API with a Claude Haiku model, asks for JSON only, strips any code fences, and parses. If parsing fails it returns the unparsed text rather than throwing away the result.

Two modes, so you can build without paying

Webhands runs on Cloudflare Workers with Browser Rendering. There are two modes. In live mode the BROWSER binding is present and it drives a real headless browser via Cloudflare's Puppeteer. In dry mode the binding is absent and it returns the plan it would have run, labelled as a dry result, instead of executing anything. That means you can author and test recipes without a paid binding, then flip to live when you are ready. Even when the binding exists but Browser Rendering is over quota or unprovisioned, the launch is wrapped so it degrades into a clean error instead of crashing.

An honest limitation

The write-gate protects you from executing a mutation you did not confirm. It does not understand what a given click means. The write: true flag is set by whoever authors the recipe. If you mark a genuinely destructive button as a read, or forget to mark it at all, Webhands will happily click it without asking, because from its point of view a click without write: true is just navigation. The safety model is only as good as the labelling. It is a forcing function for deliberate writes, not a classifier that detects danger on its own. I chose that tradeoff deliberately, since guessing intent from a button label is far less reliable than an explicit flag, but it does mean the human writing the recipe still owns the judgment call.

The other honest caveat is that this operates real UIs. Selectors break when dashboards change, logins get challenged, and a page that renders slowly can time out. Screenshot proof and the step log exist precisely because runs against real portals are messier than runs against an API contract.

Closing

"There's no API" becomes "there's an agent." Webhands is my attempt to make that swap safe enough to actually use against production dashboards, by making reads free and writes something you have to ask for twice. The recipe format is small on purpose, the gate is a few lines, and both modes let you iterate cheaply before anything touches a live account.

Code is here: https://github.com/AgentPostmortem/Webhands