DEV Community

Cover image for Make pages browser agents can click through
Conor Bronsdon
Conor Bronsdon

Posted on Originally published at chainofthought.show

Make pages browser agents can click through

Most websites will never expose APIs for AI agents. Dhruv Batra, co-founder and chief scientist of Yutori, argues that infrastructure changes slowly. The head of the distribution, the top 50, 100, 500 sites holding enterprise and customer data, has the incentive and maturity to build agent pathways or charge tolls for access. The long tail does not. He points to roughly 20,000 to 40,000 school district sites in the US, plus government offices and hundreds of thousands of e-commerce pages, each built for people.

The Golden Gate Park tennis court reservation system website was written by some indie developer. That website has a perfectly functioning calendar view reservation system. It's not going to reinvent itself overnight.

Dhruv Batra, co-founder and chief scientist of Yutori, on Chain of Thought episode 63

If you run a site in that long tail, I would make the page workable for a browser agent that acts like a person. Batra's model is pixels in, clicks out. If a machine can perceive the screen and click the buttons, that capability is the API. Some of the holdout, in his words, is socio-political, and pointing coding agents at these sites does not solve that part. Rollout across distributed human organizations takes time. He reaches for autonomous driving: dedicated sensor lanes would make self-driving easy, and that infrastructure is too expensive, so people and autonomous cars share the roads. Thirty years of browsers and websites made rich visual information efficient for the human brain. That shared roadway is what agents will use for a long time.

Why would I skip a separate agent API?

Tool use, also called function calling, is how a model outputs a structured request, such as a function name plus arguments, and your code runs it and returns the result. On its own, a language model can only produce text. Tool use is how it acts. Open standards such as the Model Context Protocol exist so the same tools can work across models and apps.

That design assumes the site exposes a structured endpoint. Batra's bet for the long tail is different. The human-visible screen is the interface, and the site does not have to be rebuilt first. He says the DOM and the accessibility tree are inconsistent, because there is no central owner of the web and no enforced standard. People propose standards, then another standard, and buttons are labeled one way on one site and another way on the next. The source of truth is what a person sees.

If a machine can perceive pixels and click buttons and do the things and report to you that yes, I'm able to do it. There, that's your API.

Dhruv Batra, co-founder and chief scientist of Yutori, on Chain of Thought episode 63

What should the agent be able to see and click?

If you can look at a screenshot and know which button to click, that is the operation a browser agent has to perform. I would make that test easy.

Use a real button, link, or form control, and put the action in visible text. Keep the words stable. "Reserve court" should stay "Reserve court" on every visit. Keep the label in the control. A placeholder that disappears when someone types is a weak label.

Put the result on the screen. Batra's shipping question is the flow I would design against. He asks what it costs to ship a product to a house, which is separate from the listed price. On many shops that figure shows up only after you add the item to the cart and enter a zip code. There is no API for the question. The agent has to do what a person does, then read the page. If the answer never appears in the document, a pixel agent cannot report that it finished.

Yutori's Navigator N1.5 can write JavaScript on the fly and fill several fields in one action. The pixels stay the source of truth. Stable ids, names, and label text give that action something steady to aim at. The page contract below is my sketch.

// Sketch only. Not a guest's code and not a vendor SDK.
const reserveCourt = {
  fields: [
    { id: "court", label: "Court", control: "select" },
    { id: "day", label: "Date", control: "date" },
  ],
  submitLabel: "Reserve court",
  // Show this string in the document after submit.
  resultText: "Reservation confirmed",
};
Enter fullscreen mode Exit fullscreen mode

Match those strings in the HTML a person sees. After submit, render resultText in the page. A success code that never becomes text on screen is invisible to this kind of agent.

How do I keep the click path short?

Batra argues that a browser agent runs a perceive-act loop over long trajectories. Each step has to be cheap and fast, or the task stops making economic sense. Yutori's Navigator is post-trained in-house for browser and computer use. On the episode he said it was slightly more accurate than Opus 4.7 and GPT-5.5 on standard browser-use benchmarks, and he declined to make much of that gap. He emphasized efficiency: 2 to 3 times faster and 4 to 5 times cheaper, because the model is small and light. N1.5 writes JavaScript so one compound action can replace a run of clicks. A pixel-only model can sit on a date picker clicking next 12 times to reach a date a year out. The code path can set the date in one step. What makes these tasks hard, in his account, is the long horizon, and a compound action cuts the trajectory.

You do not pick the model that visits you. You pick how many clicks you require. I would put one primary action on each view, ask for fields in a fixed order, and render the outcome in the document when the step fails as well as when it works. If a form can collect the same facts, skip a hover menu or a slider. On a site you own, treat a label change like a broken contract with every return visitor who only has the screen.

When should I declare a browser tool?

Ship the click path first. A declared tool is an experiment on top of it. The WebMCP lab compares buttons with structured WebMCP browser tools in a fictional agent code-review game. I would copy that scope: one task, two interfaces, and the buttons still complete the task.

Chrome's WebMCP early preview is the lighter middle path in this story. A site can describe its forms and actions to agents without building a separate API. Batra's argument applies there too. It still asks each site to do the work, and most sites will not. Declare only actions you will keep aligned with the visible controls: the same field names, the same button text, the same result string. If the tool is absent, the page should still work. Where an API already exists, use it. Batra's product uses APIs when they are there, and browser agents for the long tail. A site you control can offer the screen for certain, and a description of that screen only where you will maintain it.

Checklist

  • Assume the long tail, including sites you depend on, will not ship an agent API.
  • Put each important action in visible text on a real control, and keep that text stable.
  • Render each step's result in the page, in words a screenshot can show.
  • Shorten the trajectory: one form and one primary button, when the task allows it.
  • If you declare a browser tool, keep the human click path working on its own.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

Top comments (0)