Letting the Agent Touch the Browser Changed My Task List

For the first month I kept the agent inside the terminal. Files, git, commands - all fine. The browser was off limits because I assumed it would be flaky, and every GUI automation story I had heard ended in a pile of brittle coordinates.

Then a task forced the issue: a weekly report that only exists behind a login, with no export button. Either the agent learns the browser or I keep doing it by hand forever.

What was actually hard

Not clicking. Clicking was fine. The hard parts were, in order:

1. Knowing when the page was ready. Sleeps are a lie. I ended up waiting on a specific element that only exists after the data loads, and failing loudly after a timeout rather than continuing into an empty table.
2. Telling a real failure from a slow one. A page that did not load looks identical to a page that is still loading. The fix was the same element wait, plus a screenshot saved on every timeout, so the log tells me which it was.
3. Finding things that do not have stable coordinates. Layout changes, banners appear, a cookie notice shifts everything down. Reading the accessibility tree and clicking by role and name was more robust than pixels, and it also produces a log I can read.
4. Anything that needs me. Two factor prompts, captchas, "confirm your identity". These are not automation problems, they are stop conditions, and the agent should stop rather than guess.

What changed about the task list

The interesting effect was not that old tasks got faster. It was that new tasks became possible.

Things I had classified as "has to be done by hand" turned out to be "has no API". Reading a dashboard, pulling numbers from a supplier portal, checking whether a listing is live - none of these are hard, they were just behind a screen.

That is a bigger deal than shaving minutes off something. The list of what is automatable was wrong, and it was wrong because I had been treating the interface as a wall.

The rules I keep

• Never cache a screenshot between steps. Read the current state every time.
• If a click produces no visible change, treat it as an error, not as done.
• Stop and ask on anything that looks like a security prompt. Do not try to click through it.
• Save a screenshot on every failure, named after the step. Half my debugging is looking at those.

I am still cautious about this. A browser agent that half-works is more dangerous than one that refuses, because a half-working flow produces plausible wrong data. The element waits and the loud failures are what make it trustworthy enough to use.