Most browser automation hands the model the page and hopes. I run it the other way round, and it holds up better in production.
The model never touches the page. It receives a text description of the state and returns a judgement. Three answer types only: a probability, a choice from a list I supply, or a score between 0 and 1. Separate code performs the action.
Why that beats an open-ended agent loop:
Every decision is loggable, because the answer is one of a closed set rather than a paragraph of reasoning.
The action layer is testable on its own. Whether a click landed is a question about the harness, and you can test that with no model in the loop at all.
The blast radius is small. A model that can only choose from a list cannot invent a step.
The failure mode is the part worth knowing. A decision model is exactly as good as the options you hand it. I once asked one to find the reply control on a webmail page. The state described the page accurately, the model answered confidently, and the answer was useless, because the control had never been in the option set.
The fix is to stop asking open questions and start enumerating. Walk the page, tag every clickable element, hand over the numbered list. On that page there were 33 candidates. Given the list, the same model picked the right one at a stated confidence of 1.00.
A second use worth more than I expected: turning a repeated judgement into a rubric. Any call a person makes over and over can usually be broken into closed questions, and then it stops being an opinion and becomes a measurement.
The honest limitation. The model has no memory between calls, so everything it needs must be in the state you send. It also cannot tell you that your option set is incomplete. That failure is silent, and it looks like a model problem when it is a design problem.
I build these systems for a living, so I have an interest in the approach. The reason I settled on it is narrow. It is the only arrangement where I can read the logs afterwards and say what was decided, by what, from which options, and whether it worked.
Originally published at *aaronstalberg.com*. If this one was useful, the rest of the series is here:
Top comments (0)