Browser Control Agent Prototype built with Playwright + LLMs

#ai #webdev #tutorial #playwright

Over the last couple of weekends I built the prototype to explore browser-control AI agents beyond the hype (because hype is really enormous).

I assume everyone here is using browser control feature in Codex or Claude code almost everyday, but for general public it's a hard sell. When people see how their browser "behaves on its own", they freak out, because it looks like a malware.

However, I see that many products are going to launch these features soon. One startup was even trying to hire me to build this end-to-end.

I began to think through this problem, and realized simple thing: most discussions assume that every interaction should be AI-driven through computer vision. And this is where something important is missing, both technologically and from the user's perspective.

So I decided to experiment with a hybrid architecture that combines deterministic web automation (in my case, Playwright) with LLM reasoning.

The core idea is a scenario router. Before any browser interaction happens, it decides which execution strategy best fits the user's intent:
– deterministic browser automation,
– AI-driven browser control,
– or even direct API/MCP integration,
– or any other execution mechanism you want.

In my opinion, choosing the right execution strategy is a much more interesting product and engineering problem than simply giving an LLM complete control over the browser.

And since I'm not interested in building and maintaining this type of products myself, I decided to open up the prototype repo, write an article and make a video about it.

I'd love to hear your thoughts on the architecture, recovery strategy, and the overall system design.

All the links are below.

Cheers,
V