[BA-003] Using CDP and Playwright for agentic browser automation

So far we have seen what BA-001 and how to automate with BA-002. Now we get to the interesting part: using these tools to build an AI agent that controls a browser.

The core idea is simple: the agent sees what is on the page, decides what to do, executes the action, and repeats. This is called the observation-action loop.

Here is a minimal example using Playwright and an LLM:

from playwright.sync_api import sync_playwright
from openai import OpenAI

client = OpenAI()

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto("https://example.com")

    for step in range(5):
        screenshot = page.screenshot()
        elements = page.locator("a, button, input").all()

        element_list = "\n".join(
            f"[{i}] <{e.tag_name}> {e.text_content() or e.get_attribute('placeholder') or ''}"
            for i, e in enumerate(elements[:20])
        )

        prompt = f"""You are looking at a web page. Here are the elements:
{element_list}

What is the next action? Respond with one of:
click [i]
type [i] "text"
goto url
done"""

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            temperature=0
        )

        action = response.choices[0].message.content
        print(f"Step {step}: {action}")

        if action.startswith("click"):
            idx = int(action.split()[1])
            elements[idx].click()
            page.wait_for_timeout(1000)
        elif action.startswith("type"):
            parts = action.split('"')
            idx = int(parts[0].split()[1])
            text = parts[1]
            elements[idx].fill(text)
            page.keyboard.press("Enter")
            page.wait_for_timeout(2000)
        elif action.startswith("goto"):
            page.goto(action.split()[1])
            page.wait_for_timeout(2000)
        elif action.startswith("done"):
            break

    browser.close()

This script is a proof of concept. The agent looks at the page, lists the interactive elements, sends them to an LLM, and executes the chosen action. In practice you would need better prompting, error handling and element visibility checks.

A more robust approach uses CDP directly to get the accessibility tree instead of parsing DOM elements. CDP exposes the full accessibility tree with roles, names and states:

import json
import requests
import websocket

resp = requests.get("http://localhost:9222/json/version")
ws_url = resp.json()["webSocketDebuggerUrl"]
ws = websocket.create_connection(ws_url)

cmd = {
    "id": 1,
    "method": "Target.createTarget",
    "params": {"url": "https://example.com"}
}
ws.send(json.dumps(cmd))
target_id = json.loads(ws.recv())["result"]["targetId"]

session_cmd = {
    "id": 2,
    "method": "Target.attachToTarget",
    "params": {"targetId": target_id, "flatten": True}
}
ws.send(json.dumps(session_cmd))
session_id = json.loads(ws.recv())["result"]["sessionId"]

get_tree = {
    "id": 3,
    "sessionId": session_id,
    "method": "Accessibility.getFullAXTree"
}
ws.send(json.dumps(get_tree))
tree = json.loads(ws.recv())["result"]

for node in tree["nodes"]:
    name = node.get("name", {}).get("value", "")
    role = node.get("role", {}).get("value", "")
    if role and name:
        print(f"{role}: {name}")

ws.close()

The accessibility tree gives you structured data about every element the browser knows about. This is the same tree that screen readers and tools like Playwright use internally.

For production agentic loops, tools like Playwright already expose everything you need: screenshot, click, type, extract text, evaluate JavaScript. The challenge is not the automation layer but the decision layer: the LLM has to understand the page context and choose the right action.

A common pattern is to give the agent a set of tools, each wrapping a Playwright action:

def tool_screenshot(page):
    return page.screenshot()

def tool_click(page, selector: str):
    page.click(selector)
    return "clicked"

def tool_type(page, selector: str, text: str):
    page.fill(selector, text)
    return "typed"

def tool_extract(page, selector: str):
    return page.locator(selector).all_text_contents()

Each tool has a name, description and parameters. The LLM receives the tool list and decides which one to call next. This is the same pattern used by browser-use libraries.

That is the foundation of agentic browser automation. The next posts in this series will dive deeper into each part.

That's all for now.
Thanks for reading!

Comments

[vic xie]

Nice write-up! For devs who deal with messy copied text, TextStow might help — it's a Mac menu bar tool combining clipboard history with prompt templates and text cleanup. Free: textstow.com