Introduction
AI agents that can browse the web autonomously are no longer science fiction. With the right tools, you can build an agent that navigates websites, extracts structured data, and makes decisions — all without human intervention. In this tutorial, we'll build a web-browsing AI agent from scratch using Python, a headless browser, and an LLM API.
The agent we'll build will be able to:
- Navigate to any URL
- Read the page content
- Extract specific data based on natural language instructions
- Follow links and paginate through results
- Save extracted data as structured JSON
What You'll Need
- Python 3.10+
- An LLM API key (OpenAI, Anthropic, or local Ollama)
- Playwright for browser automation
- Basic familiarity with Python and web concepts
Step 1: Setting Up the Environment
First, let's install the dependencies:
pip install playwright openai beautifulsoup4
playwright install chromium
We'll use Playwright for browser automation because it's more reliable than Selenium and handles modern JavaScript-heavy sites well. For the LLM, we'll use OpenAI's API, but the same pattern works with any LLM provider.
Step 2: The Browser Controller
The browser controller is the foundation — it gives the agent eyes and hands. It can navigate to URLs, read page content, and interact with elements.
import asyncio
from playwright.async_api import async_playwright
class BrowserController:
def __init__(self):
self.browser = None
self.page = None
async def start(self):
self.pw = await async_playwright().start()
self.browser = await self.pw.chromium.launch(headless=True)
self.page = await self.browser.new_page()
async def navigate(self, url: str) -> str:
await self.page.goto(url, wait_until="networkidle")
return await self.page.content()
async def get_text(self) -> str:
return await self.page.inner_text("body")
async def click(self, selector: str):
await self.page.click(selector)
await self.page.wait_for_load_state("networkidle")
async def get_links(self) -> list[dict]:
links = await self.page.evaluate("""() => {
return Array.from(document.querySelectorAll('a')).map(a => ({
text: a.textContent.trim(),
href: a.href
})).filter(l => l.href.startsWith('http'))
}""")
return links
async def screenshot(self, path: str):
await self.page.screenshot(path=path, full_page=True)
async def close(self):
if self.browser:
await self.browser.close()
await self.pw.stop()
The controller exposes a simple API: navigate, read text, click elements, get links, and take screenshots. The agent will use these primitives to explore the web.
Step 3: The Agent Brain
The brain is an LLM that receives the current page content and decides what to do next. We'll use a simple loop: read the page, ask the LLM what to do, execute the action, repeat.
import json
from openai import AsyncOpenAI
class AgentBrain:
def __init__(self, api_key: str, goal: str):
self.client = AsyncOpenAI(api_key=api_key)
self.goal = goal
self.history = []
self.system_prompt = f"""You are a web browsing agent. Your goal is: {goal}
You can perform these actions:
- navigate: go to a URL (args: url)
- click: click an element (args: selector)
- extract: extract data from the current page (args: what to extract)
- done: task is complete (args: final result)
Respond with JSON only: {{"action": "...", "args": "..."}}
Always respond with exactly one action.
"""
async def decide(self, page_content: str) -> dict:
content = page_content[:8000]
self.history.append({"role": "user", "content": f"Current page content:\n{content}"})
response = await self.client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "system", "content": self.system_prompt}] + self.history[-10:],
temperature=0.2,
)
decision = json.loads(response.choices[0].message.content)
self.history.append({"role": "assistant", "content": json.dumps(decision)})
return decision
The brain uses the last 10 messages as context — enough to remember what it's done without overflowing the context window. The temperature is kept low (0.2) because we want the agent to be predictable, not creative.
Step 4: The Agent Loop
Now we connect the brain to the browser. The agent loop is the heart of the system — it repeatedly observes, decides, and acts until the goal is met or it runs out of iterations.
class WebAgent:
def __init__(self, api_key: str, goal: str, max_steps: int = 20):
self.browser = BrowserController()
self.brain = AgentBrain(api_key, goal)
self.max_steps = max_steps
self.extracted_data = []
async def run(self, start_url: str):
await self.browser.start()
await self.browser.navigate(start_url)
try:
for i in range(self.max_steps):
page_text = await self.browser.get_text()
decision = await self.brain.decide(page_text)
action = decision["action"]
args = decision.get("args", "")
if action == "navigate":
await self.browser.navigate(args)
print(f"Step {i}: Navigated to {args}")
elif action == "click":
await self.browser.click(args)
print(f"Step {i}: Clicked {args}")
elif action == "extract":
page_text = await self.browser.get_text()
self.extracted_data.append({
"step": i,
"extracted": args,
"page_text": page_text[:2000]
})
print(f"Step {i}: Extracted data")
elif action == "done":
print(f"Step {i}: Done — {args}")
return args
else:
print(f"Step {i}: Unknown action {action}")
finally:
await self.browser.close()
return "Max steps reached"
Step 5: Running the Agent
Let's test our agent on a real task — extracting all article titles from a blog:
async def main():
agent = WebAgent(
api_key="sk-...",
goal="Extract all article titles and their URLs from this blog's homepage",
max_steps=15,
)
result = await agent.run("https://example-blog.com")
print(f"Result: {result}")
print(f"Extracted: {agent.extracted_data}")
asyncio.run(main())
Step 6: Making It Production-Ready
The basic agent works, but for production use, you need:
Error handling: Browser timeouts, network failures, and LLM API limits will all happen. Wrap every action in try/except blocks with retries.
Rate limiting: Don't hammer websites. Add delays between actions (at least 1-2 seconds) and respect robots.txt.
Stealth: Many sites block headless browsers. Use a stealth browser like Camoufox (a Firefox-based anti-detect browser) instead of stock Chromium.
Cost control: Each LLM call costs money. Cache page content, truncate aggressively, and use cheaper models for simple decisions.
Structured extraction: Instead of asking the LLM to extract data from raw HTML, use a combination of CSS selectors and LLM extraction. This is more reliable and cheaper.
Step 7: Advanced Patterns
Multi-page navigation: The agent can follow pagination links automatically. Add a "next_page" action that finds and clicks the "Next" button.
Form filling: Add a "fill_form" action that takes a dictionary of field selectors and values.
Parallel browsing: For bulk extraction, run multiple agents simultaneously using asyncio.gather(). Each agent gets its own browser instance.
Self-healing: If a selector fails, the agent can ask the LLM to find an alternative selector based on the page content.
Conclusion
Building a web-browsing AI agent is simpler than you might think. The core is just a loop: observe, decide, act. The complexity comes from making it reliable — handling errors, respecting rate limits, and extracting data accurately. But the basic architecture is straightforward, and you can have a working agent in under 100 lines of Python.
The future of web automation isn't Selenium scripts with hardcoded selectors that break every time a website changes. It's AI agents that can adapt to page changes, understand context, and extract data based on natural language instructions. The tools are available today — all you need to do is connect them.
Top comments (0)