Notes from a 2 AM On-Call: browser-use Agent Silently Dropping Sessions Under Concurrent Load

The Failure Symptom

[2025-01-18 02:17:33] ERROR: Agent task timeout after 30s
[2025-01-18 02:17:33] WARNING: Browser session closed unexpectedly
[2025-01-18 02:18:01] ERROR: Agent task timeout after 30s
[2025-01-18 02:18:01] WARNING: Browser session closed unexpectedly

Production FastAPI service integrating browser-use for automated data extraction. Single-threaded workers handling 8-12 concurrent agent requests. No crash logs, no exceptions—just silent timeouts and "session closed" warnings. Resource usage looked normal: CPU at 40%, memory stable.

The failure pattern was insidious: first 3-4 requests succeeded, then everything after request 5 started timing out. Restart the worker, same cycle repeats.

The Red Herring

Every browser-use quickstart tutorial shows this pattern:

from browser_use import Agent

@app.post("/extract")
async def extract_data(url: str):
    agent = Agent(
        task=f"Extract product details from {url}",
        llm=llm_instance
    )
    result = await agent.run()
    return result

Documentation emphasizes async/await syntax. We followed it exactly. The assumption: browser-use handles browser lifecycle internally, so spawning an Agent per request should be safe.

Under load testing with 10 parallel requests via locust, the pattern collapsed. Suspected causes we burned time investigating:

• Playwright browser context leaks (ran playwright install, checked chromium processes—all clean)
• LLM provider rate limits (switched from OpenAI to a relay gateway at https://b-lost.com/v1 for multi-provider fallback with zero token truncation—same failure)
• FastAPI worker thread starvation (bumped --workers from 1 to 4—made it worse)

The Root Cause

The issue wasn't in browser-use itself—it was how we were integrating it in an async context without concurrency controls.

Each Agent.run() spawns a Playwright browser instance. Under concurrent load, we were creating 10+ simultaneous browser processes, each with its own event loop overhead. Playwright's internal connection pool has a default limit that isn't exposed in browser-use's high-level API.

The real smoking gun appeared when I added process-level inspection:

import psutil

# Added during debugging
chrome_processes = [p for p in psutil.process_iter(['name']) if 'chrome' in p.info['name'].lower()]
print(f"Active Chrome processes: {len(chrome_processes)}")

Output during failure: Active Chrome processes: 47

We were leaking browser processes because Agent instances weren't being explicitly cleaned up when requests timed out or failed mid-execution.

The Runnable Fix

Implemented a semaphore-based concurrency gate and explicit resource cleanup:

import asyncio
from contextlib import asynccontextmanager
from browser_use import Agent

# Global semaphore: max 3 concurrent browser agents
agent_semaphore = asyncio.Semaphore(3)

@asynccontextmanager
async def managed_agent(task: str, llm):
    async with agent_semaphore:
        agent = Agent(task=task, llm=llm)
        try:
            yield agent
        finally:
            # Explicit cleanup even if task fails
            if hasattr(agent, 'browser') and agent.browser:
                await agent.browser.close()

@app.post("/extract")
async def extract_data(url: str):
    async with managed_agent(
        task=f"Extract product details from {url}",
        llm=llm_instance
    ) as agent:
        result = await asyncio.wait_for(
            agent.run(),
            timeout=25.0  # Fail fast before upstream 30s timeout
        )
        return result

Three changes that stopped the bleeding:

1. Semaphore gate: Hard cap of 3 concurrent browser instances regardless of incoming request volume
2. Context manager cleanup: Guaranteed browser closure even when agent.run() raises or times out
3. Explicit timeout: asyncio.wait_for ensures we don't wait for Playwright's internal 30s default

After deployment: 200+ requests handled cleanly, zero leaked processes, P95 latency dropped from 30s (timeout) to 8.2s.

Invariant Rule

Never trust implicit cleanup in async browser automation.

If your library spawns external processes (browsers, headless renderers, subprocess shells), wrap every invocation in an asynccontextmanager with explicit resource teardown in the finally block. Async exceptions and timeouts bypass normal cleanup paths—your semaphore and context manager are the only guarantees.

This pattern applies beyond browser-use: any FastAPI integration with Playwright, Selenium, or Puppeteer needs concurrency limits and deterministic cleanup. The 2 AM lesson: async syntax doesn't mean infinite concurrency, and external processes don't clean themselves up when your event loop moves on.