[BA-008] Running multiple browsers in parallel

When you need to scrape multiple pages or test across different scenarios, running one browser at a time is too slow. Playwright supports concurrent contexts and browsers natively.

The simplest way to run tasks in parallel is with Python threads:

from playwright.sync_api import sync_playwright
from concurrent.futures import ThreadPoolExecutor

def check_page(url: str) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url, wait_until="domcontentloaded")
        result = {
            "url": url,
            "title": page.title(),
            "status": page.evaluate("document.readyState")
        }
        browser.close()
        return result

urls = [
    "https://example.com",
    "https://example.org",
    "https://example.net",
]

with ThreadPoolExecutor(max_workers=3) as pool:
    results = list(pool.map(check_page, urls))

for r in results:
    print(f"{r['url']}: {r['title']}")

Playwright sync API is thread safe when you create a separate browser instance per thread. Each thread gets its own browser process.

For better performance you can reuse browser instances across tasks. This avoids the overhead of launching a browser for every task:

from playwright.sync_api import sync_playwright

class BrowserPool:
    def __init__(self, size=3):
        self.playwright = sync_playwright().start()
        self.browsers = [
            self.playwright.chromium.launch()
            for _ in range(size)
        ]
        self.index = 0

    def get_page(self):
        browser = self.browsers[self.index % len(self.browsers)]
        self.index += 1
        return browser.new_page()

    def close(self):
        for b in self.browsers:
            b.close()
        self.playwright.stop()

pool = BrowserPool(3)

pages = [pool.get_page() for _ in range(6)]
for page in pages:
    page.goto("https://example.com")
    print(page.title())

for page in pages:
    page.close()
pool.close()

This keeps 3 browser processes running and distributes pages across them.

A common production pattern is to use Playwright async API with asyncio:

import asyncio
from playwright.async_api import async_playwright

async def scrape(url: str):
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto(url)
        title = await page.title()
        await browser.close()
        return title

async def main():
    tasks = [
        scrape("https://example.com"),
        scrape("https://example.org"),
        scrape("https://example.net"),
    ]
    results = await asyncio.gather(*tasks)
    for url, title in zip(urls, results):
        print(f"{url}: {title}")

asyncio.run(main())

The async version uses less memory because it runs in a single event loop instead of one thread per task.

A few things to keep in mind when running parallel browsers:

• Each browser process uses about 200-400 MB of RAM
• Websites rate limit requests per IP, so parallel requests from the same IP may be blocked
• Always call browser.close() to free resources
• For large scale work, use a queue based system instead of unbounded parallelism

Parallel browsers turn a 60 second scraping job into 10 seconds.

That's all for now.
Thanks for reading!