Scrape JavaScript-Heavy Sites with Playwright in Python (Developer Guide)

You fire off a requests.get(), parse the response with BeautifulSoup, and find an empty <div id="root"></div>. The data you can see in your browser never made it into the HTML. This is the everyday reality of scraping modern sites built with React, Vue, Angular, or any framework that renders content in the browser.

Playwright solves this by driving a real browser engine. It runs the page's JavaScript, waits for the content to appear, and lets you extract it just as a user would see it. This guide walks through the full workflow in Python: setup, waiting strategies, infinite scroll, intercepting API calls, performance tuning, and making your scraper reliable enough to run unattended.

Why requests and BeautifulSoup fail on dynamic pages

A traditional HTTP client downloads the initial HTML document and stops. On a client-side rendered site, that document is mostly a shell. The real content arrives later through JavaScript that fetches JSON from an API, builds DOM nodes, and injects them into the page.

You can confirm this quickly. Open DevTools, choose "View Source" (not "Inspect"), and search for a piece of text you can see on screen. If it's missing from the source, the page is rendered by JavaScript, and you need either a browser or a way to call the underlying API directly. Playwright gives you both options.

Setting up Playwright for Python

Install the library and download the browser binaries:

pip install playwright
playwright install chromium

Playwright also supports Firefox and WebKit, but Chromium is the usual choice for web scraping because it's the most widely tested and matches what most visitors use. Here is a minimal script:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products", wait_until="domcontentloaded")
    page.wait_for_selector(".product-card")
    print(page.title())
    browser.close()

The flow is always the same: launch a browser, open a page, navigate, wait for the content you need, extract, and close. The sync API is easiest for scripts. We'll switch to async later when concurrency matters.

Wait for elements the right way

Most flaky scrapers fail because of bad waiting. Three common mistakes: reading the DOM too early, sleeping for a fixed time, and relying on networkidle for everything.

Playwright's locators auto-wait. When you call locator.click() or locator.inner_text(), Playwright waits until the element exists, is visible, and is stable. This is the behavior you want in most cases:

name = page.locator("h1.product-title").inner_text()

When you need to wait explicitly, choose the narrowest condition that proves the data is ready:

• page.wait_for_selector(".product-card") waits for the first item to render.
• page.wait_for_function("document.querySelectorAll('.product-card').length >= 20") waits for a specific amount of content.
• page.wait_for_load_state("networkidle") waits until there's no network activity for 500 ms. It's a blunt tool. Sites with analytics pings, WebSockets, or polling never go idle, so use it sparingly.

Avoid time.sleep() and page.wait_for_timeout() as your main strategy. A fixed delay is either too short and breaks, or too long and wastes time on every page. Use it only for deliberate pauses, such as spacing out requests.

Extract data from the rendered DOM

Once the content has rendered, you can pull it out with locators:

cards = page.locator(".product-card")
for i in range(cards.count()):
    card = cards.nth(i)
    name = card.locator("h2").inner_text()
    price = card.locator(".price").inner_text()

This works, but every inner_text() call is a round trip between Python and the browser. On a page with 500 items and five fields each, that adds up. A faster approach is to run one JavaScript function on the page and return everything at once:

rows = page.locator(".product-card").evaluate_all("""
  els => els.map(el => ({
    name: el.querySelector('h2')?.textContent.trim(),
    price: el.querySelector('.price')?.textContent.trim(),
    url: el.querySelector('a')?.href
  }))
""")

One call returns a list of dictionaries. For large pages, this is often ten times faster than looping over locators in Python. Use optional chaining (?.) so a missing field returns null instead of throwing an error and killing the whole extraction.

Check for a hidden API before parsing HTML

Before you write a single selector, open the Network tab in DevTools, filter by Fetch/XHR, and reload the page. Dynamic sites usually load their data from a JSON endpoint. If you can see that endpoint, you can often skip the HTML parsing entirely.

Playwright lets you capture those responses while the page runs, which means you get clean JSON without reverse-engineering authentication or request signing:

with page.expect_response(
    lambda r: "/api/products" in r.url and r.status == 200
) as resp_info:
    page.goto("https://example.com/products")

data = resp_info.value.json()
for item in data["items"]:
    print(item["name"], item["price"])

JSON is far more stable than CSS selectors. Designers change class names constantly, but API response shapes tend to survive redesigns. When a clean API response is available, prefer it. Fall back to DOM extraction when the data is only assembled on the client or the API is heavily protected.

Handle infinite scroll

Infinite scroll loads more items as the user nears the bottom of the page. To scrape it, you scroll, wait for new content, and stop when nothing new appears. Counting items is more reliable than comparing page height:

from playwright.sync_api import TimeoutError as PWTimeout

def scroll_to_end(page, selector=".product-card", max_rounds=100):
    previous = 0
    for _ in range(max_rounds):
        page.mouse.wheel(0, 25000)
        try:
            page.wait_for_function(
                "([sel, prev]) => document.querySelectorAll(sel).length > prev",
                arg=[selector, previous],
                timeout=4000,
            )
        except PWTimeout:
            break  # no new items appeared, we've reached the end
        previous = page.locator(selector).count()
    return previous

The loop scrolls, then waits up to four seconds for the item count to increase. If the timeout fires, the list has ended. The max_rounds cap protects you from feeds that never stop.

For "Load more" buttons, the pattern is similar. Click the button inside expect_response or wait for the count to grow, and break when the button disappears or becomes disabled.

Handle pagination and multiple pages

For classic pagination, loop until there is no next link. Always wait for the new page's content, not just for the click to finish:

results = []
while True:
    page.wait_for_selector(".product-card")
    results += page.locator(".product-card").evaluate_all(EXTRACT_JS)

    next_btn = page.locator("a[rel='next']")
    if next_btn.count() == 0:
        break
    next_btn.click()
    page.wait_for_load_state("domcontentloaded")

If the URL contains a page number, it is usually faster and more robust to generate URLs directly and open them in parallel contexts than to click through one at a time.

Make it faster: block junk and go async

A browser downloads images, fonts, video, and trackers you don't need. Blocking them cuts load time and bandwidth sharply:

def block_heavy(route):
    if route.request.resource_type in {"image", "font", "media"}:
        route.abort()
    else:
        route.continue_()

page.route("**/*", block_heavy)

Next, reuse the browser and create lightweight contexts rather than launching a new browser per URL. A context is an isolated session with its own cookies and storage, and it starts in milliseconds. Combine this with the async API and a semaphore to control concurrency:

import asyncio
from playwright.async_api import async_playwright

async def scrape(browser, url, sem):
    async with sem:
        context = await browser.new_context()
        page = await context.new_page()
        try:
            await page.goto(url, wait_until="domcontentloaded")
            await page.wait_for_selector(".product-card")
            return await page.locator(".product-card").evaluate_all(EXTRACT_JS)
        finally:
            await context.close()

async def main(urls):
    sem = asyncio.Semaphore(5)
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        results = await asyncio.gather(*(scrape(browser, u, sem) for u in urls))
        await browser.close()
    return results

Start with five concurrent pages and increase slowly. Each page consumes real memory, often 100 to 300 MB, so concurrency is limited by RAM before it's limited by your network.

Build reliability into the scraper

A script that works once is a prototype. A script that works every night is a pipeline. A few habits make the difference.

• Retry with backoff: Network errors and timeouts are normal. Wrap navigation in a retry helper (the tenacity library works well) and use exponential backoff so you don't hammer a struggling server.
• Set explicit timeouts: The default 30 seconds per action is generous. Set tighter limits with page.set_default_timeout(10000) so a stuck page fails fast and moves to the retry queue.
• Validate what you extract: Check that required fields are present and that prices parse as numbers. A selector that silently returns None on every row is the most common sign a site has changed. Fail loudly when the percentage of empty fields crosses a threshold.
• Save as you go: Write each batch to JSON Lines or a database as it arrives instead of holding everything in memory. If the run crashes at item 9,000, you keep the first 8,999.
• Be a polite client: Respect robots.txt where it applies, space out requests, and identify yourself sensibly. Aggressive scraping gets blocked faster, and it burdens the sites you depend on.

Debug like a developer

Playwright ships with excellent tooling. Three features are worth learning early:

• Headed mode: Launch with headless=False and slow_mo=250 to watch the browser work step by step.
• Codegen: Run playwright codegen https://example.com to record your clicks and generate locator code you can adapt.
• Trace viewer: Start a trace with context.tracing.start(screenshots=True, snapshots=True), save it on failure, and open it with playwright show-trace trace.zip. You get a timeline of every action, a DOM snapshot at each step, and the network log. It is the fastest way to understand why a run failed on a server you can't watch.

Know the limits of browser scraping

Playwright handles rendering well, but it isn't magic. Many sites run bot detection that examines TLS fingerprints, browser properties, request patterns, and IP reputation. A default headless browser from a datacenter IP will be flagged on protected sites regardless of how well your selectors work. Solving that usually involves residential or rotating proxies, realistic browser profiles, and careful pacing. It is a separate topic with its own trade-offs.

Browsers are also expensive. If your target exposes a stable JSON API, calling it directly with httpx will be orders of magnitude cheaper than rendering pages. Use Playwright to discover the API, then decide whether you still need the browser in production.

Putting it together

Scraping JavaScript-heavy sites comes down to a handful of principles:

1. Check the Network tab first. If there's a clean API, use it.
2. Rely on auto-waiting locators and wait for specific conditions rather than fixed delays.
3. Extract in bulk with evaluate_all instead of many small calls.
4. Handle infinite scroll by counting items and stopping when growth ends.
5. Block unneeded resources, reuse the browser, and cap concurrency.
6. Validate output, retry failures, and save progress continuously.

With these patterns, Playwright turns the hardest category of pages into a manageable engineering problem. Start with one page, get the extraction right, then layer in scrolling, concurrency, and monitoring as the job grows. When maintaining proxies, browsers, and breakage handling starts to cost more than the data is worth, that's the point to consider a managed scraping service.

Frequently asked questions

Can I use AI to scrape websites with Playwright?

Yes. Playwright handles the hard part for AI-powered web scraping: it renders the JavaScript and gives you the final DOM. You then pass the cleaned content to an LLM and ask it to return structured JSON. This works well on messy or inconsistent pages where writing CSS selectors for every layout is impractical.

A reliable pattern is to render the page, trim it to the relevant section, send it with a strict schema, and validate the output:

from pydantic import BaseModel

class Product(BaseModel):
    name: str
    price: float | None = None
    currency: str | None = None

html = page.locator("main").inner_html()  # trim before sending
# send `html` (or its text) to your LLM with the schema,
# then validate the reply instead of trusting it:
product = Product.model_validate_json(llm_response)

The trade-offs are cost, latency, and occasional wrong values. For high-volume pages with a stable layout, CSS selectors or a hidden JSON API are cheaper and more predictable. Many teams use selectors as the default and call the LLM only as a fallback when extraction fails or the layout varies.

Should I use Playwright, Selenium, or Puppeteer for scraping?

For new Python projects, Playwright is usually the better choice. It has built-in auto-waiting, cleaner async support, isolated browser contexts, and tools like codegen and the trace viewer. Selenium is still reasonable if you have a large existing test suite or need its wider ecosystem. Puppeteer is a solid option if your stack is Node.js, since Playwright's Python bindings are the main reason to pick it for Python work.

How do I scrape pages that require a login?

Log in once, save the session state, and reuse it so you don't authenticate on every run:

# One time: log in, then save cookies and local storage
context.storage_state(path="auth.json")

# Later runs: start already authenticated
context = browser.new_context(storage_state="auth.json")

Only scrape content you're authorized to access, and check the site's terms first. Sessions expire, so add a check that detects a logged-out page and refreshes the state.

Why does my scraper work locally but fail on a server?

The usual causes are missing system dependencies, a datacenter IP that gets flagged, and low memory. Install the browser with its OS dependencies using playwright install --with-deps chromium, or build from the official Playwright Python Docker image. If pages load but return blocks or CAPTCHAs, the problem is IP reputation and browser fingerprinting, not your code. Also check RAM, because each page can use 100 to 300 MB.

How do I avoid getting blocked while scraping?

Start with the basics: lower your concurrency, add delays between requests, block assets you don't need, and reuse sessions where it makes sense. Prefer an official or hidden API over rendering pages when one exists. If a site still blocks you, you're likely dealing with fingerprinting or IP reputation, which usually needs rotating proxies and more realistic browser profiles. Always respect robots.txt and the site's terms, and stop if a site clearly doesn't want automated access.

Comments

[Alina Shah]

Thanks for checking out this guide!

One important takeaway: Before scraping a JavaScript-heavy website with Playwright, always check whether the required data is available through an API. It can save significant time and resources compared to browser-based extraction.

I'd love to hear from other developers:

Do you prefer Playwright, Selenium, or direct API extraction for handling dynamic websites?

Share your experience in the comments! 👇

[Dev Stack Hub]

The evaluate_all section is the most useful part for me. Each inner_text() call is a separate CDP round trip, so batching the extraction into one in-page function removes most of the latency on large listings. The ?. tip also matters, because one missing node would otherwise throw and kill the whole batch.

To answer your question, my default is direct API extraction. I use Playwright’s expect_response or page.on("response") to find the endpoint, then replay it with httpx in production. This gives much lower memory use per request than a 100-300 MB page, and the JSON schema usually survives redesigns that would break CSS selectors.

I fall back to Playwright only when:

• The payload is signed or tokenized per request (HMAC headers, short-lived tokens), so replaying it outside the browser context is impractical
• The data is assembled client-side from several calls or from WebSocket frames
• The site relies on fingerprinting that a plain HTTP client fails

Two additions that have helped me:

Use context.route() instead of page.route() so request blocking applies to every page in the context, including popups
Persist storage_state and add a logged-out check (for example, a redirect to /login), since expired sessions otherwise show up as silently empty results

Have you benchmarked async Playwright with a semaphore against httpx plus a hidden API at scale? I’d be curious about the cost-per-1k-pages difference.

[Jennifer Smith]

The “check the Network tab first” tip is the one I wish I’d learned earlier. I’ve lost hours writing selectors for pages that were just rendering a clean JSON response.

To answer your question, my order is usually:

1. Direct API extraction if there’s a stable endpoint. It’s faster, cheaper, and survives redesigns much better than CSS selectors.
2. Playwright when the data only exists after client-side rendering, or when I need to handle scrolling, logins, or session state. Auto-waiting alone makes it far less flaky than Selenium for me.
3. Selenium only when I’m working in a codebase that already uses it.

I like the hybrid approach you mention: use Playwright to discover the API, then switch to httpx for production runs.

Have you run into cases where the hidden API was too heavily protected (signed requests, rotating tokens) to be worth calling directly? I’d be curious how you decide when to give up on the API route and just render the page.