How to Scrape JavaScript-Heavy Sites with Playwright

How to Scrape JavaScript-Heavy Sites with Playwright

You’ve probably hit this wall: you write a clean script with requests or httpx, fire it off, and get back a blank page or a skeleton HTML with zero data. That’s because the site you’re targeting doesn’t render its content until JavaScript runs in the browser. Traditional scrapers can’t execute that JS, so they see nothing. The solution? Use a tool that actually runs a browser. Enter Playwright—a modern, fast, and reliable browser automation library that lets you scrape JavaScript-heavy sites, single-page applications (SPAs), and dynamic content with ease.

Unlike older tools like Selenium, Playwright is built for speed, supports multiple browsers (Chromium, Firefox, WebKit), and has native async support. It’s also smart about waiting: you don’t need to guess how long to wait for content to load. Playwright can wait for selectors, network events, or even specific DOM states. Let’s get you building a scraper that works today.

Why Playwright Beats Traditional Scrapers

Most HTTP-based scrapers just fetch the raw HTML response. But modern websites—especially those built with React, Vue, or Angular—load content dynamically via AJAX or WebSockets. The initial HTML might be empty, and the real data only appears after JS executes.

Playwright solves this by controlling a real browser. It launches Chromium (or another browser), navigates to the page, executes all JavaScript, and gives you the fully rendered DOM. This means:

• You get the actual content users see.
• You can interact with the page: click buttons, fill forms, scroll, and handle pop-ups.
• You avoid many blocking issues because Playwright mimics human behavior more closely than raw HTTP requests.

According to industry guides, Playwright is “highly effective for web scraping JavaScript-heavy sites, SPAs, and any dynamic content page” [3]. It’s also a great alternative to Selenium because of its asynchronous nature and cross-browser support [5].

Setting Up Your Python Environment

Before writing code, let’s set up a clean Python environment. You’ll need:

• Python 3.8+
• playwright package
• pandas (optional, for data handling)

Here’s how to install everything:

python -m venv playwright_scraper
source playwright_scraper/bin/activate  # On Windows: playwright_scraper\Scripts\activate
pip install playwright pandas
playwright install

The playwright install command downloads the optimized browser binaries Playwright uses [6].

Building Your First Scraper: A Real Example

Let’s scrape a simple JavaScript-heavy e-commerce sandbox that mimics real product listings. We’ll use the Oxylabs sandbox site, which is perfect for practice [6].

Our goals:

1. Launch a browser and navigate to the URL.
2. Wait for product items to load.
3. Extract product name, price, and URL.
4. Save results to a CSV.

Here’s a complete, working Python script using Playwright’s sync API (simpler for beginners):

from playwright.sync_api import sync_playwright
import csv

URL = "https://sandbox.oxylabs.io/products"
OUTPUT_FILE = "products.csv"

def scrape_products():
    products = []

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)  # Set headless=False to see the browser
        page = browser.new_page()
        page.goto(URL)

        # Wait for product items to be rendered
        page.wait_for_selector('[data-content="product-item"]', state="attached")

        # Get all product locators
        product_items = page.locator('[data-content="product-item"]')

        for item in product_items.all():
            name = item.locator("h2 a").text_content()
            price = item.locator(".price").text_content()
            url = item.locator("h2 a").get_attribute("href")

            products.append({
                "name": name,
                "price": price,
                "url": url
            })

        browser.close()

    # Write to CSV
    with open(OUTPUT_FILE, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["name", "price", "url"])
        writer.writeheader()
        writer.writerows(products)

    print(f"Saved {len(products)} products to {OUTPUT_FILE}")

if __name__ == "__main__":
    scrape_products()

Run this script, and you’ll get a products.csv file with real product data. No manual waits, no guesswork—just reliable scraping [6].

Handling Dynamic Content and Waiting Strategies

One of Playwright’s biggest strengths is its smart waiting. You don’t need time.sleep(5) anymore. Instead, use:

• page.wait_for_selector() – waits for an element to appear.
• page.wait_for_load_state("networkidle") – waits until network activity stops.
• page.locator() – automatically waits and retries until the element is ready [6].

For example, if your site loads content after scrolling, you can scroll and then wait:

page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
page.wait_for_load_state("networkidle")

This ensures all AJAX-loaded products are visible before scraping [4].

Avoiding Detection and Rate Limiting

Since Playwright runs a real browser, it’s harder for sites to detect you as a bot. But you can still get blocked if you’re too aggressive. Here are pro tips:

• Use headless mode in production (headless=True) to reduce overhead.
• Add random delays between requests to mimic human behavior.
• Implement retry logic with exponential backoff for 403 errors [2].
• Block unnecessary resources (images, fonts) to speed up scraping:

def block_resources(page):
    page.route(".*", lambda route, request: 
        route.abort() if request.resource_type in ["image", "font", "stylesheet"] 
        else route.continue_()
    )

page = browser.new_page()
block_resources(page)

This technique reduces load time and bandwidth [6].

Debugging and Testing Selectors

Before writing code, inspect the page in Browser DevTools:

1. Right-click an element → “Inspect”.
2. Find a stable selector (e.g., data-content="product-item").
3. Copy it and drop it into page.locator().

Playwright also offers helper methods like:

• get_by_role() – finds elements by accessibility role.
• get_by_text() – matches visible text.
• get_by_test_id() – targets elements with a test ID [6].

These make your scraper more robust against DOM changes.

When to Use Playwright vs. Other Tools

Tool	Best For	Limitations
Playwright	JS-heavy sites, SPAs, interactions	Slower than HTTP scrapers
Selenium	Legacy automation, cross-browser	Less modern, slower sync
Puppeteer	Node.js-only JS scraping	Chromium-only, no Firefox/WebKit
Requests	Static HTML, APIs	Can’t handle JS

Playwright is the best choice when you need real browser execution and interaction capabilities [2][3].

Take Action Today

You don’t need to reverse-engineer APIs or struggle with blank pages. With Playwright, you can scrape any JavaScript-heavy site by simply running a browser. Try the script above on a sandbox site, then adapt it to your target.

Your next step: Pick a real JavaScript-heavy site you’ve struggled with, install Playwright, and run the scraper. Share your results or questions in the comments—let’s build together.

Got a tricky site you’re scraping? Drop the URL and what you’re trying to extract. I’ll help you craft a selector strategy. Let’s make scraping easier, one page at a time.

---

If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming!

---

🛠️ Useful resource: **Content Creator Ultimate Bundle (Save 33%)* — $29.99. Check it out on Gumroad!*