DEV Community

Python Tools Lab
Python Tools Lab

Posted on

How to Scrape Dynamic JavaScript Websites with Python and Playwright (2026 Guide)

Most web scraping tutorials show you how to use requests and BeautifulSoup. That works great for static HTML, but modern websites built with React, Vue, or Next.js load data dynamically via JavaScript—leaving you with empty tags.

Here is a clean, production-ready template using Playwright that renders JavaScript, handles infinite scrolling, and exports directly to CSV:

import time
import pandas as pd
from playwright.sync_api import sync_playwright

TARGET_URL = "https://quotes.toscrape.com/js/"
WAIT_SELECTOR = "div.quote"

with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(TARGET_URL, wait_until="networkidle")
page.wait_for_selector(WAIT_SELECTOR)

for _ in range(3):
    page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    time.sleep(2)

items = page.query_selector_all(WAIT_SELECTOR)
data = [{"Quote": i.query_selector("span.text").inner_text()} for i in items]
browser.close()
Enter fullscreen mode Exit fullscreen mode

pd.DataFrame(data).to_csv("output.csv", index=False)

Enter fullscreen mode Exit fullscreen mode




Want the Complete Playbook & 4 Ready-to-Use Templates?

If you want the full step-by-step guide—including static catalog scrapers, HTML table extractors, and anti-block User-Agent rotation scripts—grab The No-BS Python Web Scraping Playbook here:

Download the Playbook & Python Templates: https://tugrulian8.gumroad.com/l/ktiqff

Top comments (0)