DEV Community

Python Tools Lab
Python Tools Lab

Posted on

How to Scrape Dynamic JavaScript Websites with Python & Playwright

When requests and BeautifulSoup return empty HTML tags, it usually means the website renders its content dynamically via JavaScript or React after the page loads.

Instead of fighting broken timers, you can use Playwright in Python to wait for network activity to settle and extract the rendered DOM cleanly:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

quotes = page.locator(".quote .text").all_inner_texts()
for q in quotes[:5]:
    print(q)

browser.close()
Enter fullscreen mode Exit fullscreen mode
Enter fullscreen mode Exit fullscreen mode




Want All 8 Python Scraping & Excel Automation Templates?

If you want ready-to-run templates for static sites, Playwright infinite scroll, anti-block User-Agent rotation, plus automated Excel/CSV cleaning and reporting, check out my complete 2-in-1 toolkit:

Top comments (0)