Most web scraping tutorials show you how to use requests and BeautifulSoup. That works great for static HTML, but modern websites built with React, Vue, or Next.js load data dynamically via JavaScript—leaving you with empty tags.
Here is a clean, production-ready template using Playwright that renders JavaScript, handles infinite scrolling, and exports directly to CSV:
import time
import pandas as pd
from playwright.sync_api import sync_playwright
TARGET_URL = "https://quotes.toscrape.com/js/"
WAIT_SELECTOR = "div.quote"
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(TARGET_URL, wait_until="networkidle")
page.wait_for_selector(WAIT_SELECTOR)
for _ in range(3):
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
time.sleep(2)
items = page.query_selector_all(WAIT_SELECTOR)
data = [{"Quote": i.query_selector("span.text").inner_text()} for i in items]
browser.close()
pd.DataFrame(data).to_csv("output.csv", index=False)
Want the Complete Playbook & 4 Ready-to-Use Templates?
If you want the full step-by-step guide—including static catalog scrapers, HTML table extractors, and anti-block User-Agent rotation scripts—grab The No-BS Python Web Scraping Playbook here:
Download the Playbook & Python Templates: https://tugrulian8.gumroad.com/l/ktiqff
Top comments (0)