When requests and BeautifulSoup return empty HTML tags, it usually means the website renders its content dynamically via JavaScript or React after the page loads.
Instead of fighting broken timers, you can use Playwright in Python to wait for network activity to settle and extract the rendered DOM cleanly:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")
quotes = page.locator(".quote .text").all_inner_texts()
for q in quotes[:5]:
print(q)
browser.close()
Want All 8 Python Scraping & Excel Automation Templates?
If you want ready-to-run templates for static sites, Playwright infinite scroll, anti-block User-Agent rotation, plus automated Excel/CSV cleaning and reporting, check out my complete 2-in-1 toolkit:
- Grab The Complete Python Data & Automation Bundle (8 Scripts + 2 PDFs): https://tugrulian8.gumroad.com/l/xhgdyr
- Or try the Free Static Web Scraper Starter ($0): https://tugrulian8.gumroad.com/l/xvcnrob
Top comments (0)