DEV Community

Zenrows profile image Benny
Benny for Zenrows

Posted on Edited on Originally published at zenrows.com

What We Learned Building a DIY Web Scraper published

#python #playwright #webscraping #tutorial

This article was originally published on the Zenrows blog.

We built a Python web scraper from scratch with Requests, BeautifulSoup, and Playwright, then tested it against three protected sites and a documented rate limit. This post walks through the build with every script and output. You'll need Python 3.10 or later and a terminal. The self-hosted part needs no API key, and a free Zenrows key comes in at the end for comparison.

We came away with four lessons.

  • Browser headers, retries, and random delays are enough for static pages.
  • Playwright handles pages that fill in their content with JavaScript.
  • On protected sites, Requests got 403s, and Playwright got a timeout, a challenge page, and an empty shell.
  • A per-IP quota, like GitHub's 60 requests an hour, stops the scraper, however carefully you set its headers.

Zenrows Fetch with mode=auto returned all three protected pages. The full code is on GitHub.

Setup 

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install requests beautifulsoup4 lxml playwright brotli
python3 -m playwright install chromium
Enter fullscreen mode Exit fullscreen mode

All the scraper code goes into one file, scraper.py, built up step by step. Each test runs as its own stage.

1. Send Browser-Like Headers 

import random
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
]


def build_headers() -> dict[str, str]:
    return {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Connection": "keep-alive",
    }
Enter fullscreen mode Exit fullscreen mode

By default, Requests identifies itself as a script. This version rotates three browser User-Agent strings and sends Accept, Accept-Language, Accept-Encoding, and Connection headers that match a Chrome request. Browser-like headers may help with simple filters, but they don't reproduce a full browser or reliably bypass bot detection.

2. Retry With Backoff 

# continuing scraper.py
import time

RETRY_STATUS = {429, 500, 502, 503, 504}


def fetch(session: requests.Session, url: str, max_retries: int = 3) -> str | None:
    for attempt in range(max_retries):
        try:
            response = session.get(url, headers=build_headers(), timeout=15)

            if response.status_code == 200:
                return response.text

            if response.status_code not in RETRY_STATUS:
                print(f"{url} returned {response.status_code}, not retrying")
                return None

            time.sleep(2 ** attempt)

        except requests.RequestException as error:
            print(f"{url} failed with {type(error).__name__}")
            time.sleep(2 ** attempt)

    return None
Enter fullscreen mode Exit fullscreen mode

fetch() retries on 429, 500, 502, 503, and 504 with exponential backoff, and the 15-second timeout keeps an unresponsive server from stalling the run. Any other status returns immediately. A 404 means the page doesn't exist, so another attempt wouldn't change the result.

3. Add Polite Delays 

# continuing scraper.py

def polite_delay(base: float = 1.5, jitter: float = 1.0) -> None:
    time.sleep(base + random.uniform(0, jitter))
Enter fullscreen mode Exit fullscreen mode

A burst of requests from one IP address can trigger site-specific rate limits or bot detection. Each pause is 1.5 seconds plus up to 1 second of random jitter, because a person reading a page doesn't click at fixed intervals.

4. Parse Products Into a Dataclass

Each product on the demo store is an li.product element holding a .product-name, a .product-price, and a product link. BeautifulSoup with the lxml parser handles malformed HTML better than the built-in parser.

# continuing scraper.py

from bs4 import BeautifulSoup
from dataclasses import dataclass, asdict


@dataclass
class Product:
    name: str
    price: str
    url: str


def parse_products(html: str) -> list[Product]:
    soup = BeautifulSoup(html, "lxml")
    products = []

    for card in soup.select("li.product"):
        name = card.select_one(".product-name")
        price = card.select_one(".product-price")
        link = card.select_one("a.woocommerce-LoopProduct-link")

        if not (name and price and link):
            continue

        products.append(Product(
            name=name.get_text(strip=True),
            price=price.get_text(strip=True),
            url=link["href"],
        ))

    return products
Enter fullscreen mode Exit fullscreen mode

5. Scrape Three Pages to CSV 

# continuing scraper.py

import csv

BASE_URL = "https://www.scrapingcourse.com/ecommerce/page/{}/"


def scrape(pages: int = 3) -> list[Product]:
    all_products = []

    with requests.Session() as session:
        for page in range(1, pages + 1):
            html = fetch(session, BASE_URL.format(page))

            if html is None:
                continue

            products = parse_products(html)
            print(f"page {page}: {len(products)} products")
            all_products.extend(products)
            polite_delay()

    return all_products


def run_static() -> None:
    products = scrape()

    with open("products.csv", "w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=["name", "price", "url"])
        writer.writeheader()
        writer.writerows(asdict(p) for p in products)

    print(f"saved {len(products)} products")
Enter fullscreen mode Exit fullscreen mode

 

# keep this block at the bottom of scraper.py
import sys

if __name__ == "__main__":
    # run the stage named on the command line, for example: python3 scraper.py static
    stage = sys.argv[1] if len(sys.argv) > 1 else "static"
    globals()[f"run_{stage}"]()
Enter fullscreen mode Exit fullscreen mode

The main block runs the stage named on the command line. Run the static stage.

python3 scraper.py static
Enter fullscreen mode Exit fullscreen mode

 

page 1: 16 products
page 2: 16 products
page 3: 16 products
saved 48 products
Enter fullscreen mode Exit fullscreen mode

It scraped 48 products from the demo store across three pages into products.csv.

6. Handle JavaScript-Rendered Pages

A static fetch of a client-rendered page returns empty fields.

# continuing scraper.py, above the main block

def run_dynamic() -> None:
    url = "https://www.scrapingcourse.com/javascript-rendering"

    with requests.Session() as session:
        html = fetch(session, url)

    if html is None:
        raise RuntimeError(f"Could not fetch {url}")

    soup = BeautifulSoup(html, "lxml")

    for card in soup.select(".product-item")[:3]:
        name = card.select_one("[data-testid=product-name]")
        price = card.select_one("[data-testid=product-price]")
        link = card.select_one("a.product-link")
        print(repr(name.get_text(strip=True)), repr(price.get_text(strip=True)), repr(link["href"]))
Enter fullscreen mode Exit fullscreen mode

 

python3 scraper.py dynamic
Enter fullscreen mode Exit fullscreen mode

 

'' '' ''
'' '' ''
'' '' ''
Enter fullscreen mode Exit fullscreen mode

The parser finds the cards, but every field is empty. Add this line to the end of run_dynamic(), outside the loop, to print one card's markup.

print(soup.select(".product-item")[0].prettify()[:400])
Enter fullscreen mode Exit fullscreen mode

 

<div class="product-item flex flex-col items-center rounded-lg" data-content="product-item" data-testid="product-item" itemscope="" itemtype="http://schema.org/Product">
 <a class="product-link" data-content="product-link" data-testid="product-link" href="">
  <img alt="" class="rounded-lg border shadow-md object-cover" data-content="product-image" data-testid="product-image" decoding="async" fetc
Enter fullscreen mode Exit fullscreen mode

The server sends the page structure with empty href, src, name, and price values, and JavaScript fills them in after the page loads.

Playwright renders the page in headless Chromium and waits for a selector before reading it, so parsing doesn't happen too early. The finally block closes the browser so a failure doesn't leave Chromium processes running.

# continuing scraper.py, above the main block

from playwright.sync_api import sync_playwright


def fetch_rendered(url: str, wait_for: str, timeout: int = 30000) -> str | None:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(user_agent=random.choice(USER_AGENTS))

        try:
            page.goto(url, timeout=timeout)
            page.wait_for_selector(wait_for, timeout=timeout)
            return page.content()
        except Exception as error:
            print(f"{url} failed to render: {type(error).__name__}")
            return None
        finally:
            browser.close()
Enter fullscreen mode Exit fullscreen mode

 

# continuing scraper.py, above the main block

def parse_rendered(html: str) -> list[Product]:
    soup = BeautifulSoup(html, "lxml")
    products = []

    for card in soup.select(".product-item"):
        name = card.select_one("[data-testid=product-name]")
        price = card.select_one("[data-testid=product-price]")
        link = card.select_one("a.product-link")

        if not (name and price and link):
            continue

        products.append(Product(
            name=name.get_text(strip=True),
            price=price.get_text(strip=True),
            url=link["href"],
        ))

    return products


def run_rendered() -> None:
    url = "https://www.scrapingcourse.com/javascript-rendering"
    html = fetch_rendered(url, wait_for="[data-testid=product-name]:not(:empty)")

    if html:
        for product in parse_rendered(html)[:3]:
            print(product)
Enter fullscreen mode Exit fullscreen mode

 

python3 scraper.py rendered
Enter fullscreen mode Exit fullscreen mode

 

Product(name='Chaz Kangeroo Hoodie', price='$52', url='https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie')
Product(name='Teton Pullover Hoodie', price='$70', url='https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie')
Product(name='Bruno Compete Hoodie', price='$63', url='https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie')
Enter fullscreen mode Exit fullscreen mode

7. Test Protected Pages With Requests 

# continuing scraper.py, above the main block

TARGETS = [
    "https://dexscreener.com/solana",
    "https://www.sofascore.com/football",
    "https://www.g2.com/categories/crm",
]


def run_protected() -> None:
    with requests.Session() as session:
        for url in TARGETS:
            try:
                r = session.get(url, headers=build_headers(), timeout=15)
            except requests.RequestException as error:
                print(f"\n{url}\nfailed: {type(error).__name__}")
                continue

            print(f"\n{url}")
            print("status:", r.status_code, "| bytes:", len(r.content))
            print("server:", r.headers.get("server"))
            print("cf-mitigated:", r.headers.get("cf-mitigated"))
            print("x-datadome:", r.headers.get("x-datadome"))
            print("cookies:", list(r.cookies.keys()))
            print("title:", BeautifulSoup(r.text, "lxml").title)
Enter fullscreen mode Exit fullscreen mode

 

python3 scraper.py protected
Enter fullscreen mode Exit fullscreen mode

 

https://dexscreener.com/solana
status: 403 | bytes: 5796
server: cloudflare
cf-mitigated: challenge
x-datadome: None
cookies: ['__cf_bm']
title: <title>Just a moment...</title>

https://www.sofascore.com/football
status: 403 | bytes: 48
server: Varnish
cf-mitigated: None
x-datadome: None
cookies: []
title: None

https://www.g2.com/categories/crm
status: 403 | bytes: 1707
server: cloudflare
cf-mitigated: None
x-datadome: protected
cookies: ['datadome', '__cf_bm']
title: <title>g2.com</title>
Enter fullscreen mode Exit fullscreen mode

All three returned 403, and each was blocked differently. Dexscreener's cf-mitigated: challenge header and "Just a moment..." title mark Cloudflare's JavaScript challenge page. Sofascore blocked the request at its Varnish edge with 48 bytes, no cookies, and no HTML. G2's x-datadome: protected header and datadome cookie show DataDome, and the response is a placeholder page with nothing to parse. Responses may differ by IP, session, time, and target configuration.

8. Hit a Documented Rate Limit 

# continuing scraper.py, above the main block

RATE_LIMIT_URL = "https://api.github.com/users/Python"


def run_ratelimit() -> None:
    with requests.Session() as session:
        for i in range(1, 71):
            r = session.get(RATE_LIMIT_URL, headers=build_headers(), timeout=15)
            remaining = r.headers.get("x-ratelimit-remaining")

            if i == 1 or i % 10 == 0 or r.status_code != 200:
                print(f"request {i}: {r.status_code} | remaining: {remaining}")

            if r.status_code != 200:
                print("limit:", r.headers.get("x-ratelimit-limit"))
                print("reset:", r.headers.get("x-ratelimit-reset"))
                break
Enter fullscreen mode Exit fullscreen mode

 

python3 scraper.py ratelimit
Enter fullscreen mode Exit fullscreen mode

 

request 1: 200 | remaining: 58
request 10: 200 | remaining: 49
request 20: 200 | remaining: 39
request 30: 200 | remaining: 29
request 40: 200 | remaining: 19
request 50: 200 | remaining: 9
request 60: 403 | remaining: 0
limit: 60
reset: 1792572851
Enter fullscreen mode Exit fullscreen mode

GitHub's API documents a limit of 60 requests an hour for unauthenticated clients, which makes it a clean test even though it's an API. The x-ratelimit-remaining header counted down from the first request, and the 60th request returned 403. The reset value is a Unix timestamp for when the window reopens. The loop leaves out the delay from step 3, and adding it back wouldn't change the result, because this limit is a per-IP quota.

9. Test Protected Pages With Playwright 

# continuing scraper.py, above the main block

def run_browser() -> None:
    for url in TARGETS:
        html = fetch_rendered(url, wait_for="body")

        if html is None:
            print(f"\n{url}\nno content returned")
            continue

        soup = BeautifulSoup(html, "lxml")
        print(f"\n{url}")
        print("bytes:", len(html))
        print("title:", soup.title)
Enter fullscreen mode Exit fullscreen mode

 

python3 scraper.py browser
Enter fullscreen mode Exit fullscreen mode

 

https://dexscreener.com/solana failed to render: TimeoutError

https://dexscreener.com/solana
no content returned

https://www.sofascore.com/football
bytes: 5563
title: <title>Sofascore</title>

https://www.g2.com/categories/crm
bytes: 2609
title: <title>g2.com</title>
Enter fullscreen mode Exit fullscreen mode

Dexscreener timed out, Sofascore returned a "Verifying you are human" page, and G2 returned a 2,609-byte empty shell.

10. Fetch the Same Pages Through Zenrows

Create a free API key on the signup page. The free plan includes 5,000 credits a month with no credit card. Install the two dependencies, add your key to .env, and add .env to .gitignore.

pip3 install requests python-dotenv
Enter fullscreen mode Exit fullscreen mode

 

ZENROWS_API_KEY=your_zenrows_api_key
Enter fullscreen mode Exit fullscreen mode

Create zenrows_fetch.py.

import os

import requests
from bs4 import BeautifulSoup
from dotenv import load_dotenv

load_dotenv()

TARGETS = [
    "https://dexscreener.com/solana",
    "https://www.sofascore.com/football",
    "https://www.g2.com/categories/crm",
]


def fetch_zenrows(url: str) -> str | None:
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "apikey": os.getenv("ZENROWS_API_KEY"),
            "url": url,
            "mode": "auto",
        },
        timeout=180,
    )

    if response.status_code == 200:
        return response.text

    print(f"{url} returned {response.status_code}: {response.text[:200]}")
    return None


for url in TARGETS:
    html = fetch_zenrows(url)

    if html:
        soup = BeautifulSoup(html, "lxml")
        print(f"\n{url}")
        print("bytes:", len(html))
        print("title:", soup.title)
Enter fullscreen mode Exit fullscreen mode

 

python3 zenrows_fetch.py
Enter fullscreen mode Exit fullscreen mode

 

https://dexscreener.com/solana
bytes: 2132120
title: <title data-rh="true">Solana DEX Screener</title>

https://www.sofascore.com/football
bytes: 183264
title: <title data-next-head="">Football Live Scores, Fixtures &amp; Odds | Sofascore | Championship League, Premier League</title>

https://www.g2.com/categories/crm
bytes: 827211
title: <title>Best CRM Software: User Reviews from October 2026</title>
Enter fullscreen mode Exit fullscreen mode

All three requests returned 200 with the full page.

Results

Target Protection Requests Playwright Zenrows Fetch (mode=auto)
Dexscreener Cloudflare challenge 403, "Just a moment..." page Timed out 200, full page (2,132,120 bytes)
Sofascore Varnish edge block, then a JavaScript challenge 403, 48 bytes "Verifying you are human" page 200, full page (183,264 bytes)
G2 DataDome 403, placeholder page 2,609-byte empty shell 200, full page (827,211 bytes)

Requests and Playwright get 403s, challenge pages, and empty shells, while Zenrows Fetch returns all three full pages

mode=auto starts with the lowest-cost configuration that works and escalates to a heavier one only when a page needs it. You're billed once, for the configuration that succeeds. It also handles IP-based restrictions like the per-IP quota from the GitHub test. The parser stays the same, and only the fetch layer changes.

Wrapping Up

A self-hosted scraper works well for open pages with bounded volume. Protected targets need stealth plugins, residential proxies, CAPTCHA solvers, or anti-detect browsers, and each one is another component to maintain whenever a protection changes. At thousands of pages, the one-browser-per-call pattern also needs bounded concurrency, browser lifecycle management, crash recovery, and capacity testing. Keep self-hosting when target access is open and bounded, and use a managed layer when protected access and ongoing maintenance are work you don't want to own.

What's Next

Top comments (0)