DEV Community: Alina Shah

Choosing an API gateway is about more than routing requests. Security, scalability, rate limiting, and failover all matter in production. This guide compares 6 options to help developers choose the right fit.

Alina Shah — Sat, 31 Oct 2026 15:45:15 +0000

Dev Stack Hub

Oct 31

6 Best API Gateways for Developers Breaking Their Routing Logic

#discuss #api #devops #backend

19 min read

Struggling with JavaScript-heavy websites? Learn how to scrape dynamic content using Playwright and Python, handle infinite scrolling, capture API responses, and build reliable web scrapers. 🚀

Alina Shah — Sat, 31 Oct 2026 14:18:14 +0000

Alina Shah

Oct 31

Scrape JavaScript-Heavy Sites with Playwright in Python (Developer Guide)

#discuss #javascript #python #playwright

9 min read

Scrape JavaScript-Heavy Sites with Playwright in Python (Developer Guide)

Alina Shah — Sat, 31 Oct 2026 14:15:42 +0000

You fire off a requests.get(), parse the response with BeautifulSoup, and find an empty <div id="root"></div>. The data you can see in your browser never made it into the HTML. This is the everyday reality of scraping modern sites built with React, Vue, Angular, or any framework that renders content in the browser.

Playwright solves this by driving a real browser engine. It runs the page's JavaScript, waits for the content to appear, and lets you extract it just as a user would see it. This guide walks through the full workflow in Python: setup, waiting strategies, infinite scroll, intercepting API calls, performance tuning, and making your scraper reliable enough to run unattended.

Why requests and BeautifulSoup fail on dynamic pages

A traditional HTTP client downloads the initial HTML document and stops. On a client-side rendered site, that document is mostly a shell. The real content arrives later through JavaScript that fetches JSON from an API, builds DOM nodes, and injects them into the page.

You can confirm this quickly. Open DevTools, choose "View Source" (not "Inspect"), and search for a piece of text you can see on screen. If it's missing from the source, the page is rendered by JavaScript, and you need either a browser or a way to call the underlying API directly. Playwright gives you both options.

Setting up Playwright for Python

Install the library and download the browser binaries:

pip install playwright
playwright install chromium

Playwright also supports Firefox and WebKit, but Chromium is the usual choice for web scraping because it's the most widely tested and matches what most visitors use. Here is a minimal script:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products", wait_until="domcontentloaded")
    page.wait_for_selector(".product-card")
    print(page.title())
    browser.close()

The flow is always the same: launch a browser, open a page, navigate, wait for the content you need, extract, and close. The sync API is easiest for scripts. We'll switch to async later when concurrency matters.

Wait for elements the right way

Most flaky scrapers fail because of bad waiting. Three common mistakes: reading the DOM too early, sleeping for a fixed time, and relying on networkidle for everything.

Playwright's locators auto-wait. When you call locator.click() or locator.inner_text(), Playwright waits until the element exists, is visible, and is stable. This is the behavior you want in most cases:

name = page.locator("h1.product-title").inner_text()

When you need to wait explicitly, choose the narrowest condition that proves the data is ready:

page.wait_for_selector(".product-card") waits for the first item to render.
page.wait_for_function("document.querySelectorAll('.product-card').length >= 20") waits for a specific amount of content.
page.wait_for_load_state("networkidle") waits until there's no network activity for 500 ms. It's a blunt tool. Sites with analytics pings, WebSockets, or polling never go idle, so use it sparingly.

Avoid time.sleep() and page.wait_for_timeout() as your main strategy. A fixed delay is either too short and breaks, or too long and wastes time on every page. Use it only for deliberate pauses, such as spacing out requests.

Extract data from the rendered DOM

Once the content has rendered, you can pull it out with locators:

cards = page.locator(".product-card")
for i in range(cards.count()):
    card = cards.nth(i)
    name = card.locator("h2").inner_text()
    price = card.locator(".price").inner_text()

This works, but every inner_text() call is a round trip between Python and the browser. On a page with 500 items and five fields each, that adds up. A faster approach is to run one JavaScript function on the page and return everything at once:

rows = page.locator(".product-card").evaluate_all("""
  els => els.map(el => ({
    name: el.querySelector('h2')?.textContent.trim(),
    price: el.querySelector('.price')?.textContent.trim(),
    url: el.querySelector('a')?.href
  }))
""")

One call returns a list of dictionaries. For large pages, this is often ten times faster than looping over locators in Python. Use optional chaining (?.) so a missing field returns null instead of throwing an error and killing the whole extraction.

Check for a hidden API before parsing HTML

Before you write a single selector, open the Network tab in DevTools, filter by Fetch/XHR, and reload the page. Dynamic sites usually load their data from a JSON endpoint. If you can see that endpoint, you can often skip the HTML parsing entirely.

Playwright lets you capture those responses while the page runs, which means you get clean JSON without reverse-engineering authentication or request signing:

with page.expect_response(
    lambda r: "/api/products" in r.url and r.status == 200
) as resp_info:
    page.goto("https://example.com/products")

data = resp_info.value.json()
for item in data["items"]:
    print(item["name"], item["price"])

JSON is far more stable than CSS selectors. Designers change class names constantly, but API response shapes tend to survive redesigns. When a clean API response is available, prefer it. Fall back to DOM extraction when the data is only assembled on the client or the API is heavily protected.

Handle infinite scroll

Infinite scroll loads more items as the user nears the bottom of the page. To scrape it, you scroll, wait for new content, and stop when nothing new appears. Counting items is more reliable than comparing page height:

from playwright.sync_api import TimeoutError as PWTimeout

def scroll_to_end(page, selector=".product-card", max_rounds=100):
    previous = 0
    for _ in range(max_rounds):
        page.mouse.wheel(0, 25000)
        try:
            page.wait_for_function(
                "([sel, prev]) => document.querySelectorAll(sel).length > prev",
                arg=[selector, previous],
                timeout=4000,
            )
        except PWTimeout:
            break  # no new items appeared, we've reached the end
        previous = page.locator(selector).count()
    return previous

The loop scrolls, then waits up to four seconds for the item count to increase. If the timeout fires, the list has ended. The max_rounds cap protects you from feeds that never stop.

For "Load more" buttons, the pattern is similar. Click the button inside expect_response or wait for the count to grow, and break when the button disappears or becomes disabled.

Handle pagination and multiple pages

For classic pagination, loop until there is no next link. Always wait for the new page's content, not just for the click to finish:

results = []
while True:
    page.wait_for_selector(".product-card")
    results += page.locator(".product-card").evaluate_all(EXTRACT_JS)

    next_btn = page.locator("a[rel='next']")
    if next_btn.count() == 0:
        break
    next_btn.click()
    page.wait_for_load_state("domcontentloaded")

If the URL contains a page number, it is usually faster and more robust to generate URLs directly and open them in parallel contexts than to click through one at a time.

Make it faster: block junk and go async

A browser downloads images, fonts, video, and trackers you don't need. Blocking them cuts load time and bandwidth sharply:

def block_heavy(route):
    if route.request.resource_type in {"image", "font", "media"}:
        route.abort()
    else:
        route.continue_()

page.route("**/*", block_heavy)

Next, reuse the browser and create lightweight contexts rather than launching a new browser per URL. A context is an isolated session with its own cookies and storage, and it starts in milliseconds. Combine this with the async API and a semaphore to control concurrency:

import asyncio
from playwright.async_api import async_playwright

async def scrape(browser, url, sem):
    async with sem:
        context = await browser.new_context()
        page = await context.new_page()
        try:
            await page.goto(url, wait_until="domcontentloaded")
            await page.wait_for_selector(".product-card")
            return await page.locator(".product-card").evaluate_all(EXTRACT_JS)
        finally:
            await context.close()

async def main(urls):
    sem = asyncio.Semaphore(5)
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        results = await asyncio.gather(*(scrape(browser, u, sem) for u in urls))
        await browser.close()
    return results

Start with five concurrent pages and increase slowly. Each page consumes real memory, often 100 to 300 MB, so concurrency is limited by RAM before it's limited by your network.

Build reliability into the scraper

A script that works once is a prototype. A script that works every night is a pipeline. A few habits make the difference.

Retry with backoff: Network errors and timeouts are normal. Wrap navigation in a retry helper (the tenacity library works well) and use exponential backoff so you don't hammer a struggling server.
Set explicit timeouts: The default 30 seconds per action is generous. Set tighter limits with page.set_default_timeout(10000) so a stuck page fails fast and moves to the retry queue.
Validate what you extract: Check that required fields are present and that prices parse as numbers. A selector that silently returns None on every row is the most common sign a site has changed. Fail loudly when the percentage of empty fields crosses a threshold.
Save as you go: Write each batch to JSON Lines or a database as it arrives instead of holding everything in memory. If the run crashes at item 9,000, you keep the first 8,999.
Be a polite client: Respect robots.txt where it applies, space out requests, and identify yourself sensibly. Aggressive scraping gets blocked faster, and it burdens the sites you depend on.

Debug like a developer

Playwright ships with excellent tooling. Three features are worth learning early:

Headed mode: Launch with headless=False and slow_mo=250 to watch the browser work step by step.
Codegen: Run playwright codegen https://example.com to record your clicks and generate locator code you can adapt.
Trace viewer: Start a trace with context.tracing.start(screenshots=True, snapshots=True), save it on failure, and open it with playwright show-trace trace.zip. You get a timeline of every action, a DOM snapshot at each step, and the network log. It is the fastest way to understand why a run failed on a server you can't watch.

Know the limits of browser scraping

Playwright handles rendering well, but it isn't magic. Many sites run bot detection that examines TLS fingerprints, browser properties, request patterns, and IP reputation. A default headless browser from a datacenter IP will be flagged on protected sites regardless of how well your selectors work. Solving that usually involves residential or rotating proxies, realistic browser profiles, and careful pacing. It is a separate topic with its own trade-offs.

Browsers are also expensive. If your target exposes a stable JSON API, calling it directly with httpx will be orders of magnitude cheaper than rendering pages. Use Playwright to discover the API, then decide whether you still need the browser in production.

Putting it together

Scraping JavaScript-heavy sites comes down to a handful of principles:

Check the Network tab first. If there's a clean API, use it.
Rely on auto-waiting locators and wait for specific conditions rather than fixed delays.
Extract in bulk with evaluate_all instead of many small calls.
Handle infinite scroll by counting items and stopping when growth ends.
Block unneeded resources, reuse the browser, and cap concurrency.
Validate output, retry failures, and save progress continuously.

With these patterns, Playwright turns the hardest category of pages into a manageable engineering problem. Start with one page, get the extraction right, then layer in scrolling, concurrency, and monitoring as the job grows. When maintaining proxies, browsers, and breakage handling starts to cost more than the data is worth, that's the point to consider a managed scraping service.

Frequently asked questions

Can I use AI to scrape websites with Playwright?

Yes. Playwright handles the hard part for AI-powered web scraping: it renders the JavaScript and gives you the final DOM. You then pass the cleaned content to an LLM and ask it to return structured JSON. This works well on messy or inconsistent pages where writing CSS selectors for every layout is impractical.

A reliable pattern is to render the page, trim it to the relevant section, send it with a strict schema, and validate the output:

from pydantic import BaseModel

class Product(BaseModel):
    name: str
    price: float | None = None
    currency: str | None = None

html = page.locator("main").inner_html()  # trim before sending
# send `html` (or its text) to your LLM with the schema,
# then validate the reply instead of trusting it:
product = Product.model_validate_json(llm_response)

The trade-offs are cost, latency, and occasional wrong values. For high-volume pages with a stable layout, CSS selectors or a hidden JSON API are cheaper and more predictable. Many teams use selectors as the default and call the LLM only as a fallback when extraction fails or the layout varies.

Should I use Playwright, Selenium, or Puppeteer for scraping?

For new Python projects, Playwright is usually the better choice. It has built-in auto-waiting, cleaner async support, isolated browser contexts, and tools like codegen and the trace viewer. Selenium is still reasonable if you have a large existing test suite or need its wider ecosystem. Puppeteer is a solid option if your stack is Node.js, since Playwright's Python bindings are the main reason to pick it for Python work.

How do I scrape pages that require a login?

# One time: log in, then save cookies and local storage
context.storage_state(path="auth.json")

# Later runs: start already authenticated
context = browser.new_context(storage_state="auth.json")

Only scrape content you're authorized to access, and check the site's terms first. Sessions expire, so add a check that detects a logged-out page and refreshes the state.

Why does my scraper work locally but fail on a server?

The usual causes are missing system dependencies, a datacenter IP that gets flagged, and low memory. Install the browser with its OS dependencies using playwright install --with-deps chromium, or build from the official Playwright Python Docker image. If pages load but return blocks or CAPTCHAs, the problem is IP reputation and browser fingerprinting, not your code. Also check RAM, because each page can use 100 to 300 MB.

How do I avoid getting blocked while scraping?

Start with the basics: lower your concurrency, add delays between requests, block assets you don't need, and reuse sessions where it makes sense. Prefer an official or hidden API over rendering pages when one exists. If a site still blocks you, you're likely dealing with fingerprinting or IP reputation, which usually needs rotating proxies and more realistic browser profiles. Always respect robots.txt and the site's terms, and stop if a site clearly doesn't want automated access.

Python web scraping is a must-have skill for developers working with data! 🐍 From choosing the right tools to writing your first scraper, this guide covers the essentials with practical tips and code examples. 🚀 Check it out 👇

Alina Shah — Tue, 27 Oct 2026 17:28:04 +0000

Alina Shah for Dev Stack Community

Oct 27

Getting Started With Python Web Scraping: Tools, Tips & Code

#discuss #python #webscraping #programming

10 min read

Getting Started With Python Web Scraping: Tools, Tips & Code

Alina Shah — Tue, 27 Oct 2026 17:21:13 +0000

Have you ever wanted to collect product prices from multiple websites, track job listings, or gather article titles without copying everything manually?

That is where Python web scraping comes in. Instead of opening dozens of pages and transferring information into a spreadsheet, you can write a script to retrieve web pages, extract the information you need, and organize it into a usable format.

The tricky part is that websites don't all store information the same way. Some use straightforward HTML, while others load content with JavaScript, block automated requests, or change their page structure without warning.

In this guide, I will walk through essential Python web scraping tools, show working code examples, and explain how to build easier-to-maintain scrapers.

What Is Python Web Scraping?

Python web scraping is the process of using Python code to retrieve information from web pages and extract specific data from their content.

For example, suppose you want to collect:

Product names and prices from a permitted product listing page.
Job titles and company names from public job listings.
Article headlines and publication dates from a news archive.
Publicly available research data from a website.
Links and metadata from a directory of resources.

You could copy the information manually, but that becomes inefficient when you need hundreds of records or want to repeat the process regularly.

A scraper automates the workflow:

Send a request to a web page.
Receive the page content.
Parse the HTML.
Identify the elements containing the information.
Extract and clean the values.
Save the results to a file or database.

Python is a popular choice because its libraries make HTTP requests, HTML parsing, browser automation, and data processing relatively straightforward.

The Best Python Web Scraping Tools to Know

You do not need a complicated stack to start scraping. The tools you choose should depend on how the target website delivers its content.

1. Requests: Retrieve Web Pages

The Requests library sends HTTP requests and provides a convenient way to access responses.

It is a good starting point when the information you need is already present in the page's HTML.

Install it with:

pip install requests

A basic example:

import requests

url = "https://example.com"

response = requests.get(
    url,
    timeout=15
)

response.raise_for_status()

print(response.status_code)
print(response.text[:500])

The get() method requests the page. The timeout prevents the request from waiting indefinitely, while raise_for_status() raises an exception for unsuccessful HTTP responses.

For real projects, handle network errors and timeouts instead of assuming every request will succeed.

2. Beautiful Soup: Extract Information From HTML

A successful HTTP request does not automatically give you clean data. The response usually contains HTML with headings, links, paragraphs, navigation, and other elements.

Beautiful Soup helps you parse that HTML and select the elements you need.

Install it:

pip install beautifulsoup4

Let's extract the title and headings from a page:

import requests
from bs4 import BeautifulSoup

url = "https://example.com"

response = requests.get(
    url,
    timeout=15
)
response.raise_for_status()

soup = BeautifulSoup(
    response.text,
    "html.parser"
)

page_title = (
    soup.title.get_text(strip=True)
    if soup.title
    else "No title found"
)

print("Page title:", page_title)

for heading in soup.find_all(["h1", "h2"]):
    print(heading.get_text(" ", strip=True))

This example demonstrates two useful techniques: checking whether an element exists before accessing it and using get_text() to remove the HTML markup from the extracted content.

Beautiful Soup is especially useful for small and medium scraping tasks involving predictable HTML.

3. lxml: Parse HTML Efficiently

The lxml library is another option for parsing HTML and XML. It supports XPath expressions, which can be useful when you need to select elements using more detailed structural rules.

Install it with:

pip install lxml

You can use it directly with BeautifulSoup:

from bs4 import BeautifulSoup

html = """
<html>
  <body>
    <h1>Python Web Scraping</h1>
    <p class="summary">A beginner's guide</p>
  </body>
</html>
"""

soup = BeautifulSoup(html, "lxml")

heading = soup.find("h1")
summary = soup.select_one(".summary")

print(heading.get_text(strip=True))
print(summary.get_text(strip=True))

For many beginners, Beautiful Soup with Python's built-in HTML parser is enough. Consider lxml when you need XPath or want to evaluate parsing performance for a larger workload.

4. Scrapy: Build Larger Scraping Projects

Scrapy is a more complete framework for crawling websites and extracting data across multiple pages. It provides request scheduling, item processing, pipelines, and result exporting.

Install it:

pip install scrapy

Create a project:

scrapy startproject my_scraper

Scrapy becomes useful when a project needs to follow pagination, visit many links, manage retries, and process a substantial number of records.

For a single page, it may be unnecessary. For a recurring crawl with several stages, its built-in structure can save time compared with writing everything from scratch.

5. Playwright: Handle JavaScript-Rendered Pages

Some websites display their main content only after JavaScript runs. A simple Requests call may return an HTML shell without the data you expected.

Playwright automates a browser, allowing you to wait for page content and interact with elements when permitted.

Install it:

pip install playwright
playwright install chromium

Example:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    page.goto(
        "https://example.com",
        wait_until="domcontentloaded"
    )

    page_title = page.title()
    print(page_title)

    browser.close()

This example opens a page in a browser and retrieves its title. If you need content loaded later by JavaScript, wait for a specific, documented page element rather than adding an arbitrary long delay.

Browser automation consumes more resources than a basic HTTP request, so use it only when the target page actually requires it.

A Complete Python Web Scraping Example

Let's combine Requests and Beautiful Soup into a small scraper.

For this example, we will use Books to Scrape, a demonstration website intended for scraping practice. The script retrieves book titles and prices from its first catalog page.

Install the dependencies:

pip install requests beautifulsoup4

Then create a Python file named scrape_books.py:

import csv
import requests
from bs4 import BeautifulSoup

URL = "https://books.toscrape.com/catalogue/page-1.html"

HEADERS = {
    "User-Agent": "LearningScraper/1.0"
}


def scrape_books():
    response = requests.get(
        URL,
        headers=HEADERS,
        timeout=15
    )
    response.raise_for_status()

    soup = BeautifulSoup(
        response.text,
        "html.parser"
    )

    books = []

    for item in soup.select("article.product_pod"):
        title_element = item.select_one("h3 a")
        price_element = item.select_one(".price_color")

        if title_element is None or price_element is None:
            continue

        books.append({
            "title": title_element.get("title", "").strip(),
            "price": price_element.get_text(strip=True)
        })

    return books


def save_to_csv(books):
    with open(
        "books.csv",
        "w",
        newline="",
        encoding="utf-8"
    ) as file:
        writer = csv.DictWriter(
            file,
            fieldnames=["title", "price"]
        )
        writer.writeheader()
        writer.writerows(books)


if __name__ == "__main__":
    books = scrape_books()
    save_to_csv(books)

    print(f"Saved {len(books)} books to books.csv")

    for book in books[:5]:
        print(book)

Run the script:

python scrape_books.py

If the request succeeds and the page structure matches the selectors, the script creates a books.csv file containing the extracted titles and prices.

How the Code Works

Step 1: Request the page. Requests downloads the HTML from the catalog URL.

Step 2: Parse the HTML. Beautiful Soup converts the response into a structure that Python can search.

Step 3: Find book listings. The CSS selector article.product_pod targets each book card.

Step 4: Extract fields. The script retrieves the book title from the link's title attribute and the displayed price from the price element.

Step 5: Save the results. Python's built-in csv module writes the records to a spreadsheet-friendly file.

The script also checks whether the expected elements exist. This matters because a missing element should not crash the entire scraping process unnecessarily.

The example is deliberately limited to one page. A production scraper would need additional handling for pagination, unexpected page layouts, network failures, and website changes.

How to Choose the Right Python Scraping Tool

Use this quick guide when deciding which library to start with.

Tool	Best use case	Learning curve
Requests	Downloading HTML	Low
Beautiful Soup	Extracting text, links, and attributes	Low
lxml	HTML/XML parsing and XPath	Low to moderate
Scrapy	Multi-page crawling and structured scraping pipelines	Moderate
Playwright	Pages that require browser rendering or interaction	Moderate

For a first project, I would start with Requests and Beautiful Soup. They cover most straightforward scraping tasks without adding unnecessary complexity.

Move to Scrapy when the project grows into a crawler. Use Playwright when you confirm that the information depends on browser execution or interaction.

Practical Tips for More Reliable Web Scraping

A script that works once is not necessarily a reliable scraper. Small changes to the website, network conditions, or response format can break it.

1. Check the Website's Rules First

Before collecting data, review the website's terms, applicable policies, and robots.txt guidance. Prefer an official API or permitted data export when available.

Do not assume that publicly visible information is automatically free to collect or reuse. Respect access restrictions, copyright, privacy requirements, and rate limits. Do not bypass authentication, CAPTCHAs, or other technical controls without authorization.

2. Add Timeouts and Error Handling

Network requests can fail because of timeouts, DNS problems, server errors, or temporary connectivity issues.

A simple pattern is:

import requests

try:
    response = requests.get(
        "https://example.com",
        timeout=15
    )
    response.raise_for_status()

except requests.exceptions.Timeout:
    print("The request timed out.")

except requests.exceptions.HTTPError as error:
    print(f"HTTP error: {error}")

except requests.exceptions.RequestException as error:
    print(f"Request failed: {error}")

For recurring jobs, record errors in logs and decide which failures are safe to retry. Respect server-provided rate-limit information and avoid retrying indefinitely.

3. Use Specific CSS Selectors

Broad selectors can collect unrelated information. For example, selecting every paragraph may capture navigation labels, advertisements, and footer text.

Inspect the page's HTML and target the elements that correspond to the data you actually need.

When possible, use stable attributes or semantic elements. Avoid depending on automatically generated CSS class names that may change frequently.

4. Handle Missing and Inconsistent Data

Not every record will contain every field. Prices might be missing, publication dates might use different formats, or a page might have an unexpected layout.

Check for missing elements before extracting values. Normalize whitespace, dates, and numbers, and validate the resulting data before saving it.

If the output will feed a dashboard or database, add tests for the fields that must always be present.

5. Avoid Sending Too Many Requests

High request volumes can overload a website and cause your IP address to be blocked. Keep request rates reasonable, follow published limits, and cache responses when appropriate.

For a small sequential scraper, even a short delay between requests may help reduce unnecessary load:

import time

for url in permitted_urls:
    # Fetch and process the page here.
    time.sleep(1)

This is a basic pacing example, not a guarantee that one request per second is permitted. Follow the target site's actual requirements.

6. Keep Scraped Data Separate From Your Code

Save collected records to CSV, JSON, or a database instead of mixing data extraction with every downstream task.

This makes it easier to inspect the results, rerun analysis without downloading the pages again, and identify whether a problem occurred during retrieval, parsing, or storage.

For recurring projects, record when the data was collected and where it came from. This provides useful context when values change.

Common Python Web Scraping Problems and Fixes

The script returns an empty list: Check the page source and confirm that your CSS selectors match the actual HTML. The data might be loaded through JavaScript, or the page may have returned an error message instead of the expected content.
The request returns a 403 or 429 response: The server may be denying access or enforcing rate limits. Review the site's rules and reduce request frequency where appropriate. Do not attempt to bypass access controls.
The extracted text contains extra whitespace: Use methods such as get_text(" ", strip=True) and normalize the values before saving them.
The scraper stops when one element is missing: Check for None before accessing an element's attributes or methods, and log records that do not match the expected structure.
The scraper becomes slow: Avoid launching a browser for pages that work with HTTP requests. For larger crawls, consider Scrapy's request scheduling and concurrency controls, configured to respect the target website's limits.
The website changes its layout: Add tests that verify important selectors and expected fields. When a scraper runs regularly, monitor record counts and unexpected missing values so structural changes do not go unnoticed.

When Should You Use an API Instead of Web Scraping?

Web scraping is not always the best solution. If a website provides an official API, that is usually the first option to investigate.

An API typically provides structured responses, documented fields, and clearer expectations about access and rate limits. Scraping may be useful when the information is available only in HTML, and you are permitted to collect it.

Before building a scraper, ask three questions:

Is there an official API or downloadable dataset?
Am I permitted to access, collect, and use this information?
Can I maintain the scraper if the page structure changes?

If an API provides the data you need, it may be more reliable and easier to maintain than parsing HTML.

Conclusion

Python web scraping is a practical way to automate data collection, whether you are learning to extract a few headlines or building a recurring data pipeline. Requests and Beautiful Soup are excellent starting points for ordinary HTML pages, while lxml, Scrapy, and Playwright address more specialized needs.

The biggest lesson is to keep the first project simple. Choose one permitted page, inspect its HTML, extract a small set of fields, and save the results to CSV. Once that works, add error handling, validation, logging, and sensible request limits.

Reliable scraping is not just about getting data out of a page. It is about collecting the right information responsibly and building a script that keeps working when conditions change.

FAQs

Is Python good for web scraping?

Yes. Python has libraries for HTTP requests, HTML parsing, browser automation, crawling, and data processing. Beginners can start with Requests and Beautiful Soup, then use tools such as Scrapy or Playwright when their requirements become more complex.

Which Python library is best for web scraping?

Beautiful Soup is a good choice for extracting information from HTML, usually alongside Requests. Scrapy suits larger crawling projects, while Playwright is useful when content requires browser rendering or interaction.

Can Python scrape JavaScript websites?

Yes, but the right method depends on how the website delivers its content. If the data is available through a permitted underlying API or in the initial HTML, a normal HTTP request may be enough. If the page requires browser-side JavaScript, a tool such as Playwright can render it.

Is web scraping legal?

It depends on the website, the data, the access method, and the intended use. Review the site's terms, access rules, privacy obligations, and applicable laws. Public visibility alone does not establish permission to collect or reuse all content.

How do I save scraped data in Python?

You can save scraped records to CSV using Python's built-in csv module or to JSON using the json module. For larger or recurring projects, a database may be a better fit.

AI design-to-code tools are evolving fast, but not all deliver the same results. This comparison explores 11 tools, their key differences, and how they perform in real-world projects to help developers choose the right workflow.

Alina Shah — Tue, 27 Oct 2026 17:02:45 +0000

Dev Stack Hub

Oct 26

I Compared 11 Best AI Design-to-Code Tools for My Client Projects

#discuss #ai #design #nocode

13 min read

What's your favorite AI site builder right now?

Alina Shah — Tue, 20 Oct 2026 13:49:02 +0000

Dev Stack Hub

Oct 20

Top 13 Web Scraping Tools Every Developer Should Know

#discuss #webscraping #programming #productivity

13 min read

The article is a practical, code-based guide to making web scrapers reliable in production. It covers timeouts and bounded retries, data validation, restart-safe job state, idempotent storage, and monitoring, with Python examples and an FAQ. It’s original,

Alina Shah — Thu, 15 Oct 2026 17:57:21 +0000

Alina Shah

Oct 14

How I Built a Web Scraper That Doesn’t Break in Production

#discuss #webscraping #productivity #devdiscuss

11 min read

How I Built a Web Scraper That Doesn’t Break in Production

Alina Shah — Wed, 14 Oct 2026 18:57:00 +0000

The first version of my web scraper worked perfectly. It sent requests, extracted the data I needed, and saved everything to a database. I tested it on a few pages, checked the output, and assumed the difficult part was over.

Then I ran it against a larger dataset. Some pages returned errors, a few records were missing, the website changed its HTML structure, and the scraper quietly saved incomplete data.

The problem wasn't that I couldn't write a scraper. It was that I had built a script that worked under ideal conditions, not a system that could survive real-world conditions.

That changed how I approached web scraping. Instead of focusing only on extracting data quickly, I started designing for failures, retries, monitoring, and recovery.

Why Most Web Scrapers Break in Production

A scraper can look reliable during development because the test environment is usually predictable. You know which URL to request, the website is responding, and the page structure hasn't changed. Production is different.

Websites change their layouts, networks fail, requests get throttled, and individual pages return unexpected content. Even a successful HTTP response doesn't guarantee that the page contains the data your scraper expects.

These were the problems I needed to solve:

Unstable requests: Temporary network failures, timeouts, and server errors interrupted scraping jobs.
Rate limiting: Sending too many requests in a short period triggered HTTP 429 responses or temporary blocks.
Changing HTML: A class name or page structure changed, and extraction started returning empty values.
Incomplete records: Some pages loaded partially, or optional fields were missing.
Duplicate data: Restarting a failed job sometimes caused the same records to be saved more than once.
Silent failures: The script kept running, but the amount or quality of extracted data dropped without an obvious error.

I realized that fixing individual exceptions wasn't enough. I needed a scraper with clear failure-handling rules.

1. I Started With a Simple, Maintainable Architecture

My first improvement was separating the scraper into smaller components. Earlier, one function handled requests, parsing, validation, and database writes. That made debugging difficult because every failure could originate from several places.

I split the workflow into five stages:

URL discovery: Identify the pages that need to be processed.
Request handling: Fetch pages with timeouts, headers, and controlled retries.
Parsing: Extract the required fields from the response.
Validation: Check whether the extracted data is complete and usable.
Storage: Save validated records without creating duplicates.

This separation made failures easier to diagnose. If a page loaded successfully but returned no product title, I could investigate the parser rather than rewriting the request logic.

It also made testing easier. I could test the parser with saved HTML files without making a network request every time.

For a small project, I didn't need a complicated distributed system. A straightforward Python application with a few focused modules was enough. The important decision was keeping responsibilities separate so that one component could change without breaking the rest.

2. I Added Timeouts and Retries, but With Limits

One of the earliest lessons was that requests should never be allowed to wait indefinitely.

A server might respond slowly, a connection could stall, or a network request might fail temporarily. Without a timeout, a worker could remain stuck and prevent the job from finishing.

I used Python's requests library for straightforward HTTP requests and added explicit connection and read timeouts. I also introduced limited retries for temporary failures.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()

retry_policy = Retry(
    total=4,
    connect=4,
    read=4,
    status=4,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET"],
    respect_retry_after_header=True,
)

adapter = HTTPAdapter(max_retries=retry_policy)

session.mount("https://", adapter)
session.mount("http://", adapter)


def fetch_page(url):
    response = session.get(
        url,
        timeout=(5, 20),
        headers={"User-Agent": "ResearchScraper/1.0"}
    )
    response.raise_for_status()
    return response.text

The timeout tuple sets a connection timeout of five seconds and a read timeout of twenty seconds. The retry policy handles selected temporary errors, with backoff to avoid immediately repeating requests.

This was a useful starting point, but retries aren't a universal fix. A missing page, invalid URL, or persistent permission error usually won't become successful just because the scraper tries again.

I also treated HTTP 429 responses carefully. When a server provides a Retry-After header, the scraper should respect it. Repeatedly hitting the same endpoint after being rate-limited only makes the problem worse.

The lesson: retries should be bounded, selective, and observable. Infinite retries can turn a temporary failure into a job that never finishes.

3. I Stopped Treating Successful Responses as Valid Data

This was one of the most important changes.

Initially, I assumed that if a request returned a successful status code, the page was ready to parse. But websites can return a normal HTML page containing a maintenance message, an access-denied screen, an empty result, or a completely redesigned layout.

The request can succeed while the extraction fails.

I added validation rules for the fields that mattered to the project. For example, if a product page was expected to contain a title and product URL, those fields had to be present before the record could be stored.

from bs4 import BeautifulSoup


def parse_product(html, page_url):
    soup = BeautifulSoup(html, "html.parser")

    title_element = soup.select_one("h1.product-title")
    price_element = soup.select_one(".product-price")

    if title_element is None:
        raise ValueError(
            f"Required product title missing on {page_url}"
        )

    title = title_element.get_text(" ", strip=True)

    if not title:
        raise ValueError(
            f"Product title is empty on {page_url}"
        )

    price = (
        price_element.get_text(" ", strip=True)
        if price_element
        else None
    )

    return {
        "url": page_url,
        "title": title,
        "price": price,
    }

The selectors here are illustrative. Real selectors depend on the target website and should be verified against its current HTML.

Notice that I treated the title as required but the price as optional. That distinction matters. If every missing field causes the entire record to fail, a scraper can discard useful data unnecessarily. If no missing field triggers an error, broken extraction can go unnoticed.

I classified fields into three groups:

Required: The record isn't useful without them.
Optional: The record can still be stored if they're missing.
Derived: The scraper calculates them from other values, so they need their own validation.

I also tracked validation failures separately from request failures. That made it much easier to identify whether a website was unavailable or my extraction logic had become outdated.

4. I Designed for Website Changes

HTML scraping is fragile by nature. A site owner can change a class name, move a field into a different element, or redesign the page without warning.

I couldn't prevent those changes, so I made them easier to detect.

First, I avoided relying on long, deeply nested CSS selectors when a simpler selector was available. I also kept selectors in a small, identifiable part of the code rather than scattering them across the application.

Second, I saved representative HTML fixtures for automated parser tests. Whenever I changed the extraction logic, I ran the parser against those saved pages to confirm that titles, prices, links, and other fields were still extracted correctly.

Third, I monitored extraction quality. A scraper that normally extracts 95% of expected titles shouldn't suddenly drop to 20% without triggering an alert.

For example, I could track:

Percentage of pages fetched successfully
Percentage of pages producing valid records
Number of missing required fields
Number of records extracted per run
Changes in the frequency of parsing errors

A drop in extraction quality is often more useful as an early warning than a basic uptime check.

I also kept a small collection of known-good test pages. When a website changed, I could compare the new HTML with previous samples and update the parser deliberately instead of guessing at the cause.

5. I Made the Scraper Safe to Restart

A production job will eventually stop unexpectedly. The machine may restart, a database connection may fail, or a deployment may interrupt a running process.

I wanted to restart the scraper without manually figuring out which URLs had already been processed.

I added a record of each URL's processing state. A simplified workflow looked like this:

State	Meaning
Pending	The URL is waiting to be processed.
In progress	A worker is handling the URL.
Completed	Data was extracted, validated, and stored.
Retry scheduled	A temporary failure may be recoverable.
Failed	The URL exceeded its retry limit or needs investigation.

The scraper could resume from pending and retry-scheduled URLs instead of starting from the beginning.

I also made database writes idempotent. In practical terms, processing the same page twice shouldn't create two copies of the same record. A stable source ID or canonical URL, combined with a database uniqueness constraint and an upsert operation, can help enforce this.

For example, a product record could use its source URL as a unique key if that URL is stable and reliably identifies the product. If the website provides a durable product ID, that may be a better choice.

I didn't rely only on an in-memory list of completed URLs because that information disappears when the process stops. Persistent job state made recovery much more predictable.

For larger scraping jobs, I would also add a lease or expiration time to the in-progress state. That allows the system to recover URLs that were assigned to a worker that crashed before finishing.

6. I Controlled Concurrency Instead of Sending Requests as Fast as Possible

It is tempting to increase the number of workers and send as many requests as possible. That can improve throughput, but excessive concurrency also increases failures, rate limiting, and pressure on the target server.

I treated concurrency as a configurable limit rather than an unlimited performance setting.

My approach was to:

Set conservative per-domain request limits.
Add delays or rate limiting where appropriate.
Reduce request pressure after repeated 429 responses.
Use exponential backoff for temporary failures.
Avoid launching duplicate jobs for the same URLs.
Increase concurrency only after measuring the effect.

The appropriate limit depends on the target site's rules, response times, and the scale of the project. There isn't one universal request-per-second value that works everywhere.

I also checked the site's terms and applicable crawling policies before automating requests. I preferred official APIs or bulk data exports when they provided the information I needed. I didn't design the scraper to bypass authentication, CAPTCHAs, or access controls.

The goal wasn't to extract the maximum number of pages in the shortest time. It was to finish reliably without creating unnecessary load or getting the job blocked.

7. I Added Logging and Monitoring That Answered Useful Questions

A scraper can fail in ways that are difficult to reproduce. Without useful logs, debugging becomes a guessing game.

I moved beyond generic messages such as Request failed and recorded enough context to identify the problem:

URL or internal job ID
HTTP status code
Attempt number
Request duration
Error category
Parsing or validation result
Number of records stored

I avoided logging credentials, session cookies, authorization headers, or other sensitive information.

I also added run-level metrics. At the end of each run, I wanted to know how many pages were discovered, attempted, completed, retried, skipped, and permanently failed.

That helped answer questions such as:

Did the target website become unavailable?
Did the scraper encounter a rate limit?
Did the HTML change?
Did the parser return empty records?
Did the database reject writes?
Did the job finish with unusually little data?

I used alerts for meaningful changes rather than every isolated error. A single failed page may not matter, but a sudden spike in failed requests or a major drop in valid records deserves attention.

Monitoring also gave me a baseline. Once I understood the scraper's normal success rate and execution time, unusual behavior became much easier to spot.

8. I Tested Failure Scenarios, Not Just the Happy Path

Testing a scraper with one working page doesn't tell you much about how it behaves in production.

I started testing cases that could realistically break it:

A request times out.
The server returns HTTP 429 or 503.
A page returns HTML without the expected title.
An optional field is missing.
The same URL is processed twice.
The database becomes temporarily unavailable.
A job stops halfway through a batch.
A malformed page contains unexpected characters or empty values.

I separated parser tests from network tests. Parser tests used saved HTML, which made them quick and deterministic. Network tests checked the request layer separately, with controlled timeouts and mocked responses where appropriate.

I also tested recovery. If a run stopped after processing half the URLs, could it resume without losing work or creating duplicates? If the parser started failing across many pages, would the monitoring reveal the problem?

Those tests were more valuable than simply confirming that the scraper worked once.

The Architecture I Would Use for a Reliable Scraper

For a small or medium-sized scraping project, I would start with this design:

URL Source
    |
    v
Job Queue or URL Table
    |
    v
Request Layer
    |
    v
HTML Parser
    |
    v
Data Validation
    |
    v
Database / Data Store
    |
    v
Logs, Metrics, and Alerts

Each stage has a clear responsibility. The request layer handles network behavior, the parser extracts fields, validation checks data quality, and storage manages persistence and duplicates.

For a small project, a single Python process and a database-backed URL table may be sufficient. For a larger workload, I might introduce a queue, multiple workers, a separate monitoring service, and a dedicated scheduler.

I wouldn't add distributed infrastructure just because it sounds production-ready. Extra components create their own operational problems. The architecture should match the volume of data, acceptable failure rate, recovery requirements, and maintenance budget.

What I Learned From Building It

The biggest lesson was that a reliable web scraper is not defined by how quickly it extracts a page. It's defined by what happens when the page is unavailable, the markup changes, the database rejects a write, or the job stops unexpectedly.

If I were starting another scraper today, I would prioritize these five things from the beginning:

Explicit timeouts and bounded retries to handle temporary network problems.
Data validation to catch broken extraction before it reaches the database.
Persistent job state to resume interrupted work safely.
Idempotent storage to prevent duplicate records during retries and restarts.
Monitoring and parser tests to detect changes before they cause widespread data loss.

I would add more workers or more complex infrastructure only after the basic system was reliable.

A scraper doesn't need to be complicated to be production-ready. It needs predictable behavior, clear failure states, useful logs, and a way to recover when things go wrong.

Conclusion

Building a web scraper that survives production taught me to think beyond requests and selectors. The real work was handling uncertainty: websites that change, servers that fail, data that arrives incomplete, and jobs that stop halfway through. By separating the request, parsing, validation, and storage layers, adding bounded retries, tracking progress, and monitoring data quality, I could make the scraper much easier to maintain and recover.

The goal isn't to build a scraper that never fails. That's unrealistic. The goal is to build one that detects failures, avoids corrupting data, recovers safely, and tells you when something needs attention. That's the difference between a script that works during a demo and a scraper you can trust in production.

FAQs

1. How do you make a web scraper reliable in production?

Use request timeouts, bounded retries, rate limiting, data validation, persistent job state, idempotent database writes, logging, and automated tests. Monitor extraction quality as well as request success so that structural changes don't silently corrupt your dataset.

2. Which Python libraries are best for building a web scraper?

requests is a good option for straightforward HTTP requests, while Beautiful Soup is useful for parsing HTML. Scrapy is designed for larger crawling workflows with built-in scheduling and concurrency controls. Playwright can handle pages that depend on browser-side JavaScript when ordinary HTTP requests aren't sufficient.

3. How do you handle website layout changes in a scraper?

Keep selectors maintainable, test parsers against saved HTML samples, validate required fields, and monitor extraction success rates. Alert when important fields suddenly become missing or the number of valid records drops unexpectedly.

4. How can you prevent duplicate data when a scraper retries?

Use a stable unique identifier, enforce uniqueness in the database, and use upsert operations instead of blindly inserting every result. Keep processing state in persistent storage so the scraper can resume after an interruption.

5. Should a web scraper use multiple threads or workers?

It depends on the workload. Concurrency can improve throughput for network-bound scraping, but too many simultaneous requests may trigger rate limits or overload the target site. Start with conservative limits, respect the site's policies, and increase concurrency only after measuring reliability and performance.

This is actually such a fun idea 😂 I definitely want to see what kind of setups people are going to share.

Alina Shah — Thu, 08 Oct 2026 18:49:37 +0000

Almost everyone uses a dark theme editor

Dev Stack Hub

Oct 8

Show Me Your Dev Setup — I'll Roast It 🖥️🔥

#discuss #devdiscuss #showdev #community

2 min read