How I Built a Web Scraper That Doesn’t Break in Production

The first version of my web scraper worked perfectly. It sent requests, extracted the data I needed, and saved everything to a database. I tested it on a few pages, checked the output, and assumed the difficult part was over.

Then I ran it against a larger dataset. Some pages returned errors, a few records were missing, the website changed its HTML structure, and the scraper quietly saved incomplete data.

The problem wasn't that I couldn't write a scraper. It was that I had built a script that worked under ideal conditions, not a system that could survive real-world conditions.

That changed how I approached web scraping. Instead of focusing only on extracting data quickly, I started designing for failures, retries, monitoring, and recovery.

Why Most Web Scrapers Break in Production

A scraper can look reliable during development because the test environment is usually predictable. You know which URL to request, the website is responding, and the page structure hasn't changed. Production is different.

Websites change their layouts, networks fail, requests get throttled, and individual pages return unexpected content. Even a successful HTTP response doesn't guarantee that the page contains the data your scraper expects.

These were the problems I needed to solve:

• Unstable requests: Temporary network failures, timeouts, and server errors interrupted scraping jobs.
• Rate limiting: Sending too many requests in a short period triggered HTTP 429 responses or temporary blocks.
• Changing HTML: A class name or page structure changed, and extraction started returning empty values.
• Incomplete records: Some pages loaded partially, or optional fields were missing.
• Duplicate data: Restarting a failed job sometimes caused the same records to be saved more than once.
• Silent failures: The script kept running, but the amount or quality of extracted data dropped without an obvious error.

I realized that fixing individual exceptions wasn't enough. I needed a scraper with clear failure-handling rules.

1. I Started With a Simple, Maintainable Architecture

My first improvement was separating the scraper into smaller components. Earlier, one function handled requests, parsing, validation, and database writes. That made debugging difficult because every failure could originate from several places.

I split the workflow into five stages:

• URL discovery: Identify the pages that need to be processed.
• Request handling: Fetch pages with timeouts, headers, and controlled retries.
• Parsing: Extract the required fields from the response.
• Validation: Check whether the extracted data is complete and usable.
• Storage: Save validated records without creating duplicates.

This separation made failures easier to diagnose. If a page loaded successfully but returned no product title, I could investigate the parser rather than rewriting the request logic.

It also made testing easier. I could test the parser with saved HTML files without making a network request every time.

For a small project, I didn't need a complicated distributed system. A straightforward Python application with a few focused modules was enough. The important decision was keeping responsibilities separate so that one component could change without breaking the rest.

2. I Added Timeouts and Retries, but With Limits

One of the earliest lessons was that requests should never be allowed to wait indefinitely.

A server might respond slowly, a connection could stall, or a network request might fail temporarily. Without a timeout, a worker could remain stuck and prevent the job from finishing.

I used Python's requests library for straightforward HTTP requests and added explicit connection and read timeouts. I also introduced limited retries for temporary failures.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()

retry_policy = Retry(
    total=4,
    connect=4,
    read=4,
    status=4,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET"],
    respect_retry_after_header=True,
)

adapter = HTTPAdapter(max_retries=retry_policy)

session.mount("https://", adapter)
session.mount("http://", adapter)


def fetch_page(url):
    response = session.get(
        url,
        timeout=(5, 20),
        headers={"User-Agent": "ResearchScraper/1.0"}
    )
    response.raise_for_status()
    return response.text

The timeout tuple sets a connection timeout of five seconds and a read timeout of twenty seconds. The retry policy handles selected temporary errors, with backoff to avoid immediately repeating requests.

This was a useful starting point, but retries aren't a universal fix. A missing page, invalid URL, or persistent permission error usually won't become successful just because the scraper tries again.

I also treated HTTP 429 responses carefully. When a server provides a Retry-After header, the scraper should respect it. Repeatedly hitting the same endpoint after being rate-limited only makes the problem worse.

The lesson: retries should be bounded, selective, and observable. Infinite retries can turn a temporary failure into a job that never finishes.

3. I Stopped Treating Successful Responses as Valid Data

This was one of the most important changes.

Initially, I assumed that if a request returned a successful status code, the page was ready to parse. But websites can return a normal HTML page containing a maintenance message, an access-denied screen, an empty result, or a completely redesigned layout.

The request can succeed while the extraction fails.

I added validation rules for the fields that mattered to the project. For example, if a product page was expected to contain a title and product URL, those fields had to be present before the record could be stored.

from bs4 import BeautifulSoup


def parse_product(html, page_url):
    soup = BeautifulSoup(html, "html.parser")

    title_element = soup.select_one("h1.product-title")
    price_element = soup.select_one(".product-price")

    if title_element is None:
        raise ValueError(
            f"Required product title missing on {page_url}"
        )

    title = title_element.get_text(" ", strip=True)

    if not title:
        raise ValueError(
            f"Product title is empty on {page_url}"
        )

    price = (
        price_element.get_text(" ", strip=True)
        if price_element
        else None
    )

    return {
        "url": page_url,
        "title": title,
        "price": price,
    }

The selectors here are illustrative. Real selectors depend on the target website and should be verified against its current HTML.

Notice that I treated the title as required but the price as optional. That distinction matters. If every missing field causes the entire record to fail, a scraper can discard useful data unnecessarily. If no missing field triggers an error, broken extraction can go unnoticed.

I classified fields into three groups:

• Required: The record isn't useful without them.
• Optional: The record can still be stored if they're missing.
• Derived: The scraper calculates them from other values, so they need their own validation.

I also tracked validation failures separately from request failures. That made it much easier to identify whether a website was unavailable or my extraction logic had become outdated.

4. I Designed for Website Changes

HTML scraping is fragile by nature. A site owner can change a class name, move a field into a different element, or redesign the page without warning.

I couldn't prevent those changes, so I made them easier to detect.

First, I avoided relying on long, deeply nested CSS selectors when a simpler selector was available. I also kept selectors in a small, identifiable part of the code rather than scattering them across the application.

Second, I saved representative HTML fixtures for automated parser tests. Whenever I changed the extraction logic, I ran the parser against those saved pages to confirm that titles, prices, links, and other fields were still extracted correctly.

Third, I monitored extraction quality. A scraper that normally extracts 95% of expected titles shouldn't suddenly drop to 20% without triggering an alert.

For example, I could track:

• Percentage of pages fetched successfully
• Percentage of pages producing valid records
• Number of missing required fields
• Number of records extracted per run
• Changes in the frequency of parsing errors

A drop in extraction quality is often more useful as an early warning than a basic uptime check.

I also kept a small collection of known-good test pages. When a website changed, I could compare the new HTML with previous samples and update the parser deliberately instead of guessing at the cause.

5. I Made the Scraper Safe to Restart

A production job will eventually stop unexpectedly. The machine may restart, a database connection may fail, or a deployment may interrupt a running process.

I wanted to restart the scraper without manually figuring out which URLs had already been processed.

I added a record of each URL's processing state. A simplified workflow looked like this:

State	Meaning
Pending	The URL is waiting to be processed.
In progress	A worker is handling the URL.
Completed	Data was extracted, validated, and stored.
Retry scheduled	A temporary failure may be recoverable.
Failed	The URL exceeded its retry limit or needs investigation.

The scraper could resume from pending and retry-scheduled URLs instead of starting from the beginning.

I also made database writes idempotent. In practical terms, processing the same page twice shouldn't create two copies of the same record. A stable source ID or canonical URL, combined with a database uniqueness constraint and an upsert operation, can help enforce this.

For example, a product record could use its source URL as a unique key if that URL is stable and reliably identifies the product. If the website provides a durable product ID, that may be a better choice.

I didn't rely only on an in-memory list of completed URLs because that information disappears when the process stops. Persistent job state made recovery much more predictable.

For larger scraping jobs, I would also add a lease or expiration time to the in-progress state. That allows the system to recover URLs that were assigned to a worker that crashed before finishing.

6. I Controlled Concurrency Instead of Sending Requests as Fast as Possible

It is tempting to increase the number of workers and send as many requests as possible. That can improve throughput, but excessive concurrency also increases failures, rate limiting, and pressure on the target server.

I treated concurrency as a configurable limit rather than an unlimited performance setting.

My approach was to:

• Set conservative per-domain request limits.
• Add delays or rate limiting where appropriate.
• Reduce request pressure after repeated 429 responses.
• Use exponential backoff for temporary failures.
• Avoid launching duplicate jobs for the same URLs.
• Increase concurrency only after measuring the effect.

The appropriate limit depends on the target site's rules, response times, and the scale of the project. There isn't one universal request-per-second value that works everywhere.

I also checked the site's terms and applicable crawling policies before automating requests. I preferred official APIs or bulk data exports when they provided the information I needed. I didn't design the scraper to bypass authentication, CAPTCHAs, or access controls.

The goal wasn't to extract the maximum number of pages in the shortest time. It was to finish reliably without creating unnecessary load or getting the job blocked.

7. I Added Logging and Monitoring That Answered Useful Questions

A scraper can fail in ways that are difficult to reproduce. Without useful logs, debugging becomes a guessing game.

I moved beyond generic messages such as Request failed and recorded enough context to identify the problem:

• URL or internal job ID
• HTTP status code
• Attempt number
• Request duration
• Error category
• Parsing or validation result
• Number of records stored

I avoided logging credentials, session cookies, authorization headers, or other sensitive information.

I also added run-level metrics. At the end of each run, I wanted to know how many pages were discovered, attempted, completed, retried, skipped, and permanently failed.

That helped answer questions such as:

• Did the target website become unavailable?
• Did the scraper encounter a rate limit?
• Did the HTML change?
• Did the parser return empty records?
• Did the database reject writes?
• Did the job finish with unusually little data?

I used alerts for meaningful changes rather than every isolated error. A single failed page may not matter, but a sudden spike in failed requests or a major drop in valid records deserves attention.

Monitoring also gave me a baseline. Once I understood the scraper's normal success rate and execution time, unusual behavior became much easier to spot.

8. I Tested Failure Scenarios, Not Just the Happy Path

Testing a scraper with one working page doesn't tell you much about how it behaves in production.

I started testing cases that could realistically break it:

• A request times out.
• The server returns HTTP 429 or 503.
• A page returns HTML without the expected title.
• An optional field is missing.
• The same URL is processed twice.
• The database becomes temporarily unavailable.
• A job stops halfway through a batch.
• A malformed page contains unexpected characters or empty values.

I separated parser tests from network tests. Parser tests used saved HTML, which made them quick and deterministic. Network tests checked the request layer separately, with controlled timeouts and mocked responses where appropriate.

I also tested recovery. If a run stopped after processing half the URLs, could it resume without losing work or creating duplicates? If the parser started failing across many pages, would the monitoring reveal the problem?

Those tests were more valuable than simply confirming that the scraper worked once.

The Architecture I Would Use for a Reliable Scraper

For a small or medium-sized scraping project, I would start with this design:

URL Source
    |
    v
Job Queue or URL Table
    |
    v
Request Layer
    |
    v
HTML Parser
    |
    v
Data Validation
    |
    v
Database / Data Store
    |
    v
Logs, Metrics, and Alerts

Each stage has a clear responsibility. The request layer handles network behavior, the parser extracts fields, validation checks data quality, and storage manages persistence and duplicates.

For a small project, a single Python process and a database-backed URL table may be sufficient. For a larger workload, I might introduce a queue, multiple workers, a separate monitoring service, and a dedicated scheduler.

I wouldn't add distributed infrastructure just because it sounds production-ready. Extra components create their own operational problems. The architecture should match the volume of data, acceptable failure rate, recovery requirements, and maintenance budget.

What I Learned From Building It

The biggest lesson was that a reliable web scraper is not defined by how quickly it extracts a page. It's defined by what happens when the page is unavailable, the markup changes, the database rejects a write, or the job stops unexpectedly.

If I were starting another scraper today, I would prioritize these five things from the beginning:

1. Explicit timeouts and bounded retries to handle temporary network problems.
2. Data validation to catch broken extraction before it reaches the database.
3. Persistent job state to resume interrupted work safely.
4. Idempotent storage to prevent duplicate records during retries and restarts.
5. Monitoring and parser tests to detect changes before they cause widespread data loss.

I would add more workers or more complex infrastructure only after the basic system was reliable.

A scraper doesn't need to be complicated to be production-ready. It needs predictable behavior, clear failure states, useful logs, and a way to recover when things go wrong.

Conclusion

Building a web scraper that survives production taught me to think beyond requests and selectors. The real work was handling uncertainty: websites that change, servers that fail, data that arrives incomplete, and jobs that stop halfway through. By separating the request, parsing, validation, and storage layers, adding bounded retries, tracking progress, and monitoring data quality, I could make the scraper much easier to maintain and recover.

The goal isn't to build a scraper that never fails. That's unrealistic. The goal is to build one that detects failures, avoids corrupting data, recovers safely, and tells you when something needs attention. That's the difference between a script that works during a demo and a scraper you can trust in production.

FAQs

1. How do you make a web scraper reliable in production?

Use request timeouts, bounded retries, rate limiting, data validation, persistent job state, idempotent database writes, logging, and automated tests. Monitor extraction quality as well as request success so that structural changes don't silently corrupt your dataset.

2. Which Python libraries are best for building a web scraper?

requests is a good option for straightforward HTTP requests, while Beautiful Soup is useful for parsing HTML. Scrapy is designed for larger crawling workflows with built-in scheduling and concurrency controls. Playwright can handle pages that depend on browser-side JavaScript when ordinary HTTP requests aren't sufficient.

3. How do you handle website layout changes in a scraper?

Keep selectors maintainable, test parsers against saved HTML samples, validate required fields, and monitor extraction success rates. Alert when important fields suddenly become missing or the number of valid records drops unexpectedly.

4. How can you prevent duplicate data when a scraper retries?

Use a stable unique identifier, enforce uniqueness in the database, and use upsert operations instead of blindly inserting every result. Keep processing state in persistent storage so the scraper can resume after an interruption.

5. Should a web scraper use multiple threads or workers?

It depends on the workload. Concurrency can improve throughput for network-bound scraping, but too many simultaneous requests may trigger rate limits or overload the target site. Start with conservative limits, respect the site's policies, and increase concurrency only after measuring reliability and performance.