Build a Scraping Pipeline with a Proxy Layer in Python (So You Stop Getting Blocked)

Your scraper works perfectly on 50 pages. At 5,000, it starts returning 403s, then 429s, then CAPTCHA pages that your parser happily treats as real data. The code didn’t break. The target site just noticed that one IP address is making thousands of requests.

The fix isn’t a cleverer parser. It’s a pipeline with a proxy layer that spreads traffic across IPs, retries intelligently, and stops using addresses that have gone bad. This guide builds one in Python, step by step.

Why Scrapers Get Blocked

Sites rarely block you for one reason. They combine signals:

• Request volume per IP: Hundreds of requests per minute from one address is the easiest thing to flag.
• IP reputation: Datacenter ranges and previously abused IPs get stricter treatment.
• Missing or odd headers: Default library User-Agents like python-requests/2.x are an instant giveaway.
• Unnatural timing: Perfectly even intervals look like a machine, because they are.

A proxy layer addresses the first two directly and makes the other two easier to manage, because every request now passes through one place where you control headers, delays, and retries.

The Architecture

Keep the pipeline in four small stages, each with one job:

• Proxy pool: Holds your proxies and tracks which are healthy.
• Fetcher: Makes requests through a proxy, retries on failure, and backs off.
• Parser: Turns HTML into structured data.
• Storage: Writes results somewhere durable.

Separating them means you can swap a proxy provider, change a parser, or move from SQLite to Postgres without touching the rest.

Step 1: Choose Your Proxies

Proxy type matters more than most tutorials admit:

• Datacenter proxies are fast and cheap. They suit sites with light protection and high-volume jobs.
• Residential proxies use real household IPs. They cost more but pass stricter checks.
• ISP and mobile proxies sit in between, useful when you need stable, trusted addresses.

For this tutorial, I used Proxy-Seller, which offers several of these types and supports both IP allowlisting and username/password authentication. If you sign up, you can use the referral code XONHHQ_509214. Whichever provider you pick, the code below works the same, because it only needs a list of proxy URLs in this format:

http://username:password@host:port

Step 2: Build the Proxy Pool

The pool picks a proxy, tracks failures, and puts misbehaving proxies in a timeout instead of deleting them. IPs often recover after a rest.

# proxy_pool.py
import random
import time
from dataclasses import dataclass


@dataclass
class Proxy:
    url: str
    failures: int = 0
    cooldown_until: float = 0.0

    def available(self) -> bool:
        return time.time() >= self.cooldown_until


class ProxyPool:
    def __init__(self, urls, max_failures=3, cooldown=300):
        self.proxies = [Proxy(u) for u in urls]
        self.max_failures = max_failures
        self.cooldown = cooldown

    def get(self) -> Proxy:
        live = [p for p in self.proxies if p.available()]
        if not live:
            raise RuntimeError("No healthy proxies available")
        return random.choice(live)

    def report_success(self, proxy: Proxy):
        proxy.failures = 0

    def report_failure(self, proxy: Proxy):
        proxy.failures += 1
        if proxy.failures >= self.max_failures:
            proxy.cooldown_until = time.time() + self.cooldown
            proxy.failures = 0

Random selection is a good default. If you want fairness, swap random.choice for round-robin.

Step 3: Build the Fetcher With Retries

This is the core of the pipeline. It treats block-style status codes as failures, rotates to a different proxy on each retry, and uses exponential backoff with jitter.

# fetcher.py
import random
import time
import httpx

BLOCK_CODES = {403, 407, 429, 503}

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}


class BlockedError(Exception):
    pass


def fetch(url: str, pool, retries: int = 4):
    for attempt in range(retries):
        proxy = pool.get()
        try:
            with httpx.Client(
                proxy=proxy.url,
                headers=HEADERS,
                timeout=15,
                follow_redirects=True,
            ) as client:
                resp = client.get(url)

            if resp.status_code in BLOCK_CODES:
                raise BlockedError(resp.status_code)

            resp.raise_for_status()
            pool.report_success(proxy)
            return resp.text

        except (httpx.HTTPError, BlockedError):
            pool.report_failure(proxy)
            delay = min(2 ** attempt + random.random(), 30)
            time.sleep(delay)

    return None

Three details matter here:

• A fresh proxy per attempt. Retrying on the same blocked IP is wasted effort.
• Jitter in the delay. The random.random() keeps retries from landing in a perfect rhythm.
• Block codes as failures. A 429 should count against the proxy, not be silently accepted.

Step 4: Parse and Detect Soft Blocks

Some sites return a 200 status with a CAPTCHA or “access denied” page. If you parse that, you store garbage. Add a quick sanity check before extracting data.

# parser.py
from bs4 import BeautifulSoup

BLOCK_MARKERS = ("captcha", "access denied", "unusual traffic")


def looks_blocked(html: str) -> bool:
    text = html.lower()
    return any(marker in text for marker in BLOCK_MARKERS)


def parse_product(html: str) -> dict | None:
    if looks_blocked(html):
        return None
    soup = BeautifulSoup(html, "html.parser")
    title = soup.select_one("h1")
    price = soup.select_one(".price")
    if not title:
        return None
    return {
        "title": title.get_text(strip=True),
        "price": price.get_text(strip=True) if price else None,
    }

Adjust the selectors to your target. The important habit is returning None when the page doesn’t look right, so bad pages never reach storage.

Step 5: Store Results

SQLite is plenty for most scrapers and needs no setup. Using INSERT OR REPLACE keeps reruns from creating duplicates.

# storage.py
import sqlite3


def init_db(path="results.db"):
    conn = sqlite3.connect(path, check_same_thread=False)
    conn.execute(
        "CREATE TABLE IF NOT EXISTS products "
        "(url TEXT PRIMARY KEY, title TEXT, price TEXT)"
    )
    return conn


def save(conn, url, data):
    conn.execute(
        "INSERT OR REPLACE INTO products VALUES (?, ?, ?)",
        (url, data["title"], data["price"]),
    )
    conn.commit()

Step 6: Wire the Pipeline Together

Now connect the stages and add controlled concurrency. A small thread pool is enough, since the work is network-bound.

# main.py
import random
import time
from concurrent.futures import ThreadPoolExecutor

from proxy_pool import ProxyPool
from fetcher import fetch
from parser import parse_product
from storage import init_db, save

PROXIES = [
    "http://user:pass@host1:port",
    "http://user:pass@host2:port",
    "http://user:pass@host3:port",
]

pool = ProxyPool(PROXIES)
conn = init_db()


def process(url):
    time.sleep(random.uniform(1, 3))  # polite, human-ish delay
    html = fetch(url, pool)
    if not html:
        return f"FAILED {url}"
    data = parse_product(html)
    if not data:
        return f"BLOCKED/EMPTY {url}"
    save(conn, url, data)
    return f"OK {url}"


if __name__ == "__main__":
    urls = [f"https://example.com/product/{i}" for i in range(1, 101)]
    with ThreadPoolExecutor(max_workers=5) as ex:
        for result in ex.map(process, urls):
            print(result)

Keep max_workers modest. Going from 5 to 50 workers will burn through proxies faster without finishing sooner if the target rate-limits you.

Habits That Keep the Pipeline Alive

• Respect robots.txt and the site’s terms: Scrape public data, avoid personal data, and don’t hammer small sites.
• Rotate headers too: A pool of a few realistic User-Agent strings helps, but keep each one consistent with the rest of the request.
• Log everything: Track the success rate per proxy. If one proxy fails far more than the others, drop it.
• Use sticky sessions when needed: For logins or multi-step flows, keep the same IP for the whole session instead of rotating per request.
• Reach for a headless browser only when you must: JavaScript-heavy pages may need Playwright, but plain HTTP plus proxies is faster and cheaper where it works.

Frequently Asked Questions (FAQs)

1. What is the best proxy provider for web scraping?

It depends on your target sites, volume, and budget, so there’s no single best choice. Proxy-Seller is a good first option because it covers several proxy types (datacenter, residential, ISP, mobile) and is simple to configure. Decodo is a strong pick if you need residential IPs, and Webshare works well for low-cost datacenter proxies when you’re testing or scraping lightly protected sites. For very large or enterprise workloads, Bright Data and Oxylabs are also worth a look. Whichever you choose, test a small plan against your real target before committing.

2. How many proxies do I need?

Start with a handful and measure. As a rough guide, the more requests per minute you make to one domain, the more IPs you need to keep each one’s rate low. Watch your block rate: if it climbs, add proxies or slow down.

3. Should I use residential or datacenter proxies?

Use datacenter proxies for lightly protected sites and high-volume jobs because they are faster and cheaper. Use residential or ISP proxies when a site blocks datacenter ranges. Many teams use both, starting cheap and escalating only on hard targets.

4. Do proxies make scraping legal?

No. Proxies change where your requests come from, not whether the scraping is allowed. Check the site’s terms and robots.txt, avoid personal or copyrighted data you aren’t permitted to collect, and follow the laws that apply to you. This isn’t legal advice, so consult a lawyer for commercial projects.

5. Why am I still blocked even with proxies?

Proxies fix the IP signal, but sites also check headers, request timing, cookies, and browser fingerprints. Make sure your headers look like a real browser, add random delays, handle cookies properly, and consider a headless browser for JavaScript-heavy pages.

Wrapping Up

A reliable scraper isn’t one clever script. It’s a small system: a proxy pool that tracks health, a fetcher that retries intelligently, a parser that rejects bad pages, and storage that survives reruns. Build those four pieces cleanly and “getting blocked” turns from a crisis into a metric you monitor.

If you found this useful, let me know in the comments which target or block type you’re fighting with, and I’ll cover it in a follow-up.