Building a Resilient Scraping Queue: Failure Isolation, Priority Lanes, and Backoff That Actually Works

A few years ago I watched a scraping platform eat itself over the course of about forty minutes. One mid-sized retailer in our crawl list — call it shop-43.example.com — deployed a broken build and started hanging on every request, never returning, never erroring, just holding connections open until the 30-second timeout. Our queue didn't crash. Nothing crashed. It just… stopped delivering results. By the time someone noticed, the fresh-data lag on the other 400 domains in our fleet had gone from minutes to hours, all because a few thousand jobs for one dying site were squatting in front of everything else.

The postmortem was embarrassing in hindsight: the fix we shipped was to raise the retry count from 3 to 5. That made things worse — more attempts per dead job, more worker time burned, longer starvation for everyone else. That day taught me the thesis that this whole post hangs on: resilience in a scraping system doesn't come from retrying harder. It comes from shrinking the blast radius of each failure to the smallest thing that can fail independently — a domain, then a job — and letting everything outside that radius keep flowing.

Let's build that queue. Redis and redis-py, no frameworks, everything runnable.

Why the Naive Retry Loop Collapses

Every scraping tutorial contains some version of this:

for url in urls:
    for attempt in range(3):
        try:
            fetch(url)
            break
        except Exception:
            time.sleep(2 ** attempt)

This is fine for a hundred URLs on a good day. Under real-world failure distributions it falls apart, for reasons that compound:

Failure mode	What the naive loop does	What actually happens
One site hangs (timeout, no error)	Occupies a worker per attempt	Head-of-line blocking: jobs behind it starve
One site hard-fails (5xx / anti-bot)	Retries 3× per job	Multiplies load on a site that's already upset
Outage ends, 10k jobs retry at once	All sleep the same duration	Thundering herd, instant re-outage
Permanent failure (404, delisted page)	Still retries 3×	Wasted spend, every time, forever

The core mistake is treating "the URL list" as one failure domain. It's N failure domains pretending to be one. Site A being down carries zero information about site B, so a global retry budget or a global worker pool guarantees that A's problems become B's problems.

Three fixes, in order of impact: priority lanes so urgency isn't a FIFO accident, per-domain circuit breakers so a dying site can't eat your fleet, and jittered backoff with a dead-letter exit so retries are cheap, spread out, and bounded.

The Shape of the Queue

Design in one paragraph: jobs are JSON blobs in Redis. Each priority lane is a sorted set (ZSET) where the score is the timestamp when the job becomes eligible — this gives you priority and delayed retries in one primitive. Circuit breaker state lives in small Redis keys per domain. After N consecutive failures, the domain trips open; jobs for it are re-queued with a far-future score instead of being attempted at all. Jobs that exhaust their retry budget land in a dead-letter sorted set for human (or script) review.

One decision worth defending before the code: retry state lives in the job payload, not in a side table. When a worker pops a job, everything it needs to decide "how many times has this been tried, when should it run again" is inside the job itself. The alternative — a job_id -> attempts hash on the side — means every code path that touches the job (requeue, dead-letter, manual replay, crash recovery) has to remember to keep the side table in sync, and the first time it doesn't, you get a job that silently retries forever or never retries at all. Self-contained payloads are harder to corrupt. The only state I keep outside the job is the circuit breaker, because that's domain state, not job state — it belongs to a different lifetime.

Priority Lanes: One ZSET, Score = Eligibility

Scores in a ZSET are just floats. Encode eligible_at as the score and you get a delay queue for free; encode lane priority by offsetting the score at enqueue time.

import json
import time
import uuid
from urllib.parse import urlparse

import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

# Lower score = popped sooner, so lower lane offset = higher priority.
LANES = {
    "critical": -1_000_000_000,   # backlog repair, re-crawls for paying alerts
    "normal":   0,
    "bulk":     2_000_000_000,    # historical backfills, sitemap sweeps
}

QUEUE_KEY = "scrape:queue"

def enqueue(url: str, lane: str = "normal", delay_s: float = 0.0,
            payload: dict | None = None) -> str:
    job_id = str(uuid.uuid4())
    job = {
        "id": job_id,
        "url": url,
        "domain": urlparse(url).netloc,
        "attempts": 0,
        "max_attempts": 5,
        "payload": payload or {},
    }
    eligible_at = time.time() + delay_s + LANES[lane]
    r.zadd(QUEUE_KEY, {json.dumps(job): eligible_at})
    return job_id

Wait — priority as a constant offset on a timestamp score? Doesn't that mean a critical job enqueued now loses to a bulk job enqueued yesterday? Yes, and that's deliberate: the offset is sized (here, ~23 days for bulk vs. critical) so lane ordering holds within any realistic backlog window, while an aging job eventually out-prioritizes fresh low-lane work. That aging property is what keeps bulk lanes from starving permanently. If you need strict lane precedence instead, run three ZSETs and pop them in order — but you'll usually want aging more than you want strictness.

Popping is the classic atomic non-blocking pattern:

def pop_job(timeout_s: int = 5) -> dict | None:
    deadline = time.time() + timeout_s
    while time.time() < deadline:
        now = time.time()
        jobs = r.zrangebyscore(QUEUE_KEY, "-inf", now, start=0, num=1)
        if not jobs:
            time.sleep(0.2)
            continue
        raw = jobs[0]
        if r.zrem(QUEUE_KEY, raw):     # atomic claim: only one worker wins
            return json.loads(raw)
    return None

The zrem return value is the mutex: ten workers can call zrangebyscore and see the same job, but only one zrem returns 1. No locks, no Lua needed for the basic case.

Per-Domain Circuit Breakers

Here's where the shop-43 incident actually gets solved. The breaker is three Redis keys per domain: a consecutive-failure counter, a state flag, and a trip timestamp. One domain trips, and every job destined for it gets bounced back into the queue with a delay — before we ever spend a worker-second or a proxy request on it.

import socket
from urllib.parse import urlparse

BREAKER_FAILS = 5            # consecutive failures before tripping
BREAKER_COOLDOWN = 120       # seconds open before we probe again

def breaker_key(domain: str, part: str) -> str:
    return f"breaker:{domain}:{part}"

def breaker_allows(domain: str) -> bool:
    state = r.get(breaker_key(domain, "state"))
    if state != "open":
        return True
    tripped_at = float(r.get(breaker_key(domain, "tripped_at") or 0))
    if time.time() - tripped_at > BREAKER_COOLDOWN:
        # half-open: let exactly one probe through, tracked via a short TTL lock
        return bool(r.set(breaker_key(domain, "probe"), 1, nx=True, ex=30))
    return False

def record(domain: str, success: bool) -> None:
    fails_key = breaker_key(domain, "fails")
    if success:
        r.delete(fails_key)
        r.delete(breaker_key(domain, "state"))
        r.delete(breaker_key(domain, "tripped_at"))
        r.delete(breaker_key(domain, "probe"))
        return
    fails = r.incr(fails_key)
    if fails >= BREAKER_FAILS:
        r.set(breaker_key(domain, "state"), "open")
        r.set(breaker_key(domain, "tripped_at"), time.time())

Why is this better than a global retry limit? Because the breaker captures the information the retry count can't: whether failures correlate by domain. Five consecutive failures on shop-43.example.com while everything else succeeds is a signal about that site, not about those five jobs. A per-job retry limit makes each job independently decide it's doomed; a per-domain breaker lets the system decide the destination is unhealthy. Same failures, completely different conclusion — and the healthy domains never pay for it.

The breaker even saves you money on retries: while open, jobs for that domain rack up delay but cost zero fetches. When the half-open probe succeeds, the backlog drains on its normal backoff schedule.

Backoff With Jitter, and the Dead-Letter Exit

The retry math needs three properties: exponential growth (be gentle on a struggling site), full jitter (no synchronized herds), and a terminal state (retry budgets are finite, or you're running a perpetual motion machine).

import random

BASE_DELAY = 15
MAX_DELAY = 3600
DEAD_LETTER_KEY = "scrape:dead"

def backoff_delay(attempts: int) -> float:
    exp = min(BASE_DELAY * (2 ** attempts), MAX_DELAY)
    return random.uniform(0, exp)          # full jitter (AWS-style)

def handle_result(job: dict, exc: Exception | None) -> None:
    if job["domain"] is None:
        job["domain"] = urlparse(job["url"]).netloc

    if exc is None:
        record(job["domain"], success=True)
        return  # success path: write your results, done

    record(job["domain"], success=False)
    job["attempts"] += 1
    job["last_error"] = f"{type(exc).__name__}: {exc}"[:500]

    if job["attempts"] >= job["max_attempts"]:
        # dead-letter: score by when it died, so review is newest-first-ish
        r.zadd(DEAD_LETTER_KEY, {json.dumps(job): time.time()})
        return

    # If the domain breaker is open, park the job for the cooldown period
    # instead of hammering the backoff curve against a wall we know about.
    delay = backoff_delay(job["attempts"])
    if r.get(breaker_key(job["domain"], "state")) == "open":
        delay = max(delay, BREAKER_COOLDOWN + random.uniform(0, 30))
    r.zadd(QUEUE_KEY, {json.dumps(job): time.time() + delay})

Full jitter (uniform(0, exp)) rather than "exponential plus a little random" matters most right after a mass outage. Suppose 8,000 jobs were mid-flight when a provider hiccuped. With deterministic backoff, all 8,000 wake up in the same second, every time, in lockstep. With full jitter, the wake-ups smear uniformly across each doubling window — the herd never forms. It costs you a slightly longer average wait per job, and it buys you an outage that ends once instead of three times.

The dead-letter set is not a failure of the system; it's a product feature. It's the list that tells you a domain banned your fingerprint, a page template changed, or a site 404'd your whole catalog. I check the size of it daily; a sudden spike in dead-letters for one domain is the earliest alarm you'll get that something upstream changed, usually before your success-rate dashboards move.

Tying It Together: The Worker Loop

import requests

def run_worker(shutdown_at: float) -> None:
    while time.time() < shutdown_at:
        job = pop_job(timeout_s=5)
        if job is None:
            continue
        if not breaker_allows(job["domain"]):
            # domain is open — park the job, don't burn a fetch
            r.zadd(QUEUE_KEY, {json.dumps(job):
                               time.time() + BREAKER_COOLDOWN + random.uniform(0, 30)})
            continue
        try:
            resp = requests.get(job["url"], timeout=30,
                                proxies={"https": "http://USER:[e-mail]:9000"})
            resp.raise_for_status()
            handle_result(job, None)          # -> extract, store, etc.
        except (requests.RequestException, socket.timeout) as e:
            handle_result(job, e)

if __name__ == "__main__":
    run_worker(shutdown_at=time.time() + 3600)

In production this loop gets more: a wrapper that marks in-flight jobs into a processing ZSET with a TTL for crash recovery, a graceful-shutdown drain, and per-domain politeness in front of the fetch. But those are refinements — the skeleton above is the load-bearing part, and it's the part most scraping stacks are missing.

Wrapping Up

The pattern in one sentence: push failure state down to where the failure actually lives. Retry counts belong to jobs, because jobs are what get retried. Circuit state belongs to domains, because domains are what break. Priority belongs to lanes, because urgency is a property of the business, not the arrival order. And every retry must be jittered, bounded, and must end somewhere — either back in the queue or in the dead letters, never in limbo.

None of this is exotic. It's one ZSET, a handful of small keys, and the discipline to not write a global retry(3) anywhere. When the next shop-43 deploys its broken build, the breaker trips, that domain's jobs park themselves, and your other 400 domains never even notice. That's what resilience actually looks like — not a system that never fails, but one where failure is small, local, and cheap.

Disclosure: I use Thordata's residential proxies as the fetch layer behind queue patterns like the one in this post. If you want to try them, they're at thordata.com, and the code **thor020* gets you 10% off.*