Cells, Not Threads: A Failure-Isolation Architecture for Amazon Price Scrapers

A previous post covered the survival basics of Amazon scraping: throttle hard, back off exponentially, rotate proxies. That's necessary but not sufficient. Everything in that architecture shares one fatal assumption — that your scraper is one thing. One pool of workers, one pool of proxies, one rate limit, one health state. The moment Amazon's anti-bot system fingerprint you, it burns the whole machine.

This post argues for a different decomposition. Don't scale your Amazon price monitor by adding concurrency to a monolith. Partition it into cells: each cell owns a slice of products, its own sticky proxy cohort, its own rate budget, and its own circuit breaker. When Amazon starts blocking one cell, that cell dies alone. The rest of the pipeline keeps producing clean data. Failure isolation — not brute force — is what keeps a long-running monitor alive.

Why a monolithic pool fails together

The standard design looks like this: a queue of ASINs, a worker pool of N asyncio tasks, and a shared proxy pool where each request grabs a random proxy. It feels robust. It's actually a correlation machine.

Three correlated failure modes:

1. Shared proxy pool. If one residential session gets flagged and the gateway starts returning CAPTCHAs, your pool doesn't know. The flagged session goes back in rotation, gets reused, and poisons more requests. Within minutes you're burning your entire proxy bandwidth on CAPTCHA pages — and paying for it.
2. Global concurrency. A single asyncio.Semaphore(50) means every detection event has access to your full request rate. Amazon's rate heuristics fire on aggregate behavior from your exit IPs. One shared budget means one shared tripwire.
3. Global circuit breaker. If you open a breaker on, say, a 30% CAPTCHA rate measured across the whole pool, a problem that originated with one proxy cohort takes down 100% of your throughput. You didn't isolate the failure; you broadcast it.

The worst version I've seen: a monitor where a single soft-block on one subnet caused a global backoff, which caused the queue to back up, which caused the retry logic to hammer harder when the breaker half-opened. Everything failed together, repeatedly, on a six-hour cycle. The data had holes exactly where it mattered — during price drops, when Amazon's defenses are most aggressive anyway.

The cell abstraction

A cell is the unit of failure. Concretely, each cell contains:

• A product slice: a fixed assignment of ASINs (or a category subtree). Cells never share products, so there's no duplicate crawling and no cross-cell contention.
• A sticky proxy cohort: a small set of residential sessions via a gateway, e.g. http://user-session-cell04-a:[e-mail]:8080. Stickiness matters — an exit IP that consistently browses the same category looks like a returning shopper; an exit IP that jumps between random product pages every request looks like a bot. I run 3–5 sessions per cell and pin them for days, not per-request.
• An independent rate budget: a per-cell token bucket, something conservative like 20–40 requests/min per cell for price checks. The point is that budgets are not pooled.
• A circuit breaker: closed → open → half-open state machine per cell, with its own thresholds.
• A health score: a decaying metric of CAPTCHA rate, HTTP 429/503 rate, and parse failures, used to decide whether to retire just the session cohort (cheap) or quarantine the whole cell (expensive).

The key property: no shared mutable state across cells. A cell's rate limiter, breaker, and cohort are private. Coordination happens only at the top level — a scheduler that decides which cells run and a sink that merges their output.

The code

Runnable, stdlib + aiohttp. This is the core machinery — the fetch/parse layer is deliberately skeletal (you bring your own selectors and CAPTCHA detector), but the isolation mechanics are complete.

import asyncio
import time
import random
from dataclasses import dataclass, field
from enum import Enum
from typing import Callable

import aiohttp

# ---------------------------------------------------------------- proxies

def session_urls_for_cell(cell_id: str, n_sessions: int = 4) -> list[str]:
    """Sticky residential sessions for one cell. Each gateway username pins
    a distinct exit IP for the session's lifetime (typically ~10-30 min,
    refreshed automatically by the provider). Keep usernames stable per cell
    so Amazon sees returning visitors."""
    return [
        f"http://user-cell{cell_id}-s{i}:[e-mail]:8080"
        for i in range(n_sessions)
    ]

# ------------------------------------------------------- rate limiting

class TokenBucket:
    """Per-cell token bucket. 0.5 tokens/sec refill = 30 req/min sustained."""

    def __init__(self, rate: float, capacity: int):
        self.rate = rate            # tokens per second
        self.capacity = capacity
        self.tokens = float(capacity)
        self.updated = time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self):
        async with self._lock:
            while True:
                now = time.monotonic()
                self.tokens = min(self.capacity,
                                  self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1.0:
                    self.tokens -= 1.0
                    return
                await asyncio.sleep(max(0.05, (1.0 - self.tokens) / self.rate))

# ---------------------------------------------------- circuit breaker

class BreakerState(Enum):
    CLOSED = "closed"      # normal operation
    OPEN = "open"          # cell tripped, all requests refused
    HALF_OPEN = "half_open"  # probing recovery

class CircuitBreaker:
    """Per-cell breaker. Trips OPEN after `fail_threshold` failures inside
    `window` seconds; probes with a single request after `cooldown`."""

    def __init__(self, fail_threshold: int = 8, window: float = 120.0,
                 cooldown: float = 900.0):
        self.fail_threshold = fail_threshold
        self.window = window
        self.cooldown = cooldown
        self.state = BreakerState.CLOSED
        self.failures: list[float] = []
        self.opened_at: float = 0.0

    def allow(self) -> bool:
        now = time.monotonic()
        if self.state == BreakerState.CLOSED:
            return True
        if self.state == BreakerState.OPEN:
            if now - self.opened_at >= self.cooldown:
                self.state = BreakerState.HALF_OPEN
                return True          # exactly one probe gets through
            return False
        return False                 # half-open: probe already in flight

    def record(self, ok: bool):
        now = time.monotonic()
        if ok:
            self.failures.clear()
            if self.state == BreakerState.HALF_OPEN:
                self.state = BreakerState.CLOSED
            return
        if self.state == BreakerState.HALF_OPEN:
            self.state = BreakerState.OPEN
            self.opened_at = now
            return
        self.failures = [t for t in self.failures if now - t < self.window]
        self.failures.append(now)
        if len(self.failures) >= self.fail_threshold:
            self.state = BreakerState.OPEN
            self.opened_at = now

# ------------------------------------------------------------- the cell

@dataclass
class Cell:
    cell_id: str
    asins: list[str]
    check_interval: float = 7200.0       # each ASIN checked every ~2h
    rate: float = 0.5                    # 30 req/min sustained
    burst: int = 5
    breaker_params: dict = field(default_factory=dict)
    cohort: list[str] = field(default_factory=list)
    bucket: TokenBucket = field(init=False)
    breaker: CircuitBreaker = field(init=False)
    health: float = 1.0                  # decaying health score in [0, 1]
    generation: int = 0                  # bumps on cohort retirement

    def __post_init__(self):
        self.bucket = TokenBucket(self.rate, self.burst)
        self.breaker = CircuitBreaker(**self.breaker_params)
        if not self.cohort:
            self.cohort = session_urls_for_cell(self.cell_id)

    def record_outcome(self, ok: bool, captcha: bool = False):
        """Health decay is faster for CAPTCHAs than for soft errors —
        CAPTCHA means detected; a 503 often means 'try again'."""
        penalty = 0.25 if captcha else (0.10 if not ok else 0.0)
        self.health = max(0.0, self.health * (1.0 - penalty) if not ok
                          else min(1.0, self.health + 0.05))
        self.breaker.record(ok and not captcha)

    async def run(self, session_factory: Callable[[str], aiohttp.ClientSession],
                  fetch: Callable, sink: Callable, stop: asyncio.Event):
        while not stop.is_set():
            for asin in self.asins:
                if stop.is_set():
                    return
                if not self.breaker.allow():
                    await asyncio.sleep(60)   # cell is open; idle cheaply
                    continue
                await self.bucket.acquire()
                proxy = random.choice(self.cohort)
                try:
                    async with session_factory(proxy) as http:
                        price = await fetch(http, asin)
                    if price is None:                     # CAPTCHA / blocked
                        self.record_outcome(False, captcha=True)
                    else:
                        self.record_outcome(True)
                        await sink(self.cell_id, asin, price)
                except (aiohttp.ClientError, asyncio.TimeoutError):
                    self.record_outcome(False)

            # Cohort retirement: sessions are the cheap thing to replace.
            if self.health < 0.5 and self.breaker.state == BreakerState.CLOSED:
                self.generation += 1
                self.cohort = session_urls_for_cell(f"{self.cell_id}g{self.generation}")
                self.health = 1.0
                await asyncio.sleep(300)   # let new IPs warm up quietly

            await asyncio.sleep(self.check_interval / max(1, len(self.asins)))

# ------------------------------------------------------------ scheduler

async def main():
    # 24 cells x ~400 ASINs = ~9,600 products, 12 req/min aggregate budget
    # per cell, comfortably under 300 req/min total.
    asins = load_catalog()                       # your ASIN universe
    size = 400
    cells = [Cell(cell_id=f"{i:02d}",
                  asins=asins[i * size:(i + 1) * size],
                  rate=0.2, burst=3,             # 12 req/min per cell
                  breaker_params=dict(fail_threshold=8, cooldown=900))
             for i in range(0, (len(asins) + size - 1) // size)]

    stop = asyncio.Event()
    connector = aiohttp.TCPConnector(limit_per_host=4)  # no global blowups

    def session_factory(proxy: str) -> aiohttp.ClientSession:
        return aiohttp.ClientSession(
            connector=connector,
            proxy=proxy,
            timeout=aiohttp.ClientTimeout(total=20),
            headers={"User-Agent": UA_POOL[cell_id_hash(proxy)]})

    async def fetch(http, asin):
        # Your real fetch: GET product page, detect CAPTCHA markers,
        # extract price. Return None on block.
        ...

    async def sink(cell_id, asin, price):
        print(f"[{cell_id}] {asin} -> {price}")  # write to your DB here

    async def fetch_all(c: Cell):
        async with aiohttp.ClientSession() as _:
            await c.run(session_factory, fetch, sink, stop)

    await asyncio.gather(*(fetch_all(c) for c in cells))

if __name__ == "__main__":
    asyncio.run(main())

A few implementation notes that matter in production:

• Thundering herd on half-open. My first version probed with the normal request rate the instant the breaker half-opened — which re-tripped it within seconds, 15 minutes of cooldown at a time. The probe must be one request, and a successful close should ramp rate back over a few minutes, not instantly.
• Session cohort poisoning. Retiring the cohort (generation bump above) is your first lever. It's cheap — new exit IPs, same product slice — and it fixes the common case where the IPs, not the crawl pattern, got flagged. Quarantining the whole cell (leaving it open for hours) is the second lever, for when a fresh cohort gets blocked within minutes. That pattern means Amazon flagged something about the behavior — pacing, header fingerprint, or the product slice itself — and rotating IPs just burns money.
• Cold cells after retirement. A retired cohort means cold TCP/TLS sessions and IPs with no browsing history. Warm them at 25% rate for the first 5 minutes; cold-starting at full rate on fresh IPs is a reliable way to get the new cohort flagged too.

Cell sizing heuristics

There's a real tradeoff. Small cells mean cheap failures but high fixed costs: every cell carries its own breaker, bucket, metrics, and cohort minimum (you can't usefully run 2 sessions). Big cells amortize those costs but each failure deletes a large chunk of your coverage.

Numbers that have worked for me on Amazon, for a ~10k-ASIN price monitor:

• 200–500 ASINs per cell. At 400 ASINs checked every 2 hours, a cell needs ~3.3 requests/min sustained — a trivial rate that looks human on a single IP.
• 3–5 sticky sessions per cell, rotated lazily (one per day) rather than all at once.
• 12–30 cells total. Past ~30 cells, per-cell observability becomes the bottleneck — you need dashboards, alerting, and postmortem data per cell, and that's real operational surface area.

The honest downside: cells add static assignment complexity. Products change popularity, categories change size, and a static partition slowly goes stale. You'll want a periodic rebalance (weekly, off-peak) that moves ASINs between cells — carefully, because moving an ASIN to a new cell puts it on new IPs, and the first few checks after a move are your least reliable data points. Mark them in your sink.

Blast-radius math

This is the part that convinced my team. Model detection events as knocking out whatever shares the flagged resource.

Monolith with a shared pool: one detection event on the shared proxy pool or global budget takes out 100% of throughput. Expected data loss per event = 1.0. Worse, recovery is global — the breaker has to be confident about everything before anything resumes, so downtime is long.

N independent cells: one event takes out one cell (or one cohort within it, which is recoverable without data loss). Expected data loss = 1/N. With 24 cells, a detection event costs you ~4% of coverage for the cooldown window, and the other 96% keeps writing rows.

For a price monitor specifically, this compounds: prices change during exactly the windows when Amazon is most defensive (drops, Lightning Deals, Prime events). A monolith tends to be fully down precisely when the data is most valuable. Cells degrade partially — you lose resolution on some categories, not the whole picture. And "some cells down during the sale" is a survivable data-quality event; "everything dark for six hours during Prime Day" is a pager and an angry stakeholder.

What this buys you operationally

• Partial degradation instead of binary uptime. Your SLA becomes "at least 90% of cells reporting," which you can actually hold.
• Clean data vs corrupted data. A blocked cell writes nothing, not CAPTCHA-page garbage. Nothing is recoverable downstream (backfill from an adjacent cell's spot-checks, or interpolation flagged as such); corrupted rows are quietly wrong.
• Cheap postmortems. Per-cell metrics turn "the scraper is flaky" into "cell 07 tripped at 02:14 after its cohort's CAPTCHA rate hit 40% — cohort retired, recovered in 22 minutes." One graph per cell, one obvious culprit.
• Horizontal scale without renegotiation. Adding coverage means adding cells, not re-tuning a global budget everyone shares.

The mental shift is the point. Stop asking "how many concurrent requests can I survive?" and start asking "when detection happens — because it will — how much of my pipeline dies with it?" If the answer is "all of it," you don't have a scraper architecture. You have a single point of failure with extra steps. Partition into cells, give each one its own budget and breaker, and let failures stay small, local, and boring.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele