Scraping at Scale with asyncio and aiohttp: 10x Throughput Without Getting Blocked

There's a moment in every scraping project when requests in a for-loop stops being enough. The job list hits ten thousand URLs, each fetch takes two seconds, and the math says your crawl finishes sometime next week.

The obvious fix — more concurrency — is also the obvious way to get blocked. This post is about getting both: async-level throughput and survival. They're not in tension if you structure the pipeline correctly.

Why asyncio Changes the Economics

Synchronous scraping spends most of its life waiting: connection setup, TLS handshake, server think-time, response transfer. Your process is idle for 90%+ of each request. Asyncio lets one worker overlap that waiting across dozens of requests — same machine, same IP budget, ten times the effective throughput.

import asyncio
import aiohttp

async def fetch(session, url, proxy):
    async with session.get(url, proxy=proxy, timeout=aiohttp.ClientTimeout(total=30)) as r:
        return await r.text()

async def main(urls):
    proxy = f"http://{USER}:{PASS}@gate.thordata.com:9000"
    async with aiohttp.ClientSession(headers=HEADERS) as session:
        tasks = [fetch(session, u, proxy) for u in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)
    return results

results = asyncio.run(main(urls))

Run that on 50 URLs and the wall-clock is barely longer than a single fetch. Now the real work begins — because naively scaling this to 10,000 requests is how you burn through a proxy budget and a target site's patience simultaneously.

The Three throttles of a Healthy Async Scraper

Throughput isn't one knob. A production crawler controls three:

Concurrency cap — how many requests are in flight at once. Use a semaphore:

async def worker(name, queue, session, sem, results):
    while True:
        url = await queue.get()
        async with sem:                      # max in-flight requests
            try:
                html = await fetch(session, url, proxy_for(url))
                results.append(parse(html))
            except Exception as e:
                log_failure(url, e)
        await asyncio.sleep(random.uniform(1.0, 3.0))   # per-task politeness
        queue.task_done()

async def crawl(urls, concurrency=20):
    queue = asyncio.Queue()
    sem = asyncio.Semaphore(concurrency)
    for u in urls:
        queue.put_nowait(u)
    results = []
    async with aiohttp.ClientSession(headers=HEADERS) as session:
        workers = [asyncio.create_task(worker(f"w{i}", queue, session, sem, results))
                   for i in range(concurrency)]
        await queue.join()
        for w in workers:
            w.cancel()
    return results

Per-request delay with jitter — the asyncio.sleep(random.uniform(...)) above. Twenty concurrent workers each pausing randomly between requests looks like twenty people browsing. Twenty workers hammering with zero gap looks like a DDoS.

Global rate ceiling — even with jitter, cap total requests per minute. A token bucket fits in ten lines:

class RateLimiter:
    def __init__(self, max_per_minute):
        self.interval = 60.0 / max_per_minute
        self._last = 0.0
        self._lock = asyncio.Lock()

    async def acquire(self):
        async with self._lock:
            now = asyncio.get_event_loop().time()
            wait = self._last + self.interval - now
            if wait > 0:
                await asyncio.sleep(wait)
            self._last = asyncio.get_event_loop().time()

Attach it to the worker loop and you have a hard ceiling regardless of worker count.

Retries That Don't Amplify

The classic async anti-pattern: one endpoint hiccups, all fifty workers retry simultaneously, the target sees a synchronized burst, and blocks you. Fix it with exponential backoff and decorrelated jitter:

async def fetch_with_retry(session, url, proxy, max_retries=4):
    for attempt in range(max_retries):
        try:
            async with session.get(url, proxy=proxy,
                                   timeout=aiohttp.ClientTimeout(total=30)) as r:
                if r.status == 200:
                    return await r.text()
                if r.status in (403, 429):
                    # backoff + jitter: never retry in lockstep
                    await asyncio.sleep((2 ** attempt) + random.uniform(0, 2))
                    continue
                return None
        except (aiohttp.ClientError, asyncio.TimeoutError):
            await asyncio.sleep((2 ** attempt) + random.uniform(0, 2))
    return None

Where the Proxy Layer Fits

Concurrency multiplies whatever your IP situation is. Twenty workers on one datacenter IP is a self-inflicted ban; twenty workers on a rotating residential pool is just... twenty users. The async crawler is where rotating residential proxies earn their keep:

def proxy_for(url):
    # gateway handles rotation per connection; geo set per target market
    user = f"{USER}-country-{geo_for(url)}"
    return f"http://{user}:{PASS}@gate.thordata.com:9000"

Two async-specific notes. First, each new connection through the gateway gets a fresh exit IP — with aiohttp's connection pooling, tune limit and limit_per_host on your TCPConnector so you're not unknowingly reusing one connection (and thus one IP) for everything:

connector = aiohttp.TCPConnector(limit=0, limit_per_host=0, force_close=True)
# force_close: new connection per request = new rotating IP per request

Second, sticky sessions compose with async fine — same username-parameter trick, pinned per logical task rather than per worker.

Sizing Guide

Workload	Workers	Delay	Result
Politeness-critical (small site)	3–5	2–5s	Safe, slow
General crawling	15–25	1–3s	Sweet spot
Bulk on rotating residential	30–50	0.5–1.5s	Throughput mode

Start conservative and raise concurrency until your block rate moves, then back off 20%. The block rate is a gauge on the dashboard, not something you discover at the end.

The Honest Caveat

Async speeds up waiting, not parsing. If your CPU is the bottleneck (heavy HTML processing), move parsing to a ProcessPoolExecutor or you'll just have very fast I/O feeding a very slow queue. Profile before scaling — the answer is sometimes multiprocessing, not asyncio.

---

Disclosure: my async pipelines run on Thordata's rotating residential proxies (from $0.65/GB; code thor020 for 10% off). The async patterns here are provider-independent.