Building a Resilient Async Web Crawler in Python: Rate Limiting, Retries, and Concurrency Control

I recently needed to crawl ~50,000 pages from a documentation site for a search index rebuild. The naive approach — requests.get() in a loop — would have taken roughly 14 hours. With asyncio and aiohttp, I got it down to 22 minutes.

But speed isn't the only challenge. Polite crawlers need rate limiting, retry logic, and error handling. Here's the production-grade pattern I landed on.

The Core Pattern: Semaphore-Controlled Concurrency

The fundamental building block is an asyncio.Semaphore to cap concurrent requests:

import asyncio
import aiohttp
from aiohttp import ClientTimeout, ClientError

class AsyncCrawler:
    def __init__(self, max_concurrent: int = 10, rate_limit: float = 0.1):
        self.semaphore = asyncio.Semaphore(max_concurrent)
        self.rate_limit = rate_limit  # seconds between requests
        self.last_request_time = 0.0
        self.session: aiohttp.ClientSession | None = None

    async def __aenter__(self):
        timeout = ClientTimeout(total=30, connect=10)
        self.session = aiohttp.ClientSession(timeout=timeout)
        return self

    async def __aexit__(self, *args):
        await self.session.close()

The semaphore acts as a gate: at most `max_concurrent] tasks can hold the semaphore at once. The rest queue up automatically.

Adding Polite Rate Limiting

A semaphore alone doesn't prevent you from hammering the server when all concurrent tasks fire simultaneously. You need a token bucket or at minimum a time-gated delay:

`python
async def _rate_limit(self):
"""Ensure minimum delay between requests."""
now = asyncio.get_event_loop().time()
wait = self.rate_limit - (now - self.last_request_time)
if wait > 0:
await asyncio.sleep(wait)
self.last_request_time = asyncio.get_event_loop().time()

async def fetch(self, url: str) -> str | None:
    async with self.semaphore:
        await self._rate_limit()
        try:
            async with self.session.get(url) as response:
                if response.status == 429:
                    retry_after = int(response.headers.get("Retry-After", 5))
                    print(f"Rate limited on {url}, waiting {retry_after}s")
                    await asyncio.sleep(retry_after)
                    return await self.fetch(url)  # retry once
                response.raise_for_status()
                return await response.text()
        except (ClientError, asyncio.TimeoutError) as e:
            print(f"Failed to fetch {url}: {e}")
            return None

`

Exponential Backoff with Retries

For transient failures (5xx, timeouts), exponential backoff prevents cascading load on the server:

python
async def fetch_with_retry(self, url: str, max_retries: int = 3) -> str | None:
for attempt in range(max_retries):
result = await self.fetch(url)
if result is not None:
return result
if attempt < max_retries - 1:
wait = 2 ** attempt # 1, 2, 4 seconds
print(f"Retrying {url} in {wait}s (attempt {attempt + 1}/{max_retries})")
await asyncio.sleep(wait)
return None


Managing the Work Queue

With hundreds of thousands of URLs, you can't load them all into memory as tasks. Use an asyncio.Queue to stream URLs:

`python
async def crawl(self, urls: list[str]) -> dict[str, str | None]:
queue = asyncio.Queue()
for url in urls:
await queue.put(url)

    results = {}

    async def worker():
        while not queue.empty():
            try:
                url = queue.get_nowait()
            except asyncio.QueueEmpty:
                break
            results[url] = await self.fetch_with_retry(url)
            queue.task_done()

    workers = [asyncio.create_task(worker()) for _ in range(self.semaphore._value)]
    await asyncio.gather(*workers)
    return results

`

Putting It All Together

`python
async def main():
urls = [f"https://example.com/page/{i}" for i in range(1000)]

async with AsyncCrawler(max_concurrent=20, rate_limit=0.05) as crawler:
    results = await crawler.crawl(urls)

successful = sum(1 for v in results.values() if v is not None)
print(f"Crawled {successful}/{len(urls)} pages successfully")

asyncio.run(main())
`

What This Avoids

1. Connection pool exhaustion — aiohttp.ClientSession reuses connections internally
2. Memory blow-up — Queue-based dispatching doesn't create all coroutines at once
3. Server hammering — Semaphore + rate limiter ensure polite crawling
4. Silent failures — Retry with backoff + explicit None returns for failed pages

Benchmarks from My Project

On a standard VPS with 4 vCPUs:

• Sequential (requests.get): ~500 pages/hour
• Semaphore (10 concurrent): ~4,500 pages/hour
• Semaphore (20 concurrent) + connection pooling: ~8,200 pages/hour

The law of diminishing returns kicks in around 30-50 concurrent connections for most servers. Beyond that, you're fighting the remote server's connection limit.

---

Originally posted on my blog at AIXHDD. Check out more Python development tools and content there.