Introduction: When One Machine Is No Longer Enough
Most Python developers write simple scrapers—requests, BeautifulSoup, a loop, CSV writer—just to get data once or twice. When scaling or running them over months, the real challenge is building a robust, scheduled system, not the parsing logic. The extraction code is tiny; the critical components are the queue, cache, storage, block detection, recovery loop, and exporters that keep the scraper alive against real‑world internet conditions.
At small scale, a single async Python process can handle thousands of pages per hour. But when you need to crawl millions of pages per day — competitor catalogues, job markets, news archives, e-commerce databases — a single machine hits hard limits: one IP address, one CPU, one point of failure.
Distributed scraping involves spreading tasks across multiple machines to increase speed and volume. Throttling and introducing random delays between requests can help to prevent IP bans, while rotating proxies help distribute requests and avoid detection. Managing sessions and leveraging parallel processing can further enhance efficiency.
This guide shows you exactly how to build a distributed scraping system that runs across multiple machines — using scrapy-redis for shared request queues, Docker for containerisation, and a production proxy management layer that survives real-world conditions.
The Architecture: One Queue, Many Workers
The core insight behind distributed scraping is simple: replace Scrapy's in-memory request queue with a shared Redis queue that every worker machine can read from.
┌─────────────────────────────────────────────────────────────┐
│ DISTRIBUTED SCRAPER │
│ │
│ Master │
│ ┌──────────┐ Seeds requests ┌─────────┐ │
│ │ Spider │──────────────────────▶│ Redis │ │
│ │ (seed) │ │ Queue │ │
│ └──────────┘ └────┬────┘ │
│ │ │
│ Workers (any number of machines) │ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Worker 1 │ │ Worker 2 │ │ Worker N │ ← pull jobs │
│ │ (spider) │ │ (spider) │ │ (spider) │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │
│ └──────────────┴──────────────┘ │
│ │ │
│ ┌─────▼──────┐ │
│ │ MongoDB │ ← all workers write here │
│ │ (results) │ │
│ └────────────┘ │
└─────────────────────────────────────────────────────────────┘
scrapy-redis lets multiple spider instances across machines pull from the same request queue with deduplication and job scheduling. This is the standard approach for large-scale web scraping that Python teams use in production.
Part 1: Setting Up scrapy-redis
pip install scrapy scrapy-redis redis pymongo
The distributed spider
# spiders/distributed_product_spider.py
import scrapy
from scrapy_redis.spiders import RedisSpider
from urllib.parse import urljoin
from datetime import datetime, timezone
class DistributedProductSpider(RedisSpider):
"""
A Scrapy spider that pulls start URLs from a Redis list
instead of a hardcoded start_urls list.
To start crawling, push seed URLs to Redis:
redis-cli lpush products:start_urls "https://example-store.com/products/"
Any number of workers running this spider will cooperatively
process the shared queue — each URL is processed exactly once.
"""
name = "distributed_products"
redis_key = "products:start_urls" # Redis list to pop URLs from
# How many URLs to pop from Redis at once per worker
redis_batch_size = 16
# Spider-level settings — override settings.py per spider
custom_settings = {
"CONCURRENT_REQUESTS": 32,
"CONCURRENT_REQUESTS_PER_DOMAIN": 8,
"DOWNLOAD_DELAY": 0.5,
"RANDOMIZE_DOWNLOAD_DELAY": True,
"AUTOTHROTTLE_ENABLED": True,
"AUTOTHROTTLE_TARGET_CONCURRENCY": 16.0,
"AUTOTHROTTLE_MAX_DELAY": 5.0,
"RETRY_TIMES": 3,
"RETRY_HTTP_CODES": [429, 500, 502, 503, 504],
}
def parse(self, response):
"""
Parse a product listing page.
Yields product detail requests AND discovers pagination links.
"""
# Follow product links to detail pages
for href in response.css("a.product-link::attr(href)").getall():
yield response.follow(href, callback=self.parse_product)
# Auto-discover pagination — push next page back to Redis queue
next_page = response.css("a[rel='next']::attr(href)").get()
if next_page:
# Use follow() to handle relative URLs
yield response.follow(next_page, callback=self.parse)
def parse_product(self, response):
"""Extract product data from a detail page."""
yield {
"url": response.url,
"title": response.css("h1::text").get("").strip(),
"price": response.css(".price::text").get("").strip(),
"sku": response.css("[data-sku]::attr(data-sku)").get(),
"in_stock": bool(response.css(".in-stock")),
"description":response.css(".product-description::text").get("").strip()[:500],
"scraped_at": datetime.now(timezone.utc).isoformat(),
"worker_id": self.settings.get("WORKER_ID", "unknown"),
}
settings.py for distributed mode
# settings.py
BOT_NAME = "distributed_scraper"
SPIDER_MODULES = ["spiders"]
# ── scrapy-redis settings ─────────────────────────────────────
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_URL = "redis://redis-host:6379"
# Keep crawl state across restarts — resumable crawls
SCHEDULER_PERSIST = True
# How long to wait for new URLs before worker shuts down
SCHEDULER_IDLE_BEFORE_CLOSE = 30
# ── MongoDB pipeline ──────────────────────────────────────────
ITEM_PIPELINES = {
"pipelines.MongoPipeline": 100,
"pipelines.DuplicateFilterPipeline": 50,
}
MONGO_URI = "mongodb://mongo-host:27017/"
MONGO_DATABASE = "distributed_scrape"
# ── Concurrency ───────────────────────────────────────────────
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 0.5
RANDOMIZE_DOWNLOAD_DELAY = True
# ── Retry ─────────────────────────────────────────────────────
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524]
# ── Logging ───────────────────────────────────────────────────
LOG_LEVEL = "INFO"
# ── Downloader middlewares ────────────────────────────────────
DOWNLOADER_MIDDLEWARES = {
"scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
"middlewares.RotatingProxyMiddleware": 350,
"middlewares.UserAgentRotationMiddleware": 400,
"scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
}
# ── User agent pool ───────────────────────────────────────────
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/119.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) Chrome/118.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
]
Part 2: Production Middlewares
# middlewares.py
import random
import logging
from scrapy import signals
from scrapy.exceptions import NotConfigured
logger = logging.getLogger(__name__)
class UserAgentRotationMiddleware:
"""Rotate User-Agent on every request."""
def __init__(self, user_agents):
self.user_agents = user_agents
@classmethod
def from_crawler(cls, crawler):
agents = crawler.settings.getlist("USER_AGENTS")
if not agents:
raise NotConfigured("USER_AGENTS not set")
return cls(agents)
def process_request(self, request, spider):
request.headers["User-Agent"] = random.choice(self.user_agents)
class RotatingProxyMiddleware:
"""
Rotate through a proxy pool.
Tracks failures per proxy and removes bad proxies from the pool.
"""
def __init__(self, proxies, max_failures=5):
self.proxies = list(proxies)
self.failures = {}
self.max_failures = max_failures
@classmethod
def from_crawler(cls, crawler):
proxies = crawler.settings.getlist("PROXY_LIST", [])
if not proxies:
raise NotConfigured("PROXY_LIST is empty")
return cls(proxies)
def process_request(self, request, spider):
if not self.proxies:
return # No proxies left — run without
proxy = random.choice(self.proxies)
request.meta["proxy"] = proxy
def process_response(self, request, response, spider):
if response.status in (403, 407, 429):
proxy = request.meta.get("proxy")
self._mark_failure(proxy)
return response
def process_exception(self, request, exception, spider):
proxy = request.meta.get("proxy")
self._mark_failure(proxy)
def _mark_failure(self, proxy):
if not proxy:
return
self.failures[proxy] = self.failures.get(proxy, 0) + 1
if self.failures[proxy] >= self.max_failures:
if proxy in self.proxies:
self.proxies.remove(proxy)
logger.warning(f"Removed bad proxy: {proxy} ({self.max_failures} failures)")
class BlockDetectionMiddleware:
"""
Detect common block patterns and trigger retries with a different proxy.
"""
BLOCK_PATTERNS = [
"access denied", "captcha", "blocked", "forbidden",
"unusual traffic", "robot", "automated queries",
"cf-browser-verification", "ddos-guard",
]
def process_response(self, request, response, spider):
body_lower = response.text[:2000].lower()
is_blocked = (
response.status in (403, 429, 503) or
any(p in body_lower for p in self.BLOCK_PATTERNS) or
len(response.text) < 300
)
if is_blocked:
logger.warning(f"Block detected on {request.url} — retrying")
request.meta["proxy"] = None # Force new proxy on retry
request.dont_filter = True
return request # Re-schedule the request
return response
Part 3: MongoDB Pipeline with Bulk Writes
# pipelines.py
import logging
from datetime import datetime, timezone
from pymongo import MongoClient, UpdateOne
from pymongo.errors import BulkWriteError
from itemadapter import ItemAdapter
logger = logging.getLogger(__name__)
class DuplicateFilterPipeline:
"""Track seen URLs in memory to drop duplicates before DB write."""
def open_spider(self, spider):
self.seen_urls = set()
def process_item(self, item, spider):
adapter = ItemAdapter(item)
url = adapter.get("url", "")
if url in self.seen_urls:
from scrapy.exceptions import DropItem
raise DropItem(f"Duplicate URL: {url}")
self.seen_urls.add(url)
return item
class MongoPipeline:
"""
Write scraped items to MongoDB using bulk operations.
Upserts on URL — safe to re-run without creating duplicates.
"""
BULK_SIZE = 200 # Flush every 200 items
COLLECTION = "products"
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
self._buffer = []
@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get("MONGO_URI", "mongodb://localhost:27017/"),
mongo_db =crawler.settings.get("MONGO_DATABASE", "scrapy_data"),
)
def open_spider(self, spider):
self.client = MongoClient(self.mongo_uri)
self.col = self.client[self.mongo_db][self.COLLECTION]
self.col.create_index("url", unique=True)
self.items_written = 0
logger.info(f"MongoDB connected: {self.mongo_db}.{self.COLLECTION}")
def close_spider(self, spider):
if self._buffer:
self._flush()
self.client.close()
logger.info(f"MongoDB closed. Total items written: {self.items_written}")
def process_item(self, item, spider):
self._buffer.append(dict(item))
if len(self._buffer) >= self.BULK_SIZE:
self._flush()
return item
def _flush(self):
ops = [
UpdateOne({"url": doc["url"]}, {"$set": doc}, upsert=True)
for doc in self._buffer
]
try:
result = self.col.bulk_write(ops, ordered=False)
count = result.upserted_count + result.modified_count
self.items_written += count
logger.info(f"Flushed {len(self._buffer)} items → MongoDB")
except BulkWriteError as e:
logger.error(f"Bulk write error: {e.details.get('writeErrors', [])[:2]}")
finally:
self._buffer.clear()
Part 4: Docker Compose — Multi-Worker Setup
# docker-compose.yml
version: "3.9"
x-worker-base: &worker-base
build: .
volumes:
- .:/app
environment:
- REDIS_URL=redis://redis:6379
- MONGO_URI=mongodb://mongo:27017/
- PROXY_LIST=${PROXY_LIST}
depends_on:
- redis
- mongo
restart: unless-stopped
services:
# ── Infrastructure ──────────────────────────────────────────
redis:
image: redis:7-alpine
ports: ["6379:6379"]
command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
volumes: ["redis_data:/data"]
mongo:
image: mongo:7
ports: ["27017:27017"]
volumes: ["mongo_data:/data/db"]
# ── Scrapy Workers ──────────────────────────────────────────
# Run as many of these as you have cores / IPs
worker-1:
<<: *worker-base
command: >
scrapy crawl distributed_products
-s WORKER_ID=worker-1
-s CONCURRENT_REQUESTS=16
environment:
- REDIS_URL=redis://redis:6379
- MONGO_URI=mongodb://mongo:27017/
- WORKER_ID=worker-1
worker-2:
<<: *worker-base
command: >
scrapy crawl distributed_products
-s WORKER_ID=worker-2
-s CONCURRENT_REQUESTS=16
environment:
- REDIS_URL=redis://redis:6379
- MONGO_URI=mongodb://mongo:27017/
- WORKER_ID=worker-2
worker-3:
<<: *worker-base
command: >
scrapy crawl distributed_products
-s WORKER_ID=worker-3
-s CONCURRENT_REQUESTS=16
environment:
- REDIS_URL=redis://redis:6379
- MONGO_URI=mongodb://mongo:27017/
- WORKER_ID=worker-3
# ── Seed Service — pushes start URLs into Redis ─────────────
seeder:
<<: *worker-base
command: python seeder.py
restart: "no" # Run once then exit
volumes:
redis_data:
mongo_data:
Part 5: The Seeder — Feeding URLs Into the Queue
# seeder.py
import redis
import time
import sys
from urllib.parse import urlencode
REDIS_URL = "redis://localhost:6379"
REDIS_KEY = "products:start_urls"
def seed_from_list(urls: list[str], batch_size: int = 500):
"""Push a list of start URLs into the Redis queue."""
r = redis.from_url(REDIS_URL)
# Clear existing queue if resuming fresh
existing = r.llen(REDIS_KEY)
if existing > 0:
print(f"Queue already has {existing:,} URLs. Adding to it.")
pushed = 0
for i in range(0, len(urls), batch_size):
batch = urls[i:i + batch_size]
r.rpush(REDIS_KEY, *batch)
pushed += len(batch)
print(f"Seeded {pushed:,}/{len(urls):,} URLs")
print(f"\nDone. Redis queue '{REDIS_KEY}' has {r.llen(REDIS_KEY):,} URLs.")
def seed_paginated_site(
base_url: str,
start_page: int = 1,
end_page: int = 500,
page_param: str = "page"
):
"""Generate paginated URLs and push to Redis."""
urls = []
for page in range(start_page, end_page + 1):
params = {page_param: page}
urls.append(f"{base_url}?{urlencode(params)}")
seed_from_list(urls)
def monitor_queue():
"""Monitor queue depth and worker progress in real time."""
r = redis.from_url(REDIS_URL)
print("Monitoring queue depth (Ctrl+C to stop)...")
try:
while True:
depth = r.llen(REDIS_KEY)
seen = r.scard(f"{REDIS_KEY}:dupefilter") or 0
print(f" Queue: {depth:,} pending | Seen: {seen:,} processed", end="\r")
time.sleep(2)
except KeyboardInterrupt:
print("\nMonitor stopped.")
if __name__ == "__main__":
mode = sys.argv[1] if len(sys.argv) > 1 else "seed"
if mode == "seed":
seed_paginated_site(
base_url = "https://example-store.com/products",
start_page = 1,
end_page = 1000,
)
elif mode == "monitor":
monitor_queue()
elif mode == "clear":
r = redis.from_url(REDIS_URL)
r.delete(REDIS_KEY)
print(f"Queue '{REDIS_KEY}' cleared.")
Part 6: Scaling to Kubernetes
For truly large-scale crawling across dozens of machines, Kubernetes is the standard deployment target. The key insight: each Scrapy worker is a stateless pod that reads from the shared Redis queue.
# k8s/scrapy-worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: scrapy-workers
labels:
app: scrapy-worker
spec:
replicas: 10 # Start with 10 workers; scale up/down with kubectl
selector:
matchLabels:
app: scrapy-worker
template:
metadata:
labels:
app: scrapy-worker
spec:
containers:
- name: scrapy-worker
image: your-registry/scrapy-worker:latest
command:
- scrapy
- crawl
- distributed_products
env:
- name: REDIS_URL
valueFrom:
secretKeyRef:
name: scraper-secrets
key: redis-url
- name: MONGO_URI
valueFrom:
secretKeyRef:
name: scraper-secrets
key: mongo-uri
- name: WORKER_ID
valueFrom:
fieldRef:
fieldPath: metadata.name # Pod name as worker ID
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
Scale workers up or down instantly:
# Scale to 20 workers
kubectl scale deployment scrapy-workers --replicas=20
# Check worker status
kubectl get pods -l app=scrapy-worker
# View logs from all workers
kubectl logs -l app=scrapy-worker --tail=50
# Auto-scale based on Redis queue depth (requires custom metrics)
kubectl autoscale deployment scrapy-workers --min=2 --max=50
Part 7: Proxy Management at Scale
For serious scraping in 2026, residential proxies are almost always the safer option. Proxy quality matters far more than proxy quantity. A smaller pool of clean residential IPs usually performs much better than massive low-quality networks.
Here's a production proxy manager with health checking:
# proxy_manager.py
import asyncio
import httpx
import random
import time
import json
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class ProxyHealth:
url: str
success_count: int = 0
failure_count: int = 0
last_used: float = 0.0
last_success: float = 0.0
avg_response_ms: float = 0.0
is_banned: bool = False
@property
def success_rate(self) -> float:
total = self.success_count + self.failure_count
return self.success_count / total if total > 0 else 0.0
@property
def score(self) -> float:
"""Composite score: higher = better proxy to use."""
if self.is_banned:
return 0.0
recency_bonus = max(0, 1 - (time.time() - self.last_success) / 3600)
speed_score = max(0, 1 - self.avg_response_ms / 5000)
return self.success_rate * 0.6 + recency_bonus * 0.2 + speed_score * 0.2
class ProxyPool:
"""
Intelligent proxy pool with health tracking and weighted selection.
Workers register success/failure, pool learns which proxies are best.
"""
BAN_THRESHOLD = 0.2 # Mark as banned if success rate drops below 20%
def __init__(self, proxy_urls: list[str]):
self.proxies = {url: ProxyHealth(url=url) for url in proxy_urls}
def get_proxy(self, strategy: str = "weighted") -> Optional[str]:
"""Select a proxy using the specified strategy."""
available = [
p for p in self.proxies.values()
if not p.is_banned
]
if not available:
return None
if strategy == "random":
return random.choice(available).url
elif strategy == "weighted":
# Weight by score — best proxies get used more often
scores = [max(p.score, 0.01) for p in available]
total = sum(scores)
weights = [s / total for s in scores]
return random.choices(available, weights=weights)[0].url
elif strategy == "round_robin":
# Sort by last_used timestamp — use least recently used
available.sort(key=lambda p: p.last_used)
return available[0].url
return available[0].url
def report_success(self, proxy_url: str, response_ms: float):
if proxy_url in self.proxies:
p = self.proxies[proxy_url]
p.success_count += 1
p.last_used = time.time()
p.last_success = time.time()
# Rolling average response time
p.avg_response_ms = (p.avg_response_ms * 0.8 + response_ms * 0.2)
def report_failure(self, proxy_url: str):
if proxy_url in self.proxies:
p = self.proxies[proxy_url]
p.failure_count += 1
p.last_used = time.time()
# Auto-ban proxies with very low success rate
if p.failure_count > 10 and p.success_rate < self.BAN_THRESHOLD:
p.is_banned = True
print(f" Proxy banned (success rate {p.success_rate:.0%}): {proxy_url}")
def get_stats(self) -> dict:
active = [p for p in self.proxies.values() if not p.is_banned]
banned = [p for p in self.proxies.values() if p.is_banned]
avg_sr = sum(p.success_rate for p in active) / len(active) if active else 0
return {
"total": len(self.proxies),
"active": len(active),
"banned": len(banned),
"avg_success_rate": f"{avg_sr:.1%}",
"best_proxy": max(active, key=lambda p: p.score).url if active else None,
}
async def health_check_proxies(pool: ProxyPool, test_url: str = "https://httpbin.org/ip"):
"""
Periodically check all proxies and un-ban those that have recovered.
Run this as a background task.
"""
async with httpx.AsyncClient(timeout=10) as client:
for url, proxy in list(pool.proxies.items()):
if not proxy.is_banned:
continue
try:
start = time.time()
r = await client.get(test_url, proxies={"https": url})
if r.status_code == 200:
elapsed_ms = (time.time() - start) * 1000
proxy.is_banned = False
pool.report_success(url, elapsed_ms)
print(f" Proxy recovered: {url}")
except Exception:
pass # Still banned
stats = pool.get_stats()
print(f"Proxy pool: {stats['active']} active, {stats['banned']} banned, "
f"avg success rate: {stats['avg_success_rate']}")
Part 8: Monitoring Your Distributed Crawl
# monitor.py — real-time crawl progress dashboard
import redis
import time
import json
from pymongo import MongoClient
from datetime import datetime
def live_dashboard(redis_url: str, mongo_uri: str, refresh_seconds: int = 5):
"""Print a live crawl progress dashboard to the terminal."""
r = redis.from_url(redis_url)
db = MongoClient(mongo_uri)["distributed_scrape"]
start_time = time.time()
prev_count = 0
try:
while True:
# Queue stats
queue_depth = r.llen("products:start_urls")
seen_count = r.scard("products:start_urls:dupefilter") or 0
# DB stats
items_stored = db["products"].count_documents({})
items_delta = items_stored - prev_count
rate_per_min = items_delta * (60 / refresh_seconds)
prev_count = items_stored
# Worker stats (scrapy-redis stores worker heartbeats)
workers = r.smembers("scrapy:workers") or set()
# Elapsed
elapsed = time.time() - start_time
h, m = divmod(int(elapsed), 3600)
m, s = divmod(m, 60)
print(f"\033[2J\033[H") # Clear screen
print(f"{'═'*55}")
print(f" DISTRIBUTED SCRAPE MONITOR — {datetime.now().strftime('%H:%M:%S')}")
print(f"{'═'*55}")
print(f" Runtime: {h:02d}h {m:02d}m {s:02d}s")
print(f" Active workers:{len(workers)}")
print(f"{'─'*55}")
print(f" Queue depth: {queue_depth:>10,} (URLs remaining)")
print(f" URLs seen: {seen_count:>10,} (deduplicated total)")
print(f" Items stored: {items_stored:>10,} (in MongoDB)")
print(f" Rate: {rate_per_min:>10.0f} items/minute")
print(f"{'─'*55}")
if rate_per_min > 0 and queue_depth > 0:
eta_mins = queue_depth / rate_per_min
h2, m2 = divmod(int(eta_mins * 60), 3600)
m2, s2 = divmod(m2, 60)
print(f" ETA: {h2:02d}h {m2:02d}m (estimated)")
print(f"{'═'*55}")
time.sleep(refresh_seconds)
except KeyboardInterrupt:
print("\nMonitor stopped.")
if __name__ == "__main__":
live_dashboard(
redis_url="redis://localhost:6379",
mongo_uri="mongodb://localhost:27017/",
)
Part 9: Resumable Crawls
One of the biggest advantages of scrapy-redis is that crawls are inherently resumable. If a worker crashes or you need to add more workers mid-crawl, simply restart:
# Start a fresh crawl
python seeder.py seed
# Launch workers (they'll pick up from where they left off if SCHEDULER_PERSIST=True)
docker-compose up --scale worker=5
# Pause all workers (Ctrl+C in docker-compose)
# Resume later — queue state is preserved in Redis
docker-compose up --scale worker=10 # Can add more workers
To completely reset a crawl:
# Clear the queue and deduplication filter
redis-cli del products:start_urls
redis-cli del products:start_urls:dupefilter
python seeder.py seed # Re-seed with fresh URLs
Performance Numbers: What to Expect
One machine first, then scale out. Tune Scrapy performance optimization settings before throwing hardware at the problem — raise CONCURRENT_REQUESTS, turn on AUTOTHROTTLE, and enable HTTP caching. If one server can't handle scraping millions of pages, set up a shared queue for distributed crawling across multiple workers.
Real-world throughput benchmarks:
| Setup | Pages/hour | Cost estimate |
|---|---|---|
| 1 worker, no proxy | ~8,000 | Free |
| 1 worker + 10 proxies | ~25,000 | ~$5/day |
| 5 workers + 50 proxies | ~120,000 | ~$20/day |
| 20 workers + 200 proxies | ~500,000 | ~$80/day |
| 100 workers (Kubernetes) | ~2,500,000 | ~$350/day |
Summary
| Component | Tool | Role |
|---|---|---|
| Spider | Scrapy + scrapy-redis | Crawl logic + distributed request handling |
| Queue | Redis | Shared URL queue with built-in deduplication |
| Worker deployment | Docker Compose / Kubernetes | Horizontal scale, stateless workers |
| Proxy management | Custom ProxyPool | Health-tracked, weighted proxy selection |
| Storage | MongoDB (bulk upserts) | Centralised, deduplicated results |
| Monitoring | Custom dashboard + Flower | Real-time progress and worker health |
| Resumability | SCHEDULER_PERSIST=True | Crash-safe, restartable crawls |
Originally published on ZyVOP
💡 For more articles like this, subscribe to the ZyVOP newsletter!
Top comments (0)