Scrapy AutoThrottle: How to tune crawl speed without getting blocked

AutoThrottle is one of Scrapy's most useful production features and one of the most commonly misconfigured. Most guides tell you to add four lines to your settings file and move on. This one explains what the algorithm is actually doing, what its assumptions are, why those assumptions sometimes break down, and how to tune it for real crawls.

If you have enabled AutoThrottle and are not sure whether it is working, or if your crawl speed is not what you expect despite having it turned on, this is the post to read.

What AutoThrottle actually measures

The most common misconception about AutoThrottle: it adjusts based on HTTP response codes. It does not. It adjusts based on response latency, the time between sending a request and receiving a response.

AutoThrottle never looks at whether you are receiving 429s, 503s, or any other status code that might indicate rate limiting or blocking. When responses come back quickly, it assumes the server has capacity and reduces the delay between requests. When responses slow down, it assumes load and increases the delay. The goal is to maintain a target number of in-flight concurrent requests to the server, adjusting the inter-request delay to keep actual concurrency close to that target.

This works well when server latency is an accurate signal of server load. It breaks in several common situations:

• Content delivery network (CDN)-cached responses. A CDN edge node returns cached content at sub-millisecond latency. AutoThrottle sees very fast responses and reduces the delay aggressively, potentially hammering the origin server, which AutoThrottle never directly observes.

• Silent rate limiting. Some sites return a 200 with a soft-block page, a CAPTCHA, a antiban challenge, or empty results, in fast response time. AutoThrottle interprets this as a healthy server and keeps the rate high. You are being blocked; the algorithm does not know.

• Rate limiting via response code. A server that returns 429 in milliseconds looks, to AutoThrottle, like a fast healthy server. Latency-based throttling is irrelevant when the server is enforcing a request cap by policy rather than by load.

Knowing what AutoThrottle measures is what lets you decide when to use it. Before tuning delay settings, it is worth understanding your request requirements thoroughly: the post "The recipe for a request: Scaling data extraction" argues for investigating those requirements carefully before committing to a crawl strategy.

The settings that matter

Five settings control AutoThrottle behavior. Three of them matter for most crawls.

AUTOTHROTTLE_ENABLED = True

# The delay before the algorithm takes over — used for the first few requests
AUTOTHROTTLE_START_DELAY = 1.0

# Ceiling on the computed delay — prevents runaway backoff on very slow servers
AUTOTHROTTLE_MAX_DELAY = 60.0

# Target number of in-flight concurrent requests to the server
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

# Log per-request throttling decisions — useful for tuning
AUTOTHROTTLE_DEBUG = False

AUTOTHROTTLE_TARGET_CONCURRENCY

This is the most commonly misunderstood setting. It is not a hard concurrency cap; it is a target. AutoThrottle computes a delay to try to maintain approximately this many in-flight requests at any given time, given the observed latency.

The default of 1.0 means AutoThrottle aims for one request in flight at a time. At 200ms average response time, that translates to roughly five requests per second. At 50ms, roughly 20 requests per second. Raising TARGET_CONCURRENCY makes the crawl more aggressive; at 4.0 with 200ms average latency, AutoThrottle aims for about 20 requests per second.

Also note the interaction with CONCURRENT_REQUESTS_PER_DOMAIN: if that cap is lower than what AutoThrottle's computed delay would allow, the cap takes precedence. Both settings need to be sized for your throughput target.

AUTOTHROTTLE_MAX_DELAY

Without this, a slow or overloaded server can push computed delays into minutes. The default of 60 seconds is reasonable for polite crawling, but it means a blocked or very slow server will stall your crawl for up to a minute between requests. Set it proportional to your acceptable throughput floor: a ceiling of 10 seconds means the slowest you crawl is one request per 10 seconds.

AUTOTHROTTLE_START_DELAY

This is the delay used for the first few requests before the algorithm has enough latency samples to make good decisions. Set it to something in the range of the delay you would use if setting DOWNLOAD_DELAY manually. Too low and the first few requests come in a burst; too high and you waste time at the start of each domain.

Reading the debug output

The most direct way to understand what AutoThrottle is doing is to enable debug logging:

AUTOTHROTTLE_DEBUG = True

This logs one line per request showing the throttle decision:

2026-01-15 09:42:11 [scrapy.extensions.throttle] DEBUG: slot: example.com
  prev/curr concurrency: 3/4
  prev/curr latency: 0.24s/0.31s
  target latency: 0.31s
  delay: 0.08s -> 0.10s

slot is the domain this decision applies to.

prev/curr concurrency shows how many requests were in-flight on the previous request versus the current one, indicating whether actual concurrency is matching the target.

prev/curr latency is the signal AutoThrottle is responding to. Rising latency drives the delay up; falling latency drives it down.

target latency is what the algorithm is aiming for, derived from TARGET_CONCURRENCY and current observed latency.

delay is the computed inter-request delay before and after this adjustment. This is the number that feeds into the actual crawl rate.

What to look for during tuning:

• Delay stuck at MAX_DELAY consistently. The server is responding slowly or you are being rate limited by latency (rare). Consider raising MAX_DELAY if the server is legitimately slow, or investigate whether you are being blocked.
• Delay near zero consistently. AutoThrottle is running with almost no restriction. Either the server is very fast or TARGET_CONCURRENCY is too high relative to CONCURRENT_REQUESTS_PER_DOMAIN.
• Concurrency consistently below target. The server is slower than expected and AutoThrottle cannot achieve the target without excessive load. Lower TARGET_CONCURRENCY.

When to use AutoThrottle, a fixed delay, or neither

AutoThrottle makes sense when you are crawling a domain where server response time is a reliable signal of server load, you want adaptive behavior that maximizes throughput without overloading the server, and you are not under a specific documented rate limit. That covers most general-purpose crawls.

Use a fixed DOWNLOAD_DELAY when you have a specific rate limit to respect: an API with documented caps, or a crawl policy agreed with the site operator. Fixed delays give you auditable, predictable behavior. The Scrapy 2026 release improved retry logic and delay handling, making fixed-delay crawls more reliable in high-volume scenarios.

DOWNLOAD_DELAY = 1.0           # one second between requests
RANDOMIZE_DOWNLOAD_DELAY = True  # randomise between 0.5s and 1.5s

RANDOMIZE_DOWNLOAD_DELAY (on by default when DOWNLOAD_DELAY is set) makes the timing pattern look less mechanical, which helps with some simple bot detection approaches.

Use neither when you are crawling many small domains in parallel where per-domain load is negligible, or when you are using a scraping API such as Zyte API that manages politeness for you. Adding a delay in those cases only reduces throughput without benefiting the target server.

Patterns that trick AutoThrottle

CDN caching

Many high-traffic sites serve content from CDN edges that return cached responses in under 10ms. AutoThrottle sees fast responses and reduces the delay, sometimes to near zero. The origin server may be under significant load from your crawl; CDN latency tells you nothing about it. If you are crawling a CDN-backed site and need to be polite to the origin, use a fixed DOWNLOAD_DELAY rather than AutoThrottle.

Silent rate limiting

A site that returns a soft-block page in fast response time looks identical to a successfully served page from AutoThrottle's perspective. You are being blocked; the algorithm sees only fast responses.

The diagnostic: if your item rate drops while response latency stays low and AutoThrottle is not backing off, you are being silently blocked. Slowing down will not fix it. The problem is elsewhere in the request fingerprint.

Variable-latency backends

Some sites have pages that respond in 30ms (static HTML, heavily cached) and pages that take 800ms (product pages with real-time inventory lookups). AutoThrottle makes decisions based on recent latency samples and will oscillate: fast pages drive the delay down, then a batch of slow pages backs it up. If the fast pages are not the ones you need to throttle for, lower TARGET_CONCURRENCY to give more headroom, or separate fast and slow page types into different crawl jobs with different throttle settings.

Settings cheat sheet

Recommended starting points for three common scenarios:

Scenario	TARGET_CONCURRENCY	MAX_DELAY	START_DELAY
Polite crawl of a single domain	1.0	10	2.0
Faster crawl of a cooperative server	4.0	30	1.0
Multi-domain crawl, many parallel slots	2.0	60	1.0

Enable AUTOTHROTTLE_DEBUG = True, run a short crawl, read the output, and adjust from there.

A note on blocking vs rate limiting

AutoThrottle addresses one problem: sending requests too fast for the server to handle comfortably. It does not address fingerprint-based detection, including TLS fingerprints, HTTP header patterns, browser behavior signatures, or IP reputation. Modern anti-bot systems primarily detect scrapers through these signals, not through request rate alone.

If you are following AutoThrottle's guidance on request rate and still getting blocked, the rate is probably not the issue. The problem is in the request fingerprint, and that requires a different solution.