Two incidents look the same on most scraping dashboards.
In the first, one spider starts failing because the site changed a CSS class. In the second, forty spiders start failing in the same hour because a bot management vendor pushed a new model. Both show up as a drop in items and a rise in "blocked". The fix for the first is a code change. The fix for the second is to change identity and wait, and touching the spider code will make it worse.
The second kind is more common than it should be, and State of Web Access, the audit Zyte (where I work) ran on 11,100 landing pages, has the numbers on why.
One vendor decides what a browser is
18.5% of landing pages run a dedicated bot manager, the kind that looks at behaviour, canvas rendering and attribute consistency rather than just IPs. One vendor is 75.9% of those deployments. The next four combined are under a quarter.
The TLS layer is more concentrated still. Of the TLS-fingerprinting sites the audit could attribute, the same vendor is 88.8%.
And at the firewall layer, 61.9% of sites run a single WAF vendor in its default mode. 1.4% run more than one. Roughly three in ten sites run a WAF and nothing else, and that WAF arrived bundled with the CDN. Nobody chose it.
Put those together and for most of the defended web, one company's model of what a real browser looks like is the definition of a real browser. When that model changes, the change lands everywhere at once. Your spiders are not failing independently. They are failing together, for one reason, and the reason is not in your repository.
Why a "blocked" counter cannot see this
Most Scrapy projects I have looked at count blocks in one bucket. Maybe two, split by status code. That hides the two things you need to know when the pager goes off: which layer refused you, and whether it is one domain or all of them.
The audit found the same four layers over and over. TLS, which shows up as a connection reset or an empty 403. WAF, which shows up as a 403 with a block page. Challenge, which shows up as a 200 or 403 carrying a JavaScript challenge. And rendering, where the response is fine but the data is not in it. Each has a different fix. A single counter makes them all look like the same fire.
Middleware that counts by layer and domain
This is a Scrapy downloader middleware. It classifies each response by the wording of the page and increments a stat per layer and per domain. Connection-level failures get their own bucket, because that is where TLS filtering hides.
import re
from urllib.parse import urlparse
from scrapy import Request
from scrapy.http import Response
from twisted.internet.error import ConnectionLost, ConnectionRefusedError, TimeoutError
SIGNATURES = {
"challenge/js": [r"checking your browser", r"enable javascript and cookies",
r"just a moment", r"verify you are human"],
"challenge/captcha": [r"captcha", r"are you a robot", r"unusual traffic"],
"waf/block-page": [r"access denied", r"request blocked", r"you have been blocked",
r"reference\s*(#|number|id)"],
}
CONNECTION_ERRORS = (ConnectionLost, ConnectionRefusedError, TimeoutError, OSError)
def classify(response: Response) -> str | None:
body = response.text[:50_000] if hasattr(response, "text") else ""
for label, patterns in SIGNATURES.items():
if any(re.search(p, body, re.I) for p in patterns):
return label
if response.status == 429:
return "ratelimit/429"
if response.status == 403:
return "waf/unattributed"
if response.status == 200 and len(body) < 2_000:
return "render/empty"
return None
class BlockLayerStatsMiddleware:
def __init__(self, stats):
self.stats = stats
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.stats)
def _record(self, layer: str, url: str) -> None:
domain = urlparse(url).netloc
self.stats.inc_value(f"blocked/{layer}")
self.stats.inc_value(f"blocked/{layer}/{domain}")
def process_response(self, request: Request, response: Response, spider):
layer = classify(response)
if layer:
self._record(layer, response.url)
return response
def process_exception(self, request: Request, exception, spider):
if isinstance(exception, CONNECTION_ERRORS):
# TLS filtering often looks like a reset, not a status code.
self._record("tls-or-network/reset", request.url)
return None
Enable it in settings:
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.BlockLayerStatsMiddleware": 543,
}
The stats land in the crawl's stats collector, so they end up wherever you already ship Scrapy stats. If you use Spidermon, that is where the rules below go.
Two rules that tell the incidents apart
The first rule is per spider. If blocked/* for one domain rises past a threshold and other domains are quiet, it is that site. Somebody changed a selector or added a rule. Open the spider.
The second rule is fleet-wide. If the same layer, say challenge/js, rises across three or more domains inside ten minutes, it is not your code. It is a vendor change. Do not deploy anything. Rotate identity, drop concurrency, and give it a few hours before anyone edits a spider. The person who owns the fetch layer gets paged, not the person who owns the parser.
The layer name in the stat key is what makes the second rule possible. blocked/403 across forty domains could be forty separate problems. blocked/challenge/js across forty domains is one problem.
The unattributed bucket is not a bug
You will see a lot of waf/unattributed. The audit could only attribute 45% of TLS-fingerprinting sites to any vendor, and plenty of block pages carry no recognisable wording at all. Keep the bucket. A spike in unattributed 403s across many domains is still a correlated failure, and the replay test from the first post in this series, one request from a different IP class, tells you whether it is reputation or something else.
The vendor shares by industry are on the antibot page and the WAF breakdown is on the WAF page. Adult content, furniture and gambling lead bot management adoption at 30% and above. Those are sectors with checkout flows and inventory, and the bot manager is guarding revenue, not text.
Disclosure: I work at Zyte. The audit is ours. The middleware and the opinions are mine.
Top comments (0)