I built this after repeatedly hitting bot detection (Akamai Bot Manager, Cloudflare) while scraping product catalogs from e-commerce/distributor sites for data pipelines I run — RS Components, DigiKey UK, Euspares, and others.
The core problem: full browser automation works against strong bot detection but is slow and resource-heavy on every request. Plain HTTP is fast but gets blocked immediately on protected sites. Most tools pick one and stick with it.
scrapy-stealth escalates through three layers instead:
Plain HTTP — for sites with no real protection
Stealth HTTP — fingerprint/header spoofing for moderate protection
Full browser automation — only as a last resort, for heavy protection
It only pays the browser-automation cost when the cheaper layers actually fail.
A few bugs along the way that might be interesting to others working on similar tooling:
An event loop teardown race on Windows that only reproduced in production, never locally
Websocket listener races causing intermittent hangs during browser automation
A recursion bug specific to headless browser automation on newer Python versions (headful was the stable workaround)
Temp browser profile accumulation on long-running jobs
It's MIT licensed, in beta (0.6.x), on PyPI: pip install scrapy-stealth
GitHub: https://github.com/fawadss1/scrapy-stealth
Happy to answer questions about the architecture or any of the debugging above.
Top comments (0)