scrapy-stealth – Layered anti-bot engine for Scrapy (HTTP stealth HTTP browser)

Fawad — Tue, 04 Aug 2026 13:55:10 +0000

I built this after repeatedly hitting bot detection (Akamai Bot Manager, Cloudflare) while scraping product catalogs from e-commerce/distributor sites for data pipelines I run — RS Components, DigiKey UK, Euspares, and others.

The core problem: full browser automation works against strong bot detection but is slow and resource-heavy on every request. Plain HTTP is fast but gets blocked immediately on protected sites. Most tools pick one and stick with it.

scrapy-stealth escalates through three layers instead:

Plain HTTP — for sites with no real protection
Stealth HTTP — fingerprint/header spoofing for moderate protection
Full browser automation — only as a last resort, for heavy protection

It only pays the browser-automation cost when the cheaper layers actually fail.

A few bugs along the way that might be interesting to others working on similar tooling:

An event loop teardown race on Windows that only reproduced in production, never locally
Websocket listener races causing intermittent hangs during browser automation
A recursion bug specific to headless browser automation on newer Python versions (headful was the stable workaround)
Temp browser profile accumulation on long-running jobs

It's MIT licensed, in beta (0.6.x), on PyPI: pip install scrapy-stealth

GitHub: https://github.com/fawadss1/scrapy-stealth

Happy to answer questions about the architecture or any of the debugging above.

DEV Community: Fawad

scrapy-stealth – Layered anti-bot engine for Scrapy (HTTP stealth HTTP browser)