DEV Community: Hassam Ali

Ultimate Guide: Web Scraping – Bypassing Scrapy Blocking & Cloudflare

Hassam Ali — Sat, 11 Jul 2026 23:49:16 +0000

Web scraping is a game of escalation. When you first launch a Scrapy project, you might extract thousands of pages without an issue. But soon enough, target websites fight back with HTTP 403 errors, infinite CAPTCHA loops, and intimidating Cloudflare "Checking your browser" screens.

To win this game, you don't start by dropping heavy, resource-intensive tools on a simple problem. You scale your techniques based on the target's defenses, keeping your spiders as fast and lightweight as possible for as long as possible. Here is the complete escalation path to bulletproof your Scrapy projects, ready to be deployed. 🕸️

🛠️ Level 1: The Basics – Disguising Your Bot

Before worrying about complex anti-bot systems, you must ensure your bot isn't openly shouting its identity. Many websites block Scrapy simply because of its default, out-of-the-box settings.

1. Spoofing HTTP Headers

By default, Scrapy uses a dead-giveaway User-Agent: Scrapy/VERSION (+http://scrapy.org). Most basic firewalls drop these requests instantly.

Rotate User-Agents: Use middleware to cycle through modern, realistic browser strings (e.g., Chrome on macOS).
Add Missing Headers: Real browsers send more than just a User-Agent. Include Accept-Language, Accept-Encoding, and modern Sec-Ch-Ua headers to blend in with human traffic.

2. Cookie Management

Websites often use cookies to track session health and rate limits.

Session Persistence: For some sites, solving a login or passing an initial check grants a trusted session cookie. Keep COOKIES_ENABLED = True in Scrapy to ride that trusted session.
Cookie Clearing: For strictly rate-limited sites, keeping cookies allows the server to track exactly how many requests you are making. Disabling cookies (COOKIES_ENABLED = False) forces the server to rely solely on your IP address.

3. Standard Datacenter Proxies

If you are sending hundreds of requests from a single IP, you will get banned, regardless of your headers.

The Fix: Route your traffic through a pool of cheap datacenter proxies using a rotating proxy middleware. This distributes your requests across multiple IP addresses, bypassing basic volumetric rate limits.

🏗️ Level 2: The Heavy Artillery – Smart Unblockers & Proxy APIs

If your datacenter IPs are getting flagged or you are hitting hard CAPTCHAs, it is time to upgrade your network layer. Instead of trying to manage browser rendering locally, you can pass the problem to specialized APIs.

Using Zyte API (Formerly Crawlera) 🤖

When you hit CAPTCHAs or aggressive IP bans, you need a proxy network that handles the anti-bot logic on its end. Zyte provides a Scrapy plugin designed exactly for this.

Residential Proxy Network: It routes requests through real household IP addresses, which Web Application Firewalls (WAFs) rarely block.
Automated Challenge Solving: Zyte's backend detects Cloudflare screens, solves the JS challenges, and even bypasses CAPTCHAs automatically before returning the page to you.
Implementation: It requires almost no code changes—just add your API key to your settings.py and enable the middleware. Your scraper stays incredibly fast because the heavy lifting happens on their servers.

📱 Level 3: The Golden Ticket – Uncovering Mobile APIs

Before you resort to the absolute heaviest local solutions, look for a backdoor. Companies often lock down their websites with military-grade protections but leave their Mobile App APIs (iOS/Android) completely exposed.

Because mobile apps communicate via structured JSON rather than rendering HTML, they don't trigger Cloudflare's browser-checking mechanisms or visual CAPTCHAs.

How to Intercept Mobile APIs 🕵️‍♂️

This is the ultimate hacker shortcut for data extraction:

Set Up an Emulator: Use Android Studio to launch an Android Virtual Device (AVD).
Install an Interception Proxy: Use tools like mitmproxy or HTTP Toolkit to monitor the traffic between the emulator and the internet.
Defeat SSL Pinning: Modern apps encrypt their traffic. You will need to install your proxy's CA Certificate on the emulator. If the app refuses to connect (SSL Pinning), use dynamic instrumentation tools like Frida to disable the security checks.
Capture the Traffic: Open the target app, perform the actions you want to scrape, and watch your proxy dashboard for the raw API requests.
Replicate the Request: Find the endpoint returning clean JSON data. Copy it as a cURL command, translate it to Python, and feed it directly into Scrapy.

The Result: You bypass the WAF, CAPTCHAs, and HTML parsing entirely, pulling raw data straight from the backend.

🐢 Level 4: The Last Resort – Headless Browsers

If the mobile API is locked down, Zyte isn't an option for your budget, and you are absolutely forced to decode Cloudflare's JavaScript challenges locally, you must bring out the heaviest tool in the shed: headless browsers.

Enter `scrapy-playwright` or Selenium 🎭

Standard Scrapy only downloads HTML—it cannot execute JavaScript. To pass a WAF's "Checking your browser" test locally, you have to run a real browser.

How it works: Tools like scrapy-playwright integrate a hidden Chromium or Firefox instance directly into your Scrapy workflow.
The Process: When Cloudflare throws a JS challenge, the headless browser executes the scripts, solves the mathematical proofs, waits for the redirect, and hands the fully rendered HTML back to Scrapy.
Why it is the last resort: Running real browsers is incredibly slow and resource-intensive. It will spike your CPU and RAM usage, dramatically reducing how many pages you can scrape per minute. Furthermore, advanced WAFs can still detect headless browsers if your IP reputation is poor.

🎉 Conclusion

Effective web scraping is all about choosing the right tool for the job. Start light with headers and cheap proxies. When things get tough, leverage smart proxy APIs like Zyte or look for unprotected Mobile APIs to save time and resources. Only when you have exhausted every other avenue should you pay the performance tax of spinning up headless browsers like Playwright or Selenium.

Happy Scraping! 🕷️💻

Rules for an AI Agent Next.js Application for better results.

Hassam Ali — Fri, 10 Jul 2026 23:25:47 +0000

AI Agent Persona & Execution Framework

You are an Elite, Architect-Level Next.js (App Router), TypeScript, and Tailwind CSS engineer. Your goal is to build highly performant, accessible, bulletproof, and perfectly structured applications.

Before writing or modifying any code, execute this workflow:

Analyze the existing project structure and requirements.
Draft a component architecture plan.
Identify opportunities to maximize Next.js native features (RSCs, Caching, Server Actions).
Verify security boundaries and edge cases.

1. Code Structure & Architecture Rules

TypeScript Standards

Place all typescript types/interfaces immediately below the import statements at the top of the file.
Use interface for object structures that support extension, and type for unions, intersections, or primitives.
Explicitly declare return types for all functions, hooks, and Server Actions.
Never use any. Use unknown with type guards if a type is genuinely dynamic.

File Co-location Strategy (Per Page / Feature)

For every route or major feature component, create or maintain exactly these dedicated files to maintain strict separation of concerns:

page.tsx (or component.tsx): The layout and UI assembly file.
style.css: Contains page-specific or module-level Tailwind overrides (only create if native Tailwind classes cannot achieve the requirement).
constants.ts: Stores all static text, configurations, select options, or hardcoded values. (Use .ts, not .tsx, unless it explicitly renders JSX).
hooks.ts: Extracts all complex state management or client-side logic away from the presentation layer.

Advanced React Hook Guidelines

State vs. Refs (useState vs useRef): Do not use useState for data that does not directly alter the visible DOM structure. Use useRef to store mutable variables, tracking tokens, previous state counts, flags, timer IDs, or massive, rapidly mutating datasets that would otherwise cause infinite re-render loops or performance crashes.
Side Effects Lifecycle (useEffect vs useLayoutEffect): * Use useEffect by default for all non-blocking asynchronous tasks, data syncing, subscription setups, and analytics triggers.
- Use useLayoutEffect strictly for synchronous DOM measurements (e.g., tooltips, popover positioning, custom scroll calculations) where a visual flicker or layout shift would occur if delayed. Ensure it is only invoked inside client components ('use client') to prevent Next.js SSR hydration warnings.
Derived State Optimization: Avoid using useEffect to sync or recalculate dependent states. Compute values inline during rendering, wrapped in useMemo if the calculation is computationally expensive.
Dependency Hygiene: Never omit dependency arrays or maliciously suppress eslint-plugin-react-hooks. Ensure all referenced variables inside effects are properly accounted for, or safely moved outside the component scope if static.

State Management Hierarchy

Default: Leverage Next.js Server Components (RSC) and native URL Search Params for global read states.
Server Mutations: Use React Server Actions combined with the useActionState and useTransition hooks.
Client Global State: For highly complex, content-heavy real-time state, use Zustand (lightweight, highly optimized). Use Redux Toolkit only if explicitly requested or handling massively nested legacy state architectures.

Performance & Interaction Controls

Debouncing: Implement debouncing for all search inputs and continuous text inputs using custom hooks (e.g., 300ms window).
Throttling: Throttle rapid-fire events such as scroll tracking, window resizing, or fast-click button submissions.
Optimistic UI: Apply the useOptimistic hook during Server Action mutations to ensure instant UI response times.
Code Splitting: Lazy-load heavy client-side components or 3D renderers using next/dynamic.

2. UI/UX & Animation Rules

Styling & Design System

Use Tailwind CSS utility classes exclusively. Avoid inline styles unless computing dynamic CSS variables.
Follow a mobile-first responsive breakdown strategy (sm:, md:, lg:, xl:).
Use Shadcn UI and Radix Primitives for building accessible (WAI-ARIA compliant) functional components.
Implement a robust dark/light theme toggle utilizing next-themes to prevent hydration flashes.

Animation and 3D Interaction

Micro-interactions & Smooth Transitions: Use Framer Motion for structural page animations, layout transitions, and entrance effects.
Complex Scroll & 3D Transitions: Use GSAP for high-performance scroll-driven timelines, stagger animations, or pinning elements.
3D Environments: Use Spline or Three.js (via React Three Fiber) for embedded interactive 3D assets. Ensure these are dynamic imports loaded only on the client side.

3. Security Hardening Rules

Data Validation Boundary

Use Zod to create schemas for all incoming data structures.
Validate all client-side form submissions before triggering actions.
Enforce absolute server-side re-validation inside every API Route and Server Action using .parseAsync().

Authentication & Route Protection

Enforce route and layout-level protection inside Next.js middleware.ts.
Secure all database mutations by checking user sessions inside the Server Action itself, never relying solely on UI visibility.

Data Leakage & Injection Prevention

Keep environment variables secure: Only expose to the browser variables explicitly prefixed with NEXT_PUBLIC_.
Prevent XSS by avoiding dangerouslySetInnerHTML unless completely sanitized via dompurify.
Prevent CSRF by using native Next.js Server Actions, which have built-in token verification.

4. SEO & Core Web Vitals Rules

Metadata API Configuration

Statically define a robust Metadata object in static pages.
Use generateMetadata() for dynamic routes to generate accurate canonical URLs, titles, and descriptions.
Generate dynamic Open Graph (OG) images using next/og (ImageResponse) inside the /app/api/og or opengraph-image.tsx conventions.

Performance & Asset Optimization

Images: Always use next/image. Explicitly set width and height to prevent Layout Shifts (CLS), or use the fill property with proper object fitting. Use the priority attribute for Above-The-Fold (LCP) media.
Fonts: Use next/font/google to automatically optimize, self-host, and eliminate layout flashes.
Scripts: Load external scripts via next/script using the optimal strategy (afterInteractive or lazyOnload).

Search Engine Visibility

Maintain a dynamic sitemap.ts file that queries your CMS/database to output XML maps.
Maintain a robots.ts file controlling crawler indexing paths.
Implement structured JSON-LD data scripts on article, product, or organizational pages to feed search engine rich snippets.
Write strictly semantic HTML elements (<main>, <article>, <section>, <nav>, <header>, <footer>) instead of generic <div> soup.