DEV Community

Cover image for 21 Ways to Avoid Getting Blocked While Web Scraping
Kevin Meneses González
Kevin Meneses González

Posted on Originally published at Medium

21 Ways to Avoid Getting Blocked While Web Scraping

#web #scraping #api #data

Your scraper doesn't fail because of your code.

It fails because it screams "I'm a bot" on every single request.

If you want to avoid getting blocked while web scraping, you need to understand one thing first: modern websites don't block scrapers, they block signals. A missing header. An IP from a datacenter. A request every 200 milliseconds, perfectly timed, 24 hours a day.

If you're:

  • scraping prices or product data,
  • building datasets for AI or RAG pipelines,
  • or monitoring competitors at scale,

this guide is for you.

Below are 21 techniques I use for web scraping without getting blocked, grouped into five blocks, each one with a short Python example you can copy.

TL;DR

  • Why you get blocked: anti-bot systems score four layers of every request: IP, HTTP headers, behavior and browser fingerprint.
  • Quick wins (free): real User-Agent, full browser headers, random delays, backoff on 429, caching, and hidden JSON APIs.
  • Medium effort: rotating proxies, residential IPs, geo-targeting and sticky sessions.
  • Hard part: TLS fingerprints, headless browser detection and CAPTCHAs.
  • Shortcut: a web scraping API like ZenRows handles proxies, fingerprints and anti-bot challenges in one request. mode=auto escalates only when needed, and the free plan includes 5,000 credits/month.

Why Websites Block Web Scrapers

Anti-bot systems like Cloudflare Bot Management, DataDome, Akamai or PerimeterX score every request. When the score crosses a threshold, you get a 403 Forbidden, a CAPTCHA, or worse: a 200 response with fake or empty data.

They look at four things:

  • Your IP: reputation, type (datacenter vs residential), request volume
  • Your HTTP identity: User-Agent, headers, header order, cookies
  • Your behavior: speed, timing patterns, navigation path
  • Your browser fingerprint: TLS handshake, JavaScript execution, canvas, WebGL

Fix one and ignore the others, and you still get blocked.

That's the real problem. Blocking is multi-layered, so your strategy to avoid getting blocked has to be too.

Block 1: HTTP Headers and Identity (Techniques 1–5)

1. Use a real, rotating User-Agent

The default python-requests/2.x User-Agent is the fastest way to get banned. Use real browser strings and rotate them.

import random
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Safari/605.1.15",
    "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0",
]

headers = {"User-Agent": random.choice(USER_AGENTS)}
r = requests.get("https://httpbin.io/headers", headers=headers)
print(r.json())
Enter fullscreen mode Exit fullscreen mode

Keep the list updated. A Chrome version from three years ago is almost as suspicious as no User-Agent at all. 📖 MDN: User-Agent header

2. Send the full set of browser headers

A real browser never sends only a User-Agent. It sends Accept, Accept-Language, Accept-Encoding, Sec-Fetch-* and more.

headers = {
    "User-Agent": USER_AGENTS[0],
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
}
Enter fullscreen mode Exit fullscreen mode

Tip: open httpbin.io/headers in your browser, then call it from your script. Compare both outputs. Every difference is a signal.

3. Keep headers consistent with the User-Agent

If your User-Agent says Firefox but you send Chrome-only headers like sec-ch-ua, you've just flagged yourself. Every header set should match the browser it claims to be.

A simple rule: build header profiles (one per browser) and rotate the full profile, never individual headers. 📖 MDN: Sec-CH-UA client hints

4. Set a realistic Referer

Real users arrive from somewhere. Usually Google, or another page of the same site.

headers["Referer"] = "https://www.google.com/"
Enter fullscreen mode Exit fullscreen mode

For deep pages, set the Referer to the category or listing page that links to them.

5. Persist cookies with a session

Many sites set cookies on the first visit and expect them back. A scraper that never returns cookies looks like a thousand brand-new visitors.

session = requests.Session()
session.headers.update(headers)

session.get("https://example.com/")             # gets cookies
r = session.get("https://example.com/products") # sends them back
Enter fullscreen mode Exit fullscreen mode

📖 Requests docs: Session objects

Headers alone won't help you avoid getting blocked, though.

If every request comes from the same IP, nothing else matters.

Block 2: Rotating Proxies and IPs (Techniques 6–9)

6. Use rotating proxies

Sending thousands of requests from one IP is the oldest red flag in web scraping. Use a proxy pool and rotate IPs.

import itertools

PROXIES = [
    "http://user:pass@proxy1.example.com:8000",
    "http://user:pass@proxy2.example.com:8000",
    "http://user:pass@proxy3.example.com:8000",
]
proxy_pool = itertools.cycle(PROXIES)

for url in urls:
    proxy = next(proxy_pool)
    r = requests.get(url, headers=headers, proxies={"http": proxy, "https": proxy}, timeout=30)
Enter fullscreen mode Exit fullscreen mode

📖 Requests docs: Proxies

7. Use residential proxies for protected sites

Datacenter IPs are cheap and fast, and anti-bot systems know every one of their ranges. Residential proxies use IPs from real ISPs, so they carry much higher trust.

Use datacenter proxies for easy targets. Switch to residential proxies when you start seeing 403 Forbidden errors.

8. Match the IP's geolocation to the target

Scraping a Spanish e-commerce site from a Singapore IP looks odd. It can also return different prices, currencies or content. Pick proxies from the country your target expects.

9. Use sticky sessions for multi-step flows

Rotating proxies are great until you need to log in, paginate or keep a cart. Changing IP mid-session is a classic bot signal. Keep the same IP for the whole flow, then rotate.

Managing a residential proxy pool, geo-targeting and sticky sessions yourself is a real project.

This is where most people quietly give up.

Skip the proxy plumbing
ZenRows gives you rotating residential proxies (premium_proxy), country targeting (proxy_country) and sticky sessions (session_id) as simple API parameters.
→ Try ZenRows for free

Block 3: Scraping Behavior and Rate Limits (Techniques 10–14)

10. Throttle your request rate

Hammering a server is both rude and obvious. Limit concurrency and requests per second per domain.

import time

for url in urls:
    r = session.get(url)
    time.sleep(1.5)
Enter fullscreen mode Exit fullscreen mode

11. Add random delays

Humans are irregular. Bots are perfectly regular. A fixed sleep(1) is a fingerprint in itself.

time.sleep(random.uniform(1.2, 4.5))
Enter fullscreen mode Exit fullscreen mode

12. Back off on 429 and 503

When the server says "slow down", listen. Retrying instantly just confirms you're a bot and gets your IP blocked faster.

def fetch_with_backoff(url, max_retries=5):
    for attempt in range(max_retries):
        r = session.get(url, timeout=30)
        if r.status_code not in (429, 503):
            return r
        wait = int(r.headers.get("Retry-After", 2 ** attempt))
        time.sleep(wait + random.random())
    return r
Enter fullscreen mode Exit fullscreen mode

📖 MDN: 429 Too Many Requests · MDN: Retry-After

13. Respect robots.txt and crawl-delay

Checking robots.txt keeps you away from the traps (honeypot URLs) and the areas the site watches most closely. It's also the ethical baseline for any web scraping project.

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://example.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://example.com/products"))
print(rp.crawl_delay("*"))
Enter fullscreen mode Exit fullscreen mode

📖 Python docs: urllib.robotparser · Google: How robots.txt works

14. Scrape during off-peak hours

Your traffic stands out less when it's spread across the day, and you put less load on the site at night in its local time zone. Big batch jobs belong there.

Behavior is where most DIY scrapers improve the fastest.

But the hardest anti-bot layer is still ahead.

Block 4: Browser Fingerprint and Anti-Bot Detection (Techniques 15–18)

15. Use a headless browser with stealth patches

Plain headless Chrome leaks signals like navigator.webdriver = true. Use stealth patches to hide them.

# pip install playwright playwright-stealth
# playwright install chromium
from playwright.sync_api import sync_playwright
from playwright_stealth import Stealth

with Stealth().use_sync(sync_playwright()) as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")
    print(page.title())
    browser.close()
Enter fullscreen mode Exit fullscreen mode

📖 Playwright for Python · playwright-stealth on PyPI

16. Fix your TLS fingerprint

This one surprises people. Even with perfect headers, the TLS handshake from Python's requests doesn't look like Chrome's. Anti-bot systems check it (JA3/JA4 fingerprints).

curl_cffi impersonates a real browser at the TLS level:

# pip install curl_cffi
from curl_cffi import requests as cffi_requests

r = cffi_requests.get("https://example.com", impersonate="chrome")
print(r.status_code)
Enter fullscreen mode Exit fullscreen mode

📖 curl_cffi on GitHub

17. Render JavaScript only when you need it

Many sites load data with JavaScript, and some anti-bot challenges require JS execution to pass. But a headless browser is slow and expensive. Use it for the pages that need it, not for everything.

18. Avoid CAPTCHAs instead of solving them

If you're seeing CAPTCHAs, the anti-bot system already distrusts you. Solving services work, but they're slow and cost money per solve. Fixing the signals from techniques 1–17 is cheaper than solving the CAPTCHA they trigger.

Stealth browsers and TLS impersonation work.

Until the anti-bot vendor ships an update and you're back to debugging on a Friday night.

Block 5: Smart Scraping Architecture (Techniques 19–21)

19. Look for hidden APIs (XHR) first

Open DevTools → Network → Fetch/XHR and reload the page. Many sites load their data from a JSON endpoint. Calling it directly is faster, cleaner and lighter than parsing HTML.

r = session.get("https://example.com/api/products?page=1")
data = r.json()
Enter fullscreen mode Exit fullscreen mode

📖 Chrome DevTools: Inspect network activity

20. Cache and deduplicate requests

The request you don't send can't get you blocked. Store responses, skip URLs you've already scraped and only refresh what changed.

# pip install requests-cache
import requests_cache

session = requests_cache.CachedSession("scrape_cache", expire_after=3600)
Enter fullscreen mode Exit fullscreen mode

📖 requests-cache docs

21. Use a web scraping API

Techniques 1 to 18 are a full-time job: rotating proxies, header profiles, TLS impersonation, headless browsers, CAPTCHA handling, retries. And all of it breaks whenever the anti-bot vendors change their rules.

A web scraping API bundles all of that behind one request.

After testing several options, ZenRows is the one I reach for when a site fights back. Here's why:

  1. Adaptive Stealth Mode (mode=auto) starts with the cheapest setup and escalates to JS rendering or premium proxies only when needed. You're billed only for the configuration that succeeds. Docs
  2. Residential proxies with geo-targeting via premium_proxy=true and proxy_country (e.g. es, us, de). Docs
  3. JS rendering (js_render=true) plus wait_for and js_instructions for dynamic pages.
  4. Sticky sessions with session_id (any number from 1 to 99999), which keeps the same IP for 10 minutes. Docs
  5. Managed browser headers: ZenRows sets User-Agent, Sec-Ch-Ua, Sec-Fetch-* and similar headers for you, so they always match the fingerprint.
  6. Clean output: CSS extraction to JSON, or Markdown (response_type=markdown) for LLM pipelines.

That single technique covers most of the other 20.

Full Example: Scrape Without Getting Blocked (DIY vs ZenRows)

Let's use ZenRows' own test page, scrapingcourse.com/antibot-challenge, which is protected by an anti-bot challenge. First, the DIY version:

import requests

url = "https://www.scrapingcourse.com/antibot-challenge"
r = requests.get(url, headers={"User-Agent": USER_AGENTS[0]}, timeout=30)

print("Status:", r.status_code)
print("Bypassed:", "You bypassed the Antibot challenge" in r.text)
Enter fullscreen mode Exit fullscreen mode

You'll get the challenge page instead of the content:

Bypassed: False
Enter fullscreen mode Exit fullscreen mode

Now with ZenRows and Adaptive Stealth Mode:

# pip install requests
import requests

params = {
    "url": "https://www.scrapingcourse.com/antibot-challenge",
    "apikey": "YOUR_ZENROWS_API_KEY",
    "mode": "auto",
}

r = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)

print("Bypassed:", "You bypassed the Antibot challenge" in r.text)
print("Cost (USD):", r.headers.get("X-Request-Cost"))
Enter fullscreen mode Exit fullscreen mode 
Bypassed: True
Cost (USD): 0.025
Enter fullscreen mode Exit fullscreen mode

No proxy list. No header profiles. No stealth patches.

The X-Request-Cost header shows what the request cost (0.001 USD = 1 credit), so you can see when the API had to escalate. Two details from the API reference: use a client timeout of 180 seconds, because protected pages can take a while, and don't send js_render or premium_proxy together with mode=auto. Auto mode manages those two for you.

Manual configuration: geo-targeted scraping with CSS extraction

Need control instead of automation? Set the parameters yourself:

params = {
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "apikey": "YOUR_ZENROWS_API_KEY",
    "js_render": "true",
    "premium_proxy": "true",
    "proxy_country": "es",
    "wait_for": ".product",
    "css_extractor": '{"names": ".product-name", "prices": ".price", "links": "a.woocommerce-LoopProduct-link @href"}',
}

r = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
print(r.json())
Enter fullscreen mode Exit fullscreen mode

A few rules worth knowing:

  • wait_for (and wait) only work with js_render=true.
  • proxy_country requires premium_proxy=true.
  • css_extractor returns JSON: one match comes back as a string, several as a list. Add @href or @src to grab attributes. Docs

Sending a Referer or cookies

ZenRows manages browser headers, but you can still send your own Referer or Cookie. Turn on custom_headers and pass them with the normal headers= argument:

params = {
    "url": "https://httpbin.io/anything",
    "apikey": "YOUR_ZENROWS_API_KEY",
    "custom_headers": "true",
}
headers = {"Referer": "https://www.google.com/"}

r = requests.get("https://api.zenrows.com/v1/", params=params, headers=headers, timeout=180)
print(r.text)
Enter fullscreen mode Exit fullscreen mode

Headers like User-Agent or Sec-Fetch-* are ignored here on purpose, so they never contradict the browser fingerprint. Docs

Prefer an SDK? 

# pip install zenrows
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
response = client.get("https://www.scrapingcourse.com/ecommerce/", params={
    "js_render": True,
    "premium_proxy": True,
    "proxy_country": "us",
})
print(response.text)
Enter fullscreen mode Exit fullscreen mode

📖 ZenRows Python SDK quickstart

From here you can build:

  • price monitoring dashboards,
  • lead generation pipelines,
  • datasets for AI agents and RAG systems.

The 21 Techniques to Avoid Getting Blocked at a Glance

# Technique Layer DIY difficulty ZenRows
1 Real, rotating User-Agent HTTP headers Easy ✅ Managed
2 Full browser headers HTTP headers Easy ✅ Managed
3 Headers consistent with UA HTTP headers Medium ✅ Managed
4 Realistic Referer HTTP headers Easy ✅ custom_headers
5 Cookies and sessions HTTP headers Easy Partial (send Cookie via custom_headers)
6 Rotating proxies Proxies Medium ✅ premium_proxy
7 Residential proxies Proxies Hard ✅ premium_proxy
8 Geo-targeted IPs Proxies Medium ✅ proxy_country
9 Sticky sessions Proxies Medium ✅ session_id
10 Rate limiting Behavior Easy Your side
11 Random delays Behavior Easy Your side
12 Backoff on 429/503 Behavior Easy Your side
13 Respect robots.txt Behavior Easy Your side
14 Off-peak scraping Behavior Easy Your side
15 Stealth headless browser Fingerprint Hard ✅ js_render
16 TLS fingerprint Fingerprint Hard ✅
17 Selective JS rendering Fingerprint Medium ✅ mode=auto
18 Avoid CAPTCHAs Fingerprint Hard ✅
19 Hidden APIs (XHR) Architecture Medium Partial (json_response, needs js_render)
20 Caching Architecture Easy Your side
21 Web scraping API Architecture Easy ✅

Key Takeaways

  • Anti-bot systems score signals across four layers (IP, headers, behavior, fingerprint). Fixing only one won't stop you from getting blocked.
  • Behavior and architecture (techniques 10–14, 19–20) are cheap wins you should always own.
  • For rotating proxies, fingerprints and CAPTCHAs, a web scraping API like ZenRows saves weeks of maintenance.

FAQs

❓ How do I avoid getting blocked while web scraping?
✅ Combine several techniques: real browser headers, rotating residential proxies, random delays with backoff on 429 errors, and a TLS fingerprint that matches a real browser. If the site uses serious anti-bot protection, a web scraping API like ZenRows handles those layers for you.

❓ Why do I get a 403 Forbidden error when web scraping?
✅ A 403 usually means the anti-bot system flagged your request. The most common causes are a default Python User-Agent, missing browser headers, a datacenter IP or a non-browser TLS fingerprint. Fix headers first, then move to residential proxies.

❓ Is web scraping legal?
✅ Scraping publicly available data is generally legal in many jurisdictions, but it depends on the data, the site's terms and local laws (personal data under GDPR, for example). Avoid scraping behind logins without permission, respect robots.txt and check with a lawyer for commercial projects.

❓ How many requests per second can I send without getting blocked?
✅ There's no universal number. Start around one request every 1–3 seconds per domain with random jitter, watch for 429 responses and adjust. Rotating proxies let you spread volume, but they don't make aggressive scraping invisible.

❓ Do I need residential proxies for web scraping?
✅ Not always. Datacenter proxies work fine for many sites. You need residential proxies when the target uses serious anti-bot protection (Cloudflare, DataDome, Akamai) or blocks datacenter IP ranges.

❓ Is there a free web scraping API?
✅ ZenRows has a free plan with 5,000 credits per month and no credit card required. A standard request costs 1 credit, JS rendering 5, premium proxies 10, and both together 25. See pricing.

Useful Resources

Final Thoughts

Avoiding blocks isn't about one clever trick.

It's about not giving the anti-bot system a single reason to doubt you. You can build that yourself, layer by layer, or hand the hardest layers to a web scraping API and focus on the data.

Stop fighting anti-bot systems
Rotating residential proxies, JS rendering and anti-bot bypass in one request. 5,000 free credits per month, no card required.
→ Start scraping with ZenRows

More practical guides on Python, APIs and automation
Tutorials, tools and real workflows for developers.
→ Visit kevinmeneses.com

Looking for technical content for your company? I can help · LinkedIn · kevinmenesesgonzalez@gmail.com

Top comments (0)