DEV Community

Cover image for Debugging 403, 429 and CAPTCHA Errors with Proxies
Dev Stack Hub
Dev Stack Hub

Posted on

Debugging 403, 429 and CAPTCHA Errors with Proxies

Your scraper worked fine on 50 requests. At 500, it started returning 403 Forbidden. At 5,000, you're staring at 429 Too Many Requests and CAPTCHA pages with a 200 status code. Adding more proxies didn't help, or helped for ten minutes.

The mistake most developers make here is changing several things at once: new proxies, new headers, more retries. This guide gives you a repeatable way to find out why you're being blocked so that you can fix the right thing.

What we'll cover

  • What 403, 429, and CAPTCHA responses actually mean
  • A step-by-step isolation process (and a small Python probe script)
  • Detecting "soft blocks" that return HTTP 200
  • Backoff, Retry-After, and pacing done properly
  • Tracking proxy health so bad IPs get benched automatically
  • When the problem isn't the IP at all

Examples use Python and requests, with placeholder proxies from the documentation range (203.0.113.x).

What each signal usually means

Signal What the server is telling you Typical root cause
429 Too Many Requests You exceeded a rate limit Too many requests per IP, per session or per account in a time window
403 Forbidden The server understood you and refuses IP reputation, bot detection, missing or inconsistent headers, geo restriction
200 with a CAPTCHA page A challenge is served instead of content Risk score too high: IP, behavior or fingerprint looks automated
407 Proxy Authentication Required The proxy rejected you, not the target Wrong credentials or non-whitelisted IP

Note that 407 is not a block at all. It's your proxy configuration, and it's the first thing to rule out.

Step 1: Isolate where the block happens

Before touching your scraper, test the target with the simplest possible client. If curl through the proxy works but your script fails, the problem is in your code. If both fail, it's the IP or the target.

# Direct (no proxy)
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/products

# Through the proxy
curl -s -o /dev/null -w "%{http_code}\n" -x http://user:pass@203.0.113.10:8080 https://example.com/products
Enter fullscreen mode Exit fullscreen mode

Then reproduce the comparison in Python with a small probe that varies one thing at a time:

import requests

TARGET = 'https://example.com/products'
PROXY = 'http://user:pass@203.0.113.10:8080'
PROXIES = {'http': PROXY, 'https': PROXY}

BROWSER_HEADERS = {
   'User-Agent': (
       'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
       '(KHTML, like Gecko) Chrome/124.0 Safari/537.36'
   ),
   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
   'Accept-Language': 'en-US,en;q=0.9',
}

def probe(label, **kwargs):
   try:
       r = requests.get(TARGET, timeout=(5, 15), **kwargs)
       print(f'{label:<28} {r.status_code}  {len(r.content):>8} bytes')
   except requests.RequestException as exc:
       print(f'{label:<28} ERROR {type(exc).__name__}')

probe('direct, default headers')
probe('direct, browser headers', headers=BROWSER_HEADERS)
probe('proxy, default headers', proxies=PROXIES)
probe('proxy, browser headers', proxies=PROXIES, headers=BROWSER_HEADERS)
Enter fullscreen mode Exit fullscreen mode

Read the results like a truth table:

  • Direct works, proxy fails: the proxy IP is the problem (reputation, type, or geo).
  • Default headers fail, browser headers pass: the site checks the User-Agent and friends. The default python-requests/x.y agent is an easy signal to filter.
  • Everything fails: the page may need JavaScript, a cookie, or a login. The proxy is not the issue.
  • Everything passes: your problem is volume or pacing, so move on to Step 3.

Also confirm the proxy really is being used by calling an IP-echo endpoint such as https://api.ipify.org?format=json through it. A surprising number of "proxy problems" are requests silently going out directly.

Step 2: Detect soft blocks

A response code of 200 doesn't mean you got the page. Many sites serve a CAPTCHA or an "access denied" page with a success status, and your parser then silently stores garbage. Classify every response before you trust it:

BLOCK_MARKERS = (
   'captcha',
   'unusual traffic',
   'access denied',
   'verify you are human',
   'are you a robot',
)

def classify(response):
   if response.status_code == 407:
       return 'proxy_auth'
   if response.status_code == 429:
       return 'rate_limited'
   if response.status_code in (401, 403):
       return 'blocked'

   snippet = response.text[:5000].lower()
   if any(marker in snippet for marker in BLOCK_MARKERS):
       return 'challenge'
   return 'ok'
Enter fullscreen mode Exit fullscreen mode

The marker list is site-specific, so tune it to the challenge pages you actually see. A cheaper complementary check is to assert that the content you expect (a product title, a JSON key) is present, and treat its absence as a failure.

Step 3: Fix 429s with backoff and pacing

A 429 is the friendliest error you'll get, because the server is telling you exactly what to do: slow down. Many servers include a Retry-After header, either as several seconds or as an HTTP date. Honor it:

import random
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def retry_after_seconds(response, default=30):
   value = response.headers.get('Retry-After')
   if not value:
       return default
   if value.isdigit():
       return int(value)
   try:
       when = parsedate_to_datetime(value)
       return max(0, (when - datetime.now(timezone.utc)).total_seconds())
   except (TypeError, ValueError):
       return default

def polite_sleep(base=1.0, jitter=1.5):
   # randomized delay between requests, so traffic isn't perfectly periodic
   time.sleep(base + random.uniform(0, jitter))
Enter fullscreen mode Exit fullscreen mode

Three principles prevent most 429s:

  1. Limit concurrency per IP, not just overall. Ten threads on one IP is a burst. Ten threads across ten IPs is not.
  2. Add jitter. A request every exactly 1.000 seconds looks robotic.
  3. Use exponential backoff on repeated failures. If a 429 comes back after waiting, wait longer next time instead of retrying at the same speed.

If you're still rate-limited at a sensible pace, the limit probably applies per IP, and the fix is a larger pool rather than a faster loop.

Step 4: Track proxy health and bench bad IPs

Random rotation keeps sending traffic to IPs that are already burned. Instead, record outcomes and give failing proxies a cooldown:

import random
import time
from collections import defaultdict

class ProxyPool:
   def __init__(self, proxies, cooldown=300, max_failures=3):
       self.proxies = list(proxies)
       self.cooldown = cooldown
       self.max_failures = max_failures
       self.failures = defaultdict(int)
       self.benched_until = {}

   def get(self):
       now = time.time()
       available = [p for p in self.proxies if self.benched_until.get(p, 0) <= now]
       if not available:
           raise RuntimeError('No healthy proxies: slow down or add more IPs')
       return random.choice(available)

   def report(self, proxy, outcome):
       if outcome == 'ok':
           self.failures[proxy] = 0
           return
       self.failures[proxy] += 1
       # a 429 benches the IP immediately; other failures need a streak
       if outcome == 'rate_limited' or self.failures[proxy] >= self.max_failures:
           self.benched_until[proxy] = time.time() + self.cooldown
           self.failures[proxy] = 0
Enter fullscreen mode Exit fullscreen mode

And a fetch function that ties the pieces together:

import requests

def fetch(url, pool, attempts=4):
   for _ in range(attempts):
       proxy = pool.get()
       try:
           r = requests.get(
               url,
               proxies={'http': proxy, 'https': proxy},
               headers=BROWSER_HEADERS,
               timeout=(5, 15),
           )
       except requests.RequestException:
           pool.report(proxy, 'error')
           continue

       outcome = classify(r)
       pool.report(proxy, outcome)

       if outcome == 'ok':
           return r
       if outcome == 'rate_limited':
           time.sleep(retry_after_seconds(r))
       elif outcome == 'proxy_auth':
           raise RuntimeError('Proxy credentials rejected (407). Check auth or IP whitelist.')

       polite_sleep()

   raise RuntimeError(f'Failed to fetch {url} after {attempts} attempts')
Enter fullscreen mode Exit fullscreen mode

Log the outcome, proxy, and URL for every request. After a few hundred requests, you'll be able to see whether failures cluster on specific IPs (bad IPs), specific URLs (protected pages), or specific times (rate-limit windows). That pattern tells you which fix to apply.

Step 5: When it's the IP type

If your probe showed that browser-like headers pass directly but fail through the proxy, you're dealing with IP reputation. Different proxy types carry different levels of trust:

Proxy type Trust with strict sites Strengths Watch out for
Datacenter / IPv6 Lower Fast, cheap, easy to scale Whole ranges are often flagged; shared IPs inherit other users' behavior
ISP (static residential) Higher Looks like a home user, stable for long sessions Priced per IP, so pool size is limited by budget
Residential (rotating) High Huge pools, precise geo-targeting Usually billed per GB; can be slower
Mobile (4G/5G) Highest Carrier IPs are shared by many real users, so sites are cautious about blocking them Most expensive; speed varies

A few related checks:

  • Dedicated vs shared: a shared IP can be burned by someone else before you send a single request.
  • Geo mismatch: if your Accept-Language says en-US but your exit IP is in another country, or your IP is in Germany while you request a US-only price page, you will get blocked or served different content.
  • Burned static IPs: a static IP that has been hammered for days may need a rest or a replacement. Rotation isn't just for volume; it's hygiene.

Step 6: When it isn't the IP at all

If you've tried clean IPs and sensible pacing and still get 403s, look beyond the proxy:

  • Session and cookies: Many sites set a cookie on the first page and expect it on the next. Use a requests.Session and keep the same IP for the whole flow (a sticky session if your provider supports one).
  • Inconsistent headers: A Chrome User-Agent with a missing Accept-Language, or headers in an unusual order, is a signal. Keep the set realistic and consistent per session.
  • TLS fingerprinting: Some bot-protection systems inspect the TLS handshake itself, and Python's default stack looks different from a real browser's. If headers and IPs are fine but a site still refuses you, a real browser (Playwright or Selenium) or an HTTP client that mimics browser TLS such as curl_cffi may be the next thing to test.
  • JavaScript-rendered content: If the data is loaded by JavaScript, plain requests get an empty shell. Look for the JSON API call the page makes in your browser's network tab, or use a headless browser.
  • Behavioral signals: Hitting deep URLs with no prior navigation, or requesting pages at inhuman speed, can raise a risk score even from a perfect IP.

A word on CAPTCHAs

A CAPTCHA is not an error to defeat. It's a signal that your traffic looks risky. The most reliable strategy is to trigger fewer of them: better IP quality, lower rate, consistent sessions, realistic headers. If you're seeing them constantly, treat that as a symptom and work back through the steps above.

It's also worth asking whether you should be scraping that page at all. Check for an official API or data export first, read the site's terms of service and robots.txt, and only collect data you're permitted to use. Circumventing access controls on sites that forbid automated access can create legal problems that no proxy will solve.

Choosing proxies for testing

To follow along, you need proxies that let you compare types on the same target. I used a provider that offers datacenter, ISP, residential, and mobile IPs with both HTTP/HTTPS and SOCKS5 so that I could swap types without changing code. Disclosure: [add your affiliate or sponsorship note here if applicable]. Whichever service you use, buy a small amount of each type you're considering, run the probe script from Step 1, and compare success rate and latency on your target before committing to a larger plan.

Quick reference

  • 407? Fix proxy auth first. It isn't a block.
  • Isolate: compare direct vs proxy, default vs browser headers.
  • Classify every response, including 200 pages that are really CAPTCHAs.
  • 429: honor Retry-After, add jitter, limit per-IP concurrency, then grow the pool.
  • Bench bad IPs with a cooldown instead of rotating unquestioningly.
  • 403 on clean IPs: check sessions, headers, TLS fingerprint, and JavaScript rendering.
  • Constant CAPTCHAs: reduce the triggers, and check whether an official API exists.

Got a blocking pattern that isn't covered here? Share it in the comments, and I'll add it to the guide.

Top comments (19)

Collapse
 
andersonkevin profile image
Kevin Anderson •

The “change one variable at a time” framing is what most scraping tutorials skip, and the direct vs. proxy, default vs. browser headers truth table is a quick way to see where the block actually is.

Two things I especially liked:

Soft-block detection. A 200 response that is really a CAPTCHA page quietly corrupts your data, and classify() catches that before it reaches the parser. Pairing it with a check that the expected content is present would make it sturdier still.
Benching proxies instead of rotating blindly. Random rotation keeps feeding traffic to burned IPs, and the cooldown approach fixes that cleanly.

One thing I’d add: confirm the proxy is really being used (the ipify check) before blaming it. I’ve seen requests silently go out direct because of a misconfigured proxies dict.

Curious how you’d handle sticky sessions with rotating residential proxies when a flow spans several pages. Do you pin one IP per session, or re-authenticate when it changes?

Collapse
 
devstackhub profile image
Dev Stack Hub • • Edited

Thanks for the thoughtful read, and good call on the ipify check. A silently misconfigured proxies dict is one of the most common reasons people end up blaming the proxy for something that never went through it. The usual culprit is a dict keyed only on "http" while the target is HTTPS` so requests quietly go out direct. I should have made that check step zero in the post.

Agreed on pairing classify() with an expected-content check too. Status code and block-page markers catch the obvious cases, but asserting that a known selector or field is present catches the subtle ones, like a trimmed page or a "lite" response served to suspected bots.

On sticky sessions: I pin one IP per flow. For anything spanning multiple pages (login, search, checkout, pagination with session state), the site ties cookies, CSRF tokens, and often fingerprint signals to the IP, so a mid-flow change usually invalidates the session or trips risk scoring. Most residential providers support this by embedding a session ID in the proxy username, which holds the same exit IP for a set window (typically 1 to 30 minutes, depending on the provider).

My rules of thumb:

  • One session ID per logical flow, with a fresh one for each new flow.
  • Keep the cookie jar and headers tied to that session, so IP, cookies, and fingerprint all rotate together, never separately.
  • Set the sticky window longer than your slowest expected flow, with some margin.
  • If the IP drops mid-flow anyway, don't try to re-authenticate and continue. Treat the flow as failed, discard the session state, and restart it cleanly on a new IP. Resuming on a new IP with old cookies is a classic way to get flagged.
  • Bench the session, not just the IP. If a session ID starts returning soft blocks, retire it using the same cooldown logic.

Re-authenticating is a fallback for flows that are expensive to restart, but I'd treat it as the exception.

How are you handling it on your end? Are your flows short enough that sticky windows aren't an issue?

Collapse
 
mayur-upadhyay profile image
Mayur Upadhyay •

The truth-table approach in Step 1 is a clean way to separate proxy issues from code issues.

A few technical details that have bitten me in similar setups:

  1. Exponential backoff isn’t in the code yet. The Step 3 text recommends it, but fetch() only sleeps for Retry-After (or the 30s default) and then polite_sleep(). If a server keeps returning 429 without a Retry-After header, every retry waits the same amount. Something like delay = min(60, 2 ** attempt) + random.uniform(0, 1) inside the loop would close that gap.

  2. Header order matters, and requests doesn’t preserve it like a browser does. Some anti-bot systems check header order along with the values, so a perfect Chrome User-Agent can still look off. httpx or curl_cffi give you more control here, and curl_cffi’s impersonate="chrome" also handles the TLS/HTTP2 fingerprint at the same time.

  3. SOCKS5 and DNS leaks. If you swap to SOCKS5 proxies as the article mentions, socks5:// makes requests resolve DNS locally, which can expose your real resolver and cause geo mismatches. Use socks5h:// so the proxy resolves the hostname.

  4. Reuse a Session per proxy. Creating a new connection for every request through a proxy adds latency and a new TLS handshake each time. A session per proxy keeps cookies consistent and makes the traffic pattern look more natural.

Have you measured how much a sticky session reduces CAPTCHA frequency compared to per-request rotation? I’d be curious about the numbers.

Collapse
 
devstackhub profile image
Dev Stack Hub •

You’ve caught a real gap, and the details you added are the ones that bite in practice.

Backoff: You’re right. The article recommends exponential backoff in Step 3, but fetch() only honors Retry-After or falls back to a flat 30s, so a server that returns 429 without the header gets the same wait every time. Your min(60, 2 ** attempt) + random.uniform(0, 1) fix is what I’d use, with Retry-After taking priority when it’s present. I’ll update the code so it matches the text.

Header order and TLS: Agreed. requests can’t control header order or the TLS/HTTP2 fingerprint, so a perfect Chrome User-Agent can still be flagged. For targets with serious bot detection, curl_cffi with impersonate="chrome" is the better tool, and I should have called that out rather than implying header values alone are enough.

socks5 vs socks5h: Good catch. With socks5:// the DNS lookup happens locally, which leaks your resolver and can cause geo mismatches. socks5h:// fixes that. I’ll add a note where SOCKS5 is mentioned.

Session per proxy: Fully agree. A session per proxy avoids a fresh TLS handshake on every request, keeps cookies consistent, and produces a more natural traffic pattern.

Sticky vs per-request rotation: I haven’t run a controlled comparison, so I don’t have clean numbers to share. What I’ve seen is anecdotal: sticky sessions noticeably cut CAPTCHA frequency on sites that tie challenges to session or cookie state, while per-request rotation worked better on simple rate-limit-by-IP targets. Both results depended heavily on the target, so any single figure would mislead. If you’ve measured it, I’d like to compare notes. I may do a proper benchmark across a few targets for a follow-up post.

Collapse
 
rafidbottler profile image
Rafid Bottler •

Solid guide, and the point about logging outcomes per proxy, URL and time is what makes debugging possible instead of guesswork.

A few things that helped me once this pattern was in production:

  1. Fingerprint block pages by hash. Soft-block markers work until the site changes its wording. Hashing a normalized version of the response body (strip digits, timestamps and tokens first) lets you spot when many “successful” responses are actually the same page. If 40 different product URLs return an identical hash, you’re being served a challenge, even if no marker matches.

  2. Pick proxies by success rate, not random.choice. ProxyPool.get() treats every healthy IP equally. Tracking a rolling success rate (say, the last 50 requests) and weighting selection toward better performers means good IPs get used more and mediocre ones fade out before they even hit the bench threshold.

  3. Reduce requests before optimizing evasion. Conditional requests (If-None-Match with the stored ETag, or If-Modified-Since) return a cheap 304 for unchanged pages. For recurring scrapes, this can cut request volume dramatically, and fewer requests means fewer 429s. Many sites that rate-limit aggressively still honor conditional headers.

  4. Apply a per-domain rate limit when you go async. If you move from requests to asyncio with httpx or aiohttp, a plain semaphore caps total concurrency but not the rate per target. A token bucket per domain (for example, 2 requests per second regardless of how many workers or proxies you have) keeps pacing predictable.

  5. Respect Crawl-delay in robots.txt as a free hint. Some sites publish it, and it’s a good starting point for your base delay, since it’s roughly what the site owner considers acceptable.

Do you also track latency per proxy? I’ve found a sudden jump in response time often shows up a few minutes before blocks start, which makes it a useful early warning.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks, this is a really useful list. A few thoughts on each:

Hash fingerprinting: Agreed, and this is a gap in the post. Marker matching breaks the moment a site rewords its challenge page. Normalizing the body before hashing and flagging when many distinct URLs collapse to one hash is a much sturdier signal. I’m adding it to the detection section.

Success-rate weighting: Fair criticism. random.choice treats every healthy IP the same, and a rolling window of the last 50 requests with weighted selection would let weak proxies fade out before they hit the bench threshold. I’ll update the ProxyPool example.

Conditional requests: This is the point I most wish I’d included. Cutting request volume with ETag and If-Modified-Since is the cheapest way to avoid 429s, and it’s far better than any evasion tweak. Worth its own section.

Per-domain token bucket: Good call. A semaphore caps concurrency but says nothing about the rate hitting a single target, so I’ll mention this for the async version.

Crawl-delay: A nice free hint for setting the base delay. Not every site publishes it, but it costs nothing to read.

On your question: I didn’t cover latency per proxy in the post, but your experience makes sense. A rising response time often means the target is throttling or queueing a proxy before it starts returning hard blocks. Tracking a rolling median latency per proxy and alerting when it jumps well above its baseline would make a good early-warning signal. I’ll look at adding it to the logging section.

Collapse
 
alexshev profile image
Alex Shev •

Useful breakdown. One caution: proxy rotation can hide the signal that a scraper should slow down or stop. I’d put rate budgets, robots/terms checks, and a clear exit condition ahead of more aggressive routing, then treat 403/429 trends as observability data rather than a bypass challenge.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks, this is a fair point, and I agree with it. Rotation can mask the very signal a scraper should be listening to. If every 429 just triggers a new IP, you've turned a "slow down" message into a retry loop.

I'd frame the post's advice as a diagnostic aid, not a license to push harder. Rate budgets per target, robots.txt and terms checks, and a clear stop condition should come first. Proxies come after that, and mainly for legitimate cases like geo-specific testing or spreading load you're already allowed to send.

Your suggestion to treat 403/429 trends as observability data is the right way to look at it. A rising 429 rate is feedback about your request pattern, and the response should be backoff, lower concurrency, or contacting the site owner, not a smarter bypass. I'll add a short section to the article on setting rate budgets and exit conditions so that framing is explicit. Appreciate you raising it.

Collapse
 
harperelise profile image
Harper Elise Callahan •

Step 1 is the right way to debug this. Run the same request direct and through the proxy, with default and browser headers, and you can tell whether the problem is the IP, the headers or the volume. Most people skip this and rotate proxies blindly. Also verify the exit IP with an echo endpoint like ipify, because mixed-up http/https keys in the proxies dict can send requests out directly without any error.

The soft-block check matters because a CAPTCHA page returned with a 200 passes every status-code check. I’d add a content assertion on top of the keyword markers, such as a required selector, JSON key or minimum body size, since marker strings break when the site changes its challenge page. Benching a proxy right after a 429 while requiring a failure streak for timeouts is the correct split, because 429s are tied to the IP and timeouts are often transient.

For 403s on clean IPs, TLS fingerprinting is the likely cause. Python’s default stack has a JA3 signature that doesn’t match a browser’s, so headers and IP quality don’t help. Switching to curl_cffi with browser impersonation is usually the fastest test before going to Playwright. For cookie-based flows, keep one sticky IP per session and rotate only between sessions, not between requests.

Collapse
 
devstackhub profile image
Dev Stack Hub •

You’ve picked up on the parts that cost people the most time in practice.

On Step 1, I agree. Rotating proxies blindly is the most common mistake, and the direct vs. proxy, default vs. browser headers matrix is the fastest way to find out whether it’s the IP, the headers, or the volume. Your ipify point is a good one too. A mixed-up http/https key in the proxies dict can send traffic out directly with no error, so verifying the exit IP should be a standard check, and I’ll add it to the article.

You’re right about the soft-block check. Keyword markers break the moment a site changes its challenge page. A required selector, a JSON key, or a minimum body size makes a sturdier second layer, and I’d recommend using both together.

On TLS fingerprinting, I agree that it’s the likely culprit when a clean IP still gets a 403. Python’s default stack has a JA3 signature that doesn’t look like a browser, so no amount of header tuning or IP swapping will fix it. Trying curl_cffi with browser impersonation before reaching for Playwright is a cheap test that saves a lot of time.

Your sticky session advice is spot on as well. For cookie-based flows, one IP per session, rotating only between sessions, avoids a lot of confusing failures.

I’ll fold the exit IP check, the content assertions, and the TLS fingerprinting note into the next revision. Thanks again for the detailed feedback.

Collapse
 
levinemundro profile image
Levine Mundro •

One caveat on Step 1: curl and requests can differ even with identical headers, because curl negotiates HTTP/2 while requests only speaks HTTP/1.1, and some WAFs score that. If curl passes and requests fails, check the protocol and TLS layer before blaming the proxy.

For the classifier, checking headers like cf-mitigated: challenge or challenge cookies (datadome, _abck) is more reliable than keyword matching on the first 5 KB of the body. I’d also track proxy health per (proxy, domain) pair, since an IP burned on one site is often fine on another.

For 403s on clean IPs, it’s usually JA3/h2 fingerprinting. curl_cffi with impersonate='chrome124' fixes it quickly, but keep the UA version matched to the TLS profile.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks, these are all good points, and a couple of them show gaps in the post.

curl vs requests (Step 1): You’re right. I framed it as “if curl passes and requests fails, it’s your code,” but that’s too loose. curl negotiates HTTP/2 and requests only speaks HTTP/1.1, and the TLS handshake differs too, so some WAFs score the two differently even with identical headers. I’ll add a note that when curl passes and requests fails, you should check the protocol and TLS layer before blaming the code. Running curl --http1.1 is a quick way to see whether the protocol is what matters.

Classifier (Step 2): Agreed that keyword matching on the first 5 KB is the weakest part of the post. I included it because it’s easy to follow, but checking cf-mitigated: challenge and challenge cookies like datadome and _abck is more reliable and has fewer false positives. I’ll update classify() to check headers and cookies first and keep the body markers only as a fallback.

Per-domain proxy health (Step 4): This is the best point. An IP burned on one site is often fine on another, so the ProxyPool should track health per (proxy, domain) pair rather than per proxy. Otherwise one hostile target benches IPs that would work elsewhere. I’ll rework the pool to key failures and benched_until on that pair.

JA3/h2 on clean IPs (Step 6): I only mentioned TLS fingerprinting briefly, and curl_cffi with impersonate='chrome124' deserves more than that. Your caveat about keeping the UA version matched to the TLS profile is important. A Chrome 124 handshake with a Chrome 110 User-Agent is itself a signal.

I’ll fold these into an update and credit you in the post. If you have a go-to way to check which layer is flagging you, such as comparing against a JA3 echo service, I’d like to hear it.

Collapse
 
sidra-jefferi profile image
Sidra Jefferi •

One gap I noticed: Step 3 recommends exponential backoff and per-IP concurrency limits, but the code in fetch() doesn’t implement either. Every retry uses the same polite_sleep() delay, so a persistently throttled target is retried at a constant pace. Something like delay = min(cap, base * 2 ** attempt) + jitter would make the code match the advice, and a per-proxy semaphore (or a simple “last request time per IP” map) would enforce the concurrency rule.

A related detail for the SOCKS5 mention: with requests, socks5:// resolves DNS on your machine, while socks5h:// resolves it through the proxy. If a site serves geo-specific content, local DNS resolution can leak your real location even though the exit IP is correct. The IP-echo check from Step 1 wouldn’t catch that.

Have you compared success rates for the same target with requests versus curl_cffi? I’d be curious how much of the 403 gap on clean IPs is TLS fingerprinting.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks, this is a fair catch on all three points.

Backoff and concurrency: You’re right. Step 3 recommends both, but fetch() only uses Retry-Afteron 429s and falls back to the same polite_sleep() for everything else, so a persistently throttled target gets retried at a constant pace. I’m updating the code to match the advice:

def backoff_delay(attempt, base=1.0, cap=60.0):
    return min(cap, base * 2 ** attempt) + random.uniform(0, 1.5)
Enter fullscreen mode Exit fullscreen mode

I’ll use it in place of polite_sleep() on retries, and take the larger of that and Retry-After for 429s. For concurrency, I’ll add a per-proxy “last request time” map to ProxyPool, since it’s the simplest version to show without pulling in asyncio.

socks5 vs socks5h: Good point, and I should have called it out. With socks5://, requests resolves DNS locally, so a geo-sensitive site can see your real resolver’s location even when the exit IP is correct. socks5h:// fixes that. You’re also right that the IP-echo check in Step 1 can’t catch it, since it only verifies the exit IP. I’ll add a note on this to the geo-mismatch section.

requests vs curl_cffi: I haven’t run a controlled comparison on the same target with the same clean IPs, so I don’t want to quote a number I can’t back up. The article only flags TLS fingerprinting as the next thing to test. If you’ve measured it, I’d love to hear what you saw. I’ll try running the same probe with both clients and share the results here or in a follow-up.

Collapse
 
debugtodeploy profile image
Vinay Shah •

The “change one variable at a time” framing in Step 1 is what most scraping tutorials skip, and the direct-vs-proxy, default-vs-browser headers truth table is a quick way to stop guessing.

A few things that stood out:

  • Soft blocks are underrated. Treating a 200 response as a failure when it’s actually a CAPTCHA page saves a lot of silent bad data. Asserting that the expected content is present (a product title or JSON key) works well alongside the marker list, since marker strings can drift when a site changes its challenge page.
  • Benching 429s immediately is a smart default. It stops one burned IP from dragging down the whole pool.
  • Small suggestions for the fetch loop: you mention exponential backoff in Step 3, but the loop uses a fixed polite_sleep() after failures, so it might be worth wiring the backoff in. If people run this with threads, ProxyPool will also need a lock around failures and benched_until.
  • Sticky sessions. Step 6 could stress that a rotating proxy mid-session often causes 403s by itself. Cookies set on one IP and replayed from another are a classic trigger.

Also glad you ended with the “check for an official API and read the ToS first” reminder. It gets skipped far too often.

Curious whether you’ve seen curl_cffi fully resolve TLS-fingerprint blocks in practice, or whether you still end up needing Playwright for the stricter sites.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks for the careful read, and you’re right on both code points.

Backoff: The loop should use the exponential backoff from Step 3 instead of a fixed polite_sleep(). Something like sleep = base * 2**attempt + random.uniform(0, 1) makes retries cheaper on the target and less predictable. I’ll update the snippet so the code matches the text.

Thread safety: You’re also right that ProxyPool isn’t safe with threads as written. Concurrent updates to failures and benched_until can race, so a threading.Lock around those reads and writes is the minimal fix. I’ll add a note, since people will definitely copy this into a threaded worker.

Your content-assertion point is a good complement to marker lists. Markers break when a site redesigns its challenge page, while “the product title or expected JSON key is present” fails closed. I’ll mention using both.

On sticky sessions, agreed: rotating mid-session is a self-inflicted 403 more often than people think. Cookies and tokens issued to one IP and replayed from another look exactly like session hijacking to a WAF. I’ll expand Step 6 to say: pin one proxy per session, rotate only between sessions.

On curl_cffi: in my experience it resolves a lot of TLS/JA3 fingerprint blocks, and often that’s all a site needs, and it’s far lighter than a full browser. But it doesn’t solve everything. Stricter sites check JavaScript execution, canvas/WebGL signals, or behavioral telemetry, and there I still end up reaching for Playwright. My rule of thumb is to try curl_cffi with browser impersonation first, and escalate to Playwright only when the response is still a challenge page.

Thanks for the kind words on the ToS reminder, too. Appreciate you taking the time to write this up.

Collapse
 
kevinhall profile image
Kevin Hall •

Soft blocks are a billing problem too. On a pay-per-result scraper a 200 CAPTCHA page can land in the dataset as a row, so I would validate a field that only a real page has, not just the status code, before a run counts as a success.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Agreed, and thanks for raising the billing angle. I only touched on it in Step 2, where I suggested asserting that the expected content is present, but you've put it better: on a pay-per-result scraper, a CAPTCHA page that lands in the dataset as a row costs you money.

I'd go further than the marker list. Block-marker strings are a deny-list, and they break whenever the site changes its challenge page. Checking for a field that only a real page has (a product title, a price, a JSON key) is an allow-list, so it fails safe: anything unfamiliar counts as a failure instead of slipping through. I'd use both, with the positive check deciding whether a run counts as a success.

A few things I'd add:

  • Track the share of failed validations per run. If a run reports "success" but 30% of its rows failed the field check, that's a block, and the pass rate is a better health signal than status codes.
  • Only count a result as billable once it passes validation, and benching the proxy that served a soft block applies here too.
  • Be careful with the required field. Pick something that is always present on real pages, or legitimately empty pages will be flagged as blocks.

I'll add a validation step to the guide's fetch function. Thanks for the suggestion!

Collapse
 
mp3sss31 profile image
mp3sss •

The isolation matrix is the most useful part — most scraper debugging is debugging-by-stabbing, and the direct-vs-proxy truth table turns it into bisection. Two additions from running desktop clients against the same walls: treat Retry-After as a first-class signal (a pool that ignores it just thrashes harder), and score proxy health on a decaying error window so bad IPs get benched automatically instead of on a fixed timer. Worth noting a 200-with-CAPTCHA shows up inside headless browsers too — behavior fingerprints apply there as much as raw HTTP.