Your scraper worked fine on 50 requests. At 500, it started returning 403 Forbidden. At 5,000, you're staring at 429 Too Many Requests and CAPTCHA pages with a 200 status code. Adding more proxies didn't help, or helped for ten minutes.
The mistake most developers make here is changing several things at once: new proxies, new headers, more retries. This guide gives you a repeatable way to find out why you're being blocked so that you can fix the right thing.
What we'll cover
- What 403, 429, and CAPTCHA responses actually mean
- A step-by-step isolation process (and a small Python probe script)
- Detecting "soft blocks" that return HTTP 200
- Backoff,
Retry-After, and pacing done properly - Tracking proxy health so bad IPs get benched automatically
- When the problem isn't the IP at all
Examples use Python and requests, with placeholder proxies from the documentation range (203.0.113.x).
What each signal usually means
| Signal | What the server is telling you | Typical root cause |
|---|---|---|
| 429 Too Many Requests | You exceeded a rate limit | Too many requests per IP, per session or per account in a time window |
| 403 Forbidden | The server understood you and refuses | IP reputation, bot detection, missing or inconsistent headers, geo restriction |
| 200 with a CAPTCHA page | A challenge is served instead of content | Risk score too high: IP, behavior or fingerprint looks automated |
| 407 Proxy Authentication Required | The proxy rejected you, not the target | Wrong credentials or non-whitelisted IP |
Note that 407 is not a block at all. It's your proxy configuration, and it's the first thing to rule out.
Step 1: Isolate where the block happens
Before touching your scraper, test the target with the simplest possible client. If curl through the proxy works but your script fails, the problem is in your code. If both fail, it's the IP or the target.
# Direct (no proxy)
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/products
# Through the proxy
curl -s -o /dev/null -w "%{http_code}\n" -x http://user:pass@203.0.113.10:8080 https://example.com/products
Then reproduce the comparison in Python with a small probe that varies one thing at a time:
import requests
TARGET = 'https://example.com/products'
PROXY = 'http://user:pass@203.0.113.10:8080'
PROXIES = {'http': PROXY, 'https': PROXY}
BROWSER_HEADERS = {
'User-Agent': (
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
'(KHTML, like Gecko) Chrome/124.0 Safari/537.36'
),
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.9',
}
def probe(label, **kwargs):
try:
r = requests.get(TARGET, timeout=(5, 15), **kwargs)
print(f'{label:<28} {r.status_code} {len(r.content):>8} bytes')
except requests.RequestException as exc:
print(f'{label:<28} ERROR {type(exc).__name__}')
probe('direct, default headers')
probe('direct, browser headers', headers=BROWSER_HEADERS)
probe('proxy, default headers', proxies=PROXIES)
probe('proxy, browser headers', proxies=PROXIES, headers=BROWSER_HEADERS)
Read the results like a truth table:
- Direct works, proxy fails: the proxy IP is the problem (reputation, type, or geo).
-
Default headers fail, browser headers pass: the site checks the User-Agent and friends. The default
python-requests/x.yagent is an easy signal to filter. - Everything fails: the page may need JavaScript, a cookie, or a login. The proxy is not the issue.
- Everything passes: your problem is volume or pacing, so move on to Step 3.
Also confirm the proxy really is being used by calling an IP-echo endpoint such as https://api.ipify.org?format=json through it. A surprising number of "proxy problems" are requests silently going out directly.
Step 2: Detect soft blocks
A response code of 200 doesn't mean you got the page. Many sites serve a CAPTCHA or an "access denied" page with a success status, and your parser then silently stores garbage. Classify every response before you trust it:
BLOCK_MARKERS = (
'captcha',
'unusual traffic',
'access denied',
'verify you are human',
'are you a robot',
)
def classify(response):
if response.status_code == 407:
return 'proxy_auth'
if response.status_code == 429:
return 'rate_limited'
if response.status_code in (401, 403):
return 'blocked'
snippet = response.text[:5000].lower()
if any(marker in snippet for marker in BLOCK_MARKERS):
return 'challenge'
return 'ok'
The marker list is site-specific, so tune it to the challenge pages you actually see. A cheaper complementary check is to assert that the content you expect (a product title, a JSON key) is present, and treat its absence as a failure.
Step 3: Fix 429s with backoff and pacing
A 429 is the friendliest error you'll get, because the server is telling you exactly what to do: slow down. Many servers include a Retry-After header, either as several seconds or as an HTTP date. Honor it:
import random
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
def retry_after_seconds(response, default=30):
value = response.headers.get('Retry-After')
if not value:
return default
if value.isdigit():
return int(value)
try:
when = parsedate_to_datetime(value)
return max(0, (when - datetime.now(timezone.utc)).total_seconds())
except (TypeError, ValueError):
return default
def polite_sleep(base=1.0, jitter=1.5):
# randomized delay between requests, so traffic isn't perfectly periodic
time.sleep(base + random.uniform(0, jitter))
Three principles prevent most 429s:
- Limit concurrency per IP, not just overall. Ten threads on one IP is a burst. Ten threads across ten IPs is not.
- Add jitter. A request every exactly 1.000 seconds looks robotic.
- Use exponential backoff on repeated failures. If a 429 comes back after waiting, wait longer next time instead of retrying at the same speed.
If you're still rate-limited at a sensible pace, the limit probably applies per IP, and the fix is a larger pool rather than a faster loop.
Step 4: Track proxy health and bench bad IPs
Random rotation keeps sending traffic to IPs that are already burned. Instead, record outcomes and give failing proxies a cooldown:
import random
import time
from collections import defaultdict
class ProxyPool:
def __init__(self, proxies, cooldown=300, max_failures=3):
self.proxies = list(proxies)
self.cooldown = cooldown
self.max_failures = max_failures
self.failures = defaultdict(int)
self.benched_until = {}
def get(self):
now = time.time()
available = [p for p in self.proxies if self.benched_until.get(p, 0) <= now]
if not available:
raise RuntimeError('No healthy proxies: slow down or add more IPs')
return random.choice(available)
def report(self, proxy, outcome):
if outcome == 'ok':
self.failures[proxy] = 0
return
self.failures[proxy] += 1
# a 429 benches the IP immediately; other failures need a streak
if outcome == 'rate_limited' or self.failures[proxy] >= self.max_failures:
self.benched_until[proxy] = time.time() + self.cooldown
self.failures[proxy] = 0
And a fetch function that ties the pieces together:
import requests
def fetch(url, pool, attempts=4):
for _ in range(attempts):
proxy = pool.get()
try:
r = requests.get(
url,
proxies={'http': proxy, 'https': proxy},
headers=BROWSER_HEADERS,
timeout=(5, 15),
)
except requests.RequestException:
pool.report(proxy, 'error')
continue
outcome = classify(r)
pool.report(proxy, outcome)
if outcome == 'ok':
return r
if outcome == 'rate_limited':
time.sleep(retry_after_seconds(r))
elif outcome == 'proxy_auth':
raise RuntimeError('Proxy credentials rejected (407). Check auth or IP whitelist.')
polite_sleep()
raise RuntimeError(f'Failed to fetch {url} after {attempts} attempts')
Log the outcome, proxy, and URL for every request. After a few hundred requests, you'll be able to see whether failures cluster on specific IPs (bad IPs), specific URLs (protected pages), or specific times (rate-limit windows). That pattern tells you which fix to apply.
Step 5: When it's the IP type
If your probe showed that browser-like headers pass directly but fail through the proxy, you're dealing with IP reputation. Different proxy types carry different levels of trust:
| Proxy type | Trust with strict sites | Strengths | Watch out for |
|---|---|---|---|
| Datacenter / IPv6 | Lower | Fast, cheap, easy to scale | Whole ranges are often flagged; shared IPs inherit other users' behavior |
| ISP (static residential) | Higher | Looks like a home user, stable for long sessions | Priced per IP, so pool size is limited by budget |
| Residential (rotating) | High | Huge pools, precise geo-targeting | Usually billed per GB; can be slower |
| Mobile (4G/5G) | Highest | Carrier IPs are shared by many real users, so sites are cautious about blocking them | Most expensive; speed varies |
A few related checks:
- Dedicated vs shared: a shared IP can be burned by someone else before you send a single request.
-
Geo mismatch: if your
Accept-Languagesaysen-USbut your exit IP is in another country, or your IP is in Germany while you request a US-only price page, you will get blocked or served different content. - Burned static IPs: a static IP that has been hammered for days may need a rest or a replacement. Rotation isn't just for volume; it's hygiene.
Step 6: When it isn't the IP at all
If you've tried clean IPs and sensible pacing and still get 403s, look beyond the proxy:
-
Session and cookies: Many sites set a cookie on the first page and expect it on the next. Use a
requests.Sessionand keep the same IP for the whole flow (a sticky session if your provider supports one). -
Inconsistent headers: A Chrome User-Agent with a missing
Accept-Language, or headers in an unusual order, is a signal. Keep the set realistic and consistent per session. -
TLS fingerprinting: Some bot-protection systems inspect the TLS handshake itself, and Python's default stack looks different from a real browser's. If headers and IPs are fine but a site still refuses you, a real browser (Playwright or Selenium) or an HTTP client that mimics browser TLS such as
curl_cffimay be the next thing to test. -
JavaScript-rendered content: If the data is loaded by JavaScript, plain
requestsget an empty shell. Look for the JSON API call the page makes in your browser's network tab, or use a headless browser. - Behavioral signals: Hitting deep URLs with no prior navigation, or requesting pages at inhuman speed, can raise a risk score even from a perfect IP.
A word on CAPTCHAs
A CAPTCHA is not an error to defeat. It's a signal that your traffic looks risky. The most reliable strategy is to trigger fewer of them: better IP quality, lower rate, consistent sessions, realistic headers. If you're seeing them constantly, treat that as a symptom and work back through the steps above.
It's also worth asking whether you should be scraping that page at all. Check for an official API or data export first, read the site's terms of service and robots.txt, and only collect data you're permitted to use. Circumventing access controls on sites that forbid automated access can create legal problems that no proxy will solve.
Choosing proxies for testing
To follow along, you need proxies that let you compare types on the same target. I used a provider that offers datacenter, ISP, residential, and mobile IPs with both HTTP/HTTPS and SOCKS5 so that I could swap types without changing code. Disclosure: [add your affiliate or sponsorship note here if applicable]. Whichever service you use, buy a small amount of each type you're considering, run the probe script from Step 1, and compare success rate and latency on your target before committing to a larger plan.
Quick reference
- 407? Fix proxy auth first. It isn't a block.
- Isolate: compare direct vs proxy, default vs browser headers.
- Classify every response, including 200 pages that are really CAPTCHAs.
-
429: honor
Retry-After, add jitter, limit per-IP concurrency, then grow the pool. - Bench bad IPs with a cooldown instead of rotating unquestioningly.
- 403 on clean IPs: check sessions, headers, TLS fingerprint, and JavaScript rendering.
- Constant CAPTCHAs: reduce the triggers, and check whether an official API exists.
Got a blocking pattern that isn't covered here? Share it in the comments, and I'll add it to the guide.
Top comments (19)
The “change one variable at a time” framing is what most scraping tutorials skip, and the direct vs. proxy, default vs. browser headers truth table is a quick way to see where the block actually is.
Two things I especially liked:
Soft-block detection. A 200 response that is really a CAPTCHA page quietly corrupts your data, and
classify()catches that before it reaches the parser. Pairing it with a check that the expected content is present would make it sturdier still.Benching proxies instead of rotating blindly. Random rotation keeps feeding traffic to burned IPs, and the cooldown approach fixes that cleanly.
One thing I’d add: confirm the proxy is really being used (the ipify check) before blaming it. I’ve seen requests silently go out direct because of a misconfigured
proxiesdict.Curious how you’d handle sticky sessions with rotating residential proxies when a flow spans several pages. Do you pin one IP per session, or re-authenticate when it changes?
Thanks for the thoughtful read, and good call on the ipify check. A silently misconfigured proxies dict is one of the most common reasons people end up blaming the proxy for something that never went through it. The usual culprit is a dict keyed only on
"http"while the target is HTTPS` so requests quietly go out direct. I should have made that check step zero in the post.Agreed on pairing
classify()with an expected-content check too. Status code and block-page markers catch the obvious cases, but asserting that a known selector or field is present catches the subtle ones, like a trimmed page or a "lite" response served to suspected bots.On sticky sessions: I pin one IP per flow. For anything spanning multiple pages (login, search, checkout, pagination with session state), the site ties cookies, CSRF tokens, and often fingerprint signals to the IP, so a mid-flow change usually invalidates the session or trips risk scoring. Most residential providers support this by embedding a session ID in the proxy username, which holds the same exit IP for a set window (typically 1 to 30 minutes, depending on the provider).
My rules of thumb:
Re-authenticating is a fallback for flows that are expensive to restart, but I'd treat it as the exception.
How are you handling it on your end? Are your flows short enough that sticky windows aren't an issue?
The truth-table approach in Step 1 is a clean way to separate proxy issues from code issues.
A few technical details that have bitten me in similar setups:
Exponential backoff isn’t in the code yet. The Step 3 text recommends it, but
fetch()only sleeps forRetry-After(or the 30s default) and thenpolite_sleep(). If a server keeps returning 429 without aRetry-Afterheader, every retry waits the same amount. Something likedelay = min(60, 2 ** attempt) + random.uniform(0, 1)inside the loop would close that gap.Header order matters, and
requestsdoesn’t preserve it like a browser does. Some anti-bot systems check header order along with the values, so a perfect Chrome User-Agent can still look off.httpxorcurl_cffigive you more control here, andcurl_cffi’simpersonate="chrome"also handles the TLS/HTTP2 fingerprint at the same time.SOCKS5 and DNS leaks. If you swap to SOCKS5 proxies as the article mentions,
socks5://makesrequestsresolve DNS locally, which can expose your real resolver and cause geo mismatches. Usesocks5h://so the proxy resolves the hostname.Reuse a
Sessionper proxy. Creating a new connection for every request through a proxy adds latency and a new TLS handshake each time. A session per proxy keeps cookies consistent and makes the traffic pattern look more natural.Have you measured how much a sticky session reduces CAPTCHA frequency compared to per-request rotation? I’d be curious about the numbers.
You’ve caught a real gap, and the details you added are the ones that bite in practice.
Backoff: You’re right. The article recommends exponential backoff in Step 3, but
fetch()only honorsRetry-Afteror falls back to a flat 30s, so a server that returns 429 without the header gets the same wait every time. Yourmin(60, 2 ** attempt) + random.uniform(0, 1)fix is what I’d use, withRetry-Aftertaking priority when it’s present. I’ll update the code so it matches the text.Header order and TLS: Agreed.
requestscan’t control header order or the TLS/HTTP2 fingerprint, so a perfect Chrome User-Agent can still be flagged. For targets with serious bot detection,curl_cffiwithimpersonate="chrome"is the better tool, and I should have called that out rather than implying header values alone are enough.socks5 vs socks5h: Good catch. With
socks5://the DNS lookup happens locally, which leaks your resolver and can cause geo mismatches.socks5h://fixes that. I’ll add a note where SOCKS5 is mentioned.Session per proxy: Fully agree. A session per proxy avoids a fresh TLS handshake on every request, keeps cookies consistent, and produces a more natural traffic pattern.
Sticky vs per-request rotation: I haven’t run a controlled comparison, so I don’t have clean numbers to share. What I’ve seen is anecdotal: sticky sessions noticeably cut CAPTCHA frequency on sites that tie challenges to session or cookie state, while per-request rotation worked better on simple rate-limit-by-IP targets. Both results depended heavily on the target, so any single figure would mislead. If you’ve measured it, I’d like to compare notes. I may do a proper benchmark across a few targets for a follow-up post.
Solid guide, and the point about logging outcomes per proxy, URL and time is what makes debugging possible instead of guesswork.
A few things that helped me once this pattern was in production:
Fingerprint block pages by hash. Soft-block markers work until the site changes its wording. Hashing a normalized version of the response body (strip digits, timestamps and tokens first) lets you spot when many “successful” responses are actually the same page. If 40 different product URLs return an identical hash, you’re being served a challenge, even if no marker matches.
Pick proxies by success rate, not
random.choice.ProxyPool.get()treats every healthy IP equally. Tracking a rolling success rate (say, the last 50 requests) and weighting selection toward better performers means good IPs get used more and mediocre ones fade out before they even hit the bench threshold.Reduce requests before optimizing evasion. Conditional requests (
If-None-Matchwith the storedETag, orIf-Modified-Since) return a cheap 304 for unchanged pages. For recurring scrapes, this can cut request volume dramatically, and fewer requests means fewer 429s. Many sites that rate-limit aggressively still honor conditional headers.Apply a per-domain rate limit when you go async. If you move from
requeststoasynciowithhttpxoraiohttp, a plain semaphore caps total concurrency but not the rate per target. A token bucket per domain (for example, 2 requests per second regardless of how many workers or proxies you have) keeps pacing predictable.Respect
Crawl-delayinrobots.txtas a free hint. Some sites publish it, and it’s a good starting point for your base delay, since it’s roughly what the site owner considers acceptable.Do you also track latency per proxy? I’ve found a sudden jump in response time often shows up a few minutes before blocks start, which makes it a useful early warning.
Thanks, this is a really useful list. A few thoughts on each:
Hash fingerprinting: Agreed, and this is a gap in the post. Marker matching breaks the moment a site rewords its challenge page. Normalizing the body before hashing and flagging when many distinct URLs collapse to one hash is a much sturdier signal. I’m adding it to the detection section.
Success-rate weighting: Fair criticism.
random.choicetreats every healthy IP the same, and a rolling window of the last 50 requests with weighted selection would let weak proxies fade out before they hit the bench threshold. I’ll update theProxyPoolexample.Conditional requests: This is the point I most wish I’d included. Cutting request volume with ETag and If-Modified-Since is the cheapest way to avoid 429s, and it’s far better than any evasion tweak. Worth its own section.
Per-domain token bucket: Good call. A semaphore caps concurrency but says nothing about the rate hitting a single target, so I’ll mention this for the async version.
Crawl-delay: A nice free hint for setting the base delay. Not every site publishes it, but it costs nothing to read.
On your question: I didn’t cover latency per proxy in the post, but your experience makes sense. A rising response time often means the target is throttling or queueing a proxy before it starts returning hard blocks. Tracking a rolling median latency per proxy and alerting when it jumps well above its baseline would make a good early-warning signal. I’ll look at adding it to the logging section.
Useful breakdown. One caution: proxy rotation can hide the signal that a scraper should slow down or stop. I’d put rate budgets, robots/terms checks, and a clear exit condition ahead of more aggressive routing, then treat 403/429 trends as observability data rather than a bypass challenge.
Thanks, this is a fair point, and I agree with it. Rotation can mask the very signal a scraper should be listening to. If every 429 just triggers a new IP, you've turned a "slow down" message into a retry loop.
I'd frame the post's advice as a diagnostic aid, not a license to push harder. Rate budgets per target, robots.txt and terms checks, and a clear stop condition should come first. Proxies come after that, and mainly for legitimate cases like geo-specific testing or spreading load you're already allowed to send.
Your suggestion to treat 403/429 trends as observability data is the right way to look at it. A rising 429 rate is feedback about your request pattern, and the response should be backoff, lower concurrency, or contacting the site owner, not a smarter bypass. I'll add a short section to the article on setting rate budgets and exit conditions so that framing is explicit. Appreciate you raising it.
Step 1 is the right way to debug this. Run the same request direct and through the proxy, with default and browser headers, and you can tell whether the problem is the IP, the headers or the volume. Most people skip this and rotate proxies blindly. Also verify the exit IP with an echo endpoint like ipify, because mixed-up
http/httpskeys in the proxies dict can send requests out directly without any error.The soft-block check matters because a CAPTCHA page returned with a 200 passes every status-code check. I’d add a content assertion on top of the keyword markers, such as a required selector, JSON key or minimum body size, since marker strings break when the site changes its challenge page. Benching a proxy right after a 429 while requiring a failure streak for timeouts is the correct split, because 429s are tied to the IP and timeouts are often transient.
For 403s on clean IPs, TLS fingerprinting is the likely cause. Python’s default stack has a JA3 signature that doesn’t match a browser’s, so headers and IP quality don’t help. Switching to
curl_cffiwith browser impersonation is usually the fastest test before going to Playwright. For cookie-based flows, keep one sticky IP per session and rotate only between sessions, not between requests.You’ve picked up on the parts that cost people the most time in practice.
On Step 1, I agree. Rotating proxies blindly is the most common mistake, and the direct vs. proxy, default vs. browser headers matrix is the fastest way to find out whether it’s the IP, the headers, or the volume. Your ipify point is a good one too. A mixed-up http/https key in the proxies dict can send traffic out directly with no error, so verifying the exit IP should be a standard check, and I’ll add it to the article.
You’re right about the soft-block check. Keyword markers break the moment a site changes its challenge page. A required selector, a JSON key, or a minimum body size makes a sturdier second layer, and I’d recommend using both together.
On TLS fingerprinting, I agree that it’s the likely culprit when a clean IP still gets a 403. Python’s default stack has a JA3 signature that doesn’t look like a browser, so no amount of header tuning or IP swapping will fix it. Trying curl_cffi with browser impersonation before reaching for Playwright is a cheap test that saves a lot of time.
Your sticky session advice is spot on as well. For cookie-based flows, one IP per session, rotating only between sessions, avoids a lot of confusing failures.
I’ll fold the exit IP check, the content assertions, and the TLS fingerprinting note into the next revision. Thanks again for the detailed feedback.
One caveat on Step 1:
curlandrequestscan differ even with identical headers, becausecurlnegotiates HTTP/2 whilerequestsonly speaks HTTP/1.1, and some WAFs score that. Ifcurlpasses andrequestsfails, check the protocol and TLS layer before blaming the proxy.For the classifier, checking headers like
cf-mitigated: challengeor challenge cookies (datadome,_abck) is more reliable than keyword matching on the first 5 KB of the body. I’d also track proxy health per (proxy, domain) pair, since an IP burned on one site is often fine on another.For 403s on clean IPs, it’s usually JA3/h2 fingerprinting.
curl_cffiwithimpersonate='chrome124'fixes it quickly, but keep the UA version matched to the TLS profile.Thanks, these are all good points, and a couple of them show gaps in the post.
curl vs requests (Step 1): You’re right. I framed it as “if curl passes and requests fails, it’s your code,” but that’s too loose. curl negotiates HTTP/2 and requests only speaks HTTP/1.1, and the TLS handshake differs too, so some WAFs score the two differently even with identical headers. I’ll add a note that when curl passes and requests fails, you should check the protocol and TLS layer before blaming the code. Running
curl --http1.1is a quick way to see whether the protocol is what matters.Classifier (Step 2): Agreed that keyword matching on the first 5 KB is the weakest part of the post. I included it because it’s easy to follow, but checking
cf-mitigated: challengeand challenge cookies likedatadomeand_abckis more reliable and has fewer false positives. I’ll update classify() to check headers and cookies first and keep the body markers only as a fallback.Per-domain proxy health (Step 4): This is the best point. An IP burned on one site is often fine on another, so the
ProxyPoolshould track health per (proxy, domain) pair rather than per proxy. Otherwise one hostile target benches IPs that would work elsewhere. I’ll rework the pool to keyfailuresandbenched_untilon that pair.JA3/h2 on clean IPs (Step 6): I only mentioned TLS fingerprinting briefly, and
curl_cffiwithimpersonate='chrome124'deserves more than that. Your caveat about keeping the UA version matched to the TLS profile is important. A Chrome 124 handshake with a Chrome 110 User-Agent is itself a signal.I’ll fold these into an update and credit you in the post. If you have a go-to way to check which layer is flagging you, such as comparing against a JA3 echo service, I’d like to hear it.
One gap I noticed: Step 3 recommends exponential backoff and per-IP concurrency limits, but the code in
fetch()doesn’t implement either. Every retry uses the samepolite_sleep()delay, so a persistently throttled target is retried at a constant pace. Something likedelay = min(cap, base * 2 ** attempt) + jitterwould make the code match the advice, and a per-proxy semaphore (or a simple “last request time per IP” map) would enforce the concurrency rule.A related detail for the SOCKS5 mention: with
requests,socks5://resolves DNS on your machine, whilesocks5h://resolves it through the proxy. If a site serves geo-specific content, local DNS resolution can leak your real location even though the exit IP is correct. The IP-echo check from Step 1 wouldn’t catch that.Have you compared success rates for the same target with
requestsversuscurl_cffi? I’d be curious how much of the 403 gap on clean IPs is TLS fingerprinting.Thanks, this is a fair catch on all three points.
Backoff and concurrency: You’re right. Step 3 recommends both, but
fetch()only usesRetry-Afteron 429s and falls back to the samepolite_sleep()for everything else, so a persistently throttled target gets retried at a constant pace. I’m updating the code to match the advice:I’ll use it in place of
polite_sleep()on retries, and take the larger of that andRetry-Afterfor 429s. For concurrency, I’ll add a per-proxy “last request time” map to ProxyPool, since it’s the simplest version to show without pulling in asyncio.socks5 vs socks5h: Good point, and I should have called it out. With
socks5://, requests resolves DNS locally, so a geo-sensitive site can see your real resolver’s location even when the exit IP is correct.socks5h://fixes that. You’re also right that the IP-echo check in Step 1 can’t catch it, since it only verifies the exit IP. I’ll add a note on this to the geo-mismatch section.requests vs curl_cffi: I haven’t run a controlled comparison on the same target with the same clean IPs, so I don’t want to quote a number I can’t back up. The article only flags TLS fingerprinting as the next thing to test. If you’ve measured it, I’d love to hear what you saw. I’ll try running the same probe with both clients and share the results here or in a follow-up.
The “change one variable at a time” framing in Step 1 is what most scraping tutorials skip, and the direct-vs-proxy, default-vs-browser headers truth table is a quick way to stop guessing.
A few things that stood out:
fetchloop: you mention exponential backoff in Step 3, but the loop uses a fixedpolite_sleep()after failures, so it might be worth wiring the backoff in. If people run this with threads,ProxyPoolwill also need a lock aroundfailuresandbenched_until.Also glad you ended with the “check for an official API and read the ToS first” reminder. It gets skipped far too often.
Curious whether you’ve seen
curl_cffifully resolve TLS-fingerprint blocks in practice, or whether you still end up needing Playwright for the stricter sites.Thanks for the careful read, and you’re right on both code points.
Backoff: The loop should use the exponential backoff from Step 3 instead of a fixed
polite_sleep(). Something likesleep = base * 2**attempt + random.uniform(0, 1)makes retries cheaper on the target and less predictable. I’ll update the snippet so the code matches the text.Thread safety: You’re also right that
ProxyPoolisn’t safe with threads as written. Concurrent updates tofailuresandbenched_untilcan race, so athreading.Lockaround those reads and writes is the minimal fix. I’ll add a note, since people will definitely copy this into a threaded worker.Your content-assertion point is a good complement to marker lists. Markers break when a site redesigns its challenge page, while “the product title or expected JSON key is present” fails closed. I’ll mention using both.
On sticky sessions, agreed: rotating mid-session is a self-inflicted 403 more often than people think. Cookies and tokens issued to one IP and replayed from another look exactly like session hijacking to a WAF. I’ll expand Step 6 to say: pin one proxy per session, rotate only between sessions.
On curl_cffi: in my experience it resolves a lot of TLS/JA3 fingerprint blocks, and often that’s all a site needs, and it’s far lighter than a full browser. But it doesn’t solve everything. Stricter sites check JavaScript execution, canvas/WebGL signals, or behavioral telemetry, and there I still end up reaching for Playwright. My rule of thumb is to try curl_cffi with browser impersonation first, and escalate to Playwright only when the response is still a challenge page.
Thanks for the kind words on the ToS reminder, too. Appreciate you taking the time to write this up.
Soft blocks are a billing problem too. On a pay-per-result scraper a 200 CAPTCHA page can land in the dataset as a row, so I would validate a field that only a real page has, not just the status code, before a run counts as a success.
Agreed, and thanks for raising the billing angle. I only touched on it in Step 2, where I suggested asserting that the expected content is present, but you've put it better: on a pay-per-result scraper, a CAPTCHA page that lands in the dataset as a row costs you money.
I'd go further than the marker list. Block-marker strings are a deny-list, and they break whenever the site changes its challenge page. Checking for a field that only a real page has (a product title, a price, a JSON key) is an allow-list, so it fails safe: anything unfamiliar counts as a failure instead of slipping through. I'd use both, with the positive check deciding whether a run counts as a success.
A few things I'd add:
I'll add a validation step to the guide's
fetchfunction. Thanks for the suggestion!The isolation matrix is the most useful part — most scraper debugging is debugging-by-stabbing, and the direct-vs-proxy truth table turns it into bisection. Two additions from running desktop clients against the same walls: treat Retry-After as a first-class signal (a pool that ignores it just thrashes harder), and score proxy health on a decaying error window so bad IPs get benched automatically instead of on a fixed timer. Worth noting a 200-with-CAPTCHA shows up inside headless browsers too — behavior fingerprints apply there as much as raw HTTP.