DEV Community

Cover image for Debugging 403, 429 and CAPTCHA Errors with Proxies

Debugging 403, 429 and CAPTCHA Errors with Proxies

Dev Stack Hub on October 28, 2026

Your scraper worked fine on 50 requests. At 500, it started returning 403 Forbidden. At 5,000, you're staring at 429 Too Many Requests and CAPTCHA ...
Collapse
 
andersonkevin profile image
Kevin Anderson •

The “change one variable at a time” framing is what most scraping tutorials skip, and the direct vs. proxy, default vs. browser headers truth table is a quick way to see where the block actually is.

Two things I especially liked:

Soft-block detection. A 200 response that is really a CAPTCHA page quietly corrupts your data, and classify() catches that before it reaches the parser. Pairing it with a check that the expected content is present would make it sturdier still.
Benching proxies instead of rotating blindly. Random rotation keeps feeding traffic to burned IPs, and the cooldown approach fixes that cleanly.

One thing I’d add: confirm the proxy is really being used (the ipify check) before blaming it. I’ve seen requests silently go out direct because of a misconfigured proxies dict.

Curious how you’d handle sticky sessions with rotating residential proxies when a flow spans several pages. Do you pin one IP per session, or re-authenticate when it changes?

Collapse
 
devstackhub profile image
Dev Stack Hub • • Edited

Thanks for the thoughtful read, and good call on the ipify check. A silently misconfigured proxies dict is one of the most common reasons people end up blaming the proxy for something that never went through it. The usual culprit is a dict keyed only on "http" while the target is HTTPS` so requests quietly go out direct. I should have made that check step zero in the post.

Agreed on pairing classify() with an expected-content check too. Status code and block-page markers catch the obvious cases, but asserting that a known selector or field is present catches the subtle ones, like a trimmed page or a "lite" response served to suspected bots.

On sticky sessions: I pin one IP per flow. For anything spanning multiple pages (login, search, checkout, pagination with session state), the site ties cookies, CSRF tokens, and often fingerprint signals to the IP, so a mid-flow change usually invalidates the session or trips risk scoring. Most residential providers support this by embedding a session ID in the proxy username, which holds the same exit IP for a set window (typically 1 to 30 minutes, depending on the provider).

My rules of thumb:

  • One session ID per logical flow, with a fresh one for each new flow.
  • Keep the cookie jar and headers tied to that session, so IP, cookies, and fingerprint all rotate together, never separately.
  • Set the sticky window longer than your slowest expected flow, with some margin.
  • If the IP drops mid-flow anyway, don't try to re-authenticate and continue. Treat the flow as failed, discard the session state, and restart it cleanly on a new IP. Resuming on a new IP with old cookies is a classic way to get flagged.
  • Bench the session, not just the IP. If a session ID starts returning soft blocks, retire it using the same cooldown logic.

Re-authenticating is a fallback for flows that are expensive to restart, but I'd treat it as the exception.

How are you handling it on your end? Are your flows short enough that sticky windows aren't an issue?

Collapse
 
mayur-upadhyay profile image
Mayur Upadhyay •

The truth-table approach in Step 1 is a clean way to separate proxy issues from code issues.

A few technical details that have bitten me in similar setups:

  1. Exponential backoff isn’t in the code yet. The Step 3 text recommends it, but fetch() only sleeps for Retry-After (or the 30s default) and then polite_sleep(). If a server keeps returning 429 without a Retry-After header, every retry waits the same amount. Something like delay = min(60, 2 ** attempt) + random.uniform(0, 1) inside the loop would close that gap.

  2. Header order matters, and requests doesn’t preserve it like a browser does. Some anti-bot systems check header order along with the values, so a perfect Chrome User-Agent can still look off. httpx or curl_cffi give you more control here, and curl_cffi’s impersonate="chrome" also handles the TLS/HTTP2 fingerprint at the same time.

  3. SOCKS5 and DNS leaks. If you swap to SOCKS5 proxies as the article mentions, socks5:// makes requests resolve DNS locally, which can expose your real resolver and cause geo mismatches. Use socks5h:// so the proxy resolves the hostname.

  4. Reuse a Session per proxy. Creating a new connection for every request through a proxy adds latency and a new TLS handshake each time. A session per proxy keeps cookies consistent and makes the traffic pattern look more natural.

Have you measured how much a sticky session reduces CAPTCHA frequency compared to per-request rotation? I’d be curious about the numbers.

Collapse
 
devstackhub profile image
Dev Stack Hub •

You’ve caught a real gap, and the details you added are the ones that bite in practice.

Backoff: You’re right. The article recommends exponential backoff in Step 3, but fetch() only honors Retry-After or falls back to a flat 30s, so a server that returns 429 without the header gets the same wait every time. Your min(60, 2 ** attempt) + random.uniform(0, 1) fix is what I’d use, with Retry-After taking priority when it’s present. I’ll update the code so it matches the text.

Header order and TLS: Agreed. requests can’t control header order or the TLS/HTTP2 fingerprint, so a perfect Chrome User-Agent can still be flagged. For targets with serious bot detection, curl_cffi with impersonate="chrome" is the better tool, and I should have called that out rather than implying header values alone are enough.

socks5 vs socks5h: Good catch. With socks5:// the DNS lookup happens locally, which leaks your resolver and can cause geo mismatches. socks5h:// fixes that. I’ll add a note where SOCKS5 is mentioned.

Session per proxy: Fully agree. A session per proxy avoids a fresh TLS handshake on every request, keeps cookies consistent, and produces a more natural traffic pattern.

Sticky vs per-request rotation: I haven’t run a controlled comparison, so I don’t have clean numbers to share. What I’ve seen is anecdotal: sticky sessions noticeably cut CAPTCHA frequency on sites that tie challenges to session or cookie state, while per-request rotation worked better on simple rate-limit-by-IP targets. Both results depended heavily on the target, so any single figure would mislead. If you’ve measured it, I’d like to compare notes. I may do a proper benchmark across a few targets for a follow-up post.

Collapse
 
rafidbottler profile image
Rafid Bottler •

Solid guide, and the point about logging outcomes per proxy, URL and time is what makes debugging possible instead of guesswork.

A few things that helped me once this pattern was in production:

  1. Fingerprint block pages by hash. Soft-block markers work until the site changes its wording. Hashing a normalized version of the response body (strip digits, timestamps and tokens first) lets you spot when many “successful” responses are actually the same page. If 40 different product URLs return an identical hash, you’re being served a challenge, even if no marker matches.

  2. Pick proxies by success rate, not random.choice. ProxyPool.get() treats every healthy IP equally. Tracking a rolling success rate (say, the last 50 requests) and weighting selection toward better performers means good IPs get used more and mediocre ones fade out before they even hit the bench threshold.

  3. Reduce requests before optimizing evasion. Conditional requests (If-None-Match with the stored ETag, or If-Modified-Since) return a cheap 304 for unchanged pages. For recurring scrapes, this can cut request volume dramatically, and fewer requests means fewer 429s. Many sites that rate-limit aggressively still honor conditional headers.

  4. Apply a per-domain rate limit when you go async. If you move from requests to asyncio with httpx or aiohttp, a plain semaphore caps total concurrency but not the rate per target. A token bucket per domain (for example, 2 requests per second regardless of how many workers or proxies you have) keeps pacing predictable.

  5. Respect Crawl-delay in robots.txt as a free hint. Some sites publish it, and it’s a good starting point for your base delay, since it’s roughly what the site owner considers acceptable.

Do you also track latency per proxy? I’ve found a sudden jump in response time often shows up a few minutes before blocks start, which makes it a useful early warning.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks, this is a really useful list. A few thoughts on each:

Hash fingerprinting: Agreed, and this is a gap in the post. Marker matching breaks the moment a site rewords its challenge page. Normalizing the body before hashing and flagging when many distinct URLs collapse to one hash is a much sturdier signal. I’m adding it to the detection section.

Success-rate weighting: Fair criticism. random.choice treats every healthy IP the same, and a rolling window of the last 50 requests with weighted selection would let weak proxies fade out before they hit the bench threshold. I’ll update the ProxyPool example.

Conditional requests: This is the point I most wish I’d included. Cutting request volume with ETag and If-Modified-Since is the cheapest way to avoid 429s, and it’s far better than any evasion tweak. Worth its own section.

Per-domain token bucket: Good call. A semaphore caps concurrency but says nothing about the rate hitting a single target, so I’ll mention this for the async version.

Crawl-delay: A nice free hint for setting the base delay. Not every site publishes it, but it costs nothing to read.

On your question: I didn’t cover latency per proxy in the post, but your experience makes sense. A rising response time often means the target is throttling or queueing a proxy before it starts returning hard blocks. Tracking a rolling median latency per proxy and alerting when it jumps well above its baseline would make a good early-warning signal. I’ll look at adding it to the logging section.

Collapse
 
alexshev profile image
Alex Shev •

Useful breakdown. One caution: proxy rotation can hide the signal that a scraper should slow down or stop. I’d put rate budgets, robots/terms checks, and a clear exit condition ahead of more aggressive routing, then treat 403/429 trends as observability data rather than a bypass challenge.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks, this is a fair point, and I agree with it. Rotation can mask the very signal a scraper should be listening to. If every 429 just triggers a new IP, you've turned a "slow down" message into a retry loop.

I'd frame the post's advice as a diagnostic aid, not a license to push harder. Rate budgets per target, robots.txt and terms checks, and a clear stop condition should come first. Proxies come after that, and mainly for legitimate cases like geo-specific testing or spreading load you're already allowed to send.

Your suggestion to treat 403/429 trends as observability data is the right way to look at it. A rising 429 rate is feedback about your request pattern, and the response should be backoff, lower concurrency, or contacting the site owner, not a smarter bypass. I'll add a short section to the article on setting rate budgets and exit conditions so that framing is explicit. Appreciate you raising it.

Collapse
 
harperelise profile image
Harper Elise Callahan •

Step 1 is the right way to debug this. Run the same request direct and through the proxy, with default and browser headers, and you can tell whether the problem is the IP, the headers or the volume. Most people skip this and rotate proxies blindly. Also verify the exit IP with an echo endpoint like ipify, because mixed-up http/https keys in the proxies dict can send requests out directly without any error.

The soft-block check matters because a CAPTCHA page returned with a 200 passes every status-code check. I’d add a content assertion on top of the keyword markers, such as a required selector, JSON key or minimum body size, since marker strings break when the site changes its challenge page. Benching a proxy right after a 429 while requiring a failure streak for timeouts is the correct split, because 429s are tied to the IP and timeouts are often transient.

For 403s on clean IPs, TLS fingerprinting is the likely cause. Python’s default stack has a JA3 signature that doesn’t match a browser’s, so headers and IP quality don’t help. Switching to curl_cffi with browser impersonation is usually the fastest test before going to Playwright. For cookie-based flows, keep one sticky IP per session and rotate only between sessions, not between requests.

Collapse
 
devstackhub profile image
Dev Stack Hub •

You’ve picked up on the parts that cost people the most time in practice.

On Step 1, I agree. Rotating proxies blindly is the most common mistake, and the direct vs. proxy, default vs. browser headers matrix is the fastest way to find out whether it’s the IP, the headers, or the volume. Your ipify point is a good one too. A mixed-up http/https key in the proxies dict can send traffic out directly with no error, so verifying the exit IP should be a standard check, and I’ll add it to the article.

You’re right about the soft-block check. Keyword markers break the moment a site changes its challenge page. A required selector, a JSON key, or a minimum body size makes a sturdier second layer, and I’d recommend using both together.

On TLS fingerprinting, I agree that it’s the likely culprit when a clean IP still gets a 403. Python’s default stack has a JA3 signature that doesn’t look like a browser, so no amount of header tuning or IP swapping will fix it. Trying curl_cffi with browser impersonation before reaching for Playwright is a cheap test that saves a lot of time.

Your sticky session advice is spot on as well. For cookie-based flows, one IP per session, rotating only between sessions, avoids a lot of confusing failures.

I’ll fold the exit IP check, the content assertions, and the TLS fingerprinting note into the next revision. Thanks again for the detailed feedback.

Collapse
 
levinemundro profile image
Levine Mundro •

One caveat on Step 1: curl and requests can differ even with identical headers, because curl negotiates HTTP/2 while requests only speaks HTTP/1.1, and some WAFs score that. If curl passes and requests fails, check the protocol and TLS layer before blaming the proxy.

For the classifier, checking headers like cf-mitigated: challenge or challenge cookies (datadome, _abck) is more reliable than keyword matching on the first 5 KB of the body. I’d also track proxy health per (proxy, domain) pair, since an IP burned on one site is often fine on another.

For 403s on clean IPs, it’s usually JA3/h2 fingerprinting. curl_cffi with impersonate='chrome124' fixes it quickly, but keep the UA version matched to the TLS profile.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks, these are all good points, and a couple of them show gaps in the post.

curl vs requests (Step 1): You’re right. I framed it as “if curl passes and requests fails, it’s your code,” but that’s too loose. curl negotiates HTTP/2 and requests only speaks HTTP/1.1, and the TLS handshake differs too, so some WAFs score the two differently even with identical headers. I’ll add a note that when curl passes and requests fails, you should check the protocol and TLS layer before blaming the code. Running curl --http1.1 is a quick way to see whether the protocol is what matters.

Classifier (Step 2): Agreed that keyword matching on the first 5 KB is the weakest part of the post. I included it because it’s easy to follow, but checking cf-mitigated: challenge and challenge cookies like datadome and _abck is more reliable and has fewer false positives. I’ll update classify() to check headers and cookies first and keep the body markers only as a fallback.

Per-domain proxy health (Step 4): This is the best point. An IP burned on one site is often fine on another, so the ProxyPool should track health per (proxy, domain) pair rather than per proxy. Otherwise one hostile target benches IPs that would work elsewhere. I’ll rework the pool to key failures and benched_until on that pair.

JA3/h2 on clean IPs (Step 6): I only mentioned TLS fingerprinting briefly, and curl_cffi with impersonate='chrome124' deserves more than that. Your caveat about keeping the UA version matched to the TLS profile is important. A Chrome 124 handshake with a Chrome 110 User-Agent is itself a signal.

I’ll fold these into an update and credit you in the post. If you have a go-to way to check which layer is flagging you, such as comparing against a JA3 echo service, I’d like to hear it.

Collapse
 
sidra-jefferi profile image
Sidra Jefferi •

One gap I noticed: Step 3 recommends exponential backoff and per-IP concurrency limits, but the code in fetch() doesn’t implement either. Every retry uses the same polite_sleep() delay, so a persistently throttled target is retried at a constant pace. Something like delay = min(cap, base * 2 ** attempt) + jitter would make the code match the advice, and a per-proxy semaphore (or a simple “last request time per IP” map) would enforce the concurrency rule.

A related detail for the SOCKS5 mention: with requests, socks5:// resolves DNS on your machine, while socks5h:// resolves it through the proxy. If a site serves geo-specific content, local DNS resolution can leak your real location even though the exit IP is correct. The IP-echo check from Step 1 wouldn’t catch that.

Have you compared success rates for the same target with requests versus curl_cffi? I’d be curious how much of the 403 gap on clean IPs is TLS fingerprinting.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks, this is a fair catch on all three points.

Backoff and concurrency: You’re right. Step 3 recommends both, but fetch() only uses Retry-Afteron 429s and falls back to the same polite_sleep() for everything else, so a persistently throttled target gets retried at a constant pace. I’m updating the code to match the advice:

def backoff_delay(attempt, base=1.0, cap=60.0):
    return min(cap, base * 2 ** attempt) + random.uniform(0, 1.5)
Enter fullscreen mode Exit fullscreen mode

I’ll use it in place of polite_sleep() on retries, and take the larger of that and Retry-After for 429s. For concurrency, I’ll add a per-proxy “last request time” map to ProxyPool, since it’s the simplest version to show without pulling in asyncio.

socks5 vs socks5h: Good point, and I should have called it out. With socks5://, requests resolves DNS locally, so a geo-sensitive site can see your real resolver’s location even when the exit IP is correct. socks5h:// fixes that. You’re also right that the IP-echo check in Step 1 can’t catch it, since it only verifies the exit IP. I’ll add a note on this to the geo-mismatch section.

requests vs curl_cffi: I haven’t run a controlled comparison on the same target with the same clean IPs, so I don’t want to quote a number I can’t back up. The article only flags TLS fingerprinting as the next thing to test. If you’ve measured it, I’d love to hear what you saw. I’ll try running the same probe with both clients and share the results here or in a follow-up.

Collapse
 
debugtodeploy profile image
Vinay Shah •

The “change one variable at a time” framing in Step 1 is what most scraping tutorials skip, and the direct-vs-proxy, default-vs-browser headers truth table is a quick way to stop guessing.

A few things that stood out:

  • Soft blocks are underrated. Treating a 200 response as a failure when it’s actually a CAPTCHA page saves a lot of silent bad data. Asserting that the expected content is present (a product title or JSON key) works well alongside the marker list, since marker strings can drift when a site changes its challenge page.
  • Benching 429s immediately is a smart default. It stops one burned IP from dragging down the whole pool.
  • Small suggestions for the fetch loop: you mention exponential backoff in Step 3, but the loop uses a fixed polite_sleep() after failures, so it might be worth wiring the backoff in. If people run this with threads, ProxyPool will also need a lock around failures and benched_until.
  • Sticky sessions. Step 6 could stress that a rotating proxy mid-session often causes 403s by itself. Cookies set on one IP and replayed from another are a classic trigger.

Also glad you ended with the “check for an official API and read the ToS first” reminder. It gets skipped far too often.

Curious whether you’ve seen curl_cffi fully resolve TLS-fingerprint blocks in practice, or whether you still end up needing Playwright for the stricter sites.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Thanks for the careful read, and you’re right on both code points.

Backoff: The loop should use the exponential backoff from Step 3 instead of a fixed polite_sleep(). Something like sleep = base * 2**attempt + random.uniform(0, 1) makes retries cheaper on the target and less predictable. I’ll update the snippet so the code matches the text.

Thread safety: You’re also right that ProxyPool isn’t safe with threads as written. Concurrent updates to failures and benched_until can race, so a threading.Lock around those reads and writes is the minimal fix. I’ll add a note, since people will definitely copy this into a threaded worker.

Your content-assertion point is a good complement to marker lists. Markers break when a site redesigns its challenge page, while “the product title or expected JSON key is present” fails closed. I’ll mention using both.

On sticky sessions, agreed: rotating mid-session is a self-inflicted 403 more often than people think. Cookies and tokens issued to one IP and replayed from another look exactly like session hijacking to a WAF. I’ll expand Step 6 to say: pin one proxy per session, rotate only between sessions.

On curl_cffi: in my experience it resolves a lot of TLS/JA3 fingerprint blocks, and often that’s all a site needs, and it’s far lighter than a full browser. But it doesn’t solve everything. Stricter sites check JavaScript execution, canvas/WebGL signals, or behavioral telemetry, and there I still end up reaching for Playwright. My rule of thumb is to try curl_cffi with browser impersonation first, and escalate to Playwright only when the response is still a challenge page.

Thanks for the kind words on the ToS reminder, too. Appreciate you taking the time to write this up.

Collapse
 
kevinhall profile image
Kevin Hall •

Soft blocks are a billing problem too. On a pay-per-result scraper a 200 CAPTCHA page can land in the dataset as a row, so I would validate a field that only a real page has, not just the status code, before a run counts as a success.

Collapse
 
devstackhub profile image
Dev Stack Hub •

Agreed, and thanks for raising the billing angle. I only touched on it in Step 2, where I suggested asserting that the expected content is present, but you've put it better: on a pay-per-result scraper, a CAPTCHA page that lands in the dataset as a row costs you money.

I'd go further than the marker list. Block-marker strings are a deny-list, and they break whenever the site changes its challenge page. Checking for a field that only a real page has (a product title, a price, a JSON key) is an allow-list, so it fails safe: anything unfamiliar counts as a failure instead of slipping through. I'd use both, with the positive check deciding whether a run counts as a success.

A few things I'd add:

  • Track the share of failed validations per run. If a run reports "success" but 30% of its rows failed the field check, that's a block, and the pass rate is a better health signal than status codes.
  • Only count a result as billable once it passes validation, and benching the proxy that served a soft block applies here too.
  • Be careful with the required field. Pick something that is always present on real pages, or legitimately empty pages will be flagged as blocks.

I'll add a validation step to the guide's fetch function. Thanks for the suggestion!

Collapse
 
mp3sss31 profile image
mp3sss •

The isolation matrix is the most useful part — most scraper debugging is debugging-by-stabbing, and the direct-vs-proxy truth table turns it into bisection. Two additions from running desktop clients against the same walls: treat Retry-After as a first-class signal (a pool that ignores it just thrashes harder), and score proxy health on a decaying error window so bad IPs get benched automatically instead of on a fixed timer. Worth noting a 200-with-CAPTCHA shows up inside headless browsers too — behavior fingerprints apply there as much as raw HTTP.