You pull a couple of Amazon product pages and it works. Then the listings turn into robot checks. Amazon runs one of the meanest anti-bot stacks on the web, and the first thing it weighs is how many requests come off your address.
Amazon flags a busy IP in minutes
Amazon counts requests per IP and personalises hard, so a burst from one address is flagged inside a few minutes. The tool is fine. The address is the problem. Everything leaves through one IP, the target counts requests per address, and past a certain rate it stops trusting you. First the responses drag. Then a 503 robot check shows up on every call. You can tune headers all day and it will not help, because this is a volume problem sitting on a single address.
What a block actually looks like
It rarely fails cleanly. It rots. The worst stage is the third, where you still get 200 OK but the numbers are wrong, so a run looks fine and is worthless. By the time you spot it, half the dataset is already poisoned.
| Stage | What you see | What the site is doing |
|---|---|---|
| 1 | Responses slow down | Rate-limiting your IP |
| 2 | A CAPTCHA on every page | Flagged the address as automated |
| 3 | Empty or decoy results | Feeding you junk to waste the run |
| 4 | 429s or refused connections | Temporary block on the IP |
Scrape Amazon from a rotating pool
The fix is not cleverer code. It is more addresses. private proxies for Amazon give you a dedicated IPv4 and SOCKS5 pool with rotation built in, so you pull rotating proxies and keep every address under the frequency that trips the anti-bot. The script you already have starts finishing its runs.
Wiring a proxy in
Two lines, not a rewrite.
import requests
PROXY = "http://USER:PASS@HOST:PORT" # WinGate, rotating
r = requests.get("https://www.amazon.com/",
proxies={"http": PROXY, "https": PROXY},
headers={"User-Agent": "Mozilla/5.0"}, timeout=30)
print(r.status_code)
Rotate every few requests, keep the per-IP rate sane, and jitter the timing so it is not machine-even. undetected-chromedriver plugs in the same way, and a Python scraping stack follows the identical pattern. Start slow, then push the rate while you watch the block count.
Rotating across the pool
Once the proxy is wired in, rotation is a few lines. Keep a list of pool endpoints and pick a fresh one per request.
import random
POOL = ["http://USER:PASS@h1:PORT", "http://USER:PASS@h2:PORT"] # WinGate
def fetch(url):
p = random.choice(POOL)
return requests.get(url, proxies={"http": p, "https": p},
headers={"User-Agent": "Mozilla/5.0"}, timeout=30)
In production you would pull the pool from config and weight it, but the shape does not change: one address per request, none of them overworked.
Backing off when a 503 robot check appears
Even with rotation you will meet the odd limit. Do not hammer through it. Back off, then let the next attempt land on a different address.
import time
for attempt in range(5):
r = fetch(url)
if r.status_code in (403, 429):
time.sleep(2 ** attempt) # exponential back-off
continue # next fetch() rotates the IP
break
Exponential back-off plus a fresh exit clears most transient blocks without a solver in sight.
Reading what the site tells you
The status line is a diagnosis if you read it. Quick key:
- 403 the address is not trusted, rotate to a clean one.
- 429 you went too fast on one IP, back off and add addresses.
- 503 or a challenge page the anti-bot flagged you, drop the rate.
- 200 with wrong data the nastiest one, you are being fed decoys.
Most debugging is matching one of these to the fix next to it, not rewriting the parser.
Rotate or pin? Depends on the job
Stateless scraping wants rotation. A logged-in session wants the opposite. Pin one address from private addresses of your own so the account never watches its IP jump and ask you to log in again. Most projects need both at once, and keeping the two apart is half the job.
| Job | Address strategy | Why |
|---|---|---|
| Stateless scraping | Rotate per request | No IP builds a rate |
| Logged-in session | One pinned IP | The account never sees the IP move |
| Mixed pipeline | Rotate collectors, pin sessions | Keeps volume and identity apart |
CAPTCHAs: cause before cure
A CAPTCHA is a symptom, and the cause is almost always frequency on one address. Clean rotation removes the signature that triggers most of them, which is why the network comes before the solver. For the leftovers, keep the solving traffic clean with anti-captcha proxies and CapMonster behind clean IPs. A 429 says the same thing: back off, add addresses.
Datacenter or residential, plainly
Datacenter is fast and cheap and carries most targets. Residential blends in where the anti-bot is fussy about ranges. What matters more than the label: the address is yours. a private IPv4 that nobody else touches beats a shared "residential" one with a spoiled past.
| Datacenter IPv4 | Residential-grade | |
|---|---|---|
| Speed | Fast | Slower |
| Cost | Low | Higher |
| Survives strict anti-bot | Sometimes | Usually |
| Best for | Most targets | The nastiest defences |
SOCKS5, and running wide
Reach for SOCKS5 whenever a tool will not take a plain HTTP proxy. It relays raw TCP, so it fits scripts, headless browsers and schedulers alike. With headroom up to 5000 threads, heavy parallelism gets served instead of queued.
Geography: the right regional data
Amazon scraping often returns different content or prices by region, so one location gives you a lopsided sample and never errors to warn you. A worldmix pool lets you place the exit where you need it and compare regions in a single run. If your buyers span several markets, that is the difference between a real picture and a guess.
How many addresses do you actually need
Rough maths beats guessing. Take your target requests per hour and divide by a safe per-IP rate the site tolerates. If you want 20,000 requests an hour and one address survives about 400, you need on the order of fifty clean addresses, not five worked to death. Size the pool to the workload, then add headroom, and you stop rediscovering the limit the hard way.
Private versus public addresses
A public proxy is shared by thousands and already flagged, so a request through it is suspect before the server answers, and you catch a 503 robot check on the first hit. A private IPv4 is yours alone. Its record is clean because no stranger spoiled it, and the pass rate holds steady enough to plan a run around.
Behaviour: headers and timing
An address fixes the network, not the manners. A perfectly even request rhythm and one static User-Agent still read as a robot. Vary the headers within reason, add a bit of jitter between calls, and keep concurrency believable. The IP, the request shape, and the timing get judged together, so all three have to look human at once.
Traffic you do not count
This work moves real bandwidth, and a metered plan taxes exactly what you came to do. With bandwidth you do not meter you size the pool by addresses, not gigabytes, and run a full crawl without watching a meter. It also makes the bill predictable, because you pay for dedicated capacity instead of guessing how many gigabytes a job will eat.
Mistakes that bring the blocks back
Too high a rate on one address, public proxies off a shared list, no pauses, one request template that ignores Amazon per-region differences. Any one of them rebuilds the signature you just cleared, and a 503 robot check is back. The cure is dull and it works: clean private addresses, a sane per-IP rate, rotation for volume, sticky IPs for logins, a solver only for the scraps.
A quick pre-run checklist
- Bind a handful of clean addresses and turn on rotation.
- Cap the per-IP rate below where a 503 robot check first showed up.
- Add jitter and realistic headers so the timing is not machine-even.
- Watch the block rate as you scale threads, not after the run.
Try it before you trust it
Do not take my word for it. WinGate gives you a trial that runs up to two hours, which is enough to run Amazon scraping against your actual target and watch the block rate. If it passes, add addresses and scale. If it does not, walk away.
Top comments (0)