Building a B2B lead list means pulling public contact and company data from directories and search across many pages fast, and from one address those sources rate-limit and block within minutes.
Directories rate limit heavy crawling
Directory and search sources count requests per IP and challenge mechanical crawling. The tool is fine. The address is the problem. Everything leaves through one IP, the target counts requests per address, and past a certain rate it stops trusting you. First the responses drag. Then a block shows up on every call. You can tune headers all day and it will not help, because this is a volume problem sitting on a single address.
Crawl directories across a pool
The fix is not cleverer code. It is more addresses. private proxies for lead generation give you a dedicated IPv4 and SOCKS5 pool with rotation built in, so you pull a rotating pool and keep every address under the frequency that trips the anti-bot. The script you already have starts finishing its runs.
Wiring a proxy in
The change is tiny.
import requests
PROXY = "http://USER:PASS@HOST:PORT" # WinGate, rotating
r = requests.get("https://example.com/",
proxies={"http": PROXY, "https": PROXY},
headers={"User-Agent": "Mozilla/5.0"}, timeout=30)
print(r.status_code)
Add a little random delay and rotate on a counter. If you use Scrapy, the proxy goes in the same place. I keep the rate low at first and only raise it once a few hundred requests come back clean. a Python scraping stack works exactly like this.
Rotating across the pool
Once the proxy is wired in, rotation is a few lines. Keep a list of pool endpoints and pick a fresh one per request.
import random
POOL = ["http://USER:PASS@h1:PORT", "http://USER:PASS@h2:PORT"] # WinGate
def fetch(url):
p = random.choice(POOL)
return requests.get(url, proxies={"http": p, "https": p},
headers={"User-Agent": "Mozilla/5.0"}, timeout=30)
In production you would pull the pool from config and weight it, but the shape does not change: one address per request, none of them overworked.
Datacenter or residential, plainly
People overthink this. A clean, dedicated a private IPv4 handles the large majority of work. Reach for residential only when a target is openly hostile to datacenter ranges. A private address you do not share beats a "residential" one a hundred strangers already burned.
| Datacenter IPv4 | Residential-grade | |
|---|---|---|
| Speed | Fast | Slower |
| Cost | Low | Higher |
| Survives strict anti-bot | Sometimes | Usually |
| Best for | Most targets | The nastiest defences |
CAPTCHAs: cause before cure
Most people fight CAPTCHAs with a bigger solver bill. Wrong order. Fix the address first, because rotation kills the pattern that summons a block in the first place. Then, for whatever still slips through, run anti-captcha proxies and CapMonster behind clean IPs so the solver is not working from a flagged IP.
Reading what the site tells you
The status line is a diagnosis if you read it. Quick key:
- 403 the address is not trusted, rotate to a clean one.
- 429 you went too fast on one IP, back off and add addresses.
- 503 or a challenge page the anti-bot flagged you, drop the rate.
- 200 with wrong data the nastiest one, you are being fed decoys.
Most debugging is matching one of these to the fix next to it, not rewriting the parser.
Region matters more than you think
Prices, stock and even layout shift by country, and scraping from one place hides all of it behind numbers that look fine. Put exits in the regions you care about and the sample stops lying. A worldmix pool makes that a config choice, not a second project.
Rotate or pin? Depends on the job
Here is the rule I use. No login, rotate hard. Behind a login, pin one address from the private pool and leave it. Mixing the two is how sessions die: a logged-in profile hates a moving IP.
| Job | Address strategy | Why |
|---|---|---|
| Stateless scraping | Rotate per request | No IP builds a rate |
| Logged-in session | One pinned IP | The account never sees the IP move |
| Mixed pipeline | Rotate collectors, pin sessions | Keeps volume and identity apart |
Backing off when a block appears
Even with rotation you will meet the odd limit. Do not hammer through it. Back off, then let the next attempt land on a different address.
import time
for attempt in range(5):
r = fetch(url)
if r.status_code in (403, 429):
time.sleep(2 ** attempt) # exponential back-off
continue # next fetch() rotates the IP
break
Exponential back-off plus a fresh exit clears most transient blocks without a solver in sight.
Behaviour: headers and timing
An address fixes the network, not the manners. A perfectly even request rhythm and one static User-Agent still read as a robot. Vary the headers within reason, add a bit of jitter between calls, and keep concurrency believable. The IP, the request shape, and the timing get judged together, so all three have to look human at once.
SOCKS5, and running wide
Reach for SOCKS5 whenever a tool will not take a plain HTTP proxy. It relays raw TCP, so it fits scripts, headless browsers and schedulers alike. With headroom up to 5000 threads, heavy parallelism gets served instead of queued.
How many addresses do you actually need
Rough maths beats guessing. Take your target requests per hour and divide by a safe per-IP rate the site tolerates. If you want 20,000 requests an hour and one address survives about 400, you need on the order of fifty clean addresses, not five worked to death. Size the pool to the workload, then add headroom, and you stop rediscovering the limit the hard way.
Traffic you do not count
This work moves real bandwidth, and a metered plan taxes exactly what you came to do. With bandwidth you do not meter you size the pool by addresses, not gigabytes, and run a full crawl without watching a meter. It also makes the bill predictable, because you pay for dedicated capacity instead of guessing how many gigabytes a job will eat.
Private versus public addresses
A public proxy is shared by thousands and already flagged, so a request through it is suspect before the server answers, and you catch a block on the first hit. A private IPv4 is yours alone. Its record is clean because no stranger spoiled it, and the pass rate holds steady enough to plan a run around.
Which address for which task
No single right proxy. Just the right one for the job in front of you.
| Situation | Reach for |
|---|---|
| High-volume lead collection | Rotating private IPv4 |
| A tool that only speaks SOCKS | SOCKS5 from the pool |
| A logged-in account | One sticky private address |
| Region-specific data | A worldmix exit in that region |
Mistakes that bring the blocks back
Too high a rate on one address, public proxies, no pauses, one request template. Any one of them rebuilds the signature you just cleared, and a block is back. The cure is dull and it works: clean private addresses, a sane per-IP rate, rotation for volume, sticky IPs for logins, a solver only for the scraps.
A quick pre-run checklist
- Bind a handful of clean addresses and turn on rotation.
- Cap the per-IP rate below where a block first showed up.
- Add jitter and realistic headers so the timing is not machine-even.
- Watch the block rate as you scale threads, not after the run.
Try it before you trust it
You can judge a proxy on your own workload in an afternoon. WinGate has a trial that runs up to two hours. Point a clean lead-gen pool at your real script, run it across a few threads, and compare the CAPTCHA and block rate to what you get now. Holds up? Scale to your volume. Does not? You lost nothing.
Top comments (0)