Every scraper eventually asks the same question: how many threads should I run? Turn it up and the job finishes faster, until it does not, because somewhere past a threshold the target starts throttling and your effective throughput drops even as your thread count climbs. The right number is not "as many as your machine can handle." It is a balance between concurrency, rate limits, and how many addresses you are spreading across. Here is how to think about tuning it.
Two different ceilings
There are two separate limits, and people confuse them constantly.
- Your machine's ceiling. CPU, memory, and open sockets. This caps how many threads you can run at all.
- The target's ceiling. How many requests it accepts from one address before throttling. This caps how many threads you should run per address.
Maxing out the first while ignoring the second is the classic mistake. You run 500 threads, they all leave from one address, and the target throttles the address, so your 500 threads spend their time waiting on 429s. High concurrency through one exit is not fast, it is just a bigger traffic jam.
Concurrency and addresses scale together
The insight that fixes this: your safe concurrency is roughly the per-address limit multiplied by the number of addresses you spread across. If one address tolerates 5 requests a second and you have a pool cycling 100 addresses, your headroom is far higher than either number alone. So the way to raise throughput is not just more threads, it is more threads paired with more exits.
With a rotating pool, this happens naturally. You raise the thread count, each request draws a fresh exit, and no single address carries enough load to trip its limit. Concurrency scales because the load is distributed, not concentrated.
Finding the number empirically
Do not guess. Measure.
- Start modest, then increase concurrency in steps while watching two metrics: successful responses per second, and the error and 429 rate.
- Effective throughput rises with concurrency up to a point, then flattens or drops as errors climb. That inflection is your ceiling.
- Add a small random delay per request so you are not a synchronized burst even through a pool.
- Tune down if the error rate climbs; a lower thread count with a 99 percent success rate beats a higher one wasting half its requests on retries.
The goal is the highest concurrency where the success rate stays clean, not the highest concurrency your machine allows.
What the pool needs to let you scale
Raising concurrency only helps if the pool behind it can absorb the load. WinGate supports up to 5000 threads and draws from a rotating worldmix pool, so you can scale concurrency while each exit stays modest. The addresses are private IPv4 rather than shared, so you are not competing with strangers for the same rate budget, traffic is unlimited so heavy runs do not meter you, and it speaks HTTP, HTTPS, and SOCKS5. That combination is what lets you turn the thread count up without every request funneling through one throttled address.
An honest note: more threads plus more exits raises your safe throughput, it does not make a target's limits disappear, and it does not exempt you from its terms. A high thread count aimed at one endpoint with no pacing will still cause harm and still get blocked. Scale concurrency and address spread together, and keep the total request rate reasonable. There is a free 2 hour test, so run the step-up measurement against your own target and find the real ceiling before you commit.
The takeaway: there are two ceilings, your machine and the target, and threads through one address hit the target's ceiling fast. Scale concurrency together with a rotating pool so load spreads, measure the point where success rate drops, and run just under it. That is the fastest a scrape can safely go.
Top comments (0)