Asyncio Semaphores Are the Wrong Knob: Per-Exit Concurrency Caps for Rotating Proxy Pools

Most beginner async scrapers cap concurrency with a single asyncio.Semaphore. It is the first piece of advice in nearly every "scraping with asyncio" tutorial: wrap your fetch function in async with sem: and you will never have more than N requests in flight. Then you add rotating residential proxies, and the semaphore stops meaning anything useful — because the thing that actually gets overwhelmed is not your event loop. It is the exit.

This article makes one argument: in a proxied pipeline, concurrency is a property of the exit, not of the worker, and you should cap it with a per-exit lease pool instead of a global semaphore.

Why the global semaphore fails once proxies enter the picture

Say you run Semaphore(20) across 8 async workers and pull a fresh rotating proxy for every request. At any moment, the 20 in-flight requests might be spread across 20 different exits — or, with a sticky session, five requests might pile onto one residential IP while fifteen pile onto another. The global semaphore bounds neither case in a way you can reason about.

From the origin's perspective, your pipeline does not exist. What exists is a set of exit IPs, each emitting requests. Rate heuristics, per-IP token buckets, and bandwidth caps on the proxy side all key off that exit. A single residential gateway will happily accept thirty concurrent TLS streams from you until the upstream connection pool behind it saturates, at which point latency doubles, then triples, then everything times out — and your global semaphore never noticed, because it was counting tasks, not sockets per exit.

The failure signature is nasty for beginners: everything "works" at Semaphore(5), falls over at Semaphore(30), and nobody can say why, because the bottleneck is not in the code you wrote.

The right model: a lease pool keyed by exit

A per-exit cap has three moving parts:

1. A registry of live exits (a proxy session or a gateway endpoint identifies one).
2. A small semaphore per exit, created lazily.
3. An acquisition step that picks an exit first, then asks that exit for permission.

Here is a complete, runnable lease pool. It caps each exit at per_exit concurrent requests and refuses to overload a dying exit:

import asyncio
import random
from collections import defaultdict


class ExitLeasePool:
    """Concurrency capping keyed to proxy exits instead of tasks."""

    def __init__(self, exits, per_exit: int = 3):
        self._exits = list(exits)                      # e.g. proxy URLs
        self._per_exit = per_exit
        self._sems = defaultdict(lambda: asyncio.Semaphore(self._per_exit))
        self._inflight = defaultdict(int)              # observability

    def _pick_exit(self) -> str:
        # Least-loaded-first beats round-robin: spreads load away
        # from exits that are already saturated or slowing down.
        candidates = sorted(
            self._exits,
            key=lambda e: self._inflight[e] / self._per_exit,
        )
        head = candidates[: max(1, len(candidates) // 2)]
        return random.choice(head)

    async def fetch(self, client, url: str, timeout: float = 12.0):
        exit_url = self._pick_exit()
        sem = self._sems[exit_url]
        async with sem:                                # per-exit cap
            self._inflight[exit_url] += 1
            try:
                async with client.get(
                    url, proxy=exit_url,
                    timeout=aiohttp_timeout(timeout),
                ) as resp:
                    return resp.status, await resp.read()
            finally:
                self._inflight[exit_url] -= 1


def aiohttp_timeout(t: float):
    import aiohttp
    return aiohttp.ClientTimeout(total=t)

And a small driver that proves the invariant — at no moment does any exit carry more than per_exit requests:

import aiohttp


async def main():
    exits = [f"http://user-portal{i}:[e-mail]:9000" for i in range(6)]
    pool = ExitLeasePool(exits, per_exit=3)
    urls = [f"https://httpbin.org/delay/1?i={i}" for i in range(60)]

    observed_max = defaultdict(int)

    async with aiohttp.ClientSession() as client:
        async def one(u):
            # record which exit took it (demo: re-pick is internal,
            # so we instrument via pool._inflight during the call)
            t = asyncio.create_task(pool.fetch(client, u))
            while not t.done():
                for e, n in pool._inflight.items():
                    observed_max[e] = max(observed_max[e], n)
                await asyncio.sleep(0.02)
            return await t

        await asyncio.gather(*(one(u) for u in urls))

    print(dict(observed_max))   # every value <= 3

Run it and every exit's peak concurrency is <= per_exit. Try the same instrumentation with a global Semaphore(18) and you will watch one exit spike to nine while another idles.

Sizing per_exit without guessing

The number is not mystical. Two measurements give you a sane starting point:

• Latency knee. Send 1, 2, 4, 8 concurrent requests through one exit to a stable endpoint and plot mean latency. The concurrency level where latency exceeds ~2x your single-request baseline is your knee. Set per_exit one step below it.
• Provider contract. Some residential plans bill or throttle per concurrent session, not per request. Read that clause; it is a hard cap that sits under whatever your latency test suggests.

Starting values that work in practice: per_exit = 2–3 for rotating residential gateways, 4–6 for static residential or ISP exits, and whatever your VPS bandwidth allows for datacenter exits. The point of the lease pool is that changing tiers means changing one number, not restructuring your code.

The bonus: a natural place for health signals

Once every request passes through ExitLeasePool.fetch, you have a single chokepoint to record per-exit telemetry: latency, 429 counts, TLS errors. When an exit's error rate crosses a threshold, bump its semaphore down or evict it from self._exits mid-run. That kind of adaptation is impossible with a bare global semaphore, because there is no object to attach state to.

A benchmark that finds the knee, not a guess

Two numbers decide per_exit: the exit's latency knee and the provider's concurrent-session limit. The first you measure yourself. Here is a compact harness — it walks concurrency levels 1, 2, 4, 8, 16 through a single exit and reports the median latency, error rate, and where the cliff appears:

import asyncio, statistics
import aiohttp

async def probe(exit_url: str, target: str, level: int) -> dict:
    lat = []
    errs = 0
    connector = aiohttp.TCPConnector(limit=level)
    async with aiohttp.ClientSession(connector=connector) as s:
        async def one():
            nonlocal errs
            loop = asyncio.get_running_loop()
            t0 = loop.time()
            try:
                async with s.get(target, proxy=exit_url, timeout=30) as r:
                    await r.read()
                    if r.status >= 500:
                        errs += 1
            except Exception:
                errs += 1
                return
            lat.append(loop.time() - t0)
        await asyncio.gather(*(one() for _ in range(level * 4)))
    base = statistics.median(lat) if lat else float("inf")
    return {"level": level, "median_ms": base * 1000,
            "p95_ms": (statistics.quantiles(lat, n=20)[-1] * 1000) if len(lat) >= 4 else base * 1000,
            "err_rate": errs / max(1, level * 4)}

async def main():
    EXIT = "http://USER-SESSION-bench:[e-mail]:9000"
    TARGET = "https://httpbin.org/delay/1"
    rows = []
    for level in (1, 2, 4, 8, 16):
        r = await probe(EXIT, TARGET, level)
        rows.append(r)
        print(f"concurrency={level:<3} median={r['median_ms']:6.0f} ms "
              f"p95={r['p95_ms']:7.0f} ms err={r['err_rate']:.0%}")
    baseline = rows[0]["median_ms"]
    for r in rows:
        if r["median_ms"] > 2 * baseline or r["err_rate"] > 0.05:
            print(f"knee at level {r['level']} -> set per_exit = {r['level'] // 2}")
            break

asyncio.run(main())

Run it once per tier and once per region you plan to use, and store the results in config next to the exit list. Then, when the provider changes hardware under the same product name — which they do — the number is stale, not your reasoning. Re-run it monthly like any other migration.

What the cap does NOT protect: the origin

A subtle trap: per-exit caps make your traffic safe for the exits, and can still be rude to the site. Ten exits at 3 concurrent requests each is thirty in-flight requests at the origin. Origin rate limits, Retry-After headers, and politeness budgets all key off the target, not the exit. So the lease pool is only half of a correct design; the other half is a per-origin token bucket that every request passes through before it is allowed to acquire an exit lease:

class OriginBudget:
    """Global per-origin rate limit, independent of how many exits you own."""
    def __init__(self, rps: float):
        self._tokens, self._cap = rps, rps
        self._last = asyncio.get_running_loop().time()
        self._lock = asyncio.Lock()

    async def acquire(self):
        async with self._lock:
            while True:
                now = asyncio.get_running_loop().time()
                self._tokens = min(self._cap, self._tokens + (now - self._last) * self._cap)
                self._last = now
                if self._tokens >= 1:
                    self._tokens -= 1
                    return
                await asyncio.sleep((1 - self._tokens) / self._cap)

Order matters: origin budget first, exit lease second. Acquiring an exit and then waiting on the budget pins scarce exits doing nothing, which is exactly the resource-starvation pattern you built the lease pool to avoid.

What beginners should take away

asyncio.Semaphore is a fine tool for the problem it solves: bounding CPU, file descriptors, or a single upstream service you own. A rotating proxy pool is none of those things. It is a fleet of independent network exits, each with its own capacity, and a fleet needs per-unit caps. Replace the one big semaphore with a lease pool keyed by exit, size per_exit from a latency knee instead of vibes, and half of the "my async scraper randomly dies at high concurrency" mystery disappears.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele