Your Residential IP Is Clean. Your Request Rate Gives You Away.

The last post I wrote here was about geo signals — how a German exit IP means nothing if your Accept-Language, timezone and locale all still say "US developer laptop." A reader pushed back with a fair point: they'd fixed all of that, every signal agreed, and they were still getting throttled through clean residential IPs. So what was left?

The exit IP was clean. Every static signal agreed. What gave them away was how the IP behaved — the rate, the concurrency, and the timing of the requests coming out of it. A residential IP doesn't just claim to be a household. It has to act like one.

An IP is a story about a person

When a site sees a residential IP, the implicit claim is "there is a human on a home connection here." Humans, and the households behind them, have a shape:

• They open a few tabs, not forty.
• They read a page for some seconds before clicking the next one.
• They go quiet for hours and then come back.
• Their requests arrive one after another with human gaps, not in perfectly-spaced 200ms intervals and not in a thundering burst.

None of this is exotic bot detection. It's arithmetic the server was already doing for capacity planning. A single residential IP that sustains 40 concurrent connections and 30 requests a second is not a suspicious fingerprint — it's a physically implausible household. The IP is clean. The story it's telling is not.

The three numbers that matter per IP

Forget global rate limits for a second. The unit that matters is per exit IP, because that's the unit the target attributes behavior to.

1. Concurrency. How many requests are in flight through one IP at once. This is the one people get most wrong, because async frameworks make it invisible. asyncio.gather over 500 URLs with one sticky session doesn't feel like 500 concurrent requests when you write it — but that's exactly what leaves the IP. Cap it. A real browser rarely has more than ~6 connections open to one host.

2. Rate. Requests per second per IP, sustained. The safe number depends entirely on the target, but the useful mental model is "how fast would one interested person click?" — not "how fast can my machine send." For most sites, sustained single-digit requests per second per IP is already faster than any human.

3. Spacing. The gap between requests. Two scrapers can have the same average rate and look completely different: one sends a request exactly every 500ms, the other sends them with jittered human-ish gaps. Metronomic timing is itself a signal — real traffic is bursty and irregular. A little randomization in the delay does more than shaving the average rate.

The trap is optimizing the first two and ignoring the third. A perfectly rate-limited scraper that fires on a clockwork interval still reads as automation, because nothing that has a human in the loop is that regular.

Why this fails softly (again)

If you've read the earlier posts you know where this goes. Cross the rate a target actually enforces and you get a 429 — loud, obvious, easy to back off from. But most sites don't hard-limit at the first sign; they degrade you. Slower responses. More challenges. Shorter sticky-session lifetimes. You get quietly moved into a bucket for traffic that's probably automated, and you stay there. Your success rate is 10% lower than it should be and you blame the proxies.

And rotating the IP doesn't buy back what you spent. If you burn an IP by hammering it, the next IP gets the same treatment the moment it behaves the same way. You're not running out of IPs; you're running the same implausible household story from a new address.

The fix is a budget, not a sleep

The instinct is to sprinkle time.sleep() around and call it rate limiting. That controls the average and nothing else. What you actually want is a small budget enforced per IP: a cap on in-flight requests, a token-bucket rate with jitter, and a rule that when one IP starts getting slow or challenged, you pace it down before you rotate it away.

Here's the shape of it — a per-IP limiter you acquire before every request through that exit:

import asyncio, random, time

class IPBudget:
    """Per-exit-IP pacing: concurrency cap + jittered token-bucket rate."""
    def __init__(self, max_concurrency=4, rps=2.0, jitter=0.4):
        self._sem = asyncio.Semaphore(max_concurrency)
        self._min_gap = 1.0 / rps
        self._jitter = jitter
        self._next_at = 0.0
        self._lock = asyncio.Lock()

    async def __aenter__(self):
        await self._sem.acquire()
        async with self._lock:
            now = time.monotonic()
            wait = max(0.0, self._next_at - now)
            gap = self._min_gap * (1 + random.uniform(0, self._jitter))
            self._next_at = max(now, self._next_at) + gap
        if wait:
            await asyncio.sleep(wait)
        return self

    async def __aexit__(self, *exc):
        self._sem.release()

One IPBudget per exit IP, not one global one. If you hold a sticky residential session, that session gets its own budget for its whole life; when you rotate, the new IP starts fresh. The concurrency cap keeps gather honest, the token bucket keeps the sustained rate human, and the jitter keeps the spacing from looking like a machine.

A runnable version — with the per-host bookkeeping and a "slow down before you rotate" hook — is in our examples repo: github.com/roamproxy/proxy-examples.

Putting it with the rest

Across these posts the theme keeps coming back: a clean IP is table stakes, and it's the cheapest part of not getting blocked. The IP has to look like a household, the client's geo signals have to agree with it, and the behavior coming out of it has to be physically plausible for one person on one connection. Get all three right and the exit IP stops being the thing anyone notices — which is the whole point.

---

We run Roam, a pay-as-you-go residential/datacenter/mobile proxy network — real home-broadband exits, per-GB pricing, balance never expires. The examples above run against any proxy; the repo is provider-agnostic.