Stop Picking One Proxy Tier: Route by Endpoint Defense Level Instead

Every scraping architecture review I do starts the same way. Someone asks "should we use datacenter, residential, or mobile proxies?" and the room splits into three camps, each defending their favorite. The datacenter camp quotes latency and price. The residential camp quotes success rates. The mobile camp quotes that one Wall Street Journal story about Airbnbs.

They are all answering the wrong question. There is no best proxy tier. There is only a match between an exit IP's reputation and the defense level of the specific endpoint you are hitting. A single scraping pipeline typically touches dozens of endpoints with wildly different defenses, and routing all of them through one tier means you are either burning money on endpoints that don't care, or burning IPs on endpoints that do.

The core idea of this post: proxy tier selection is a per-endpoint routing decision, not an infrastructure decision. Build a router that scores each endpoint's defense level and picks the cheapest tier that clears it. Everything else follows.

Why one tier never fits a real pipeline

Consider a realistic price-monitoring pipeline for an e-commerce client. In a single crawl cycle it hits:

1. A product sitemap endpoint — static XML, no bot defense at all, served from a CDN.
2. Product detail pages — behind Akamai, aggressive TLS fingerprinting and IP reputation scoring.
3. A JSON search API — rate limited per IP but no reputation wall.
4. A login-gated seller portal — behavioral analysis, device fingerprinting, the works.
5. An image CDN for product photos — literally does not care who you are.

Now price it. Residential traffic runs roughly 10-30x the cost per GB of datacenter traffic, and mobile is another 3-10x on top of residential. If you route the sitemap and image traffic through residential "to be safe," you are paying a 20x premium for bytes that a $0.50/GB datacenter IP would have delivered identically. If you route the Akamai-protected pages through datacenter to save money, you will burn through a blocklist in an afternoon and your success rate collapses.

The teams that win on both cost and reliability do the same thing: they classify endpoints and route each class through the cheapest tier that works. This isn't exotic engineering. It's the same logic your CDN uses for cache tiers.

Step 1: Define the tiers honestly

Before routing, be precise about what each tier actually buys you, because the marketing copy won't:

• Datacenter: IPs registered to hosting ASNs (OVH, Hetzner, AWS, DigitalOcean). Lowest latency, lowest cost, near-zero reputation with any bot mitigation vendor. Datacenter ranges are published and trivially blockable.
• Residential: IPs registered to consumer ISPs, exiting from real home routers. High reputation, higher latency variance, costs per GB. Sub-types worth distinguishing: rotating (new IP per request), sticky sessions (same IP for a window), and static residential (a fixed IP you keep).
• Mobile: IPs registered to carrier-grade NAT ranges (T-Mobile, Verizon). Highest reputation because carrier NAT means thousands of real users share each IP, which forces defenders to be lenient. Highest cost.
• ISP (a.k.a. static residential hosted in datacenters): Datacenter hardware, but IP ranges registered to consumer ISPs. Speed close to datacenter, reputation close to residential. Often sold per IP per month rather than per GB.

Step 2: Score your endpoints

You don't need a fancy classifier. Four defense levels cover 95% of real-world endpoints:

• L0 — No defense: sitemaps, public APIs without rate limits, static assets. Any working IP clears it. Route: datacenter (or no proxy at all).
• L1 — Rate limiting only: per-IP quotas, 429s, but no reputation wall. Route: datacenter or ISP with concurrency spread across IPs.
• L2 — IP reputation + fingerprinting: the Akamai/Cloudflare/PerimeterX tier. Datacenter fails, residential and mobile pass. Route: residential.
• L3 — Behavioral + account-bound analysis: login portals, checkout flows, anything where the defender correlates identity across signals. Route: residential sticky sessions or mobile, with cookie and fingerprint hygiene.

Here is a compact router implementing exactly this. It's dependency-light and thread-safe:

import time
import random
import threading
from dataclasses import dataclass, field
from enum import Enum
from collections import defaultdict

class Tier(Enum):
    DATACENTER = "dataproxy.thordata.com:9000"   # datacenter port
    RESIDENTIAL = "resi.thordata.com:9001"       # rotating residential
    RESIDENTIAL_STICKY = "resi.thordata.com:9001" # sticky via session suffix
    ISP = "isp.thordata.com:9002"
    MOBILE = "mobile.thordata.com:9003"

class Defense(Enum):
    L0 = 0  # no defense
    L1 = 1  # rate limiting only
    L2 = 2  # IP reputation + fingerprinting
    L3 = 3  # behavioral / account-bound

# The routing table: cheapest tier that historically clears each level.
DEFAULT_ROUTING = {
    Defense.L0: [Tier.DATACENTER],
    Defense.L1: [Tier.DATACENTER, Tier.ISP],
    Defense.L2: [Tier.RESIDENTIAL, Tier.MOBILE],
    Defense.L3: [Tier.RESIDENTIAL_STICKY, Tier.MOBILE],
}

@dataclass
class EndpointProfile:
    host: str
    defense: Defense
    sticky_sessions: bool = False       # L3 usually wants identity continuity
    recent_failures: int = 0
    recent_successes: int = 0

class TierRouter:
    """Routes each target host through the cheapest tier that clears its defense level."""

    def __init__(self, username: str, password: str, routing=DEFAULT_ROUTING):
        self.creds = f"{username}:{password}"
        self.routing = routing
        self.profiles: dict[str, EndpointProfile] = {}
        self._lock = threading.Lock()
        # session id per host for sticky tiers
        self._session_ids: dict[str, str] = defaultdict(
            lambda: f"sess-{random.randint(10**9, 10**10)}"
        )

    def register(self, host: str, defense: Defense, sticky: bool = False):
        self.profiles[host] = EndpointProfile(host, defense, sticky)

    def proxy_url(self, host: str) -> str:
        profile = self.profiles[host]
        candidates = list(self.routing[profile.defense])

        # Escalation: if this host burned through the cheap tier recently,
        # move up one candidate. This is the whole ballgame.
        with self._lock:
            if profile.recent_failures >= 3 and len(candidates) > 1:
                candidates = candidates[1:]
                profile.recent_failures = 0  # give the expensive tier a clean slate

        tier = candidates[0]
        auth = self.creds
        if tier is Tier.RESIDENTIAL_STICKY:
            auth = f"{self.creds}-session-{self._session_ids[host]}"
        return f"http://{auth}@{tier.value}"

    def report(self, host: str, ok: bool):
        with self._lock:
            p = self.profiles[host]
            if ok:
                p.recent_successes += 1
                p.recent_failures = 0
            else:
                p.recent_failures += 1

Usage with requests looks like this:

import requests

router = TierRouter("your_username", "your_password")
router.register("shop.example.com", Defense.L0)                    # sitemap host
router.register("catalog.example.com", Defense.L2)                 # Akamai-protected PDPs
router.register("seller-portal.example.com", Defense.L3, sticky=True)

def fetch(url, host):
    proxies = {"http": router.proxy_url(host), "https": router.proxy_url(host)}
    try:
        r = requests.get(url, proxies=proxies, timeout=20)
        r.raise_for_status()
        router.report(host, ok=r.status_code == 200)
        return r
    except Exception:
        router.report(host, ok=False)
        raise

The report/escalation loop is the part most teams skip. A static routing table rots: endpoints add defenses, defenders reclassify your residential ranges, tiers change. The router treats tier selection as a control loop — cheap tier by default, escalate on sustained failure. That single mechanism is worth more than any amount of upfront tier agonizing.

Step 3: Verify the ASN claim, don't trust the label

One engineering detail that saves real pain: verify what you're actually buying. A "residential" proxy that exits through a hosting ASN is a datacenter proxy with a markup. Check the ASN of your exit IPs against a public database before you commit traffic. Here's a probe that runs through each tier and reports the network:

import requests

def audit_exit(tier_proxy: str) -> str:
    r = requests.get(
        "https://ipinfo.io/json",
        proxies={"http": tier_proxy, "https": tier_proxy},
        timeout=20,
    )
    info = r.json()
    return f"ip={info['ip']} org={info.get('org', '?')} country={info.get('country', '?')}"

for name, proxy in {
    "datacenter": "http://user:[e-mail]:9000",
    "residential": "http://user:[e-mail]:9001",
    "isp": "http://user:[e-mail]:9002",
}.items():
    print(name, "->", audit_exit(proxy))

If the org field for your "residential" tier shows a hosting provider, you've found a billing problem disguised as an engineering one. Run this probe on a schedule — mixups happen, and per-GB pricing makes them expensive.

What this buys you

I've watched this pattern play out on real pipelines. A client was running everything through rotating residential at roughly 1.8 TB/month. We profiled their endpoints and found 60% of their traffic (images, sitemaps, an unfenced public API) sailed through on datacenter IPs. Same success rate, total bandwidth bill dropped by about 70%. Conversely, a team running everything through datacenter kept "solving" blocks with header rotation and retry storms; moving just their L2 endpoints to residential cut their compute spend in half because they stopped re-scraping blocked pages.

The deeper point is an organizational one. "Which proxy tier should we buy?" invites a vendor-flavored religious debate. "Which tier clears each endpoint's defense level, and what does each route cost us per 1,000 successful pages?" is an engineering question with a measurable answer. Put the routing table in code, the escalation loop in code, and the ASN audit in CI, and the debate ends.

---

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele