Your scraper isn't blocked because of your headers — it's blocked because of your ASN

You've been here. The scraper works fine on your laptop. You deploy it to a VPS and every response comes back 403. So you start the ritual: copy the exact User-Agent from Chrome DevTools, add Accept-Language, fix the header ordering, add Sec-Fetch-*, swap requests for curl_cffi to match the TLS fingerprint.

Still 403.

The reason is usually simpler and lower down the stack than the thing you're debugging. The target decided about your request before it ever looked at your headers, based on which autonomous system your packets came from.

A 60-second refresher on ASNs

The internet is a network of networks. Each of those networks — an ISP, a hosting company, a university, a mobile carrier — is an autonomous system, identified by an ASN, an autonomous system number. Every routable IP address belongs to a prefix, and every prefix is announced to the global routing table by some AS.

This mapping is public. It has to be, because BGP is how packets find their way anywhere. Which means anyone can take your IP and, in one lookup, learn that you are Hetzner (AS24940), DigitalOcean (AS14061), AWS (AS16509), or Comcast (AS7922).

That last distinction is the whole article.

Check what you're actually egressing from

Before theorising, look at your own traffic. Team Cymru runs a free whois service that maps IP to ASN:

whois -h whois.cymru.com " -v $(curl -s ifconfig.me)"

Output from a cheap VPS looks something like:

AS      | IP           | AS Name
24940   | 116.x.x.x    | HETZNER-AS, DE

Or hit RIPEstat, which needs no auth and returns JSON:

curl -s "https://stat.ripe.net/data/network-info/data.json?resource=1.1.1.1" | jq '.data.asns'

Now run the same check on your home connection. You'll get something like AS7922 COMCAST-CABLE-1 or AS3320 DTAG. Different bucket entirely.

What the anti-bot vendor sees

Commercial IP intelligence databases — the ones sitting behind Cloudflare, DataDome, PerimeterX, Akamai and friends — don't just store geolocation. Each prefix gets tagged with a usage type. The categories vary by vendor but they always include some version of:

• hosting / datacenter — cloud providers, VPS shops, colo
• isp / residential — consumer broadband
• mobile — cellular carriers
• education, government, business

A human buying sneakers does not browse from AS16509. A human reading a job listing does not browse from AS24940. So "usage type is hosting" becomes an extremely cheap, extremely high-precision signal. It costs one lookup against a preloaded dataset, it happens before any JS challenge, and it has almost no false positives on consumer-facing sites.

That's why it runs first. Your beautifully crafted headers are being evaluated by a code path you never reached.

Why fixing headers feels like it should work

Header and fingerprint work does matter — just not at this layer. Think of detection as a stack:

Layer	Signal	Cost to evaluate	When it runs
1	IP reputation / ASN usage type	~0	Before anything
2	TLS fingerprint (JA3/JA4)	very low	On handshake
3	HTTP headers, ordering, HTTP/2 frames	low	On request
4	JS challenge, canvas, behavioural	high	After the above pass

Every guide you've read optimises layers 2 and 3, because those are the fun ones with clever code. But if layer 1 fails, nothing downstream gets a chance. The frustrating part is that the failure mode looks identical: a 403, a CAPTCHA, an empty page. There's no error message that says "your ASN is registered to a hosting provider."

A quick way to confirm you're stuck at layer 1: run the identical request from your home connection over a VPN-free network. If it sails through with the exact same headers your VPS is sending, headers were never the problem.

The three-bucket model

Once you internalise the usage-type tagging, proxy shopping stops being mysterious:

Datacenter proxies. Fast, cheap ($0.02/IP territory), and announced by hosting ASNs. You are trading one hosting ASN for another. Fine for targets that don't check — internal APIs, open data portals, sites with no bot vendor in front of them. Useless on anything that filters by usage type.

Rotating residential proxies. IPs from real consumer connections, so the usage type is isp. Trust is high. The trade-off is that the exit changes constantly, latency is variable and higher, and you're billed by bandwidth.

ISP proxies. The interesting middle. The IP address is allocated from a range registered to a consumer internet service provider, so a lookup returns an ISP ASN — but the machine actually answering is a server in a datacenter with real bandwidth behind it. You get the usage-type classification of residential with the latency and stability of hosting.

ISP proxies are a concrete example of this shape: static IPs on premium ASNs (AT&T, Verizon, Orange, Tele2 among them), sub-0.2s response times, HTTP(S) and SOCKS5, across 17 countries. Because the IP doesn't rotate, the ASN lookup and the IP itself both stay constant across a session — which matters a lot for the next section.

Static matters as much as the ASN

There's a second reason ISP proxies work where rotation doesn't, and it's about session continuity rather than trust.

If your workflow involves logging in, filling a cart, paging through an account area, or running a multi-step agent loop, a changing egress IP is itself a red flag. Consumer accounts don't teleport between subnets every 800ms. Rotating through residential exits mid-session will trigger re-authentication, device verification emails, and step-up challenges that you then have to solve — a problem you created.

Rough rule:

• Rotate when you need volume and every request is independent and anonymous.
• Stay static when there's a session, a login, or state carried across requests.

Wiring it up

Nothing exotic. The proxy is an HTTP proxy, so every client already speaks it:

import requests

USER = "your-username"
PASS = "your-password"

proxy = f"http://{USER}:{PASS}@isp.decodo.com:10001"
proxies = {"http": proxy, "https": proxy}

# confirm what the world sees
print(requests.get("https://ip.decodo.com/json", proxies=proxies).json())

Then verify the ASN actually changed, which is the only check that matters here:

import requests

def asn_for(ip: str) -> list[str]:
    r = requests.get(
        "https://stat.ripe.net/data/network-info/data.json",
        params={"resource": ip},
        timeout=10,
    )
    return r.json()["data"]["asns"]

direct = requests.get("https://api.ipify.org").text
via_proxy = requests.get("https://api.ipify.org", proxies=proxies).text

print("direct :", direct, asn_for(direct))
print("proxied:", via_proxy, asn_for(via_proxy))

If the proxied ASN still resolves to a hosting provider, you bought datacenter IPs with a nicer label. Check before you build on top of it.

For Playwright, pass it at browser launch so both the navigation and the XHRs inherit it:

browser = p.chromium.launch(proxy={
    "server": "http://isp.decodo.com:10001",
    "username": USER,
    "password": PASS,
})

And keep a Session around so cookies and connection reuse survive across requests:

s = requests.Session()
s.proxies.update(proxies)
s.headers.update({"Accept-Language": "en-US,en;q=0.9"})

What ASN doesn't fix

Being honest about the limits, because "buy ISP proxies" is not a complete anti-blocking strategy:

• Rate. 200 requests per second from a single residential IP is not residential behaviour. The ASN gets you past the front door; volume gets you thrown out anyway.
• TLS fingerprint. Python's requests has a JA3 signature that no browser produces. If the target checks, use curl_cffi or drive a real browser. Layers 2 and 3 still exist — they're just not first.
• Shared IP reputation. On a shared pool, someone else's behaviour on your IP is your problem. If a target matters, use a dedicated IP.
• Behavioural signals. Perfectly uniform timing, no mouse movement, identical navigation paths. None of that is an IP problem.
• The rules. Check robots.txt and terms of service, respect rate limits, and don't scrape personal data you have no basis to collect. Being technically unblocked and being in the clear are different things.

The debugging order that actually saves time

Next time something 403s, work bottom-up instead of top-down:

1. Resolve your egress IP's ASN. Hosting provider? Stop — that's it. Fix that first.
2. Make the same request from a residential connection with identical headers. Works? Confirmed layer 1.
3. Only now start on TLS fingerprint and headers.
4. Then rate limiting and timing.
5. Then behavioural and JS challenges.

Most people run this list in reverse, spend a day on Sec-Fetch-Site, and never look at the routing table entry that decided the whole thing in the first microsecond.