One TLS handshake predicts the whole anti-bot stack

When I size up a new website I used to start with headers. Copy them out of DevTools, match the order, add the cookies, see what happens. Now I start with one TLS handshake, because it tells me more about the site than an afternoon of header work.

The reason is a set of numbers from State of Web Access, the audit Zyte (my employer) ran on 11,100 popular landing pages. 13.8% of them screen the TLS handshake. Look at what else those sites run compared to everyone else:

Barrier	TLS-fingerprinting sites	All other sites
Rate limiting	78.1%	13.2%
Antibot	45.0%	14.2%
CAPTCHA	37.3%	20.2%
JavaScript required	27.0%	42.8%

A TLS check is the strongest single predictor in the whole dataset. If the handshake is being inspected, you are almost certainly going to hit rate limiting too, and you have a coin-flip chance of a dedicated bot manager on top. If it is not being inspected, the site probably stops at a CDN firewall. One request and you know which world you are in.

The JavaScript row runs the other way, and I find it the most interesting. TLS-checking sites need a browser less often. They are server-rendered, performance-minded operations that enforce access at the connection, not in the page.

Why nobody notices this layer

TLS fingerprinting fails before HTTP starts. Your client sends a ClientHello announcing which TLS versions, cipher suites and extensions it supports, in a particular order. Python on OpenSSL, Go's crypto/tls and Chrome's BoringSSL all produce different ones. A site can see "Chrome user agent, OpenSSL handshake" and know the client is lying before it reads a single header.

JA3 hashed those fields into a fingerprint in 2017. Chrome started shuffling its extension order in 2023, which broke JA3, and JA4 replaced it by sorting the fields first. The detail does not matter much for scraping. What matters is that the check is silent. Some sites return a bare 403 with an empty body. Some just close the connection, and your logs say "connection reset by peer", which looks like a network blip. Nothing in the failure says TLS.

Vendor attribution was only possible for 45% of the TLS-checking sites in the audit. The other 55% show the behaviour with no named signature. Where it could be attributed, a single bot management vendor was 88.8% of it. So this is mostly one vendor's check, bundled into a product a lot of sites bought for other reasons.

The probe

This is the method the report used, cut down. Two requests, identical headers, from the same IP. The only thing that differs is the handshake. aiohttp presents a standard Python TLS signature. curl_cffi presents Chrome's. If the first fails and the second passes, the site is looking at TLS.

import asyncio
import sys

import aiohttp
from curl_cffi.requests import AsyncSession

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}


async def python_tls(url: str) -> tuple[int, int]:
    try:
        async with aiohttp.ClientSession(headers=HEADERS) as s:
            async with s.get(url, timeout=aiohttp.ClientTimeout(total=20)) as r:
                return r.status, len(await r.read())
    except aiohttp.ClientError:
        return 0, 0          # reset or refused: count it as a block


async def chrome_tls(url: str) -> tuple[int, int]:
    async with AsyncSession(impersonate="chrome", headers=HEADERS) as s:
        r = await s.get(url, timeout=20)
        return r.status_code, len(r.content)


async def probe(url: str) -> dict:
    py, ch = await asyncio.gather(python_tls(url), chrome_tls(url))
    filtered = py[0] != ch[0] and ch[0] < 400
    return {"python_tls": py, "chrome_tls": ch, "tls_filtered": filtered}


if __name__ == "__main__":
    print(asyncio.run(probe(sys.argv[1])))

Install with pip install aiohttp curl_cffi. The impersonate part follows whatever Chrome version curl_cffi ships, so it drifts less than a hand-maintained profile would.

Because both requests leave the same machine, IP reputation is held constant. This is a controlled experiment with one variable, which is more than most scraping diagnostics manage.

How I read the result

If tls_filtered is true, I plan for the stack the table above predicts. Browser-shaped TLS from request one, session handling, per-domain concurrency of one or two, content validation on every response. I do not spend a day on header tweaks because headers were never the problem.

If it is false and I am still blocked, I have ruled out one layer for the cost of one request. The block is IP class, headers, cookies or behaviour, and I look there instead.

If it is false and both requests pass, I have a Simple-tier site and I should stop reaching for Playwright. The audit's tier data puts 85% of landing pages at plain HTTP. Rendering when you do not need to is the most common way I see scraping budgets disappear.

One caveat the report makes and I will repeat. Everything here is landing pages, scanned once, from datacentre IPs. Search and product pages are usually defended harder than the front door. Treat the percentages as a floor.

The industry split is on the TLS page. Jewellery and luxury leads at 34%. Reference sites sit at 3%, because blocking crawlers would kill the thing they exist for.

Disclosure: I work at Zyte. The dataset is ours. The probe is a cut-down version of what the audit ran, and the opinions are mine.