Does Your TLS Impersonation Actually Work? Building a Fingerprint Conformance Test for Your Scraper

Last March, my price-monitoring scraper — 14 months of quiet, dependable operation — started getting 403s on every request to one specific retail site. No code changes. No new rate limiting announced. The logs showed the same proxied IPs, the same headers, the same crawl pacing. What had changed was a routine pip install -U three days earlier that bumped curl_cffi from 0.6 to 0.7. Nothing in the changelog mentioned handshakes. But buried in that upgrade was a rebuild of the underlying curl-impersonate binary, and the impersonate="chrome110" profile I'd been using now produced a subtly different TLS fingerprint — one the target's bot detection had apparently never seen from a real Chrome on that OS version. To the anti-bot system, every request my scraper made was now an unknown client. Block, block, block.

That outage cost me a day of debugging and taught me a lesson I should have learned much earlier: TLS impersonation is not a set-and-forget flag. It's a contract with your dependencies, and dependencies drift. The fix isn't more careful upgrading — it's treating fingerprint parity as a continuously tested contract, exactly like a unit test in CI.

What JA3 and JA4 actually summarize

When your client opens an HTTPS connection, the TLS ClientHello message carries a list of cipher suites, TLS extensions, and elliptic curves — and crucially, the order they appear in. Real browsers have quirky, version-specific orderings. Most HTTP libraries have different, equally identifiable ones. A fingerprint is just a hash summarizing all of that.

JA3 concatenates the TLS version, cipher suites, extensions, and curves into a string and MD5s it. It's been the industry standard for years, but it has a real flaw: the hash is all-or-nothing. If a single extension changes position — say, Chrome reorders key_share and psk_key_exchange_modes — the JA3 changes completely, and you can't tell from the two hashes what changed.

JA4 (from FoxIO/John Althouse) fixes this by splitting the fingerprint into readable, separable parts: t13d1516h2_8daaf6152771_b0da82dd1658. The first chunk encodes protocol and extension counts, the second is a SHA-256 truncation of the cipher list, the third covers extensions and signature algorithms. Because it's segmented and sortable, you can diff two JA4 strings and immediately see "the cipher set changed but the extension set didn't." For conformance testing, that difference between "it changed" and "here's what changed" is everything. I pin my tests to JA4 and use JA3 only as a secondary sanity check.

One more fingerprint worth tracking: the Akamai HTTP/2 fingerprint (akamai_fingerprint in most test endpoints), which encodes SETTINGS values, the priority tree, and pseudo-header order. Modern anti-bot systems fingerprint the H2 layer too — a perfect JA4 with a broken HTTP/2 fingerprint is still a detectable client.

Seeing your own fingerprint

The fastest way to make this concrete is tls.peet.ws, which echoes back your complete client fingerprint as JSON. Compare plain requests against curl_cffi's impersonation:

import requests
from curl_cffi import requests as cffi_requests

# Baseline: Python's default TLS stack — instantly recognizable as a script
r = requests.get("https://tls.peet.ws/api/all", timeout=15)
print(r.json()["ja4"])        # t13d1517h2_... (the "python-requests" look)

# Now with impersonation
r = cffi_requests.get("https://tls.peet.ws/api/all",
                      impersonate="chrome124", timeout=15)
d = r.json()
print(d["ja3"])               # cd1a8f59d1c...
print(d["ja4"])               # t13d1516h2_8daaf6152771_b0da82dd1658
print(d["http_version"])      # h2
print(d["akamai_fingerprint"]) # 1:65536;2:0;4:131072;6:262144|15663105|0|m,a,s,p

The fields that matter for your harness: ja4, akamai_fingerprint, and http_version. The plain-requests JA4 differs from Chrome's in the extension-count segment alone, and its Akamai fingerprint is a giveaway (0:m,a,s,p ordering instead of Chrome's m,a,s,p with WINDOW_UPDATE frames — and older urllib3 stacks often negotiate HTTP/1.1 where Chrome would use h2). Anti-bot vendors run these exact comparisons server-side on every handshake.

The conformance harness

The core idea: store one "known-good" fingerprint profile per browser target in a JSON file, fetch the echo endpoint the same way production does, and fail loudly on any mismatch. Here's the harness I run, trimmed to essentials but fully runnable:

"""tls_conformance.py — fail CI if our fingerprint drifts from known-good."""
import json
import sys
from pathlib import Path

from curl_cffi import requests as cffi_requests

PROFILES_FILE = Path(__file__).parent / "tls_profiles.json"
API_URL = "https://tls.peet.ws/api/all"
PROXY = None  # or "http://user:pass@egress-host:port" in CI runs


def fetch_profile(impersonate: str) -> dict:
    resp = cffi_requests.get(
        API_URL,
        impersonate=impersonate,
        proxies={"https": PROXY, "http": PROXY} if PROXY else None,
        timeout=20,
    )
    resp.raise_for_status()
    d = resp.json()
    return {
        "ja4": d["tls"]["ja4"],
        "akamai_fingerprint": d["http2"]["akamai_fingerprint"],
        "http_version": d["http_version"],
    }


def main() -> int:
    expected = json.loads(PROFILES_FILE.read_text())
    failures = []

    for impersonate, want in expected.items():
        try:
            got = fetch_profile(impersonate)
        except Exception as exc:
            failures.append(f"{impersonate}: request failed: {exc}")
            continue

        for field in ("ja4", "akamai_fingerprint", "http_version"):
            if got[field] != want[field]:
                failures.append(
                    f"{impersonate}.{field}: "
                    f"expected {want[field]}, got {got[field]}"
                )

    if failures:
        print("TLS FINGERPRINT DRIFT DETECTED:", file=sys.stderr)
        for f in failures:
            print(f"  - {f}", file=sys.stderr)
        return 1

    print(f"OK: {len(expected)} profile(s) match known-good fingerprints.")
    return 0


if __name__ == "__main__":
    sys.exit(main())

And the tls_profiles.json it validates against — captured once, by hand, on a day I verified impersonation was working end-to-end:

{
  "chrome124": {
    "ja4": "t13d1516h2_8daaf6152771_b0da82dd1658",
    "akamai_fingerprint": "1:65536;2:0;4:131072;6:262144|15663105|0|m,a,s,p",
    "http_version": "h2"
  },
  "safari17_0": {
    "ja4": "t13d1715h2_5b57614c22b0_3d5424432f57",
    "akamai_fingerprint": "1:65536;2:0;3:1000;4:1048576;6:262144|15663105|0|m,a,s,p",
    "http_version": "h2"
  }
}

Two design choices worth calling out. First, I compare field-by-field rather than diffing whole blobs — when it fails, the error message tells me exactly which layer drifted (chrome124.ja4: expected ...b0da82dd1658, got ...a7b3e2d1c9f4 points at cipher/extension changes; an akamai_fingerprint mismatch with clean JA4 points at the HTTP/2 layer). Second, the harness goes through the same proxy egress production uses, because proxies are a fingerprint wildcard (more below).

One caveat: JA4 strings above are illustrative — don't copy them from this post. Capture your own on day one, from the exact library versions you pin, and treat those as the contract.

Wiring it into CI

Fingerprints drift for reasons other than your own upgrades — curl_cffi tracks upstream curl-impersonate releases, profile names get remapped, and Chrome itself changes handshakes every few versions. A test you only run when you remember is not a defense. I run mine on a schedule:

# .github/workflows/tls-conformance.yml
name: TLS fingerprint conformance
on:
  schedule:
    - cron: "0 6 * * *"   # daily, before my morning coffee
  pull_request:
    paths: ["requirements.txt"]

jobs:
  fingerprint-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python tls_conformance.py
        env:
          PROXY_URL: ${{ secrets.PROXY_URL }}

The pull_request trigger on requirements.txt is the important one: it catches the "routine upgrade" case at review time, before it ships. The daily cron catches the cases nobody initiated — upstream binary swaps, profile renames, or a proxy provider silently changing their egress behavior. When it fires, I either pin the old version or re-capture the profiles deliberately, with a commit message that says why.

Engineering details that matter

Pin your library versions exactly. curl_cffi==0.7.1, not curl_cffi>=0.6. The whole point of the conformance test is that upgrades are events, and you can't have an event against a floating range. Same for requests, urllib3, and anything else that touches the wire.

Treat profile names as browser-version-specific, not generic. impersonate="chrome124" means "Chrome 124's handshake as curl-impersonate implemented it," not "Chrome." When Chrome 131 ships and your target site's detection model updates, a stale-but-consistent 124 fingerprint starts looking like an old, unpatched browser — which real users do have, but in dwindling numbers. Plan profile upgrades as routine maintenance, gated by your conformance test.

Test through the same proxy egress you use in production. This bit me after the March outage: my conformance test ran direct from CI while production traffic went through residential proxies, so it validated a code path production never used. Some proxies terminate and re-originate TLS (fine if the egress handshake is still Chrome-like; fatal if it's the proxy vendor's own HAProxy fingerprint), and some mangle HTTP/2 into HTTP/1.1 downgrades. The PROXY variable in the harness exists because the only fingerprint that matters is the one the target server actually sees — which is the one leaving the proxy.

Keep one known-good profile per target browser, captured deliberately. Don't accumulate five snapshots from different dates and guess which is right. One canonical capture, verified end-to-end against a real target that was previously blocking you, stored in version control with a README note on how it was produced. When it changes, you want the diff to mean something.

Wrapping up

The 403 storm taught me that "it works" is a property that silently expires. TLS impersonation sits at the bottom of your stack, depends on a binary you didn't compile, and is evaluated adversarially by systems that update without telling you. A 70-line conformance test, a JSON file of known-good fingerprints, and a daily cron job turn that hidden dependency into an explicit, monitored contract. My scraper has survived two curl_cffi upgrades since March — both times, the test caught the drift in CI before a single target site could. Build the test before you need it; the 403s won't send a calendar invite.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele