I had to built a scraper which was supposed to extract data from an old asp.net based site but kept getting blocked by cloudflare. Today's bot have evolved far more than just checking headers and basic connection types. Detection systems nowadays analyze the cryptographic characteristics of connections themselves, creating a new challenge for developers building HTTP clients that just want to fetch simple pages.
I'll try to explain how TLS and HTTP/2 fingerprinting work, why standard library implementations are easily detected, and the approach I took to build httpcloak, Golang HTTP client designed to produce browser-identical network fingerprints.
The Mechanics of TLS Fingerprinting
Every TLS handshake begins with a ClientHello message. This message contains several fields that, while functionally necessary for establishing a secure connection, also serve as an identification mechanism.
The ClientHello includes an ordered list of cipher suites the client supports, TLS extensions it wishes to use, elliptic curves for key exchange, and signature algorithms it accepts. The specific values, their order, and their presence or absence create a unique signature.
This signature gets hashed into standardized formats. JA3, developed by Salesforce, was the original standard but JA4, its successor, provides more granularity and has become the preferred method for fingerprint analysis. Bot detection services maintain extensive databases mapping these hashes to known clients—every Chrome version, every Firefox release, every Python requests library, and yes, every Go net/http client.
When Go's standard TLS implementation connects to a server, it presents a ClientHello that differs from browsers in several ways. It offers fewer cipher suites (13 versus Chrome's 16), includes fewer extensions (12 versus 18), lacks GREASE values (randomized dummy values browsers include for extensibility), and doesn't support post-quantum key exchange algorithms that Chrome has implemented since version 131.
These differences are immediately apparent to any server running fingerprint analysis.
HTTP/2 Fingerprinting: The Second Layer
Passing TLS fingerprint checks is necessary but not sufficient. HTTP/2 connections have their own fingerprint based on the SETTINGS frame sent immediately after connection establishment. This frame contains parameters like HEADER_TABLE_SIZE, INITIAL_WINDOW_SIZE, MAX_CONCURRENT_STREAMS, and others. Go's defaults differ substantially from browser values. The most obvious example is INITIAL_WINDOW_SIZE: Go uses 65,535 bytes while Chrome uses 6,291,456 bytes—a difference of nearly 100x.
Akamai developed a fingerprinting methodology specifically for HTTP/2 that hashes these settings values. A mismatch here, even with a perfect TLS fingerprint, results in detection.
Beyond settings, browsers exhibit specific behaviors in header ordering. Chrome sends pseudo-headers in the order :method, :authority, :scheme, :path. It sends regular headers in a specific sequence with sec-ch-ua client hints appearing before other headers. Go's HTTP/2 implementation uses different ordering, providing yet another detection vector. Btw chrome used the sequence m,a,s,p.
The Architecture of httpcloak
Building a browser-identical HTTP client requires intervention at multiple layers of the network stack.
For TLS fingerprinting, httpcloak builds on uTLS, a fork of Go's crypto/tls package that allows specification of custom ClientHello parameters. This provides the foundation for mimicking browser TLS handshakes, including support for the X25519MLKEM768 post-quantum key exchange that Chrome 131 and later versions use.
The more complex challenge lies in HTTP/2. Go's x/net/http2 package doesn't expose the controls needed for fingerprint matching. httpcloak implements custom HTTP/2 framing that sends the exact SETTINGS values Chrome uses, manages WINDOW_UPDATE frames appropriately, and includes PRIORITY frame data that Chrome 143 (version 143 was released when this article was written) sends but Go omits.
The library ensures pseudo-headers appear in browser order and regular headers follow Chrome's specific sequencing, including proper placement of client hints.
Session management follows patterns familiar to anyone who has used Python's requests library. A session maintains cookies across requests, handles redirects appropriately, and manages connection pooling for efficiency.
Verification Methodology
Claims of browser-identical fingerprints require verification. I used tls.peet.ws, a service that analyzes TLS and HTTP/2 characteristics and returns fingerprint hashes, to compare httpcloak against both Go's standard library and actual Chrome.
The results confirmed fingerprint matching across all three major hashing algorithms: JA4 for TLS, Akamai's hash for HTTP/2, and peetprint for combined analysis. The hashes httpcloak produces are identical to those from Chrome 143 running on the same platform.
One important thing, JA3 hashes will vary between connections even for identical clients because they include GREASE values, which are intentionally randomized. This is expected and correct behavior using the same GREASE values repeatedly would itself be a fingerprint.
Practical Considerations
httpcloak addresses connection-level fingerprinting, which represents the first several layers of bot detection. It does not address JavaScript-based detection that examines canvas rendering, WebGL capabilities, or browser API availability. Those checks require an actual browser environment.
The library supports HTTP/1.1, HTTP/2 and HTTP/3, with automatic protocol negotiation and fallback. Proxy support covers HTTP, HTTPS, and SOCKS5 protocols. Standard features like automatic decompression, configurable redirects, and retry with exponential backoff are included.
For use from other languages, an IPC daemon communicates via JSON over stdin/stdout, allowing Python, Node.js, or any language capable of spawning processes to leverage the fingerprinting capabilities.
Availability
The library is available at github.com/sardanioss/httpcloak under the MIT license. Documentation includes implementation details for those interested in the technical approach, along with usage examples for common scenarios.
Contributions and feedback are welcome.
Tags: go, golang, networking, security, http, cloudflare, bypass, bot, cloudflare bypass, requests
Top comments (0)