DEV Community

Cover image for Cloudflare 403s Your HttpClient. Here's the Four-Rung Ladder That Finally Got the PDF.
Gabriel Le Roux
Gabriel Le Roux

Posted on

Cloudflare 403s Your HttpClient. Here's the Four-Rung Ladder That Finally Got the PDF.

#dotnet #webscraping #cloudflarechallenge #tls

A user saved a medRxiv paper to my app. PDF extraction failed — not a parse error, a 403 Forbidden on the download itself. The host sits behind Cloudflare, and Cloudflare had decided my server wasn't a person.

This is the normal case now, not the exception. Preprint servers, industry reports, government documents — a growing share of the PDFs people actually want live behind bot detection. A "save this PDF" feature that only works on politely-served PDFs fails on all the interesting ones.

What follows is the four-rung escalation ladder I ended up with. Each rung exists because the one below it failed on a real URL, and the whole thing is sorted by cost, which turns out to be the most important design decision in it.

Rung 1: HttpClient with realistic browser headers

The baseline. A normal .NET HttpClient, a real Chrome user-agent string, the full header set a browser would send.

var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.UserAgent.ParseAdd(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
    "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36");
request.Headers.Accept.ParseAdd("application/pdf,text/html;q=0.9,*/*;q=0.8");
request.Headers.AcceptLanguage.ParseAdd("en-US,en;q=0.9");
request.Headers.Add("Sec-Fetch-Dest", "document");
request.Headers.Add("Sec-Fetch-Mode", "navigate");
Enter fullscreen mode Exit fullscreen mode

This still works for most of the web and costs nothing. medRxiv 403'd it without blinking.

Why it fails: because modern bot detection doesn't read your headers. It reads your handshake.

Rung 2: curl-impersonate, because TLS is the real fingerprint

Here's the thing that took me embarrassingly long to internalise:

Your TLS handshake identifies your HTTP client before a single byte of HTTP is sent.

Every client negotiates TLS with a characteristic combination of cipher suites, extensions, elliptic curves, and — crucially — the ordering of all of them. That combination is stable per library and is trivially fingerprintable (JA3/JA4). .NET's SslStream announces "I am a program written in .NET" in the ClientHello. You can spoof every HTTP header you like; you're still holding up a sign.

curl-impersonate is curl rebuilt against the same TLS stack Chrome uses, with the handshake tuned to match byte for byte. You shell out to curl_chrome116 (or whichever build) and you look, at the transport layer, exactly like Chrome.

var tmp = Path.GetTempFileName();
var psi = new ProcessStartInfo("curl_chrome116")
{
    RedirectStandardError = true,
    RedirectStandardOutput = false,   // important — see below
};
psi.ArgumentList.Add("-sL");
psi.ArgumentList.Add("--max-time"); psi.ArgumentList.Add("60");
psi.ArgumentList.Add("-o");          psi.ArgumentList.Add(tmp);
psi.ArgumentList.Add(url);

using var proc = Process.Start(psi)!;
await proc.WaitForExitAsync(ct);

var bytes = await File.ReadAllBytesAsync(tmp, ct);
Enter fullscreen mode Exit fullscreen mode

The implementation detail that bites everyone: do not pipe a binary response through anything that touches strings. RedirectStandardOutput = true plus ReadToEndAsync() will hand you a string, and the moment a PDF's bytes go through UTF-8 decoding they are silently, irreversibly corrupted. Write to a temp file, read it back as byte[].

Rung 3: the same fingerprint, from a residential IP

Some hosts don't care what you sound like, only where you're calling from. Whole datacenter ASNs are blocked outright — AWS, Hetzner, DigitalOcean, all of it.

So rung 3 is rung 2 again, through a residential proxy:

psi.ArgumentList.Add("--proxy");
psi.ArgumentList.Add($"http://{proxyUser}:{proxyPass}@{proxyHost}:{proxyPort}");
Enter fullscreen mode Exit fullscreen mode

Chrome's TLS fingerprint. A residential IP. A full browser header set.

medRxiv 403'd that too.

At which point the lesson lands: you are not going to out-imitate the browser. You have to be the browser.

Rung 4: an actual browser — and the indirection that makes it work

The last rung drives a real stealth browser (I use Browserless; Playwright or Puppeteer with a stealth plugin gets you the same place).

But you cannot just point a headless browser at the PDF URL and grab the response. The naive version fails exactly like the others, because a direct hit on the PDF has no session behind it. The move that matters is indirection:

  1. Navigate to the article page — the HTML landing page, not the PDF.
  2. Let the browser sit through and solve the Cloudflare challenge, the way any visitor does. It gets a clearance cookie.
  3. Then fetch the PDF from inside that page's JavaScript context, so the request carries the cookies the challenge just minted.
  4. Return the bytes as base64, because the query layer between you and the browser only speaks scalars. 
// evaluated inside the page, after the challenge has been solved
const res = await fetch(pdfUrl, { credentials: 'include' });
const buf = await res.arrayBuffer();
return btoa(String.fromCharCode(...new Uint8Array(buf)));
Enter fullscreen mode Exit fullscreen mode

Now the PDF request arrives as what it claims to be: a follow-on request from a browser that has already proved itself. That's the difference between imitating trust and acquiring it.

The medRxiv paper that had failed in production for three days came through at 189 KB with a clean %PDF header.

Why the order matters more than the rungs

The ladder isn't "try things until one works." It's sorted by cost.

Rung Mechanism Cost Defeats
1 HttpClient + headers Free Naive UA checks
2 curl-impersonate Free TLS fingerprinting
3 curl_chrome + residential proxy Cheap Datacenter IP blocks
4 Stealth browser Metered Full JS challenges

Rungs 1–3 are effectively free. Rung 4 is a paid, metered service with a per-session price. Escalating in cost order means the expensive tier only runs when every cheap option has genuinely failed — and it's quota-checked and usage-recorded like every other paid call.

The expensive path being last is what makes it affordable to have at all. Put the stealth browser first and you have a beautiful, reliable, financially ruinous fetcher.

The single most important line: validate the bytes, not the status code

This is the part I'd tattoo on people if it were legal.

Bot walls routinely return 200 OK with an HTML block page where the PDF should be. Trust the status code and that garbage flows downstream into your PDF parser, which fails later with a baffling error pointing nowhere near the actual cause. You will spend an afternoon debugging a "corrupt PDF" that was never a PDF.

So every rung validates the payload:

private static bool LooksLikePdf(ReadOnlySpan<byte> bytes) =>
    bytes.Length > 4 &&
    bytes[0] == 0x25 && bytes[1] == 0x50 &&   // %P
    bytes[2] == 0x44 && bytes[3] == 0x46;     // DF
Enter fullscreen mode Exit fullscreen mode

Four bytes. A block page fails immediately, at the rung that received it, and the attempt log reads like a story:

[1] plain      → 403
[2] curl_chrome → 403
[3] proxied     → 403
[4] stealth     → 200, %PDF, 189KB ✓
Enter fullscreen mode Exit fullscreen mode

When some future host needs a rung five, that log is where I'll find out what to build.

The general shape

Cheap thing first. Escalate on failure. Validate the payload, not the envelope. Log every attempt.

None of that is PDF-specific — it's the shape of any fetch pipeline that touches the adversarial web. The web you build against in development is cooperative. The web your users actually save from is not.

Build the ladder before you need the top rung.

Top comments (0)