Libme

Posted on Sep 6

curl Returns 200 but Your HTTP Client Gets a 404: Debugging Vary and CDN Cache Variants

#performance #api #devops #webdev

If the same URL returns different responses depending on which client asks, you are almost certainly hitting two different cache entries, not two different origin states. A CDN keys its cache on the URL plus whichever request headers the origin listed in Vary — most often Accept-Encoding. curl sends no Accept-Encoding by default; nearly every HTTP library sends gzip, deflate. That one difference puts them on opposite sides of a cache split, and a stale error response can sit in one variant for hours while the other is perfectly healthy.

I lost most of an afternoon to this. A publishing script called an API endpoint for an article that had just gone live and got a clean 404 every single time. Pasting the identical URL into curl returned 200 with the full payload. Same machine, same network, same second.

Why do two clients get different responses from the same URL?

The cache key. RFC 9110 says a response's Vary header lists the request headers that participate in matching a stored response. So when the origin sends:

Vary: Accept-Encoding, Origin, X-Loggedin

the CDN stops storing "one response for /api/articles/123" and starts storing "one response per distinct combination of those three headers." Those are variants. They have independent TTLs, independent ages, and independent contents.

Now the client difference matters:

# curl sends no Accept-Encoding by default -> the "identity" variant
curl -sI https://example.com/api/articles/123

# --compressed sends Accept-Encoding: gzip -> a different variant
curl -sI --compressed https://example.com/api/articles/123

import requests
# requests/urllib3 send "Accept-Encoding: gzip, deflate" unless told otherwise
r = requests.get("https://example.com/api/articles/123")
print(r.status_code)

In my case, the resource had been requested during the brief window right after publication when the origin still answered 404. That 404 got stored in the gzip variant with a long edge TTL (x-accel-expires: 172800), and every subsequent library call matched it. The identity variant was populated later, after the origin was consistent, and served a fresh 200. Nothing retried its way out of that — the two variants had no idea the other existed.

The takeaway: when one client succeeds and another fails on the same URL, compare the request headers before you touch the origin.

How do you confirm it's a cache variant and not a flaky origin?

Diff the response headers of the two clients. Age, X-Cache, CF-Cache-Status, and Vary will tell you almost immediately.

for enc in "identity" "gzip"; do
  echo "--- Accept-Encoding: $enc"
  curl -sI -H "Accept-Encoding: $enc" https://example.com/api/articles/123 \
    | grep -iE '^(HTTP/|age|vary|x-cache|cf-cache-status|x-accel-expires)'
done

Two signals confirm the diagnosis:

The two runs return different status codes or different Age values. A large Age (mine was over 58,000 seconds) on the failing variant means you're reading something cached long before your current problem started.
Vary names a header that differs between your clients.

If both variants return identical status and near-zero Age, stop here — the problem is at the origin, and you should be reading origin logs instead.

The takeaway: a large Age on the failing request and a small one on the succeeding request is a cache-variant fingerprint, not an origin bug.

What can you actually do when the bad variant belongs to someone else's CDN?

This is the uncomfortable case: it's a third-party API, you can't purge their cache, and support tickets take longer than your deploy. Your only real lever is changing which variant you land on, by controlling the headers you send.

import requests

session = requests.Session()
# Bypass the poisoned gzip variant entirely by matching the identity cache key.
session.headers["Accept-Encoding"] = "identity"

r = session.get("https://example.com/api/articles/123", timeout=10)
r.raise_for_status()

That is what fixed my publishing script: one helper that every outbound call to that API goes through, pinning Accept-Encoding: identity. The cost is real — you give up response compression, so this is a reasonable trade for small JSON payloads and a bad one for large ones. Do it in a single wrapper function, not scattered at each call site, so you can reverse it in one edit later.

Options, honestly compared:

Approach	Works when	Real cost
Change your `Accept-Encoding`	Third-party CDN, `Vary: Accept-Encoding`	Loses compression; still cached, just a different variant
Add a cache-busting query param	You control nothing but the URL	Pollutes their cache, may violate rate limits or ToS
`Cache-Control: no-cache` request header	CDN honors it (many don't for anonymous traffic)	Frequently ignored; unreliable to depend on
Purge by URL	You own the CDN	Often purges only one variant — verify it clears all
Retry with backoff	The origin is genuinely flaky	Useless here; every retry matches the same poisoned key

That last row is the trap. Retries feel like the safe generic fix, and against a cache-key problem they are pure latency with a guaranteed failure at the end.

The takeaway: retries cannot fix a cache-key mismatch, because every retry computes the same key.

How do you stop your own service from creating this problem?

If you operate the origin, the fix is to keep the variant count small and predictable.

Normalize Accept-Encoding at the edge rather than passing raw client values through. Client header values are wildly diverse (gzip;q=1.0, deflate;q=0.8, br, and so on), and if each string becomes its own cache key you get a low hit ratio plus many more chances to store a bad response somewhere. Varnish's default VCL normalizes this for exactly that reason, and Fastly documents the same pattern:

sub vcl_recv {
    if (req.http.Accept-Encoding) {
        if (req.http.Accept-Encoding ~ "br") {
            set req.http.Accept-Encoding = "br";
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } else {
            unset req.http.Accept-Encoding;
        }
    }
}

Three more rules that have saved me repeatedly:

Never let an error response inherit a success TTL. A 404 or 5xx cached for two days is the actual damage here; the variant split only decided who saw it. Set short negative-caching TTLs explicitly.
Never Vary on high-cardinality headers. Vary: User-Agent multiplies your cache by thousands of variants and effectively disables it. Vary: Cookie on an authenticated endpoint is worse — one wrong key and you serve someone else's data.
Vary: Origin needs care with CORS. If you echo the request Origin into Access-Control-Allow-Origin without varying on it, a cached response hands the wrong origin's CORS header to the next caller. Varying on it is correct, but it is another cache split.

If you want a managed edge where this behavior is configurable rather than emergent, both Fastly (through VCL) and AWS CloudFront (through cache policies that let you specify exactly which headers enter the key) give you explicit control of the cache key instead of leaving it to whatever Vary your framework happens to emit. As of mid-2026, check your provider's current documentation before assuming a specific normalization default — this is exactly the kind of behavior that changes between platform versions.

The takeaway: your cache key should be something you designed, not a side effect of your framework's default headers.

FAQ

Why does curl work but Python requests return 404?
Because curl sends no Accept-Encoding header by default while requests sends gzip, deflate, and if the response carries Vary: Accept-Encoding those two requests hit different CDN cache entries. Reproduce it with curl --compressed — if that also fails, you've confirmed the cache variant is the cause rather than anything client-specific.

Does the Vary header affect the CDN cache key?
Yes. Every header listed in Vary becomes part of the cache key, so the CDN stores a separate copy of the response for each distinct combination of those header values. This is why purging "the URL" sometimes fails to clear the copy your client is actually receiving.

How do I force a request to bypass a bad cached variant?
Change a header that appears in the response's Vary list — most commonly by sending Accept-Encoding: identity — so your request maps to a different cache key. You cannot rely on Cache-Control: no-cache in the request, because many CDNs ignore it for anonymous traffic.

Bottom line

If two clients disagree about the same URL, diff their request headers and their Age values before you suspect the origin. When the poisoned variant lives on a CDN you don't control, your only reliable move is to change the headers that make up the cache key — pin Accept-Encoding in one shared HTTP helper and accept the loss of compression for small payloads. When you own the origin, normalize Accept-Encoding at the edge, keep Vary to low-cardinality headers, and give error responses their own short TTL. Retries are the wrong instinct here; they recompute an identical key and fail identically.

DEV Community