DEV Community

MyTaghazout
MyTaghazout

Posted on

The header I trusted was lying by accident

#ai #seo #webdev #cloudflare

If you have Cloudflare in front of your site, and at some point you wrote a cache rule that matches bots, go and read that rule again. I wrote one to be helpful to AI crawlers. What it actually did was store my error responses at the edge and replay them to every AI crawler for an hour at a time.

I found out because OpenAI's crawler could not read my site. GPTBot was getting HTTP 429. Amazonbot and Bytespider were getting 403. I run taghazout.io alone, so there was nobody to hand the problem to. I did what most people do first: I looked at the response headers.

That is where I got it wrong.

The header I trusted

The 429 responses carried this:

x-turbo-charged-by: LiteSpeed
Enter fullscreen mode Exit fullscreen mode

LiteSpeed is the web server my host runs. I read that header, decided the origin was producing the 429, and opened a ticket asking about rate limiting on their side. It felt like a closed case. The server had signed its name on the error.

The flaw took me too long to see. A response served from a CDN cache is a stored copy of an origin response, headers included. If Cloudflare stored a 429 an hour ago, it hands you that copy later, x-turbo-charged-by: LiteSpeed and all. That header tells you which server generated the body at some point in the past. It tells you nothing about whether this response, the one in my terminal right then, came from the origin or from a cache in another city.

I had used a header as evidence for a claim it cannot make. It was not lying to me on purpose — server-identifying headers get copied along with everything else, and they will point you at an innocent party by accident.

What the host actually said

Namecheap support came back and said the domain is proxied through Cloudflare, that a "Cache Everything" style rule can cause 429 or 403 responses to be cached at the edge, and that this looked more like a stale or corrupted cached response at the CDN than something LiteSpeed generated.

They were right and I was wrong. The easy version of this story is "host blames the CDN, developer is vindicated". That is not what happened. I arrived with a bad theory and their support gave me a better one.

The header that actually answers the question 

curl -s -D - -o /dev/null \
  -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot' \
  https://taghazout.io/ \
  | grep -iE '^(HTTP/|cf-cache-status|cf-ray|age|cache-control|x-turbo-charged-by)'
Enter fullscreen mode Exit fullscreen mode

If cf-cache-status says HIT, you are holding a stored copy and the status code may be older than the problem you are debugging. If it says BYPASS or MISS, Cloudflare went to the origin and what you see is live. Look the other values up rather than guessing: EXPIRED does not mean what it sounds like — the object was in cache, had gone stale, and was served from the origin anyway. A non-zero age is a second hint; cf-cache-status answers directly.

Use -D - -o /dev/null, not curl -I: -I sends a HEAD request, and HEAD can be routed and cached differently from the GET a real crawler sends. Measure the request the crawler actually makes.

Root cause 1: I had cached my own errors

The rule was mine. In Cloudflare's cache rules sat one I had written, ai-crawlers-edge-cache. It matched 13 AI crawler user agents and set Edge TTL to "Ignore cache-control header and use this TTL: 1 hour". My origin sends no-store on HTML. The rule overrode it.

The trap has a simple shape:

  1. Your origin returns an error to one bot. A rate limit, a security block, one bad minute.
  2. Your edge rule says "ignore the origin's cache-control, use this TTL instead".
  3. Cloudflare stores that error the same way it would store a page.
  4. Until the TTL expires, every request matching the same cache key gets the error back, including from crawlers that never did anything wrong.

Caching a 200 is the point of the rule. Caching a 429 is the bug. Nothing in the dashboard separates the two for you.

I had built a machine for turning one bad response into an hour of bad responses, and I had built it on purpose, for exactly the bots the rule was written to help.

What changed:

  • ai-crawlers-edge-cache Edge TTL: Ignore cache-control header → Respect origin headers
  • A full cache purge afterwards, because the poisoned entries survive the rule change
  • Verification is now per-crawler, not "AI crawlers are fine"

ClaudeBot, OAI-SearchBot, PerplexityBot and Googlebot all went from 429 or 403 to 200. Amazonbot and Bytespider did not move: both still return 403, and on a cache miss, which means those are coming from the origin too. At the time I read that as leftover noise. It was the second cause, already visible, and I did not look at it.

I wrote it up as solved. It was not solved.

GPTBot alone still gets 429

Four crawlers turning green felt like proof. It was proof about four crawlers.

So I ran an isolation test built to separate three explanations: a cached error, a flat block, and a rate limit. Ten distinct URLs — distinct matters, or you are testing your own cache — six seconds apart.

UA_GPTBOT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot'

# ten distinct URLs, pulled from the site's own sitemap
# (if yours is a sitemap index, pull one child sitemap instead)
curl -s https://taghazout.io/sitemap.xml \
  | grep -o '<loc>[^<]*' | sed 's/<loc>//' | head -10 > urls.txt

while read -r url; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -A "$UA_GPTBOT" "$url" </dev/null)
  printf '%s  %s  %s\n' "$(date +%T)" "$code" "$url"
  sleep 6
done < urls.txt
Enter fullscreen mode Exit fullscreen mode

First request: 200. The next nine: 429 — roughly one successful request per minute, which is what the timestamps are there to show. The same URLs with a normal Chrome user agent returned 200 every time.

Then the same URL again, six user agents back to back. Running them concurrently is tighter, and it is how I would do it now:

URL='https://taghazout.io/'

while IFS='|' read -r name ua; do
  ( printf '%-16s %s\n' "$name" \
      "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" "$URL" </dev/null)" ) &
done <<'EOF'
GPTBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot
OAI-SearchBot|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
ClaudeBot|Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
PerplexityBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Googlebot|Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Chrome|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
EOF
wait
Enter fullscreen mode Exit fullscreen mode

Vendors bump their user-agent versions, so check the current published strings before copying. And </dev/null is not decoration: without it the backgrounded subshells eat the heredoc and you lose rows.

User agent (same URL, seconds apart) Status
GPTBot 429
ClaudeBot 200
OAI-SearchBot 200
PerplexityBot 200
Googlebot 200
Chrome 200

Same URL, same edge, same origin, all within a minute of each other. Only the user agent string differs.

And the header that started all this, on those 429s: cf-cache-status came back BYPASS / MISS. Cloudflare went to the origin for them. Live origin 429s, not replays. Cloudflare's AI Crawl Control dashboard also showed every crawler, GPTBot included, set to Allowed.

So: not the edge, and not a flat block. A user-agent-scoped rate limit at the origin, letting through roughly one GPTBot request per minute.

I took that evidence back to support, and this time I described the behaviour instead of my theory: same URL, same second, GPTBot 429 while five other user agents get 200, on a cache miss. The answer came back quickly — ModSecurity. The web application firewall was matching GPTBot's requests and refusing them, and the fix is to whitelist the rule that fires.

Which turns out to be a decision rather than a button. Whitelisting narrowly — one rule ID, scoped to that one user agent — is close to free: GPTBot only ever issues GET requests for public pages, so the exemption has almost no surface. Whitelisting the same rule for all traffic is a different trade entirely, and worth pushing back on. Ask which rule ID fired, and ask for it scoped to the user agent. That request is with the host now.

Amazonbot and Bytespider, still on 403, are almost certainly the same layer.

Why crawler access is worth this much trouble

taghazout.io is not a weather app, although you would not guess that from the search traffic. It sells surf weeks, surf lessons, board rental and airport transfers around Taghazout, and bookings run over WhatsApp with a quote I review myself and a 20% deposit. Almost everything Google sends me is a swell, webcam or sea-state query — the forecast side of the stack is in my earlier post, I run a surf forecast for 20 breaks in Morocco on EUR 0/month.

Two numbers turned crawler access from a hygiene item into a revenue question. In a 16-day window, Bing Webmaster Tools recorded 80 citations in Microsoft Copilot and partner AI surfaces, against 21 Bing search clicks — assistants cited the site about four times more often than Bing search sent people to it. For scale, Google Search Console over 28 days: 430 clicks, 25,100 impressions, average position 11. Total external backlinks: 3, from 2 domains.

Three backlinks from two domains is an ordinary link profile, and that is the point: if an assistant can read your pages, it can recommend you without anyone having linked to you first. I am not going to out-link anybody. Being readable by assistants is the channel I actually have, and someone asking an assistant where to book a surf week in Morocco is much closer to a booking than someone checking whether tomorrow is offshore.

So a crawler getting a 429 is not a log-noise problem. It is a distribution problem.

What I would tell myself before opening that ticket

  1. Read cf-cache-status first. It answers origin-versus-replay directly and costs thirty seconds. x-turbo-charged-by, server and friends are stored copies, and they will point you at an innocent party by accident.
  2. An edge rule that ignores origin cache-control caches your errors as cheerfully as your successes. Caching 200s is the intent; caching 429s is the bug, same rule doing both.
  3. "The obvious cause was real" and "the obvious cause was the only cause" are different claims. Verify per-crawler: a fix that moves four crawlers from red to green will hide the fifth that did not move.
  4. In the support ticket, describe the behaviour, not your theory. Mine was confident and wrong, and the host's read of it was correct — and when I went back the second time with a reproducible test instead of a hunch, the real cause came back in one reply.
  5. If you care about being cited by AI assistants, crawler access is a revenue path, not a nice-to-have — and it can be broken by a rule you wrote yourself to help them.

Top comments (1)

Collapse
 
citedy profile image
Dmitry Sergeev • 
{ "content": "I debugged a similar issue where X-Forwarded-Host was overriding the Host header and now I never trust default cache rules without double checking the actual payload"
}
Enter fullscreen mode Exit fullscreen mode