DEV Community: MyTaghazout

The header I trusted was lying by accident

MyTaghazout — Sat, 29 Aug 2026 11:39:02 +0000

If you have Cloudflare in front of your site, and at some point you wrote a cache rule that matches bots, go and read that rule again. I wrote one to be helpful to AI crawlers. What it actually did was store my error responses at the edge and replay them to every AI crawler for an hour at a time.

I found out because OpenAI's crawler could not read my site. GPTBot was getting HTTP 429. Amazonbot and Bytespider were getting 403. I run taghazout.io alone, so there was nobody to hand the problem to. I did what most people do first: I looked at the response headers.

That is where I got it wrong.

The header I trusted

The 429 responses carried this:

x-turbo-charged-by: LiteSpeed

LiteSpeed is the web server my host runs. I read that header, decided the origin was producing the 429, and opened a ticket asking about rate limiting on their side. It felt like a closed case. The server had signed its name on the error.

The flaw took me too long to see. A response served from a CDN cache is a stored copy of an origin response, headers included. If Cloudflare stored a 429 an hour ago, it hands you that copy later, x-turbo-charged-by: LiteSpeed and all. That header tells you which server generated the body at some point in the past. It tells you nothing about whether this response, the one in my terminal right then, came from the origin or from a cache in another city.

I had used a header as evidence for a claim it cannot make. It was not lying to me on purpose — server-identifying headers get copied along with everything else, and they will point you at an innocent party by accident.

What the host actually said

Namecheap support came back and said the domain is proxied through Cloudflare, that a "Cache Everything" style rule can cause 429 or 403 responses to be cached at the edge, and that this looked more like a stale or corrupted cached response at the CDN than something LiteSpeed generated.

They were right and I was wrong. The easy version of this story is "host blames the CDN, developer is vindicated". That is not what happened. I arrived with a bad theory and their support gave me a better one.

The header that actually answers the question

curl -s -D - -o /dev/null \
  -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot' \
  https://taghazout.io/ \
  | grep -iE '^(HTTP/|cf-cache-status|cf-ray|age|cache-control|x-turbo-charged-by)'

If cf-cache-status says HIT, you are holding a stored copy and the status code may be older than the problem you are debugging. If it says BYPASS or MISS, Cloudflare went to the origin and what you see is live. Look the other values up rather than guessing: EXPIRED does not mean what it sounds like — the object was in cache, had gone stale, and was served from the origin anyway. A non-zero age is a second hint; cf-cache-status answers directly.

Use -D - -o /dev/null, not curl -I: -I sends a HEAD request, and HEAD can be routed and cached differently from the GET a real crawler sends. Measure the request the crawler actually makes.

Root cause 1: I had cached my own errors

The rule was mine. In Cloudflare's cache rules sat one I had written, ai-crawlers-edge-cache. It matched 13 AI crawler user agents and set Edge TTL to "Ignore cache-control header and use this TTL: 1 hour". My origin sends no-store on HTML. The rule overrode it.

The trap has a simple shape:

Your origin returns an error to one bot. A rate limit, a security block, one bad minute.
Your edge rule says "ignore the origin's cache-control, use this TTL instead".
Cloudflare stores that error the same way it would store a page.
Until the TTL expires, every request matching the same cache key gets the error back, including from crawlers that never did anything wrong.

Caching a 200 is the point of the rule. Caching a 429 is the bug. Nothing in the dashboard separates the two for you.

I had built a machine for turning one bad response into an hour of bad responses, and I had built it on purpose, for exactly the bots the rule was written to help.

What changed:

ai-crawlers-edge-cache Edge TTL: Ignore cache-control header → Respect origin headers
A full cache purge afterwards, because the poisoned entries survive the rule change
Verification is now per-crawler, not "AI crawlers are fine"

ClaudeBot, OAI-SearchBot, PerplexityBot and Googlebot all went from 429 or 403 to 200. Amazonbot and Bytespider did not move: both still return 403, and on a cache miss, which means those are coming from the origin too. At the time I read that as leftover noise. It was the second cause, already visible, and I did not look at it.

I wrote it up as solved. It was not solved.

GPTBot alone still gets 429

Four crawlers turning green felt like proof. It was proof about four crawlers.

So I ran an isolation test built to separate three explanations: a cached error, a flat block, and a rate limit. Ten distinct URLs — distinct matters, or you are testing your own cache — six seconds apart.

UA_GPTBOT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot'

# ten distinct URLs, pulled from the site's own sitemap
# (if yours is a sitemap index, pull one child sitemap instead)
curl -s https://taghazout.io/sitemap.xml \
  | grep -o '<loc>[^<]*' | sed 's/<loc>//' | head -10 > urls.txt

while read -r url; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -A "$UA_GPTBOT" "$url" </dev/null)
  printf '%s  %s  %s\n' "$(date +%T)" "$code" "$url"
  sleep 6
done < urls.txt

First request: 200. The next nine: 429 — roughly one successful request per minute, which is what the timestamps are there to show. The same URLs with a normal Chrome user agent returned 200 every time.

Then the same URL again, six user agents back to back. Running them concurrently is tighter, and it is how I would do it now:

URL='https://taghazout.io/'

while IFS='|' read -r name ua; do
  ( printf '%-16s %s\n' "$name" \
      "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" "$URL" </dev/null)" ) &
done <<'EOF'
GPTBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot
OAI-SearchBot|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
ClaudeBot|Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
PerplexityBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Googlebot|Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Chrome|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
EOF
wait

Vendors bump their user-agent versions, so check the current published strings before copying. And </dev/null is not decoration: without it the backgrounded subshells eat the heredoc and you lose rows.

User agent (same URL, seconds apart)	Status
GPTBot	429
ClaudeBot	200
OAI-SearchBot	200
PerplexityBot	200
Googlebot	200
Chrome	200

Same URL, same edge, same origin, all within a minute of each other. Only the user agent string differs.

And the header that started all this, on those 429s: cf-cache-status came back BYPASS / MISS. Cloudflare went to the origin for them. Live origin 429s, not replays. Cloudflare's AI Crawl Control dashboard also showed every crawler, GPTBot included, set to Allowed.

So: not the edge, and not a flat block. A user-agent-scoped rate limit at the origin, letting through roughly one GPTBot request per minute.

I took that evidence back to support, and this time I described the behaviour instead of my theory: same URL, same second, GPTBot 429 while five other user agents get 200, on a cache miss. The answer came back quickly — ModSecurity. The web application firewall was matching GPTBot's requests and refusing them, and the fix is to whitelist the rule that fires.

Which turns out to be a decision rather than a button. Whitelisting narrowly — one rule ID, scoped to that one user agent — is close to free: GPTBot only ever issues GET requests for public pages, so the exemption has almost no surface. Whitelisting the same rule for all traffic is a different trade entirely, and worth pushing back on. Ask which rule ID fired, and ask for it scoped to the user agent. That request is with the host now.

Amazonbot and Bytespider, still on 403, are almost certainly the same layer.

Why crawler access is worth this much trouble

taghazout.io is not a weather app, although you would not guess that from the search traffic. It sells surf weeks, surf lessons, board rental and airport transfers around Taghazout, and bookings run over WhatsApp with a quote I review myself and a 20% deposit. Almost everything Google sends me is a swell, webcam or sea-state query — the forecast side of the stack is in my earlier post, I run a surf forecast for 20 breaks in Morocco on EUR 0/month.

Two numbers turned crawler access from a hygiene item into a revenue question. In a 16-day window, Bing Webmaster Tools recorded 80 citations in Microsoft Copilot and partner AI surfaces, against 21 Bing search clicks — assistants cited the site about four times more often than Bing search sent people to it. For scale, Google Search Console over 28 days: 430 clicks, 25,100 impressions, average position 11. Total external backlinks: 3, from 2 domains.

Three backlinks from two domains is an ordinary link profile, and that is the point: if an assistant can read your pages, it can recommend you without anyone having linked to you first. I am not going to out-link anybody. Being readable by assistants is the channel I actually have, and someone asking an assistant where to book a surf week in Morocco is much closer to a booking than someone checking whether tomorrow is offshore.

So a crawler getting a 429 is not a log-noise problem. It is a distribution problem.

What I would tell myself before opening that ticket

Read cf-cache-status first. It answers origin-versus-replay directly and costs thirty seconds. x-turbo-charged-by, server and friends are stored copies, and they will point you at an innocent party by accident.
An edge rule that ignores origin cache-control caches your errors as cheerfully as your successes. Caching 200s is the intent; caching 429s is the bug, same rule doing both.
"The obvious cause was real" and "the obvious cause was the only cause" are different claims. Verify per-crawler: a fix that moves four crawlers from red to green will hide the fifth that did not move.
In the support ticket, describe the behaviour, not your theory. Mine was confident and wrong, and the host's read of it was correct — and when I went back the second time with a reproducible test instead of a hunch, the real cause came back in one reply.
If you care about being cited by AI assistants, crawler access is a revenue path, not a nice-to-have — and it can be broken by a rule you wrote yourself to help them.

I run a surf forecast for 20 breaks in Morocco on EUR 0/month. Here's the stack.

MyTaghazout — Mon, 17 Aug 2026 00:04:53 +0000

I live on the Taghazout coast in Morocco - a strip of Atlantic between Agadir and Imsouane that's basically one long right-hand point break after another. Two years ago the only way to know if tomorrow was worth it was to check three different global forecast sites, none of which knew the difference between Anchor Point and the beach break 400m south of it.

So I built taghazout.io. It now covers 20 named breaks, runs in 10 languages, and costs me nothing per month. Here's how it's actually put together - including the parts I'd do differently.

The stack is deliberately boring

Hand-rolled PHP. No framework, no build step, no node_modules. About 4,800 files, server-rendered, no hydration.

That sounds like a confession, but it was the right call for one reason: my readers are on phones, on cafe Wi-Fi, often on 3G. A server-rendered page that ships HTML and a little CSS beats anything I could have built with a client-side framework in that environment. Time-to-content is the only metric that matters when someone is standing on the beach deciding whether to paddle out.

The hosting is a cheap shared plan. The forecast data is free and open. The whole thing runs at EUR 0/month recurring, which was a hard constraint from day one.

The interesting part: two ocean models that disagree

The forecast blends two sources:

Open-Meteo (CC BY 4.0) - the primary, with a marine endpoint that covers our coastal cells.
NOAA WaveWatch III via PacIOOS - the second opinion.

Here's the thing nobody tells you: they disagree, a lot. On the same hour at the same break I've seen WaveWatch read ~55% higher than Open-Meteo (1.36m vs 0.88m). Offshore models resolve coastal bathymetry badly, and our points are exactly the kind of close-in, shallow-reef setups where that bias shows up.

The wrong fix is to pick one and pretend. What I did instead:

Run both, cache both.
Compute agreement over a 72-hour window - a Pearson correlation on the swell rhythm plus a circular difference on direction (you can't average 350 and 10 degrees naively).
Show the user a confidence flag: agree / mixed / diverge.

A recent live read for Anchor Point: r=0.88, 8 degrees direction difference over 73 hours, so "agree". When it says diverge, that's genuinely useful information - it means treat the numbers with suspicion, not that the site is broken.

Some gotchas from the WaveWatch integration, since these cost me a full evening:

The ERDDAP query brackets must be percent-encoded as %5B / %5D.
The time stop must be the literal (last), not a future timestamp you computed.
The coastal grid cell you'd expect (lon 350.5) is dry land. You have to fall back half a degree offshore to get swell at all.

Honest data beats impressive data

The tide readout says, in the payload itself:

"modelled sea level, relative to this window's low"

Because that's what it is. It isn't a harbour gauge, and pretending otherwise would get someone hurt on a reef. Every derived number on the site carries its caveat, and the public feed carries the same note in a source_note field. I'd rather be trusted than look authoritative.

I opened up the feed

The same data the site's own forecast pages run on is public, keyless JSON:

curl "https://taghazout.io/weather-data/_feed.php?loc=anchor-point"

{
  "ok": true,
  "spot": { "name": "Anchor Point", "area": "North Taghazout" },
  "swell_m": 0.66,
  "period_s": 8.4,
  "wind_kmh": 11.2,
  "water_temp_c": 21.6,
  "surf_potential_percent": 82,
  "tide": { "label": "Rising tide", "trend": "rising", "height_m": 2.53 }
}

There's a small MIT-licensed client for it in Python and JS: github.com/MyTaghazout/taghazout-surf, a full API reference, and the spot registry is published as a dataset on Hugging Face.

If you build something with it, carry the attribution: weather by Open-Meteo (CC BY 4.0), wave model NOAA WaveWatch III via PacIOOS.

What I'd do differently

Ten languages was too many, too early. Nine of them started as machine output and I have spent months undoing chimera sentences - English word order with individual words swapped. If I started again: two languages done properly, and add the rest only when each one has real readers.
I measured nothing for six months. The analytics tag was gated behind cookie consent in a way that made most visitors invisible, so I was optimising blind. Fix your measurement before you optimise anything.

Happy to answer anything about the model-blending or the caching layer in the comments.

Weather data by Open-Meteo.com (CC BY 4.0). Wave model NOAA WaveWatch III via PacIOOS.