Cloudflare was 403-ing ChatGPT, Perplexity and Claude on my site, and my logs never knew

For three weeks I wrote content aimed squarely at answer engines. An llms.txt,
FAQPage JSON-LD on two pages, a comparison page built to be quotable, a "how it
works" page with HowTo markup. The site is a small dating app I run on my own
server, loviam.com, Symfony and PostgreSQL on a single
box behind Cloudflare.

Referrals stayed at zero. The only external referrer my analytics table had ever
recorded, over its whole history, was chatgpt.com: three visits, one day in
July, never seen again.

I assumed the content was not good enough. The robots had never read it.

The five minute probe that should have been the first thing I did

Nothing in my usual instruments could see the problem, so I stopped looking at
them and asked the site directly, once per user agent, at its real public
address:

for ua in \
  "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; [e-mail])" ; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://www.loviam.com/)
  echo "$code  $ua"
done

200  Googlebot
403  OAI-SearchBot
403  PerplexityBot
403  Claude-SearchBot

Same result for ChatGPT-User, Claude-User and MistralAI-User. Meanwhile
bingbot, DuckDuckBot, Applebot, YandexBot and an ordinary browser all
got a 200.

The 403 body was eight bytes long, blocked., with server: cloudflare in the
headers, and there was no matching line in the Apache access log. The refusal
happened at the edge. My origin never heard about it.

Why every instrument I had was blind to this

This is the part worth stealing, because the failure mode generalises.

Application logs cannot record a request that never arrives. Obvious once
said, easy to forget when you are grepping the access log for PerplexityBot
and concluding "it never came".

First party analytics is worse than blind, it is reassuring. My beacon fires
from the page. A robot that gets a 403 never gets the page, so it never appears,
so the dashboard looks exactly like "nobody is interested".

My deployment smoke tests talked to the wrong server. They run with
curl --resolve www.loviam.com:443:127.0.0.1, which is the right call for a
deployment gate: you want to test the code you just shipped, not the CDN cache.
But it means the whole test suite speaks to Apache directly and Cloudflare is
invisible to it by construction. Thirty-eight green checks, every deploy, on a
path no visitor uses.

And the managed robots.txt actively pointed the wrong way. Cloudflare
injects its own block above yours. Mine named GPTBot, ClaudeBot, CCBot,
Bytespider, Amazonbot, meta-externalagent, Google-Extended and
Applebot-Extended, all of them training crawlers. Blocking those is a
defensible editorial decision and I stand by it.

But the HTTP filter was blocking something else: the search and on-demand
reading robots, which were not named in that file at all. So a perfectly
obedient crawler read "you are allowed", requested the page, and got a 403. That
mismatch between the two settings is the tell. It is not a policy, it is a
misconfiguration, and reading either file alone will never show it to you.

The fix is one toggle, and it is not the one named after crawlers

Cloudflare dashboard, zone level: Security → Settings → Block AI bots.

Not "AI Crawl Control", which is where I looked first and where the interesting
per-crawler table lives. The switch that returns the 403 is the plain one in
Security Settings, and it treats "AI bot" as a single category: the crawler that
builds ChatGPT's search index and the crawler that scrapes you for training data
are the same thing to it, even though for a publisher they are close to
opposites. One sends you traffic. The other does not.

Also worth checking afterwards: the managed robots.txt block, so it does not
still name a robot you have just decided to let through.

The guard I put in, and why it is a timer and not a test

What broke here is not my code. It is an edge configuration that can change
without a deploy, from a dashboard, possibly by a provider default I never
chose. A test in CI would only prove it was fine at build time.

So it is a daily command instead, app:seo:check-crawlers, which:

• probes the real public URL, through Cloudflare, never --resolve;
• walks sixteen user agents in three families: search, answer engines, training;
• checks two independent things per robot, the actual HTTP status and whether the served robots.txt names it in a Disallow, because those two disagreeing is the exact signature I missed;
• keeps an ordinary browser user agent as a control, so that "the robots are blocked" is never confused with "the site is down";
• fails for search and answer engines, and never for training crawlers, which stay blocked on purpose.

A systemd timer runs it every morning and mails me only on state change.
An alert that fires every day is an alert nobody reads.

The second thing Cloudflare was doing, which was worse

While I was in there, I checked what else the proxy changed about requests, and
found the site had been taking itself down for weeks.

Every request reached Apache carrying the IP of a Cloudflare relay.
mod_remoteip was loaded but never configured, so nothing restored the real
client address:

172.71.135.63 - - [25/Jul/2026:12:21:03 +0000] "GET /fr/ HTTP/2.0" 200 10399

fail2ban reads those logs. So fail2ban was banning Cloudflare's own relays.
Cloudflare could then no longer reach my origin, and served HTTP 521 to every
visitor routed through that relay. The correlation was exact, to the second:

18/07 14:50:39  fail2ban  [apache-auth] Ban 141.101.98.192   (a Cloudflare range)
18/07 14:50:39  uptime probe: DOWN (HTTP 521)
18/07 15:49     unban cascade
18/07 15:54:02  uptime probe: RECOVERED

Five episodes in one week, the longest 42 minutes. My apache-auth jail triggers
at maxretry = 2, and my staging host sits behind htpasswd and therefore
manufactures 401s: two fat-fingered logins behind a shared relay were enough to
take production down for everyone else behind it.

Two quieter casualties of the same root cause: every rate limiter keyed on
$request->getClientIp() was bucketing the entire planet into a handful of
relays, and the visitor hash behind my "unique visitors" number was hashing the
relay, not the visitor. The weekly figure I had been reading to judge growth was
not counting what I thought it counted.

The fix is to trust CF-Connecting-IP, but only when the request comes from a
published Cloudflare range. Without that restriction anyone could forge the
header and walk straight past your rate limiting and your bans. Second layer, as
a net: put those same ranges in fail2ban's ignoreip, so that if the first layer
ever stops working you ban nobody rather than banning your own CDN. Both files
are generated by a monthly script, because the ranges do move, and a frozen list
would quietly reopen the trap.

What I would tell myself in June

If something sits between your users and your server, test through it, with
the user agent of the thing you care about. Not from your laptop, not with
--resolve, not from the logs. The whole class of bug here is that the failure
happens in a place none of your instruments can observe, and every instrument
you own will report a calm, plausible, completely wrong "nobody came".