Your robots.txt might not say what you think it says

Jan Driessen — Thu, 10 Sep 2026 03:11:20 +0000

I ship a bunch of small free tools, and last week I wanted a quick way to double-check what robots.txt a site was actually serving to crawlers — not what's sitting in the repo, what's actually on the wire.

Turns out those aren't always the same thing. Cloudflare has a feature called AI Crawl Control that can rewrite robots.txt at the edge — inject its own rules for AI crawlers — without touching your origin file at all. If you're only ever checking the file in your repo (or your CMS, or wherever it's generated), you'd never see that. The version Googlebot, GPTBot, or your own CI sees can be different from the one you think you're serving.

Every robots.txt tool I could find is a generator — it helps you build a static file. None of them check what's actually live. So I built a tiny one that does:

npx github:janibert1/robots-check yourdomain.com

Zero dependencies, just fetches the live file and tells you what's there, including whether anything looks CDN-injected. There's also a --ci flag for build pipelines (exits 1 if the whole site is accidentally disallowed) and a stricter --fail-on-cloudflare-managed flag if you want CI to fail on any Cloudflare-injected rewrite specifically, plus a matching GitHub Action (janibert1/robots-check@v1) if you'd rather wire it into a workflow than run it by hand.

Repo (MIT): https://github.com/janibert1/robots-check

If you run a site behind Cloudflare, it's worth a 10-second check — worst case it confirms nothing weird is happening.

DEV Community: Jan Driessen

Your robots.txt might not say what you think it says