Quick recap if you're new here: last month I tested whether five different AI engines (Claude, ChatGPT, Gemini, Perplexity, Bing Copilot) would cite my own websites when answering people's questions. Mostly, they didn't. I chalked some of that up to normal stuff, small sites, not enough content out there about them yet.
Then a reader named riley craig left a comment that changed the story.
He'd checked two files on my sites that I hadn't thought to check myself: robots.txt and llms.txt. Both are supposed to be small text files that live at the root of a website (yoursite.com/robots.txt) and tell automated visitors, like search engines and AI crawlers, what they're allowed to look at and where to find the important stuff. robots.txt has been standard since the 1990s. llms.txt is a newer, similar idea, specifically aimed at AI tools, and it's meant to be a plain, boring text file, nothing fancy.
Mine weren't text files. They were my homepage.
Every time anything, human or bot, requested robots.txt or llms.txt on my site, my server handed back the exact same 35KB webpage you'd get by just visiting the homepage, titled "Remote AI Agent Developer." My other site, naija-vpn.com, had the same problem on llms.txt (its robots.txt was fine, that one worked correctly).
Modern websites are often built as what's called a single-page app, or SPA. One HTML page loads, and then JavaScript swaps content in and out as you click around, instead of loading a fresh page every time. To make that work, the hosting platform (mine is Cloudflare Pages) is usually set up to hand you the main app page whenever you ask for a page that doesn't exist as a real file. The assumption is that you're a person who clicked a broken link and should land somewhere useful.
That fallback is helpful for people. It's a problem for robots.txt and llms.txt, because those specifically need to be small, plain text files, not a webpage. My site's build process, the step that turns source code into the files that get uploaded and served, never generated those two files at all. So every request for them just fell through to the homepage fallback. An AI crawler asking "what am I allowed to read here" got back a full HTML page instead of a straight answer.
I fixed both sites. Confirmed the fix works by checking the files myself afterward. That part took an afternoon.
Fixing my own two small sites isn't much of a story on its own. Anyone could have that exact same bug and never notice, because nothing about it throws an error. Your site still loads fine for anyone visiting it normally. It just quietly gives bots the wrong answer.
So I built a small tool to check for this automatically: ai_crawlability.py, a new piece of an open-source SEO toolkit I maintain. Point it at a domain and it requests robots.txt, llms.txt, and sitemap.xml (a third standard file, this one lists all the pages on a site so search engines can find them), then checks whether each one comes back as a real text file or as a webpage in disguise.
If a site simply doesn't have an llms.txt file at all, that's not this bug. Most sites don't have one yet since it's a new convention, and a normal "file not found" response is completely different from what happened to me. The bug I'm looking for is this: the server says "yes, here's your file" (a status code of 200, which just means "success, request completed normally") but hands you a full webpage instead of the plain text you asked for.
The tool can check specific sites you give it directly. It can also do something more interesting: pull a real, unbiased sample of domains straight from Google search results (using SearchApi, the search API company sponsoring this series of articles) for whatever search terms you give it, then check every domain in that sample. That second mode is the point. Checking my own two sites tells you nothing about how common this bug really is. Checking a real sample does.
What a real sample turned up
I picked ten completely unrelated search topics on purpose, developer tools, travel booking, productivity software, laptop reviews, coffee subscriptions, specifically so the sample wouldn't just turn into a pile of other companies similar to mine. SearchApi's search results gave me 50 unique domains across all ten searches. I checked every one.
| Result | Count |
|---|---|
| Working correctly | 39 |
| Has the bug (webpage instead of text file) | 5 |
| Blocked the request or errored out (a different, unrelated problem) | 6 |
Five broken out of fifty checked. That's one in ten.
reddit.com has the bug on both llms.txt and sitemap.xml. So does skyscanner.com. paymoapp.com has it on llms.txt. And two different Medium.com blogs, run by two different people (ilampadmanabhan.medium.com and navanathjadhav.medium.com), both have it on sitemap.xml, with the broken webpage they each got back weighing in at 41,975 and 41,972 bytes respectively.
Two unrelated writers didn't independently make the identical mistake. Those numbers being that close means it's almost certainly Medium's own platform serving the same broken response to everyone who points a custom domain at a Medium blog. If that's true, it's not affecting two people. It's affecting every writer who's done that.
Not every failure was this bug, though. Six domains failed outright: w3schools.com, pcmag.com, united.com, cheapoair.com, drinktrade.com, and beanbox.com. Before trusting my own tool's judgment on those, I checked two of them by hand. w3schools.com sends back a "403 Forbidden," which means it's actively refusing the request. pcmag.com does something similar behind what looks like bot-detection software. Neither of those is the bug I'm measuring. They're sites that block anything that doesn't look like a real browser, which is a completely different, unrelated situation, and my tool correctly recognized that and left them out of the "broken" count instead of lumping them in.
That distinction turned out to matter more than I expected going in. A tool that can't tell "this site is blocking me" apart from "this site is actually broken" would have either made the 10% number look bigger than it really is, or hidden real bugs inside a pile of unrelated failures.
I haven't gone back and re-run the original 5-engine test on my own two sites since fixing them. Search engines and AI crawlers don't revisit a site the moment you change something, they come back on their own schedule, which can take days or weeks. Testing again too soon would just show the same zero-citation results as before, but for a completely different reason than last time, and that would be a misleading comparison, not a real one. That follow-up test is its own piece, once enough time has gone by for it to mean something.
For now, here's what I can say with confidence: this bug is real, it's not rare, and reddit.com has it right now.
Where to look
- The code: github.com/dannwaneri/seo-agent, specifically
modules/ai_crawlability.py - The tool behind the domain discovery: SearchApi's Google search engine
*This piece was produced as part of SearchApi's Developer Ambassador program. They provided API credits; I built and tested the integration myself.
Top comments (1)
Ngl, that's the sort of thing that slips between the cracks... Good job taking the initiative once someone brought it up, there's alot of sites that likely need a refresh!