I ship small sites and APIs for a living, and like most people I checked them the way I was taught: Google Search Console, a Lighthouse run, view-source once in a while. Everything looked fine.
Then I asked a different question: not "can Google see this site?" but "can ChatGPT's crawler?" Those are not the same audience, and I had never once checked the second one.
So I wrote down the checklist I now run before calling any site "launched", automated the parts that can be automated, and ran it on my own sites first. My API hub scored 75/100. It had no llms.txt at all — the file whose entire job is introducing your site to LLMs, 404ing on the site of someone who sells APIs for a living. Another site scored 95, missing only a canonical tag. Both had been live for weeks.
The checker is open source (MIT, single-file Python, zero dependencies) if you want to run it on your own sites: https://github.com/binbin7418-dotcom/newkwlab-ai-site-check — but this post is about how the checks were chosen, because that's the part worth arguing about.
The four check groups, and why
1. Fetch the homepage as GPTBot (5 pts). Before anything else, the tool fetches your homepage with GPTBot's user agent. A 401/403 here means every other check is academic — the crawler is refused at the door. It happens more than you'd think, usually thanks to a CDN rule someone added during a bot scare.
2. robots.txt, resolved the way crawlers resolve it (25 pts). Existence is 10 points. The other 15 check six crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended — against the / path using longest-match semantics: when an Allow and a Disallow both match, the longer path wins, and ties go to Allow. A naive "does the file contain the string GPTBot" check gets real sites wrong in both directions.
3. llms.txt (25 pts). The joint-heaviest item, deliberately. In every batch of sites I've run, this is the biggest binary gap: the file either exists with an H1, real markdown links and actual substance, or it's a 404. Partial credit exists (a thin file earns some points), but a missing one is the single most common reason a site is mute to LLMs.
4. Sitemap + raw HTML (45 pts). The sitemap check doesn't stop at "file exists": it samples the first listed URLs and requires them to answer 200. The raw-HTML group (title, meta description, exactly one H1, canonical, and a word count of the HTML as served) exists because most AI crawlers don't execute JavaScript. If your content isn't in the served HTML, it isn't in the answer. Roughly 300 words is the pass line; under ~120 the page is a shell.
Bugs the tool found in itself
Two worth mentioning, because both produced confidently wrong results before they produced errors:
- My first H1 extractor was regex-based. A > inside a tag attribute (perfectly legal HTML) ended the match early and produced garbage headings. The fix was boring and correct: a real HTMLParser subclass. Parse HTML with an HTML parser — I know, I know.
- Truncated HTTP transfers. A partial read looks like a short page, and a short page looks like a fail. The fetcher now keeps partial bodies when they're usable, and retries network errors with backoff — a dropped packet is not a finding about someone's website.
The scoreboard (real runs, 2026-11-16)
| Site | Score | Deciding factor |
|---|---|---|
| firecrawl.dev | 100/100 (A) | llms.txt with 205 links; 4,046 words in raw HTML |
| pitchreel.newkwlab.com (mine) | 95/100 (A) | Missing canonical — everything else passed |
| api.newkwlab.com (mine) | 75/100 (B) | No llms.txt at all (fixed the same day) |
| example.com | 21/100 (F) | No llms.txt, no sitemap, almost no content |
What a score doesn't mean
A 100 means crawlers can fetch, crawl and understand the site. It does not mean ChatGPT will cite you — citation depends on content, authority and the question being asked, and anyone selling a guarantee is selling fiction. The score is a "you're not invisible" check, nothing more. I think that distinction matters, and it's why the tool is a checker and not a "rank in ChatGPT" dashboard.
What's next
The check list grows from real failures: sites that score wrong, checks people ask for. If you run it and something looks off, an issue with the URL is gold. And if you disagree with the weights — especially llms.txt at 25 — I want to hear why.
Top comments (1)
Since you asked about the llms.txt weight: I'd put it lower. It's a proposal (llmstxt.org, 2024), and as far as I know none of the operators of the crawlers you test documents reading it as an input. I'd move those points to the raw-HTML group, where the effect on what crawlers actually receive is clear.
Two checks that tend to go wrong in both directions:
*group is ignored entirely for GPTBot, so a Disallow under*no longer applies to it. Worth a test case next to longest-match.__NEXT_DATA__JSON.Also, a GPTBot user agent sent from your own IP isn't the same request as GPTBot from OpenAI's IP ranges. WAF rules that verify bot IPs can block the spoofed request, or let it through while the real crawler gets a 403.