AI assistants are becoming a referral channel whether sites prepare for it or not. People ask ChatGPT, Claude and Perplexity for recommendations, and those assistants read the web to answer. Which raises a question almost nobody is checking: can they actually read YOUR site?
There are three mundane, checkable things that decide it.
The three checks
llms.txt. A plain-text file at yoursite.com/llms.txt that tells AI systems what your site is and where the important content lives, like robots.txt but for guidance instead of prohibition. It went from proposal to quiet standard fast: in our scans this month, Shopify, Stripe's docs, Anthropic's docs and WordPress.org all serve one.
AI crawler access in robots.txt. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, CCBot: each can be allowed or blocked in robots.txt, and many sites block them without remembering they did (often a blanket rule added in 2023). Blocking is a legitimate choice for some businesses. Blocking accidentally while your marketing team invests in "AI visibility" is just self-harm.
A sitemap. Boring, decades old, still how any crawler, AI or classic, discovers your content systematically.
What we found
We ran a scored audit (0 to 100 across those checks) on a mix of docs sites and publishers. WordPress.org scored 88: llms.txt present and well-formed, every major AI crawler explicitly allowed, sitemap in place. Stripe's and Anthropic's docs scored 75 and 85.
The Irish Times scored 33: no llms.txt, no llms-full.txt, and GPTBot explicitly blocked in robots.txt. A national newspaper, invisible by choice or accident to the fastest-growing answer engine there is. (For a publisher wanting licensing leverage, that may be strategy. If so, the llms.txt file stating terms is still missing.)
Check your own site in 30 seconds
Manually: visit yoursite.com/llms.txt, yoursite.com/robots.txt (search it for "GPTBot" and "ClaudeBot"), and yoursite.com/sitemap.xml. That's the whole audit for one site.
At scale (client portfolios, prospect lists, agencies selling AI-visibility work): I built llms.txt Checker, which runs all the checks on a list of URLs and returns the 0-100 score with a plain-English fix list, about $5 per 1,000 sites. The example run on the listing shows the exact newspaper result above.
The fixes are usually a day's work: write an llms.txt, audit the robots.txt rules against actual intent, confirm the sitemap. Cheap insurance on a channel that's only growing.
Top comments (0)