Search is changing: more people ask ChatGPT, Claude or Perplexity instead of opening Google. If those assistants cannot read your site, you will not be quoted. Here is a five-minute checklist.
1. robots.txt: are AI bots blocked?
Open yoursite.com/robots.txt and look for these user agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot. A blanket Disallow: / under User-agent: * blocks all of them. Many CMS security plugins add it silently.
2. Is your text in the HTML?
Run curl -s https://yoursite.com | grep -i "your headline". If the headline is missing, your content is rendered by JavaScript, and most AI crawlers do not run JavaScript. They see an empty page.
3. llms.txt
A plain markdown file at /llms.txt that lists your key pages and describes the site. It is not a standard everyone honors yet, but it costs ten minutes and some assistants read it.
4. Structured data
JSON-LD (Organization, Product, Article, FAQPage) tells a crawler what a page is without guessing. Validate it: a missing required property usually means the markup is ignored.
5. Email authentication (bonus)
If your domain sends mail, check SPF, DKIM and DMARC. Without them a lot of your mail lands in spam, and Google and Yahoo now require them for bulk senders.
Automating it
I got tired of doing this by hand, so I wrapped the checks into small tools on Apify. Each takes a domain and returns a 0-100 score with a concrete fix for every issue:
- AI crawler access (robots.txt, llms.txt, JSON-LD, server-rendered text): https://apify.com/rodorn/ai-crawler-access-checker
- SPF, DKIM, DMARC, MTA-STS, BIMI: https://apify.com/rodorn/email-authentication-checker
- HTTP security headers: https://apify.com/rodorn/security-headers-checker
- JSON-LD validator: https://apify.com/rodorn/structured-data-validator
- Broken links crawler: https://apify.com/rodorn/broken-links-checker
They are pay-per-result (0.005 USD per result), so scanning a handful of domains costs cents. Feedback and feature requests are welcome in the comments.
Top comments (0)