Can ChatGPT and Claude read your website? A 5-minute checklist

Paweł Iwanek — Fri, 16 Oct 2026 14:38:22 +0000

Search is changing: more people ask ChatGPT, Claude or Perplexity instead of opening Google. If those assistants cannot read your site, you will not be quoted. Here is a five-minute checklist.

1. robots.txt: are AI bots blocked?

Open yoursite.com/robots.txt and look for these user agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot. A blanket Disallow: / under User-agent: * blocks all of them. Many CMS security plugins add it silently.

2. Is your text in the HTML?

Run curl -s https://yoursite.com | grep -i "your headline". If the headline is missing, your content is rendered by JavaScript, and most AI crawlers do not run JavaScript. They see an empty page.

3. llms.txt

A plain markdown file at /llms.txt that lists your key pages and describes the site. It is not a standard everyone honors yet, but it costs ten minutes and some assistants read it.

4. Structured data

JSON-LD (Organization, Product, Article, FAQPage) tells a crawler what a page is without guessing. Validate it: a missing required property usually means the markup is ignored.

5. Email authentication (bonus)

If your domain sends mail, check SPF, DKIM and DMARC. Without them a lot of your mail lands in spam, and Google and Yahoo now require them for bulk senders.

Automating it

I got tired of doing this by hand, so I wrapped the checks into small tools on Apify. Each takes a domain and returns a 0-100 score with a concrete fix for every issue:

AI crawler access (robots.txt, llms.txt, JSON-LD, server-rendered text): https://apify.com/rodorn/ai-crawler-access-checker
SPF, DKIM, DMARC, MTA-STS, BIMI: https://apify.com/rodorn/email-authentication-checker
HTTP security headers: https://apify.com/rodorn/security-headers-checker
JSON-LD validator: https://apify.com/rodorn/structured-data-validator
Broken links crawler: https://apify.com/rodorn/broken-links-checker

They are pay-per-result (0.005 USD per result), so scanning a handful of domains costs cents. Feedback and feature requests are welcome in the comments.