DEV Community

Eyvind Barthelemy
Eyvind Barthelemy

Posted on

I scanned 136 companies to see if AI search can actually read their sites - the data (and how to fix yours)

#webdev #beginners #seo #ai

AI answer engines (ChatGPT, Perplexity, Google AI Overviews, Claude) are turning into a real referral source. But they can only cite pages they can actually crawl and parse. So I built a tiny, open-source checker and ran it against the homepages of 136 well-known companies across 7 industries to see how AI-ready the web really is.

The results weren't what I expected.

What I measured

Each homepage scored 0-100 across six things AI crawlers and answer-engines rely on:

  • AI-crawler access - does robots.txt allow GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended, ClaudeBot?
  • Structured data - JSON-LD (Organization, WebSite, FAQPage)
  • Title + meta description
  • Open Graph tags
  • XML sitemap
  • llms.txt

The findings

Average score by industry (higher = easier for AI to read):

Industry Avg score
Marketing agencies 92
SaaS 88
Dev tools 86
E-commerce 85
AI startups 83
Fintech 74
Healthtech 63

What jumped out:

  • Structured data is the #1 gap. A large share of sites ship no Organization or FAQPage JSON-LD - which is exactly the format AI answers like to quote.
  • Plenty of well-known tech companies score in the C/D range, almost always because of missing schema, not anything hard.
  • The bar is low. A deliberate, clean setup puts a tiny site ahead of companies a thousand times its size in how readable it is to AI search.

How to fix yours (copy-paste)

1. Let the AI crawlers in - robots.txt:

User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

2. Tell engines who you are - Organization JSON-LD in your <head>:

<script type="application/ld+json">
{"@context":"https://schema.org","@type":"Organization","name":"YOUR COMPANY","url":"https://yourdomain.com","sameAs":["https://www.linkedin.com/company/yourco","https://x.com/yourco"]}
</script>
Enter fullscreen mode Exit fullscreen mode

3. Add FAQPage JSON-LD to any page with Q&A - it's the structure AI answers quote most.

Check your own site (free, open source)

I open-sourced the checker. No install needed:

npx github:epistemedeus/ai-readiness https://yourdomain.com
Enter fullscreen mode Exit fullscreen mode

It prints a 0-100 score plus the exact gaps and fixes. Browser version (no install): https://samedaydesk.com/tools/ai-readiness

If you'd rather grab every template ready to paste (robots.txt, all the JSON-LD, sitemap, meta/OG) in one file, I bundled them into a $9 kit: https://buy.stripe.com/9B66oI1BEdTV6116oieZ20j - but the free checker plus the snippets above honestly get most sites 80% of the way there.

What does your site score? The thing that surprised me most: a bunch of AI companies can't be read by AI. 🤔

Top comments (0)