Your website looks great. Clean design, fast load times, maybe even a perfect Lighthouse score.
But to an AI agent? It might as well not exist.
AI agents don't have eyes
Here's something most developers haven't internalized yet: AI agents don't open a browser. They don't render your beautiful CSS. They don't execute your JavaScript.
They send an HTTP request. They get back HTML (hopefully). They try to parse it into something meaningful.
That's it. No clicking cookie banners. No waiting for React to hydrate. No solving CAPTCHAs.
And this is where things fall apart for most of the web.
We tested 834 websites. The results were brutal.
At SiliconFriendly, we built a scoring system (L0 to L5) that rates how well a website works with AI agents. Then we pointed it at 834 real websites.
The breakdown:
- 50.4% scored L0-L1 — essentially hostile to agents. Blocked crawlers, no structured data, JavaScript-only rendering.
- 35.2% scored L2-L3 — partially accessible, but missing key pieces.
- 13.6% scored L4 — good, with structured APIs or solid metadata.
- 0.8% scored L5 — fully agent-friendly. That's 7 websites out of 834.
Let that sink in. 96% of the websites we tested are partially or fully invisible to AI agents.
What actually blocks AI agents?
It's not one thing. It's a stack of problems:
1. JavaScript-rendered content. If your content only exists after JS executes, agents see an empty page. SPAs are the worst offenders. Your React app with client-side rendering? To an AI agent, it's a blank <div id="root"></div>.
2. Aggressive bot blocking. CAPTCHAs, Cloudflare challenges, rate limiting — all designed to stop bots. They work. They stop AI agents too. The irony: you're blocking the same AI systems that could be recommending your product to millions of users.
3. No structured data. Schema.org markup, OpenGraph tags, clean semantic HTML — these are how agents understand what your page is about. Without them, agents are guessing.
4. No llms.txt. This is the new robots.txt for the AI era. A simple file at your domain root that tells AI agents what your site does, what content matters, and how to interact with it. Almost nobody has one yet.
5. No machine-readable APIs. If the only way to get data from your site is to scrape HTML, you're making agents work ten times harder than they need to.
What makes a website agent-friendly?
The websites that scored L4-L5 had a few things in common:
They serve real HTML. Server-side rendering or static generation. Content is in the initial response, not loaded by JavaScript after the fact.
They have an llms.txt file. A simple, markdown-formatted file that tells AI systems: here's who we are, here's what we offer, here's our most important content. Think of it as a cover letter for AI agents visiting your site.
They expose structured data. JSON-LD schema markup on key pages. Not just basic Organization schema — product info, FAQs, how-to guides, all marked up.
They have documented APIs. Even a simple REST API with clear docs gives agents a clean way to interact with your service.
They don't block AI crawlers by default. Check your robots.txt. If you're blocking GPTBot, ClaudeBot, or similar user agents, you're opting out of the AI-powered web.
Why this matters now
AI agents are going to be a massive traffic and conversion channel. Not next year — now.
When someone asks ChatGPT "what's the best project management tool for small teams?", the answer comes from somewhere. If your website is invisible to the systems generating that answer, you don't exist in that conversation.
This isn't theoretical. Companies are already seeing measurable traffic from AI referrals. The ones that show up are the ones whose sites are actually parseable by these systems.
Check your own site
We built SiliconFriendly specifically for this. Punch in your URL and get an instant score (L0-L5) with specific recommendations.
It checks your llms.txt, structured data, JavaScript dependency, crawler accessibility, and more. Takes about 30 seconds.
The fix isn't hard for most sites. Add an llms.txt. Add some Schema.org markup. Make sure your critical content is in the HTML response. Unblock AI crawlers in robots.txt.
The web is splitting into two: sites that AI agents can work with, and sites that are invisible to them. Right now, 96% of sites are on the wrong side.
Don't be part of the 96%.
Check your site's AI-agent readiness at siliconfriendly.com
Top comments (0)