DEV Community: Shubham Gupta

Why 96% of the Web Is Invisible to AI Agents (And What You Can Do About It)

Shubham Gupta — Thu, 26 Mar 2026 11:12:50 +0000

Your website looks great. Clean design, fast load times, maybe even a perfect Lighthouse score.

But to an AI agent? It might as well not exist.

AI agents don't have eyes

Here's something most developers haven't internalized yet: AI agents don't open a browser. They don't render your beautiful CSS. They don't execute your JavaScript.

They send an HTTP request. They get back HTML (hopefully). They try to parse it into something meaningful.

That's it. No clicking cookie banners. No waiting for React to hydrate. No solving CAPTCHAs.

And this is where things fall apart for most of the web.

We tested 834 websites. The results were brutal.

At SiliconFriendly, we built a scoring system (L0 to L5) that rates how well a website works with AI agents. Then we pointed it at 834 real websites.

The breakdown:

50.4% scored L0-L1 — essentially hostile to agents. Blocked crawlers, no structured data, JavaScript-only rendering.
35.2% scored L2-L3 — partially accessible, but missing key pieces.
13.6% scored L4 — good, with structured APIs or solid metadata.
0.8% scored L5 — fully agent-friendly. That's 7 websites out of 834.

Let that sink in. 96% of the websites we tested are partially or fully invisible to AI agents.

What actually blocks AI agents?

It's not one thing. It's a stack of problems:

1. JavaScript-rendered content. If your content only exists after JS executes, agents see an empty page. SPAs are the worst offenders. Your React app with client-side rendering? To an AI agent, it's a blank <div id="root"></div>.

2. Aggressive bot blocking. CAPTCHAs, Cloudflare challenges, rate limiting — all designed to stop bots. They work. They stop AI agents too. The irony: you're blocking the same AI systems that could be recommending your product to millions of users.

3. No structured data. Schema.org markup, OpenGraph tags, clean semantic HTML — these are how agents understand what your page is about. Without them, agents are guessing.

4. No llms.txt. This is the new robots.txt for the AI era. A simple file at your domain root that tells AI agents what your site does, what content matters, and how to interact with it. Almost nobody has one yet.

5. No machine-readable APIs. If the only way to get data from your site is to scrape HTML, you're making agents work ten times harder than they need to.

What makes a website agent-friendly?

The websites that scored L4-L5 had a few things in common:

They serve real HTML. Server-side rendering or static generation. Content is in the initial response, not loaded by JavaScript after the fact.

They have an llms.txt file. A simple, markdown-formatted file that tells AI systems: here's who we are, here's what we offer, here's our most important content. Think of it as a cover letter for AI agents visiting your site.

They expose structured data. JSON-LD schema markup on key pages. Not just basic Organization schema — product info, FAQs, how-to guides, all marked up.

They have documented APIs. Even a simple REST API with clear docs gives agents a clean way to interact with your service.

They don't block AI crawlers by default. Check your robots.txt. If you're blocking GPTBot, ClaudeBot, or similar user agents, you're opting out of the AI-powered web.

Why this matters now

AI agents are going to be a massive traffic and conversion channel. Not next year — now.

When someone asks ChatGPT "what's the best project management tool for small teams?", the answer comes from somewhere. If your website is invisible to the systems generating that answer, you don't exist in that conversation.

This isn't theoretical. Companies are already seeing measurable traffic from AI referrals. The ones that show up are the ones whose sites are actually parseable by these systems.

Check your own site

We built SiliconFriendly specifically for this. Punch in your URL and get an instant score (L0-L5) with specific recommendations.

It checks your llms.txt, structured data, JavaScript dependency, crawler accessibility, and more. Takes about 30 seconds.

The fix isn't hard for most sites. Add an llms.txt. Add some Schema.org markup. Make sure your critical content is in the HTML response. Unblock AI crawlers in robots.txt.

The web is splitting into two: sites that AI agents can work with, and sites that are invisible to them. Right now, 96% of sites are on the wrong side.

Don't be part of the 96%.

Check your site's AI-agent readiness at siliconfriendly.com

How to make your website AI-agent friendly in 30 minutes

Shubham Gupta — Fri, 20 Mar 2026 12:34:55 +0000

AI agents are the new browsers. They're crawling, reading, and trying to interact with your site right now — and over 50% of websites are basically unusable for them. If you want agents to actually work with your site, here's how to fix that in 30 minutes.

Step 1: Add llms.txt (5 min)

Create a /llms.txt file at your site root. This is a plain-text file that tells AI agents what your site does and how to use it. Think of it as robots.txt, but instead of saying "don't crawl this," you're saying "here's how to understand me."

The spec lives at llmstxt.org. Here's what a good one looks like:

# Acme API

> Acme provides a REST API for managing invoices and payments.

## Docs

- [API Reference](/docs/api): Full endpoint documentation
- [Authentication](/docs/auth): How to authenticate requests
- [Webhooks](/docs/webhooks): Event notification setup

## Key Endpoints

- POST /api/invoices - Create a new invoice
- GET /api/invoices/{id} - Retrieve invoice details
- GET /api/customers - List customers

## Notes

- All endpoints return JSON
- Rate limit: 100 requests/minute
- Auth: Bearer token in Authorization header

Drop this at yourdomain.com/llms.txt. That's it. Agents that support the spec will read it before trying to parse your HTML.

You can also add a /llms-full.txt with your complete documentation in markdown — useful for agents that want to ingest everything at once.

Step 2: Add structured data (10 min)

Agents parse your HTML, but they understand structured data much better. The fastest win is adding JSON-LD markup to your pages using schema.org vocabulary.

For a SaaS product page:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Acme Invoicing",
  "description": "Automated invoice management for small businesses",
  "applicationCategory": "BusinessApplication",
  "operatingSystem": "Web",
  "offers": {
    "@type": "Offer",
    "price": "29.00",
    "priceCurrency": "USD"
  },
  "url": "https://acme.com",
  "documentation": "https://acme.com/docs"
}
</script>

For a blog post, use Article. For an API, use WebAPI. For a business, use Organization.

Other quick wins:

Use semantic HTML: <nav>, <main>, <article>, <header> — not <div class="nav-wrapper-v2">.
Add descriptive meta tags: <meta name="description"> actually matters again.
Use clear heading hierarchy: Agents use h1→h6 to build a content tree, just like screen readers do.

Step 3: Make your API agent-accessible (10 min)

If you have an API, three things make it dramatically easier for agents to use:

Add an OpenAPI spec. Host it at a known location like /openapi.json:

{
  "openapi": "3.0.0",
  "info": {
    "title": "Acme API",
    "version": "1.0.0"
  },
  "paths": {
    "/api/invoices": {
      "post": {
        "summary": "Create an invoice",
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "type": "object",
                "required": ["customer_id", "amount"],
                "properties": {
                  "customer_id": { "type": "string" },
                  "amount": { "type": "number" }
                }
              }
            }
          }
        }
      }
    }
  }
}

Return structured JSON errors, not HTML error pages:

{
  "error": "invalid_request",
  "message": "customer_id is required",
  "status": 400
}

Agents can't parse your pretty 404 page. They need machine-readable errors to self-correct.

Document rate limits in response headers:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 87
X-RateLimit-Reset: 1710892800

If you don't have an API, consider adding a single read-only endpoint — even a /api/info.json that returns basic metadata about your site is useful for agent discovery.

Step 4: Check your score (5 min)

Go to siliconfriendly.com/check and enter your domain. It evaluates your site across 30 criteria on an L0–L5 scale:

L0 – No agent support (most sites today)
L1 – Basic: has llms.txt or structured metadata
L2 – Navigable: semantic HTML, clear content hierarchy
L3 – Interactable: API with OpenAPI spec, structured errors
L4 – Integrated: agent.json, A2A support
L5 – Fully agent-native

Most sites start at L0. Following steps 1–3 above should get you to L2 or L3.

Bonus: Add agent.json

For agent-to-agent (A2A) discovery, you can add a /.well-known/agent.json file. This tells other AI agents what your agent (or service) can do and how to communicate with it:

{
  "name": "Acme Invoicing Agent",
  "description": "Manages invoices and payments",
  "url": "https://acme.com",
  "capabilities": ["create_invoice", "lookup_customer"],
  "protocol": "https",
  "auth": "bearer"
}

This is newer and less standardized, but the /.well-known/ convention is gaining traction. If you're building anything with agent interoperability in mind, it's worth adding now.

TL;DR

Step	Time	What
llms.txt	5 min	Tell agents what your site does
Structured data	10 min	JSON-LD + semantic HTML
API access	10 min	OpenAPI spec + JSON errors
Check score	5 min	siliconfriendly.com

The bar is low right now — most sites do none of this. Spending 30 minutes puts you ahead of half the web.

Check where you stand: siliconfriendly.com

Your AI Agents Are Flying Blind. Here's a Pre-Flight Check.

Shubham Gupta — Sat, 28 Feb 2026 14:14:15 +0000

Your agent hits a 403. It retries. Gets a CAPTCHA. Retries again. Finally gets some HTML back — 200KB of JavaScript-rendered noise it burns 4,000 tokens trying to parse before concluding it can't extract anything useful.

You've been there. I've been there. Every agent builder has been there.

The problem isn't your agent. It's that there's no way to know if a website will cooperate before your agent wastes time and tokens on it.

The Spectrum of Agent-Friendliness

Not all websites treat agents the same. Some have structured APIs, machine-readable docs, and llms.txt files. Others serve you a Cloudflare challenge page and call it a day.

Silicon Friendly rates 834+ websites on a scale from L0 to L5:

Level	What it means
L0	Hostile — blocks agents outright
L1	Passive — no agent support, you're scraping
L2	Basic — has an API, but docs are human-only
L3	Structured — good APIs, machine-readable content
L4	Verified — strong agent support across 30 criteria
L5	Agent-native — built for programmatic access first

Each rating is based on 30 concrete criteria: structured error responses, rate limit headers, auth documentation, machine-readable pricing, llms.txt support, and more.

Wire It Into Your Agent in 5 Minutes

Silicon Friendly exposes an MCP server your agents can query at runtime. If you're using CrewAI (which supports MCP natively), here's the setup:

pip install crewai crewai-tools[mcp]

from crewai import Agent, Task, Crew
from crewai_tools import MCPServerAdapter

sf_server = {
    "url": "https://siliconfriendly.com/mcp",
    "transport": "streamable_http",
}

with MCPServerAdapter(sf_server, connect_timeout=30) as tools:
    scout = Agent(
        role="Integration Scout",
        goal="Evaluate services for agent compatibility before integration",
        backstory=(
            "You check whether websites and APIs are built for AI agents. "
            "You use Silicon Friendly to look up agent-friendliness ratings "
            "and search for alternatives when a service scores poorly."
        ),
        tools=tools,
        verbose=True,
    )

    task = Task(
        description=(
            "We need to integrate a payment processor. "
            "Check if stripe.com is agent-friendly. "
            "Then search for other payment processors and compare."
        ),
        expected_output=(
            "Stripe's agent-friendliness level, "
            "alternatives with their levels, "
            "and a recommendation."
        ),
        agent=scout,
    )

    crew = Crew(agents=[scout], tasks=[task])
    result = crew.kickoff()
    print(result)

The agent calls check_agent_friendliness on stripe.com (L4), then search_websites for "payment processing" to find alternatives like Razorpay (L5) and Square Developer (L5). It makes an informed recommendation instead of guessing.

Key Tools Available via MCP

Tool	Purpose
`check_agent_friendliness`	Quick L0-L5 check for any domain
`search_websites`	Semantic search by use case ("email API", "cloud storage")
`get_website`	Full 30-criteria breakdown for a domain
`get_levels_info`	Explains the rating system

No auth required for reads. The MCP endpoint is https://siliconfriendly.com/mcp using Streamable HTTP transport.

Works with smolagents Too

If you're using HuggingFace's smolagents instead of CrewAI, the same MCP server works — smolagents has native MCP support. Any framework that speaks MCP can connect.

Practical Advice

L3+ is the sweet spot. Anything rated L3 or above has structured data, clear APIs, and machine-readable content. Below that, your agent is scraping HTML and guessing.

Use search instead of hardcoding. Instead of assuming which service to use, let your agent search for "email API" or "cloud storage" and pick the highest-rated option. The directory has 834+ websites indexed.

Check the details when it matters. get_website returns the full 30-criteria breakdown — you'll see exactly what works (structured error responses, rate limit headers) and what doesn't (no machine-readable pricing, no agent auth flow).

Check Your Own Stack

Before your next agent build, check whether the services you depend on will actually cooperate:

👉 siliconfriendly.com/llms.txt

If a site you use isn't rated yet, you can submit it through the MCP server or the website. You get bonus API queries for each site you verify.

Silicon Friendly is an open directory. The MCP server is free to use. Registry entry: com.siliconfriendly/directory.

How AI agents browse the web (and why your site might be invisible to them)

Shubham Gupta — Fri, 27 Feb 2026 07:53:49 +0000

How AI agents browse the web (and why your site might be invisible to them)

Most sites are built for humans sitting in front of a browser. That's fine, except AI agents aren't humans, and they don't use browsers.

When an agent needs to interact with your product, it sends an HTTP request, reads the response, and tries to extract something useful. No JavaScript engine, no mouse clicks, no scroll events. If your content only exists after a React component mounts, the agent sees an empty shell. If your site is behind Cloudflare's bot challenge, the agent never gets past the waiting room.

How an agent actually 'sees' a website

The sequence is roughly this: HTTP GET to a URL, parse the HTML or JSON response, check for an llms.txt file at the root, and look for any documented API it can call. That's the whole pipeline.

llms.txt is a simple convention, similar in spirit to robots.txt, that gives agents a curated map of what your site contains and how to use it. If you haven't heard of it, that's part of the problem. Most sites don't have one. Agents fall back to guessing, and guessing usually fails.

The three ways it breaks

In our data, agent failures almost always fall into one of three categories.

Access problems come first. Your robots.txt blocks known AI crawlers. Cloudflare fingerprints the client and serves a challenge page instead of content. Some sites block ClaudeBot in robots.txt while simultaneously using the Claude API in their own backend. That's incoherent, and it's more common than you'd expect.

Parsing problems come second. The page loads fine, technically, but all the meaningful content is rendered client-side. A simple HTTP client gets back a <div id="root"></div> and nothing else. Navigation, pricing, documentation, none of it visible. 50.4% of the sites we've analyzed are in this state: discoverable, but useless to an agent.

Action problems come third. Even if an agent can read your site, there's nothing it can actually do. No API. No structured way to initiate a trial, submit a form, or retrieve account data. The agent can look, but not touch.

The spectrum from broken to useful

Think of it as levels. At L0, your site is actively blocking agents. At L1, it's accessible but the content is unreadable. L2 is readable but static, no way to act. L3 means you have an API, but it's not documented in a way agents can discover. L4 is a documented, agent-accessible API with an llms.txt pointing to it. L5 adds things like OAuth flows and structured error messages that make autonomous operation reliable.

Only 4.2% of sites we've seen are at L4 or above. Most are stuck at L2.

What you can add in a week

None of this requires a rewrite. A few specific additions get you most of the way.

Add or update robots.txt to allow the major AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Googlebot-Extended
Create llms.txt at your domain root with a plain-language description of what your product does and links to your API docs
Add JSON-LD structured data to your homepage, pricing page, and any key product pages, Product, SoftwareApplication, or WebPage schemas work
Make sure your core documentation is server-rendered or available as static HTML, not just client-side rendered
If you have a public API, document it in OpenAPI format and link to it from llms.txt

These are not difficult changes. Most can be done in an afternoon. The payoff is that agents can actually use your product when they're trying to help someone who needs what you offer.

Check where you stand

If you want to see how your site scores across these levels, run it through siliconfriendly.com. It'll tell you exactly which layer you're blocked at and what to fix.

Originally published at Silicon Friendly

Is GPTBot blocked on your site? Here's what that means for AI agents

Shubham Gupta — Fri, 27 Feb 2026 07:53:14 +0000

Is GPTBot blocked on your site? How to check and fix it in 5 minutes

Right now, at least a dozen AI crawlers are probably hitting your site. Some are scraping your content to train models. Others are fetching pages in real time so ChatGPT or Perplexity can answer questions about you. Most site owners have no idea which ones they're allowing and which ones they've accidentally blocked.

Your robots.txt file controls all of this. And there's a good chance yours is either wide open or blocking the wrong things.

Here's how to check and fix it.

The bots you should know about

Not all AI crawlers do the same thing. Some collect training data. Others fetch your pages live when a user asks a question. That difference matters, because you might want to block one but not the other.

Training bots scrape your content to build or fine-tune AI models. Your pages get absorbed into the model's weights. You don't get a link, attribution, or traffic in return.

User-agent bots fetch your pages in real time when someone asks an AI tool a question. These work more like search engines - they can send you traffic, and they usually link back to your site.

Here's the list that actually matters:

Bot	Company	What it does
GPTBot	OpenAI	Training data collection + powers ChatGPT's web browsing
OAI-SearchBot	OpenAI	Fetches pages for SearchGPT results
ChatGPT-User	OpenAI	ChatGPT browsing plugin, real-time page fetches
ClaudeBot	Anthropic	Training data collection
anthropic-ai	Anthropic	Older training crawler, still active
Google-Extended	Google	Gemini training data. This is NOT Googlebot - blocking it won't affect your search rankings
Googlebot	Google	Regular search indexing. Don't block this unless you want to disappear from Google
CCBot	Common Crawl	Open dataset used by many AI labs for training
Bytespider	ByteDance	TikTok's parent company, training data collection
PerplexityBot	Perplexity AI	Real-time search, links back to sources
Applebot-Extended	Apple	Apple Intelligence training. Separate from the regular Applebot used for Siri/Spotlight

GPTBot gets the most attention, but it's only one of many. If you block GPTBot and leave CCBot wide open, your content is still ending up in training datasets.

How to check what you're currently blocking

Open your browser and go to:

https://yourdomain.com/robots.txt

That's it. The file is public. You'll see something like this:

User-agent: *
Disallow: /admin/
Disallow: /private/

If you don't see any AI bot names in there, you're allowing all of them. If you see User-agent: GPTBot with a Disallow: /, that one is blocked.

Most sites I check fall into one of two buckets: they either block nothing, or they block GPTBot specifically because someone read a headline about it in 2023 and forgot about the other ten crawlers doing the exact same thing.

Training vs. browsing - why it matters

Here's the decision most people skip: do you actually want AI tools to be able to reference your content?

If you run a recipe blog and someone asks ChatGPT "how do I make sourdough starter," you probably want ChatGPT-User and OAI-SearchBot to be able to fetch your page and link to it. That's traffic. What you probably don't want is GPTBot scraping your entire archive so the model can answer sourdough questions without ever sending anyone to your site.

The distinction is between bots that take your content for training and bots that bring users to your content in real time.

Block the training bots. Think twice before blocking the browsing ones.

Copy-paste robots.txt configs

Pick the scenario that fits your situation. Add the relevant lines to the end of your existing robots.txt file.

Option A: Block AI training, allow AI search (recommended for most sites)

This blocks bots that scrape for training while still letting AI search tools reference your pages.

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Allow AI search/browsing bots
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

Option B: Block everything AI-related

If you don't want any AI system crawling your site, training or otherwise:

# Block all AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Option C: Allow everything

If you want maximum AI visibility and don't mind your content being used for training:

# Allow all AI crawlers
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Google-Extended
Allow: /

If your robots.txt doesn't mention a bot at all, it's allowed by default. So Option C is really just making your intent explicit.

The catch: robots.txt is voluntary

robots.txt is a protocol, not a firewall. It's a polite request. Well-behaved bots respect it. The major companies - OpenAI, Google, Anthropic, Apple - do follow robots.txt directives. But nothing technically prevents a crawler from ignoring the file entirely.

Some lesser-known crawlers don't check robots.txt at all. If you need hard enforcement, you'll need server-side blocking by user-agent string or IP range. Cloudflare and several other CDNs offer bot management tools that can handle this. But for most sites, robots.txt covers the crawlers that matter.

Blocking is step one. Being agent-friendly is the bigger move.

Controlling which bots can access your site is the defensive play. It's worth doing, but it's only part of the picture.

The sites that will get the most out of AI - not just search engines, but AI agents that browse, compare, and buy on behalf of users - are the ones that make themselves easy for machines to understand. That means structured data, clean APIs, and files like llms.txt that tell AI systems what your site is about and how to use it.

If you want to see where your site stands on all of this, Silicon Friendly rates websites across 30 criteria on a scale from L0 to L5 for agent-friendliness. It checks robots.txt configuration, but also structured data, API availability, and a lot more. Worth a look if you've just fixed your robots.txt and want to know what else you might be missing.

Quick checklist

Visit yourdomain.com/robots.txt right now
Check which AI bots are mentioned (probably none or just GPTBot)
Decide: do you want to block training, browsing, or both?
Copy the relevant config above and add it to your file
Verify the change by revisiting the URL

Five minutes. That's all it takes to go from "I have no idea what's crawling my site" to having a clear, intentional policy about it.

Originally published at Silicon Friendly

What is llms.txt and does your SaaS website need one?

Shubham Gupta — Fri, 27 Feb 2026 07:52:38 +0000

What is llms.txt and does your SaaS website need one?

What is llms.txt

llms.txt is a proposed web standard for telling AI agents about your website. You put a plain-text markdown file at yourdomain.com/llms.txt, and it describes your site's content hierarchy in a format that's easy for language models to parse. Think of it as a plain-language map of your site, written for machines that read, not crawl.

How it differs from robots.txt

robots.txt is about permissions. It tells crawlers what they're allowed to access. llms.txt is about structure and meaning. It doesn't block or allow anything. It just gives AI agents context: what your site is, what pages matter, and how things are organized. Two different jobs.

What it actually looks like

Here's a minimal example:

# Acme Analytics

> Acme Analytics helps SaaS teams track product usage and reduce churn.

## Docs
- [Getting started](https://acme.com/docs/start): How to install and configure Acme
- [API reference](https://acme.com/docs/api): Full API documentation
- [Integrations](https://acme.com/docs/integrations): Connect with Segment, Mixpanel, and others

## About
- [Pricing](https://acme.com/pricing): Plans and pricing
- [Blog](https://acme.com/blog): Product updates and guides

That's it. A short description, then organized links with one-line summaries. No special syntax to learn.

Does it actually change anything for AI agents today

Somewhat. Claude has adopted the standard and uses llms.txt when it's present. Other major models are catching up but aren't there yet. So right now you're mostly optimizing for Claude users, which is a real and growing segment. The spec is being actively developed, and adoption is moving. I'd rather have it in place now than scramble later.

At Silicon Friendly, we've analyzed 832 websites and only 30.8% have llms.txt. That's a low bar to clear, and clearing it matters for how AI agents interact with your product.

How to create one

Create a markdown file. Name it llms.txt. Put it at the root of your domain so it's accessible at yourdomain.com/llms.txt. That's the whole process. Write a one-sentence description of what your product does, then list your most important pages with short labels. Twenty minutes, tops.

Should your SaaS add one

Yes. If you want AI agents, Claude-powered tools, or any AI assistant to understand your product correctly when someone asks about it, you want llms.txt. It costs nothing. It takes under an hour. And it's one of the cleaner signals you can send that your site is built for how people actually use software in 2025 and beyond.

Adding llms.txt moves a site from L1 to potentially L2 on our AI-agent compatibility taxonomy. It's a small file that does real work.

Want to see where your site stands? Check your AI-agent compatibility score at siliconfriendly.com.

Originally published at Silicon Friendly

robots.txt vs llms.txt: What's the difference and why it matters

Shubham Gupta — Fri, 27 Feb 2026 07:52:03 +0000

robots.txt vs llms.txt: what's the difference and which do you need?

Your website talks to two audiences now: search engine crawlers and large language models. They want different things, they read content differently, and the files that serve them have almost nothing in common.

Most developers know robots.txt. Fewer know llms.txt. Almost nobody understands why you need both.

robots.txt: the bouncer

robots.txt has been around since 1994. Martijn Koster, a Dutch engineer, proposed it after web crawlers kept hammering his server. The idea was simple: put a text file at your site root that tells bots which pages they can and can't visit.

For 28 years it existed as an informal standard. Everyone followed it, nobody had formally ratified it. The IETF finally published RFC 9309 in September 2022, making it official.

A typical robots.txt in 2025 looks like this:

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

It's a bouncer. It says "you can come in" or "you can't." That's it. No nuance, no context, no guidance. And compliance is voluntary. There is no enforcement mechanism. Bots follow it because they choose to.

The AI era has turned robots.txt into a battlefield. As of late 2025, 79% of top news sites block AI training bots. GPTBot went from zero mentions to 578,000 websites in about 15 months. ChatGPT-User requests surged 2,825% in a single year.

Blocking a crawler is not the same as helping one. And that's where robots.txt runs out of ideas.

llms.txt: the tour guide

In September 2024, Jeremy Howard (co-founder of Answer.AI and fast.ai) published a proposal for a new file: llms.txt. The idea came from a specific frustration. LLMs increasingly need to use website content at inference time, when they're actively generating a response. Not for training. For answering questions right now.

And they're terrible at it.

Web pages are built for humans with browsers. They have nav bars, footers, ad scripts, cookie banners, tracking pixels, JavaScript bundles. When an LLM tries to read a typical HTML page, most of the tokens go to junk. Converting HTML to clean text can reduce token consumption by 20-30%, but the conversion itself is unreliable.

llms.txt solves this differently. Instead of telling bots what to avoid, it tells them what to read. It's a Markdown file at your site root that curates the most important content and provides clean, structured links.

A well-structured one looks like this:

# Acme API

> Cloud infrastructure API for deploying and managing containers.

Acme API provides RESTful endpoints for container orchestration,
monitoring, and scaling. Supports Docker and OCI images.

## Documentation

- [Getting Started](https://acme.dev/docs/quickstart): Set up your first container in 5 minutes
- [API Reference](https://acme.dev/docs/api): Complete endpoint documentation
- [Authentication](https://acme.dev/docs/auth): API keys, OAuth, and JWT setup

## Guides

- [Scaling Guide](https://acme.dev/guides/scaling): Auto-scaling configuration
- [Migration from AWS](https://acme.dev/guides/aws-migration): Step-by-step migration

## Optional

- [Changelog](https://acme.dev/changelog): Release notes
- [Status Page](https://status.acme.dev): Current system status

The format is simple. An H1 with your project name (the only required element). A blockquote summary. Then H2 sections with curated links and descriptions. The "Optional" section marks stuff that can be skipped when context windows are tight.

The spec also encourages companion files: an llms-full.txt containing all your documentation in one Markdown file, and .md mirrors of individual pages. Anthropic's docs site, for example, has an llms-full.txt that runs to 481,349 tokens.

Why LLMs need something different

This is worth spelling out, because I keep seeing people ask "isn't sitemap.xml enough?"

No. And the reason matters.

A search crawler indexes content. It visits pages, extracts text, stores it in a database for later. The crawler has unlimited time and near-unlimited storage. It doesn't care about token efficiency. It can visit every page on your site and sort out relevance later.

An LLM synthesizes content. It needs to ingest information, reason about it, and produce an answer, often in real time. Context windows are finite. Even Gemini's 1M token window can't hold a medium-sized documentation site. The model needs a curated view: what are the most important pages, what do they cover, and where are the clean Markdown versions?

robots.txt can't express any of that. It's binary. Allow or disallow. It knows nothing about content priority, page relationships, or which resources matter most for understanding your product.

robots.txt is the bouncer checking IDs at the door. llms.txt is the building directory that tells you accounting is on the third floor.

The adoption picture

BuiltWith reports 844,000+ sites now have an llms.txt file. That sounds impressive until you dig in. Most implementations are bare-bones or broken. Common mistakes include wrong filenames (llm.txt or LLMS.txt), missing the required H1 heading, or just dumping a flat list of URLs with no descriptions.

The sites doing it well tend to be developer-facing: Anthropic, Cloudflare, Stripe, Supabase, Cursor. They have extensive documentation that AI coding assistants actively consume. For them, llms.txt is already useful, because tools like Cursor and Windsurf actually read it to pull in documentation context.

The skeptics have fair points too. Google's John Mueller compared llms.txt to the discredited <meta name="keywords"> tag. SE Ranking studied 300,000 domains and found no correlation between having llms.txt and being cited in LLM answers. Then, in December 2025, Google quietly added an llms.txt to its own Search Central documentation. When an SEO professional pointed out the irony, Mueller responded with a cryptic "hmmn :-/".

I think the truth is boring but practical: llms.txt matters right now for developer tools and documentation-heavy products, where AI coding assistants are the primary consumers. For a restaurant website or a local business, probably not worth the effort yet. But AI agents are getting better at using web content every month, and the sites that give them clean structured context will have an advantage when that usage becomes mainstream.

Do you need both?

Yes. They solve entirely different problems.

robots.txt controls access. Without it, you have no way to tell AI training crawlers to stay away from your content. If you care about that (and most publishers do), you need a robots.txt that specifically addresses GPTBot, ClaudeBot, Google-Extended, CCBot, and the growing list of AI crawlers.

llms.txt controls understanding. If you want AI systems to accurately represent your product, answer questions about your API, or help developers use your tools, llms.txt gives those systems the curated context they need.

Blocking training crawlers while providing an llms.txt is not contradictory. You can say "don't scrape my site for training data" in robots.txt while also saying "here's the best way to understand my product" in llms.txt. One is about what bots take from you. The other is about what you give to them.

Where agent-friendliness goes beyond both files

robots.txt and llms.txt are two pieces of a bigger puzzle. A truly agent-friendly website also considers structured data, API access, authentication flows for AI agents, and how well its pages render without JavaScript.

At Silicon Friendly, we rate websites across 30 criteria on an L0-L5 scale, where L0 means "completely invisible to agents" and L5 means "built for agents from the ground up." Having both robots.txt and llms.txt properly configured is part of that picture, but far from the whole thing.

Checklist: making your site work for both crawlers and LLMs

robots.txt

File exists at yourdomain.com/robots.txt
Explicitly addresses AI training bots (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, anthropic-ai)
Separates training bots from search/user bots (OAI-SearchBot and ChatGPT-User are different from GPTBot)
Includes a Sitemap directive
Review and update quarterly as new bots appear

llms.txt

File exists at yourdomain.com/llms.txt (exact filename, lowercase)
Starts with an H1 heading containing your site or project name
Includes a blockquote summary explaining what you do
Links are curated, not exhaustive. 10-30 of your most important pages, not 500
Each link has a short description
Uses an "Optional" section for secondary content
Companion llms-full.txt exists if you have extensive documentation
Linked pages have .md mirror versions
File is reviewed when you add or remove major content

Neither file is complicated. A developer can set up both in an afternoon. The hard part is curation: deciding which content matters most, writing useful descriptions, and keeping the file current as your site evolves. But that's what makes llms.txt valuable in the first place. Automated discovery is noisy. Human curation is signal.

Originally published at Silicon Friendly

We checked 832 websites for AI-agent compatibility. Here's what we found.

Shubham Gupta — Fri, 27 Feb 2026 07:51:27 +0000

We checked 832 websites for AI-agent compatibility. Here's what we found.

Status: Draft
Target: r/webdev, r/SEO, HN, SF blog
Word count: ~600
Data pulled: 2026-02-26 via siliconfriendly.com API (full dataset, 832 sites, 42 pages)

Check where your site stands: siliconfriendly.com

Submit your site, see your level, and see what's missing.

Raw data — full 832-site dataset (2026-02-26)

Level distribution:

L0: 123 sites (14.8%)
L1: 178 sites (21.4%)
L2: 419 sites (50.4%)
L3: 77 sites (9.3%)
L4: 28 sites (3.4%)
L5: 7 sites (0.8%)
Total: 832 sites

Criteria pass rates (all 832 sites):

L1 — Accessible:

clean_urls: 97.6%
ssr_content: 93.6%
meta_tags: 83.8%
no_captcha: 82.7%
semantic_html: 79.2%
schema_org: 43.0%

L2 — Discoverable:

text_content: 94.6%
robots_txt: 84.6%
sitemap: 78.8%
documentation: 73.6%
llms_txt: 30.8%
openapi_spec: 8.3%

L3 — API-ready:

json_responses: 78.0%
search_filter_api: 53.5%
structured_api: 44.2%
structured_errors: 17.7%
rate_limits_documented: 14.5%
a2a_agent_card: 6.1%

L4 — Agent-native:

write_api: 59.9%
agent_auth: 51.0%
mcp_server: 21.0%
webhooks: 19.2%
idempotency: 1.7%
webmcp: 0.2%

L5 — Agentic-first:

event_streaming: 79.0%
workflow_orchestration: 47.6%
subscription_api: 8.3%
proactive_notifications: 2.6%
cross_service_handoff: 1.8%
agent_negotiation: 1.0%

Note on l5_event_streaming (79%) and l5_workflow_orchestration (47.6%): These high numbers reflect sites that check individual criteria boxes, but don't qualify as L5 overall because they fail other L4/L5 requirements. The level rating requires meeting a threshold across all criteria at each level.

Verified: 832/832 (100%)
Verification count: 831 sites with 1 verification, 1 site with 2 verifications

Originally published at Silicon Friendly

The only thing you need to master React! (from my 5 years of experience)

Shubham Gupta — Sat, 11 Nov 2023 14:52:32 +0000

I've been working in and around React since the time it got popular. I've dug in every corner of it, from reading the source code to creating my own libraries to simplify the mess React can create at times.

But this was all possible because of one thing I did, and even to this date, I continue to do it when I want to learn something at its core in the least time possible.

Create your own (mini) version of it.

whY?

React, at its core, is just a way to write HTML using JS. Why JS? Because it's a scripting language and you can write logic in it, while HTML is a declarative language and you can simply declare everything you want.

React combines the two. You can declare with logic, empowering everything. That's it. That's all the React ever was, is, and will be.

What to do?

I get it. The first thought of creating your own mini-version of React can seem both exciting and heart-pounding at the same time. Just like talking to your crush. But here, we will learn all about masterful flirting, i.e. how to masterfully create your own version of React.

Steps:

Let's stop calling it "your version of React.".
Let's call it "GoodAct".
Second, this is where you come in. I am not going to give you any code because reading my code won't make you "master" react. You will have to open up VS Code and write your own code. I, however, will do one thing for you. If you write your code, share the github repo with me. I'd give it a try and feature some good ones in an article. Also, shoot me any questions or if you're stuck somewhere. Let's get you resolved.

My email is shubham@goodgit.io

Basics to understand

To keep things simple, let's not use JSX. It's basically a really complex version of "replace". as in, replace this tag with this code all the way down to HTML.

<Blog title="Hello World" image="/static/image.png">

is replaced to

<div class="blog">
  <h1>Hello World</h1>
<image src="/static/image.png" />
</div>

What to build?

2 things. Virtual DOM and state That's it.

To keep things simple and core, all will be there in one HTML file only. The following weird-looking syntax should be understood by goodact:

<html>
  <head>
    <title>GoodAct</title>
  </head>
  <body>
    <h1>{{ title }}</h1>
    <p>{{ body }}</p>
    <script src="goodact.js"></script>
  </body>
</html>

<html>
  <head>
    <title>GoodAct</title>
  </head>
  <body>
    <input type="number" value="{{ userN }}"/>
    <h1>Your number times 2 is {{ userN * 2}}</h1>
    <script src="goodact.js"></script>
  </body>
</html>

<html>
  <head>
    <title>GoodAct</title>
  </head>
  <body>
    {% for i in range(10) %}
      <h1>Let's count {{ i }}. </h1>
    {% endfor %}
    <script src="goodact.js"></script>
  </body>
</html>

That's it. 😂 I know, I know. it's crazy. but I also know you can do it. It's not easy but it is going to ramp up all the muscles in your brain, but by the end of it, you'll be smarter, sharper and amaster.😉

start on a weekend. Think about how you can build something like this. For those of you familiar with Django or Flask, it will feel a lot of jinja-templating, and you're correct; that's where I drew my inspiration from.

Once you can build a system like this, you'll know exactly how React works under the hood, and you will have a newfound appreciation for the tool you are so used to using.

See you on the other side with your projects working. Shoot me an email if you need any help, just want to talk, or are excited to share what you built!

Also, also, also

Use GoodGit to push your code to GitHub.
Just install it with

pip install goodgit

and then

gg add.

GoodGit takes care of the rest. You don't even have to write a commit message. Just go and use it. You can read about it at https://goodgit.io.