DEV Community: Andriy Pyvovarchuk

How to test your llms.txt before you ship it

Andriy Pyvovarchuk — Thu, 06 Aug 2026 13:29:41 +0000

An llms.txt is easy to write and easy to get subtly wrong. The file can look perfectly fine in your editor and still be malformed, or point at pages that 404 — and unlike a broken web page, nothing visibly breaks, so you never notice. Here's exactly what to check before (and after) you publish.

1. Structure

Confirm the spec basics:

exactly one # H1 title,
a > blockquote summary right under it,
## sections that each contain a Markdown list of links.

Multiple H1s and empty sections are the two most common structural mistakes. If your file has two # lines, demote one to ##.

2. Link reachability (the check almost everyone skips)

This is the big one. Every URL you link should actually resolve. A flawless-looking file that points at 404s wastes a crawler's budget and quietly erodes trust — and most "llms.txt checkers" only lint the syntax, so dead links sail straight through them.

You have to actually request each linked URL and look for anything that's broken or redirecting. This is where doing it by hand falls apart: eyeballing structure is fine, but clicking twenty links to check status codes is tedious and easy to get wrong.

3. Best practices

Absolute https:// URLs — not relative paths, not http://.
A short : description after each link so the model knows what it is.
Served as Content-Type: text/plain; charset=utf-8 (a lot of servers wrongly return text/html).

Manual vs automated

You can verify all of the above by hand. Structure, sure. But link reachability across dozens of URLs, checking content type, re-checking every time you edit the file — that's exactly the kind of tedious, error-prone work worth automating.

The fast way

Paste your domain (or the raw text) into an llms.txt validator. It checks the structure, fetches every linked URL concurrently, applies the best-practice rules above, and returns a 0-100 score with a prioritized, fixable list of findings. Re-run it whenever you change the file.

What a clean result looks like: 100/100, every link fetched, with the HTTP status, size, content type and link count broken out. A single broken link drags the score down even when the Markdown is perfect.

The 30-second checklist

[ ] One # H1, one > summary, ## sections with links
[ ] Every linked URL returns 200 (no 404s, no redirect chains)
[ ] All links absolute https://
[ ] A : description on each link
[ ] Served as text/plain; charset=utf-8
[ ] Re-checked after your latest edit

Get those six right and your llms.txt is in better shape than most on the web today. Test it once, fix what's flagged, ship it.

Which AI crawlers actually read your llms.txt? (GPTBot, ClaudeBot & friends)

Andriy Pyvovarchuk — Thu, 06 Aug 2026 05:55:44 +0000

You added an llms.txt. Reasonable next question: who's actually reading it? The honest answer surprises people — so let's clear it up, and more importantly sort out which bots you should care about and how you actually control them (spoiler: not with llms.txt).

First, split the bots into two piles

Lumping all "AI crawlers" together is where people get confused. There are two kinds, and you treat them very differently:

Answer / search crawlers — they fetch a page to answer a user's question right now, and they're the ones that get you cited in AI answers:

OpenAI: OAI-SearchBot, ChatGPT-User
Anthropic: Claude-SearchBot, Claude-User
Perplexity: PerplexityBot, Perplexity-User

You generally want these.

Training crawlers — they collect data to train future models:

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent

Whether to allow these is a separate, opt-out decision — and blocking them does not hurt your AI-search visibility.

So which of them read llms.txt today?

Straight answer: adoption is still early. llms.txt is a young, voluntary convention. It's picking up support in AI developer tools and agent frameworks, but most of the big crawlers above do not yet fetch it automatically. No bot is required to read it.

That sounds like a reason to skip it. It isn't — for two reasons:

The clean pages it points to help models right now. Even a crawler that ignores the file itself benefits from the well-structured, Markdown-friendly pages you curated for it.
It's cheap readiness. A correct llms.txt costs ten minutes and positions you for the moment adoption tips — which, given how fast this space moves, won't be long.

The part people get wrong: llms.txt doesn't control access

This is the big misconception. llms.txt never blocks or grants anything. It only curates — a suggested reading list. If you want to control which AI crawlers may fetch your pages, that's robots.txt's job.

A visibility-friendly setup allows the answer crawlers and makes a deliberate call on the training ones:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

That example says "let AI search engines cite me, but keep my content out of model training." Your call on that last line — just know it's a training opt-out, not a visibility switch.

The short checklist

Allow the answer/search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, …) so AI tools can cite you.
Decide training opt-outs (GPTBot, Google-Extended) deliberately — blocking them doesn't touch AI-search visibility.
Publish a curated llms.txt and then actually check it — the most common failure is a valid-looking file pointing at dead links. Run it through an llms.txt validator that fetches every URL.

llms.txt is the map; robots.txt is the gate. Keep the map honest and the gate deliberate, and you're ahead of nearly everyone.

Does llms.txt help SEO? An honest answer

Andriy Pyvovarchuk — Tue, 04 Aug 2026 06:22:19 +0000

If you searched this, you've probably also seen the other kind of post — "add llms.txt and watch your rankings climb." Let me save you the disappointment and then tell you why the file is still worth ten minutes of your time.

The short answer: no, not for classic SEO

llms.txt is not a Google ranking signal. Google has said Googlebot doesn't use it for search ranking. It's not a meta tag, not a sitemap replacement, not a backlink, and no, it won't move you up the results page. Anyone selling "llms.txt → #1 on Google" is selling snake oil.

So if your only goal is blue-link rankings, llms.txt isn't your lever. Close the tab, go fix your Core Web Vitals.

The longer answer: it helps a different kind of visibility

Still here? Good — because the question is slightly out of date. More and more, people don't get an answer about your product from a ranked list of links. They ask ChatGPT, Claude, Perplexity, or Google's AI overview, and read whatever those models say about you.

Being cleanly readable by those models is its own discipline now — people call it GEO (generative engine optimization), the AI-answer cousin of SEO. And that's exactly what llms.txt is for. A tidy one:

points models at your best, most current pages instead of letting them guess,
gives them clean context so answers about you are accurate rather than hallucinated,
pairs with Markdown versions of your pages so the model reads content, not your nav bar, cookie banner, and ads.

It won't rank you. It changes what the AI says when someone asks about you — which, increasingly, is the thing that actually drives the click.

So is it worth it?

Simple test: does your audience ask AI tools about your domain — your docs, your API, your product? If yes, llms.txt is a small file with real upside and basically no downside. Write it once, keep it curated, done.

If you're a purely local business with no documentation and nobody's asking an LLM about you, the payoff is smaller — spend the time elsewhere.

What to do instead of chasing rankings with it

Treat llms.txt as one piece of AI-readiness, alongside the things that genuinely help discovery:

a clean robots.txt that actually lets the AI crawlers you want in,
schema.org structured data,
a real sitemap,
and, yes, a curated llms.txt on top.

Then make sure the file is actually correct — the most common failure is a perfectly-formatted llms.txt that points at 404s, which does the opposite of building trust. Run it through an llms.txt validator that fetches every linked URL, and fix whatever it flags.

The honest pitch: llms.txt won't help your SEO. It'll help the thing SEO is slowly turning into. That's a better bet anyway.

I made my website callable by another AI agent — here's the actual JSON-RPC

Andriy Pyvovarchuk — Tue, 04 Aug 2026 06:01:22 +0000

Most "AI integration" still means a human reads your docs and writes a client. A2A flips that: another agent reads a small machine description of what your service does and calls it directly — no browser, no scraping, no human in the loop. I wired it up on a small tool I run (a free llms.txt validator) and it's simpler than the acronyms suggest. Here's the whole thing, with the real requests.

What A2A actually is

A2A (Agent2Agent) is an open protocol — now under the Linux Foundation — that lets AI agents discover and call each other over plain HTTP. Think of it as the agent-to-agent counterpart of a public API: instead of publishing OpenAPI docs for humans, you publish a machine description another agent can consume and invoke.

It has exactly two moving parts:

The agent card — a discovery document.
The skill — a callable JSON-RPC endpoint.

That's it. You can ship a useful A2A surface with one skill and a single method.

Part 1 — the agent card (discovery)

You serve a JSON file at /.well-known/agent-card.json describing who you are and what you can do. The required fields are name, description, version, url (your endpoint) and at least one skill:

{
  "protocolVersion": "0.3.0",
  "name": "llms.txt Validator",
  "description": "Validate a website's llms.txt and return a score with findings.",
  "url": "https://llms-txt-validator.dev/a2a",
  "skills": [{
    "id": "validate_llms_txt",
    "name": "Validate llms.txt",
    "description": "Given a domain or URL, fetch and validate its llms.txt."
  }]
}

The card is a contract, not a meta tag. An agent fetches it, sees a validate_llms_txt skill, and knows both what you offer and where to call.

Part 2 — the endpoint (a callable skill)

The card's url points at a JSON-RPC 2.0 endpoint. An agent invokes the message/send method with a message; you do the work and return a Task. Here's a real call to my agent:

POST /a2a
{ "jsonrpc": "2.0", "id": "1", "method": "message/send",
  "params": { "message": { "role": "user",
    "parts": [{ "kind": "text", "text": "validate llmstxt.org" }] } } }

…and the reply — a completed task carrying a human-readable summary and structured data the calling agent can use directly:

{ "result": { "kind": "task", "status": { "state": "completed" },
  "artifacts": [{ "parts": [
    { "kind": "text", "text": "Validated llmstxt.org: score 100/100..." },
    { "kind": "data", "data": { "ok": true, "report": { "scores": { "overall": 100 } } } }
  ] }] } }

No HTML, no parsing, no guessing. The agent asked a question in natural language and got back exactly the data it needed. You can curl this yourself.

The minimum viable A2A

You don't need the whole spec to start. Pick one real thing your service does and:

Serve a valid agent card at /.well-known/agent-card.json with one skill.
Implement the synchronous message/send method wrapping that capability.
Return a completed Task with the result as an artifact.

Streaming, task history, and push notifications are all optional — set capabilities.streaming to false and add them only when you actually need them.

A2A vs WebMCP vs MCP (they're not competitors)

A2A — networked agents calling your service over HTTP (this article).
WebMCP — an agent running inside the browser calling tools on your open page.
MCP — connecting tools to a single model or app, usually locally.

They stack rather than compete: A2A reaches networked agents, WebMCP reaches in-browser ones, MCP wires tools into one model.

The one rule: don't publish a card you can't back

It's tempting to drop an agent card to look modern and leave the endpoint returning 501. Don't. An agent that fetches your card and calls a dead url trusts you less afterward — you've spent its call for nothing. Every skill you advertise should resolve to real, working behavior. (That's why the validator I built reports an A2A signal as "present" only when a live endpoint actually answers message/send.)

If you ship one, drop your agent card in the comments — I'd like to call it.

7 llms.txt mistakes I see constantly (and the 30-second fix for each)

Andriy Pyvovarchuk — Sun, 26 Jul 2026 06:57:50 +0000

llms.txt is deceptively simple — it's just Markdown — so most people write one in five minutes and move on. The problem: a file can look perfectly valid and still be broken in ways that quietly waste an AI crawler's time or get it ignored. Here are the seven mistakes I run into most, and the quick fix for each.

1. More than one H1

The spec wants exactly one top-level # Title. Extra # headings scramble the structure a model relies on. Fix: keep a single #, demote every other heading to ##.

2. No blockquote summary

The one- or two-sentence > blockquote right under the title is the first context a model reads. Skipping it is the single most common miss. Fix: add one line.

# Acme

> Acme turns CSV files into charts. These links cover setup, the API, and pricing.

3. Sections with no links (or no sections at all)

Empty ## sections add nothing; a wall of links with no structure is just as bad. Fix: group your key pages under real headings (## Docs, ## Guides) with a Markdown bullet list of links under each.

4. Broken links

This is the big one. A file can be flawless Markdown and still point at 404s — which burns crawl budget and erodes trust. Most linters check syntax and never actually fetch the URLs, so broken links sail straight through. Fix: verify every link resolves. (This is the check I built my validator around — it fetches each one.)

5. Relative or `http://` links

Models may not resolve relative paths like /docs/start, and plain http:// links look untrustworthy and can break under HTTPS-only crawling. Fix: use absolute https:// URLs everywhere.

6. The wrong content type

Serve the file as text/plain; charset=utf-8. A surprising number of servers return text/html for it (a default misconfiguration), and some consumers then won't treat it as llms.txt at all. Fix: set the content type explicitly.

7. A bloated, uncurated dump

llms.txt is a curated index, not a sitemap. Hundreds of undescribed links bury the signal you're trying to send. Fix: keep the best pages, add a short : description after each, and push secondary links to an ## Optional section.

Catch all seven at once

You don't have to eyeball these by hand. Paste your domain into an llms.txt validator — it scores structure, fetches every linked URL, and tells you exactly which of the above to fix, in plain language.

A clean report: 100/100 with every link fetched. Broken links drag the score down even when the Markdown is perfect — which is exactly the failure mode syntax-only checkers miss.

Fix these seven and you're ahead of most llms.txt files on the web today. The bar is genuinely low right now — which is the opportunity.

llms.txt vs robots.txt vs sitemap.xml: what actually does what

Andriy Pyvovarchuk — Sun, 26 Jul 2026 06:26:18 +0000

If you run a website in 2026 you now have three plain-text files that can live at your root and all look suspiciously similar: robots.txt, sitemap.xml, and the newcomer, llms.txt. They get confused constantly — I've seen people add llms.txt thinking it blocks AI crawlers (it doesn't) or skip it thinking robots.txt already covers it (it doesn't). Here's the clean mental model.

The one-line version

robots.txt → controls crawler access: which paths bots may or may not fetch.
sitemap.xml → lists every indexable page so search engines miss nothing.
llms.txt → curates a shortlist of high-signal pages for AI models at answer time.

Access, coverage, curation. Three different jobs.

robots.txt is about permission

robots.txt is an access-control convention: "Googlebot may crawl /blog but not /admin." It's a bouncer at the door. It doesn't describe your content, doesn't rank it, and isn't written for language models in particular. Blocking or allowing a path there says nothing about what the page is — only whether a bot may request it.

Crucially: llms.txt is not an access-control file. It can't stop an AI crawler from reading anything. If that's your goal, robots.txt (and server-side rules) is still the tool.

sitemap.xml is about coverage

A sitemap tries to be exhaustive: every URL you want indexed, so search engines don't overlook a page. It's machine-oriented XML for crawlers, not a human-readable summary, and it makes no attempt to say which pages matter most — they're all just listed.

llms.txt is about curation

llms.txt is the opposite of exhaustive. It's a hand-picked, Markdown list of the few pages that best help a model understand and use your site, each with a short description. It exists for inference — the moment a user asks ChatGPT or Claude "how do I do X with your product" — not for training datasets and not for access control.

# Acme

> Acme turns CSV files into charts. These links cover setup, the API, and pricing.

## Docs
- [Quickstart](https://example.com/quickstart): first chart in 5 minutes
- [API reference](https://example.com/api): endpoints and auth

Where a sitemap says "here is everything," llms.txt says "here is what's worth your limited context window."

You don't choose — you use all three

This is the part people miss. A strong 2026 setup has all three, each doing its job:

File	Question it answers	Audience
`robots.txt`	May you fetch this?	all crawlers
`sitemap.xml`	What should you index?	search engines
`llms.txt`	What matters most here?	AI models

robots.txt governs access, sitemap.xml ensures coverage, and llms.txt curates signal for AI. They complement each other; none replaces another.

Getting llms.txt right

The catch with llms.txt is that it's easy to write one that looks valid but points at dead links or ships as the wrong content type — and unlike the other two, almost nobody checks it. After you write yours, run it through an llms.txt validator that fetches every linked URL and confirms it resolves, then fix whatever it flags. Aim for no broken links first; structure second.

Three files, three jobs. Once that clicks, llms.txt stops being mysterious — it's just the curation layer the web didn't have until AI needed one.

I made my website callable by AI agents — the three layers (with the code I shipped)

Andriy Pyvovarchuk — Sun, 26 Jul 2026 05:26:06 +0000

"Make your site agent-ready" gets thrown around a lot, but it actually means three different things to three different kinds of AI agent — and they stack. I built a small tool (a free llms.txt validator) and wired up all three layers on it, so this is the copy-the-code version, not the hand-wavy one.

But before the three layers, let's start at the bottom — because the whole stack rests on one small file most sites still don't have.

First: what is llms.txt, and why should you care?

llms.txt is a plain-text file you put at yoursite.com/llms.txt that tells AI models what your site is about and which pages actually matter. That's it. It's written in Markdown, and it's read by AI assistants and crawlers — ChatGPT, Claude, Perplexity, and the growing crowd of AI agents — when they try to understand your site.

Here's the problem it solves. When an AI tries to "read" your website today, it gets a mess: HTML wrappers, navigation menus, cookie banners, ads, JavaScript that may not even run. It has to guess what's important and burns a big chunk of its limited context window doing it. Often it guesses wrong, or gives up.

llms.txt hands the AI a clean, curated map instead — you decide what your site is and which pages are the good ones, in priority order:

# Example — Project Management for Teams

> Example is a project-management app. This file points AI assistants
> at our most useful docs and product pages.

## Docs
- [Quickstart](https://example.com/quickstart): get set up in 5 minutes
- [API reference](https://example.com/api): every endpoint, with examples

## Product
- [Pricing](https://example.com/pricing): plans and limits

Two useful mental models:

It's the AI counterpart to sitemap.xml. A sitemap helps search engines find every page; llms.txt helps AI models understand the few pages that matter, in the format they read most cheaply.
It's the flip side of robots.txt. robots.txt says what crawlers may not touch; llms.txt says what your site is and where the value is.

Why it matters now: people increasingly get answers about your product from an AI assistant instead of clicking to your site. If the AI can't cheaply and correctly understand you, it either skips you or describes you wrong. llms.txt is your chance to feed it an accurate, curated summary — so you're represented the way you intend. It's a context layer, not a ranking trick: you're not gaming anything, you're just making your site legible to machines.

(There's an optional companion, llms-full.txt, that inlines the full text of those pages so an AI can ingest everything in one fetch. Start with llms.txt.)

That's the foundation. Now the three layers that build on it.

The tool this article is about: paste a domain, it fetches /llms.txt, checks it against the spec, and tests every link.

Here are the three independent layers. You can adopt any one on its own:

llms.txt — context for crawlers reading your site.
WebMCP — tools for an agent driving your UI inside the browser.
A2A — a callable endpoint for another agent hitting you over the network.

Crawler → browser agent → networked agent. Different agents, different mechanisms, zero overlap.

Layer 1 — llms.txt: context for crawlers

The baseline, and the highest-value one for most sites. A plain-text llms.txt at your root tells an AI crawler what your site is and which pages matter, in priority order. It's read at fetch time — no JavaScript, no protocol, no SDK. It's literally Markdown:

# Example Docs

> Developer documentation for the Example API.

## Core
- [Quickstart](https://example.com/quickstart): 5-minute setup
- [Authentication](https://example.com/auth): API keys and OAuth

## Reference
- [API reference](https://example.com/api): every endpoint

Rules that actually matter in practice:

One # H1 title, one > blockquote summary.
## sections with real Markdown links.
Absolute HTTPS URLs — relative links are the single most common failure.
Serve it as text/plain.

That's the whole layer. If you do nothing else, do this one.

Layer 2 — WebMCP: tools for in-browser agents

WebMCP lets a page expose callable tools to an AI agent running inside the browser — so instead of the agent guessing at your form fields and clicking around, it invokes a typed action you defined. There are two ways to expose a tool, and I shipped both.

Declarative — annotate a real form and the browser synthesizes a tool from it. Zero JavaScript:

<form toolname="open_llms_txt_report"
      tooldescription="Validate a site's llms.txt and open its report.">
  <input name="url" toolparamdescription="Domain or URL to validate">
</form>

Imperative — register a tool in JavaScript that returns structured data directly:

navigator.modelContext?.registerTool({
  name: "validate_llms_txt",
  description: "\"Validate a website's llms.txt; returns a 0-100 score and findings.\","
  inputSchema: {
    type: "object",
    properties: { url: { type: "string" } },
    required: ["url"]
  },
  execute: (input) =>
    fetch("/api/validate?url=" + encodeURIComponent(input.url))
      .then(r => r.json())
      .then(data => ({ content: [{ type: "text", text: JSON.stringify(data) }] }))
});

Guard the imperative call behind if (navigator.modelContext) so it's a no-op where WebMCP isn't present. Reality check: it's a Chrome origin trial today, so actual reach is tiny. Treat it as a forward signal and a clean reference, not a traffic source.

Layer 3 — A2A: a callable agent for other agents

The Agent2Agent (A2A) protocol — now under the Linux Foundation — lets other agents discover and call your service over plain HTTP. No browser, no scraping. It's the agent-to-agent counterpart of a public API.

Two parts: a discovery document (the agent card) and a callable endpoint (the skill).

The agent card — a JSON file at /.well-known/agent-card.json:

{
  "protocolVersion": "0.3.0",
  "name": "llms.txt Validator",
  "description": "Validate a website's llms.txt and return a score with findings.",
  "url": "https://llms-txt-validator.dev/a2a",
  "skills": [{
    "id": "validate_llms_txt",
    "name": "Validate llms.txt",
    "description": "Given a domain or URL, fetch and validate its llms.txt."
  }]
}

The endpoint — the card's url points at a JSON-RPC 2.0 endpoint. An agent calls message/send; you do the work and return a Task. A real call:

POST /a2a
{ "jsonrpc": "2.0", "id": "1", "method": "message/send",
  "params": { "message": { "role": "user",
    "parts": [{ "kind": "text", "text": "validate llmstxt.org" }] } } }

…and the reply — a completed task with a human-readable summary and structured data:

{ "result": { "kind": "task", "status": { "state": "completed" },
  "artifacts": [{ "parts": [
    { "kind": "text", "text": "Validated llmstxt.org: score 100/100..." },
    { "kind": "data", "data": { "ok": true, "report": { "scores": { "overall": 100 } } } }
  ] }] } }

The agent gets exactly the data it asked for. You don't need the whole protocol to start — pick one real capability, serve a card with one skill, implement the synchronous message/send, return a completed Task. Streaming, task history and push notifications are optional (capabilities.streaming: false) until you need them.

The one rule: don't publish a signal you can't back

It's tempting to drop an empty agent card or fake form annotations to make some audit go green. Don't. An agent that fetches your card and calls a dead endpoint — or invokes a tool that does nothing — trusts you less afterward. Every signal should resolve to something real. (This is why the validator I built reports a WebMCP or A2A signal as "present" only when a working implementation actually backs it.)

How to verify each layer

llms.txt — run your domain through an llms.txt validator; aim for 100/100 with no broken links.

A clean report: 100/100, with Structure / Links / Best practices scored separately. The tool fetches every linked URL, so a broken link drags the score down even when the Markdown is perfect.

WebMCP — in Chrome 149+, enable chrome://flags/#enable-webmcp-testing and run Lighthouse's Agentic Browsing audit; your tools should be listed.
A2A — fetch your /.well-known/agent-card.json, then POST a message/send call to its url and confirm you get a completed task back.

That's the whole stack: llms.txt for crawlers, WebMCP for in-browser agents, A2A for networked ones. Start with llms.txt (real value today), add WebMCP and A2A as forward signals when you have a genuine action or service to expose.

If you build one, I'd love to see it — drop your agent card in the comments.