DEV Community: Andriy Pyvovarchuk

7 llms.txt mistakes I see constantly (and the 30-second fix for each)

Andriy Pyvovarchuk — Sun, 26 Jul 2026 06:57:50 +0000

llms.txt is deceptively simple — it's just Markdown — so most people write one in five minutes and move on. The problem: a file can look perfectly valid and still be broken in ways that quietly waste an AI crawler's time or get it ignored. Here are the seven mistakes I run into most, and the quick fix for each.

1. More than one H1

The spec wants exactly one top-level # Title. Extra # headings scramble the structure a model relies on. Fix: keep a single #, demote every other heading to ##.

2. No blockquote summary

The one- or two-sentence > blockquote right under the title is the first context a model reads. Skipping it is the single most common miss. Fix: add one line.

# Acme

> Acme turns CSV files into charts. These links cover setup, the API, and pricing.

3. Sections with no links (or no sections at all)

Empty ## sections add nothing; a wall of links with no structure is just as bad. Fix: group your key pages under real headings (## Docs, ## Guides) with a Markdown bullet list of links under each.

4. Broken links

This is the big one. A file can be flawless Markdown and still point at 404s — which burns crawl budget and erodes trust. Most linters check syntax and never actually fetch the URLs, so broken links sail straight through. Fix: verify every link resolves. (This is the check I built my validator around — it fetches each one.)

5. Relative or `http://` links

Models may not resolve relative paths like /docs/start, and plain http:// links look untrustworthy and can break under HTTPS-only crawling. Fix: use absolute https:// URLs everywhere.

6. The wrong content type

Serve the file as text/plain; charset=utf-8. A surprising number of servers return text/html for it (a default misconfiguration), and some consumers then won't treat it as llms.txt at all. Fix: set the content type explicitly.

7. A bloated, uncurated dump

llms.txt is a curated index, not a sitemap. Hundreds of undescribed links bury the signal you're trying to send. Fix: keep the best pages, add a short : description after each, and push secondary links to an ## Optional section.

Catch all seven at once

You don't have to eyeball these by hand. Paste your domain into an llms.txt validator — it scores structure, fetches every linked URL, and tells you exactly which of the above to fix, in plain language.

A clean report: 100/100 with every link fetched. Broken links drag the score down even when the Markdown is perfect — which is exactly the failure mode syntax-only checkers miss.

Fix these seven and you're ahead of most llms.txt files on the web today. The bar is genuinely low right now — which is the opportunity.

llms.txt vs robots.txt vs sitemap.xml: what actually does what

Andriy Pyvovarchuk — Sun, 26 Jul 2026 06:26:18 +0000

If you run a website in 2026 you now have three plain-text files that can live at your root and all look suspiciously similar: robots.txt, sitemap.xml, and the newcomer, llms.txt. They get confused constantly — I've seen people add llms.txt thinking it blocks AI crawlers (it doesn't) or skip it thinking robots.txt already covers it (it doesn't). Here's the clean mental model.

The one-line version

robots.txt → controls crawler access: which paths bots may or may not fetch.
sitemap.xml → lists every indexable page so search engines miss nothing.
llms.txt → curates a shortlist of high-signal pages for AI models at answer time.

Access, coverage, curation. Three different jobs.

robots.txt is about permission

robots.txt is an access-control convention: "Googlebot may crawl /blog but not /admin." It's a bouncer at the door. It doesn't describe your content, doesn't rank it, and isn't written for language models in particular. Blocking or allowing a path there says nothing about what the page is — only whether a bot may request it.

Crucially: llms.txt is not an access-control file. It can't stop an AI crawler from reading anything. If that's your goal, robots.txt (and server-side rules) is still the tool.

sitemap.xml is about coverage

A sitemap tries to be exhaustive: every URL you want indexed, so search engines don't overlook a page. It's machine-oriented XML for crawlers, not a human-readable summary, and it makes no attempt to say which pages matter most — they're all just listed.

llms.txt is about curation

llms.txt is the opposite of exhaustive. It's a hand-picked, Markdown list of the few pages that best help a model understand and use your site, each with a short description. It exists for inference — the moment a user asks ChatGPT or Claude "how do I do X with your product" — not for training datasets and not for access control.

# Acme

> Acme turns CSV files into charts. These links cover setup, the API, and pricing.

## Docs
- [Quickstart](https://example.com/quickstart): first chart in 5 minutes
- [API reference](https://example.com/api): endpoints and auth

Where a sitemap says "here is everything," llms.txt says "here is what's worth your limited context window."

You don't choose — you use all three

This is the part people miss. A strong 2026 setup has all three, each doing its job:

File	Question it answers	Audience
`robots.txt`	May you fetch this?	all crawlers
`sitemap.xml`	What should you index?	search engines
`llms.txt`	What matters most here?	AI models

robots.txt governs access, sitemap.xml ensures coverage, and llms.txt curates signal for AI. They complement each other; none replaces another.

Getting llms.txt right

The catch with llms.txt is that it's easy to write one that looks valid but points at dead links or ships as the wrong content type — and unlike the other two, almost nobody checks it. After you write yours, run it through an llms.txt validator that fetches every linked URL and confirms it resolves, then fix whatever it flags. Aim for no broken links first; structure second.

Three files, three jobs. Once that clicks, llms.txt stops being mysterious — it's just the curation layer the web didn't have until AI needed one.

I made my website callable by AI agents — the three layers (with the code I shipped)

Andriy Pyvovarchuk — Sun, 26 Jul 2026 05:26:06 +0000

"Make your site agent-ready" gets thrown around a lot, but it actually means three different things to three different kinds of AI agent — and they stack. I built a small tool (a free llms.txt validator) and wired up all three layers on it, so this is the copy-the-code version, not the hand-wavy one.

But before the three layers, let's start at the bottom — because the whole stack rests on one small file most sites still don't have.

First: what is llms.txt, and why should you care?

llms.txt is a plain-text file you put at yoursite.com/llms.txt that tells AI models what your site is about and which pages actually matter. That's it. It's written in Markdown, and it's read by AI assistants and crawlers — ChatGPT, Claude, Perplexity, and the growing crowd of AI agents — when they try to understand your site.

Here's the problem it solves. When an AI tries to "read" your website today, it gets a mess: HTML wrappers, navigation menus, cookie banners, ads, JavaScript that may not even run. It has to guess what's important and burns a big chunk of its limited context window doing it. Often it guesses wrong, or gives up.

llms.txt hands the AI a clean, curated map instead — you decide what your site is and which pages are the good ones, in priority order:

# Example — Project Management for Teams

> Example is a project-management app. This file points AI assistants
> at our most useful docs and product pages.

## Docs
- [Quickstart](https://example.com/quickstart): get set up in 5 minutes
- [API reference](https://example.com/api): every endpoint, with examples

## Product
- [Pricing](https://example.com/pricing): plans and limits

Two useful mental models:

It's the AI counterpart to sitemap.xml. A sitemap helps search engines find every page; llms.txt helps AI models understand the few pages that matter, in the format they read most cheaply.
It's the flip side of robots.txt. robots.txt says what crawlers may not touch; llms.txt says what your site is and where the value is.

Why it matters now: people increasingly get answers about your product from an AI assistant instead of clicking to your site. If the AI can't cheaply and correctly understand you, it either skips you or describes you wrong. llms.txt is your chance to feed it an accurate, curated summary — so you're represented the way you intend. It's a context layer, not a ranking trick: you're not gaming anything, you're just making your site legible to machines.

(There's an optional companion, llms-full.txt, that inlines the full text of those pages so an AI can ingest everything in one fetch. Start with llms.txt.)

That's the foundation. Now the three layers that build on it.

The tool this article is about: paste a domain, it fetches /llms.txt, checks it against the spec, and tests every link.

Here are the three independent layers. You can adopt any one on its own:

llms.txt — context for crawlers reading your site.
WebMCP — tools for an agent driving your UI inside the browser.
A2A — a callable endpoint for another agent hitting you over the network.

Crawler → browser agent → networked agent. Different agents, different mechanisms, zero overlap.

Layer 1 — llms.txt: context for crawlers

The baseline, and the highest-value one for most sites. A plain-text llms.txt at your root tells an AI crawler what your site is and which pages matter, in priority order. It's read at fetch time — no JavaScript, no protocol, no SDK. It's literally Markdown:

# Example Docs

> Developer documentation for the Example API.

## Core
- [Quickstart](https://example.com/quickstart): 5-minute setup
- [Authentication](https://example.com/auth): API keys and OAuth

## Reference
- [API reference](https://example.com/api): every endpoint

Rules that actually matter in practice:

One # H1 title, one > blockquote summary.
## sections with real Markdown links.
Absolute HTTPS URLs — relative links are the single most common failure.
Serve it as text/plain.

That's the whole layer. If you do nothing else, do this one.

Layer 2 — WebMCP: tools for in-browser agents

WebMCP lets a page expose callable tools to an AI agent running inside the browser — so instead of the agent guessing at your form fields and clicking around, it invokes a typed action you defined. There are two ways to expose a tool, and I shipped both.

Declarative — annotate a real form and the browser synthesizes a tool from it. Zero JavaScript:

<form toolname="open_llms_txt_report"
      tooldescription="Validate a site's llms.txt and open its report.">
  <input name="url" toolparamdescription="Domain or URL to validate">
</form>

Imperative — register a tool in JavaScript that returns structured data directly:

navigator.modelContext?.registerTool({
  name: "validate_llms_txt",
  description: "\"Validate a website's llms.txt; returns a 0-100 score and findings.\","
  inputSchema: {
    type: "object",
    properties: { url: { type: "string" } },
    required: ["url"]
  },
  execute: (input) =>
    fetch("/api/validate?url=" + encodeURIComponent(input.url))
      .then(r => r.json())
      .then(data => ({ content: [{ type: "text", text: JSON.stringify(data) }] }))
});

Guard the imperative call behind if (navigator.modelContext) so it's a no-op where WebMCP isn't present. Reality check: it's a Chrome origin trial today, so actual reach is tiny. Treat it as a forward signal and a clean reference, not a traffic source.

Layer 3 — A2A: a callable agent for other agents

The Agent2Agent (A2A) protocol — now under the Linux Foundation — lets other agents discover and call your service over plain HTTP. No browser, no scraping. It's the agent-to-agent counterpart of a public API.

Two parts: a discovery document (the agent card) and a callable endpoint (the skill).

The agent card — a JSON file at /.well-known/agent-card.json:

{
  "protocolVersion": "0.3.0",
  "name": "llms.txt Validator",
  "description": "Validate a website's llms.txt and return a score with findings.",
  "url": "https://llms-txt-validator.dev/a2a",
  "skills": [{
    "id": "validate_llms_txt",
    "name": "Validate llms.txt",
    "description": "Given a domain or URL, fetch and validate its llms.txt."
  }]
}

The endpoint — the card's url points at a JSON-RPC 2.0 endpoint. An agent calls message/send; you do the work and return a Task. A real call:

POST /a2a
{ "jsonrpc": "2.0", "id": "1", "method": "message/send",
  "params": { "message": { "role": "user",
    "parts": [{ "kind": "text", "text": "validate llmstxt.org" }] } } }

…and the reply — a completed task with a human-readable summary and structured data:

{ "result": { "kind": "task", "status": { "state": "completed" },
  "artifacts": [{ "parts": [
    { "kind": "text", "text": "Validated llmstxt.org: score 100/100..." },
    { "kind": "data", "data": { "ok": true, "report": { "scores": { "overall": 100 } } } }
  ] }] } }

The agent gets exactly the data it asked for. You don't need the whole protocol to start — pick one real capability, serve a card with one skill, implement the synchronous message/send, return a completed Task. Streaming, task history and push notifications are optional (capabilities.streaming: false) until you need them.

The one rule: don't publish a signal you can't back

It's tempting to drop an empty agent card or fake form annotations to make some audit go green. Don't. An agent that fetches your card and calls a dead endpoint — or invokes a tool that does nothing — trusts you less afterward. Every signal should resolve to something real. (This is why the validator I built reports a WebMCP or A2A signal as "present" only when a working implementation actually backs it.)

How to verify each layer

llms.txt — run your domain through an llms.txt validator; aim for 100/100 with no broken links.

A clean report: 100/100, with Structure / Links / Best practices scored separately. The tool fetches every linked URL, so a broken link drags the score down even when the Markdown is perfect.

WebMCP — in Chrome 149+, enable chrome://flags/#enable-webmcp-testing and run Lighthouse's Agentic Browsing audit; your tools should be listed.
A2A — fetch your /.well-known/agent-card.json, then POST a message/send call to its url and confirm you get a completed task back.

That's the whole stack: llms.txt for crawlers, WebMCP for in-browser agents, A2A for networked ones. Start with llms.txt (real value today), add WebMCP and A2A as forward signals when you have a genuine action or service to expose.

If you build one, I'd love to see it — drop your agent card in the comments.