DEV Community

Cover image for I made my website callable by AI agents — the three layers (with the code I shipped)
Andriy Pyvovarchuk
Andriy Pyvovarchuk

Posted on • Originally published at llms-txt-validator.dev

I made my website callable by AI agents — the three layers (with the code I shipped)

#ai #webdev #javascript #tutorial

"Make your site agent-ready" gets thrown around a lot, but it actually means three different things to three different kinds of AI agent — and they stack. I built a small tool (a free llms.txt validator) and wired up all three layers on it, so this is the copy-the-code version, not the hand-wavy one.

But before the three layers, let's start at the bottom — because the whole stack rests on one small file most sites still don't have.

First: what is llms.txt, and why should you care?

llms.txt is a plain-text file you put at yoursite.com/llms.txt that tells AI models what your site is about and which pages actually matter. That's it. It's written in Markdown, and it's read by AI assistants and crawlers — ChatGPT, Claude, Perplexity, and the growing crowd of AI agents — when they try to understand your site.

Here's the problem it solves. When an AI tries to "read" your website today, it gets a mess: HTML wrappers, navigation menus, cookie banners, ads, JavaScript that may not even run. It has to guess what's important and burns a big chunk of its limited context window doing it. Often it guesses wrong, or gives up.

llms.txt hands the AI a clean, curated map instead — you decide what your site is and which pages are the good ones, in priority order:

# Example — Project Management for Teams

> Example is a project-management app. This file points AI assistants
> at our most useful docs and product pages.

## Docs
- [Quickstart](https://example.com/quickstart): get set up in 5 minutes
- [API reference](https://example.com/api): every endpoint, with examples

## Product
- [Pricing](https://example.com/pricing): plans and limits
Enter fullscreen mode Exit fullscreen mode

Two useful mental models:

  • It's the AI counterpart to sitemap.xml. A sitemap helps search engines find every page; llms.txt helps AI models understand the few pages that matter, in the format they read most cheaply.
  • It's the flip side of robots.txt. robots.txt says what crawlers may not touch; llms.txt says what your site is and where the value is.

Why it matters now: people increasingly get answers about your product from an AI assistant instead of clicking to your site. If the AI can't cheaply and correctly understand you, it either skips you or describes you wrong. llms.txt is your chance to feed it an accurate, curated summary — so you're represented the way you intend. It's a context layer, not a ranking trick: you're not gaming anything, you're just making your site legible to machines.

(There's an optional companion, llms-full.txt, that inlines the full text of those pages so an AI can ingest everything in one fetch. Start with llms.txt.)

That's the foundation. Now the three layers that build on it.

The llms.txt validator homepage — paste a domain, get a score in seconds
The tool this article is about: paste a domain, it fetches /llms.txt, checks it against the spec, and tests every link.

Here are the three independent layers. You can adopt any one on its own:

  1. llms.txt — context for crawlers reading your site.
  2. WebMCP — tools for an agent driving your UI inside the browser.
  3. A2A — a callable endpoint for another agent hitting you over the network.

Crawler → browser agent → networked agent. Different agents, different mechanisms, zero overlap.

Layer 1 — llms.txt: context for crawlers

The baseline, and the highest-value one for most sites. A plain-text llms.txt at your root tells an AI crawler what your site is and which pages matter, in priority order. It's read at fetch time — no JavaScript, no protocol, no SDK. It's literally Markdown:

# Example Docs

> Developer documentation for the Example API.

## Core
- [Quickstart](https://example.com/quickstart): 5-minute setup
- [Authentication](https://example.com/auth): API keys and OAuth

## Reference
- [API reference](https://example.com/api): every endpoint
Enter fullscreen mode Exit fullscreen mode

Rules that actually matter in practice:

  • One # H1 title, one > blockquote summary.
  • ## sections with real Markdown links.
  • Absolute HTTPS URLs — relative links are the single most common failure.
  • Serve it as text/plain.

That's the whole layer. If you do nothing else, do this one.

Layer 2 — WebMCP: tools for in-browser agents

WebMCP lets a page expose callable tools to an AI agent running inside the browser — so instead of the agent guessing at your form fields and clicking around, it invokes a typed action you defined. There are two ways to expose a tool, and I shipped both.

Declarative — annotate a real form and the browser synthesizes a tool from it. Zero JavaScript:

<form toolname="open_llms_txt_report"
      tooldescription="Validate a site's llms.txt and open its report.">
  <input name="url" toolparamdescription="Domain or URL to validate">
</form>
Enter fullscreen mode Exit fullscreen mode

Imperative — register a tool in JavaScript that returns structured data directly:

navigator.modelContext?.registerTool({
  name: "validate_llms_txt",
  description: "\"Validate a website's llms.txt; returns a 0-100 score and findings.\","
  inputSchema: {
    type: "object",
    properties: { url: { type: "string" } },
    required: ["url"]
  },
  execute: (input) =>
    fetch("/api/validate?url=" + encodeURIComponent(input.url))
      .then(r => r.json())
      .then(data => ({ content: [{ type: "text", text: JSON.stringify(data) }] }))
});
Enter fullscreen mode Exit fullscreen mode

Guard the imperative call behind if (navigator.modelContext) so it's a no-op where WebMCP isn't present. Reality check: it's a Chrome origin trial today, so actual reach is tiny. Treat it as a forward signal and a clean reference, not a traffic source.

Layer 3 — A2A: a callable agent for other agents

The Agent2Agent (A2A) protocol — now under the Linux Foundation — lets other agents discover and call your service over plain HTTP. No browser, no scraping. It's the agent-to-agent counterpart of a public API.

Two parts: a discovery document (the agent card) and a callable endpoint (the skill).

The agent card — a JSON file at /.well-known/agent-card.json:

{
  "protocolVersion": "0.3.0",
  "name": "llms.txt Validator",
  "description": "Validate a website's llms.txt and return a score with findings.",
  "url": "https://llms-txt-validator.dev/a2a",
  "skills": [{
    "id": "validate_llms_txt",
    "name": "Validate llms.txt",
    "description": "Given a domain or URL, fetch and validate its llms.txt."
  }]
}
Enter fullscreen mode Exit fullscreen mode

The endpoint — the card's url points at a JSON-RPC 2.0 endpoint. An agent calls message/send; you do the work and return a Task. A real call:

POST /a2a
{ "jsonrpc": "2.0", "id": "1", "method": "message/send",
  "params": { "message": { "role": "user",
    "parts": [{ "kind": "text", "text": "validate llmstxt.org" }] } } }
Enter fullscreen mode Exit fullscreen mode

…and the reply — a completed task with a human-readable summary and structured data:

{ "result": { "kind": "task", "status": { "state": "completed" },
  "artifacts": [{ "parts": [
    { "kind": "text", "text": "Validated llmstxt.org: score 100/100..." },
    { "kind": "data", "data": { "ok": true, "report": { "scores": { "overall": 100 } } } }
  ] }] } }
Enter fullscreen mode Exit fullscreen mode

The agent gets exactly the data it asked for. You don't need the whole protocol to start — pick one real capability, serve a card with one skill, implement the synchronous message/send, return a completed Task. Streaming, task history and push notifications are optional (capabilities.streaming: false) until you need them.

The one rule: don't publish a signal you can't back

It's tempting to drop an empty agent card or fake form annotations to make some audit go green. Don't. An agent that fetches your card and calls a dead endpoint — or invokes a tool that does nothing — trusts you less afterward. Every signal should resolve to something real. (This is why the validator I built reports a WebMCP or A2A signal as "present" only when a working implementation actually backs it.)

How to verify each layer

  • llms.txt — run your domain through an llms.txt validator; aim for 100/100 with no broken links.

A 100/100 llms.txt report broken down by Structure, Links and Best practices
A clean report: 100/100, with Structure / Links / Best practices scored separately. The tool fetches every linked URL, so a broken link drags the score down even when the Markdown is perfect.

  • WebMCP — in Chrome 149+, enable chrome://flags/#enable-webmcp-testing and run Lighthouse's Agentic Browsing audit; your tools should be listed.
  • A2A — fetch your /.well-known/agent-card.json, then POST a message/send call to its url and confirm you get a completed task back.

That's the whole stack: llms.txt for crawlers, WebMCP for in-browser agents, A2A for networked ones. Start with llms.txt (real value today), add WebMCP and A2A as forward signals when you have a genuine action or service to expose.

If you build one, I'd love to see it — drop your agent card in the comments.

Top comments (0)