If you’ve spent any time poking around the AIE World’s Fair 2026 website, you may have come across the llms.md page. If you’ve clicked the link, you may have an idea of the page’s purpose already. There is a distinct shift from energetic copy text, neat layouts, and advanced styling to focused, accurate details and well-labeled links. This whole page, while being a good resource for you, is not designed for you. It’s designed for AI.
It follows a standard proposal called the “/llms.txt file,” which you can read about at llmstxt.org. In fact, even though the AIEWF 2026 site has its page at llms.md, if you visit llms.txt instead, it handles it smoothly, redirecting to the llms.md page. The idea is that, rather than filling up an AI’s context window with footers, navs, sidebars, and styles, it is a focused and simple entry point for AIs visiting a site. The entry llms.txt (or .md) contains the minimum necessary information to use the site as well as clear links pointing to where the AI can get more information specifically about the topics it needs. A common choice is to link to “/llms-full.txt,” which has much more detail (which the AIEWF 2026 site does). Another recommendation from the standard itself is to make all content pages mirror a simple text version with an appended “.md” suffix for the same reasons (e.g., “posts.html” becomes “posts.html.md”).
The recommended layout looks something like this: one H1 header with the overall site/project title, a blockquote with a short summary of the most important information, zero or more markdown blocks with more details, followed by zero or more markdown sections with H2 headers containing lists of URLs pointing to further detail.
To be clear, these minimal, AI-first text files are simply a proposal, not a standard, and there is mixed adoption. Most chat-based tools have not committed to looking for llms.txt by default. Some people claim that they are “a solution in search of a problem.” This is reasonable, as the primary selling point is reduction of context clutter, and context efficiency is currently a popular research topic. (Look at how many talks at AIE 2026 are about context!) However, these files are relatively painless to autogenerate — although most aren’t quite as pretty as ai.engineer’s — and coding tools like Cursor and others actually do claim to reference them, especially when looking up library documentation. Their use does seem to be trending upward. Google even announced it as a new Lighthouse signal under the new Agentic Browsing category in May of this year, and it seems likely more tooling and standardization are coming in this area.
Every source on the subject, however, seems to agree on one thing that is the most important: llms.txt files are not replacements for current standards like robots.txt or sitemap.xml files. They are best when used together. A quick conversation with Gemini revealed that it was able to make use of the AIEWF 2026 llms.md file because it was already indexed by Google and showed up specifically in the search results. So, actions like listing it in your sitemap, ensuring crawlers can see it, and linking to it from landing pages will go a long way toward helping the AIs that use your site.
Top comments (9)
started noticing this when my agents pull docs and burn context on nav elements and cookie banners. llms.txt is the right call - curious if sites actually adopt it or it stays an enthusiast thing.
Good balanced take, and the "complement, not replace" landing is the right one. The part I'd add from the consumer side, since I build the scrapers and agents that are supposed to read these: almost nothing looks for llms.txt today. When I point an agent at a site I'm hitting rendered HTML or an API, not sniffing for a curated entry file, and that's true of most frameworks by default. So right now llms.txt is mostly producer-side optimism, you can publish a beautiful one and still have every agent walk straight past it. The Cursor-for-library-docs case works precisely because that's a closed loop where the consumer opted in, which is the exception, not the norm yet.
The other thing that nags me is drift, and it's why I'd be careful ingesting llms-full.txt into anything downstream. It's a second copy of your content with nothing enforcing that it still matches the HTML, so the "clean source for AI" quietly becomes a stale source of truth the moment someone edits a page and forgets the mirror. Hand-maintained, it will drift, guaranteed. The only version I'd trust is one generated from the canonical content at build time, which is really the same discipline as your discoverability point, it only works when it's wired into the pipeline instead of bolted on. Autogenerate it from the source and list it in the sitemap and it's genuinely useful. Maintain it by hand and it's a liability wearing a helpful hat.
What interests me about initiatives like
llms.txtisn't the file format itself so much as what it represents.For decades, we optimized content for search engines whose primary job was ranking documents. Increasingly, we're optimizing content for systems whose primary job is synthesizing answers.
That's a subtle but important shift.
The question stops being:
And starts becoming:
In that sense,
llms.txtfeels less like an SEO tool and more like an attempt to improve machine-readable context. Not unlike schema markup, except aimed at helping answer engines navigate information rather than helping search engines rank it.The bigger trend I see is that websites are gradually becoming sources rather than destinations. Traffic still matters, but increasingly the challenge is ensuring that concepts survive retrieval and synthesis while retaining their meaning.
Whether
llms.txtbecomes the standard or not, I suspect we'll see more efforts in this direction as organizations realize that discoverability in the age of answer engines is increasingly about semantic clarity rather than keyword visibility.I’m really hoping that there’s a shift into tools surfacing their sources, being able to attach a link to a statement of fact so that the user can investigate a claim further. But I agree. The LLMs.txt is more of an indicator right now of people’s desire to mold how they show up to an agent and how an agent consumes their content. I’m not sure it fully solves the problem though.
I keep snagging on the maintenance side of this. An llms.txt written by hand, and worse a parallel .md for every page, is another copy of your content that can quietly drift out of sync with the real thing, and a confident but stale summary an agent trusts is arguably worse than making it read the messy HTML. Your point that Gemini only used the file because Google had already indexed it kind of gives the game away, doesn't it? It makes me wonder if the durable win is just "ship clean markdown and a good sitemap," with llms.txt riding along rather than doing the heavy lifting. Have you seen a tool actually fetch it on its own, without the search index sitting in the middle?
The llms.txt approach is elegant because it solves the wrong-abstraction problem. Agents don't need your CSS, your nav, or your footer â they need a machine-readable index of what's actually on the site. The redirect from llms.txt to llms.md is a nice touch that makes discovery path-independent.
What I find interesting is the storage implication. When an agent scrapes llms.txt, where does it keep that knowledge across sessions? Right now most agent frameworks either re-scrape every time or cache in memory, which means restart = amnesia. For agents that maintain long-running context â think a personal assistant that remembers which docs it's already indexed â you need a persistent store that lives alongside the agent, not behind a server API.
This is the kind of scenario we designed moteDB for: an embedded database that stores structured metadata (page indexes, fetch timestamps, content hashes) and vector embeddings for semantic retrieval, all in a single Rust binary you embed directly. No Redis, no Postgres, no external service to manage. The agent's knowledge graph lives where the agent lives.
Has anyone in the llms.txt community started thinking about the "agent state" layer â how agents should persist and share what they've learned from these files across sessions?
Interesting take on agent-driven LLM optimization—have you considered how confidential computing could enhance trust in agent interactions, especially when dealing with sensitive data? At work, we're exploring how secure enclaves can isolate agent logic without sacrificing performance, and it’s fascinating to see how that might align with projects like llms.txt.
The drift point above gets worse the second you're multilingual. I build FR/EN/KA sites, and a hand-kept llms.txt means the same content can now drift in three languages instead of one, plus you have to decide which language version an agent lands on when there's no hreflang equivalent for a plain text file. The only version that holds up is generated per-locale at build time from the same source as the pages. Has anyone landed on a clean convention for pointing llms.txt at language variants yet?
The interesting part of llms.txt is not just discoverability, it is intent. A good file should tell agents what is stable, what is canonical, and what should not be guessed from a random page scrape.