llms.txt is a plain Markdown file, usually published at the root of a website, that tells AI agents what the site is and which pages to read first; it blocks nothing, and in the sources Canlah AI reviewed on September 23, 2026, a statement by any major answer engine that it uses the file to choose citations was not found.
The takeaway
robots.txt decides which crawlers may enter your site; llms.txt hands an AI agent a short, curated map once it is inside. The two are complementary, not substitutes. llms.txt is cheap to publish and useful to coding agents and browser agents, but the public evidence reviewed on September 23, 2026 shows no measurable link between publishing one and being cited more often in AI answers.
Disclosure: Canlah AI publishes this guide, operates canlah.ai/llms.txt and sells the SEO + GEO work described below. Third-party studies and vendor positions were reviewed on September 23, 2026 from their public pages; Canlah AI figures are first-party and directional, not an independent benchmark.
You published llms.txt, so why hasn’t AI started citing you?
Publishing llms.txt does not make ChatGPT, Perplexity or Gemini recommend a site, because the file tells an agent where to read, not which brand to name. Many teams expect the opposite, and treat the file going live as the moment AI answers should start improving.
A site can serve a tidy llms.txt, rank well on Google and pass every technical check, and still be absent when a prospect puts buyer questions like these to ChatGPT, Perplexity, Gemini or Google AI Mode:
- “Which GEO agencies in Singapore are worth shortlisting?”
- “Is llms.txt enough to get my brand cited?”
- “What should a B2B SaaS team fix first for AI search?”
- “Who measures AI visibility with evidence?”
Canlah AI measured this on its own site. canlah.ai has served an llms.txt since at least August 2026, and on September 7, 2026 it still scored 4 out of 100 on the GEO dimension of Canlah AI’s own audit, with the brand named in 7 of 177 grounded answers to open buyer questions.
robots.txt determines whether AI crawlers may read your pages; llms.txt determines how easily an agent finds the right ones; neither determines whether AI recommends you.
The file is not useless; it answers a different question from the one most buyers are asking.
llms.txt in 30 seconds
llms.txt is a reading guide written for language models. It sits beside robots.txt and sitemap.xml, but it does a narrower job than either: it summarises the site in a few lines and links to the pages an agent should fetch when it needs detail.
Put simply:
- robots.txt tells crawlers where they may and may not go;
- sitemap.xml lists every indexable URL for search engines;
- llms.txt gives an agent a short, human-written summary plus a curated list of links;
- llms-full.txt puts the full text of those pages into one file an agent can load in one request;
- none of the four decides whether an AI engine cites or recommends your brand.
The four root files compared on format, reader, blocking power and evidence of effect on AI citations, reviewed September 23, 2026
| File | What it is | Main reader | Can it block access? | Google position for AI features | Evidence it raises AI citations | What to verify on your own site |
|---|---|---|---|---|---|---|
| robots.txt | Crawl-access rules as plain-text directives (RFC 9309) | Search and AI crawlers at crawl time | Yes, for compliant crawlers | Access must be allowed for AI Overviews and AI Mode | Indirect: blocking a crawler removes you from its answers | That GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot are not disallowed by accident |
| sitemap.xml | Complete XML inventory of indexable URLs | Search engine crawlers | No | Optional | Not found in the sources reviewed | That every listed URL is live, canonical and indexable |
| llms.txt | Curated Markdown map: H1, blockquote, H2 link lists | Agents and AI tools at the moment of a task | No | Not needed; Google’s AI optimization guide says it can be ignored | Not found in the sources reviewed | That it returns 200 as text and its summary matches the About page and schema |
| llms-full.txt | Full text of the key pages in one Markdown file, often megabytes | Coding agents, IDE assistants, retrieval pipelines | No | Not needed | Not found in the sources reviewed | That it is generated on every deploy rather than written by hand |
“Not found in the sources reviewed” means the public studies and vendor documentation listed in Method and sources did not report that effect when they were checked on September 23, 2026. It is not proof that the effect is absent. Provenance: llmstxt.org v2, Google Search Central, IETF RFC 9309, Chrome Lighthouse 13.3 documentation and the seven studies in the evidence table below.
In one line:
robots.txt controls access; llms.txt controls orientation; citations are decided by the evidence the engine finds.
Why llms.txt became a standard talking point in 2026
llms.txt became a standard talking point in 2026 because the reader of a commercial web page is increasingly a model rather than a person. Buyers put a full question to an assistant instead of scanning a results page, and agents fetch pages on a buyer’s behalf to check integrations, prices, framework compatibility and release notes.
For those readers, HTML built for people wraps its content in navigation, scripts and layout, and converting it back into clean text wastes tokens. That is the genuine problem the file addresses, and it is a real one for documentation-heavy sites.
It is also why the file became a sales line. Agencies and plugin vendors found a deliverable that is cheap to produce and hard for a client to evaluate, and SEO tools started flagging its absence. Google’s Search Central team, in its May 2026 guide to optimizing for generative AI features, tells site owners they can ignore “creating unnecessary AI text files (like llms.txt)”.
Being easy for an agent to read is not the same as being chosen by an engine to cite.
What Is llms.txt?
llms.txt is a proposed convention, not a ratified standard. Jeremy Howard of Answer.AI published the proposal at llmstxt.org on September 3, 2024; version 2 of the page was last modified on August 10, 2026. It proposes a Markdown file at /llms.txt, or at any subpath such as /docs/llms.txt, that gives agents “brief background information, guidance, and links to detailed markdown files”.
The format is deliberately simple, in a fixed order:
- an H1 with the name of the site or project, which is the only required element;
- a blockquote with a short summary that an agent needs in order to read the rest;
- optional paragraphs or lists with more detail, but no headings;
- H2 sections containing “file lists”, each item a Markdown link with an optional note after a colon;
- an H2 named Optional for secondary links an agent can skip when context is short.
The proposal also recommends serving a clean Markdown copy of important pages at the same URL with .md appended, and linking pages to their llms.txt with standard link relations.
The spec is explicit about its intended use. The llmstxt.org page says the file is used “on demand, when an agent needs information about a topic while assisting a user”, and that it was expected to matter mainly for inference rather than training. It’s used most heavily for software documentation, where coding agents follow it to API references.
Adoption is real, if concentrated. Anthropic, OpenAI, Stripe, Cloudflare and Google’s ai.google.dev all served llms.txt files for their developer documentation when checked on September 23, 2026. Mintlify, GitBook, Yoast SEO, AIOSEO and Wix generate the file automatically, according to the integrations listed on llmstxt.org.
llms.txt is a reading aid for agents, not a permission system and not a ranking signal.
What Is llms-full.txt?
llms-full.txt is a companion file that isn’t in the core llms.txt spec but spread through documentation platforms. Mintlify announced on November 20, 2024 that it automatically generates and hosts both /llms.txt and /llms-full.txt for every docs site it hosts. Where llms.txt is an index of links, llms-full.txt concatenates the full text of those pages into a single Markdown file.
The trade-off is size. Vercel’s /docs/llms-full.txt returned about 9.5 MB on September 23, 2026, far larger than most context windows can hold in one pass. One Singapore GEO agency served an 827 KB llms-full.txt containing every article it publishes, checked on September 22, 2026.
The use case is narrow but genuine: a developer pastes one URL into an assistant or IDE and the whole corpus loads without crawling. For a marketing site, llms-full.txt mainly guarantees that an agent reading your content sees the full article body, with dates and summaries attached, rather than a truncated page.
Canlah AI builds its own llms-full.txt from every blog post in English, Simplified Chinese and Traditional Chinese, each preceded by URL, language, topics, type, published date, last-updated date and summary. When this page was reviewed on September 23, 2026, the live canlah.ai/llms-full.txt still returned a 404, and the generator that fixes it shipped with the September 23, 2026 update of this page. That is the kind of gap a file audit catches and a strategy deck does not.
The Core Differences Between llms.txt and robots.txt
The core differences between llms.txt and robots.txt on purpose, format, standard status, reader, timing, blocking power and Google’s position, reviewed September 23, 2026
| Dimension | robots.txt | llms.txt | What the difference means for you |
|---|---|---|---|
| Purpose | States which automated clients may access which paths | Summarises the site and points agents to the pages worth reading | Fix access first; a reading list helps nobody who was refused entry |
| Format | User-agent groups with Allow and Disallow lines | Markdown with H1, blockquote and H2 link lists | A Disallow line inside llms.txt has no effect anywhere |
| Standard status | IETF RFC 9309, published September 2022 | Community proposal at llmstxt.org, v2 modified August 10, 2026 | robots.txt behaviour is predictable; llms.txt support varies by agent and can change |
| When it is read | Before a crawl, by crawlers such as Googlebot, GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot | On demand, when an agent is assisting a user with a task | A stale llms.txt is read at the worst moment, mid-task |
| Can it block AI crawlers? | Yes, for crawlers that honour it | No; it has no directive syntax | Opting out of AI training is a robots.txt job, never an llms.txt one |
| Google Search position | Respected; blocking Googlebot removes pages from AI Overviews and AI Mode | Not used for Search; Lighthouse 13.3 checks for it in an experimental Agentic Browsing audit | Publish llms.txt for agent readiness, not in the expectation of Google rankings |
| Typical mistake | Accidentally blocking the answer-engine crawler you want citations from | Filling it with robots-style directives; an agent that parses those lines was not found in the sources reviewed | Audit both files in the same pass, on a fixed date |
Provenance: IETF RFC 9309, the llmstxt.org v2 page modified August 10, 2026, Google Search Central’s AI optimization guide and the Chrome Lighthouse 13.3 documentation, all reviewed on September 23, 2026.
In one sentence:
robots.txt manages the “door.” llms.txt manages the “reading list.”
robots.txt is like the sign at a library entrance saying which rooms are open; llms.txt is like the librarian’s one-page guide to the ten shelves worth your time. The first determines whether a reader gets in at all; the second determines how fast a reader who is already inside finds what matters.
That distinction is where the earlier version of this page went wrong. It described llms.txt with directives such as User-agent, Disallow, Attribution and Commercial-Use. Those lines belong to robots.txt or to nobody: an agent that reads an attribution or licensing directive from llms.txt was not found in the sources reviewed, and the llmstxt.org format has no directive syntax at all.
If your goal is to stop a model from training on your content, the tool is robots.txt with named user agents such as GPTBot, Google-Extended, CCBot and anthropic-ai. If your goal is to help an agent that’s already allowed in, the tool is llms.txt.
If Google Ignores llms.txt, Why Does Anyone Still Publish One?
Google’s own products disagree about llms.txt, which is the main reason teams keep publishing it after the Search team dismissed it. Search Engine Journal reported on May 20, 2026 that the Search team’s new guide lists llms.txt among tactics to skip, while Chrome’s Lighthouse 13.3 added an llms.txt check to its experimental Agentic Browsing category. The Lighthouse documentation says that without the file, “agents may spend more time crawling the site to understand its high-level structure and primary content.”
Across Canlah AI’s own site and the sites it reviews, three patterns explain why the file keeps getting published.
Pattern 1: Coding agents and IDE assistants read it routinely
Developer documentation is where llms.txt earns its keep. A coding agent asked to call an API can fetch docs.stripe.com/llms.txt or docs.anthropic.com/llms.txt, find the right reference page and load its Markdown version instead of scraping HTML.
This usually isn’t a search visibility question. It’s a developer experience question, and for API-first companies it’s a sensible one.
Pattern 2: Browser agents and audits reward a clean map
Lighthouse 13.3 reports whether a site provides llms.txt and flags server errors when fetching it. As browser agents complete tasks on a user’s behalf, a short map of the site’s structure reduces wasted crawling.
This is readiness for agents that act, not ranking in engines that answer.
Pattern 3: Platforms publish it by default
Mintlify, GitBook, Wix, Yoast SEO and AIOSEO generate the file automatically, so many sites have one without anyone deciding to. In Canlah AI’s August 2026 agent-readiness dataset, 26 of 47 reachable cross-border DTC storefronts served /llms.txt, and 23 of those 26 also carried a Shopify-issued commerce manifest.
That pattern suggests platform defaults rather than 26 separate strategic decisions.
The Most Common llms.txt Problem in Real Projects: Treating a Map as a Ranking Signal
The observations below come from Canlah AI’s audit of its own site and from Canlah AI’s open agent-readiness dataset. No client data is used; canlah.ai is the only property named, and the storefront dataset is public under CC BY 4.0.
1. The file was in place, and open-question visibility was still near zero
canlah.ai publishes a 113-line, 15,953-byte llms.txt with eight H2 sections, from Key pages and Services through Open data, the whitepaper index and Optional.
On September 7, 2026, Canlah AI probed canlah.ai with 39 buyer questions, three runs each, on ChatGPT and Gemini. The site scored 85 out of 100 on the SEO dimension and 4 out of 100 on the GEO dimension. It was named in 39 of 39 grounded answers to branded questions and 7 of 177 grounded answers to open questions, a 4.0% non-brand mention rate.
This shows:
An agent-friendly map doesn’t make an engine recommend you on questions where the buyer hasn’t named you.
The engines cited 381 distinct third-party domains instead, mostly listicles and competitor pages. That’s where the gap was; the llms.txt couldn’t close it.
2. The llms.txt summary disagreed with the site’s own entity description
The blockquote in canlah.ai/llms.txt describes Canlah AI as operating inside “ChatGPT, Perplexity, Gemini and Google AI Overviews” and probing “three engines”. Canlah AI’s organisation description elsewhere names eight engines. Both were accurate when written.
This kind of drift matters more than the file’s existence. Engines assemble a brand’s description from many sources, and an llms.txt that contradicts the About page, the schema and third-party profiles adds noise rather than clarity.
llms.txt is only as useful as its consistency with everything else you publish.
3. The file you think is live might not be
Canlah AI’s llms-full.txt route returned a 404 on canlah.ai on September 23, 2026. Nobody had noticed, because nothing measured it.
A monthly llms.txt check confirms five things:
- that /llms.txt returns 200 with a text content type rather than an HTML error page;
- that its summary matches the entity description on the site and in the Organization schema;
- that its links resolve and point to Markdown or clean pages;
- that it was updated the last time pricing, services or key pages changed;
- that robots.txt allows the answer-engine crawlers you want citations from.
That is the difference between publishing a file and managing a signal: the file is a deliverable; the recurring check is the work.
Four common problems with how sites handle llms.txt
1. Robots-style directives in a Markdown file
Many llms.txt files, including earlier guidance on this page, contain Disallow, Crawl-delay or Attribution lines. Documentation of any agent honouring those lines was not found in the sources reviewed. If a directive matters, it belongs in robots.txt, where compliant crawlers read it.
2. A link dump instead of a curated map
Some files list hundreds of URLs, effectively a sitemap in Markdown. The spec intends a short file that fits in context; if everything is listed, nothing is prioritised.
3. Stale facts that outlive the page they came from
An llms.txt written once and never revisited quietly becomes the most outdated description of your company that an agent can find. canlah.ai showed this drift: its llms.txt blockquote named three probed engines on September 23, 2026 while the site’s organisation description named eight. Generate the summary from the same source as the About page, and re-read the file whenever pricing or positioning changes.
4. Paying for it as a citation lever
Canlah AI’s vendor evidence checklist flags llms.txt sold as a reason you’ll be cited as a proposal warning sign. When a proposal leads with it, ask which measurement data the recommendation is based on.
What the evidence says about llms.txt and AI citations
The question buyers care about is whether the file changes AI answers. Five public sources and two Canlah AI measurements address it, and none shows a positive effect on citations.
Public evidence on llms.txt adoption, crawler requests and AI citations, reviewed September 23, 2026
| Source | Date | What was measured | Finding | Caveat that limits the finding |
|---|---|---|---|---|
| SE Ranking | November 7, 2025 | About 300,000 domains, citation frequency in sampled LLM answers | 10.13% had llms.txt; no correlation with AI citations in Spearman, XGBoost or SHAP analysis | Correlational, and SE Ranking sells SEO software; no correlation in a sample is not proof of no effect |
| OtterlyAI | February 5, 2026 | One test site, 90 days of AI bot server logs | 84 of 62,100+ AI bot requests hit /llms.txt, about 0.1%; an average page drew about 265 | One site, one log window; OtterlyAI sells AI visibility monitoring |
| Google Search Central | May 2026 | Guide to optimizing for generative AI features | Lists llms.txt among tactics site owners can ignore for Google Search | Covers Google Search only, not coding agents or browser agents |
| Chrome Lighthouse 13.3 | May 2026 | Experimental Agentic Browsing audit | Checks whether llms.txt exists and whether it errors; not an SEO audit | Measures agent readiness, not citations, and the category is labelled experimental |
| Ahrefs | Updated June 15, 2026 | About 38,000 domains with a valid llms.txt | 97% received zero requests for the file in May 2026 | Counts requests, not answers; Ahrefs sells SEO software |
| Canlah AI agent-readiness dataset | August 2026 | 47 reachable DTC storefronts | 26 of 47 served llms.txt, mostly alongside platform-issued manifests | Cross-border storefronts only, not a random sample of the web; first-party |
| Canlah AI self-audit | September 7, 2026 | canlah.ai, 39 questions, ChatGPT and Gemini, 3 runs | 4.0% non-brand mention rate with llms.txt in place | One site, two engines, no control site; first-party and directional |
Provenance: every row links to its source in Method and sources, and each page was checked on September 23, 2026. Third-party findings are reported as their publishers state them and were not independently replicated. “No correlation” means the study detected no measurable relationship under its own method, which is not proof that llms.txt can never matter.
The three vendor studies earn their place on scale and directness: SE Ranking and Ahrefs each cover tens of thousands to hundreds of thousands of domains, and OtterlyAI reads server logs rather than asking practitioners what they believe. Two cautions still apply. All three sell SEO or AI visibility tools, so their studies are not neutral market consensus, and every source here is correlational; none ran a controlled experiment where the only change was adding the file.
Across seven sources and four different methods, no source reports a positive effect of llms.txt on AI citations, and two independently show that AI crawlers rarely request the file at all.
llms.txt priorities are not the same across AI engines
Documentation naming llms.txt as an input to answer selection was not found for any of the answer engines below, in the sources reviewed on September 23, 2026. What each engine does document differs enough to change where the work goes.
- ChatGPT: OpenAI documents GPTBot and OAI-SearchBot together with robots.txt controls, and publishes an llms.txt for its own developer docs. Reading third-party llms.txt for ChatGPT search answers: not found in the sources reviewed.
- Perplexity: Perplexity’s crawler documentation states that PerplexityBot respects robots.txt, and citations follow retrievable, current pages. Use of a map file in answer selection: not found in the sources reviewed.
- Gemini: grounding runs through Google Search, so Google’s Search guidance applies unchanged. A separate Gemini position on llms.txt: not found in the sources reviewed.
- Google AI Overviews and Google AI Mode: Google Search states that llms.txt is not needed, and eligibility follows indexing and standard SEO requirements. Chrome’s Lighthouse 13.3 checks the file for browser agents instead.
- DeepSeek, Qwen and Doubao: these engines retrieve from Chinese-language sources first, so a Chinese-language page body matters far more than an English llms.txt. A documented llms.txt input: not found in the sources reviewed.
An llms.txt review that stops at confirming the file exists misses every question that decides citations. It should check, engine by engine:
- whether each answer engine’s crawler is allowed in robots.txt;
- whether the pages that engine cites for your category include yours;
- whether your own description is consistent across site, schema and llms.txt;
- whether coding and browser agents can load the pages your llms.txt lists.
Should my website have llms.txt?
It depends on who reads your site, but most sites shouldn’t treat the file as a strategy.
Case 1: You publish developer documentation or an API
Yes, publish llms.txt, and consider llms-full.txt and Markdown versions of your docs pages. Coding agents are the one audience with documented, routine use of the file. If your docs platform generates it, check that the output is current and not a raw link dump.
Case 2: You run a marketing, services or ecommerce site
Publish a short one if it takes an hour, and don’t pay a retainer for it. Put the hours into what engines evidently use: crawler access in robots.txt, indexable pages with clear answers, consistent entity facts and third-party sources that engines already cite for your category. Canlah AI’s guide to how AI engines choose sources covers those levers.
Case 3: You want to keep AI from using your content
llms.txt won’t help. Use robots.txt rules for named AI user agents, and understand the trade: blocking an answer engine’s crawler also removes you from the answers it writes.
How to create llms.txt in six steps
Step 1: Decide who the file is for
Write for an agent that knows nothing about you and has a small context budget: what you are, what you sell, where you operate and which pages hold the detail.
Step 2: Write the H1 and the summary blockquote
Use the exact brand name as the H1. In the blockquote, write two or three factual sentences that match the entity description on your About page and in your Organization schema, word for word where possible.
Step 3: Add curated H2 link lists
Group 10 to 40 links under plain H2s such as Key pages, Services, Documentation or Company facts. Give every link a one-line note after a colon explaining what the page answers.
Step 4: Move secondary links under Optional
Anything an agent can skip, such as archives, older posts or legal pages, belongs under an H2 named Optional so a context-limited agent can drop it.
Step 5: Publish at the root and verify the response
Serve the file at /llms.txt with a text content type and a 200 status. Fetch it from outside your network, confirm it isn’t blocked by robots.txt or a firewall rule and check it in Lighthouse’s Agentic Browsing audit.
Step 6: Decide on llms-full.txt and keep both current
If you have documentation or a substantial article library, generate llms-full.txt from the same source as your pages rather than by hand. Regenerate on every deploy, and add both files to whatever checks catch broken pages.
A compact version of canlah.ai’s structure, with the summary aligned to the entity description used across the site, looks like this:
# Canlah AI
> Canlah AI is a Singapore-based SEO + GEO agency that measures and improves brand visibility inside AI answers, reporting AI visibility as re-verifiable ranges with timestamped evidence.
Contact: admin@canlah.ai · Singapore
## Key pages
- [AI Info](https://canlah.ai/ai-info/): Plain-language facts about Canlah AI, its method and pricing structure.
- [GEO Services](https://canlah.ai/geo/): The four-stage SEO + GEO programme and measurement protocol.
- [Pricing](https://canlah.ai/pricing/): Package structure for SEO + GEO engagements.
## Open data
- [Agent-Readiness of 50 Cross-Border DTC Brands](https://canlah.ai/data/agent-readiness-2026/): CC BY 4.0 dataset, 50 rows x 22 fields.
## Optional
- [Blog](https://canlah.ai/blog/): Essays and data reports on AI visibility.
Summary: robots.txt governs access, llms.txt guides reading, evidence decides citation
robots.txt addresses whether AI crawlers may read your site.
llms.txt addresses how quickly an agent finds the right pages once it can.
Neither addresses the question most brands are really asking, which is whether an AI engine names them when a buyer asks for a recommendation. That’s decided by what the engine retrieves: indexable pages that answer the question, consistent facts about the brand and the third-party sources the engine already trusts.
Canlah AI is a Singapore-based SEO + GEO agency that measures and improves brand visibility inside ChatGPT, Gemini, Google AI Overviews and Google AI Mode, reporting AI visibility as re-verifiable ranges with timestamped evidence. Canlah AI ships llms.txt as one small part of the on-site foundation in its programme, and measures whether anything changed rather than assuming it did.
The limitation is worth stating plainly: as of September 23, 2026, Canlah AI had not run a controlled test isolating llms.txt on a client site, so it cannot tell you the file will move your numbers, and no other source reviewed for this guide can either.
Verify before you pay anyone, including Canlah AI. Fetch your own /llms.txt and /robots.txt, confirm both return the status and content you expect, put your buyers’ real questions to the engines they use and record which domains come back. If a provider says llms.txt will raise your citations, ask for the probe data measured before and after the file went live.
Frequently asked questions
What is llms.txt in simple terms?
llms.txt is a Markdown file at a website’s root that summarises the site and links to its most useful pages, written for AI agents rather than people. It was proposed by Jeremy Howard at llmstxt.org in September 2024. It guides reading; it doesn’t grant or deny access.
What is the difference between llms.txt and robots.txt?
robots.txt is a formal standard that tells crawlers which paths they may access, and compliant crawlers obey it. llms.txt is a community proposal that tells an agent what the site is and which pages to read, and it contains no rules to obey. Use robots.txt to allow or block AI crawlers and llms.txt to orient the agents you’ve let in.
Should my website have llms.txt?
If you publish developer documentation or an API, yes, because coding agents routinely use it. For most marketing and ecommerce sites, a short llms.txt is harmless and quick to add, but it isn’t a priority. Crawler access, indexable answer pages and consistent brand facts matter more for AI search visibility.
Does llms.txt improve AI citations or rankings?
No. Documentation of llms.txt as a citation input was not found for any major answer engine in the sources reviewed on September 23, 2026, SE Ranking found no correlation across about 300,000 domains and Google Search says the file is not needed for its AI features. llms.txt may still help an agent navigate a site, which is a different outcome from being cited.
What is llms-full.txt and do I need one?
llms-full.txt is a single Markdown file containing the full text of a site’s key pages, popularised by documentation platforms such as Mintlify. It’s most useful for developer docs that people load into coding assistants. A marketing site can publish one generated from its article library, but it should be built automatically so it never goes stale.
How do I create llms.txt for my website?
Write a Markdown file with the brand name as an H1, a two- or three-sentence summary in a blockquote and H2 sections listing key pages with one-line notes. Put secondary links under an H2 named Optional. Publish it at /llms.txt with a 200 status and check that its facts match your About page and schema.
Does Google use llms.txt?
Google Search doesn’t use llms.txt for AI Overviews, AI Mode or rankings, and its May 2026 guide lists the file among tactics site owners can ignore. Chrome’s Lighthouse 13.3 does check for llms.txt in an experimental Agentic Browsing audit aimed at browser agents. The two positions cover different products, not a change of policy.
Can llms.txt stop AI from training on my content?
No. llms.txt has no directive syntax, so it can’t block anything. To limit AI training, use robots.txt rules for named user agents such as GPTBot, Google-Extended, CCBot and anthropic-ai, and accept that blocking an answer engine’s crawler also removes you from its answers.
Is Canlah AI going to sell me an llms.txt file?
Canlah AI includes llms.txt in its on-site foundation work at no separate charge and doesn’t sell it as a citation lever. Engagements start with a baseline of buyer questions probed across AI engines, and on-site files are judged against that baseline. If a proposal from any provider leads with llms.txt, ask which measurement supports it.
Method and sources
This guide was built from the llms.txt proposal, Google’s public documentation, the most-cited public studies on llms.txt and AI citations and Canlah AI’s own files and audit archive. Public pages were reviewed on September 23, 2026, and live llms.txt files were fetched the same day. Third-party study findings are reported as their publishers state them and were not independently replicated. Canlah AI figures are first-party and directional.
- llmstxt.org, “The /llms.txt file, v2”, Jeremy Howard. Origin, format, intended use at inference time and the list of generating platforms.
- Google Search Central, guide to optimizing for generative AI features. The statement that llms.txt can be ignored for Google Search.
- Chrome for Developers, Lighthouse llms.txt audit. The Agentic Browsing check and its description of the file.
- Search Engine Journal, “Google’s llms.txt Guidance Depends On Which Product You Ask”, May 20, 2026. The Search versus Lighthouse split and earlier Google statements.
- Search Engine Roundtable, “Google Search Team Does Not Endorse LLMs.txt Files”, January 20, 2026. John Mueller’s statement that files on Google developer properties are not an endorsement.
- SE Ranking, “LLMs.txt: Why Brands Rely On It and Why It Doesn’t Work”, November 7, 2025. The 300,000-domain adoption and correlation study.
- OtterlyAI, “Llms.txt Experiment”, February 5, 2026. The 90-day server-log experiment.
- Ahrefs, “What Is llms.txt, and Should You Care About It?”, updated June 15, 2026. The 38,000-domain request analysis.
- Mintlify, llms.txt documentation. Automatic llms.txt and llms-full.txt generation.
- IETF, RFC 9309 Robots Exclusion Protocol. The robots.txt standard.
- Canlah AI llms.txt and robots.txt. The worked example; 113 lines and 15,953 bytes on September 23, 2026.
- Canlah AI, Agent-Readiness of 50 Cross-Border DTC Brands. The 26 of 47 llms.txt adoption figure.
- Canlah AI audit archive for canlah.ai, September 7, 2026. 39 questions, ChatGPT and Gemini, 3 runs; the 7 of 177 non-brand mention rate and the 85 and 4 dimension scores.
See where your brand stands in AI answers. Run the free AI visibility audit →
Related articles
- We ran our own GEO audit
- How AI engines choose sources
- SEO vs AEO vs GEO: what’s the difference
- The GEO vendor evidence checklist
- Why is my business not showing up in ChatGPT?
Originally published at canlah.ai.
Top comments (0)