Imagine an AI visibility audit landing in your backlog. Alongside indexing issues, outdated documentation, and weak internal links, there is a new urgent task: “Add llms.txt so AI can find the website.”
The implementation looks easy. Create a text file, add a few links, deploy it, and mark the ticket complete.
The harder question is what you have actually improved.
For Google Search, the answer is clear: you do not need an llms.txt file to appear in AI Overviews or AI Mode. Google says its Search systems do not use the file.
For a particular coding agent or documentation workflow, there may be a useful application. That requires a different justification and a different test.
What Google actually says about llms.txt
Google’s official guide to generative AI search optimization addresses llms.txt directly. Its position is that special AI text files and Markdown formats are unnecessary for Google Search, including its generative AI capabilities.
The guide also says maintaining these files for other systems will neither help nor harm your visibility or rankings in Google Search, because Search ignores them.
Google may still discover, crawl, or index a text file. That does not mean it treats the file as a special instruction source or ranking signal.
For a Google-focused SEO backlog, my recommendation is to prioritize confirmed content and indexing problems over an llms.txt implementation.
What an llms.txt file is intended to do
The llms.txt proposal describes a Markdown file containing background information, guidance, and links to detailed resources. It is intended to help agents use a website.
The current proposal allows the file at the site root or a relevant subpath, such as /docs/llms.txt. It also proposes clean Markdown versions of useful pages.
A simplified, hypothetical documentation example looks like this:
# Example API
> Documentation for the Example API, including authentication,
> event delivery, and integration guides.
Use the current API reference for supported endpoints.
Check the migration guide before using examples from older versions.
## Documentation
- [Authentication](https://example.com/docs/authentication.md): Credentials and request signing.
- [Webhooks](https://example.com/docs/webhooks.md): Delivery behavior and event handling.
- [Migration guide](https://example.com/docs/migration.md): Changes between supported versions.This gives a compatible tool a curated starting point. It does not establish that every AI assistant will discover the file, read it, or follow its guidance.
If you are considering adoption, name the intended consumer. “Our documentation agent reads this URL” is a concrete use case. “AI likes text files” is not.
Why Lighthouse’s guidance is not a Search requirement
There is a detail that can make the discussion confusing: Chrome’s Lighthouse has documentation for an llms.txt audit under its agentic browsing checks.
According to the Lighthouse documentation, the audit flags server errors when retrieving the file. If the file is absent and returns a 404, the audit is marked not applicable. Providing it is optional.
These documents address different questions. Search Central explains what Google Search uses. Lighthouse discusses a file that may assist agents interacting with websites.
An optional agent-focused audit does not establish a requirement for Google rankings or AI citations. When someone points to “Google’s recommendation,” check which Google product the documentation describes.
Do you need llms.txt for your website?
Use the intended outcome to make the decision:
| Your goal | Suggested decision | What to verify |
|---|---|---|
| Improve Google rankings or AI Search visibility | Do not treat llms.txt as a ranking task | Google Search’s documented requirements and your actual site issues |
| Help a coding agent navigate product documentation | Consider a small implementation | Whether that agent reads the file and completes tasks more accurately |
| Support a custom assistant you control | Use it if it fits your retrieval workflow | Whether your integration fetches and uses the resources correctly |
| Enable automatic output from a documentation platform | Review its quality and maintenance cost | Current links, version accuracy, and intended consumers |
| Satisfy a generic “AI readiness” checklist | Request a specific use case first | The consuming system and a measurable acceptance criterion |
A small file may cost little to create. Maintaining another representation of your documentation can cost more, especially if releases change endpoints, limits, or recommended workflows.
Evaluate the ongoing work alongside the possible benefit.
What to fix first for Google’s AI features
Google’s AI features documentation says supporting pages must be indexed and eligible to appear in Search with a snippet. Eligibility does not guarantee inclusion.
That gives you a more useful starting point than adding a file to a site whose important pages are not indexed.
Confirm that important pages can be found
Inspect your priority URLs in Search Console. Investigate exclusions, unexpected canonical selections, access problems, and missing content before changing the site’s publishing format.
Work from observed problems. A sitewide assumption that “AI cannot read our HTML” is not a diagnosis.
Fix answers that leave readers stuck
Review the questions your users ask before purchasing or implementing your product. Does the page explain the limitations, prerequisites, and failure cases?
For developer documentation, that might mean adding a tested migration example. For a service business, it could mean clarifying scope, pricing assumptions, or who the service suits.
A file that points to incomplete answers preserves the underlying problem.
Connect related content
Link prerequisites, implementation guides, troubleshooting pages, and product details where readers need them. Strengthening coverage across related tasks can make a content library more useful without publishing another page for every keyword variation.
Make the next step clear for someone who arrives halfway through the journey.
If you create the file, test the actual workflow
Use a small experiment with a named tool and defined tasks. For example, ask a documentation assistant to identify the supported authentication method, explain retry behavior, and locate the current migration instructions.
Compare runs with and without access to the file while keeping the model, tool permissions, and task wording consistent. Repeat the tasks enough to avoid drawing conclusions from one unusually good answer.
Measure correct answers, supported references, version mistakes, and task completion. If efficiency matters, also record requests, tokens, or time using whatever your tool exposes.
Separate two experiments: explicitly supplying the file tests its usefulness as context; allowing autonomous discovery tests whether the agent finds it.
Server logs can show requests to the file. A request alone does not prove that its contents influenced an answer.
Keep the file aligned with the source documentation
If the experiment justifies maintaining llms.txt, generate it from the same source as your public documentation where practical.
Check that linked resources exist, describe the correct product version, and remain consistent with the pages users read. Avoid putting private endpoints, credentials, or unpublished details into a public file.
Do not use llms.txt as an access-control mechanism. Enforce authentication and permissions on the resources themselves.
Assign ownership. A file without a maintenance process can become a confident guide to outdated information.
Keep traffic experiments separate from agent experiments
Website teams often bundle AI visibility, search traffic, and engagement into one project. That makes attribution difficult.
Keep visibility, visits, and business outcomes separate. An assistant completing an integration task correctly is a different result from more search visits or more qualified inquiries.
For teams running a separate targeted search-traffic experiment, SearchSEO’s Bing campaigns offer keyword and geographic targeting. Record campaign activity separately so it does not cloud your assessment of content changes or agent behavior.
Targeted traffic does not validate an llms.txt implementation, and the file is not evidence that a traffic campaign will improve AI citations.
Make the decision around a consumer, not a checklist
Do you need an llms.txt file? For Google Search, Google’s guidance says no. For a specific agent workflow, it is worth considering when you can identify the consumer, test the benefit, and maintain the content.
Before creating the file, write one sentence explaining who will use it and what should improve. If that sentence is vague, revisit the task.
Resolve the content and access problems you can verify. If a separate targeted traffic test fits your broader strategy, explore SearchSEO with its own objectives and measurement plan.
A completed AI-readiness checklist is less useful than documentation that answers the question and helps someone finish the job.
Top comments (0)