When the llms.txt proposal emerged as a standard to help LLMs and autonomous agents navigate web documentation efficiently, it promised an alternative to messy HTML scraping.
In August 2026, the specification was formally updated to v2, introducing two major changes:
- Standardized discovery via HTTP headers and HTML
<link rel="describedby">tags. - Markdown twin conventions for individual documentation routes.
Our research team at GrowNexus audited 25 major SaaS platforms to measure how the industry is actually adopting the standard.
đź“Š Key Findings from the Dataset
We examined 25 prominent platforms across developer tooling, CMS, infrastructure, and productivity software:
-
Adoption Rate: 18 of 25 (72%) publish an
llms.txtfile. - File Size Variance: File sizes ranged from 2,088 bytes to 210,736 bytes (median: 16,299 bytes).
- Spec Structure Compliance: Only 12 of 18 (66.7%) follow the section hierarchy (H1, blockquote summary, curated H2 list) outlined in the spec.
-
Header Discoverability (
rel="describedby"): Only 2 of 18 (11.1%) emit the header. -
HTML-Level Discoverability: 0 of 18 (0%) implemented
<link rel="describedby">in their page markup.
⚠️ The Problem with 100KB+ llms.txt Files
The most significant operational issue discovered is using llms.txt as a raw sitemap dump.
When autonomous agents (like Claude Code, Codex, or Hermes) parse context, scratchpads are token-budgeted. Files over 50KB consume 15,000+ tokens before the agent executes a single tool call, resulting in aggressive truncation.
A well-architected llms.txt should act as a curated index of core endpoints—not a full site mirror.
🛠️ Open-Source Benchmark Dataset
We have released the complete dataset and analysis scripts under the CC BY 4.0 license:
- GitHub Repository: github.com/grownexus/llms-txt-benchmark
- Complete Implementation Guide & Code: GrowNexus Insights
How is your engineering team approaching llms.txt implementation for your APIs?
Top comments (0)