DEV Community

Baris Sozen
Baris Sozen

Posted on

Our llms-full.txt Is the Most-Seen URL on Our Site. It Contributes Zero to Google's AI Answers.

We publish an llms.txt and an llms-full.txt at the root of hashlock.markets. The full file is about 123 KB of plain Markdown. We wrote it for AI agents, on the theory that an agent asking "what is Hashlock and what does it do" should get a clean answer instead of parsing our HTML.

Yesterday we opened Google Search Console and read 28 days of data properly for the first time. Two numbers came out of it, and they point in opposite directions.

In ordinary web search, /llms-full.txt gets more impressions than any other URL we own. More than the homepage. More than /about. More than the docs.

In Google's generative AI features, /llms-full.txt has zero impressions. Not few. Zero. It does not appear in the report at all.

Here is the whole table, unedited, 30 July to 26 August 2026.

What Google's AI features actually used

Search Console's Search Generative AI performance report covers impressions inside AI Overviews and AI Mode. Google shipped it on 3 June 2026. Ours, over 28 days:

Page AI feature impressions
/about 13
/ 9
/methodology 6
/terms 2
/docs 1
/privacy 1
/sign/login 1
/llms-full.txt absent

Twenty-two impressions total. We are a small site; this is a small number and we are not going to dress it up. But the shape of the number is the interesting part, and the shape does not depend on the magnitude.

Three things fall out of it.

The file written for machines is invisible on the machine-facing surface. Everything Google's AI features touched is a human-readable HTML page. The plain-Markdown file we built specifically so a model would not have to parse HTML is the one thing they did not touch.

Our AEO surface is /about, not the homepage. Thirteen versus nine. We have been optimising the front door while Google has been quoting the second room.

/methodology beats /docs six to one. One page, written to explain how we count volume for a third-party listing, produces six times the AI visibility of our entire documentation tree.

This is not a surprise if you have been reading the research

Our numbers are small, and small numbers lie. So the honest move is to check them against people with real sample sizes. They agree, and they agree hard.

SE Ranking crawled roughly 300,000 domains and found llms.txt on 10.13% of them, with no relationship between having the file and how often a domain gets cited in major LLM answers. Among the fifty most AI-cited domains, one had the file.

Limy analysed 515,382,577 LLM bot traffic events. Filtering for the user agents that actually drive citations, GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, only 408 requests across a 90-day window touched /llms.txt at all.

And Google has said so out loud. Gary Illyes confirmed Search does not support llms.txt and is not planning to. John Mueller compared it to the keywords meta tag, on the grounds that a file the site operator fully controls is a file no search engine can trust. His second objection is the sharper one: if the crawler has already fetched your HTML and your structured data, what is the Markdown copy adding, other than a surface on which you could show bots one thing and humans another?

So the "does llms.txt help AI search" question is settled, and it has been settled for a while. Our data is one more small confirmation from a site nobody was measuring.

The half that gets left out

Here is where most write-ups stop, and where they get it wrong.

llms.txt is not doing badly at its job. It is doing well at a different job than the one people keep measuring.

Jeremy Howard proposed the format at Answer.AI in September 2024, and the layer that adopted it is not search. It is tool-use. Cursor, Claude Code, GitHub Copilot, Windsurf, Cline and Aider look for /llms.txt and /llms-full.txt when you point them at a documentation site. LangChain shipped mcpdoc, an MCP server whose entire purpose is exposing llms.txt files to host applications as a fetch_docs tool. Mintlify, Fern, GitBook, Vercel and Supabase generate the file by default, which is why Anthropic, Stripe, Cloudflare and Coinbase all ship one.

That is a real and growing consumer base. It is just not a search consumer base, and no analytics product on the market reports it to you. Search Console will not show you a Cursor fetch. GA4 will not show you an agent that read your file, answered its user, and never sent a browser.

Which is the actual finding, and it is not about llms.txt at all.

We spent 138 days measuring the agent layer with a human-layer instrument

Two days ago we discovered our npm download count had been read wrong. Yesterday we discovered our Search Console numbers had been wrong by a factor of twelve, because our probe summed the query table instead of reading the summary card, and Google anonymises any query below a disclosure threshold. On our property, 92% of impressions and all eleven of our clicks live in that anonymous pool. The searches that convert are, by construction, the ones we cannot see.

The generative AI report has a harder version of the same problem: it is UI only. As of 11 August 2026 neither the Search Analytics API nor the BigQuery bulk export returns generative AI data, which is verifiable by anyone with a property. That is why our automated daily probe reported nothing about AI visibility for 138 consecutive days. Not because there was nothing. Because the pipe does not exist and we never checked whether it did.

Three instruments, three ways of being blind, all in one week. The pattern is not that we are careless with numbers, though we were. The pattern is that every measurement tool we reached for was built to count humans arriving in browsers, and the thing we are trying to build for does not arrive in a browser.

That is not a marketing problem. It is the same structural problem we work on.

Why an atomic settlement team cares about a Markdown file

We build the settlement layer for the agent economy: sealed-bid RFQ fused with HTLC atomic settlement, so two agents that have no reason to trust each other can trade across chains without a bridge, a custodian or an arbiter. Settlement runs live end-to-end on Ethereum mainnet today. Our Sui contracts are deployed and CLI-tested, with gateway wiring in progress, and they are not live. Bitcoin is signet-validated with mainnet pending.

The reason that architecture exists is that when an agent transacts, nobody is watching. There is no human to notice the counterparty went quiet at step three, no support queue, no chargeback window that a person remembers to open. The safety has to be in the mechanism, because there is no observer to catch the failure.

llms.txt is the low-stakes version of the same fact. An agent reads your file, uses it, and leaves no trace in any dashboard you own. That is fine for documentation. It is not fine for money. If your design for agent-to-agent value transfer assumes someone will see the failure and intervene, you have built the observability assumption into the trust model, and the observability is not there.

What we are actually changing

Not the file. We are keeping llms.txt and llms-full.txt, for exactly the reason the research supports: they are business-to-agent infrastructure, they cost almost nothing to maintain, and the consumers are IDE agents and MCP hosts, not Google. Cutting them because Google ignores them would be optimising for the audience we are explicitly not building for.

What changes is the measurement and the targets:

  • Read totals from the Search Console summary card, never from the sum of a query table. Query tables answer "which queries are named", not "how many".
  • Treat the generative AI report as a manual read, because there is no API. If it is not automatable, put it on a human's calendar rather than pretending the automation covers it.
  • Point content at /about, which is our highest AI-visibility page, and stop linking to /docs, which earned one AI impression in 28 days.
  • Instrument the CDN for hits to /llms.txt and /llms-full.txt by user agent. That is the only measurement that will ever see the audience the file is for, and it is the one we do not have.

That last one is the whole post in a sentence. We wrote a file for an audience, and then measured it with a tool that cannot perceive that audience, and concluded things for 138 days on that basis.

If you ship an llms.txt, check your CDN logs before you check your rankings. And if you are building anything where agents move value rather than just reading docs, ask what happens on the failure path when nobody is watching, because that is the default condition and not the edge case.

Protocol: https://hashlock.markets/about?utm_source=devto&utm_medium=article&utm_campaign=2026-08-30-llms-txt-ai-answers
MCP server on npm: hashlock-tech/mcp (scoped), v0.6.0
Volume methodology: https://hashlock.markets/methodology?utm_source=devto&utm_medium=article&utm_campaign=2026-08-30-llms-txt-ai-answers
Whitepaper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6712722

One question, and I want real answers rather than agreement: if you ship an llms.txt, have you ever checked your server logs for who actually fetches it? What user agents show up, and at what rate? Our own number is the one measurement we do not have yet, and I would rather learn it from your logs than from another vendor study.

Top comments (0)