I'm a computer science student. I work on side projects, and these days I'm building a small platform to help content creators. And like others before me, I noticed something: what we publish on the Internet isn't really read by humans anymore. It gets read by AIs, digested, then served back to a human as a recap. Since then, I've been optimizing everything I put online for them. Here's why.
Look at Google: ask a question, and the first thing you see is no longer a list of links — it's an "AI Overview" that answers you directly. Most of the time, that's enough: you leave without visiting a single site. The numbers back it up: when an AI Overview shows, only 8% of people still click a result, versus 15% without one (Pew Research, 68,879 real searches). As for me, sometimes I don't even type keywords anymore: I ask full questions, because I know an LLM will answer.
The new visitor doesn't click, it reads
In the human's place arrives a new kind of visitor: the AI agent. ChatGPT, Claude or Perplexity visit the page on their behalf, keep the essentials and hand back a summary. The volume is still modest — around 1% of web traffic (Conductor) — but it's climbing fast (+357% in a year, Similarweb) and it converts hard: 15.9% for traffic coming from ChatGPT, versus 1.76% for Google organic (Seer Interactive). Low volume, huge intent: now is when there's a head start to be had.
LLMO, and what it isn't
The SEO community has a name for this: LLMO, Large Language Model Optimization. In one sentence: SEO optimizes for ranking; LLMO optimizes for meaning, structure, and extractability by AIs.
The key point for developers: an LLM doesn't "browse". It sees neither your hero section, nor your animations, nor your dark mode. It receives text, splits it into tokens, and every token has a cost. Your beautiful interface is, from its point of view: useless noise, billed by the token, stretching the context; zero benefit.
The real cost: tokens
I ran the test on a real page of my own platform — dunga.io/midwiq — with OpenAI's tokenizer. The same information, three representations:
- Full page (HTML + UI/UX): 1,904 tokens — and nearly 2,900 on the legacy GPT-4 tokenizer
- Minimal semantic HTML: 521 tokens
- Clean Markdown: 283 tokens
That's 85% fewer tokens — up to ×10 depending on the tokenizer. Which makes sense: navigation, CSS, JavaScript artifacts… none of it carries any information, but the agent pays for it anyway. Clean Markdown removes all of it.
llms.txt: the obvious fix that isn't
The fashionable answer is called llms.txt: a Markdown file at the root of your site, meant to guide AIs. On paper, appealing. In the data, it doesn't hold up.
SE Ranking (300,000 domains analyzed): no measurable effect on citations — of the 50 domains most cited by AI, only one had the file. Limy: out of 500 million AI bot visits over 90 days, 408 requested llms.txt; bots crawl the HTML directly. And Google has ruled: no support — John Mueller even compared it to the old meta keywords tag.
The problem isn't the intention, it's the architecture: agents don't go looking for a side file, they read the URL you hand them. That's where the right representation needs to be served.
And yet, I have one on my site — and I stand by it, for two reasons. First: when a convention emerges on the web, I'd rather follow it; that's how conventions end up becoming standards. Second: mine isn't trying to get cited, it serves as the README of the machine interface — it's where an agent learns that every page exists as .md, and how to ask for it. Let's be honest: according to the numbers, right now, it does nothing. But it's one static file, zero maintenance — if the standard takes off, we're ready; if it dies, we've lost nothing.
What it looks like in practice (e.g. dunga.io)
That's what I did on dunga.io, the small smart-links platform (fanlinks, link-in-bio, deeplinks) I'm building for creators and artists. A link page is the textbook case: a name, some links, a bio — three useful pieces of information drowned in design.
Every link page exists as a semantic HTML version and a Markdown version, generated on the fly at request time — always up to date, nothing to store. And it's not even the server doing the work: an edge middleware (a few dozen lines running on the CDN, before everything else) looks at the User-Agent. A human gets the design, a social crawler gets its Open Graph card, an AI agent gets the stripped-down version. Same URL, three representations — 100% of the information, ~15% of the tokens. You can see it for yourself: add .md or ?format=md to any public dunga.io page. And the creator didn't have to lift a finger. That's the part I care about: LLMO shouldn't be one more chore for creators — it's a platform's responsibility.
Your turn
I don't know whether the term "LLMO" will survive past 2027. But the gap between a 2,000-token DOM and 300 tokens of Markdown isn't going anywhere. Three simple things to try this week:
- Check your logs: filter by AI User-Agent (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) and measure that traffic.
- Tokenize one of your pages: run the raw HTML through a tokenizer and compute the useful-tokens / total-tokens ratio.
- Serve an alternative representation on the same URL: content negotiation on the User-Agent fits in a few lines of middleware.
And the trap I only discovered by testing with real agents: the Content-Type. Serving text/markdown is clean on paper — except some LLM fetchers refuse any response that isn't HTML. And the reverse is worse: raw Markdown labeled text/html gets flattened by their HTML parser — line breaks swallowed, the whole content on a single line. What ended up working for me: convert the Markdown into minimal semantic HTML (headings, lists, links — zero CSS, zero scripts) served by default, and only return true text/markdown to clients that ask for it (Accept: text/markdown, or an explicit .md in the URL). Two details that go with it: a Vary: User-Agent, Accept header, without which your CDN will eventually serve the robot version to a human; and check that your own anti-bot layer isn't quietly strangling the non-human readers of your public pages — mine was, and I found out late.
Here's the gist:
// middleware.ts — Vercel Edge (same logic on Cloudflare Workers, Nginx, etc.)
const AI_UA = /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|CCBot/i;
export default async function middleware(req: Request) {
const url = new URL(req.url);
const wantsMd =
url.pathname.endsWith(".md") ||
url.searchParams.get("format") === "md" ||
/text\/markdown/i.test(req.headers.get("accept") || "");
const isAgent = AI_UA.test(req.headers.get("user-agent") || "");
if (!wantsMd && !isAgent) return; // human → the app, as before
const md = await buildMarkdown(url.pathname); // your content, stripped down
if (wantsMd) return respond(md, "text/markdown"); // explicit ask: real Markdown
return respond(mdToMinimalHtml(md), "text/html"); // agents: minimal semantic HTML
}
const respond = (body: string, type: string) =>
new Response(body, {
headers: {
"Content-Type": `${type}; charset=utf-8`,
Vary: "User-Agent, Accept", // NEVER mix human/robot caches
},
});
buildMarkdown and mdToMinimalHtml are the part that depends on your product: the first builds the Markdown from your data (served by your backend, not from your DOM), the second converts it into basic HTML — headings, lists, links, nothing else.
Top comments (0)