DEV Community

Cover image for Should You Serve Markdown to AI Agents? llms.txt, Accept: text/markdown and What Google Actually Said
Furqan Khalid
Furqan Khalid

Posted on AI-assisted

Should You Serve Markdown to AI Agents? llms.txt, Accept: text/markdown and What Google Actually Said

#nextjs #seo #ai #webdev

There's a 2026 argument that runs roughly like this:

"LLMs prefer Markdown. HTML is 80% noise. So serve Markdown to AI bots and they'll understand you better and cite you more."

Half of that is true. This post separates the half that is from the half that isn't, then shows how to do it safely if you decide to.

The three things people mean by "Markdown for AI"

Approach What it is Who reads it
/llms.txt A curated Markdown index of your site at the root (llmstxt.org) Mostly coding assistants and tools a user points at your docs
.md twins Every page also available as Markdown, e.g. /docs/auth.md Same as above, plus agents that know the convention
Content negotiation Same URL, Markdown returned when the request sends Accept: text/markdown Agents that ask for it; Cloudflare's "Markdown for Agents" does this at the edge

What the evidence says

On llms.txt for search visibility: not much.

  • Google's Gary Illyes said in July 2025 that Google doesn't support llms.txt and has no plans to.
  • Google's May 2026 guide to AI features in Search goes further. It says optimizing for AI features "is optimizing for the search experience, and thus still SEO," and lists llms.txt and special machine-readable formats among the things you don't need for Google (Google's AI optimization guide).
  • Log studies published in 2026 found the big AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) rarely request /llms.txt and mostly crawl HTML. An SE Ranking study of ~300k domains reported roughly 10% adoption and no measurable link between having the file and being cited.

On Markdown for agents: real, but a different job.

  • Cloudflare launched Markdown for Agents in 2026. If a request includes text/markdown in its Accept header, Cloudflare fetches your HTML, converts it, and returns Markdown with an x-markdown-tokens header estimating the size (Cloudflare docs). Their own example blog post went from about 16,000 tokens as HTML to about 3,000 as Markdown.
  • Google's John Mueller was blunt about serving Markdown to bots for SEO, reportedly calling it "a stupid idea" (Search Engine Journal).

Both of these can be true at once. Search crawlers are built to parse HTML and don't need help. Agents and coding assistants working within a context window benefit a lot from getting 3k tokens instead of 16k.

So when is it worth doing?

Worth it:

  • Developer docs, API references, SDK guides. Developers paste your docs URL into Cursor, Claude Code or ChatGPT all day. A clean Markdown version means fewer tokens, less nav junk and better answers about your product. Stripe, Anthropic, Cloudflare and Vercel all publish llms.txt or Markdown docs for this reason.
  • Agent-heavy flows, such as product catalogues or knowledge bases that agents query on users' behalf.

Not worth it (on its own):

  • A marketing site hoping Markdown will rank it higher in Google or get it cited more by ChatGPT. There's no evidence it does. Spend the time on content and crawlability.

Decision guide: when serving Markdown is worth it, when it isn't, and rules to avoid cloaking

How to do it without getting into trouble

The risk is cloaking: showing bots different content from users. Google's spam policies forbid that, and since May 2026 they explicitly apply to AI Overviews and AI Mode as well. Markdown is fine if it's the same content in a different format. It stops being fine the moment the Markdown version has extra claims, keywords or "notes for AI" that the HTML doesn't.

Rules I'd follow:

  1. Generate Markdown from the same source as the HTML (your MDX, CMS, or the rendered HTML). Never maintain it by hand.
  2. Send Vary: Accept so caches don't serve Markdown to browsers or HTML to agents.
  3. Point to it with rel="alternate" so it's discoverable rather than hidden.
  4. Keep the HTML canonical. Add Link: <https://example.com/page>; rel="canonical" on Markdown responses, or add X-Robots-Tag: noindex to keep .md URLs out of search results.
  5. Don't sniff user agents. Respond to what the client asks for (Accept), not to who you think it is.

Implementation: Next.js content negotiation

Assume your docs are MDX files. We'll serve the raw Markdown (front matter stripped) at /docs/foo.md and also when someone requests /docs/foo with Accept: text/markdown.

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";

export function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl;
  if (!pathname.startsWith("/docs")) return NextResponse.next();

  const accept = req.headers.get("accept") ?? "";
  const wantsMarkdown =
    pathname.endsWith(".md") ||
    // Only when Markdown is explicitly listed. Browsers send text/html and */*.
    /(^|,)\s*text\/markdown/i.test(accept);

  if (!wantsMarkdown) return NextResponse.next();

  const slug = pathname.replace(/^\/docs\/?/, "").replace(/\.md$/, "") || "index";
  const url = req.nextUrl.clone();
  url.pathname = `/api/md/${slug}`;
  return NextResponse.rewrite(url);
}

export const config = { matcher: ["/docs/:path*"] };
Enter fullscreen mode Exit fullscreen mode 
// app/api/md/[...slug]/route.ts
import { readFile } from "node:fs/promises";
import path from "node:path";
import matter from "gray-matter";

const DOCS_DIR = path.join(process.cwd(), "content/docs");

export async function GET(_req: Request, { params }: { params: Promise<{ slug: string[] }> }) {
  const { slug } = await params;
  const rel = slug.join("/");

  // Basic path-traversal guard
  const file = path.normalize(path.join(DOCS_DIR, `${rel}.mdx`));
  if (!file.startsWith(DOCS_DIR)) return new Response("Not found", { status: 404 });

  let source: string;
  try {
    source = await readFile(file, "utf8");
  } catch {
    return new Response("Not found", { status: 404 });
  }

  const { content, data } = matter(source);
  const body = `# ${data.title ?? rel}\n\n${stripJsx(content)}`;
  const canonical = `https://example.com/docs/${rel === "index" ? "" : rel}`;

  return new Response(body, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Vary": "Accept",
      "Link": `<${canonical}>; rel="canonical"`,
      "Cache-Control": "public, max-age=3600",
    },
  });
}

// Remove MDX component tags but keep their text content.
function stripJsx(md: string) {
  return md
    .replace(/^import .*$/gm, "")
    .replace(/<([A-Z][A-Za-z0-9]*)[^>]*\/>/g, "")
    .replace(/<\/?[A-Z][A-Za-z0-9]*[^>]*>/g, "");
}
Enter fullscreen mode Exit fullscreen mode

And in the HTML page's <head>:

// app/docs/[...slug]/page.tsx
export async function generateMetadata({ params }) {
  const { slug } = await params;
  return {
    alternates: {
      types: { "text/markdown": `/docs/${slug.join("/")}.md` },
    },
  };
}
Enter fullscreen mode Exit fullscreen mode

Test it:

curl -s https://example.com/docs/auth | head -c 200                           # HTML
curl -s -H "Accept: text/markdown" https://example.com/docs/auth | head -20  # Markdown
curl -sI -H "Accept: text/markdown" https://example.com/docs/auth | grep -i -E "content-type|vary|link"
Enter fullscreen mode Exit fullscreen mode

On Cloudflare? If you're on a plan that includes Markdown for Agents, you can switch it on in the dashboard (AI Crawl Control) instead of writing this. Writing it yourself only pays off if you want cleaner output from your source files than an HTML-to-Markdown conversion gives you.

And llms.txt?

If you run a docs site, a short llms.txt is cheap and helpful for coding assistants. Generate it at build time from your nav config so it never drifts out of sync:

# Acme API

> Acme is a payments API for marketplaces. These docs cover authentication, payouts and webhooks.

## Core docs
- [Authentication](https://example.com/docs/auth.md): API keys, OAuth, key rotation
- [Payouts](https://example.com/docs/payouts.md): schedules, currencies, limits
- [Webhooks](https://example.com/docs/webhooks.md): events, retries, signature verification

## Optional
- [Changelog](https://example.com/changelog.md)
Enter fullscreen mode Exit fullscreen mode

Just don't sell it to your boss as an SEO project.

Conclusion

  • For Google rankings and AI Overviews: Markdown does nothing. HTML is fine.
  • For agents and coding assistants using your docs: Markdown helps a lot. Do it.
  • To stay out of trouble: same content, Vary: Accept, canonical to the HTML, no bot-only text.

Next in the series: Hidden prompts in your HTML are now spam.

Top comments (0)