How to serve Markdown to AI agents: Making your docs more AI-friendly

Introduction

If you've been on developer Twitter recently, you might have seen Bun's tweet about serving Markdown directly to AI coding assistants like Claude Code.

Both Mintlify and Fumadocs have already immediately implemented the feature in their documentation platforms, and I just implemented it myself for Lingo.dev.

Here's why it matters and how to implement it yourself.

The problem: HTML bloat

AI agents can fetch the content of web pages to help developers. Most of the time, this content comes back as HTML. But HTML contains markup that consume tokens without adding meaningful information, degrading performance.

The solution: Content negotiation

The elegant solution is content negotiation. This is a standard HTTP mechanism where clients tell servers what format they prefer using the Accept header.

When an AI agent requests your documentation with Accept: text/markdown or Accept: text/plain, your server can respond with Markdown instead of HTML.

This means:

• Fewer tokens for the same information
• Easier parsing and comprehension
• More documentation fits in the context window

Implementation guide

The exact implementation detail depends on your programming language and framework, but the general approach is the same.

Let's walk through an example with Express.js.

1. Detect the Accept header

When a request arrives, check if the Accept header contains text/markdown or text/plain:

app.get("/docs/*", (req, res) => {
  const acceptHeader = req.headers.accept || "";

  if (acceptHeader.includes("text/markdown")) {
    // Serve Markdown
  } else if (acceptHeader.includes("text/plain")) {
    // Serve plain text
  } else {
    // Serve HTML (default behavior)
  }
});

2. Serve the raw Markdown

Load and return the raw Markdown content for the requested page:

if (acceptHeader.includes("text/markdown")) {
  const markdownContent = await loadMarkdownForPage(req.path);
  res.setHeader("Content-Type", "text/markdown; charset=utf-8");
  res.send(markdownContent);
  return;
}

3. Fallback to existing behavior

For all other requests (browsers, crawlers, etc.), continue with your existing HTML rendering:

res.render("docs", { content: processedContent });

4. Test your implementation

Use curl to verify your implementation works correctly:

# Request Markdown
curl -H 'Accept: text/markdown' https://lingo.dev/en/cli

# Request plain text
curl -H 'Accept: text/plain' https://lingo.dev/en/cli

# Request HTML (default)
curl -H 'Accept: text/html' https://lingo.dev/en/cli

For Markdown requests, you should see:

• Response header: Content-Type: text/markdown; charset=utf-8
• Body: Raw Markdown content without HTML tags

For HTML requests, you should see your normal rendered page.

Further reading and resources

To learn more about how to implement this, check out the following resources:

• Accept header documentation
• Next.js template by @cramforce
• Set-up guide for Laravel by @retlehs
• Set-up guide for static websites, Cloudflare Workers, and Caddy by @skeptrune.

(If you've created your own guide, example, or template, share it in the comments so I can add it to the post.)

Comments

[ImTheDeveloper]

I would caveat that if you have heavy table based content you may find that HTML structure is better recognised than a markdown equivalent.

Generally best practice is to ditch the bloat tags i.e. script, ads and for the rest of them it's dependent on structure.

Theres a lot of research in this space and a hybrid approach can often yield the best results. I agree with the post direction though there is a clear benefit to removing non-required tags and the bloat of classes/styling they bring.

[Max Prilutskiy Lingo.dev]

text/markdown > text/html

[Sumit Saurabh Lingo.dev]

Markdown all the way!!!!

[Barry Grenon]

This assumes the docs are GEO friendly in content already. What about capturing anti-patterns and deprecations to make sure the models don't default to "common knowledge" when your docs are net new?

What other patterns do docs need to include to ensure llm success, based on the unique "psychology" of how models process information, versus a human reader?

I wonder if the ideal will be to serve up an AI-only addendum or even completely alternate version of content to be successful.

I hope but we don't end up doing is idealizing the docs around the needs of AI (repetition, verbose procedures) versus human readers, in the same content output. The constraints on making reader friendly docs becomes: we can't do x because of localization, we can't do y visually because of accessibility, we can't do z because of llms. It is more work to provide alternates for these variety of constraints, so most writing teams shift their guidelines downwards so the same content serves all these constraints in their "single source". While the main use case - creating pleasant, compelling content according to the 80/20 rule of usage - this ends up coming last. Don't do that!

[Ellis]

What? Doesn't make sense to me. This assumes you have your page as markdown in the first place? I need a tool that converts HTML to markdown so I can even use it for websites I don't have control of.

[David Turnbull Lingo.dev]

Some of these tools already convert from HTML to Markdown already. The point of this post is to show how to provide Markdown directly, which is faster, more consistent, and gives you more control. So not something for use with third-party websites.

[Hashbyt]

Content negotiation to serve AI-friendly formats demonstrates how subtle backend enhancements can significantly boost frontend AI integrations and user experience.

[The Arkitekt]

Yes. Keep finding ways to bow to our AI overlords.

Good. Good.