How to serve Markdown to AI agents: Making your docs more AI-friendly

#tutorial #javascript #ai #webdev

Introduction

If you've been on developer Twitter recently, you might have seen Bun's tweet about serving Markdown directly to AI coding assistants like Claude Code.

Both Mintlify and Fumadocs have already immediately implemented the feature in their documentation platforms, and I just implemented it myself for Lingo.dev.

Here's why it matters and how to implement it yourself.

The problem: HTML bloat

AI agents can fetch the content of web pages to help developers. Most of the time, this content comes back as HTML. But HTML contains markup that consume tokens without adding meaningful information, degrading performance.

The solution: Content negotiation

The elegant solution is content negotiation. This is a standard HTTP mechanism where clients tell servers what format they prefer using the Accept header.

When an AI agent requests your documentation with Accept: text/markdown or Accept: text/plain, your server can respond with Markdown instead of HTML.

This means:

Fewer tokens for the same information
Easier parsing and comprehension
More documentation fits in the context window

Implementation guide

The exact implementation detail depends on your programming language and framework, but the general approach is the same.

Let's walk through an example with Express.js.

1. Detect the Accept header

When a request arrives, check if the Accept header contains text/markdown or text/plain:

app.get("/docs/*", (req, res) => {
  const acceptHeader = req.headers.accept || "";

  if (acceptHeader.includes("text/markdown")) {
    // Serve Markdown
  } else if (acceptHeader.includes("text/plain")) {
    // Serve plain text
  } else {
    // Serve HTML (default behavior)
  }
});

2. Serve the raw Markdown

Load and return the raw Markdown content for the requested page:

if (acceptHeader.includes("text/markdown")) {
  const markdownContent = await loadMarkdownForPage(req.path);
  res.setHeader("Content-Type", "text/markdown; charset=utf-8");
  res.send(markdownContent);
  return;
}

3. Fallback to existing behavior

For all other requests (browsers, crawlers, etc.), continue with your existing HTML rendering:

res.render("docs", { content: processedContent });

4. Test your implementation

Use curl to verify your implementation works correctly:

# Request Markdown
curl -H 'Accept: text/markdown' https://lingo.dev/en/cli

# Request plain text
curl -H 'Accept: text/plain' https://lingo.dev/en/cli

# Request HTML (default)
curl -H 'Accept: text/html' https://lingo.dev/en/cli