The problem: HTML bloat
AI agents can fetch the content of web pages to help developers. Most of the time, this content comes back as HTML. But HTML contains markup that consume tokens without adding meaningful information, degrading performance.
The solution: Content negotiation
The elegant solution is content negotiation. This is a standard HTTP mechanism where clients tell servers what format they prefer using the Accept header.
When an AI agent requests your documentation with Accept: text/markdown or Accept: text/plain, your server can respond with Markdown instead of HTML.
This means:
Fewer tokens for the same information
Easier parsing and comprehension
More documentation fits in the context window
Implementation guide
The exact implementation detail depends on your programming language and framework, but the general approach is the same.
Let's walk through an example with Express.js.
1. Detect the Accept header
When a request arrives, check if the Accept header contains text/markdown or text/plain:
app.get("/docs/*", (req, res) => {
const acceptHeader = req.headers.accept || "";
if (acceptHeader.includes("text/markdown")) {
// Serve Markdown
} else if (acceptHeader.includes("text/plain")) {
// Serve plain text
} else {
// Serve HTML (default behavior)
}
});
2. Serve the raw Markdown
Load and return the raw Markdown content for the requested page:
if (acceptHeader.includes("text/markdown")) {
const markdownContent = await loadMarkdownForPage(req.path);
res.setHeader("Content-Type", "text/markdown; charset=utf-8");
res.send(markdownContent);
return;
}
**3. Fallback to existing behavior
**For all other requests (browsers, crawlers, etc.), continue with your existing HTML rendering:
res.render("docs", { content: processedContent });
4. Test your implementation
Use curl to verify your implementation works correctly:
# Request Markdown
curl -H 'Accept: text/markdown' https://lingo.dev/en/cli
# Request plain text
curl -H 'Accept: text/plain' https://lingo.dev/en/cli
**# Request HTML (default)
**curl -H 'Accept: text/html' https://lingo.dev/en/cli
For Markdown requests, you should see:
Response header: Content-Type: text/markdown; charset=utf-8
Body: Raw Markdown content without HTML tags
For HTML requests, you should see your normal rendered page.
Further reading and resources
To learn more about how to implement this, check out the following resources:
**
Accept header documentation**
Next.js template by @cramforce
Set-up guide for Laravel by @retlehs
Set-up guide for static websites, Cloudflare Workers, and Caddy by @skeptrune.
Top comments (1)
Super useful — I just implemented this in a small Flask app, and it works perfectly. Might be worth mentioning FastAPI too — its content negotiation is really straightforward to extend. 🙌