The web already has the content
The web already has the content. The challenge is not that AI agents cannot fetch a URL. The challenge is that the representation they receive, typically the same HTML a browser renders, is optimized for visual display, not for machine consumption. HTML pages are laden with navigation, scripts, styles, and layout markup that add token cost and dilute semantic signal for retrieval or summarization. The interesting engineering problem is serving a cleaner, more token-efficient representation from the same URL while keeping existing caches intact. Content negotiation via the Accept header provides a standard mechanism, but it requires careful implementation to avoid fragmenting or poisoning caches.
What Accept: text/markdown changes
The Accept request header tells a server which media types the client can process. Browsers typically send a broad value like text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. An AI agent that prefers Markdown can send Accept: text/markdown, text/html;q=0.9,*/*;q=0.8 or a simpler Accept: text/markdown. This is a signal, not a command. The server is free to respond with text/html if that is all it provides. The media type text/markdown is registered for Markdown documents and is the appropriate type to use for this representation.
What changes when a server supports this is the representation it returns. Instead of the full HTML page, the server responds with a Markdown document containing the core content. This is not a replacement for HTML; it is an alternative representation optimized for a specific use case. HTTP already has a mechanism for this kind of negotiation. The server decides which representation to serve based on the client's expressed preferences.
The HTTP contract clients and servers need
Content negotiation relies on a few precise rules. When a server supports both HTML and Markdown representations for a resource, it must evaluate the client's Accept header to choose the appropriate one. Parsing Accept is not a simple substring match. It requires parsing the media types and their associated quality values (q-values) to determine the client's preference order. A client sending text/markdown;q=0.8, text/html;q=1.0 prefers HTML, so the server should choose HTML when it has that representation. The */* wildcard can match any media type, but a specific media range takes precedence over it.
The response must then declare what it is serving via the Content-Type header. For Markdown, this is text/markdown; charset=utf-8. For HTML, it is text/html; charset=utf-8. The server must not return one type while declaring another. A Content-Type: text/html response to a request for text/markdown is a contract violation, not a fallback strategy. If no available representation matches the client's constraints and the server is unwilling to supply a default, it can return 406 Not Acceptable, as defined by RFC 9110. It should not silently label HTML as Markdown.
Caching is part of the feature
A Vary: Accept response header is not optional here. Without it, a cache, whether a browser cache, a reverse proxy, or a CDN, might store the Markdown response for a URL and serve it to a subsequent request that expects HTML, or vice versa. The Vary header tells the cache that the response varies based on the Accept request header. The cache can then use that request-header value when deciding whether a stored response matches, keeping HTML and Markdown variants from being mixed.
This is where the engineering gets precise. Vary: Accept enables correct caching, but it can also cause cache fragmentation. Clients can send many different Accept strings that describe roughly the same preference, leading to multiple cache entries for one URL and a lower hit rate. The cure is provider-specific cache-key policy or normalization, not an assumption that every CDN treats Vary the same way. A good rollout checks the CDN's documentation and inspects real cache behavior with more than one header spelling.
CDNs also have specific handling for the Vary header. Akamai's current documentation describes a policy in which arbitrary Vary values can prevent caching unless the property is configured for them. Other providers document different behavior, and those policies change. Treat Vary: Accept as an application contract and a CDN configuration item, not as a guarantee that every edge will cache the variants in the same way.
A small implementation that does not lie
The simplest implementation involves checking the Accept header in a middleware or request handler and serving the appropriate representation. Many frameworks have libraries for parsing Accept headers; do not write a custom parser unless you enjoy handling edge cases. The route fragment below uses Express and the accepts package. The content lookup is application-specific, but the negotiation and response headers are the part worth copying.
This example uses Node.js with Express and the accepts library:
const express = require('express');
const accepts = require('accepts');
const app = express();
app.get('/article/:slug', (req, res) => {
const content = getContentForSlug(req.params.slug); // application-specific lookup
const available = ['text/html'];
if (content.markdown) available.push('text/markdown');
// The package handles q-values and media-range specificity.
const choice = accepts(req).types(available);
res.vary('Accept');
if (choice === false) {
return res
.status(406)
.set('Content-Type', 'text/plain; charset=utf-8')
.send('No acceptable representation is available.');
}
if (choice === 'text/markdown') {
return res
.set('Content-Type', 'text/markdown; charset=utf-8')
.send(content.markdown);
}
return res
.set('Content-Type', 'text/html; charset=utf-8')
.send(content.html);
});
The available list contains only representations that the origin can actually produce. That detail prevents a request for Markdown from being selected when no Markdown body exists. It also makes an unsupported Accept choice visible as 406 instead of silently returning a mislabeled body.
For a static documentation site, the same idea can be implemented with explicit .html and .md files. The NGINX configuration below deliberately supports one narrow policy: it chooses the Markdown file only when text/markdown appears with no lower q-value. It is not a general Accept parser.
# http {} context. Each document has matching .html and .md files.
map $http_accept $doc_file {
default $uri.html;
"~*^text/markdown([[:space:]]*;[[:space:]]*q=1([.]0*)?)?([[:space:]]*,|$)" $uri.md;
}
server {
root /srv/site;
# /docs/guide -> /srv/site/docs/guide.html or /srv/site/docs/guide.md
location /docs/ {
types {
text/html html;
text/markdown md;
}
try_files $doc_file =404;
add_header Vary Accept always;
}
}
The example is useful when the representation is already stored as a file. If the endpoint must honor every q-value combination, keep negotiation in application code or a tested edge function instead of growing an untested regular expression. Either way, preserve Vary: Accept on successful responses and on errors generated by the variant route.
When a separate Markdown URL is safer
Content negotiation is elegant but not the only solution. Some sites may find it simpler or safer to publish their Markdown content at a separate, predictable URL. For example, example.com/article.md or example.com/markdown/article-slug. This approach avoids all the complexity of Accept header parsing and Vary header management.
It is also more predictable for agents. An agent can fetch the Markdown representation directly without relying on the server correctly implementing content negotiation. This is a defensible choice, especially for documentation sites or content repositories that already manage content in Markdown. A separate URL eliminates the risk of cache fragmentation because the URL itself is the differentiator. There is no need for Vary: Accept; the URL is the cache key. However, it also breaks the principle of one resource, one URL, and can lead to link rot if the Markdown URL is not treated as a canonical representation.
The decision is an engineering trade-off. Content negotiation is more elegant and maintains a single canonical URL. A separate .md URL is more robust and simpler to cache and test. For sites with existing, heavy CDN caching and complex request routing, the separate URL often proves safer to implement, as it sidesteps potential CDN quirks with Vary handling.
How to test an AI-ready endpoint
A testing checklist is essential for validating a content-negotiation endpoint. Run these requests against staging before production, and inspect both the status and the response headers.
For a default HTML request, omit Accept or use the broad value a browser sends:
curl -sI https://example.com/article | grep -E "Content-Type|Vary"
Expected: Content-Type: text/html; charset=utf-8 and Vary: Accept.
For an explicit Markdown request, ask for the representation directly:
curl -sI -H "Accept: text/markdown" https://example.com/article | grep -E "Content-Type|Vary"
Expected: Content-Type: text/markdown; charset=utf-8 and Vary: Accept.
Then test a q-value fallback. This client prefers HTML but can still process Markdown:
curl -sI -H "Accept: text/markdown;q=0.8, text/html;q=1.0" https://example.com/article | grep -E "Content-Type|Vary"
Expected: Content-Type: text/html; charset=utf-8 and Vary: Accept. The server should respect the higher q-value for HTML.
Finally, ask for a type the server does not provide and inspect the cache headers:
curl -sI -H "Accept: application/json" https://example.com/article
curl -sI -H "Accept: text/markdown" https://example.com/article | grep -E "Vary|CF-Cache-Status|X-Cache"
The first request should return 406 Not Acceptable when the server declines to supply a default. The second should show Vary: Accept plus whatever cache status headers the provider documents. A passing curl command does not prove that a CDN is configured correctly. Make repeated requests with different Accept strings and confirm that the edge serves the right variant without mixing them.
What adoption has to prove
Adopting text/markdown content negotiation is a practical step, but it solves only one problem: delivering a cleaner representation to clients that request it. It does not make a site "AI-ready" by itself. The Markdown representation must be of high quality. It must strip out the cruft of templates and scripts while preserving the essential content hierarchy. The conversion process from HTML to Markdown is itself a separate engineering task, and a poorly generated Markdown document can be as useless as the raw HTML.
The caching implications are real. A site that serves Vary: Accept will see additional cache entries. This is the cost of doing business. The benefit is a better experience for AI agents and, potentially, lower token consumption and faster processing for the client. The adoption has to prove that the benefit outweighs the cost. For many content-heavy sites, the cost is negligible when a CDN normalizes Accept headers. For others, the fragmentation could be a concern. The only way to know is to measure. Monitor cache hit ratios after enabling the feature. Watch for an increase in cache misses. If the hit ratio drops unacceptably, consider normalizing the Accept header at the CDN level or using a separate Markdown URL instead.
The web has always supported multiple representations of a resource. The Accept header is a mature part of HTTP, not a new agent-only protocol. If you want to try this pattern, start with one content route, generate Markdown from the same source as HTML, implement the negotiation with a tested library, set Vary: Accept, and observe the cache before expanding the rollout. If your CDN cannot give you a clear variant policy, use an explicit .md URL instead.
Originally published on Dispatch.
Top comments (0)