Originally published at webofmike.com on 2026-09-24. The demo repo and every command in it were run before publishing.
Agents parse HTML badly and expensively. The usual fix is to make the site emit markdown: add a per-page index.md, publish an llms.txt, change the build. That works when you own the site. It does not work for the docs site you inherited, the vendor knowledge base, or the internal wiki nobody will redeploy this quarter.
So I put the decision in the gateway instead. One origin that serves HTML and only HTML, one 200-line shim, and an agentgateway route that picks between them on the Accept header. The origin is not modified and does not know any of it happened. Code at themsquared/agent-content-negotiation.
The interesting part is not that it works. It is the two ways the naive version fails quietly, and a third failure I shipped into my own converter and did not notice.
How the routing works
The gateway matches a regex against the request's Accept header. Requests that mention text/markdown go to the shim, which fetches from the origin and converts. Everything else goes straight to the origin. Vary: Accept is set so caches do not serve one variant to the other audience.
Five scenarios against a single URL, changing nothing but the header:
SCENARIO 2 an agent that asks -> Accept: text/markdown
PASS served the markdown variant
PASS content-type is text/markdown
PASS body is a markdown heading
PASS code fence survived
PASS link became a markdown link
PASS no HTML doctype leaked
PASS markdown body contains no HTML
PASS site chrome (nav/script/footer) was dropped
SCENARIO 5 payload size, same page, same origin
html 894 bytes
markdown 420 bytes
PASS markdown variant is smaller (474 bytes less)
assertions: 15 passed, 0 failed
That size difference is the whole economic argument in one line. Same page, same origin, less than half the bytes, and none of the remaining bytes are nav, script tags or footer boilerplate that an agent has to read and discard.
Gotcha one: your agents are asking for */*
This is the one that will actually bite you.
SCENARIO 3 THE GOTCHA -> Accept: */* (curl's default, and a lot of agent HTTP clients)
PASS served the HTML variant
*/* does not mention markdown. The agent route does not match. The agent gets the HTML site and a parsing problem it will blame on your docs.
What makes this bad is that it is invisible from the server side. There is no error, no 406, nothing in the logs that looks wrong. The request is well formed and the response is a valid 200. You will conclude the feature works, because when you test it you will type Accept: text/markdown by hand, and your agents never will.
If you deploy this, the first thing to do is not to celebrate the markdown route. It is to look at the distribution of Accept headers you are actually receiving and find out how much of your agent traffic is wildcards.
Gotcha two: a header regex is not content negotiation
SCENARIO 4 THE LIMITATION -> Accept: text/html, text/markdown;q=0.1
PASS served the markdown variant anyway
The client said it would much rather have HTML. q=0.1 on markdown is close to "only if you have nothing else." A route match cannot read that, because a regex tests for presence, not preference.
This is a routing match, not RFC 9110 proactive negotiation, and the distinction matters the moment a well-behaved client sends a weighted header. It is a real limitation of doing this at the route layer rather than in an application that parses the header properly. Worth knowing before you tell people the gateway does content negotiation, because it does not. It does routing that resembles it.
The bug I shipped into my own converter
The shim strips site chrome: nav, script, footer. It did this by opening a skip region when it saw a chrome tag and closing it on the matching end tag.
<link> is a void element. It has no end tag. So the skip region opened and never closed, the converter consumed the rest of the document, and it returned an empty document under HTTP 200 with Content-Type: text/markdown and one newline of body.
Every status-code check passed. The content type was right. The response was fast. An agent consuming this gets a page that exists and says nothing, and the most likely outcome is that it concludes your documentation is empty rather than that your gateway is broken.
Two lessons, and the second is the general one:
- Separate container tags from void elements when you walk HTML.
<link>,<meta>,<img>,<br>,<hr>and friends never close. - Assert on body content, not on status. A 200 with the right content type is not evidence that anything was served. This is the same shape as the false-pass I hit in a different demo the same week, where an assertion on the absence of a string passed against a 404 page. Both bugs are a test that cannot distinguish success from a specific kind of nothing.
A smaller one worth ten minutes of your life
agentgateway lowercases response header names that you inject in config. X-Served-Variant comes back as x-served-variant. HTTP header names are case-insensitive so nothing is wrong, but my first assertion pass grepped for the capitalised form and failed on a response that was completely correct. Compare header names case-insensitively in tests.
Running it
Docker with Compose v2, nothing else. The shim is Python standard library. Validated on Docker 29.7.2 and Compose v5.4.0, arm64 macOS, against agentgateway v1.5.0 and python:3.12-slim.
git clone https://github.com/themsquared/agent-content-negotiation
cd agent-content-negotiation
docker compose up -d --build
./scripts/demo.sh
Expected: assertions: 15 passed, 0 failed.
docker compose down
I re-ran the whole suite while writing this post rather than quoting the README, so every number above is from a run on the day of publication.
What this is not
It is not RFC 9110 content negotiation, per scenario 4. It is not a general HTML-to-markdown converter either; the shim is deliberately small enough to read in one sitting, and a real deployment would put something more capable behind the same route.
What the pattern gives you is the ability to make an origin you do not control agent-readable at the traffic layer, and to see what your agents are actually requesting while you do it. Those two things are worth more together than separately, because the header distribution is what tells you whether the markdown route is doing anything at all.
For the gateway side of the same product, per-key LLM budgets covers the other thing I keep wanting at this layer: limits that live with the caller's identity rather than the route.
The repo is themsquared/agent-content-negotiation, Apache-2.0, and the demo takes about two minutes.
Frequently asked questions
How do I serve markdown to AI agents without changing my website?
Put the decision in the gateway. Match on the Accept request header and route agent traffic through a converter while browser traffic goes to the origin unchanged. In this demo an agentgateway route matches a regex against Accept and forwards to a 200-line Python shim, so the HTML-only origin is never modified and does not know it happened.
Why does my agent still get HTML after I set up markdown content negotiation?
Almost certainly because it sends Accept: /, which is curl's default and the default of many agent HTTP clients. A wildcard does not mention text/markdown, so a header match on markdown does not fire and the agent silently receives HTML. The failure is invisible from the server side, since the request looks completely normal.
Is Accept header route matching the same as HTTP content negotiation?
No. Route matching tests whether a string appears in the header. RFC 9110 proactive negotiation weighs media types by their q-values. A client sending Accept: text/html, text/markdown;q=0.1 is saying it strongly prefers HTML, but a regex match on markdown routes it to markdown anyway. The demo reproduces this as its fourth scenario.
Canonical version, with machine-readable markdown at https://webofmike.com/markdown-for-agents-at-the-gateway/index.md: https://webofmike.com/markdown-for-agents-at-the-gateway/
Top comments (1)
The wildcard header failure is the sharpest operational lesson here because it can return a valid page while silently defeating the agent route. I would make header distribution a first class metric in production so the route is measured by real client behavior rather than a hand tested happy path.