Serving Markdown to AI agents from a Cloudflare Worker, without a second copy of the site

AI agents read web pages the expensive way. They download the HTML, throw away the navigation, scripts and styles, and try to recover the text. Some of them now send Accept: text/markdown and take Markdown if the server offers it.

I wanted our landing page to answer that request without keeping a second Markdown copy of the site in sync by hand. The site is static files behind a Cloudflare Worker, so the Worker converts the HTML on the way out. This post walks through that module: about 160 lines, no dependencies, built on HTMLRewriter.

Step 1: decide when to convert

The rule is narrow on purpose. Convert only a successful GET for an HTML asset, and only when the client actually asked for Markdown:

export async function serveMarkdownForAgents(request, assets) {
  const response = await assets.fetch(request);
  if (request.method !== 'GET' || !wantsMarkdown(request) || !response.ok
      || !response.headers.get('content-type')?.toLowerCase().startsWith('text/html')) {
    return response;
  }
  // ...convert
}

Images, CSS, the sitemap and everything else pass through untouched, even when the request prefers Markdown. Browsers never send text/markdown, so people keep getting HTML.

wantsMarkdown reads the Accept header properly instead of searching it for a substring. It splits the media ranges, looks for text/markdown, and respects the quality value:

function wantsMarkdown(request) {
  return request.headers.get('accept')?.split(',').some((part) => {
    const [mediaType, ...parameters] = part.trim().toLowerCase().split(';');
    if (mediaType !== 'text/markdown') return false;
    const quality = parameters.find((parameter) => parameter.trim().startsWith('q='));
    if (!quality) return true;
    const value = Number(quality.trim().slice(2));
    return value > 0 && value <= 1;
  }) ?? false;
}

The q=0 case matters. Accept: text/html, text/markdown;q=0 means "anything but Markdown", and a naive includes('text/markdown') would send exactly the wrong thing. There is a test for that case, and one confirming that a non-HTML asset is unchanged when Markdown is requested.

Step 2: convert with HTMLRewriter

HTMLRewriter is a streaming parser. You register handlers for selectors and get callbacks for elements and text chunks. There is no DOM tree, so the converter keeps a little state of its own:

• an output array of string chunks;
• a stack of list counters, so <ol> items get 1., 2. and <ul> items get -;
• a skippedDepth counter for subtrees that shouldn't appear at all.

Everything that is page chrome is skipped: head, nav, footer, script, style, noscript, svg, button, template, plus anything marked hidden, aria-hidden="true", role="navigation" or with a skip-link class. When such an element opens, the depth goes up, and onEndTag brings it back down. Text callbacks only write when the depth is zero.

if (OMIT.has(tag) || hidden) {
  element.remove();
  if (!VOID.has(tag)) {
    skippedDepth += 1;
    element.onEndTag(() => { skippedDepth -= 1; });
  }
  return;
}

The VOID check is easy to miss. Void elements such as <img> or <br> have no end tag, so registering onEndTag on them would leave the counter stuck and silently drop the rest of the page.

The mapping itself is ordinary. Headings become # prefixes. Paragraphs and sections become blank lines. strong and em become ** and *, and pre becomes a fenced block. Two cases need care:

• Links are rewritten to absolute URLs, and only http, https, mailto and tel survive. A relative /privacy becomes https://.../privacy, which an agent can follow, and a javascript: link is dropped.
• Images appear only when they have both alt text and a resolvable src. A decorative image without alt text carries nothing an agent can use.

Text nodes are decoded (&,  , numeric entities) and whitespace is collapsed. After the stream ends, one cleanup pass trims spaces around newlines and caps blank lines at one.

Step 3: keep the metadata

The parts of <head> that matter to an agent are kept, just not as HTML. Separate handlers collect the <title>, the meta description, the canonical URL and the text of any application/ld+json blocks. The output starts with front matter:

---
title: "..."
description: "..."
url: "https://example.com/"
---

The JSON-LD is appended at the end under a ## Structured data heading. The structured data a search engine sees, such as the FAQ and the product details, also reaches an agent that never parses HTML.

Step 4: fix the headers

This is the part that breaks caches if you skip it. The body changed, so several headers copied from the HTML asset are now lies:

headers.set('content-type', 'text/markdown; charset=utf-8');
// add Vary: Accept unless it is already there (or Vary: *)
for (const name of ['content-length', 'content-encoding', 'content-md5',
                    'etag', 'content-range', 'accept-ranges', 'digest']) headers.delete(name);

Vary: Accept is what stops a shared cache from serving the Markdown version to the next browser that asks for the same URL. Dropping the old ETag avoids the opposite mistake: a conditional request answered with a 304 that points at the HTML body.

Last, the whole conversion is wrapped in try/catch. If anything throws, the original HTML response goes out. An agent getting HTML is a degraded result; a 500 is a broken one.

Trying it

curl -s -H "Accept: text/markdown" https://talkaside.com/ | head -20

The same URL in a browser still returns the normal page.

Limits

The converter is tuned to this site's semantic HTML, and the source says so in a comment. Tables come out as pipe-separated rows without the |---| header separator, so they aren't strictly valid Markdown tables. Anything rendered by JavaScript after load isn't there, because the Worker only sees the static HTML. A site with rich widgets or complex tables would be better served by a real HTML-to-Markdown library, at the cost of a bundled dependency.

For a marketing page made of headings, paragraphs, lists and an FAQ, 160 lines of HTMLRewriter were enough, and there is only one source of truth.

I'm building Talkaside, a Windows app for job interviews and calls in another language.

Drafted with AI assistance from this project's code and logs.