Many sites render their content with JavaScript in the browser, which means the first response an agent receives is an almost empty shell. A person waits a moment and the page fills in. An agent that reads the raw response sees a loading state and little else, and it judges the site on that.
This is the single most common reason a capable site is invisible to agents. The content exists, but it arrives after the agent has already read and moved on. Search crawlers have partly adapted to this over years. Many AI agents and fetchers have not, and they take the first response at face value.
The fix is to serve the real content in the first response for clients that need it. Prerendering renders the page on the server or at the edge and returns finished HTML, so an agent reads the content immediately. A cleaner option for agents is to serve a markdown version of the page on request, which skips the rendering question entirely and costs a fraction of the tokens.
The decision is not all or nothing. A site can keep its interactive experience for people and serve prerendered or markdown content to agents and bots, deciding by the request. On turva.dev that decision lives in a Cloudflare Worker that detects the client and returns the right form.
For an audit of how a site renders for agents, contact info@turva.dev.
Related
- Serving markdown to agents
- Response headers that help agents
- Common agent-readiness gaps on marketing sites
Originally published at https://turva.dev/guides/prerendering-for-agents
Top comments (1)
This is directly relevant to something I've been meaning to check on my own site. I run a Next.js tools site, and I've been assuming the App Router's server components mean I'm automatically safe here, but that assumption probably doesn't hold for anything that hydrates client-side after the initial HTML, which is most of the interactive tool pages themselves.
The client-detection-in-a-Worker approach is a clean solve for the "not all or nothing" problem, but I'd guess the harder part is keeping the served content in sync, if the markdown/prerendered version and the live interactive page can drift (a tool description gets updated in one but not the other), that seems like a maintenance burden that grows quietly until an agent cites something stale. Curious whether that's handled by generating both from the same source at build time, or if it's a separate content path that has to be kept in sync manually.
Also curious about the detection mechanism itself, is it User-Agent sniffing (which agents can trivially spoof or fail to set correctly), or something more structural like request headers or behavior patterns? That feels like the part most likely to have false negatives as new agents show up that don't match whatever signature the Worker is checking for.