Everyone's writing think-pieces about "optimizing for AI search." I wanted numbers. So I pulled together the 2025-2026 data from Cloudflare, Vercel/MERJ, Ahrefs and the AI providers' own docs. Some of it genuinely changed how I build sites. Here's the reality check, dev to dev.
1. The AI crawlers do not run your JavaScript
This is the big one. A Vercel/MERJ analysis of 500M+ crawler fetches found that none of the major dedicated AI crawlers execute JavaScript — not OpenAI's GPTBot/OAI-SearchBot/ChatGPT-User, not Anthropic's ClaudeBot, not PerplexityBot, not Meta or ByteDance's Bytespider. They fetch your raw HTML and leave. They do download JS files (~11.5% of ChatGPT's fetches, ~23.8% of Claude's) — they just never run them.
The exception is Google: its Web Rendering Service is shared across Search and Gemini, so Google-Extended renders JS the same way Googlebot does.
Translation: if your content only exists after client-side hydration, every AI engine except Google sees an empty shell.
# see what the bots see
curl -A "GPTBot" https://yoursite.com/page | grep "your headline"
If that's empty, you need SSR, SSG, or prerendering.
2. AI crawl traffic is exploding — and barely sends anything back
Cloudflare's data shows AI crawler activity up sharply year over year, with GPTBot's volume up ~305% and reaching ~11.7% of all crawler requests by mid-2025. But roughly 86% of that crawling is for model training, not live search retrieval.
And the referrals? Lopsided doesn't cover it. Measured crawl-to-referral ratios hit ~38,065:1 for Anthropic and ~1,091:1 for OpenAI. They read enormously, cite rarely. AI tools still send under 1% of outbound web traffic today.
3. The SERP is going zero-click
By early 2026, roughly 68% of Google searches ended without a click, and pages with an AI Overview above them saw click-through drop by around 60%. Being "on page one" is worth less every quarter; being the cited source is worth more.
4. Two overhyped tactics, with the data attached
-
llms.txt: adoption grew ~8.8x, but in one May 2026 measurement 97% of llms.txt files received zero requests. OpenAI, Anthropic and Google all point site owners to
robots.txtinstead. Add it if you like — it's a cheap static file — but don't expect it to do the work. - Schema/JSON-LD as a citation hack: controlled analysis found schema presence does not independently predict AI citation once you correct for content quality. Schema still earns rich results and disambiguates entities — just don't treat it as a magic AI-ranking lever.
What actually moves the needle
Boring, durable engineering:
-
Server-render your content — verify with
curl, not DevTools. -
Allow the retrieval bots you want cited by in
robots.txt; block only what you must. - Answer-first, semantic HTML — clean headings, real lists, a direct answer per section.
- Core Web Vitals, INP included — fast pages get crawled more thoroughly.
None of this is a hack. It's the same discipline as good, crawlable web engineering — which is why developers, not marketers, tend to be the ones who move AI visibility. It's the technical backbone of modern AI SEO and it overlaps almost entirely with classic technical SEO.
Build for the crawler that can't run your JavaScript, and you're ready for whatever the answer engines do next.
Sources: Cloudflare Radar, Vercel/MERJ crawler study, Ahrefs brand-mentions analysis, and OpenAI/Anthropic/Google bot docs (2025-2026).
Top comments (0)