How to create a page that gets read by AI assistants

bilal ikram — Tue, 29 Sep 2026 23:43:18 +0000

An AI assistant does not read your whole page. It takes one section out and answers from that.

We read 15,592 sections the way a crawler does. Half were under 40 words. Only a quarter carried a number, a quotation or a link out. And 76.6% never named the brand whose site they were sitting on, because the copy says "we" and "our platform" instead.

None of that matters while a human is reading top to bottom. It matters when one section gets lifted out on its own and has to answer the question by itself.

Six things that change it:

Put the question in the H1
Give every heading an answer under it
Write sections of 40 to 350 words
One checkable fact in each section
Name yourself instead of saying "we"
Keep the facts out of JavaScript

If you only do one, do the third and fourth together: take the ten sections you most want quoted and get each one past 40 words with a number in it. The section, not the page.

Run a free AI AEO audit on your site here: https://babel42.io/ai-seo-audit

AEO #LLMSEO #GEO

Cloudflare blocks AI agent traffic by default from 15 September

bilal ikram — Tue, 08 Sep 2026 21:24:31 +0000

From 15 September, Cloudflare blocks "Agent" traffic by default on pages that carry ads. Agent means bots browsing live on someone's behalf, which includes the crawlers AI assistants send out to check a product before recommending it. Search crawlers stay allowed.

The default lands on new domains onboarding to Cloudflare, new sites added by existing customers, and every existing free-tier account. A configured paid zone is untouched on the day.

What Cloudflare counts as an agent

Cloudflare now sorts AI traffic into Search, Agent and Training. Search indexes your content to answer questions about it later. Training absorbs it into a model. Agent is the new one: automated activity acting in real time on a person's behalf, with a human waiting on the other end. Cloudflare's own examples are chat fetch bots such as ChatGPT-User, and browser-use agents driving Chrome.

That is the fetch that runs when someone asks an assistant about your product and it goes to look at your pricing page before answering. Block it and the assistant still answers. It just answers from whatever it already had.

robots.txt does not cover this

The block happens at Cloudflare's edge, before the request reaches your origin. robots.txt is a request you make of a crawler that chooses to honour it, and it has no bearing on what the edge does with the connection. Allowing has to be configured per zone, in Cloudflare's AI traffic controls or as a WAF rule matching the user agent.

The agents worth allowing

ChatGPT-User          # ChatGPT (also ChatGPT-User/2.0)
OAI-SearchBot         # ChatGPT
Claude-User           # Claude
Claude-SearchBot      # Claude
Perplexity-User       # Perplexity
PerplexityBot         # Perplexity
Google-Agent          # Gemini
GoogleAgent-Mariner   # Gemini
Googlebot             # Gemini
GrokBot               # Grok
xAI-Grok              # Grok
Grok-DeepSearch       # Grok

The training crawlers, if you still want those blocked

GPTBot
ClaudeBot
anthropic-ai
meta-externalagent
Bytespider
Google-Extended

Watch the multi-purpose crawlers

Cloudflare applies the most restrictive matching rule to a crawler that does more than one job, and names Googlebot, Applebot and Bingbot as ones that get caught by a Training block. A blanket "block training" can therefore cost you ordinary search indexing. Set the categories separately and check what each named agent falls under before you save.

Decide it rather than inherit it

If you sell ad space, the block may be exactly what you want, and Cloudflare is handing it to you for free. The bad outcome is having the default applied to you on the fifteenth without a decision, then working out months later why assistants describe your product from a stale snapshot while your search traffic looks fine.

Check the zone before the fifteenth. It is a five-minute job now and an invisible problem afterwards.

DEV Community: bilal ikram

How to create a page that gets read by AI assistants

AEO #LLMSEO #GEO

Cloudflare blocks AI agent traffic by default from 15 September

The agents worth allowing

The training crawlers, if you still want those blocked