DEV Community: bilal ikram

How we use AI assistant data to find backlinks, fix our own site and see why competitors get recommended

bilal ikram — Wed, 11 Nov 2026 00:45:10 +0000

As a new business, we didn't show up in AI assistants. We build Babel42, an AI visibility tool, and what we could see is what got our competitors mentioned, and better still, what got them recommended. So we're using our own tool on ourselves.

The setup is simple. We create a few different buyer personas, and each one asks the assistants several questions. Then we see who gets recommended and the reason the assistant gave for it. With normal SEO you'd just go after keywords. But no one uses AI search the same way, so picking questions to target is a bit of a guessing game. Running several personas and informing prompts with keyword volume data from search helps.

Very useful for us has been the list of cited sources. We pull the list of sources the assistants cited through our MCP.

Then we remove the obvious ones: Reddit, YouTube, competitors, and the big publications that are out of reach for now. What's left is a list of smaller sites we can actually pitch. The full list ran to 66 pages, which would be a pain to sort by hand. Through the MCP it's a couple of prompts.

Then we look at why the competitors won. We started with the smaller fish in our pond rather than the big names. A few of the reasons the buyers gave:

PingMyBrand won because it was the only option with confirmed agency pricing, $199 a month for up to 10 brands. tutka won on an agency plan at $149 a month with GA4 built in. LLM Metrix won because its case study was the most transparent, with an actual number and timeframe in it. ClayHog won partly because it offered to share anonymised case studies.

We use that information to improve our site, add content thats mentioned regularly, add clarity and it's why are publishing original research.

How to create a page that gets read by AI assistants

bilal ikram — Tue, 29 Sep 2026 23:43:18 +0000

An AI assistant does not read your whole page. It takes one section out and answers from that.

We read 15,592 sections the way a crawler does. Half were under 40 words. Only a quarter carried a number, a quotation or a link out. And 76.6% never named the brand whose site they were sitting on, because the copy says "we" and "our platform" instead.

None of that matters while a human is reading top to bottom. It matters when one section gets lifted out on its own and has to answer the question by itself.

Six things that change it:

Put the question in the H1
Give every heading an answer under it
Write sections of 40 to 350 words
One checkable fact in each section
Name yourself instead of saying "we"
Keep the facts out of JavaScript

If you only do one, do the third and fourth together: take the ten sections you most want quoted and get each one past 40 words with a number in it. The section, not the page.

Run a free AI AEO audit on your site here: https://babel42.io/ai-seo-audit

AEO #LLMSEO #GEO

Cloudflare blocks AI agent traffic by default from 15 September

bilal ikram — Tue, 08 Sep 2026 21:24:31 +0000

From 15 September, Cloudflare blocks "Agent" traffic by default on pages that carry ads. Agent means bots browsing live on someone's behalf, which includes the crawlers AI assistants send out to check a product before recommending it. Search crawlers stay allowed.

The default lands on new domains onboarding to Cloudflare, new sites added by existing customers, and every existing free-tier account. A configured paid zone is untouched on the day.

What Cloudflare counts as an agent

Cloudflare now sorts AI traffic into Search, Agent and Training. Search indexes your content to answer questions about it later. Training absorbs it into a model. Agent is the new one: automated activity acting in real time on a person's behalf, with a human waiting on the other end. Cloudflare's own examples are chat fetch bots such as ChatGPT-User, and browser-use agents driving Chrome.

That is the fetch that runs when someone asks an assistant about your product and it goes to look at your pricing page before answering. Block it and the assistant still answers. It just answers from whatever it already had.

robots.txt does not cover this

The block happens at Cloudflare's edge, before the request reaches your origin. robots.txt is a request you make of a crawler that chooses to honour it, and it has no bearing on what the edge does with the connection. Allowing has to be configured per zone, in Cloudflare's AI traffic controls or as a WAF rule matching the user agent.

The agents worth allowing

ChatGPT-User          # ChatGPT (also ChatGPT-User/2.0)
OAI-SearchBot         # ChatGPT
Claude-User           # Claude
Claude-SearchBot      # Claude
Perplexity-User       # Perplexity
PerplexityBot         # Perplexity
Google-Agent          # Gemini
GoogleAgent-Mariner   # Gemini
Googlebot             # Gemini
GrokBot               # Grok
xAI-Grok              # Grok
Grok-DeepSearch       # Grok

The training crawlers, if you still want those blocked

GPTBot
ClaudeBot
anthropic-ai
meta-externalagent
Bytespider
Google-Extended

Watch the multi-purpose crawlers

Cloudflare applies the most restrictive matching rule to a crawler that does more than one job, and names Googlebot, Applebot and Bingbot as ones that get caught by a Training block. A blanket "block training" can therefore cost you ordinary search indexing. Set the categories separately and check what each named agent falls under before you save.

Decide it rather than inherit it

If you sell ad space, the block may be exactly what you want, and Cloudflare is handing it to you for free. The bad outcome is having the default applied to you on the fifteenth without a decision, then working out months later why assistants describe your product from a stale snapshot while your search traffic looks fine.

Check the zone before the fifteenth. It is a five-minute job now and an invisible problem afterwards.