DEV Community

Cover image for How ChatGPT Actually Finds Your Page: A Developer's Walkthrough of RAG
Asad Abdullah Zafar
Asad Abdullah Zafar

Posted on Originally published at loop2tech.com

How ChatGPT Actually Finds Your Page: A Developer's Walkthrough of RAG

#ai #llm #seo #webdev

ChatGPT doesn't read your website the way a person does. When it searches, it uses retrieval augmented generation (RAG): it rewrites the question into search queries, pulls a small set of passages from an index, and writes an answer grounded in those passages. The unit it retrieves is a passage of a few hundred words, not your page.

If you build or maintain sites, that changes what "SEO-ready" means.

The pipeline in four steps

  1. Query fan-out. One conversational message becomes several literal searches. "Best Shopify agency in Karachi for a skincare brand" turns into separate queries for agency, location and vertical.
  2. Candidate retrieval. Each query returns dozens of candidate passages, scored by vector similarity plus classic relevance signals.
  3. Re-ranking. A second model scores candidates for usefulness and trust and throws most of them away. The context window is finite, so usually fewer than ten sources survive.
  4. Grounded generation. The answer is written from the survivors, with citations to their URLs.

Your content competes twice: once to be retrieved, once to survive re-ranking.

The chunk is the unit, not the page

Retrieval systems split documents into chunks of roughly 200 to 800 words and embed each one separately. A 3,000-word guide enters the index as eight or nine independent passages.

So a section that only makes sense after the previous one gets retrieved alone, stripped of context, and judged incoherent. Start every H2 section by naming its subject. "Retrieval augmented generation pulls passages from a live index" survives extraction. "It pulls them from a live index" doesn't.

Three OpenAI user agents, three different jobs

User agent What it does Blocking it means
GPTBot Crawls content that may train models You opt out of training, not search
OAI-SearchBot Surfaces sites in ChatGPT search You disappear from ChatGPT search
ChatGPT-User Fetches a page for a user action Live lookups of your URL fail

A permissive baseline:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /
Enter fullscreen mode Exit fullscreen mode

The failure nobody checks: the edge

robots.txt is rarely the culprit. On one ecommerce build we inherited, the CDN's bot protection returned 403 to every AI fetcher while robots.txt was wide open and Googlebot passed fine. Rankings looked healthy; AI engines got nothing.

A quick first test from your terminal:

curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://yoursite.com/some-page
Enter fullscreen mode Exit fullscreen mode

A 403 or a challenge page here means something at the edge is filtering by user agent. It won't catch every rule, since some bot protection checks IP ranges instead, so confirm in your CDN's firewall logs too.

A retrieval readiness pass

  • Make sure the main content is in the server-rendered HTML, not injected after load
  • Restate the subject at the top of every section
  • Put numbers, sources and named entities in the paragraph that makes the claim
  • Check robots.txt and the edge layer for each AI user agent

Full article: Retrieval augmented generation explained: how ChatGPT finds your website

I'm Asad Abdullah Zafar. I build AEO, GEO and SEO systems at Loop2Tech, a technology studio in Karachi, Pakistan.

Top comments (0)