DEV Community

Juan Camilo Auriti
Juan Camilo Auriti

Posted on

The 4-Layer Model for AI Search Readiness: What I Learned Auditing 360 Sites

#ai #seo #webdev #programming

The 4-Layer Model for AI Search Readiness: What I Learned Auditing 360 Sites

I audited 360 domains against a structured AI-search-readiness framework. The average score was 54.1 out of 100. Only 24.7% reached "Good" or above. The first perfect 100/100 didn't appear until July 2026.

This post walks through the four-layer model I used — Access, Orientation, Understanding, Quotability — with real code examples for each layer, the benchmark data behind it, and the infrastructure mistakes I kept seeing.

If you're a developer or tech lead, this is the framework to operationalize before your marketing team asks why ChatGPT doesn't cite your docs.

The Core Thesis: AI Search Selects, It Doesn't Rank

Traditional SEO is a ranking problem: optimize pages, climb positions, compete for slots on a results page.

Generative search — ChatGPT, Perplexity, Google AI Overviews — is a selection problem. A model receives a query, decides which sources to consult, extracts a passage, and either cites the source or paraphrases without attribution. There is no page two. You are either quoted, or you are invisible.

Getting selected requires four things to work in sequence:

  1. Access — the AI crawler can reach your content
  2. Orientation — it can find what matters on your site
  3. Understanding — it can parse what your organization and pages are about
  4. Quotability — it can extract a self-contained passage to quote

The key principle: fix access before schema, fix schema before content. Wrong order = wasted work. I saw this pattern repeatedly — teams rewriting content for "AI optimization" while their robots.txt blocked GPTBot.

Let's go layer by layer with code.

Layer 1: Access — Can the AI Crawler Reach You?

Access is robots.txt, static HTML delivery, server response behavior, and rendering. It's the least glamorous layer and the most common failure point.

There are at least 11 AI crawlers actively indexing the web. Each has a distinct user-agent token:

Crawler User-Agent Token Operator
GPTBot GPTBot OpenAI
OAI-SearchBot OAI-SearchBot OpenAI (Search)
PerplexityBot PerplexityBot Perplexity
ClaudeBot ClaudeBot Anthropic
Claude-SearchBot Claude-SearchBot Anthropic (Search)
Googlebot Googlebot Google
Google-Extended Google-Extended Google (AI training)
Applebot Applebot Apple
CCBot CCBot Common Crawl
Bytespider Bytespider ByteDance
Diffbot Diffbot Diffbot

The benchmark found an average of 23.2 bots allowed per site — but that average masks a long tail of sites blocking exactly the crawlers they need.

robots.txt: Allow the AI Crawlers

Here's a minimal robots.txt that explicitly allows the major AI crawlers while keeping your private paths locked down:

# Allow major AI crawlers access to public content
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: Diffbot
Allow: /

# Block private/admin paths from all crawlers
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/internal/

# Sitemap
Sitemap: https://example.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

The Rendering Trap

A subtler access issue: client-side rendering. If your site is a SPA that returns an empty <div id="root"></div> on the initial HTML response, many AI crawlers see nothing. GPTBot and PerplexityBot do not execute JavaScript reliably. They read the static HTML.

If your content lives behind a React/Vue/Svelte hydration step, you need either:

  • SSR (server-side rendering) or SSG (static site generation) so the HTML contains the content, or
  • A prerendering layer that serves cached HTML to bot user-agents

Check what a crawler actually sees:

curl -A "GPTBot" https://example.com | grep -i "your main heading"
Enter fullscreen mode Exit fullscreen mode

If that returns nothing, your content is invisible to the crawler. Fix this before anything else.

Layer 2: Orientation — Can It Find What Matters?

Once a crawler reaches your site, it needs to know what's important. Orientation covers llms.txt, sitemaps, RSS feeds, and priority URL signals.

This was the worst-performing category in the entire audit:

  • AI Discovery adoption: 17.5%
  • AI Discovery efficiency: 10%

The llms.txt numbers tell a specific story:

  • llms.txt adoption: 54.2% — but down from 58.3% in June. Sites are removing it.
  • Full llms.txt: only 26.9% — the rest are partial or malformed.

A broken llms.txt is worse than none. It sends a model a map with missing streets.

Minimal llms.txt Example

The llms.txt standard is a plain-text file at the root of your site that gives AI crawlers a structured summary of your content:

# Example Company

> Example Company builds developer infrastructure for AI-powered search.

## Docs
- [Getting Started](https://example.com/docs/getting-started): Quick start guide
- [API Reference](https://example.com/docs/api): Full REST API documentation
- [SDK Guide](https://example.com/docs/sdk): SDK installation and usage

## Product
- [Features](https://example.com/features): Feature overview and comparison
- [Pricing](https://example.com/pricing): Pricing tiers and FAQ

## Blog
- [Blog Index](https://example.com/blog): Engineering and product blog

## Optional
- [About](https://example.com/about): Company background and team
- [Contact](https://example.com/contact): Contact information
Enter fullscreen mode Exit fullscreen mode

Key rules:

  • The # line is the site title
  • The > line is a one-sentence summary
  • ## sections group links
  • Each link is - [Title](URL): Description — the description matters, it gives the model context about what's at that URL

Don't list every blog post. List the pages that answer "what is this site?" and "what does it do?"

Layer 3: Understanding — Can It Parse What You Are?

A crawler has reached your site and found your priority content. Now it needs structured data to understand entities: who you are, what your site does, what questions your pages answer.

Understanding is Schema.org JSON-LD, meta tags, and entity signals.

Benchmark Data

  • Schema adoption: 75.6% — up from 70.1% in June. Trending positive.
  • Organization schema: 52.2% — barely half of sites tell crawlers who they are.
  • WebSite schema: 58.9%
  • FAQ schema: 18.1% — up from 13.2%, but still remarkably low for a schema type directly designed for question-answer extraction.

The gap between "has some schema" (75.6%) and "has the schema types that matter for AI citation" (52% / 59% / 18%) is where most sites lose ground.

Organization + WebSite JSON-LD Template

Drop this in the <head> of your homepage. Replace the values with your real data.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://example.com/#organization",
      "name": "Example Company",
      "url": "https://example.com",
      "logo": "https://example.com/logo.png",
      "description": "Example Company builds developer infrastructure for AI-powered search.",
      "sameAs": [
        "https://github.com/example",
        "https://x.com/example",
        "https://www.linkedin.com/company/example"
      ],
      "contactPoint": {
        "@type": "ContactPoint",
        "contactType": "support",
        "email": "support@example.com",
        "url": "https://example.com/contact"
      }
    },
    {
      "@type": "WebSite",
      "@id": "https://example.com/#website",
      "url": "https://example.com",
      "name": "Example Company",
      "description": "Developer infrastructure for AI-powered search.",
      "publisher": { "@id": "https://example.com/#organization" },
      "potentialAction": {
        "@type": "SearchAction",
        "target": {
          "@type": "EntryPoint",
          "urlTemplate": "https://example.com/search?q={search_term_string}"
        },
        "query-input": "required name=search_term_string"
      }
    }
  ]
}
</script>
Enter fullscreen mode Exit fullscreen mode

The @graph structure lets you declare multiple entities (Organization + WebSite) in one block and cross-reference them with @id. This is how you tell a model "this organization publishes this website" — a relationship that matters for entity disambiguation.

For FAQ pages, add this on the relevant page:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is AI search readiness?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "AI search readiness is the degree to which a website can be discovered, understood, and cited by generative AI systems like ChatGPT, Perplexity, and Google AI Overviews."
      }
    },
    {
      "@type": "Question",
      "name": "How is GEO different from SEO?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "SEO optimizes for ranking positions on a search results page. GEO optimizes for selection and citation by generative models that synthesize answers from multiple sources."
      }
    }
  ]
}
</script>
Enter fullscreen mode Exit fullscreen mode

FAQ schema is your direct line to question-answer extraction. When a model sees FAQPage with Question and acceptedAnswer pairs, it can pull those answers verbatim. That's why the 18.1% adoption number is so painful — it's the schema type most directly tied to getting quoted, and 82% of sites don't have it.

Layer 4: Quotability — Can It Extract a Self-Contained Passage?

The final layer is content architecture. A model can only quote you cleanly if your content is structured to be quoted: direct answers, bottom-line-up-front (BLUF), short self-contained paragraphs.

This is the layer most teams skip — they fix robots.txt, add llms.txt, implement schema, and then leave their content as 2,000-word narratives that bury the answer in paragraph five.

Before: Narrative Structure (Hard to Quote) 

# How Our API Handles Rate Limiting

When we first launched our API in 2023, we didn't have any rate limiting
in place. After a few incidents where a single client overwhelmed the
auth service, we realized we needed a more robust approach. We
experimented with token bucket algorithms, considered sliding window
loggers, and eventually settled on a fixed window counter approach
combined with exponential backoff. Here's how it works...

[800 more words of context, history, and implementation details]
Enter fullscreen mode Exit fullscreen mode

A model reading this has to synthesize the answer from scattered sentences. It will likely paraphrase without citing, or skip this source for one that's cleaner.

After: BLUF Structure (Easy to Quote) 

# How Our API Handles Rate Limiting

Our API enforces rate limiting using a **fixed window counter** with
exponential backoff. The default limit is **100 requests per minute**
per API key. When the limit is exceeded, the API returns HTTP 429 with
a `Retry-After` header indicating the wait time in seconds.

## How It Works

Rate limits are calculated per API key, not per IP address. Each
request increments a counter that resets at the start of each
60-second window. When the counter exceeds 100, subsequent requests
receive a 429 response until the window resets.

## Handling 429 Responses

Clients should implement exponential backoff: wait 1 second before
the first retry, then double the wait on each subsequent retry, up to
a maximum of 60 seconds. The `Retry-After` header provides the exact
wait time for the current window.
Enter fullscreen mode Exit fullscreen mode

The first paragraph is self-contained. A model can extract it as a direct quote: "Our API enforces rate limiting using a fixed window counter with exponential backoff. The default limit is 100 requests per minute per API key." It makes sense outside the context of the full page. That's quotability.

The rule: every page should open with a standalone answer to its core question. Details go below. The answer goes first.

The Correlation Data: What Actually Moves the Score

The audit measured not just adoption but impact. The correlations are the strongest evidence for the 4-layer model:

Signal With Without Gap
llms.txt 63.3 43.3 +20.0 pts
Schema 61.0 32.8 +28.2 pts

Sites with llms.txt score 20 points higher than sites without. Sites with schema score 28 points higher. These aren't magic — a llms.txt file doesn't fix your server. But they show that the sites investing in orientation and understanding signals are the same sites that perform well across the board. The layers compound.

Common Infrastructure Mistakes

1. Blocking AI crawlers in robots.txt by accident. A CMS, security plugin, or boilerplate template added Disallow: / for GPTBot and nobody reviewed it. Check your robots.txt with curl https://example.com/robots.txt — actually read it.

2. Client-side rendering with no prerender fallback. Your SPA returns an empty shell. AI crawlers see nothing. Either use SSR/SSG or serve prerendered HTML to bot user-agents. Verify with curl -A "GPTBot" https://example.com.

3. Partial llms.txt. A llms.txt with broken links, missing sections, or only the homepage listed is worse than no llms.txt. The 27-point gap between adoption (54.2%) and full implementation (26.9%) means half the llms.txt files out there are broken maps.

4. Generic schema only. Article schema on blog posts is table stakes. The schema types that drive AI citation are Organization (who are you?), WebSite (what is this site?), and FAQPage (what's the answer?). Most sites have none of these.

5. Fixing content before access. I saw teams spend weeks rewriting content for "AI optimization" while their robots.txt blocked every AI crawler. The order is non-negotiable: Access → Orientation → Understanding → Quotability. Skip ahead and you're building on broken foundations.

Where the Benchmark Stands

360 domains audited. Average score 54.1/100. 75.3% at Foundation or Critical. The first 100/100 appeared in July.

The gap between traditional SEO and GEO is measurable and structural. It's not about keywords or backlinks — it's about whether an AI crawler can reach your content, orient itself, understand your entities, and extract a clean passage. Four layers, in order, each one enabling the next.

The good news for developers: every layer is fixable with infrastructure changes. robots.txt is a text file. llms.txt is a text file. JSON-LD is a script tag. BLUF is an editing pattern. None of this requires a marketing agency. It requires an engineer who knows the framework.

Now you do.

The full manual is 160 pages — 15 chapters covering all four layers in depth, 11 AI crawlers with user-agent tokens, 12 schema types with JSON-LD templates, 8 prompt injection attack vectors, and a citation measurement workflow. Free download at geoready.dev/geo-readiness-manual/

Top comments (1)

Collapse
 
citedy profile image
Dmitry Sergeev •

that part about the content structure for ai crawlers is interesting, wonder if this actually helps with perpsplexity rankings or just standard seo