Eduard

Posted on Sep 13 Originally published at auditme.dev

How AI Systems Read the Web in 2026: Discovery, Retrieval, Citations and Agents

#ai #productivity #webdev #agents

From discovery and retrieval to citations, recommendations and agent actions

A practical, evidence-first framework for making websites easier for search systems, AI answer engines, retrieval pipelines and browser agents to discover, understand, verify and use.

TL;DR: There is no universal "AI ranking factor," no single AI crawler, and no markup trick that guarantees a citation. Different machine systems perform different jobs. Search crawlers discover and refresh documents. AI search systems retrieve sources for questions. Retrieval-augmented systems select passages from a corpus. Browser agents can inspect pages and attempt actions. The durable strategy is therefore not "optimize for ChatGPT." It is to make important information reachable, explicit, unambiguous, evidence-backed, retrievable, citable and actionable.

Important research note: This article does not claim that AuditMe has already completed a cross-model benchmark. The AuditMe AI Web Intelligence Benchmark v1 described below is a proposed open methodology and dataset specification. No benchmark results, percentages or causal claims are presented here as observed AuditMe results unless explicitly labeled as such.

Last reviewed: September 13, 2026.

TL;DR at a glance

Question	Best current answer	Confidence / evidence
Does Google require special GEO markup for AI Overviews or AI Mode?	No. Google says the normal technical and quality requirements for Search still apply and there are no additional technical requirements for these AI features.	Official Google documentation
Is `llms.txt` a universal AI-search standard?	No. It is a community proposal. It can be useful as a documentation aid, but it is not a guaranteed citation or ranking mechanism.	Proposal + provider docs
Is there one "AI crawler"?	No. Providers expose multiple crawlers, fetchers and agent mechanisms with different purposes.	Official provider documentation
Does being crawled mean being cited?	No. Discovery, retrieval and citation are separate outcomes.	Architecture distinction
Does structured data guarantee AI visibility?	No. Structured data helps machines and supported Search features interpret content; it is not a universal citation guarantee.	Google + Schema.org
Does blocking `robots.txt` always remove a page from every AI product?	No. Effects are provider- and mechanism-specific.	Provider-specific documentation
Can a model without live web access discover a new page from the page itself?	No. It needs a later model update or an external retrieval path.	Basic system architecture
Does semantic HTML matter to agents?	Yes as an engineering foundation. Native controls, names and states make interfaces more explicit. OpenAI specifically recommends accessibility and ARIA best practices for its agent experience.	Web standards + provider documentation
Does canonicalization matter to AI?	Yes for machine clarity in a broad engineering sense, but do not claim that a canonical tag directly controls every AI model. Canonicalization is an established web/search mechanism for consolidating duplicate URLs.	Google documentation; AI implication is an inference
What should teams measure?	Visibility, retrieval/citation quality and task completion separately.	Proposed measurement framework

The one-sentence model

A page is useful to a machine when it can be found, fetched, parsed, retrieved, understood, verified, cited and — when the workflow requires it — acted upon.

The web no longer has one machine reader
The AI Web Evidence Chain
Discovery and access: can the machine reach the page?
Retrieval and entity clarity: can the machine select the right information?
Evidence and citations: can the page support the answer?
Write for humans and machines without writing like a robot
The technical stack: HTML, structured data, robots.txt, sitemaps and llms.txt
The agentic web: can an AI actually use your website?
Measurement: from AI visibility to Agent Task Completion Rate
The AuditMe playbook: benchmark, fix the bottleneck, publish the evidence
References, documentation and further reading

1. The web no longer has one machine reader

For a long time, website optimization could be explained with a simple model:

Publisher → Search Engine → Search Result → Human

That model is still important. It is just no longer sufficient.

A modern public website can be consumed through several distinct machine paths:

a conventional search crawler that discovers and refreshes documents;
an AI-search system that retrieves web sources for a user query;
a user-triggered fetcher that retrieves a URL because a person asked a product to use it;
a retrieval-augmented generation pipeline that selects passages from a corpus;
a language model answering from information already in its model state;
a browser agent that can inspect, navigate and attempt tasks on the page.

These systems may share infrastructure, but they do not have identical goals.

A page can be perfectly crawlable for Google Search and still fail to become a useful source in a particular AI answer. A page can be selected as evidence and still be awkward or impossible for an agent to operate. A model can know a company name while using stale information. A product page can be visible in search results while a checkout interaction is inaccessible to automation.

The first conceptual error in modern GEO is therefore treating AI as a single destination.

It is better to think in terms of machine access paths.

A practical map of machine consumption

System class	Primary purpose	Typical website concern	What success looks like
Search crawler	Discover, refresh and index documents	Crawlability, internal links, HTTP, canonicalization, sitemaps	The page can be fetched and considered for Search
AI search / answer engine	Retrieve sources and synthesize an answer	Relevance, source quality, clarity, freshness, retrievability	The page is selected as useful evidence and may be cited
Model without live retrieval	Generate from existing model state	Public availability before training/update, external retrieval where supported	The model has accurate knowledge — but the website cannot force this directly
RAG pipeline	Retrieve passages from a corpus and generate	Chunkability, metadata, stable URLs/documents, semantic clarity	Relevant passages are retrievable and faithfully used
Browser agent	Navigate and perform tasks	Semantic controls, accessible names/states, predictable flows, errors/recovery	The agent can complete the intended task safely
Tool / WebMCP-style integration	Expose explicit site capabilities to an agent	Tool definitions, deterministic inputs/outputs, safety and permissions	The agent uses a site-provided tool instead of guessing the UI

The last row is especially important because the agentic web is evolving beyond pure screen interaction. OpenAI currently documents WebMCP-based site tools in its desktop app as a proposed web standard for letting sites expose capabilities directly to AI agents. Chrome also describes WebMCP as part of its agent-ready direction. See OpenAI: Using site tools in the ChatGPT desktop app and Chrome for Developers: A developer toolkit to make your website agent-ready.

The provider docs make the distinction explicit

This is not a theoretical taxonomy invented by SEO consultants.

OpenAI: different crawling and access purposes

OpenAI currently documents OAI-SearchBot for ChatGPT search discovery and separately documents GPTBot as a control related to potential model training use. OpenAI also documents user-directed browser access and accessibility considerations for agent compatibility. For publishers, the practical lesson is simple: do not assume that "the OpenAI bot" is one thing with one purpose.

See the current OpenAI Publishers and Developers FAQ.

Anthropic: multiple user agents, different roles

Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User. Their stated purposes differ: model-development collection, search-oriented crawling and user-directed retrieval are not interchangeable jobs.

See Anthropic: Does Anthropic crawl data from the web?.

Perplexity: search crawling vs user-directed retrieval

Perplexity documents PerplexityBot for surfacing and linking websites in search results and separately documents user-triggered access mechanisms.

See Perplexity Crawlers.

Google: Search, Gemini-related controls and user-triggered fetchers

Google documents Googlebot for Search, Google-Extended as a separate control for certain Gemini-related uses, and user-triggered fetchers/agents for fetches initiated by user actions in Google products. Google explicitly says Google-Extended is not a Search ranking signal.

See Google's crawling infrastructure and Google user-triggered fetchers.

A better evidence hierarchy for GEO claims

When writing about AI search, use a stricter hierarchy than generic marketing content.

Evidence level	Example	How to state it
Officially documented behavior	Google says there are no extra technical requirements for AI Overviews / AI Mode	State it directly and link the source
Web standard / normative specification	RFC 9309 defines the Robots Exclusion Protocol	State the mechanism and scope precisely
Controlled observation	A reproducible test produces a repeated citation pattern	Publish sample, dates, prompts and method
Engineering inference	Explicit UI states are easier for automation to interpret	Label it as an engineering recommendation
Hypothesis	A certain paragraph structure might improve passage retrieval	Say it is a hypothesis until tested
Folklore	"Every LLM prefers exactly 1,500-word articles"	Do not publish as fact

This hierarchy is not pedantry. It is the foundation of credible AI-search content.

2. The AI Web Evidence Chain

The central framework in this article is the AI Web Evidence Chain:

flowchart LR
    A[DISCOVER] --> B[ACCESS]
    B --> C[PARSE]
    C --> D[RETRIEVE]
    D --> E[UNDERSTAND]
    E --> F[VERIFY]
    F --> G[SYNTHESIZE]
    G --> H[CITE]
    H --> I[RECOMMEND]
    I --> J[ACT]

    B -. failure .-> X1[Blocked / denied]
    C -. failure .-> X2[JS-only / malformed]
    F -. failure .-> X3[Weak or conflicting evidence]
    J -. failure .-> X4[Unclear or inaccessible UI]

Different products may merge, reorder or skip stages. The framework is not a claim about hidden proprietary internals. It is a way to identify where a website becomes less useful to a machine.

The ten states

Stage	Question	Typical failure	Useful engineering response
1. Discover	Can the system find the URL?	Orphan page, weak internal linking, blocked crawl path	Internal links, sitemap, clean URL architecture
2. Access	Can it fetch the resource?	403, 429, CDN/WAF block, auth wall	Review access policy, status codes and bot handling
3. Parse	Can useful content be extracted?	JS-only content, malformed HTML, noisy UI	Server-render important content, semantic HTML
4. Retrieve	Can the relevant page/passage be selected?	Ambiguous wording, fragmented content, poor information scent	Clear headings, answer-first sections, descriptive links
5. Understand	Can the system identify meaning and entities?	Conflicting names, duplicate pages, unclear relationships	Consistent entities, canonical URLs, structured data
6. Verify	Is there enough evidence to trust the claim?	Unsupported numbers, weak sourcing, outdated facts	First-party evidence, independent sources where appropriate
7. Synthesize	Can the evidence be used without losing context?	Overly broad claims, missing qualifiers	Precise definitions, scope, dates and limitations
8. Cite	Is the page useful as a source?	Generic marketing copy, no concrete evidence	Make claims specific, attributable and inspectable
9. Recommend	Is the entity suitable for the user's decision?	Weak fit despite visibility	Explain use cases, trade-offs, constraints
10. Act	Can an agent complete the required task?	Unlabeled controls, hidden state, brittle flows	Native controls, accessible names/states, deterministic workflows

Found is not Retrieved. Retrieved is not Cited. Cited is not Actionable.

This distinction is the most important practical idea in the entire framework.

Imagine a product page:

FOUND       ✓
FETCHED     ✓
PARSED      ✓
RETRIEVED   ✓
UNDERSTOOD  ✓
VERIFIED    ?
CITED       ✗
RECOMMENDED ✗
ACTIONABLE  ✗

Nothing contradictory happened.

The system may have found and read the page, but another source may have been better evidence for the user's specific question. The page may also have been perfectly informative while having a poor interface for automation.

That is why the question:

"Is my site visible in AI?"

is incomplete.

The stronger diagnostic question is:

"At which stage of the evidence chain does my site stop being useful?"

That question leads directly to engineering work.

3. Discovery and access: can the machine reach the page?

The glamorous part of GEO usually starts with content. The unglamorous failure is often earlier: the system never gets useful access to the page.

Google's current guidance for AI Overviews and AI Mode is intentionally conservative: existing SEO fundamentals continue to apply, and Google says there are no additional technical requirements specifically for eligibility in those AI features. A page needs to be indexed and eligible to appear in ordinary Search with a snippet. See Google Search Central: AI features and your website.

That does not mean technical SEO is boring. It means the fundamentals are doing more work than the GEO industry often admits.

The access stack

For an important page, check these layers in order:

URL
 ↓
DNS / TLS
 ↓
HTTP response
 ↓
Robots policy
 ↓
Authentication / WAF / CDN policy
 ↓
HTML / renderability
 ↓
Indexability
 ↓
Retrievability

A failure near the top can make everything below it irrelevant.

HTTP status is information

A crawler or agent needs predictable semantics.

200 should mean the requested resource is available.
301 / 308 can express permanent URL consolidation.
302 / 307 are temporary redirects.
401 and 403 communicate access restrictions.
404 and 410 communicate missing resources.
429 communicates rate limiting.
5xx communicates server-side failure.

See RFC 9110: HTTP Semantics and MDN: HTTP status codes.

robots.txt is a crawling control, not a universal visibility switch

The Robots Exclusion Protocol is documented in RFC 9309. Google explains that robots.txt controls crawling, but a blocked URL can still be known to the search engine through other signals. If you need a page excluded from Google's index, noindex is a different mechanism — and the crawler must be able to fetch the page to see that directive.

See:

Practical rule

Do not write:

"Add robots.txt and AI will understand your site."

Write:

"Use robots.txt to express crawling preferences for compatible crawlers, then verify the behavior of each provider you actually care about."

That statement is much harder to break.

The CDN / WAF trap

One of the worst failure modes is allowing a crawler in theory while blocking it in practice.

Common causes include:

a WAF rule triggered by user-agent strings;
bot protection returning challenges instead of HTML;
rate limiting at the edge;
country/IP restrictions;
authentication middleware applied to public pages;
inconsistent behavior between HTML and API routes;
overly aggressive security rules that treat all non-browser traffic as hostile.

The safe engineering approach is not "whitelist every bot." It is to understand the provider's published verification guidance, use least-privilege exceptions where appropriate, and monitor logs.

Google documents crawler verification methods and publishes IP ranges. OpenAI, Anthropic and Perplexity also document their web crawlers and access controls. Start from their primary documentation rather than from a third-party list of supposed AI bot names.

Search access and agent access are not the same problem

Google's current generative-AI Search documentation says normal Search technical requirements remain the basis for AI Overviews and AI Mode. Meanwhile, Chrome's 2026 agent-ready toolkit focuses on the separate problem of agents using the web after they have found it.

That gives us two very different jobs:

SEARCH DISCOVERY
Can the system find and retrieve the page?

AGENT INTERACTION
Can the system understand and operate the page?

A good website needs both only when its business workflow needs both.

4. Retrieval and entity clarity: can the machine select the right information?

Once a system can access a page, the next problem is not "Is the text good?"

It is:

Can the system reliably select the right piece of information for the query?

This is where GEO overlaps with information retrieval, information architecture and entity clarity.

Retrieval rewards explicitness, not mystery

A machine can work with prose. It can also work with highly creative prose. But if an important fact is hidden inside a page's narrative, the page gives the retrieval system more work to do.

Compare:

Weak:

Our platform gives ambitious teams a powerful experience for discovering new ways to improve websites.

Strong:

AuditMe is a website SEO auditing platform. It checks technical SEO, content, performance and related website signals and produces structured audit results.

The second sentence is not prettier. It is more useful as evidence.

Answer the exact question near the top

For important informational pages, a robust pattern is:

H1: Exact topic

1–3 sentence definition

Key facts / answer

Scope / caveats

Detailed explanation

Evidence / sources

Examples

FAQ

This works for humans because it reduces search cost. It also creates clean semantic units for retrieval systems and accessibility tools.

There is no evidence that every AI product requires this exact structure. The point is engineering practicality: clear information is easier to inspect, quote, summarize and verify than vague information.

Entity clarity is bigger than keywords

Traditional SEO often asks:

Which keyword should this page rank for?

An AI-facing content model should also ask:

What entity is this page about, what does the entity do, and how does it differ from nearby entities?

That includes:

the canonical name;
aliases and product names;
category;
product vs company vs article vs tool distinctions;
relationships to parent organizations;
official URL;
author or organization where relevant;
dates and version information;
explicit capabilities and limitations.

This is not a magic "LLM entity score." It is a way to remove semantic ambiguity from the web representation of an entity.

Canonicalization: a real SEO mechanism with an important AI implication

Google describes canonicalization as the process of selecting a representative URL among duplicate or near-duplicate pages. Google also emphasizes that the canonical URL is a hint, not an absolute rule.

See Google: How to specify a canonical URL.

Why does that matter for AI-oriented content strategy?

Because machines perform better when the same concept is represented consistently.

Consider a hypothetical site with three pages:

/site/seo-checker
/site/seo-audit-tool
/site/ai-seo-audit

Suppose all three pages mostly target the same underlying product intent.

A cleaner information architecture might be:

/site/website-seo-checker   ← canonical destination

301 from:
/site/seo-checker
/site/seo-audit-tool
/site/ai-seo-audit

Important: this example is illustrative, not a reported AuditMe experiment. No before/after ranking, traffic or citation result is being claimed here.

The engineering lesson is still sound: when multiple URLs represent substantially the same resource or intent, deliberate consolidation can reduce duplication and make the site's own representation more coherent.

The AI-specific conclusion must be stated carefully:

Canonicalization is a documented web/search mechanism. Any claim that it directly causes a specific AI model to cite a URL is an additional hypothesis that requires measurement.

That sentence is much more defensible than declaring canonicalization an "LLM ranking factor."

Use descriptive links as information, not decoration

Compare:

<a href="/docs">Read more</a>

with:

<a href="/docs/robots-txt-guide">Read the complete robots.txt guide</a>

The second link communicates destination and context directly.

Good internal links improve navigation for people and create clearer relationships between documents for automated systems that parse links.

See Google: Links and crawling.

5. Evidence and citations: can the page support the answer?

The goal of GEO is not merely to be mentioned.

The stronger goal is to become useful evidence.

That sounds like a subtle distinction until you look at the quality of the web.

A machine answer may encounter:

a company homepage with vague claims;
a vendor landing page;
a technical documentation page;
a government source;
an academic paper;
a first-party benchmark;
an independent test;
a community discussion;
an outdated article;
a page that merely repeats another source.

These are not interchangeable evidence types.

A simple evidence hierarchy

For a factual claim, ask what source type naturally owns the truth.

Claim type	Often strongest starting evidence
Product feature	Official product documentation
API behavior	Official API documentation
Regulation	Government / regulatory source
Standard	Standards body / RFC / specification
Company founding / leadership	Official company source + independent corroboration where needed
Measured performance	Reproducible benchmark methodology and raw results
User experience	Independent testing / reviews / user research
Historical fact	Primary or high-quality secondary sources
Medical / scientific claim	Peer-reviewed research or authoritative health institution

The point is not "always use primary sources." Independent evidence can be essential. The point is to match the claim to the right evidence.

Citation presence is weaker than citation relevance

Suppose an AI answer cites your homepage because your company name appears there.

That is not the same as your technical guide being selected to support a detailed explanation of an API behavior.

For a source to be genuinely useful, the cited page should answer the claim being attributed to it.

That suggests a more useful internal metric:

Citation Support Rate

citations that actually support the associated claim
------------------------------------------------------ × 100
all citations attributed to your content

This is a proposed metric, not an industry standard.

Make evidence inspectable

For important facts, prefer:

exact dates;
precise numbers with methodology;
named versions;
direct quotes only when necessary and short;
tables when comparison matters;
links to primary documentation;
explicit limitations;
clear distinction between measurement and interpretation.

Avoid:

"Experts say..."

when you can say:

"Google's documentation states..."

or:

"In our test protocol, measured on [date], the result was..."

The second pattern gives a machine and a human a better evidentiary handle.

Don't manufacture original research

This is the rule that should govern the entire AuditMe content strategy.

If you have not measured it, do not write:

"Our data proves..."

Write:

"Our proposed methodology would measure..."

or:

"Current provider documentation indicates..."

or:

"A useful engineering hypothesis is..."

This is especially important in GEO because the industry is full of tiny samples presented as universal laws.

6. Write for humans and machines without writing like a robot

There is a false choice in AI-content discussions:

Write for humans or write for machines.

Good technical content can do both.

The trick is not to make prose robotic. The trick is to make meaning explicit.

The anti-AI-content formula

For each important section, answer five questions:

What is it?
Why does it matter?
What is actually documented?
What should I do?
What should I not assume?

Example:

What is `llms.txt`?

A proposed Markdown-based convention for giving agents a concise map of important site information and resources.

Why might it matter?

It may reduce the effort required for an agent to locate useful documentation on a site.

What is documented?

The proposal defines a format and describes intended uses.

What should I do?

Publish a small, accurate llms.txt if it helps your documentation ecosystem and keep it consistent with the actual site.

What should I not assume?

Do not claim that llms.txt is a universal ranking signal or a guaranteed citation mechanism.

That is clear writing. It is also machine-friendly writing.

Use tables where comparison is the point

Tables are especially useful when a question has mutually comparable dimensions.

Search vs AI Search vs RAG vs Browser Agent

Dimension	Search engine	AI search	RAG	Browser agent
Core job	Discover / rank documents	Retrieve evidence and answer	Retrieve passages from a corpus	Perform tasks
Needs indexing?	Usually yes	Often depends on source architecture	Depends on corpus	Not necessarily
Needs live access?	Crawling is asynchronous	Often at query time	Depends on pipeline	Usually yes
Main failure	Not found / not indexed	Wrong source / weak evidence	Wrong chunk / retrieval miss	Interaction failure
Important site property	Crawlability	Relevance + evidence	Chunkable documents	Semantic, deterministic UI
Typical output	Links	Answer + sources	Generated answer	Action / task result

Bot identity table

Provider	Mechanism / crawler	Broad purpose	Practical publisher concern
OpenAI	`OAI-SearchBot`	ChatGPT search discovery	Allow when you want ChatGPT search visibility
OpenAI	`GPTBot`	Potential model-training collection	Separate policy from search visibility
Anthropic	`ClaudeBot`	Model-development collection	Separate from search-oriented crawling
Anthropic	`Claude-SearchBot`	Search relevance	Search discovery path
Anthropic	`Claude-User`	User-directed web access	A user-requested retrieval path
Perplexity	`PerplexityBot`	Search discovery / linking	Search visibility path
Google	`Googlebot`	Search crawling	Core Google Search access
Google	`Google-Extended`	Gemini-related control	Not a Google Search ranking signal
Google	User-triggered fetchers / agents	User-requested fetching or navigation	Different access semantics

Always verify the current provider documentation before changing production access rules. Bot names, products and policies can change.

The writing style that travels best through systems

Strong technical pages tend to share these properties:

one obvious topic per page;
direct definitions;
consistent terminology;
short sections with meaningful headings;
data in tables when appropriate;
explicit dates and versions;
concrete examples;
original explanations rather than endless paraphrasing;
source links near important claims;
a references section for deeper research.

This is not a recipe for "gaming AI." It is simply good information architecture.

7. The technical stack: HTML, structured data, robots.txt, sitemaps and `llms.txt`

Technical implementation should answer one question:

Can a machine reliably recover the meaning and state of the page without guessing?

Semantic HTML beats decorative HTML

Prefer native semantics when they already express the interaction you need.

Better:

<button type="submit">Run audit</button>

Riskier:

<div role="button" tabindex="0">Run audit</div>

MDN explicitly recommends native <button> elements where possible because they provide built-in browser and accessibility behavior. See MDN: button role.

For a form:

<form aria-label="Run website audit">
  <label for="url-input">Website URL</label>
  <input
    id="url-input"
    name="url"
    type="url"
    autocomplete="url"
    required
    aria-describedby="url-hint"
  />
  <p id="url-hint">Enter the full URL, including https://</p>

  <button type="submit">Run audit</button>
</form>

This is good accessibility and good interface engineering regardless of AI.

ARIA should describe real state, not invented state

aria-expanded, aria-controls, aria-pressed and aria-busy can communicate state to assistive technologies when used correctly.

For expandable UI:

<button
  type="button"
  aria-expanded="false"
  aria-controls="advanced-options"
>
  Advanced options
</button>

<section id="advanced-options" hidden>
  ...
</section>

For loading state:

<div aria-live="polite" aria-busy="true">
  Analyzing your website…
</div>

MDN explains that aria-busy indicates that an element is being modified and can help assistive technologies avoid announcing incomplete updates. That does not prove that every browser agent will literally "wait for aria-busy=false". The safe claim is narrower: explicit state makes the interface more observable and deterministic. See MDN: aria-busy.

JSON-LD: useful context, not a magic AI switch

Google uses structured data to understand page content and support eligible Search features. Schema.org defines the vocabulary.

Example:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Example Audit Tool",
  "applicationCategory": "BusinessApplication",
  "description": "A website auditing tool for technical and content analysis.",
  "url": "https://example.com/tool"
}
</script>

The crucial rule is consistency: structured data should describe the same thing users can actually see on the page.

See:

Next.js App Router: dynamic metadata and JSON-LD

Next.js provides built-in metadata APIs and a documented JSON-LD pattern. The current docs recommend rendering JSON-LD in layout.js or page.js, and recommend sanitizing payloads before injecting them into the document.

Dynamic metadata

import type { Metadata } from 'next'

export async function generateMetadata(): Promise<Metadata> {
  const title = 'Website SEO Audit'

  return {
    title,
    description:
      'Audit technical SEO, content and website quality with a structured report.',
  }
}

export default function Page() {
  return <main>...</main>
}

See Next.js: generateMetadata.

Dynamic JSON-LD

export default async function Page() {
  const product = await getProduct()

  const jsonLd = {
    '@context': 'https://schema.org',
    '@type': 'SoftwareApplication',
    name: product.name,
    description: product.description,
    url: product.url,
  }

  const jsonLdString = JSON.stringify(jsonLd).replace(/</g, '\\u003c')

  return (
    <main>
      <script
        type="application/ld+json"
        dangerouslySetInnerHTML={{ __html: jsonLdString }}
      />
      {/* page content */}
    </main>
  )
}

See Next.js: JSON-LD.

Sitemaps still matter

Sitemaps are a straightforward way to expose canonical URLs you want crawled. They do not guarantee indexing, but they are useful discovery infrastructure.

See Google: Build and submit a sitemap and sitemaps.org.

`llms.txt`: useful proposal, not a universal protocol

The llms.txt proposal uses Markdown to provide a concise, human- and machine-readable index of important resources. The proposal has evolved and is now maintained as a community convention rather than a universal web standard.

See llms.txt v2 and the core format.

A sensible implementation might look like:

# Example Documentation

> Official documentation for Example API and SDKs.

## Documentation
- [Getting started](https://example.com/docs/start): First steps
- [Authentication](https://example.com/docs/auth): API authentication
- [API reference](https://example.com/docs/api): Complete endpoint reference

## Optional
- [Changelog](https://example.com/changelog): Product and API changes

Do not use llms.txt to duplicate the entire site. Its value is in curation and clarity.

Build a source-of-truth hierarchy

For an important product or organization, create one canonical source for each major claim class:

Product identity ───────→ canonical product page
API behavior ───────────→ official documentation
Pricing ────────────────→ current pricing page
Company facts ──────────→ organization/about page
Research findings ──────→ benchmark/report page
Change history ─────────→ changelog

This prevents the same fact from being written six different ways across six pages.

8. The agentic web: can an AI actually use your website?

Search visibility is only half of the emerging machine-web problem.

The other half is interaction.

Chrome's 2026 agent-ready toolkit describes an important transition: agents first had to search the web; increasingly they also need to use the web. The tooling now includes agentic browsing audits and Chrome DevTools workflows for testing how agents interact with pages.

See Chrome: A developer toolkit to make your website agent-ready.

OpenAI's current publisher guidance makes a similar engineering point for its agent experience: accessibility helps the agent understand page structure and interactive elements, including roles, labels and states. See OpenAI: Publishers and Developers FAQ.

A page can be readable but not operable

Consider this flow:

Agent finds pricing page       ✓
Agent reads pricing             ✓
Agent compares plans            ✓
Agent clicks "Start trial"     ✓
Agent sees form                 ✓
Agent understands fields        ✗
Agent cannot recover validation ✗
Task completed                  ✗

Traditional SEO would rarely capture this failure.

An agentic test should.

Design for explicit state

A good machine-operable interface exposes:

an accessible name for the control;
the control's role;
its current state;
the relationship to the content it controls;
a predictable action;
validation feedback;
a visible or programmatically exposed success state;
a recovery path when the action fails.

Example: an audit form in React

'use client'

import { useState } from 'react'

export default function AuditForm() {
  const [status, setStatus] = useState<'idle' | 'running' | 'completed' | 'error'>('idle')
  const [message, setMessage] = useState('')

  async function runAudit(event: React.FormEvent<HTMLFormElement>) {
    event.preventDefault()
    setStatus('running')
    setMessage('')

    try {
      // Perform the actual request here.
      await runWebsiteAudit()
      setStatus('completed')
      setMessage('Audit completed successfully.')
    } catch {
      setStatus('error')
      setMessage('The audit failed. Check the URL and try again.')
    }
  }

  const busy = status === 'running'

  return (
    <form aria-label="Run website audit" onSubmit={runAudit}>
      <label htmlFor="url-input">Website URL</label>

      <input
        id="url-input"
        name="url"
        type="url"
        required
        autoComplete="url"
        aria-describedby="url-hint"
      />

      <p id="url-hint">Use a complete HTTPS URL.</p>

      <button type="submit" disabled={busy} aria-busy={busy}>
        {busy ? 'Analyzing…' : 'Start audit'}
      </button>

      <p aria-live="polite" aria-busy={busy}>
        {message}
      </p>
    </form>
  )
}

The important point is not React, Next.js or a particular state library. You could implement this with React state, Zustand, Redux, server actions or another approach.

The agent-facing property is explicit behavior: the button is a button, the field has a label, the running state is observable, and success/error states are communicated.

There is no basis for claiming that this exact code guarantees success in every AI agent. It simply follows established web accessibility and interaction principles that make automation less dependent on visual guessing.

WebMCP and explicit agent tools

The web is also moving toward explicit machine interfaces. OpenAI's current site-tools documentation describes WebMCP as a proposed web standard allowing sites to expose tools directly to agents.

This suggests a useful progression:

Screen scraping
      ↓
Semantic HTML
      ↓
Predictable interaction
      ↓
Machine-readable state
      ↓
Explicit site tools

The endpoint of that evolution may not be "make the agent better at clicking." It may be make the website less dependent on clicking at all.

Agent UX checklist

Before calling a page agent-ready, test:

Test	Pass condition
Navigation	Major actions are discoverable without visual-only cues
Labels	Form controls have clear accessible names
Buttons	Native controls are used where possible
State	Loading/success/error state is explicit
Validation	Errors explain exactly what needs fixing
Recovery	Failed actions can be retried safely
Confirmation	High-impact actions require appropriate confirmation
Determinism	The same input produces predictable outcomes
Focus	Keyboard and focus behavior are coherent
Success	Completion is clearly represented in DOM/UI state

9. Measurement: from AI visibility to Agent Task Completion Rate

The industry already has many ways to count traffic and rankings. The emerging machine web needs additional measures.

The danger is inventing a giant score and pretending it is an industry standard.

A better approach is to define narrow metrics with explicit formulas.

1. AI Mention Rate

queries where the target entity is mentioned
--------------------------------------------- × 100
eligible test queries

Useful for measuring brand presence.

Not equivalent to recommendation quality.

2. Citation Rate

queries where a target URL is cited
----------------------------------- × 100
eligible test queries

Useful, but simplistic.

3. Citation Support Rate

cited answers where the page actually supports the claim
---------------------------------------------------------- × 100
all cited answers

This is more meaningful because it distinguishes a real citation from a decorative citation.

4. Retrieval Hit Rate

For a controlled retrieval system:

queries where the correct passage/page is retrieved
----------------------------------------------------- × 100
eligible queries

This should be measured against a known relevance rubric.

5. Entity Consistency Rate

A proposed metric for auditing whether important facts are represented consistently across canonical sources.

consistent entity facts across audited pages
--------------------------------------------- × 100
entity facts checked

Example facts:

official name;
product category;
current URL;
parent organization;
version;
pricing date;
capability/limitation.

6. Agent Task Completion Rate

This is the metric worth testing seriously.

successfully completed eligible tasks
-------------------------------------- × 100
eligible agent task attempts

A task should have a clear start and a clearly defined successful end state.

Examples:

find the pricing plan that includes feature X;
run a website audit;
create a report;
locate API documentation;
submit a support request;
add an item to a cart;
compare two plans.

7. Agent Step Success Rate

successful task transitions
---------------------------- × 100
attempted transitions

This lets you diagnose where a task breaks.

Example:

Step	Human	Agent
Find pricing	100%	100%
Open plan comparison	99%	91%
Start signup	98%	83%
Complete required fields	97%	72%
Recover from validation	94%	41%
Reach confirmation	96%	66%

These numbers are illustrative only. They are not AuditMe measurements.

8. Agent Time to Completion

Measure:

first task event → successful completion

Do not optimize only for raw speed. A fast wrong action is worse than a slower correct one.

9. Agent Recovery Rate

tasks successfully recovered after recoverable failure
-------------------------------------------------------- × 100
tasks that encountered a recoverable failure

This is especially useful for forms, search interfaces and multi-step workflows.

10. Machine Referral Share

Where referral data can be reliably identified:

visits attributed to a tracked AI/referral source
-------------------------------------------------- × 100
all tracked visits

OpenAI currently documents utm_source=chatgpt.com on ChatGPT search referrals, which makes one provider-level measurement path practical. Provider-specific analytics should be documented rather than inferred.

An Agent Telemetry Schema

A useful implementation pattern is to record task semantics, not just clicks.

Suggested event structure

type AgentEvent = {
  event:
    | 'task_started'
    | 'step_viewed'
    | 'action_started'
    | 'validation_failed'
    | 'action_succeeded'
    | 'task_completed'
    | 'task_abandoned'

  taskId: string
  stepId?: string
  route: string
  timestamp: string
  outcome?: 'success' | 'failure' | 'cancelled'
  errorCode?: string
  durationMs?: number
}

Send these events from the workflow boundary, not from arbitrary UI motion.

Example

function track(event: AgentEvent) {
  navigator.sendBeacon(
    '/api/telemetry',
    JSON.stringify(event)
  )
}

track({
  event: 'task_started',
  taskId: 'audit-2026-09',
  route: '/website-seo-checker',
  timestamp: new Date().toISOString(),
})

The key distinction is:

Telemetry should tell you whether the business task succeeded, not merely whether somebody clicked a pixel.

How do you know an agent performed the task?

This requires caution.

A website should not assume that a request is human or automated based only on a user-agent string. Browser agents may render pages using ordinary browser stacks, and privacy/security controls can change the observable network signature.

Better options include:

explicit agent tool calls when the integration supports them;
server-side task IDs;
authenticated application events;
signed callbacks where appropriate;
trace/context IDs propagated through a workflow;
provider-specific headers or metadata when officially documented.

The exact mechanism depends on the product architecture.

The metric itself remains useful even when the actor classification is imperfect: track the task; then separately classify the execution channel with evidence.

10. The AuditMe playbook: benchmark, fix the bottleneck, publish the evidence

The previous sections explain the system. This final section turns it into a practical operating model.

Step 1: Audit the evidence chain, not just the homepage

Create a matrix for your most important URLs:

URL	Discover	Access	Parse	Retrieve	Understand	Verify	Cite	Act
Product page	✓	✓	✓	?	?	?	?	✓
API docs	✓	✓	✓	✓	✓	✓	✓	N/A
Pricing	✓	✓	✓	✓	✓	?	?	?
Signup	✓	✓	✓	N/A	✓	N/A	N/A	?

Use ✓, ?, ✗ and a note.

This is more actionable than a single "AI readiness score."

Step 2: Fix the earliest meaningful failure

If the page is blocked at Access, do not spend three days rewriting the copy.

If the page is readable but ambiguous, do not start by adding more schema.

If the page is frequently cited but fails agent tasks, more content is probably not the immediate bottleneck.

A good order is:

ACCESS
  ↓
PARSE
  ↓
RETRIEVE
  ↓
UNDERSTAND
  ↓
VERIFY
  ↓
CITE
  ↓
ACT

Fix the earliest meaningful break first.

Step 3: Create canonical sources for critical facts

For every business-critical claim, decide where the truth lives.

One product → one canonical product description
One API behavior → one official endpoint/reference page
One pricing model → one current pricing source
One research result → one dated benchmark/report

Other pages can summarize it, but should link back to the source of truth.

Step 4: Make the page easy to extract

A strong reference page usually includes:

a precise title;
one-sentence definition;
TL;DR;
comparison table where appropriate;
concrete examples;
source links;
dates and versions;
limitations;
FAQ for genuinely recurring questions;
stable headings and anchors.

Do this for people first. Machines benefit because the information architecture is explicit.

Step 5: Make the interface semantically honest

Use:

<button> for buttons;
<a> for navigation;
<label> for form labels;
<form> for forms;
headings in a logical hierarchy;
aria-* states only when they describe real state;
visible and programmatic error messages;
predictable success/failure states.

This is accessibility engineering, not an AI hack.

Step 6: Treat `llms.txt` as optional documentation infrastructure

Create it when it helps agents and documentation consumers find your most important resources.

Do not depend on it.

Do not promise it.

Do not turn it into a second sitemap full of every URL on the site.

Step 7: Define your benchmark before collecting your data

This is where the AuditMe AI Web Intelligence Benchmark v1 comes in.

Status: proposed methodology — not yet executed

AuditMe does not currently claim to have completed the cross-model dataset described in this article.

The purpose of publishing the methodology first is to make future measurements reproducible instead of inventing conclusions first and looking for a methodology afterward.

Proposed dataset fields

benchmark_version
run_id
run_timestamp
query_id
query_text
query_intent
engine
model
retrieval_mode
source_url
source_domain
source_type
source_position
citation_present
citation_relevance
claim_supported
entity_match
content_date
freshness_bucket
structured_data_present
first_party_source
external_corroboration
access_status
parse_status
retrieval_status
agent_task_id
agent_step
agent_step_success
task_success
duration_ms
notes

Proposed evaluation dimensions

Dimension	Core question
Discovery	Was the source findable?
Access	Could the system retrieve it?
Retrieval	Was the relevant page/passage selected?
Understanding	Was the entity/claim interpreted correctly?
Verification	Did the source support the claim?
Citation	Was the source exposed and relevant?
Recommendation	Was the recommendation a good fit?
Action	Could an agent complete the task?

Proposed query classes

A future benchmark should not use only generic head terms.

Include:

definition queries
comparison queries
best-of queries
problem-solving queries
commercial-intent queries
technical queries
entity disambiguation queries
freshness-sensitive queries
source-verification queries
agent-task queries

Why release methodology before results?

Because it prevents a common research error:

deciding what the study should prove before deciding how the study should be run.

A public methodology lets other teams criticize the design before the results acquire authority.

Step 8: Release the dataset openly when it exists

The ideal public package is:

/auditme-ai-web-benchmark-v1
├── README.md
├── LICENSE
├── methodology.md
├── schema.json
├── prompts/
├── raw/
├── normalized/
├── evaluations/
├── analysis/
└── examples/

Potential distribution targets include GitHub and Hugging Face Datasets, with stable versioning.

Do not publish credentials, private user data or provider-restricted material. Publish only what the collection method and source licenses allow.

Step 9: Turn original research into a living reference

When the benchmark eventually exists, publish:

Benchmark v1.0
     ↓
Dataset
     ↓
Methodology
     ↓
Results
     ↓
Limitations
     ↓
Replication guide
     ↓
Benchmark v1.1 / v2

A static "ultimate guide" is useful.

A living research asset is more defensible.

Step 10: Measure the bottleneck and repeat

The practical loop is:

OBSERVE
  ↓
DIAGNOSE
  ↓
PRIORITIZE
  ↓
FIX
  ↓
VERIFY
  ↓
PUBLISH THE EVIDENCE
  ↓
REPEAT

That is the operating system for AI-ready websites.

The 100-point AI Web Checklist

Use this as an implementation checklist, not as a universal ranking score.

A. Discovery — 10 points

[ ] Important URLs are internally linked
[ ] No critical page is orphaned
[ ] XML sitemap exists and is current
[ ] Canonical URLs are deliberate
[ ] Important pages return stable URLs
[ ] Redirect chains are minimized
[ ] No accidental noindex on critical pages
[ ] Public content is not hidden behind accidental auth
[ ] Search-facing URLs are understandable
[ ] URL changes have a migration plan

B. Access — 10 points

[ ] Critical pages return correct HTTP status codes
[ ] CDN/WAF does not accidentally block legitimate crawlers
[ ] Rate limiting is monitored
[ ] HTTPS works consistently
[ ] Bot policy is documented internally
[ ] Provider-specific crawler requirements are reviewed
[ ] Server logs can diagnose failed requests
[ ] Error pages do not masquerade as successful HTML
[ ] Authentication is required only where appropriate
[ ] Public content can be retrieved without unnecessary interaction

C. Parsing — 10 points

[ ] Primary content exists in usable HTML
[ ] Important text is not only inside canvas/image assets
[ ] Headings are semantic
[ ] Lists use list semantics
[ ] Tables are real tables when tabular data is presented
[ ] Forms use real form controls
[ ] Links use anchors
[ ] Images have meaningful alternatives where needed
[ ] Dynamic content has a coherent rendered state
[ ] Page source and rendered content are not contradictory

D. Retrieval — 10 points

[ ] H1 states the exact topic
[ ] Important answers appear early
[ ] Headings describe the question being answered
[ ] Paragraphs are self-contained enough to quote
[ ] Links describe their destinations
[ ] Key concepts use consistent terminology
[ ] Tables are used for actual comparisons
[ ] FAQ questions are real user questions
[ ] Important definitions are explicit
[ ] No important fact depends on vague surrounding context

E. Entity clarity — 10 points

[ ] Canonical name is consistent
[ ] Product/company/entity type is explicit
[ ] Official URL is clear
[ ] Parent organization relationships are clear
[ ] Aliases are intentional
[ ] Dates and versions are explicit
[ ] Duplicate pages are reviewed
[ ] Structured data matches visible content
[ ] Different pages do not contradict core facts
[ ] One obvious source of truth exists for important claims

F. Evidence — 10 points

[ ] Major factual claims have an appropriate source
[ ] First-party claims are identified as such
[ ] Independent evidence is used when useful
[ ] Numbers include context or methodology
[ ] Time-sensitive claims include dates
[ ] Version-sensitive technical claims include versions
[ ] Limitations are stated
[ ] Experimental claims are labeled experimental
[ ] Hypotheses are not written as facts
[ ] Sources remain accessible and relevant

G. Citability — 10 points

[ ] Pages contain concrete facts, not only slogans
[ ] Definitions are concise
[ ] Tables summarize important relationships
[ ] Important claims are attributable
[ ] Sources are easy to inspect
[ ] Page scope is clear
[ ] Author/organization context exists when relevant
[ ] Update date is visible where useful
[ ] The page provides something worth citing
[ ] The page does not overclaim what the evidence proves

H. Agent interaction — 10 points

[ ] Native buttons are used where possible
[ ] Form fields have labels
[ ] Controls have accessible names
[ ] Expand/collapse state is explicit
[ ] Loading state is observable
[ ] Validation errors are explicit
[ ] Success state is explicit
[ ] Retry/recovery paths exist
[ ] High-impact actions have appropriate confirmation
[ ] Core tasks can be completed predictably

I. Measurement — 10 points

[ ] Important tasks have measurable start events
[ ] Important tasks have measurable success events
[ ] Errors use stable codes where useful
[ ] Durations are captured
[ ] Agent channel is classified carefully
[ ] AI referrals can be analyzed where provider data allows
[ ] Citation audits use a consistent rubric
[ ] Retrieval tests use fixed prompts and versions
[ ] Benchmarks record dates
[ ] Results distinguish observation from interpretation

J. Governance — 10 points

[ ] Documentation has an owner
[ ] Critical facts have canonical sources
[ ] Content changes are versioned where needed
[ ] Provider policy changes are monitored
[ ] Security rules are reviewed before bot exceptions
[ ] Personal/private data is excluded from public benchmarks
[ ] Research data has a license/usage policy
[ ] Benchmark methodology is public
[ ] Known limitations are documented
[ ] The score is treated as a diagnostic, not a search-engine fact

What not to do in 2026

Don't build a "GEO hack stack"

There is no defensible evidence that one magic combination of llms.txt, FAQ markup, exact word count, or paragraph formula forces all AI systems to cite a site.

Don't confuse Google AI Search with every other AI system

Google's AI Overviews and AI Mode operate inside Google's Search ecosystem. Other providers have different crawlers, retrieval methods, products and policies.

Don't call every AI crawler a "training bot"

Providers themselves distinguish search-oriented crawling from model-development collection and user-triggered access.

Don't claim that a source was used just because it was cited

Citation is evidence of source presentation, not a transparent window into every internal step of a model's reasoning.

Don't publish benchmark numbers you did not measure

This should be non-negotiable for a serious research brand.

Don't build an inaccessible interface and then blame the agent

A confusing form is a confusing form. Fix the interface.

What a genuinely AI-ready website looks like

A strong AI-ready site is surprisingly normal.

It has:

Fast, stable pages
        +
Clear information architecture
        +
Accessible semantic HTML
        +
Canonical sources of truth
        +
Useful structured data
        +
Strong internal linking
        +
Evidence-backed content
        +
Transparent dates / versions
        +
Predictable interactions
        +
Measured workflows

Notice what is missing:

There is no magic AI tag.

That is the point.

The deeper shift: websites are becoming knowledge interfaces

The most important change is not that ChatGPT or Gemini can summarize pages.

The deeper change is that software can increasingly:

find information;
retrieve evidence;
compare alternatives;
navigate interfaces;
invoke tools;
complete workflows.

That means a website now has at least three audiences:

People who read it.

Machines that retrieve it.

Agents that may operate it.

A page that serves all three well does not need a separate "AI version" of the web.

It needs clear information and explicit interfaces.

A practical final test

Take one critical page and ask these ten questions:

Can a crawler discover it?
Can the intended systems fetch it?
Can useful content be parsed without guessing?
Can the exact answer be retrieved?
Is the entity unambiguous?
Can important claims be verified?
Would another engineer cite this page as evidence?
Can a user understand it without reading everything?
Can an agent operate the relevant interface?
Can you measure whether the job actually succeeded?

If the answer to number 1 is no, start there.

If numbers 1–8 are yes but number 9 is no, you have an agent UX problem.

If 1–9 are yes but 10 is no, you have a measurement problem.

That is much more useful than asking whether your website is "AI optimized."

Final takeaway

The web is not being replaced by one giant AI search engine.

It is becoming a layered machine-readable environment where different systems discover, retrieve, interpret, verify, cite, recommend and sometimes act on the same underlying documents.

That changes the optimization question.

The old question was:

How do I rank this page?

The newer question is:

How do I make this page useful to a retrieval system?

And the emerging question is:

How do I make this website useful to a machine that must finish a task?

The strongest answer is not a bag of GEO tricks.

It is an engineering discipline:

Make the right information easy to discover, easy to access, easy to parse, easy to retrieve, hard to misunderstand, easy to verify, worth citing and safe to act upon.

That is the foundation of the machine-readable web.

And it is a much more durable goal than optimizing for any single model, crawler or product.

About the practical tools

For a website-level technical baseline, the AuditMe Website SEO Checker can be used as a practical starting point for reviewing core SEO and website signals.

For a fast score-oriented overview, the AuditMe SEO Score Checker provides another starting point before deeper investigation.

The broader AuditMe blog contains the supporting SEO, GEO and AI-search research that sits around this framework.

11. References, documentation and further reading

The links below are intentionally weighted toward primary documentation, standards and first-party engineering resources. Check them directly before making production changes because web-agent ecosystems evolve quickly.

Google Search, crawling and AI Search

OpenAI

Anthropic

Perplexity

Agentic web and browser automation

HTML and accessibility

Next.js and structured data

HTTP, robots and sitemaps standards

`llms.txt`

Performance and user experience

Common Crawl and open-web research

Search quality and research literacy

Research status and editorial standard

This article intentionally separates:

provider-documented behavior;
web standards;
engineering recommendations;
proposed metrics;
future benchmark methodology;
illustrative examples.

It does not present an unexecuted benchmark as completed research.

The AuditMe AI Web Intelligence Benchmark v1 in this article is a proposed methodology. No cross-model benchmark results are claimed here.

When results eventually exist, they should be published with the prompts, test dates, model/provider versions where observable, sampling rules, evaluation rubric, raw or appropriately licensed data, limitations and replication instructions.

That is the standard required if this work is going to be useful as a serious reference rather than another round of GEO folklore.

Last reviewed: September 13, 2026.

From discovery and retrieval to citations, recommendations and agent actions

TL;DR at a glance

The one-sentence model

Table of contents

1. The web no longer has one machine reader

A practical map of machine consumption

The provider docs make the distinction explicit

OpenAI: different crawling and access purposes

Anthropic: multiple user agents, different roles

Perplexity: search crawling vs user-directed retrieval

Google: Search, Gemini-related controls and user-triggered fetchers

A better evidence hierarchy for GEO claims

2. The AI Web Evidence Chain

The ten states

Found is not Retrieved. Retrieved is not Cited. Cited is not Actionable.

3. Discovery and access: can the machine reach the page?

The access stack

HTTP status is information

robots.txt is a crawling control, not a universal visibility switch

Practical rule

The CDN / WAF trap

Search access and agent access are not the same problem

4. Retrieval and entity clarity: can the machine select the right information?

Retrieval rewards explicitness, not mystery

Answer the exact question near the top

Entity clarity is bigger than keywords

Canonicalization: a real SEO mechanism with an important AI implication

Use descriptive links as information, not decoration

5. Evidence and citations: can the page support the answer?

A simple evidence hierarchy

Citation presence is weaker than citation relevance

Citation Support Rate

Make evidence inspectable

Don't manufacture original research

6. Write for humans and machines without writing like a robot

The anti-AI-content formula

What is llms.txt?

Why might it matter?

What is documented?

What should I do?

What should I not assume?

Use tables where comparison is the point

Search vs AI Search vs RAG vs Browser Agent

Bot identity table

The writing style that travels best through systems

7. The technical stack: HTML, structured data, robots.txt, sitemaps and llms.txt

Semantic HTML beats decorative HTML

ARIA should describe real state, not invented state

JSON-LD: useful context, not a magic AI switch

Next.js App Router: dynamic metadata and JSON-LD

Dynamic metadata

Dynamic JSON-LD

Sitemaps still matter

llms.txt: useful proposal, not a universal protocol

Build a source-of-truth hierarchy

8. The agentic web: can an AI actually use your website?

A page can be readable but not operable

Design for explicit state

Example: an audit form in React

WebMCP and explicit agent tools

Agent UX checklist

9. Measurement: from AI visibility to Agent Task Completion Rate

1. AI Mention Rate

2. Citation Rate

3. Citation Support Rate

4. Retrieval Hit Rate

5. Entity Consistency Rate

6. Agent Task Completion Rate

7. Agent Step Success Rate

8. Agent Time to Completion

9. Agent Recovery Rate

10. Machine Referral Share

An Agent Telemetry Schema

Suggested event structure

Example

How do you know an agent performed the task?

10. The AuditMe playbook: benchmark, fix the bottleneck, publish the evidence

Step 1: Audit the evidence chain, not just the homepage

Step 2: Fix the earliest meaningful failure

Step 3: Create canonical sources for critical facts

What is `llms.txt`?

7. The technical stack: HTML, structured data, robots.txt, sitemaps and `llms.txt`

`llms.txt`: useful proposal, not a universal protocol

Step 6: Treat `llms.txt` as optional documentation infrastructure

`llms.txt`