DEV Community: Eduard

The Iterative Creation Loop: How to Build Better with Plan Mode, Multi-Agent Dialogues, and Verification

Eduard — Tue, 22 Sep 2026 20:59:03 +0000

A practical 2026 field guide for coding, research, content, products, knowledge systems, and AI-native workflows

There is a point where adding more intelligence to an AI workflow stops helping.

Not because the model is weak.

Because the workflow is weak.

You can give one agent a huge context window, a dozen tools, a long instruction, and a carefully engineered prompt. It can still misunderstand the target, over-trust an assumption, duplicate research, change something that was supposed to stay untouched, or declare success before anyone has actually checked the result.

The instinctive answer is often:

Add another agent.

Then another.

Then a reviewer.

Then a “senior architect.”

Then a final editor.

Eventually you have built a tiny artificial company that spends more time coordinating than doing the work.

That is not the goal.

The more useful idea is simpler:

Do not optimize for the number of agents. Optimize for the quality of state transitions between intention, evidence, action, and verified completion.

This guide calls that system the Iterative Creation Loop.

OBSERVE → CONTRACT → PLAN → CHALLENGE → SPECIALIZE
        → REFINE → APPROVE → EXECUTE → VERIFY → LEARN ↺

The agents are optional.

The loop is not.

For some tasks, one strong agent should run most of the loop. For others, independent researchers, critics, specialists, and verifiers deserve their own contexts. The architecture should follow the shape of the work.

The result is not a “swarm.” It is something more useful: a repeatable process for turning uncertain work into checked work.

TL;DR

The useful shift is from answer generation to artifact progression.

Instead of:

Human → Prompt → Model → Answer

use:

Goal Contract
     ↓
Observe real evidence
     ↓
Create a falsifiable plan
     ↓
Attack the plan
     ↓
Delegate only real uncertainties
     ↓
Execute in checkpoints
     ↓
Verify independently
     ↓
Record what was learned

A few current findings explain why this is worth taking seriously without turning the article into hype.

Current evidence	What it actually supports
Anthropic reported a 90.2% improvement over its single-agent baseline on one internal research evaluation	Multi-agent research can produce large gains on the right task class. It is not a universal benchmark for all agent workloads.
Anthropic reported roughly 15× the token usage of normal chat for its multi-agent Research system	Coordination buys capacity, but the cost is real. The task has to justify it.
The 2026 PACT paper reported substantially lower communication cost when agent outputs are projected into compact action-state records	Communication design is part of agent architecture; forwarding full transcripts is not the only option.
MAST research identified 14 failure modes in multi-agent systems, spanning system design, inter-agent misalignment, and task verification	Adding agents introduces new failure surfaces instead of removing all failure.
A September 2026 study argues that multi-agent gains are strongest on long-horizon, sparse-dependency work and can diminish on tightly coupled sequential tasks	Task topology matters more than agent count.

Primary sources: Anthropic, PACT, MAST, Rethinking Multi-Agent Collaboration.

Practical rule: start with one agent, find the bottleneck, add the smallest amount of coordination that removes it, and measure whether the extra complexity pays for itself.

The core idea: build a loop, not a council
When multi-agent actually helps — and when it makes things worse
Plan Mode: the boundary between thinking and changing the world
The artifact-first architecture
Roles and topologies: choose structure from the task
The dialogue protocol: Action, State, Result
The complete Iterative Creation Loop
How to run the loop today: Cursor, Claude Code, SDKs, and custom harnesses
What the 2025–2026 evidence actually tells us
Four practical playbooks: software, research, content, and products
Verification and measurement: make “done” observable
Failure modes: how good-looking agent systems break
SEO, GEO, and AI-readable publishing without the folklore
The reusable operating kit

1. The core idea: build a loop, not a council

The phrase “multi-agent system” makes the agents sound like the main object.

They are not.

The main object is the workflow state.

A specialist agent is simply a temporary context boundary. A critic is a role with an adversarial objective. A verifier is a gate that requires evidence. A manager is a routing mechanism.

What survives from one stage to the next should be the state of the work.

INTENTION
   ↓
OBSERVED STATE
   ↓
HYPOTHESIS / PLAN
   ↓
CHALLENGE
   ↓
ACTION
   ↓
NEW OBSERVATION
   ↓
VERIFICATION
   ↓
LEARNING

That pattern is not new. Software engineering has tests and code review. Science has hypotheses and experiments. Operations has incident response and postmortems. Product teams prototype, observe, and iterate.

Agentic systems make one part of the loop dramatically cheaper: you can run more investigation, criticism, synthesis, and verification without having a human perform every intermediate step.

The hard problem therefore moves from “Can a model generate something impressive?” to:

Can the system preserve the right state while moving from uncertain intent to checked outcome?

1.1 Answer generation vs artifact progression

A chat answer is a terminal object. An artifact can be inspected, changed, versioned, tested, and reused.

Compare:

Question → Answer

with:

Goal Contract
→ Plan
→ Evidence Ledger
→ Decision Log
→ Implementation / Draft
→ Verification Report

The second pipeline creates objects that future agents and humans can work with.

That is the key design shift.

1.2 The three questions every stage must answer

At any point in the loop, the next participant should be able to answer:

What are we trying to achieve?
What do we currently know?
What is the next justified action?

Most bad agent workflows fail because one of those becomes implicit.

The goal gets lost in a long context.

The evidence gets mixed with speculation.

The next action is chosen because the previous model sounded confident.

The loop exists to keep those three questions explicit.

1.3 Why “more agents” is the wrong objective

Suppose one agent can solve a task in eight tool calls.

You create four agents:

Planner
Researcher
Critic
Verifier

Now you have:

five contexts instead of one,
more routing,
more state transfer,
more tokens,
more latency,
more opportunities for contradiction.

Unless those extra contexts buy something real — independent evidence, parallel work, specialized tools, better verification, or useful isolation — you have increased complexity without increasing capability.

That is why the architecture should be demand-driven.

2. When multi-agent actually helps — and when it makes things worse

The question is not whether multi-agent systems are powerful.

They are.

The question is where the power comes from.

A useful model is:

Multi-agent benefit
≈
parallelism
+ context isolation
+ specialization
+ independent criticism
+ additional tool capacity
− coordination cost
− token cost
− latency
− new failure modes

There is no universal numeric formula. The point is architectural: every additional agent introduces a cost that should have a reason to exist.

2.1 The task topology test

Before creating a team, classify the task.

Task characteristic	Single agent	Multi-agent
Short and self-contained	Usually sufficient	Often unnecessary
Tightly coupled sequence	Often preferable	Can add synchronization cost
Many independent research branches	Limited by sequential time/context	Strong fit
Different domains/tools	Possible but context-heavy	Strong fit when boundaries are real
Need independent adversarial review	Self-review possible	Separate critic can help
High-risk execution	Needs gates	Separate verifier/HITL can help
Large context exceeds one useful working set	Harder	Context partitioning can help
Deterministic pipeline	Usually better in code	Multi-agent may be overkill

The September 2026 paper Rethinking Multi-Agent Collaboration: When More Is Less makes this same distinction more formally: benefits are strongest on long-horizon tasks with sparse dependencies, while tightly coupled sequential workflows can favor single-agent systems because coordination overhead becomes dominant.

Source: Rethinking Multi-Agent Collaboration: When More Is Less.

2.2 Sparse dependency vs dense dependency

This distinction is one of the fastest ways to decide.

Sparse dependency

A ───┐
B ───┼──→ synthesis
C ───┤
D ───┘

A, B, C, and D can work independently.

Multi-agent is attractive.

Dense dependency

A → B → C → D
    ↑   ↓
    └───┘

Each stage depends on details from the previous one.

A single agent or deterministic workflow may be easier to manage.

2.3 The independence test

Before spawning a specialist, ask:

Could this person work independently for ten minutes and return something another person can actually consume?

If the answer is no, it probably is not a good parallel subtask.

2.4 The context test

Ask:

Does the specialist need the whole conversation?

If not, isolate it.

A performance researcher probably does not need the entire product brief.

A fact checker probably does not need the entire brainstorming transcript.

A security specialist may need the architecture and changed files, but not the marketing discussion.

Context isolation is not just a cost optimization. It is often a quality optimization.

2.5 The “do not spawn” rule

Create no extra agent when:

- the task is simple,
- there is no meaningful parallelism,
- all stages need the same context,
- deterministic tools already solve the verification problem,
- or the expected benefit is purely stylistic.

A good orchestrator can say:

No specialist required.
Proceed with single-agent execution.

That is mature orchestration.

3. Plan Mode: the boundary between thinking and changing the world

Plan Mode is useful for a deceptively simple reason: it makes the plan visible before the implementation becomes expensive to undo.

Cursor describes Plan Mode as a flow in which the agent researches the codebase, asks questions, creates a detailed plan, lets you review or edit it, and then builds from that plan. Plans can be saved to the workspace. Cursor's CLI also exposes plan mode through commands such as /plan and --mode=plan.

Claude Code similarly supports a plan permission mode, as well as subagents and agent teams for more involved workflows.

Sources:

3.1 Reversible reasoning vs irreversible action

Without a planning boundary:

request
  ↓
agent edits
  ↓
discovers constraint
  ↓
edits again
  ↓
discovers dependency
  ↓
repairs previous edit
  ↓
human untangles diff

With one:

request
  ↓
inspect
  ↓
identify constraints
  ↓
compare approaches
  ↓
plan
  ↓
review
  ↓
execute

The model did not become smarter.

The workflow became safer.

3.2 The Goal Contract

Before the plan, freeze the problem.

goal: "Refactor the authorization layer without changing behavior"
constraints:
  - "preserve public API"
  - "preserve current authorization semantics"
  - "keep existing tests passing"
forbidden:
  - "framework migration"
  - "database schema change"
acceptance:
  - "all existing tests pass"
  - "new regression tests pass"
  - "duplicate authorization branches removed"
unknowns:
  - "legacy route behavior"

The contract should be easy to quote and hard to misinterpret.

3.3 A plan should predict the future

Weak:

Improve architecture and reliability.

Falsifiable:

Replace duplicated permission resolution in modules A and B with the existing canonical resolver, preserve public signatures, add regression coverage for guest/admin boundaries, then compare the approved file scope against the final diff.

The second version can fail.

That is precisely why it is useful.

3.4 Planning anti-patterns

The architecture fan fiction problem

The agent assumes abstractions exist because they would be sensible.

Fix:

Inspect first.
If a component is not found, mark it UNKNOWN.
Do not invent it.

The “everything is in scope” problem

The user asks for a refactor. The agent silently redesigns three systems.

Fix:

approved files
approved services
forbidden changes

The plan that cannot be tested

If you cannot explain how a step will be verified, the step is not ready.

4. The artifact-first architecture

The most important design decision is to make the shared state visible.

Do not design around “who talks to whom.”

Design around what artifacts move through the system.

I recommend five canonical artifacts:

1. Goal Contract
2. Plan
3. Evidence Ledger
4. Decision Log
5. Verification Report

4.1 Goal Contract

It is the stable definition of success.

# Goal Contract

Goal:
Refactor the reporting module without changing public behavior.

Constraints:
- preserve API
- preserve error semantics
- preserve current output format

Forbidden:
- schema migration
- framework replacement
- unrelated cleanup

Definition of done:
- existing tests pass
- regression suite passes
- public API diff is zero
- performance remains within target

4.2 Plan

The plan is a hypothesis, not a promise that reality will obey it.

It should contain:

observed locations,
dependencies,
assumptions,
sequence,
risks,
acceptance tests,
rollback/recovery path.

When new evidence invalidates the plan, revise the plan instead of pretending the old one still describes reality.

4.3 Evidence Ledger

This is the artifact that prevents repetition from turning into truth.

Field	Purpose
Claim	The exact statement being asserted
Source	URL, file, test, benchmark, tool result
Source type	Official docs / code / experiment / research / secondary
Published / observed	Freshness
Evidence	What the source actually establishes
State	VERIFIED / SUPPORTED / PLAUSIBLE / UNKNOWN
Conflicts	Contradictory evidence
Used for	Decision or plan step

Example:

claim: "OAI-SearchBot is relevant to OpenAI web search discovery"
source: "OpenAI Publishers and Developers FAQ"
source_type: "official-documentation"
state: "VERIFIED"
used_for: "AI crawler preflight"

The source can support the claim without supporting a much stronger claim such as “this guarantees citation.”

That distinction is the entire point of an evidence ledger.

4.4 Evidence states

Use a small vocabulary.

State	Meaning	Allowed use
VERIFIED	Directly observed, mechanically tested, or clearly established by a primary source for the exact claim	Can drive decisions
SUPPORTED	Strong evidence, but not fully reproduced or narrower than the claim	Can inform decisions
PLAUSIBLE	Reasonable interpretation	Must remain labeled
UNKNOWN	Evidence is missing	Must not silently become fact

A downstream agent should never be able to transform UNKNOWN into VERIFIED merely by repeating it.

4.5 Decision Log

Record choices and rejected alternatives.

Decision: Use manager + specialist-as-tool orchestration.

Rejected: unrestricted peer handoffs.

Reason: final synthesis needs one owner and specialists have bounded tasks.

Evidence: OpenAI Agents SDK manager-style orchestration guidance.

Decision logs become more valuable over time because they explain not just the current architecture, but why it exists.

4.6 Verification Report

The final report should read like a proof, not a celebration.

Criterion 1 — PASS
Evidence: 128/128 tests passed.

Criterion 2 — PASS
Evidence: public API snapshot unchanged.

Criterion 3 — FAIL
Evidence: duplicate branch remains in module C.

Decision: NO-GO
Repair: remove duplication and rerun targeted suite.

That document is reusable by the next agent, reviewer, or human.

5. Roles and topologies: choose structure from the task

Roles are useful as long as they represent different responsibilities, context, or authority.

The classic four-role model is a good starting point, not a law:

Role	Primary job	Input	Output	Authority boundary
Planner	Decompose goal and propose sequence	Goal + evidence	Plan + risks + criteria	Cannot silently redefine goal
Critic	Attack assumptions and plan	Goal + plan	Failure modes + evidence	Cannot rewrite requirements
Specialist	Resolve a narrow uncertainty	Question + scoped context	Evidence-backed recommendation	Cannot expand scope
Executor	Make approved changes	Approved plan	Implementation/draft	Cannot redefine contract
Verifier	Test the result against contract	Goal + result + raw evidence	PASS/FAIL/UNKNOWN	Cannot silently waive criteria
Human	Final authority on ambiguous or high-impact decisions	Full relevant state	Binding decision	—

5.1 Sequential orchestration

Planner → Specialist → Executor → Verifier

Use it when each step depends on the prior output.

5.2 Concurrent orchestration

        ┌→ Research A ─┐
Planner ┼→ Research B ─┼→ Synthesizer
        └→ Research C ─┘

Use it when branches are independent.

5.3 Handoff

Triage → specialist

Use it when a specialist should take over the active context or user interaction.

5.4 Manager + agents-as-tools

                 ┌→ Specialist A
Manager ─────────┼→ Specialist B
                 └→ Specialist C
                     ↓
                 final synthesis

Use it when one agent should own the final answer and decide when specialists are needed.

OpenAI's Agents SDK explicitly documents both manager-style “agents as tools” and handoff patterns, as well as code-driven orchestration where the application owns the workflow.

Source: OpenAI Agents SDK — Agent orchestration.

5.5 Group chat

A ↔ B ↔ C ↔ Manager

Group chat can work for genuine deliberation, but it has a nasty failure mode: everyone keeps talking because nobody owns stopping.

Use explicit round limits and a decision owner.

5.6 Dynamic / magentic orchestration

Microsoft's Agent Framework describes magentic orchestration as a manager coordinating specialized agents dynamically based on evolving task state.

That is useful when the path is not known in advance.

It is not automatically better than a simpler pipeline.

Source: Microsoft — Magentic orchestration.

5.7 The topology decision matrix

Question	If YES	If NO
Can branches run independently?	Consider concurrent agents	Prefer sequential/single-agent
Does a specialist need unique context/tools?	Isolate the specialist	Keep context unified
Must one agent own the final answer?	Manager + agents-as-tools	Handoff/peer topology possible
Can the next step be coded deterministically?	Orchestrate in code	LLM can help route
Is open-ended planning unavoidable?	Dynamic manager may fit	Simpler topology is preferable
Can a test answer the question?	Use deterministic verification	Model/human evaluation may be needed

6. The dialogue protocol: Action, State, Result

A multi-agent system can fail even when every individual model response looks good.

Why?

Because communication itself is a system resource.

Consider:

Agent A → 2,000-word explanation
Agent B reads it and writes 1,800 words
Agent C receives A+B and writes 1,500 words
Agent D receives everything and decides what matters

Much of that text is explanation of reasoning history, not state required to continue.

A June 2026 paper, What Should Agents Say? Action-state Communication for Efficient Multi-Agent Systems, introduces PACT — Protocolized Action-state Communication and Transmission. The paper studies several communication strategies and argues that useful inter-agent messages should preserve action-centered state rather than blindly forwarding free-form outputs. Its experiments report a substantially better performance-cost trade-off and reduced token consumption in tested settings. The paper also emphasizes that no single fixed communication strategy is optimal everywhere.

Source: PACT.

The part worth stealing is straightforward:

Pass the state needed to continue, not the transcript that produced it.

6.1 The Action-State-Result message

A practical handoff record:

### Action-State Message

From: Critic
To: Planner

Action:
Reject plan step 3.2.

State:
The plan assumes generated routes are all crawlable. Repository inspection shows that one route family can produce orphaned URLs without canonical links.

Result:
Add a route-crawlability acceptance test and an explicit canonical URL requirement.

Evidence:
route manifest + crawler output + relevant source file.

Confidence:
High.

Next needed:
Revise step 3.2 before execution.

6.2 Why “Result” is useful

“Action” says what happened.

“State” says why.

“Result” says what artifact or decision the receiver should carry forward.

Without Result, the next agent has to reconstruct the intended handoff.

6.3 Keep messages receiver-oriented

A sender may care about fifty details.

The receiver may need five.

A good message answers:

What changed?
Why does it matter?
What should I use?
What remains unresolved?

6.4 Do not pass private chain-of-thought

Structured coordination does not require passing hidden reasoning traces.

Pass:

observations,
tool outputs,
conclusions,
evidence,
decisions,
unresolved questions,
next actions.

That is enough to coordinate.

6.5 A compact JSON form

When you need machine-readable handoffs:

{
  "action": "reject_plan_step",
  "state": "route family can produce orphaned URLs",
  "result": "add crawlability + canonical acceptance test",
  "evidence": [
    "route-manifest.json",
    "crawler-output.json"
  ],
  "confidence": "high",
  "next_needed": "revise_plan"
}

This is especially useful when the orchestrator is code-driven.

7. The complete Iterative Creation Loop

The loop is a sequence of gates, not a ritual for its own sake.

7.1 Step 0 — Freeze the contract

Write:

Goal
Constraints
Forbidden changes
Definition of done
Unknowns

Do this before the team gets creative.

7.2 Step 1 — Observe

Rule:

No invention where inspection is possible.

For software, inspect:

source tree,
relevant files,
dependencies,
tests,
configuration,
current abstractions,
recent changes.

For research, inspect:

primary sources,
official documentation,
current product behavior,
research papers,
contradictory evidence.

For content, inspect:

search intent,
authoritative references,
terminology,
competing interpretations,
audience questions.

The output should begin with Observed Facts.

7.3 Step 2 — Plan

The Planner produces:

observed facts
assumptions
proposed sequence
dependencies
risks
acceptance tests
rollback path

Anything unverified stays labeled.

7.4 Step 3 — Challenge

The Critic gets one instruction:

Assume the plan is wrong. Find the smallest number of high-impact reasons it can fail.

Prioritize:

goal violations
wrong assumptions
hidden dependencies
missing tests
security/reliability risks
scope creep
unnecessary complexity

Do not ask the Critic to rewrite the plan.

Ask it to break the plan.

7.5 Step 4 — Specialize

Spawn specialists only for unresolved uncertainty.

Examples:

Performance question
Security question
Framework compatibility question
Research evidence question
Accessibility question
Crawler behavior question

A specialist assignment should be narrow enough to fit in one sentence.

7.6 Step 5 — Resolve disagreements

Do not use:

latest response wins

Use an evidence priority ladder:

1. direct mechanical evidence
2. reproducible experiment
3. official / primary documentation
4. source-code inspection
5. peer-reviewed or clearly identified research
6. strong secondary analysis
7. expert judgment
8. model intuition

This is a practical heuristic, not a universal scientific ranking. Its job is to stop rhetorical confidence from outranking evidence.

7.7 Step 6 — Refine the canonical plan

Do not append more debate to a giant transcript.

Update the Plan.

The plan should remain the canonical current state.

7.8 Step 7 — Approve a checkpoint

Do not approve an entire risky implementation as one atomic promise.

Approve a small unit:

Checkpoint 1
→ execute
→ verify
→ update state

Checkpoint 2
→ execute
→ verify
→ update state

7.9 Step 8 — Execute

Execution should now have a contract, scope, and test conditions.

7.10 Step 9 — Verify

The verifier should see:

Goal Contract,
acceptance criteria,
resulting artifact,
relevant raw evidence,
deterministic test output.

It should not be told the conclusion in advance.

7.11 Step 10 — Learn

Record:

what failed
which assumption was wrong
which test caught it
which communication was wasteful
whether the specialist was necessary
whether the overall topology paid off

That becomes future organizational memory.

7.12 The complete state diagram

┌──────────┐
│  OBSERVE │
└────┬─────┘
     ↓
┌──────────┐
│ CONTRACT │
└────┬─────┘
     ↓
┌──────────┐
│   PLAN   │
└────┬─────┘
     ↓
┌──────────┐
│ CHALLENGE│
└────┬─────┘
     ↓
┌──────────┐
│SPECIALIZE│  only where needed
└────┬─────┘
     ↓
┌──────────┐
│  REFINE  │
└────┬─────┘
     ↓
┌──────────┐
│ APPROVE  │
└────┬─────┘
     ↓
┌──────────┐
│ EXECUTE  │
└────┬─────┘
     ↓
┌──────────┐
│ VERIFY   │
└────┬─────┘
     ↓
┌──────────┐
│  LEARN   │
└────┬─────┘
     │
     └────────────→ OBSERVE

8. How to run the loop today: Cursor, Claude Code, SDKs, and custom harnesses

The framework should come after the workflow.

8.1 Cursor

Cursor's Plan Mode is a natural home for the first half of the loop:

Plan Mode
→ inspect repository
→ ask questions
→ create/edit Markdown plan
→ review
→ build
→ review diff
→ run checks

Cursor's documentation specifically recommends Plan Mode for complex features, multi-file changes, uncertain requirements, and architecture decisions. For small familiar edits, it says Agent mode can be appropriate instead.

Useful references:

Practical setup

1. Write Goal Contract.
2. Enter Plan Mode.
3. Ask for observed facts first.
4. Ask for plan + acceptance tests.
5. Run a fresh-context Critic pass.
6. Revise the plan.
7. Build one checkpoint.
8. Run deterministic checks.
9. Run verifier.

8.2 Claude Code

Claude Code supports plan-oriented permission control, subagents with separate working contexts, and agent teams for more independent coordination.

The useful distinction is:

Subagent
= isolated worker for a bounded task

Agent team
= multiple independent sessions coordinating on a broader problem

Use subagents when the intermediate work is large but only the result needs to return to the main context.

Use teams when independent workers need to coordinate around a shared task.

Useful references:

8.3 OpenAI Agents SDK

OpenAI's current SDK docs describe two broad orchestration strategies:

LLM-driven: the model decides which agents/tools to invoke.

Code-driven: application logic determines the flow.

It also distinguishes:

Agents as tools: manager retains control.

Handoffs: specialist takes over.

The same docs describe chaining, evaluator loops, and parallel agent execution as common code-driven patterns.

That gives you a straightforward implementation of the Iterative Creation Loop:

planner
→ critic
→ refiner
→ executor
→ evaluator
→ repair/replan

References:

8.4 LangChain and LangGraph

LangChain's current documentation explicitly treats multi-agent design as a context-engineering problem and documents patterns such as:

subagents,
handoffs,
routers,
skills.

LangGraph gives more explicit graph/state control for applications that need persistent, inspectable workflow transitions.

References:

8.5 CrewAI

CrewAI is a role-and-task-oriented way to prototype crew workflows.

The important question is not whether the framework calls your process a “crew.” The important question is whether you can represent:

Goal
→ task decomposition
→ ownership
→ dependencies
→ outputs
→ verification

Reference: CrewAI documentation.

8.6 Microsoft Agent Framework

Microsoft's current Agent Framework documentation exposes several orchestration patterns:

Sequential
Concurrent
Handoff
Group Chat
Magentic

It also supports human-in-the-loop workflow interactions.

This vocabulary is useful even if you never use the framework because it gives you names for different coordination shapes.

References:

8.7 Framework selection table

Need	Reasonable starting point	Core strength
Interactive coding + planning	Cursor	Plan → Build workflow
Terminal-first coding + workers	Claude Code	Subagents / teams / permission controls
Lightweight custom Python agent workflow	OpenAI Agents SDK	Small primitives + tracing/orchestration
Stateful workflow graph	LangGraph	Explicit transitions and state
Crew-style rapid prototype	CrewAI	Role/task abstraction
Enterprise orchestration patterns	Microsoft Agent Framework	Rich topology vocabulary + HITL

This is not a ranking. It is a fit map.

9. What the 2025–2026 evidence actually tells us

The multi-agent field has accumulated enough evidence that “more agents = smarter” is no longer a serious design principle.

The interesting question is where the gains come from and where they disappear.

9.1 Anthropic: more compute can buy more research capacity

Anthropic's engineering report on its multi-agent Research system is one of the clearest public examples.

The architecture uses a lead agent to plan research and delegates directions to subagents that investigate in parallel.

Anthropic reports:

90.2% improvement over its single-agent baseline on its internal BrowseComp evaluation,
roughly 15× the token usage of ordinary chat,
major gains from parallelization and additional context capacity,
poor fit for some tightly coupled tasks where agents need heavy shared context.

The right lesson is not “use many agents.”

It is:

When a task has high value, lots of parallelizable information gathering, and a bottleneck in one context, multi-agent execution can buy useful additional reasoning capacity.

Source: Anthropic — How we built our multi-agent Research system.

9.2 PACT: communication is an optimization surface

PACT asks a narrower question:

What should agents actually say to each other?

The answer is not “send everything.”

The paper evaluates multiple strategies and proposes Action-State communication as a way to preserve decision-relevant information while reducing unnecessary context transfer. It reports substantial token savings in its experiments, including improvements on evaluated coding harnesses.

The most transferable idea is:

public state update
> conversation transcript

Source: PACT.

9.3 MAST: coordination creates failure modes

The paper Why Do Multi-Agent LLM Systems Fail? analyzed more than 150 traces in detail to build a taxonomy of 14 failure modes grouped into:

specification and system design,
inter-agent misalignment,
task verification and termination.

The project later made a larger annotated trace dataset available for broader evaluation.

The important point is architectural:

A multi-agent system is a new system. It needs its own reliability engineering.

Source: MAST.

9.4 MultiAgentBench: coordination itself can be measured

The ACL 2025 MultiAgentBench work evaluates not only task completion but also coordination behavior. It studies star, chain, tree, and graph coordination protocols and uses milestone-oriented metrics.

That is useful because final accuracy alone hides system behavior.

A workflow can produce the right answer for the wrong reasons.

It can also spend enormous resources to produce a modest result.

Source: MultiAgentBench.

9.5 September 2026: “when more is less” becomes the central question

The September 2026 study Rethinking Multi-Agent Collaboration: When More Is Less directly examines scaling agent pools and recursion depth.

Its key message is that task structure defines the capability boundary. More agents do not consistently create better outcomes.

That is almost exactly the reason this guide refuses to make “four agents” or “ten agents” into a universal recipe.

Source: Rethinking Multi-Agent Collaboration.

9.6 The evidence pattern

Across these sources, a coherent engineering picture emerges:

Multi-agent helps when:
+ work can be decomposed
+ branches are reasonably independent
+ contexts benefit from isolation
+ additional tool capacity matters
+ independent checking has real value

Multi-agent hurts when:
− dependencies are dense
− everyone needs the same context
− communication dominates work
− verification is weak
− agents duplicate each other
− the task is too small to justify the overhead

That is a much more useful conclusion than “multi-agent is the future.”

10. Four practical playbooks: software, research, content, and products

The loop is abstract. These playbooks make it concrete.

10.1 Software: refactor without behavioral drift

Goal

Split a large reporting module without changing its public behavior.

Planner

Inspect:

public exports,
callers,
shared state,
tests,
performance-sensitive paths,
configuration.

Output:

current boundaries
candidate extraction points
dependency order
regression tests
rollback path

Critic

Attack:

hidden coupling
changed error behavior
circular dependencies
performance regressions
snapshot drift
unapproved files

Specialist

Performance specialist gets one question:

Does extracting the formatter move work onto a hot path or duplicate serialization?

Executor

Changes only approved files.

Verifier

Prefer deterministic gates:

unit tests
integration tests
type checking
lint
build
API snapshot
benchmark
diff scope

Example verification record

PASS — 128/128 unit tests
PASS — 24/24 integration tests
PASS — 0 type errors
PASS — public API unchanged
PASS — benchmark within target
PASS — changed files within approved scope

The LLM is not the whole verifier.

It is the layer that interprets results the deterministic tools cannot fully interpret.

10.2 Research: build an evidence map instead of a pile of summaries

Question:

How do current AI search systems discover and use web sources?

Do not launch six generic “researchers.”

Partition the uncertainty:

Worker	Question
A	What do official search/platform docs say?
B	What do primary academic studies measure?
C	What current crawler/access controls exist?
D	What contradicts the optimistic interpretation?
E	How should citation visibility actually be measured?

Each worker returns structured records:

claim: "X"
source: "https://example.com/source"
source_type: "official-docs"
published: "2026-08-12"
evidence: "exact observation"
state: "SUPPORTED"
limitations:
  - "platform-specific"

The most valuable research role is often not another researcher.

It is the disconfirmation researcher.

Give one worker this job:

Find evidence that would make the emerging conclusion wrong or materially weaker.

That reduces confirmation cascades.

10.3 Content: separate epistemic work from style work

A strong technical article can use this pipeline:

Question map
    ↓
Evidence map
    ↓
Argument map
    ↓
Outline
    ↓
Draft
    ↓
Fact check
    ↓
Human-voice edit
    ↓
SEO/GEO preflight

The bad pipeline is:

SEO agent
→ writer
→ humanizer
→ GEO agent
→ headline agent
→ final polish agent

Why?

Because every agent is optimizing surface form. Nobody is clearly responsible for epistemic truth.

Content claim labels

A useful internal label set:

FACT
REPORTED RESULT
OBSERVATION
INTERPRETATION
OPINION
PROPOSAL
UNKNOWN

The published article does not need to display every label. The internal workflow should know them.

10.4 Product creation: give each role a different failure target

Use four perspectives:

Role	Question
Planner	What should exist?
User Advocate	Where will users misunderstand or abandon it?
Technical Specialist	What is expensive, risky, or fragile?
Verifier	What evidence proves the product solved the stated problem?

The User Advocate should have a concrete assignment:

Find every place where the product asks the user to understand our internal architecture instead of understanding their own job.

That produces much more actionable feedback than “review UX.”

11. Verification and measurement: make “done” observable

A workflow becomes an engineering system when it can explain why it believes it is complete.

11.1 Verification hierarchy

Use the cheapest reliable gate first.

LEVEL 1 — Mechanical
schema validation
unit tests
type checks
lint
HTTP checks
file existence

LEVEL 2 — Deterministic comparison
snapshots
diffs
benchmarks
invariants
regression datasets

LEVEL 3 — Model evaluation
semantic quality
classification
summarization fidelity
comparative judgment
style compliance

LEVEL 4 — Human review
high-stakes decisions
ambiguous interpretation
final publication
material production changes

Do not use a model to answer a question a compiler can answer.

11.2 Criterion-by-criterion verification

Bad:

Everything looks good.

Good:

Criterion: Preserve public API
Status: PASS
Evidence: API snapshot diff = 0

Criterion: Remove duplication
Status: FAIL
Evidence: legacy branch remains in src/auth/legacy.ts

Criterion: Existing tests pass
Status: PASS
Evidence: 128/128

11.3 PASS / FAIL / UNKNOWN

UNKNOWN is important.

A verifier should be allowed to say:

I could not establish this criterion.

That is better than a fabricated PASS.

11.4 Fresh context does not automatically mean independence

A fresh verifier can still be biased if you feed it the creator's conclusion.

Avoid:

The implementation succeeded. Please verify it.

Prefer:

Original goal:
...

Acceptance criteria:
...

Result:
...

Raw test evidence:
...

Determine PASS / FAIL / UNKNOWN.

The verifier gets evidence, not the verdict.

11.5 Core metrics

Metric	Definition	Why it matters
First-pass success	Verified successes on first major execution	Planning quality
Rework ratio	Reworked changes / total changes	Downstream waste
Verification catch rate	Defects found by verification / defects discovered later	Value of verification
Tokens per successful task	Total tokens / verified successes	Economics
Time to verified result	Start → verification pass	Real speed
Human escalation rate	Human interventions / tasks	Autonomy and ambiguity
Scope violation rate	Out-of-contract changes / tasks	Particularly important for coding
Evidence coverage	Sourced material claims / material claims	Research/content quality

11.6 Run an A/B test on your workflow

Instead of debating whether multi-agent is “better,” compare:

A — one agent
B — planner + critic
C — planner + critic + specialist

For the same task class, measure:

cost
latency
pass/fail
rework
defects
verification catches
human interventions

Keep additional coordination only if it improves the verified outcome enough to justify its cost.

12. Failure modes: how good-looking agent systems break

Adding agents creates a new system. New systems create new failure modes.

MAST formalized this problem with 14 failure modes across specification/system design, inter-agent misalignment, and task verification/termination.

Source: Why Do Multi-Agent LLM Systems Fail?.

The following operational table turns that research into engineering checks.

Failure mode	Typical symptom	Prevention
Infinite dialogue	Agents keep discussing after information stops changing	Hard round cap + stop condition
Role collapse	Every agent produces the same generic review	Narrow objectives + authority boundaries
Duplicate research	Multiple workers investigate the same thing	Explicit research partitions
Context leakage	Agents lose focus in irrelevant history	Scoped context + structured handoffs
Consensus theater	Agents agree because previous text sounded confident	Adversarial critic + evidence hierarchy
Plan drift	Goal changes silently during implementation	Immutable Goal Contract
Hallucination propagation	Unsupported claim becomes accepted downstream	Evidence states
LLM-only verification	Fluent “PASS” with no real test evidence	Deterministic gates
Tool hallucination	Agent invents or misuses tools	Closed-world tool registry
Shared-state corruption	Multiple workers overwrite canonical artifacts	Ownership / single-writer rule
Token explosion	Communication costs exceed useful work	Action-State compression
Latency cascade	Sequential workers multiply wait time	Parallelize independent tasks
Premature termination	Manager stops before criteria are met	Explicit acceptance tests
Framework gravity	Workflow infrastructure exceeds task complexity	Start simpler

12.1 Infinite dialogue

Set:

max_rounds = 3

Then:

if no acceptance-blocking issue remains:
    approve
else:
    escalate

Do not let the system invent reasons to keep debating.

12.2 Consensus theater

A critic should be allowed to reject a plan.

But the Planner should be allowed to reject the Critic too when the objection is unsupported.

Good adversarial collaboration is not “everyone disagrees.”

It is:

claim
→ evidence
→ challenge
→ resolution

12.3 Hallucination cascades

One unsupported claim becomes dangerous when it passes through multiple agents:

UNKNOWN
 ↓
plausible
 ↓
supported-sounding
 ↓
“fact”
 ↓
implementation decision

Your state model should make that conversion explicit.

12.4 Shared-state corruption

For important artifacts, use a single canonical writer.

Critic      → proposes change
Specialist  → proposes evidence
Planner     → updates canonical plan
Verifier    → updates verification report
Human       → approves/rejects high-impact decisions

Others propose. One owner commits.

12.5 Tool hallucination

Maintain a closed-world registry:

tool: run_tests
description: "Runs the repository's configured test suite"
permissions: [read, execute]
side_effects: "may create temporary files"

Do not let an agent assume that “there must be a tool for that.”

12.6 Token economics

A workflow can be technically successful and economically absurd.

Anthropic's published 15× token figure for multi-agent Research is a useful warning, even though it is specific to Anthropic's architecture and evaluation.

Measure:

cost per verified success

not merely:

cost per run

13. SEO, GEO, and AI-readable publishing without the folklore

Agent-created work increasingly becomes public content.

That creates a second problem: once the workflow produces a great article, how do you make it easy for people, search engines, and AI systems to discover and understand without turning the article into “SEO soup”?

The answer starts with a distinction:

Machine-readable information architecture is useful. Magic GEO hacks are not a substitute for it.

13.1 What Google actually says in 2026

Google's current guidance on AI features is unusually direct:

existing SEO best practices remain relevant,
pages need to be indexed and eligible for Search to be supporting links in AI Overviews or AI Mode,
there are no extra technical requirements specifically required for those AI features,
important content should be available in textual form,
internal links, crawlability, page experience, and useful original content remain important,
there is no special schema.org markup required for AI Overviews or AI Mode.

Google also says meeting best practices does not guarantee crawling, indexing, or serving.

Sources:

That last point matters because it kills one of the worst forms of AI-search marketing:

“Do these three things and Google AI will cite you.”

There is no such universal guarantee.

13.2 Search Console now gives a better measurement layer

Google introduced Search Generative AI performance reporting in June 2026 and said it had rolled those insights out worldwide by August 31, 2026.

The reports expose visibility within generative AI features on Search, including AI Overviews and AI Mode, inside Search Console's performance reporting system.

That is a major practical improvement because it gives publishers a first-party measurement surface rather than forcing all AI-search analysis into third-party guesses.

Source: Google Search Central — Search Generative AI performance reports.

Use that data where available.

13.3 What “AI-readable” should mean

A useful page should let a machine answer:

What is this page about?
Who wrote it?
When was it published/updated?
What are the major claims?
What evidence supports them?
Which sections answer which questions?
Which parts are facts versus interpretations?
Where are the primary sources?
What is the canonical version?

That is not special AI formatting.

It is good information architecture.

13.4 Structure for long-form reference content

A durable article structure:

Title
↓
TL;DR
↓
Table of Contents
↓
Definition / thesis
↓
Evidence
↓
Counterexamples
↓
Architecture
↓
Implementation
↓
Examples
↓
Failure modes
↓
Measurement
↓
Limitations
↓
Templates
↓
Sources

That structure helps humans and retrieval systems for the same underlying reason: it reduces ambiguity.

13.5 Structured data: useful, not magical

For an article, relevant Schema.org properties may include:

Article
author
datePublished
dateModified
headline
image
mainEntityOfPage

But the markup should describe the visible content accurately.

Do not claim that Article schema guarantees AI citations or rankings.

Google's structured-data documentation explicitly says structured data helps it understand content, while eligibility and appearance depend on additional systems and requirements.

Sources:

13.6 robots.txt is one layer, not the whole stack

Google says robots.txt controls crawler access; it is not a general noindex mechanism.

To keep pages out of Google Search, Google points to noindex or authentication rather than relying on robots.txt alone.

Sources:

For AI crawlers, the same practical truth applies: the page can be allowed by robots.txt and still fail through other infrastructure.

Think in layers:

robots.txt
   ↓
WAF / CDN
   ↓
bot mitigation
   ↓
authentication
   ↓
rate limiting
   ↓
JavaScript challenges
   ↓
HTTP response
   ↓
rendering / retrieval

OpenAI documents OAI-SearchBot for web search-related discovery and provides publisher guidance about allowing access. Anthropic documents separate crawler identities such as ClaudeBot, Claude-User, and Claude-SearchBot.

Sources:

13.7 `llms.txt`: reasonable experiment, not magic SEO

The llms.txt project is a community proposal for presenting curated machine-oriented information about a site.

It can be useful as documentation.

It is not a universal Google requirement.

Google's current generative AI guidance explicitly says you do not need to create special AI text files such as llms.txt to appear in its generative AI search features.

Sources:

Use it because it serves a clear information-architecture purpose, not because someone sold it as a ranking switch.

13.8 GEO measurement is more than citation count

Recent 2026 research is pushing toward a more precise model of AI-search visibility.

One April 2026 study separates:

citation selection
≠
citation absorption

A page can be retrieved and cited without materially contributing to the generated answer.

A July 2026 survey similarly describes GEO as a multistage process involving discoverability, retrieval, reranking, citation, prominence, factual absorption, and downstream user behavior.

Sources:

This suggests a better measurement stack:

Discoverability
↓
Retrieval
↓
Citation
↓
Citation position / prominence
↓
Answer contribution
↓
Factual fidelity
↓
Traffic / conversions

That is much more informative than “AI visibility score = 83.”

13.9 What recent GEO research suggests — carefully

2026 studies are finding associations and controlled effects involving factors such as topical relevance, context position, explicit factual information, structure, and evidence richness. One competitive GEO study evaluated 252,000 controlled trials across six LLMs; another large ACL 2026 study explores latent user-demand alignment; a separate framework studies how citation influence differs from citation selection.

These are valuable research directions.

But they should not be flattened into universal advice such as:

“Write exactly X words, add Y headings, and every AI will cite you.”

The stronger practical claim is safer:

Make important information explicit, relevant, well-supported, easy to retrieve, and faithful to the source material.

Sources:

13.10 Practical GEO/SEO preflight

TECHNICAL
[ ] Crawlable when intended
[ ] Indexable when intended
[ ] Canonical URL is correct
[ ] Important text is accessible
[ ] No accidental noindex
[ ] Internal links are present
[ ] HTTP responses are healthy

CONTENT
[ ] Clear title
[ ] One clear H1
[ ] Coherent heading hierarchy
[ ] Direct answers to major questions
[ ] Definitions before jargon
[ ] Material claims supported by sources
[ ] Facts separated from interpretation
[ ] Publication/update date
[ ] Author / provenance

AI ACCESS
[ ] Intended AI crawler policy is deliberate
[ ] WAF/CDN does not silently block intended access
[ ] Authentication is understood
[ ] Rate limits are sane
[ ] JavaScript challenges are not accidental blockers

STRUCTURE
[ ] Accurate Article/Organization/etc. structured data where appropriate
[ ] Structured data matches visible content
[ ] Stable URLs
[ ] Duplicate versions controlled
[ ] Sources are easy to follow

For a practical public-page preflight, AuditMe's Website SEO Checker can inspect a URL and surface issues across technical SEO, content structure, schema, performance, and related readiness dimensions.

That makes it useful as a diagnostic pass.

It is not proof that an AI system will cite the page.

AuditMe links for the workflow

The right way to use an audit tool in this process is as preflight evidence: find technical and structural problems before asking a search engine or AI system to discover the page.

14. The reusable operating kit

This section is designed to be copied.

You can put the templates in a repository, a knowledge base, a prompt library, or a workflow engine.

14.1 Goal Contract template

# Goal Contract

## Goal
[One precise sentence]

## Why
[Why the work matters]

## Constraints
- [constraint]
- [constraint]

## Forbidden Changes
- [forbidden change]
- [forbidden change]

## Definition of Done
- [measurable criterion]
- [measurable criterion]

## Unknowns
- [unknown]
- [unknown]

14.2 Planner prompt

You are the Planner.

Goal:
[goal]

Constraints:
[list]

Forbidden changes:
[list]

Definition of done:
[criteria]

First inspect the real evidence available to you.
Do not execute implementation.
Do not invent architecture.
Mark assumptions and unknowns explicitly.

Return:
1. Observed facts
2. Assumptions
3. Proposed plan
4. Dependencies
5. Risks
6. Acceptance tests
7. Rollback/recovery path

14.3 Critic prompt

You are the Critic.

Assume the current plan is wrong.
Find the smallest number of high-impact reasons it could fail.

Prioritize:
- goal violations
- incorrect assumptions
- hidden dependencies
- missing tests
- security/reliability risks
- unnecessary complexity
- scope creep

Do not rewrite the plan.
Return each issue as:

Action:
State:
Result:
Evidence:
Confidence:
Next needed:

14.4 Specialist prompt

You are a domain specialist.

Question to resolve:
[one specific question]

Do not review the entire project.
Do not redesign unrelated systems.
Prefer primary documentation, code inspection, tests, or reproducible measurements.

Return:
Action:
State:
Result:
Evidence:
Confidence:
Open questions:

14.5 Verifier prompt

You are the Verifier.

Original Goal:
[goal]

Definition of Done:
[criteria]

Inspect the result independently.
Do not rely on the creator's explanation.
Prefer deterministic evidence whenever available.

For every criterion return:
PASS / FAIL / UNKNOWN

For each result include:
- evidence
- blockers
- defects found
- repair needed

Final decision:
GO / CONDITIONAL GO / NO-GO

14.6 Human-voice editor prompt

You are the final human-voice editor.

Do not make the article more enthusiastic.
Make it more credible and authored.

Remove:
- generic AI transitions
- inflated adjectives
- repeated conclusions
- fake certainty
- unnecessary headings
- repetitive sentence rhythms

Preserve:
- technical specificity
- source attribution
- disagreement
- uncertainty
- concrete examples
- author judgment

Do not invent personal experience, clients, benchmarks, or case studies.

14.7 Agent handoff schema

from: "critic"
to: "planner"
action: "reject_plan_step"
state: "orphaned routes are possible"
result: "add crawlability and canonical acceptance test"
evidence:
  - "route-manifest.json"
  - "crawler-output.json"
confidence: "high"
next_needed: "revise_plan"

14.8 Suggested repository layout

.agent/
├── goal-contract.md
├── plan.md
├── evidence-ledger.md
├── decisions.md
├── verification.md
├── prompts/
│   ├── planner.md
│   ├── critic.md
│   ├── specialist.md
│   ├── verifier.md
│   └── human-voice-editor.md
└── runs/
    ├── 2026-09-22-run-001.md
    └── 2026-09-23-run-002.md

The exact directory does not matter.

The separation does.

14.9 Minimum Viable Loop — 15 to 30 minutes

0–5 minutes

Write:

Goal
Constraints
Forbidden changes
Definition of done
Unknowns

5–10 minutes

Plan Mode / planner pass.

10–15 minutes

Fresh-context Critic.

15–20 minutes

Resolve evidence-backed corrections.

20–25 minutes

Execute one checkpoint.

25–30 minutes

Verify against the original contract.

That is enough to start.

You do not need a distributed orchestration platform to get the core benefit.

14.10 Compact checklist

Before execution

[ ] Goal Contract exists
[ ] Constraints explicit
[ ] Forbidden changes explicit
[ ] Definition of done measurable
[ ] Real evidence inspected
[ ] Unknowns labeled
[ ] Plan falsifiable
[ ] Critic allowed to reject
[ ] Specialists have narrow assignments
[ ] Canonical plan has one owner

During execution

[ ] Independent work parallelized only where useful
[ ] Handoffs pass state, not speeches
[ ] Evidence provenance preserved
[ ] Tool permissions scoped
[ ] Checkpoints explicit
[ ] Scope remains inside contract
[ ] Raw test outputs retained

Before shipping

[ ] Deterministic checks passed
[ ] Material claims verified
[ ] Contradictions resolved or labeled
[ ] Final artifact matches Goal Contract
[ ] Unknowns documented
[ ] Verification is independent enough to matter
[ ] Cost / latency / rework measured
[ ] Final artifact is reusable

14.11 The 12 rules worth remembering

1. Start with one agent.

Do not architect a council before you can name the bottleneck.

2. Split by uncertainty, not by job title.

A specialist exists because something requires different context, tools, expertise, or independent judgment.

3. Parallelize independent work.

That is one of the clearest reasons to use multiple agents.

4. Keep the Goal Contract stable.

Otherwise the system can “succeed” by changing the problem.

5. Make the plan canonical.

Do not make chat history your database.

6. Pass state, not speeches.

Action + State + Result is a practical default.

7. Attack before executing.

A wrong assumption in Markdown is cheap. A wrong assumption inside a production diff is not.

8. Verify with the cheapest reliable method.

Compiler over conversation. Test over opinion. Source over memory.

9. Treat UNKNOWN as a valid state.

Missing evidence is information.

10. Measure verified outcomes.

Tool calls are activity. A passing acceptance test is evidence of progress.

11. Keep the framework smaller than the problem.

If orchestration is harder than the underlying task, simplify.

12. Stop when the goal is met.

More dialogue is not automatically more quality.

14.12 Final thought

The agent era will produce a lot of spectacular demos.

Some of them will be useful.

A lot of them will also confuse activity with progress.

Ten agents can talk for twenty minutes and produce nothing that a human can safely ship.

One agent can solve a hard problem with a good plan and a real test.

The interesting engineering problem is therefore not:

How many agents can I make collaborate?

It is:

What information must survive from one stage to the next so that the system can move from intention to evidence-backed action and finally to verified completion?

That is the Iterative Creation Loop.

The simplest implementation is still often the best:

Goal Contract
→ Plan
→ Adversarial Critique
→ Narrow Specialist when needed
→ Checkpoint
→ Deterministic Verification
→ Independent Review
→ Learn

Start there.

Measure it against the single-agent version.

Add complexity only when it buys something you can observe.

That is the part worth scaling.

Not the number of agents.

The quality of the loop.

Editorial note

This guide intentionally separates vendor-reported engineering results, academic findings, official product documentation, and practical heuristics.

Vendor metrics are not universal benchmarks. Academic results depend on benchmark design and experimental conditions. Product documentation describes the publisher's current stated behavior, not every possible real-world outcome. GEO measurements can vary by query, engine, source set, retrieval process, model, and time.

The most useful experiment remains the least glamorous one:

Run the same class of task with and without additional coordination. Measure the verified outcome. Keep only the complexity that actually improves it.

SEO Didn’t Die. Websites Got Harder to Understand. — The 2026 Field Guide to SEO, GEO, AI Search & Website Intelligence

Eduard — Tue, 22 Sep 2026 00:22:57 +0000

SEO Didn’t Die. Websites Got Harder to Understand.

The 2026 field guide to SEO, GEO, AI search visibility, agent readiness, and website intelligence — built from real evidence, not SEO theatre.

There is a strange problem with the SEO industry right now: everyone is using the same vocabulary while talking about different measurements.

A platform says AI visibility. Another says GEO. Another talks about SEO intelligence. Another measures citations. An analytics product reports AI traffic. An agency promises to make a company "the answer" in ChatGPT. A crawler checks thousands of URLs and calls that intelligence.

All of these can be useful. They are not the same job.

That distinction is becoming more important as search moves from a simple list of blue links toward a mixture of search results, AI-generated answers, recommendations, citations, browser-based research, and agentic interactions.

The question is no longer simply:

Does my website rank?

A modern growth team needs to ask several different questions:

Can search engines crawl and understand the site?
Is the page technically healthy?
Can humans understand the proposition quickly?
Can machines extract the facts accurately?
Is important information represented consistently across HTML and structured data?
Does an AI system mention or cite the brand for the prompts that matter?
Can an agent discover pricing, documentation, products, actions, and APIs?
When something is wrong, can a developer see the evidence and know exactly what to change?
After the fix, can the system verify that the state actually improved?

Those questions belong to different layers of the stack.

This guide maps that stack, compares eight real products that a buyer may encounter while researching it, and explains where AuditMe fits without pretending to be something it is not.

Research note: Product capabilities and public positioning change quickly. The competitor sections below reflect public pages checked around September 21, 2026. They are descriptions of product scope, not independent performance certifications. Pricing is omitted where a current public price was not clear enough to quote responsibly.

A note from the builder: I built AuditMe because I kept running into the same irritating gap: a site could receive a neat SEO score while the underlying evidence was fragmented across HTML, headers, performance tools, Search Console, structured data and — increasingly — AI systems. The score was convenient. The investigation was not. This article is an attempt to document the investigation itself.

This is not a list of "10 hacks for GEO." It is a field guide for people who want to know what a website actually exposes to search engines, AI systems, developers and humans — and what can be proved from the available evidence.

What you will get: a market map, a practical measurement model, a real AuditMe self-audit, implementation examples, official documentation, and a reference architecture that can survive beyond the current SEO buzz cycle.

What you will not get: a fake winner, invented case studies, guaranteed AI citations, or a claim that one dashboard can replace an entire growth stack.

Contents

TL;DR: the useful mental model
The web changed before most SEO dashboards did
Eight products, eight different jobs
Where AuditMe is genuinely different — and where it is not
The comparison matrix that actually helps
SEO, GEO, AI visibility, and agent readiness are related — but they are not synonyms
The idea that can make AuditMe memorable: evidence provenance
A real AuditMe audit is more revealing than a perfect demo
The 12-section report: from PDF dump to decision system
Developers should be able to turn findings into code and tickets
What “agent-ready” should mean in practice
What AuditMe should borrow from the competition
The 2026 workflow: how the layers should work together
The reference library: official docs first, product claims second
The conclusion: the next generation of SEO software is not a bigger checklist

01 — TL;DR: the useful mental model

The market is easier to understand when you stop treating every product as an "SEO tool."

Think in layers:

Layer	Main question	Examples in this comparison
Technical website intelligence	What is actually present, broken, missing, or measurable on the site?	AuditMe
Site-wide SEO operations	What is happening across the whole site, and what should the SEO team do next?	Foudroyer
Traditional SEO research	Which keywords, rankings, pages, and backlinks matter?	KatLinks
AI answer visibility	When buyers ask AI systems, does the brand appear, where, and against whom?	Beamtrace, WildSEO
Human strategy + execution	What should the business do, and who will implement it?	MagicSpace
Marketing attribution	Which campaigns and channels produce conversions?	Captflow
Privacy-first web analytics	What traffic, behavior, and conversions happen on the site?	Trackboxx

AuditMe's intended center is the first layer, with important overlap into AI/GEO and agent readiness.

The core workflow is simple:

URL → evidence → signals → diagnosis → priority → fix → verify

That does not make AuditMe universally "better" than the other products. It makes the product useful for a specific job: turning a website into an evidence-backed diagnostic model that a human, developer, or automated workflow can act on.

The competitive mistake would be to turn that identity into a feature-shopping contest and then chase every neighboring category at once.

The useful differentiator is not the number of boxes on the dashboard. It is the quality of the chain from observation to action.

The better strategy is to make the evidence layer so clear and useful that other layers can plug into it.

02 — The web changed before most SEO dashboards did

For a long time, a website optimization workflow could be summarized as:

crawl → optimize → rank → get traffic

That workflow still exists. It is not obsolete.

What changed is what can happen after the page becomes discoverable.

A user might now:

search Google and see a classic result;
receive an AI-generated overview;
ask ChatGPT for a recommendation;
ask Gemini or Claude a product-comparison question;
click a citation instead of a traditional organic result;
use an AI browser or agent to inspect a site;
arrive through an AI referral that looks like normal web traffic in the analytics layer;
never visit the site at all, because the answer was sufficient.

This creates a measurement problem.

A Google position is not an AI answer position. An AI citation is not a click. A click is not a conversion. A referral visit is not proof that a page was technically strong. And a technical audit does not tell you how often competitors are recommended in a given set of AI prompts.

That sounds obvious, but product marketing constantly blends these signals together because a single number is easier to sell than a system of measurements.

The result is a growing number of dashboards where different kinds of evidence are presented as if they were interchangeable.

They are not.

The five questions a modern website stack should keep separate

1. Access: Can the relevant systems retrieve and process the page?

2. Quality: What does the site actually contain, and what is wrong with it?

3. Interpretation: Can machines understand entities, relationships, claims, pricing, documentation, and context?

4. Visibility: Do search engines or AI systems actually surface the brand or its content for important questions?

5. Outcome: Did any of this produce qualified traffic, leads, conversions, or revenue?

A serious system can connect those questions without pretending they are one metric.

That separation is the foundation for the rest of this article.

03 — Eight products, eight different jobs

The easiest way to make a bad software decision is to compare screenshots instead of jobs.

The products in the original AuditMe comparison are particularly useful because several of them are not direct substitutes at all.

A useful test: finish the sentence "I need this tool because I want to ___". If the blank is "track AI answers," "manage indexation," "understand conversions," or "get an SEO strategy implemented," you are already describing different product categories.

AuditMe — website intelligence and evidence-backed auditing

AuditMe starts with a URL and examines a broad set of website signals: technical SEO, metadata, content quality, performance, links, images, structured data, accessibility, security headers, user-experience signals, knowledge-graph signals, AI search readiness, and agent-readiness signals.

Its differentiator is not "more checks." It is the attempt to connect observations into a useful chain:

evidence → finding → severity → impact → fix → verification

The public product currently promotes a roughly 60-second URL audit with no signup required for the free path, plus a multi-section PDF report and an API/MCP direction. AuditMe AuditMe API

Beamtrace — AI search visibility and competitive analysis

Beamtrace is much more focused on the question of how a brand appears inside AI-generated answers.

Its public product pages describe:

visibility scores;
prompt-level performance;
competitor benchmarking;
average position in AI answers;
mentions;
citation analysis;
original answer context;
trend analysis;
cross-platform AI search analytics.

Beamtrace explicitly lets users inspect prompts where competitors outperform them and look at the cited sources behind answers. Beamtrace competitor analysis Beamtrace citation analysis

That is a different measurement from a technical website audit.

Foudroyer — SEO operations, crawling, analytics, keywords, and indexation

Foudroyer presents itself as an all-in-one AI-assisted SEO platform. Its public homepage currently highlights full-site audits, analytics, keyword tracking, indexation management, sitemap monitoring, and task management. It says its audits crawl an entire website, similar to a crawler such as Screaming Frog, and it also advertises Search Console-related indexation workflows. Foudroyer

The important distinction is scale and operating model.

Foudroyer is trying to run the SEO operation across the site. AuditMe's free public product starts from the single URL and builds a deeply explained evidence report around what was captured.

MagicSpace — managed SEO and AI-search execution

MagicSpace is an agency rather than a self-serve diagnostic SaaS product.

Its current public positioning is explicit: it works with SaaS companies to drive signups from Google and AI, offers live audits and SEO services, helps with implementation, provides coding and product guidance, and runs link-building and AI SEO initiatives. It also sells training products around AI/LLM SEO and programmatic SEO. MagicSpace

The distinction matters because an agency can own decisions and implementation in a way software cannot automatically do.

KatLinks — accessible traditional SEO tooling

KatLinks focuses on traditional SEO workflows: keyword rank tracking, keyword research, backlinks, on-page audits, backlink gaps, backlink opportunities, and SEO checklists. Its public homepage currently advertises a 58-point on-page inspection. KatLinks

Its usefulness is not in pretending to be a universal website intelligence system. It is in making familiar SEO workflows comparatively straightforward and affordable.

Captflow — privacy-first marketing attribution

Captflow is a different layer again. Its homepage emphasizes privacy-first analytics without cookies or pixels, channel and campaign measurement, goals, conversion funnels, and a GDPR/CCPA-oriented positioning. Captflow

Captflow answers:

Which marketing efforts produced conversions?

AuditMe answers a prior question:

What is happening on the website itself, and what should we fix?

WildSEO — ongoing AI search intelligence

WildSEO is firmly in the AI-search monitoring category. Its public homepage currently lists tracking across ChatGPT, Google AI Mode, Gemini, Claude, Perplexity, DeepSeek, Grok, Meta AI, Mistral, and Qwen, plus prompt-level reporting, citation monitoring, competitor benchmarking, AI crawler analytics, Search Console insights, answer auditing, alerts, and content workflows. WildSEO

Its central promise is ongoing visibility intelligence:

See why AI recommends your competitors. Then change the answer.

That is highly relevant to AuditMe, but it is still a different primary question from a site-level evidence audit.

Trackboxx — privacy-oriented web analytics

Trackboxx positions itself as a German, cookie-free, GDPR-oriented alternative to conventional web analytics. Its current public site highlights traffic sources, content performance, ecommerce funnels, conversions, and a live demo. Trackboxx

Again, this is outcome measurement rather than diagnosis.

04 — Where AuditMe is genuinely different — and where it is not

This is the section most product comparisons get wrong.

A company has a much stronger long-term position when it can say, plainly, "this is ours, this is theirs, and here is the boundary."

AuditMe's strongest territory

AuditMe is unusually well positioned around evidence-rich site diagnosis.

The current engine already spans a broad range of dimensions, but the more interesting part is what happens after a check fails.

For example, a good finding is not:

Schema issue — fix schema.

It is closer to:

Visible page price: $29. Structured-data offer price: $0.

Why it matters: different systems can receive conflicting commercial information.

Fix: align the JSON-LD Offer price with the value actually shown on the page.

Verify: rerun the audit and re-check the structured data.

That is an engineering artifact, not just a score.

AuditMe is also broader than a conventional SEO audit

The product includes dimensions and checks around:

structured data;
semantic HTML;
accessibility;
security headers;
AI search readiness;
entity clarity;
content answerability;
machine readability;
agent-oriented discoverability.

The distinction matters because AI systems are still systems operating on web documents. A page that cannot expose its facts clearly, that contradicts itself, or that is impossible to navigate reliably is not magically fixed by adding an "AI" label.

Where AuditMe is currently behind specialized AI-visibility products

This should be said directly.

If the job is:

Track a large set of commercial prompts across AI engines every day; show where competitors appear; inspect answer positions; monitor citation changes; and alert me when visibility drops.

then Beamtrace and WildSEO are closer to that dedicated monitoring job today. Their public products are built around prompt-level AI-answer visibility, competitor context, citations, and recurring monitoring. Beamtrace WildSEO

AuditMe's public center of gravity is still the website audit.

That is not an embarrassment. It is a product boundary.

Where AuditMe is currently behind site-wide SEO operations

If the requirement is:

Crawl the whole domain, watch keyword movement, manage indexation, handle Search Console workflows, and turn findings into a recurring SEO operations queue.

then Foudroyer is closer to that job. Its public site explicitly markets full-site crawling, keyword tracking, analytics, indexation management, sitemap monitoring, and tasks. Foudroyer

A one-URL audit should never be marketed as equivalent to a full-site crawl.

Where AuditMe is not trying to compete

With MagicSpace, the comparison is mostly about service architecture. MagicSpace can provide strategy, coding help, content work, links, AI SEO execution, and an ongoing relationship. AuditMe can automate diagnostics and produce evidence; it does not magically become a human agency because its PDF is 20 pages long. MagicSpace

With Captflow and Trackboxx, the difference is even clearer: they measure acquisition and conversion behavior after traffic reaches a property. Captflow Trackboxx

With KatLinks, AuditMe's overlap is strongest around on-page diagnosis, while KatLinks goes deeper into traditional keyword and backlink workflows. KatLinks

05 — The comparison matrix that actually helps

A giant feature checklist can create the illusion of objectivity while hiding the most important fact: the products are optimized for different units of analysis.

The more honest matrix is capability-by-capability.

Detailed 2026 Comparison Matrix (2 Parts)

Capability	AuditMe	Beamtrace	Foudroyer	MagicSpace
URL-level technical audit	Core	Secondary	Core	Service
Whole-site crawl	Crawl workflows available; single-page audit is distinct	No	Core	Service
Keyword rank tracking	Not primary positioning	No	Core	Service
Keyword research	Not primary	No	Core	Service
Backlink intelligence	Supporting layer	Limited	SEO workflow	Service
AI answer visibility	AI/GEO readiness layer	Core	Secondary	Service
Prompt-level AI tracking	Not the primary public workflow	Core	No	Service
AI competitor visibility	Emerging / indirect	Core	Traditional SEO competitor workflows	Service
Citation/source analysis	Site citation potential and provenance	Core	Secondary	Service
AI crawler signals	Core	Secondary	Limited	Service
Schema / structured data	Core	Secondary	Core	Service
Performance diagnostics	Core	Secondary	Core	Service
Accessibility	Core dimension	Secondary	SEO-adjacent	Service
Security headers	Core dimension	No	SEO-adjacent	Service
Agent-readiness checks	Core direction	Limited	Limited	Human service
Evidence provenance	Signature	Answer/source context	Operational reporting	Human interpretation
Developer fix guidance	Core report behavior	Optimization recommendations	Task workflows	Human implementation
Structured PDF report	Core artifact	Reporting	Reporting	Service deliverables
Self-serve	Core	Core	Core	No — managed service
Human strategy / execution	Not core	No	Workflow support	Core
Privacy-first analytics	No	No	No	No
API / automation	API + MCP direction	Integrations	Integrations/workflows	Service integration

Capability	KatLinks	Captflow	WildSEO	Trackboxx
URL-level technical audit	Core	No	Secondary	No
Whole-site crawl	More limited	No	Secondary	No
Keyword rank tracking	Core	No	AI-search focus	GSC/analytics context
Keyword research	Core	No	No	No
Backlink intelligence	Core	No	Secondary	No
AI answer visibility	Limited	No	Core	AI traffic context
Prompt-level AI tracking	No	No	Core	No
AI competitor visibility	Traditional SEO	No	Core	No
Citation/source analysis	Limited	No	Core	Traffic-level signals
AI crawler signals	No	No	Core	AI traffic
Schema / structured data	On-page	No	GEO/semantic	No
Performance diagnostics	On-page	No	Secondary	No
Accessibility	On-page	No	Secondary	No
Security headers	No	No	Secondary	No
Agent-readiness checks	No	No	Partial / adjacent	No
Evidence provenance	Audit detail	Attribution	Citation/answer detail	Attribution
Developer fix guidance	Checklist/fixes	No	Content recommendations	No
Structured PDF report	Audit reporting	Reporting	Reporting	Reporting
Self-serve	Core	Core	Core	Core
Human strategy / execution	No	No	Some workflow support	No
Privacy-first analytics	No	Core	No	Core
API / automation	Varies	Analytics integrations	Enterprise integrations	Analytics integrations

The point of the matrix is not to produce a ranking. It is to prevent a category error.

A dedicated AI-visibility product can have a better prompt-monitoring workflow while offering a weaker technical audit. A crawler can have stronger site-wide operations while offering less evidence provenance. An agency can provide more implementation capacity while offering less automation.

That is exactly what a mature buying guide should tell the reader.

06 — SEO, GEO, AI visibility, and agent readiness are related — but they are not synonyms

These terms are often collapsed into one bucket. That makes technical discussions less precise.

SEO

SEO is the broad discipline of improving a site's ability to be crawled, understood, indexed, surfaced, and used in search systems.

Google's documentation remains clear that the fundamentals still matter for AI-powered search experiences. Existing SEO practices are not made irrelevant just because the result page now includes generative features.

Sources:

GEO

"GEO" is an industry term with no single universal technical specification.

Depending on who uses it, GEO can mean:

generative engine optimization;
answer-engine optimization;
AI-search visibility work;
citation acquisition;
entity and semantic optimization;
content designed to be extracted accurately by generative systems.

A responsible GEO strategy should therefore state exactly what is being measured.

AI visibility

AI visibility is narrower and easier to define:

How often, where, and in what context a brand appears in AI-generated answers to a defined set of prompts.

Products such as Beamtrace and WildSEO are much closer to this measurement. Their public pages emphasize prompt tracking, answer position, competitors, citations, and trends. Beamtrace WildSEO

Agent readiness

Agent readiness asks a different question:

Can an automated agent understand and operate the website reliably enough to accomplish a task?

That includes more than crawler access.

An agent may need to:

identify the site's entity;
understand product or service information;
find pricing;
find documentation;
follow navigation;
interact with buttons and forms;
discover APIs;
complete a task;
understand success or failure;
operate with minimal dependence on fragile client-side rendering.

This is why AuditMe's current agent-readiness layer includes signals around JavaScript dependency, no-JS content, navigation, URL predictability, product discovery, pricing discovery, documentation, actions, task completion, API endpoints, machine-readable relationships, and hidden content.

Those are not the same thing as SEO ranking signals. They are an attempt to measure whether a machine can actually use the web property as a system.

07 — The idea that can make AuditMe memorable: evidence provenance

Most audit products are proud of how many checks they run.

I am more interested in what happens when a check cannot be run.

That is where trust begins.

While building AuditMe, one lesson kept repeating: a diagnostic number is only as useful as its provenance. A synthetic LCP, a field LCP, a DOM observation, a Search Console metric, and an inferred conclusion are not interchangeable just because all five can be printed inside a card.

So the audit engine already has an evidence model that distinguishes concepts such as:

captured versus inferred;
available versus unavailable;
lab versus field;
provider/source;
confidence.

This should become a visible product language, not a hidden implementation detail.

The difference is easy to see

Bad:

LCP: 15.31s

Better:

LCP: 15.31s — LAB / PageSpeed Insights

Better still:

LCP: 15.31s — LAB / PageSpeed Insights — captured 2026-09-20

And when the data does not exist:

CrUX: Not captured for this URL

That last state is important. A platform becomes less trustworthy when it silently fills an empty cell with a guess.

Provenance should be a first-class UI pattern

Use a compact vocabulary consistently:

Badge	Meaning
`LAB`	Synthetic or laboratory measurement
`FIELD`	Real-user or field measurement
`CAPTURED`	Directly observed evidence
`INFERRED`	Derived from available evidence rather than directly observed
`NOT CAPTURED`	Data source unavailable in this audit

The same discipline belongs in GEO.

Instead of:

AI visibility = 63%

show:

AI visibility = 63% · 400 tracked prompt runs · ChatGPT · Sep 1–21, 2026

The number becomes interpretable because the measurement protocol is visible.

Why this matters for AI systems too

Retrieval systems benefit from explicit facts, dates, definitions, and source boundaries.

An AI-friendly article is not an article stuffed with the phrase "GEO optimization" 40 times. It is a page where an agent can reliably answer:

What is this product?
What does it measure?
What data supports the claim?
Which part is documented versus inferred?
What changed over time?
Where is the source?

That is why evidence provenance is not just a reporting feature. It is part of the information architecture.

08 — A real AuditMe audit is more revealing than a perfect demo

Here is where the marketing story could have taken an easy shortcut.

It could have published a 100/100 audit of AuditMe itself.

Instead, the September 20, 2026 snapshot used for the current product discussion reported:

Metric	Real AuditMe snapshot
Overall score	83 / 100
Grade	B
Checks passed	106 / 158
AI / GEO Readiness	89 / 100
LCP	15.31s — LAB / PSI
Speed Index	7.71s — LAB / PSI
TBT	300ms — LAB
Potential score recovery	up to +17 points

The same report surfaced concrete issues including:

Largest Contentful Paint;
Image Alt Text;
Speed Index;
Schema Price Consistency;
Hidden Content Detection.

The audit also showed that some sources were unavailable in that run: no CrUX coverage for the target URL, no connected Search Console dataset, and no full-site crawl in that specific scan.

That distinction is important because it exposes a rule I would like more SEO software to adopt:

A long report is not proof of deep evidence. Coverage is proof of coverage.

The report itself is a dated snapshot, not a universal claim about customer sites.

The schema example is especially useful

The report found a visible product price of $29 while structured data claimed $0.

That sounds trivial until you consider how many systems can consume those conflicting facts.

A human sees one value. A parser sees another. A search engine or downstream agent may have to choose.

That is exactly the kind of issue that disappears inside a vague "schema score" but becomes obvious inside an evidence-first finding.

And yes, it is slightly embarrassing that AuditMe's own site had it.

I kept the finding in the report on purpose. While building the parser, we had to decide whether a mismatch like $29 on the page versus $0 in JSON-LD was worth a dedicated check or should disappear inside a generic schema score. We kept it. A buyer does not experience "schema quality" as an abstract percentage; they experience a page whose important facts either agree or do not.

That small decision captures the product philosophy better than a slogan does: when a contradiction is observable, surface it.

The same thinking applies to performance, indexation, AI-readiness and agent workflows. The report should expose the awkward parts because those are usually the parts somebody needs to fix.

That is useful.

A diagnostic tool that never discovers anything wrong with itself is not automatically impressive. It may simply be grading its own homework.

09 — The 12-section report: from PDF dump to decision system

A serious report should not feel like a pile of screenshots exported to PDF.

The goal is to move a reader through a sequence of decisions.

AuditMe's intended public architecture is a 12-section report. A September 20 internal PDF snapshot contained additional top-level diagnostic blocks, which exposed exactly why a centralized report-sections registry is needed: the homepage promise and PDF structure should never drift apart.

The consolidated architecture is:

01 — Executive Intelligence

The answer to:

What is happening, what matters, and what do I do first?

Include:

overall score and grade;
checks passed;
confidence;
strongest and weakest dimensions;
top issues;
top quick wins;
recovery potential;
AI/GEO snapshot.

02 — Dimension Health

Show the site's health matrix across all audit dimensions.

Avoid an enormous radar chart as the only representation. A radar can be a useful visual summary, but the underlying rows should remain readable and accessible.

03 — Findings & Fix Queue

Every important finding should expose:

Evidence → Why → Impact → Fix → Effort → Confidence → Verify

This is where AuditMe has a chance to turn the report into a handoff artifact for marketing and engineering.

04 — Priority Map

Use impact × effort to explain sequencing.

The objective is not to create another attractive 2×2 matrix. It is to make the prioritization logic inspectable.

05 — Search Result Preview

Show how the current page can appear in search:

title;
description;
URL;
character length;
canonical state;
Open Graph consistency;
structured metadata.

06 — 30-Day Action Plan

A prioritized schedule is more useful than 40 warnings dumped into one list.

The real September snapshot already contains a 30-day sequence, with weeks organized by the estimated impact and effort of fixes.

07 — Revenue Impact Scenario

This section must remain explicitly hypothetical.

A model can say:

If organic revenue is $10,000/month and the modeled uplift assumption is 14%, that scenario corresponds to approximately $1,400/month.

It must not say:

Fix these issues and you will make $1,400 more per month.

One is a scenario. The other is an unjustified forecast.

08 — Crawl & Indexation Health

This section should make it obvious whether the audit actually captured:

robots.txt;
sitemap;
indexability;
canonicalization;
redirects;
crawl data;
Search Console state.

09 — Performance Intelligence

Separate:

LAB;
FIELD;
NOT CAPTURED.

Do not hide the source under a tooltip.

10 — AI/GEO + Agent Readiness

Keep the dimensions distinct.

The current engine can expose AI/GEO, AI Search Readiness, and Agent Readiness as different concepts. Those should not collapse into one generic "AI score" simply because one number fits a hero better.

11 — Detailed Findings

The full category-by-category diagnostics live here.

12 — Evidence Quality & Methodology

This is where the report explains:

what was measured;
what was inferred;
what was unavailable;
which providers were used;
how scores were calculated;
how benchmarks should be interpreted.

The methodology section is not an appendix nobody needs.

For a product built around evidence, it is part of the product.

10 — Developers should be able to turn findings into code and tickets

This is one of the clearest ways AuditMe can move beyond generic SEO content.

A developer does not need another paragraph saying "optimize your metadata."

They need enough evidence to act.

Example: structured-data consistency

An actionable finding looks like this:

ISSUE
Schema Price Consistency

SEVERITY
High

EVIDENCE
Visible price: $29
JSON-LD Offer price: $0

WHY
Two machine-readable representations describe different commercial values.

ACTION
Align the structured-data Offer price with the visible page price.

VERIFY
Re-run the audit and validate the JSON-LD after deployment.

That structure can become:

a GitHub issue;
a Linear ticket;
a Jira task;
an n8n workflow item;
a CI failure;
an agent instruction.

AuditMe API: the programmatic layer

AuditMe's public API is deliberately simple: the current documentation exposes a GET endpoint, requires no API key for the basic public audit, and returns structured JSON. The documented public rate limit is 10 requests per minute per IP. That makes the API useful for lightweight automation, prototypes, QA checks, and internal tooling without first setting up an account or credential vault. See the AuditMe SEO Audit API and API documentation.

Minimal JavaScript

async function auditUrl(url) {
  const response = await fetch(
    `https://www.auditme.dev/api/v1/audit?url=${encodeURIComponent(url)}`
  );

  if (!response.ok) {
    throw new Error(`AuditMe API failed: ${response.status}`);
  }

  const data = await response.json();

  console.log("Overall score:", data.score.percentage);
  console.log("Title:", data.page.title);
  console.log("Load time:", `${data.page.load_time_ms}ms`);

  return data;
}

auditUrl("https://example.com");

cURL

curl "https://www.auditme.dev/api/v1/audit?url=https://example.com"

The documented response includes page metrics, a 0–100 score, prioritized recommendations and AI insights. The exact response contract should be treated as versioned API documentation rather than copied indefinitely into an article. AuditMe API docs

A practical CI quality gate

name: AuditMe Website Quality Gate

on:
  workflow_dispatch:
  push:
    branches: [main]

jobs:
  audit:
    runs-on: ubuntu-latest

    steps:
      - name: Run AuditMe
        run: |
          set -euo pipefail
          curl --fail --silent --show-error \
            "https://www.auditme.dev/api/v1/audit?url=https://example.com" \
            > audit.json

      - name: Fail on a broken response
        run: |
          node - <<'NODE'
          const fs = require('fs');
          const audit = JSON.parse(fs.readFileSync('audit.json', 'utf8'));
          if (audit.ok === false) process.exit(1);
          console.log(`AuditMe score: ${audit.score?.percentage ?? 'n/a'}`);
          NODE

      - name: Store audit evidence
        uses: actions/upload-artifact@v4
        with:
          name: auditme-audit
          path: audit.json

The point of a CI gate is not to impose a universal "90+ or fail" rule. A serious engineering team should choose policies for the defects that matter to its product. A broken canonical, accidental noindex, invalid deployment URL or contradictory structured data may deserve a hard failure; a readability warning probably does not.

AuditMe exposes more than one machine-facing surface

The public docs also expose focused endpoints for AI/GEO visibility and AI-readiness checks, alongside the core SEO audit. For example, the documented GEO visibility endpoint sends a defined query set to Gemini and returns a visibility score plus query-level observations. That is useful as a lightweight diagnostic, but it is not equivalent to the continuous, cross-engine prompt monitoring offered by dedicated AI-visibility platforms such as Beamtrace or WildSEO.

curl -X POST "https://www.auditme.dev/api/geo-check" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

The AI-readiness endpoint focuses on signals such as llms.txt, AI-bot rules and semantic HTML:

curl -X POST "https://www.auditme.dev/api/ai-readiness" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

This distinction is worth keeping public. An AI-readiness diagnostic is not the same product job as an AI-visibility monitoring platform.

API and MCP are different surfaces

The REST API is for applications, scripts and CI systems. MCP is for tool-aware AI clients and agentic workflows.

AuditMe's current documentation exposes an MCP endpoint at /api/mcp and documents a seo_audit tool. A minimal discovery call is:

curl -X POST "https://www.auditme.dev/api/mcp" \
  -H "Content-Type: application/json" \
  -d '{"method":"tools/list"}'

A minimal client configuration documented by AuditMe is:

{
  "mcpServers": {
    "auditme": {
      "url": "https://www.auditme.dev/api/mcp"
    }
  }
}

Client configuration formats change, so developers should verify the current Cursor/Claude Desktop format in both AuditMe's documentation and the client documentation before production rollout. AuditMe API docs

The larger idea

Whether the caller is a GitHub Action, n8n workflow, internal dashboard or an AI agent, the useful pattern is the same:

machine → audit → structured evidence → policy → action → verification

That is the web-native version of an engineering feedback loop.

11 — What "agent-ready" should mean in practice

The phrase is dangerously easy to misuse.

Allowing an AI crawler does not automatically make a website agent-ready.

An agent might need to navigate a product catalog, identify a product variant, understand pricing, find documentation, choose an action, submit a form, or call an API.

That means the site must expose enough structure for a machine to build a useful model of the property.

Before the fix: a conceptual example

Imagine an ecommerce page where:

visible product price is $29;
JSON-LD says $0;
important product details are loaded only after a large client-side bundle executes;
documentation is buried in an ambiguous navigation structure;
the success state after a workflow is not clearly expressed.

A browser agent may still manage the task. It now has to infer more and verify more.

The failure mode is not necessarily "the agent cannot use the site." The problem can be ambiguity, unnecessary exploration, or conflicting facts.

After the fix

Suppose the same page now has:

visible and structured data aligned;
critical product facts in initial HTML;
clear semantic headings and navigation;
stable URLs;
discoverable documentation;
machine-readable relationships;
explicit action and completion states.

The agent has a cleaner state space.

That does not guarantee successful automation. It reduces ambiguity.

And that distinction is exactly how agent readiness should be discussed: as a property of the interface and information architecture, not as a magical certification badge.

The practical checklist

For an agent-oriented website, examine:

Identity

Who owns this site?
What product or service does it represent?
What are the core entities?

Information

Can a machine find pricing?
Can it find documentation?
Are important claims directly extractable?
Are facts consistent across visible content and structured data?

Navigation

Are important destinations discoverable?
Are URLs predictable?
Are links descriptive?

Actions

Are controls semantic?
Does the interface clearly communicate what an action does?
Is completion detectable?

Machine access

Is useful content present in initial HTML where appropriate?
Is the site unnecessarily dependent on JavaScript?
Are APIs documented?

Governance

Which bots are allowed?
Which content is intended for which systems?
Are robots rules deliberate rather than copied blindly?

That is much more concrete than simply saying "optimize for AI agents."

12 — What AuditMe should borrow from the competition

Good competitive research should change the roadmap, not just decorate a blog post.

From Beamtrace: prompt-level explainability

The compelling pattern is not the aggregate visibility score.

It is the path from:

score → prompt → answer → competitor → source/citation

AuditMe can borrow the concept without copying the product.

A future version of the evidence chain could say:

Your AI visibility dropped for these prompts.

Here are the answers.

Here are the sources.

Here are the corresponding site-level evidence gaps.

Here are the changes that should be tested.

That would connect AI visibility measurement to website diagnosis.

From WildSEO: monitoring instead of snapshots

A report tells you the state of a site at a moment in time.

A monitoring system tells you what changed.

WildSEO's public positioning makes that recurring workflow explicit: track AI answers, competitors, citations, prompts, share of voice, and alerts over time. WildSEO

AuditMe's obvious long-term bridge is:

AUDIT → BASELINE → MONITOR → CHANGE DETECTION → RE-AUDIT

From Foudroyer: operational closure

Foudroyer makes the execution loop visible: crawl, identify problems, manage indexation, track tasks, and monitor SEO activity. Foudroyer

AuditMe should keep pushing findings toward implementation and verification.

A diagnostic that ends as a PDF eventually becomes a PDF graveyard.

From MagicSpace: tie technical work to business outcomes

MagicSpace is explicit about the business outcome it sells: signups, demos, revenue, and growth rather than traffic for traffic's sake. It also combines strategy with hands-on implementation. MagicSpace

AuditMe should borrow the discipline, while keeping the claims evidence-based.

Instead of:

+14% traffic guaranteed.

Use:

modeled scenario under stated assumptions.

From KatLinks: keep the simple path simple

A free URL audit should not become a cockpit requiring a training course.

KatLinks is a useful reminder that traditional SEO products can win users by being straightforward about the core workflow. KatLinks

AuditMe's advanced evidence model should exist behind a clean front door.

From Captflow and Trackboxx: separate diagnosis from outcome measurement

These products reinforce an important boundary.

Analytics tells you what visitors did.

An audit tells you what the site contains and where the implementation can improve.

A mature stack can connect the two without collapsing them into one score. Captflow Trackboxx

13 — The 2026 workflow: how the layers should work together

The most useful SEO/GEO architecture is not necessarily one giant application.

A practical workflow looks like this:

Step 1 — Establish technical and content truth

Run a website audit.

Inspect:

crawlability;
indexability;
metadata;
content structure;
links;
schema;
performance;
accessibility;
semantic identity;
AI/GEO signals;
agent readiness.

AuditMe can serve this baseline layer. AuditMe

Step 2 — Establish search performance truth

Connect Search Console and your analytics stack.

Measure:

impressions;
clicks;
CTR;
positions;
landing pages;
conversions;
revenue.

Never confuse these metrics with AI citations.

Step 3 — Establish AI visibility truth

Track a defined prompt set across relevant AI systems.

Record:

mention frequency;
position;
competitor presence;
cited URLs;
changes over time.

Dedicated tools such as Beamtrace and WildSEO are designed for that job. Beamtrace WildSEO

Step 4 — Connect the observations

Now ask the useful question:

When the AI visibility is weak, what site-level evidence is also weak?

That is where a website intelligence layer can become more valuable than either dashboard alone.

Step 5 — Fix and verify

A finding should produce an implementation ticket.

After deployment:

re-run the audit;
confirm the original issue disappeared;
check performance and crawl state;
watch search performance;
watch AI visibility trends.

This turns SEO/GEO from a publishing ritual into an engineering loop.

AuditMe's first-party research illustrates the need to keep layers separate

AuditMe's September 2026 first-party research snapshot reported:

1,737 AI citation events over 27 days;
a peak of 141 citation events in one day;
116,181 Google Search impressions;
9 Google Search clicks;
81.31 weighted average position.

Those datasets are useful precisely because they are different.

The AI citation observation is not a Google Search impression. The impression is not a click. The click is not a conversion.

A mature article should preserve those distinctions instead of turning them into one giant "AI growth" number.

What this means for GEO content

The best GEO content is not the content that repeats the term GEO the most.

It is content that is:

specific;
source-linked;
dated where needed;
explicit about definitions;
clear about measurement units;
honest about missing evidence;
connected to real implementation examples.

That is also what search engines and human readers can use.

14 — The reference library: official docs first, product claims second

A technical article should make it easy for a reader or agent to verify important claims.

Google Search

Google's June 2026 documentation update is particularly relevant to GEO discussions: Google clarified that llms.txt is not needed for Google Search and does not positively or negatively affect Search visibility or rankings. Google says publishers can still maintain it for other systems that may use it. Google Search updates

That is exactly the kind of nuance an evidence-first GEO article should preserve.

Performance

Remember the measurement distinction:

LAB ≠ FIELD.

Structured data

Structured data should represent accurate page information. It is not a secret channel for facts that contradict what users can actually see.

OpenAI and ChatGPT discovery

OpenAI's current publisher guidance explains how website discovery and crawler access work and notes that publishers who allow OAI-SearchBot can track ChatGPT referral traffic through analytics platforms. OpenAI Publishers and Developers FAQ

The practical point is not "allow every AI bot." It is to make an explicit governance decision for the services you actually care about.

Google-Extended

Google documents Google-Extended as a robots.txt control token for certain Gemini-related content usage contexts. It is not a synonym for Googlebot and should not be described as a generic AI ranking signal. Google common crawlers

AuditMe

The competitors

FAQ: concise answers for search, developers, and AI agents

What is AuditMe?

AuditMe is a website intelligence and SEO audit platform that analyzes a URL across technical, content, performance, structured-data, accessibility, security, AI/GEO, knowledge-graph, and agent-readiness signals, then turns the findings into a prioritized report.

Is AuditMe an AI visibility tracker?

Not primarily. AuditMe includes AI/GEO and agent-readiness analysis, while products such as Beamtrace and WildSEO focus more directly on recurring AI-answer visibility, prompts, competitors, citations, and trends.

Is GEO the same as SEO?

No. SEO is the broader discipline covering crawlability, indexing, relevance, technical quality, content, and search visibility. GEO is a broad industry term for optimization around generative or AI-mediated discovery and is used inconsistently, so the specific measurement should always be defined.

Does llms.txt improve Google rankings?

Google's current documentation says llms.txt is not needed for Google Search and does not itself provide a positive or negative effect on Search visibility or rankings. It can still be maintained for other systems that may use it. Google Search updates

Should I allow OAI-SearchBot?

If you want public content to be discoverable for ChatGPT search use cases, consult OpenAI's current publisher guidance and make the robots decision that matches your publishing goals. OpenAI Publishers and Developers FAQ

Does structured data guarantee AI citations?

No. Structured data can make information and relationships easier for systems to interpret, but it does not guarantee rankings, recommendations, or AI citations.

Is an AI visibility score the same as a Google ranking?

No. Prompt-level AI visibility, AI answer position, citations, Google impressions, clicks, sessions, and conversions are different measurements.

Can an SEO audit predict traffic?

Not by itself. An audit is primarily diagnostic. Any traffic or revenue projection should be labelled as a model with explicit assumptions.

Can AuditMe replace an SEO agency?

Not automatically. AuditMe can automate evidence collection, diagnosis, prioritization, and developer-oriented reporting. An agency can add business strategy, content production, implementation, authority building, and accountability.

Does every AuditMe audit include a full-site crawl?

No. Audit coverage depends on the scan configuration and available data sources. A single-URL audit should not be described as equivalent to a full-site crawl.

Why does an audit sometimes say "Not captured"?

Because unavailable data should be distinguished from measured data. A visible missing-data state is more trustworthy than a guessed number.

15 — The conclusion: the next generation of SEO software is not a bigger checklist

The easiest thing for an SEO product to add is another check.

The harder thing is to explain what the check means, prove how it was measured, tell the user what should happen next, and then verify whether the problem was actually fixed.

The same is true of AI search.

A visibility dashboard can tell you:

We appeared in 38% of tracked answers.

That is useful.

It becomes much more useful when the team can continue asking:

Which prompts?

Which competitors?

Which answers?

Which citations?

Which pages support the claim?

Which site-level weaknesses are visible?

What changed after the fix?

That is the larger idea behind website intelligence.

The goal is not to pretend that SEO, GEO, AI visibility, analytics, and agent behavior are one thing. They are not.

The goal is to connect them without destroying the distinctions that make their measurements meaningful.

For AuditMe, that creates a clear product direction:

Observe → Understand → Prioritize → Fix → Verify → Monitor

The first five steps already make a useful product. The sixth is the bridge to a much larger system.

That is also the test I would apply to the next AuditMe feature, the next SEO platform, and the next piece of GEO advice:

What exactly did we observe?

Where did the evidence come from?

What is known versus inferred?

Why does it matter?

What should change?

How will we know the change worked?

A website is no longer optimized for one audience.

It is simultaneously a document for search engines, a source for AI systems, an interface for humans, a dataset for analytics, and increasingly a surface that software agents may be asked to navigate.

That does not kill SEO.

It makes lazy definitions of SEO obsolete.

And it makes evidence more valuable than ever.

AuditMe reference card

Field	Current documented description
Product	AuditMe
Category	Website intelligence / SEO audit
Primary input	URL
Primary workflow	Audit → evidence → priority → fix
Public free flow	URL-based audit, no signup for the public audit path
Audit speed claim	Approximately 60 seconds for the public workflow
Engine snapshot discussed in this article	158 checks
Dimensions in the current public product positioning	16
Showcase audit snapshot date	September 20, 2026
Showcase audit score	83 / 100
Showcase checks passed	106 / 158
Showcase AI/GEO Readiness	89 / 100
Showcase lab LCP	15.31s
Showcase lab Speed Index	7.71s
Showcase lab TBT	300ms
Showcase potential score recovery model	Up to +17 points
API	https://www.auditme.dev/seo-audit-api
Documentation	https://www.auditme.dev/docs
GitHub	https://github.com/Edo911/AISeoAudit

Important: The showcase metrics above are a dated audit snapshot of AuditMe itself, not universal claims about customer sites.

Editorial note

This article is deliberately written as a field guide and capability map rather than a winner list. Product scope changes, public pages change, and feature boundaries overlap. Competitor descriptions are based on their public positioning as checked around September 21, 2026. Where a product claims a capability, that claim should be understood as provider-reported unless independently demonstrated here.

The strongest way to evaluate any of these products is to test the exact workflow you need on your own site and keep the measurement boundaries clear.

Your Website Was Seen 116,181 Times and Clicked 9 Times. Here's What Search Engines and AI Systems Are Actually Doing.

Eduard — Wed, 16 Sep 2026 11:05:56 +0000

Your Website Was Seen 116,181 Times and Clicked 9 Times

What Search Engines and AI Systems Are Actually Doing With Your Pages

September 2026 · first-party AuditMe data · Google Search Console + AI-performance observations

Most websites have a dashboard problem.

They tell you what happened without telling you which layer of the system failed.

You see traffic.

You see rankings.

You see an SEO score.

You see “AI visibility.”

And then someone asks the most dangerous question in SEO:

“So... are we visible?”

The answer is usually a useless “it depends.”

So I wanted a better answer.

I build AuditMe, a website intelligence platform that crawls and analyzes websites across technical SEO, content, performance, structured data, links, accessibility, security, AI search readiness, and agent readiness.

I pulled a fresh three-month Google Search Console export for AuditMe and compared it with a separate AI-performance export covering August 18–September 13, 2026.

The result looked almost absurd:

Signal	AuditMe data
Google Search impressions	116,181
Google Search clicks	9
Weighted average position	81.31
Impressions per click	12,909
AI citation events	1,737
AI observation period	27 days
Peak daily AI citations	141
Maximum cited pages in one day	8
U.S. impressions	49,970
Desktop impressions	92,235

That is not a typo.

AuditMe was appearing in Google Search often enough to accumulate 116,181 impressions, yet the export contains only 9 clicks.

At the same time, a separate AI-performance dataset recorded 1,737 citation events.

The wrong reaction is:

“SEO is dead.”

The other wrong reaction is:

“AI citations solved SEO.”

The interesting reaction is:

“These systems are measuring different layers of visibility. Let's map the layers.”

That is what this article does.

More importantly, it gives you a repeatable way to run the same investigation on your own website.

TL;DR — The Entire Article in 90 Seconds

A website does not have one visibility score in the real world.

It has a sequence of states:

ACCESS
  ↓
CRAWL
  ↓
DISCOVER
  ↓
INDEX
  ↓
RETRIEVE
  ↓
SURFACE
  ↓
CITE
  ↓
CLICK
  ↓
TRUST
  ↓
CONVERT
  ↓
VERIFY
  ↓
MONITOR

A page can succeed at one layer and fail at another.

AuditMe's September 2026 data demonstrates the point:

116,181 Google Search impressions
9 Google Search clicks
81.31 weighted average position
1,737 AI citation events in the separate AI-performance export

Google defines an impression as a result shown to a user in Search and a click as a user clicking a link from Google Search. Search Console's average position is an aggregate metric based on the topmost result from your property for each impression, so it should not be read as “this page ranked at exactly X for every query.” See Google's documentation on impressions, position and clicks and Performance report data.

The Visibility Gap is the practical gap between machine exposure and meaningful human response.

I use this simple diagnostic ratio:

Visibility Gap Ratio = Impressions / Clicks

For this AuditMe export:

116,181 / 9 = 12,909 impressions per click

This is not a Google ranking factor.

It is not an industry benchmark.

It is not proof that 12,909 people “saw” a page in the normal human sense.

It is simply a useful diagnostic derived from the supplied Search Console export.

The second big lesson is that AI citation events must also be kept separate from traffic and conversions. A citation event is not automatically a person, a click, a signup, or revenue.

The practical strategy is therefore not “optimize for Google” and then separately “hack ChatGPT.”

It is to build documents that are:

accessible,
crawlable,
indexable,
semantically clear,
evidence-rich,
internally connected,
fast enough,
accessible to people,
and attributable to a real author and organization.

Google's current guidance says the same foundational SEO best practices remain relevant for AI features and that there are no additional technical requirements or special AI schema required specifically to appear in Google AI Overviews or AI Mode. Read Google's AI features documentation.

The surprising conclusion is this:

The future-proof SEO strategy is not “optimize for AI.” It is “build information that machines can retrieve correctly and humans can trust.”

The 12,909:1 Problem
The Visibility Stack: One Website, Twelve States
What the AuditMe Data Actually Says
The Visibility Gap Experiment You Can Run in 30 Minutes
Why “SEO vs GEO vs AEO” Is the Wrong Fight
Designing Pages That Survive Retrieval
Build a Website Knowledge Graph, Not a Blog Graveyard
Technical SEO in the AI Era: What Actually Matters
AI Crawlers, robots.txt, noindex, llms.txt and the Myths
Performance, Accessibility, Security and Trust
The Website Visibility Operating System
The 2026 Builder's Checklist and the Rule I Would Bet On

1. The 12,909:1 Problem

Let's start with the number that made me stop.

1.1 116,181 impressions sounds impressive. It isn't enough information.

Search Console reported:

116,181 impressions.

The natural marketing sentence would be:

“AuditMe got more than 116K Google impressions.”

Technically true.

But it can create a completely wrong mental picture.

An impression is a Search Console measurement of a result being shown in Google Search; the details vary by result type. It is not the same thing as 116,181 people reading your homepage. Google documents the definition and counting rules here.

In the same export there were only:

9 clicks.

So the first question should not be:

“How do we celebrate 116K impressions?”

It should be:

“Why is exposure so much larger than response?”

That question is useful even if your website gets 10 impressions rather than 10 million.

1.2 The ratio is useful because it exposes a gap

I call this the Visibility Gap Ratio:

VGR = Search Impressions / Search Clicks

AuditMe:

VGR = 116,181 / 9
    ≈ 12,909

Interpretation:

In this export, AuditMe generated roughly 12,909 recorded Google Search impressions for every recorded click.

Again, this is an observational diagnostic, not an SEO score.

It tells us there is a large gap between being surfaced and receiving a click.

It does not tell us which cause dominates that gap.

Possible causes include:

very low average positions,
query mismatch,
low click intent,
SERP composition,
snippets that fail to earn attention,
brand unfamiliarity,
pages being surfaced for broad long-tail variants,
or simple measurement realities in Search Console.

That uncertainty is exactly why an audit has to inspect multiple layers.

1.3 The two biggest pages explain most of the exposure

The page export makes the concentration obvious:

URL	Impressions	Clicks	CTR	Avg. position
`/seo-checker`	51,933	2	0.00%	87.31
`/website-seo-checker`	30,709	4	0.01%	82.04
`/api-docs`	4,304	1	0.02%	74.75
`/seo-audit-tool`	3,982	1	0.03%	86.23
`/free-seo-tools`	1,266	1	0.08%	83.83

Those first two URLs alone account for 82,642 impressions.

And both sit, on average, far below the first page.

That is a much more precise story than:

“Our SEO needs work.”

It says:

Google is already associating specific AuditMe pages with a lot of search demand, but those pages are usually surfacing too low to generate meaningful click volume.

That is actionable.

1.4 The query table tells us what the machine already associates with AuditMe

Here are some of the largest query groups in the supplied export:

Query	Impressions	Clicks	Avg. position
`seo checker`	3,176	1	87.12
`website seo checker`	2,014	1	87.74
`seo page checker`	1,574	0	85.13
`seo check`	1,274	0	80.36
`seo checker api`	999	0	71.77
`seo checker website`	966	0	86.37
`check website seo`	945	0	76.38
`seo score`	861	0	84.27
`seo checker online`	760	0	88.58
`seo website checker`	748	0	84.60
`seo score checker`	685	1	86.31
`ai seo analysis`	42	1	66.17

The obvious beginner move would be to create more pages:

seo-checker
seo-checker-online
seo-checker-free
seo-checker-tool
best-seo-checker
seo-checker-website
seo-page-checker

I would not do that by default.

That can turn a semantic opportunity into a cannibalization and maintenance problem.

The smarter question is:

What distinct user intents are hidden inside this query neighborhood?

That is a very different content strategy.

2. The Visibility Stack: One Website, Twelve States

A website is not “ranked” or “not ranked.”

Those are coarse labels for a multi-stage system.

I use this stack:

#	Layer	Question	Typical evidence
1	Access	Can a machine reach it?	HTTP, TLS, DNS
2	Crawl	Can a crawler fetch it?	crawl logs, robots
3	Discovery	Can important URLs be found?	links, sitemap
4	Index	Can the URL be indexed/served?	noindex, canonical, inspection
5	Retrieval	Does it match a need?	queries, relevance
6	Surface	Does a system present it?	Search appearance
7	Rank	Where does it appear?	position
8	Citation	Is it used as source material?	citation observations
9	Click	Do people visit?	clicks, sessions
10	Trust	Can the claim be verified?	author, sources, method
11	Convert	Does the visit create value?	leads, signup, revenue
12	Verify/Monitor	Did the fix persist?	re-audit, trend history

The important insight is that a failure at one layer can masquerade as a failure at another.

2.1 Access is not ranking

If DNS is broken, ranking advice is irrelevant.

If a page returns errors to a crawler, copywriting is not the first problem.

Useful references:

2.2 Crawlability is not indexability

This distinction causes endless confusion.

robots.txt controls crawling access.

noindex is an indexing directive.

They solve different problems.

Google explicitly documents that robots.txt is not a mechanism for removing a page from Search. When you need a page excluded from indexing, the noindex guidance is the relevant documentation.

That difference is simple, but a surprising number of “SEO fixes” are built on mixing these concepts together.

2.3 Indexing is not retrieval

Being indexed does not mean a page will appear for every concept it mentions.

A page may be indexed yet be irrelevant for a particular query.

This is why content architecture matters.

A good page should have one primary job.

2.4 Ranking is not clicking

AuditMe's own numbers make this painfully obvious.

A page can accumulate tens of thousands of impressions while averaging around position 80.

Google's Search Console documentation recommends focusing on trends in impressions and clicks rather than treating average position as a complete standalone success metric. See Common tasks and use cases.

2.5 Citation is not traffic

An AI system can use a document as a source without the user becoming a site visitor.

That is especially important when reading AI visibility reports.

A citation metric should answer:

“Was this source used?”

It should not automatically be translated to:

“A customer came from AI.”

Keep the two ledgers separate.

3. What the AuditMe Data Actually Says

Now let's look at the dataset as an engineer would, not as a marketer would.

3.1 Search performance is heavily desktop-weighted

The device export:

Device	Clicks	Impressions	CTR	Avg. position
Desktop	7	92,235	0.01%	80.88
Mobile	2	23,383	0.01%	82.96
Tablet	0	563	0%	83.61

Desktop contributes almost 79% of impressions in the supplied export.

That does not prove the product should be “desktop-first.”

It does mean that the current Search Console distribution is strongly desktop-heavy, so analyzing only mobile performance would hide most of the observed search exposure.

3.2 The U.S. dominates the geographic exposure

The countries export shows:

Country	Clicks	Impressions	Avg. position
United States	1	49,970	84.34
Ukraine	2	1,619	76.08
Turkey	1	635	76.51

This is another reason not to reduce “SEO performance” to a single global number.

A website can have very different demand distributions by country, device, language, and query intent.

Search Console explicitly provides these dimensions for analysis; see the Performance report.

3.3 AI visibility is rising in a different measurement system

The separate AI-performance export covers 27 dates from August 18 through September 13, 2026.

Total citation events:

1,737

The daily series included:

Date	Citations	Cited pages
Aug 19	8	2
Aug 22	13	1
Aug 23	22	1
Aug 27	63	3
Aug 30	70	4
Aug 31	74	4
Sep 2	84	5
Sep 3	89	5
Sep 4	122	4
Sep 7	141	5
Sep 8	100	4
Sep 10	110	8
Sep 11	118	6
Sep 13	115	6

The pattern is interesting because the site is developing a measurable AI citation footprint while classic Search clicks remain tiny.

But again:

What this does NOT prove

It does not prove:

1,737 unique people saw AuditMe,
1,737 people clicked an AuditMe citation,
AI caused the Search Console impressions,
AI caused signups,
or AI caused revenue.

It proves that the supplied AI-performance measurement system recorded 1,737 citation events.

That is the level of certainty we should keep.

3.4 Why the discrepancy is actually useful

If all your metrics moved together, diagnosis would be easy.

But real websites are messy.

You can see:

High impressions
+ low clicks
+ growing AI citations
+ low average position

That is not one problem.

It is a map of several different problems and opportunities.

This is exactly what website intelligence should surface.

4. The Visibility Gap Experiment You Can Run in 30 Minutes

Here is the experiment I think is the most useful thing in this article.

It requires no paid SEO suite.

You can use Google Search Console, your browser, a spreadsheet, and a crawler/audit tool.

4.1 Step 1 — Export Search Console data

Open the Search Console Performance report and export:

queries,
pages,
devices,
countries,
dates.

Google's own documentation explains how to configure and interpret the Performance report:

4.2 Step 2 — Calculate the Visibility Gap Ratio

In a spreadsheet:

=IF(Clicks=0, "∞", Impressions/Clicks)

Example:

Page	Impressions	Clicks	VGR
Page A	10,000	100	100
Page B	10,000	10	1,000
Page C	10,000	1	10,000

Page C has a huge exposure-response gap.

That does not automatically mean the title is bad.

It tells you where to investigate.

4.3 Step 3 — Add position, because the ratio alone can mislead

This is critical.

Compare:

Page	Impressions	Clicks	VGR	Avg. position
A	10,000	10	1,000	3
B	10,000	10	1,000	82

Same VGR.

Completely different diagnosis.

Page A may deserve a snippet/title/intent investigation.

Page B may simply have a large amount of low-position exposure.

This is why I would never use the Visibility Gap Ratio alone.

Use a matrix.

4.4 Step 4 — Run a page audit

For each high-gap URL, inspect:

title,
description,
H1/H2 structure,
canonical,
robots directives,
content accessibility,
internal links,
schema,
images,
mobile setup,
performance,
security headers.

You can run a real page through the AuditMe Website SEO Checker.

The AuditMe checker currently exposes a 16-dimension model that includes meta tags, content quality, technical SEO, links, performance, schema, images, social media, E-E-A-T, accessibility, security headers, user experience, CRO, knowledge graph, AI search readiness and agent readiness. See the live Website SEO Checker.

4.5 Step 5 — Run the content through the “single-excerpt test”

Pick any important paragraph.

Imagine the reader only gets that paragraph.

Can they tell:

what topic it is about?
what the claim is?
who made the claim?
what evidence supports it?
what the scope is?

If not, the paragraph depends too much on context.

I call this retrieval resilience.

It is not a ranking factor.

It is a writing property.

And it is increasingly valuable whenever content is consumed in snippets, summaries, passages, answers, documentation systems, or agent interfaces.

4.6 Step 6 — Check internal links

Ask:

“What page would a reader logically need next?”

Then link to it.

For AuditMe that might be:

These are useful links because they represent actual next actions.

4.7 Step 7 — Create the scorecard

Use this exact table:

URL	Intent	Impressions	Clicks	VGR	Position	Audit finding	Fix	Expected evidence
`/example`	informational	12,400	6	2,067	52	weak answer structure	rewrite	query/click change
`/product`	commercial	4,900	2	2,450	18	snippet mismatch	rewrite metadata	CTR change
`/guide`	research	9,200	0	∞	75	weak internal graph	add links	impressions/position

Now you have an actual research instrument.

4.8 Step 8 — Change one major variable

Do not rewrite 30 pages at once.

Pick one high-value page.

Change one major class of variable:

search intent alignment,
title/heading clarity,
internal graph,
content evidence,
technical blockers,
performance bottleneck.

Then measure again.

The goal is not to prove your theory right.

The goal is to find out whether it was wrong.

That is a much better engineering mindset.

5. Why “SEO vs GEO vs AEO” Is the Wrong Fight

The internet has a naming problem.

We now have:

SEO
AEO
GEO
LLM SEO
AI SEO
AI search optimization
answer engine optimization
generative engine optimization
AI visibility optimization

Some of these labels describe slightly different workflows.

But they share the same underlying object:

information published on the web.

5.1 SEO is about discoverability and search performance

The classic SEO system asks whether content can be discovered, crawled, indexed, retrieved and served in Search.

Google's Search Essentials and SEO Starter Guide remain the right starting points.

5.2 AEO is a useful writing discipline

Answer Engine Optimization is most useful to me as an editorial principle:

Answer explicit questions clearly and early.

That means:

definitions near the top,
direct answers,
concise examples,
useful tables,
clear scope,
and source links.

It is good writing whether or not an AI system exists.

5.3 GEO should not be treated as a magic switch

Google currently says the same foundational SEO practices remain relevant for AI features and that there are no additional technical requirements or special schema needed specifically for AI Overviews or AI Mode.

See:

That does not make “AI visibility” meaningless.

It means the durable strategy is not a secret tag.

It is:

useful information
+ clear structure
+ accessible content
+ evidence
+ entity clarity
+ strong architecture
+ trustworthy attribution

5.4 One page can serve all three goals

A well-built page can simultaneously:

Goal	Page property
SEO	crawlable + relevant + indexable
AEO	direct, structured answers
GEO	retrievable, attributable evidence
UX	readable + fast
Trust	author + sources + methodology
Conversion	clear next action

That is why I prefer Website Visibility Intelligence as the larger category.

It avoids pretending that Google, AI search and humans live in separate universes.

6. Designing Pages That Survive Retrieval

This is the most technical part of the writing strategy.

And it is surprisingly simple.

6.1 The “standalone paragraph” rule

Imagine a retrieval system extracts one paragraph.

The paragraph should ideally survive without 15 paragraphs of setup.

Bad:

“This has a significant impact.”

What is “this”?

Better:

“A missing canonical URL can create URL-selection ambiguity when multiple URLs represent substantially similar content; inspect canonicalization before treating a ranking problem as a content problem.”

The second sentence carries its subject with it.

6.2 The Answer → Evidence → Limitation pattern

For important claims, use this structure:

ANSWER
↓
EVIDENCE
↓
LIMITATION

Example:

Google's AI features do not require a special AI schema. Google says the same foundational SEO best practices remain relevant for AI features, and there are no additional technical requirements to appear in AI Overviews or AI Mode. This does not mean every well-optimized page will appear in AI results; visibility still depends on Google's systems and the page's relevance and quality. Source

Notice what this does:

It answers the question.

It provides the source.

It states the boundary of the claim.

That is excellent material for humans and much safer material for AI systems to summarize.

6.3 Use tables as compression, not decoration

A table should answer several questions at once.

Weak table

Metric	Meaning
SEO	SEO
GEO	GEO

Useful table

Signal	Definition	Example source	Common mistake
Impression	Search result shown	Search Console	treating it as a visit
Click	Search link clicked	Search Console	treating it as a conversion
Citation	source attributed in an AI measurement system	AI-performance dataset	treating it as unique traffic
Position	aggregate Search position	Search Console	reading it as an exact universal rank

That is the kind of table people screenshot and AI systems can parse cleanly.

6.4 The definition block is underrated

For every important concept, answer:

Term:
Definition:
What it is not:
How to measure it:
Why it matters:

Example:

Visibility Gap Ratio

Definition: Search impressions divided by Search clicks for a selected property/page/time range.

Not: A Google ranking factor or an industry benchmark.

Measure: Search Console export.

Use: Identify pages where exposure and human response diverge.

That is a highly reusable information object.

7. Build a Website Knowledge Graph, Not a Blog Graveyard

Most content teams ask:

“What should we publish next?”

I prefer:

“What knowledge node is missing from the graph?”

7.1 The AuditMe example

A coherent graph might look like:

                      AUDITME
                         │
           ┌─────────────┼─────────────┐
           │             │             │
           ▼             ▼             ▼
       TOOLS          RESEARCH      DOCUMENTATION
           │             │             │
           ▼             ▼             ▼
     SEO Checker    Benchmarks      API Docs
           │             │
           ├─────────────┤
           ▼             ▼
      SEO Concepts   AI Visibility
           │             │
           └──────┬──────┘
                  ▼
                Guides
                  │
                  ▼
              Next Action

The exact topology should follow the real site.

The principle is universal.

A page should not be an island.

7.2 Use links to answer the next question

There are three good reasons to add an internal link:

The reader needs more context.
The reader needs evidence.
The reader needs an action.

Examples:

Definition → SEO Checker
Score explanation → SEO Score Checker
Full analysis → Website SEO Checker
Crawl problem → Crawl Audit
Implementation → API Docs
Broader discovery → AuditMe Blog

Use descriptive anchor text rather than “click here.”

7.3 Don't create pages because a keyword exists

This is one of the most expensive mistakes small websites can make.

Suppose Google shows:

seo checker
website seo checker
seo website checker
seo page checker
seo checker online
seo check website

Those are not necessarily six page intents.

They may be one dominant intent with minor lexical variations.

Before creating a page, ask:

Does this query require a genuinely different answer, tool, dataset, comparison or workflow?

If not, strengthen the existing node.

7.4 Create original nodes

This is where a small site can beat a giant site.

Publish:

Asset	Why it is defensible
First-party benchmark	others can reference the data
Reproducible experiment	readers can test it
Failure analysis	concrete and specific
Engineering teardown	shows implementation detail
Public methodology	creates transparency
Dataset	creates a durable research object
Tool	turns theory into action

AuditMe can become more than an SEO blog if the content itself generates new information.

8. Technical SEO in the AI Era: What Actually Matters

The good news is that the fundamentals are still boring.

Boring is good.

Boring survives hype cycles.

8.1 Robots.txt is access control for crawlers, not security

Google's robots guidance is explicit: robots.txt controls crawler access but is not a security mechanism.

If information must be private, use authentication and authorization.

References:

8.2 Sitemaps help discovery, but don't replace architecture

Google documents sitemaps as a way to tell search engines about URLs you consider important.

Start with:

But don't build a site that needs a sitemap as its only navigation structure.

Internal links should still make the important graph understandable.

8.3 Canonicals should describe reality

Canonicalization is not a “ranking boost.”

It is a way to help search systems understand preferred URL representation when duplicate or similar URLs exist.

Useful Google documentation:

Do not canonicalize every variant into the homepage just because it is convenient.

A canonical should reflect the actual relationship between URLs.

8.4 Structured data is useful when it describes visible reality

Google's structured-data guidance says markup should represent the visible page content and follow the relevant feature guidelines.

Useful references:

The strategic rule is:

Don't make the JSON-LD smarter than the page. Make the page clearer and let the markup describe it.

8.5 Don't confuse “eligible” with “guaranteed”

Structured data can make content eligible for some search features.

It does not guarantee that a search feature will appear.

That distinction should appear in technical writing because it prevents a large amount of bad SEO advice.

9. AI Crawlers, robots.txt, noindex, llms.txt and the Myths

AI systems add more names to the crawler conversation.

That makes it more important to be precise.

9.1 OpenAI's crawler distinction matters

OpenAI's current publisher/developer documentation says public websites can appear in ChatGPT search and specifically discusses OAI-SearchBot for content discovery, surfacing and citations. It also distinguishes crawler access from other uses of content.

See the current OpenAI publisher FAQ:

OpenAI — Publishers and Developers FAQ

This is an excellent example of why “AI bot” should not be treated as one generic entity.

Policies can be different by crawler and product.

9.2 The right crawler strategy is policy, not paranoia

Build an explicit matrix:

Goal	Mechanism
Allow normal Search crawling	robots/server policy
Prevent indexing	`noindex`
Keep private content private	authentication/authorization
Control snippet behavior	robots meta / X-Robots-Tag where supported
Help discovery	internal links + sitemap
Detect abuse	logs + WAF + rate limits
Support AI discovery	intentional crawler access + useful content

Don't treat robots.txt like a firewall.

9.3 `llms.txt` is interesting, but don't turn it into folklore

The llms.txt proposal is an attempt to give language-model tooling a compact, structured view of a website and its important resources.

See:

This is worth experimenting with.

But there is a crucial distinction:

A useful interoperability proposal is not the same thing as an official ranking factor.

Google's AI feature guidance does not say that sites need an llms.txt file to appear in AI Overviews or AI Mode.

So my recommendation is simple:

Build one if it improves machine access to your information.

Don't build one because somebody promised guaranteed rankings.

9.4 The “AI schema” myth

There is no universal schema field that says:

CITE THIS WEBSITE FIRST

There is no magic JSON-LD object that guarantees a model will quote you.

There is no honest SEO consultant who can promise that a single markup change will force an external answer system to cite your site.

The controllable part is the source itself:

clear entity
+ clear claim
+ evidence
+ accessible content
+ stable URL
+ useful context

That is the part worth investing in.

10. Performance, Accessibility, Security and Trust

If you only think about SEO, you will miss half the quality problem.

10.1 Core Web Vitals

The current Core Web Vitals set is:

Metric	Measures	Good target
LCP	loading	≤ 2.5s
INP	responsiveness	≤ 200ms
CLS	visual stability	≤ 0.1

See:

Field data and lab diagnostics answer different questions.

Lighthouse can tell you what is happening in a controlled test.

Real-user data tells you how users actually experience the page.

Don't substitute one for the other.

10.2 Accessibility is not an “SEO hack”

WCAG 2.2 is the current W3C WCAG Recommendation line.

See:

The useful connection is architectural:

Semantic, keyboard-accessible, clearly structured content tends to be easier for humans to use and easier for machines to interpret.

That does not mean every accessibility criterion is a direct Google ranking factor.

It means accessibility is part of a quality website system.

10.3 Security is part of trust infrastructure

The current OWASP Top 10 release is the 2025 edition.

See:

The 2025 list includes risks such as Broken Access Control, Security Misconfiguration, Software Supply Chain Failures, Cryptographic Failures, Injection, Insecure Design, Authentication Failures, Software or Data Integrity Failures, Security Logging and Alerting Failures, and Mishandling of Exceptional Conditions.

A site that is fast but compromised is not a high-quality website.

A site that ranks but serves incorrect information is not trustworthy.

Technical quality and content trust eventually meet in the same place: the user.

10.4 Trust is easier to prove than to claim

I would rather see:

Author
Date
Methodology
Data source
Limitations
Update history
Contact

than:

“We are a world-class trusted authority.”

This is especially important for research-heavy content.

Authority is more durable when the reader can verify it.

11. The Website Visibility Operating System

Now put everything together.

A modern audit should not end with:

“Your score is 73.”

A score is a summary.

A workflow is a system.

11.1 Observe → Understand → Prioritize → Fix → Verify → Monitor

This is the operating loop I use in AuditMe:

OBSERVE
  ↓
UNDERSTAND
  ↓
PRIORITIZE
  ↓
FIX
  ↓
VERIFY
  ↓
MONITOR
  ↺

Each stage has a different job.

Stage	Question
Observe	What is happening?
Understand	Why might it be happening?
Prioritize	Which issue deserves attention first?
Fix	What exactly changes?
Verify	Did the system respond?
Monitor	Did the improvement persist?

This is more useful than a giant list of “SEO issues.”

11.2 Prioritize by impact, not warning count

A practical model is:

Priority = Severity × Impact × Confidence ÷ Effort

Example:

Issue	Severity	Impact	Confidence	Effort	Priority logic
Important page blocked by noindex	5	5	5	1	fix immediately
Broken canonical	5	5	4	2	very high
Weak internal linking	3	4	4	2	high
Generic article intro	2	3	4	1	moderate
Decorative animation	1	1	3	3	low

This is not a universal scoring standard.

It is a decision aid.

The important idea is to avoid treating 100 warnings as 100 equal problems.

11.3 Separate diagnostics from business outcomes

Keep these ledgers separate:

Technical ledger

errors
crawlability
indexability
performance
schema
accessibility
security

Visibility ledger

queries
impressions
position
citations
clicks

Business ledger

sessions
signups
leads
activation
revenue

Only then connect them with experiments.

This prevents “SEO vanity math.”

11.4 The evidence chain

For every major recommendation, try to maintain:

OBSERVATION
   ↓
HYPOTHESIS
   ↓
CHANGE
   ↓
MEASUREMENT
   ↓
RESULT
   ↓
LIMITATION

Example:

Observation:
High impressions, low clicks.

Hypothesis:
The page is surfacing for broad intent but not offering a compelling result match.

Change:
Rewrite title, intro, headings and internal anchor path.

Measurement:
Search Console clicks + impressions + position.

Result:
Compare pre/post windows.

Limitation:
Correlation does not prove the title rewrite caused the change.

This is how SEO becomes engineering instead of astrology.

12. The 2026 Builder's Checklist and the Rule I Would Bet On

Here is the complete practical checklist.

12.1 Crawl and index

[ ] Important pages return correct HTTP responses
[ ] HTTPS is valid
[ ] Important resources are crawlable
[ ] robots.txt rules are intentional
[ ] noindex is deliberate
[ ] canonical URLs reflect real preferred URLs
[ ] XML sitemap is accurate
[ ] important pages have internal links
[ ] orphan pages are identified

References:

12.2 Search visibility

[ ] Search Console is configured
[ ] query data is reviewed
[ ] page data is reviewed
[ ] high-impression / low-click pages are identified
[ ] position is interpreted with context
[ ] desktop and mobile are separated when useful
[ ] country differences are understood
[ ] branded and non-branded demand are separated

Useful Google references:

12.3 Content

[ ] Every important page has a primary intent
[ ] Direct answer appears early
[ ] Headings describe actual content
[ ] Important claims are specific
[ ] Sources are linked
[ ] First-party data is identified as such
[ ] Methodology is explained
[ ] Limitations are explicit
[ ] Author and update date are visible
[ ] Important paragraphs survive the single-excerpt test

12.4 AI / GEO

[ ] Important information is accessible as text
[ ] entities are named consistently
[ ] source claims are attributable
[ ] internal links connect related concepts
[ ] crawler policy is intentional
[ ] OAI-SearchBot policy is understood where relevant
[ ] AI citation metrics are kept separate from traffic
[ ] llms.txt is treated as optional interoperability
[ ] no “magic AI schema” is assumed
[ ] AI visibility is measured rather than guessed

References:

12.5 Performance

[ ] LCP monitored
[ ] INP monitored
[ ] CLS monitored
[ ] real-user data considered
[ ] lab tests used for diagnosis
[ ] JavaScript cost monitored
[ ] third-party scripts reviewed

References:

12.6 Accessibility and security

[ ] semantic structure is correct
[ ] keyboard navigation works
[ ] form labels are present
[ ] contrast is checked
[ ] images have appropriate text alternatives
[ ] authentication and authorization are correct
[ ] security headers are reviewed
[ ] software dependencies are monitored
[ ] logging and alerting exist for important failures

References:

The Rule I Would Bet On

Here is the idea I would keep even if every AI search interface changed tomorrow:

Don't optimize your website for an algorithm. Optimize the information so that an independent machine can find it, understand it, verify it, and tell a human where it came from.

That principle survives:

Google changes,
new answer engines,
new AI models,
new browser agents,
new crawlers,
and new ranking interfaces.

Because it is not actually about one algorithm.

It is about information quality.

The Bigger Idea: Build a Website That Can Explain Itself

A high-quality website should be able to answer, in machine-readable and human-readable form:

What is this?
Who made it?
What does it claim?
Why should I trust it?
Where did the data come from?
What is the limitation?
What should I do next?

That is the website intelligence problem.

And once you think about the web this way, several old debates become less interesting.

“SEO versus GEO?”

Too narrow.

“Should I add one more keyword?”

Too narrow.

“Does this one schema type unlock AI?”

Usually the wrong question.

The better question is:

Can the entire information system of this website be inspected, understood, connected and verified?

That is a much harder problem.

It is also a much more valuable product category.

What I Would Build Next From This Dataset

The obvious move is to publish another “SEO guide.”

I would not.

I would turn the observation into a public research program.

Research series

Project	Core question
Visibility Gap Index	How often does machine exposure diverge from human response?
Website Visibility Benchmark	What does visibility look like across different site types?
AI Citation Reliability Study	How stable are citation observations across repeated questions?
Search-to-Citation Map	Which content structures appear across Search and AI visibility?
Technical Failure Census	Which recurring site defects correlate with visibility problems?
Agent Readiness Benchmark	Can agents navigate and act on real websites reliably?

The important part is methodology.

Every report should publish:

dataset definition,
collection window,
population,
formulas,
limitations,
code or reproducibility where practical,
and update history.

That is how a product becomes a source of information instead of merely a source of marketing.

A Note for Non-Technical Readers

You do not need to know what a canonical URL is to understand the core problem.

Imagine you own a shop.

Google's system walks past your storefront 116,181 times.

It recognizes the shop exists.

It associates the shop with relevant categories.

Some other machine systems even mention your shop when answering questions.

But only 9 people actually walk through the door from those Google appearances.

Would you say:

“My shop is invisible”?

Not exactly.

Would you say:

“My shop is thriving”?

Also no.

You would ask:

“Why are people seeing us but not entering?”

That is the Visibility Gap.

The technical web version simply has more layers:

Can the building be reached?
Can the sign be read?
Can the address be found?
Does the shop have what the visitor wants?
Does the storefront look relevant?
Does the visitor trust it?
Can they find the door?
Do they actually enter?
Do they buy?

SEO is not magic.

It is infrastructure plus information plus user behavior.

The AuditMe Workflow in One Screen

If you remember only one diagram from this article, make it this one:

             ┌─────────────────────┐
             │      OBSERVE        │
             │ Search • Crawl • AI │
             └──────────┬──────────┘
                        ↓
             ┌─────────────────────┐
             │     UNDERSTAND      │
             │   What failed?      │
             └──────────┬──────────┘
                        ↓
             ┌─────────────────────┐
             │     PRIORITIZE      │
             │ Impact / effort     │
             └──────────┬──────────┘
                        ↓
             ┌─────────────────────┐
             │        FIX          │
             │ Code / content / UX │
             └──────────┬──────────┘
                        ↓
             ┌─────────────────────┐
             │      VERIFY         │
             │ Re-crawl / measure  │
             └──────────┬──────────┘
                        ↓
             ┌─────────────────────┐
             │      MONITOR        │
             │ Detect regressions  │
             └──────────┬──────────┘
                        │
                        └────────────↺

AuditMe's own product is built around this same progression: Observe → Understand → Prioritize → Fix → Verify → Monitor. The live platform describes the workflow and its 16 audit dimensions.

For a quick starting point:

Check a page: Website SEO Checker

Get a baseline: SEO Score Checker

Explore tools: Free SEO Tools

Read more research: AuditMe Blog

Sources Worth Bookmarking

These are the references I would keep close while building a modern website. I intentionally prefer first-party documentation, standards, and primary project sources.

Google Search

Search Console

Web platform

Accessibility and security

AI and interoperability

AuditMe resources

Methodology and Limitations

This article uses two first-party AuditMe data exports supplied in September 2026.

Search dataset

A Google Search Console Web Search export covering the last three months, with:

daily chart data,
page data,
query data,
country data,
device data,
Search type = Web.

The chart export contains 116,181 impressions and 9 clicks. The page and query tables are dimensioned views and should not be naively summed against the property-level chart; Google documents these aggregation differences in Performance report data methodology.

AI dataset

A separate AI-performance overview export covering August 18–September 13, 2026 contains daily Citations and Cited Pages values.

The Citations column sums to 1,737 across the supplied period.

The Cited Pages values sum to 94 daily observations; that is not a claim that AuditMe had 94 unique cited URLs across the whole period.

What the data does not establish

The datasets do not establish causal relationships between:

SEO changes and Google performance,
AI citations and website traffic,
AI citations and conversions,
or individual content changes and ranking changes.

The article deliberately avoids making those claims.

The Visibility Gap Ratio is an original descriptive metric used here as a diagnostic convenience. It is not a Google metric, ranking signal, industry standard, or prediction model.

Final Thought

The weirdest thing about modern search is not that machines are becoming more intelligent.

It is that we are still using dashboards designed for an older version of the web.

We ask:

“What is our ranking?”

“What is our traffic?”

“What is our SEO score?”

Those are useful questions.

But they are incomplete.

A better question is:

Where does information about my website stop flowing?

Does it fail at access?

Crawling?

Discovery?

Indexing?

Relevance?

Ranking?

Citation?

Click?

Trust?

Conversion?

Verification?

Once you can answer that, SEO becomes less mystical.

You can see the system.

You can test the system.

You can improve the system.

And you can tell the difference between a metric that looks impressive and a change that actually matters.

That is the real job of a modern website intelligence platform.

And that is the experiment I am running with AuditMe.

The web is becoming machine-readable. The real competitive advantage is becoming machine-understandable without becoming human-unreadable.

How AI Systems Read the Web in 2026: Discovery, Retrieval, Citations and Agents

Eduard — Sun, 13 Sep 2026 09:35:58 +0000

From discovery and retrieval to citations, recommendations and agent actions

A practical, evidence-first framework for making websites easier for search systems, AI answer engines, retrieval pipelines and browser agents to discover, understand, verify and use.

TL;DR: There is no universal "AI ranking factor," no single AI crawler, and no markup trick that guarantees a citation. Different machine systems perform different jobs. Search crawlers discover and refresh documents. AI search systems retrieve sources for questions. Retrieval-augmented systems select passages from a corpus. Browser agents can inspect pages and attempt actions. The durable strategy is therefore not "optimize for ChatGPT." It is to make important information reachable, explicit, unambiguous, evidence-backed, retrievable, citable and actionable.

Important research note: This article does not claim that AuditMe has already completed a cross-model benchmark. The AuditMe AI Web Intelligence Benchmark v1 described below is a proposed open methodology and dataset specification. No benchmark results, percentages or causal claims are presented here as observed AuditMe results unless explicitly labeled as such.

Last reviewed: September 13, 2026.

TL;DR at a glance

Question	Best current answer	Confidence / evidence
Does Google require special GEO markup for AI Overviews or AI Mode?	No. Google says the normal technical and quality requirements for Search still apply and there are no additional technical requirements for these AI features.	Official Google documentation
Is `llms.txt` a universal AI-search standard?	No. It is a community proposal. It can be useful as a documentation aid, but it is not a guaranteed citation or ranking mechanism.	Proposal + provider docs
Is there one "AI crawler"?	No. Providers expose multiple crawlers, fetchers and agent mechanisms with different purposes.	Official provider documentation
Does being crawled mean being cited?	No. Discovery, retrieval and citation are separate outcomes.	Architecture distinction
Does structured data guarantee AI visibility?	No. Structured data helps machines and supported Search features interpret content; it is not a universal citation guarantee.	Google + Schema.org
Does blocking `robots.txt` always remove a page from every AI product?	No. Effects are provider- and mechanism-specific.	Provider-specific documentation
Can a model without live web access discover a new page from the page itself?	No. It needs a later model update or an external retrieval path.	Basic system architecture
Does semantic HTML matter to agents?	Yes as an engineering foundation. Native controls, names and states make interfaces more explicit. OpenAI specifically recommends accessibility and ARIA best practices for its agent experience.	Web standards + provider documentation
Does canonicalization matter to AI?	Yes for machine clarity in a broad engineering sense, but do not claim that a canonical tag directly controls every AI model. Canonicalization is an established web/search mechanism for consolidating duplicate URLs.	Google documentation; AI implication is an inference
What should teams measure?	Visibility, retrieval/citation quality and task completion separately.	Proposed measurement framework

The one-sentence model

A page is useful to a machine when it can be found, fetched, parsed, retrieved, understood, verified, cited and — when the workflow requires it — acted upon.

The web no longer has one machine reader
The AI Web Evidence Chain
Discovery and access: can the machine reach the page?
Retrieval and entity clarity: can the machine select the right information?
Evidence and citations: can the page support the answer?
Write for humans and machines without writing like a robot
The technical stack: HTML, structured data, robots.txt, sitemaps and llms.txt
The agentic web: can an AI actually use your website?
Measurement: from AI visibility to Agent Task Completion Rate
The AuditMe playbook: benchmark, fix the bottleneck, publish the evidence
References, documentation and further reading

1. The web no longer has one machine reader

For a long time, website optimization could be explained with a simple model:

Publisher → Search Engine → Search Result → Human

That model is still important. It is just no longer sufficient.

A modern public website can be consumed through several distinct machine paths:

a conventional search crawler that discovers and refreshes documents;
an AI-search system that retrieves web sources for a user query;
a user-triggered fetcher that retrieves a URL because a person asked a product to use it;
a retrieval-augmented generation pipeline that selects passages from a corpus;
a language model answering from information already in its model state;
a browser agent that can inspect, navigate and attempt tasks on the page.

These systems may share infrastructure, but they do not have identical goals.

A page can be perfectly crawlable for Google Search and still fail to become a useful source in a particular AI answer. A page can be selected as evidence and still be awkward or impossible for an agent to operate. A model can know a company name while using stale information. A product page can be visible in search results while a checkout interaction is inaccessible to automation.

The first conceptual error in modern GEO is therefore treating AI as a single destination.

It is better to think in terms of machine access paths.

A practical map of machine consumption

System class	Primary purpose	Typical website concern	What success looks like
Search crawler	Discover, refresh and index documents	Crawlability, internal links, HTTP, canonicalization, sitemaps	The page can be fetched and considered for Search
AI search / answer engine	Retrieve sources and synthesize an answer	Relevance, source quality, clarity, freshness, retrievability	The page is selected as useful evidence and may be cited
Model without live retrieval	Generate from existing model state	Public availability before training/update, external retrieval where supported	The model has accurate knowledge — but the website cannot force this directly
RAG pipeline	Retrieve passages from a corpus and generate	Chunkability, metadata, stable URLs/documents, semantic clarity	Relevant passages are retrievable and faithfully used
Browser agent	Navigate and perform tasks	Semantic controls, accessible names/states, predictable flows, errors/recovery	The agent can complete the intended task safely
Tool / WebMCP-style integration	Expose explicit site capabilities to an agent	Tool definitions, deterministic inputs/outputs, safety and permissions	The agent uses a site-provided tool instead of guessing the UI

The last row is especially important because the agentic web is evolving beyond pure screen interaction. OpenAI currently documents WebMCP-based site tools in its desktop app as a proposed web standard for letting sites expose capabilities directly to AI agents. Chrome also describes WebMCP as part of its agent-ready direction. See OpenAI: Using site tools in the ChatGPT desktop app and Chrome for Developers: A developer toolkit to make your website agent-ready.

The provider docs make the distinction explicit

This is not a theoretical taxonomy invented by SEO consultants.

OpenAI: different crawling and access purposes

OpenAI currently documents OAI-SearchBot for ChatGPT search discovery and separately documents GPTBot as a control related to potential model training use. OpenAI also documents user-directed browser access and accessibility considerations for agent compatibility. For publishers, the practical lesson is simple: do not assume that "the OpenAI bot" is one thing with one purpose.

See the current OpenAI Publishers and Developers FAQ.

Anthropic: multiple user agents, different roles

Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User. Their stated purposes differ: model-development collection, search-oriented crawling and user-directed retrieval are not interchangeable jobs.

See Anthropic: Does Anthropic crawl data from the web?.

Perplexity: search crawling vs user-directed retrieval

Perplexity documents PerplexityBot for surfacing and linking websites in search results and separately documents user-triggered access mechanisms.

See Perplexity Crawlers.

Google: Search, Gemini-related controls and user-triggered fetchers

Google documents Googlebot for Search, Google-Extended as a separate control for certain Gemini-related uses, and user-triggered fetchers/agents for fetches initiated by user actions in Google products. Google explicitly says Google-Extended is not a Search ranking signal.

See Google's crawling infrastructure and Google user-triggered fetchers.

A better evidence hierarchy for GEO claims

When writing about AI search, use a stricter hierarchy than generic marketing content.

Evidence level	Example	How to state it
Officially documented behavior	Google says there are no extra technical requirements for AI Overviews / AI Mode	State it directly and link the source
Web standard / normative specification	RFC 9309 defines the Robots Exclusion Protocol	State the mechanism and scope precisely
Controlled observation	A reproducible test produces a repeated citation pattern	Publish sample, dates, prompts and method
Engineering inference	Explicit UI states are easier for automation to interpret	Label it as an engineering recommendation
Hypothesis	A certain paragraph structure might improve passage retrieval	Say it is a hypothesis until tested
Folklore	"Every LLM prefers exactly 1,500-word articles"	Do not publish as fact

This hierarchy is not pedantry. It is the foundation of credible AI-search content.

2. The AI Web Evidence Chain

The central framework in this article is the AI Web Evidence Chain:

flowchart LR
    A[DISCOVER] --> B[ACCESS]
    B --> C[PARSE]
    C --> D[RETRIEVE]
    D --> E[UNDERSTAND]
    E --> F[VERIFY]
    F --> G[SYNTHESIZE]
    G --> H[CITE]
    H --> I[RECOMMEND]
    I --> J[ACT]

    B -. failure .-> X1[Blocked / denied]
    C -. failure .-> X2[JS-only / malformed]
    F -. failure .-> X3[Weak or conflicting evidence]
    J -. failure .-> X4[Unclear or inaccessible UI]

Different products may merge, reorder or skip stages. The framework is not a claim about hidden proprietary internals. It is a way to identify where a website becomes less useful to a machine.

The ten states

Stage	Question	Typical failure	Useful engineering response
1. Discover	Can the system find the URL?	Orphan page, weak internal linking, blocked crawl path	Internal links, sitemap, clean URL architecture
2. Access	Can it fetch the resource?	403, 429, CDN/WAF block, auth wall	Review access policy, status codes and bot handling
3. Parse	Can useful content be extracted?	JS-only content, malformed HTML, noisy UI	Server-render important content, semantic HTML
4. Retrieve	Can the relevant page/passage be selected?	Ambiguous wording, fragmented content, poor information scent	Clear headings, answer-first sections, descriptive links
5. Understand	Can the system identify meaning and entities?	Conflicting names, duplicate pages, unclear relationships	Consistent entities, canonical URLs, structured data
6. Verify	Is there enough evidence to trust the claim?	Unsupported numbers, weak sourcing, outdated facts	First-party evidence, independent sources where appropriate
7. Synthesize	Can the evidence be used without losing context?	Overly broad claims, missing qualifiers	Precise definitions, scope, dates and limitations
8. Cite	Is the page useful as a source?	Generic marketing copy, no concrete evidence	Make claims specific, attributable and inspectable
9. Recommend	Is the entity suitable for the user's decision?	Weak fit despite visibility	Explain use cases, trade-offs, constraints
10. Act	Can an agent complete the required task?	Unlabeled controls, hidden state, brittle flows	Native controls, accessible names/states, deterministic workflows

Found is not Retrieved. Retrieved is not Cited. Cited is not Actionable.

This distinction is the most important practical idea in the entire framework.

Imagine a product page:

FOUND       ✓
FETCHED     ✓
PARSED      ✓
RETRIEVED   ✓
UNDERSTOOD  ✓
VERIFIED    ?
CITED       ✗
RECOMMENDED ✗
ACTIONABLE  ✗

Nothing contradictory happened.

The system may have found and read the page, but another source may have been better evidence for the user's specific question. The page may also have been perfectly informative while having a poor interface for automation.

That is why the question:

"Is my site visible in AI?"

is incomplete.

The stronger diagnostic question is:

"At which stage of the evidence chain does my site stop being useful?"

That question leads directly to engineering work.

3. Discovery and access: can the machine reach the page?

The glamorous part of GEO usually starts with content. The unglamorous failure is often earlier: the system never gets useful access to the page.

Google's current guidance for AI Overviews and AI Mode is intentionally conservative: existing SEO fundamentals continue to apply, and Google says there are no additional technical requirements specifically for eligibility in those AI features. A page needs to be indexed and eligible to appear in ordinary Search with a snippet. See Google Search Central: AI features and your website.

That does not mean technical SEO is boring. It means the fundamentals are doing more work than the GEO industry often admits.

The access stack

For an important page, check these layers in order:

URL
 ↓
DNS / TLS
 ↓
HTTP response
 ↓
Robots policy
 ↓
Authentication / WAF / CDN policy
 ↓
HTML / renderability
 ↓
Indexability
 ↓
Retrievability

A failure near the top can make everything below it irrelevant.

HTTP status is information

A crawler or agent needs predictable semantics.

200 should mean the requested resource is available.
301 / 308 can express permanent URL consolidation.
302 / 307 are temporary redirects.
401 and 403 communicate access restrictions.
404 and 410 communicate missing resources.
429 communicates rate limiting.
5xx communicates server-side failure.

See RFC 9110: HTTP Semantics and MDN: HTTP status codes.

robots.txt is a crawling control, not a universal visibility switch

The Robots Exclusion Protocol is documented in RFC 9309. Google explains that robots.txt controls crawling, but a blocked URL can still be known to the search engine through other signals. If you need a page excluded from Google's index, noindex is a different mechanism — and the crawler must be able to fetch the page to see that directive.

See:

Practical rule

Do not write:

"Add robots.txt and AI will understand your site."

Write:

"Use robots.txt to express crawling preferences for compatible crawlers, then verify the behavior of each provider you actually care about."

That statement is much harder to break.

The CDN / WAF trap

One of the worst failure modes is allowing a crawler in theory while blocking it in practice.

Common causes include:

a WAF rule triggered by user-agent strings;
bot protection returning challenges instead of HTML;
rate limiting at the edge;
country/IP restrictions;
authentication middleware applied to public pages;
inconsistent behavior between HTML and API routes;
overly aggressive security rules that treat all non-browser traffic as hostile.

The safe engineering approach is not "whitelist every bot." It is to understand the provider's published verification guidance, use least-privilege exceptions where appropriate, and monitor logs.

Google documents crawler verification methods and publishes IP ranges. OpenAI, Anthropic and Perplexity also document their web crawlers and access controls. Start from their primary documentation rather than from a third-party list of supposed AI bot names.

Search access and agent access are not the same problem

Google's current generative-AI Search documentation says normal Search technical requirements remain the basis for AI Overviews and AI Mode. Meanwhile, Chrome's 2026 agent-ready toolkit focuses on the separate problem of agents using the web after they have found it.

That gives us two very different jobs:

SEARCH DISCOVERY
Can the system find and retrieve the page?

AGENT INTERACTION
Can the system understand and operate the page?

A good website needs both only when its business workflow needs both.

4. Retrieval and entity clarity: can the machine select the right information?

Once a system can access a page, the next problem is not "Is the text good?"

It is:

Can the system reliably select the right piece of information for the query?

This is where GEO overlaps with information retrieval, information architecture and entity clarity.

Retrieval rewards explicitness, not mystery

A machine can work with prose. It can also work with highly creative prose. But if an important fact is hidden inside a page's narrative, the page gives the retrieval system more work to do.

Compare:

Weak:

Our platform gives ambitious teams a powerful experience for discovering new ways to improve websites.

Strong:

AuditMe is a website SEO auditing platform. It checks technical SEO, content, performance and related website signals and produces structured audit results.

The second sentence is not prettier. It is more useful as evidence.

Answer the exact question near the top

For important informational pages, a robust pattern is:

H1: Exact topic

1–3 sentence definition

Key facts / answer

Scope / caveats

Detailed explanation

Evidence / sources

Examples

FAQ

This works for humans because it reduces search cost. It also creates clean semantic units for retrieval systems and accessibility tools.

There is no evidence that every AI product requires this exact structure. The point is engineering practicality: clear information is easier to inspect, quote, summarize and verify than vague information.

Entity clarity is bigger than keywords

Traditional SEO often asks:

Which keyword should this page rank for?

An AI-facing content model should also ask:

What entity is this page about, what does the entity do, and how does it differ from nearby entities?

That includes:

the canonical name;
aliases and product names;
category;
product vs company vs article vs tool distinctions;
relationships to parent organizations;
official URL;
author or organization where relevant;
dates and version information;
explicit capabilities and limitations.

This is not a magic "LLM entity score." It is a way to remove semantic ambiguity from the web representation of an entity.

Canonicalization: a real SEO mechanism with an important AI implication

Google describes canonicalization as the process of selecting a representative URL among duplicate or near-duplicate pages. Google also emphasizes that the canonical URL is a hint, not an absolute rule.

See Google: How to specify a canonical URL.

Why does that matter for AI-oriented content strategy?

Because machines perform better when the same concept is represented consistently.

Consider a hypothetical site with three pages:

/site/seo-checker
/site/seo-audit-tool
/site/ai-seo-audit

Suppose all three pages mostly target the same underlying product intent.

A cleaner information architecture might be:

/site/website-seo-checker   ← canonical destination

301 from:
/site/seo-checker
/site/seo-audit-tool
/site/ai-seo-audit

Important: this example is illustrative, not a reported AuditMe experiment. No before/after ranking, traffic or citation result is being claimed here.

The engineering lesson is still sound: when multiple URLs represent substantially the same resource or intent, deliberate consolidation can reduce duplication and make the site's own representation more coherent.

The AI-specific conclusion must be stated carefully:

Canonicalization is a documented web/search mechanism. Any claim that it directly causes a specific AI model to cite a URL is an additional hypothesis that requires measurement.

That sentence is much more defensible than declaring canonicalization an "LLM ranking factor."

Use descriptive links as information, not decoration

Compare:

<a href="/docs">Read more</a>

with:

<a href="/docs/robots-txt-guide">Read the complete robots.txt guide</a>

The second link communicates destination and context directly.

Good internal links improve navigation for people and create clearer relationships between documents for automated systems that parse links.

See Google: Links and crawling.

5. Evidence and citations: can the page support the answer?

The goal of GEO is not merely to be mentioned.

The stronger goal is to become useful evidence.

That sounds like a subtle distinction until you look at the quality of the web.

A machine answer may encounter:

a company homepage with vague claims;
a vendor landing page;
a technical documentation page;
a government source;
an academic paper;
a first-party benchmark;
an independent test;
a community discussion;
an outdated article;
a page that merely repeats another source.

These are not interchangeable evidence types.

A simple evidence hierarchy

For a factual claim, ask what source type naturally owns the truth.

Claim type	Often strongest starting evidence
Product feature	Official product documentation
API behavior	Official API documentation
Regulation	Government / regulatory source
Standard	Standards body / RFC / specification
Company founding / leadership	Official company source + independent corroboration where needed
Measured performance	Reproducible benchmark methodology and raw results
User experience	Independent testing / reviews / user research
Historical fact	Primary or high-quality secondary sources
Medical / scientific claim	Peer-reviewed research or authoritative health institution

The point is not "always use primary sources." Independent evidence can be essential. The point is to match the claim to the right evidence.

Citation presence is weaker than citation relevance

Suppose an AI answer cites your homepage because your company name appears there.

That is not the same as your technical guide being selected to support a detailed explanation of an API behavior.

For a source to be genuinely useful, the cited page should answer the claim being attributed to it.

That suggests a more useful internal metric:

Citation Support Rate

citations that actually support the associated claim
------------------------------------------------------ × 100
all citations attributed to your content

This is a proposed metric, not an industry standard.

Make evidence inspectable

For important facts, prefer:

exact dates;
precise numbers with methodology;
named versions;
direct quotes only when necessary and short;
tables when comparison matters;
links to primary documentation;
explicit limitations;
clear distinction between measurement and interpretation.

Avoid:

"Experts say..."

when you can say:

"Google's documentation states..."

or:

"In our test protocol, measured on [date], the result was..."

The second pattern gives a machine and a human a better evidentiary handle.

Don't manufacture original research

This is the rule that should govern the entire AuditMe content strategy.

If you have not measured it, do not write:

"Our data proves..."

Write:

"Our proposed methodology would measure..."

or:

"Current provider documentation indicates..."

or:

"A useful engineering hypothesis is..."

This is especially important in GEO because the industry is full of tiny samples presented as universal laws.

6. Write for humans and machines without writing like a robot

There is a false choice in AI-content discussions:

Write for humans or write for machines.

Good technical content can do both.

The trick is not to make prose robotic. The trick is to make meaning explicit.

The anti-AI-content formula

For each important section, answer five questions:

What is it?
Why does it matter?
What is actually documented?
What should I do?
What should I not assume?

Example:

What is `llms.txt`?

A proposed Markdown-based convention for giving agents a concise map of important site information and resources.

Why might it matter?

It may reduce the effort required for an agent to locate useful documentation on a site.

What is documented?

The proposal defines a format and describes intended uses.

What should I do?

Publish a small, accurate llms.txt if it helps your documentation ecosystem and keep it consistent with the actual site.

What should I not assume?

Do not claim that llms.txt is a universal ranking signal or a guaranteed citation mechanism.

That is clear writing. It is also machine-friendly writing.

Use tables where comparison is the point

Tables are especially useful when a question has mutually comparable dimensions.

Search vs AI Search vs RAG vs Browser Agent

Dimension	Search engine	AI search	RAG	Browser agent
Core job	Discover / rank documents	Retrieve evidence and answer	Retrieve passages from a corpus	Perform tasks
Needs indexing?	Usually yes	Often depends on source architecture	Depends on corpus	Not necessarily
Needs live access?	Crawling is asynchronous	Often at query time	Depends on pipeline	Usually yes
Main failure	Not found / not indexed	Wrong source / weak evidence	Wrong chunk / retrieval miss	Interaction failure
Important site property	Crawlability	Relevance + evidence	Chunkable documents	Semantic, deterministic UI
Typical output	Links	Answer + sources	Generated answer	Action / task result

Bot identity table

Provider	Mechanism / crawler	Broad purpose	Practical publisher concern
OpenAI	`OAI-SearchBot`	ChatGPT search discovery	Allow when you want ChatGPT search visibility
OpenAI	`GPTBot`	Potential model-training collection	Separate policy from search visibility
Anthropic	`ClaudeBot`	Model-development collection	Separate from search-oriented crawling
Anthropic	`Claude-SearchBot`	Search relevance	Search discovery path
Anthropic	`Claude-User`	User-directed web access	A user-requested retrieval path
Perplexity	`PerplexityBot`	Search discovery / linking	Search visibility path
Google	`Googlebot`	Search crawling	Core Google Search access
Google	`Google-Extended`	Gemini-related control	Not a Google Search ranking signal
Google	User-triggered fetchers / agents	User-requested fetching or navigation	Different access semantics

Always verify the current provider documentation before changing production access rules. Bot names, products and policies can change.

The writing style that travels best through systems

Strong technical pages tend to share these properties:

one obvious topic per page;
direct definitions;
consistent terminology;
short sections with meaningful headings;
data in tables when appropriate;
explicit dates and versions;
concrete examples;
original explanations rather than endless paraphrasing;
source links near important claims;
a references section for deeper research.

This is not a recipe for "gaming AI." It is simply good information architecture.

7. The technical stack: HTML, structured data, robots.txt, sitemaps and `llms.txt`

Technical implementation should answer one question:

Can a machine reliably recover the meaning and state of the page without guessing?

Semantic HTML beats decorative HTML

Prefer native semantics when they already express the interaction you need.

Better:

<button type="submit">Run audit</button>

Riskier:

<div role="button" tabindex="0">Run audit</div>

MDN explicitly recommends native <button> elements where possible because they provide built-in browser and accessibility behavior. See MDN: button role.

For a form:

<form aria-label="Run website audit">
  <label for="url-input">Website URL</label>
  <input
    id="url-input"
    name="url"
    type="url"
    autocomplete="url"
    required
    aria-describedby="url-hint"
  />
  <p id="url-hint">Enter the full URL, including https://</p>

  <button type="submit">Run audit</button>
</form>

This is good accessibility and good interface engineering regardless of AI.

ARIA should describe real state, not invented state

aria-expanded, aria-controls, aria-pressed and aria-busy can communicate state to assistive technologies when used correctly.

For expandable UI:

<button
  type="button"
  aria-expanded="false"
  aria-controls="advanced-options"
>
  Advanced options
</button>

<section id="advanced-options" hidden>
  ...
</section>

For loading state:

<div aria-live="polite" aria-busy="true">
  Analyzing your website…
</div>

MDN explains that aria-busy indicates that an element is being modified and can help assistive technologies avoid announcing incomplete updates. That does not prove that every browser agent will literally "wait for aria-busy=false". The safe claim is narrower: explicit state makes the interface more observable and deterministic. See MDN: aria-busy.

JSON-LD: useful context, not a magic AI switch

Google uses structured data to understand page content and support eligible Search features. Schema.org defines the vocabulary.

Example:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Example Audit Tool",
  "applicationCategory": "BusinessApplication",
  "description": "A website auditing tool for technical and content analysis.",
  "url": "https://example.com/tool"
}
</script>

The crucial rule is consistency: structured data should describe the same thing users can actually see on the page.

See:

Next.js App Router: dynamic metadata and JSON-LD

Next.js provides built-in metadata APIs and a documented JSON-LD pattern. The current docs recommend rendering JSON-LD in layout.js or page.js, and recommend sanitizing payloads before injecting them into the document.

Dynamic metadata

import type { Metadata } from 'next'

export async function generateMetadata(): Promise<Metadata> {
  const title = 'Website SEO Audit'

  return {
    title,
    description:
      'Audit technical SEO, content and website quality with a structured report.',
  }
}

export default function Page() {
  return <main>...</main>
}

See Next.js: generateMetadata.

Dynamic JSON-LD

export default async function Page() {
  const product = await getProduct()

  const jsonLd = {
    '@context': 'https://schema.org',
    '@type': 'SoftwareApplication',
    name: product.name,
    description: product.description,
    url: product.url,
  }

  const jsonLdString = JSON.stringify(jsonLd).replace(/</g, '\\u003c')

  return (
    <main>
      <script
        type="application/ld+json"
        dangerouslySetInnerHTML={{ __html: jsonLdString }}
      />
      {/* page content */}
    </main>
  )
}

See Next.js: JSON-LD.

Sitemaps still matter

Sitemaps are a straightforward way to expose canonical URLs you want crawled. They do not guarantee indexing, but they are useful discovery infrastructure.

See Google: Build and submit a sitemap and sitemaps.org.

`llms.txt`: useful proposal, not a universal protocol

The llms.txt proposal uses Markdown to provide a concise, human- and machine-readable index of important resources. The proposal has evolved and is now maintained as a community convention rather than a universal web standard.

See llms.txt v2 and the core format.

A sensible implementation might look like:

# Example Documentation

> Official documentation for Example API and SDKs.

## Documentation
- [Getting started](https://example.com/docs/start): First steps
- [Authentication](https://example.com/docs/auth): API authentication
- [API reference](https://example.com/docs/api): Complete endpoint reference

## Optional
- [Changelog](https://example.com/changelog): Product and API changes

Do not use llms.txt to duplicate the entire site. Its value is in curation and clarity.

Build a source-of-truth hierarchy

For an important product or organization, create one canonical source for each major claim class:

Product identity ───────→ canonical product page
API behavior ───────────→ official documentation
Pricing ────────────────→ current pricing page
Company facts ──────────→ organization/about page
Research findings ──────→ benchmark/report page
Change history ─────────→ changelog

This prevents the same fact from being written six different ways across six pages.

8. The agentic web: can an AI actually use your website?

Search visibility is only half of the emerging machine-web problem.

The other half is interaction.

Chrome's 2026 agent-ready toolkit describes an important transition: agents first had to search the web; increasingly they also need to use the web. The tooling now includes agentic browsing audits and Chrome DevTools workflows for testing how agents interact with pages.

See Chrome: A developer toolkit to make your website agent-ready.

OpenAI's current publisher guidance makes a similar engineering point for its agent experience: accessibility helps the agent understand page structure and interactive elements, including roles, labels and states. See OpenAI: Publishers and Developers FAQ.

A page can be readable but not operable

Consider this flow:

Agent finds pricing page       ✓
Agent reads pricing             ✓
Agent compares plans            ✓
Agent clicks "Start trial"     ✓
Agent sees form                 ✓
Agent understands fields        ✗
Agent cannot recover validation ✗
Task completed                  ✗

Traditional SEO would rarely capture this failure.

An agentic test should.

Design for explicit state

A good machine-operable interface exposes:

an accessible name for the control;
the control's role;
its current state;
the relationship to the content it controls;
a predictable action;
validation feedback;
a visible or programmatically exposed success state;
a recovery path when the action fails.

Example: an audit form in React

'use client'

import { useState } from 'react'

export default function AuditForm() {
  const [status, setStatus] = useState<'idle' | 'running' | 'completed' | 'error'>('idle')
  const [message, setMessage] = useState('')

  async function runAudit(event: React.FormEvent<HTMLFormElement>) {
    event.preventDefault()
    setStatus('running')
    setMessage('')

    try {
      // Perform the actual request here.
      await runWebsiteAudit()
      setStatus('completed')
      setMessage('Audit completed successfully.')
    } catch {
      setStatus('error')
      setMessage('The audit failed. Check the URL and try again.')
    }
  }

  const busy = status === 'running'

  return (
    <form aria-label="Run website audit" onSubmit={runAudit}>
      <label htmlFor="url-input">Website URL</label>

      <input
        id="url-input"
        name="url"
        type="url"
        required
        autoComplete="url"
        aria-describedby="url-hint"
      />

      <p id="url-hint">Use a complete HTTPS URL.</p>

      <button type="submit" disabled={busy} aria-busy={busy}>
        {busy ? 'Analyzing…' : 'Start audit'}
      </button>

      <p aria-live="polite" aria-busy={busy}>
        {message}
      </p>
    </form>
  )
}

The important point is not React, Next.js or a particular state library. You could implement this with React state, Zustand, Redux, server actions or another approach.

The agent-facing property is explicit behavior: the button is a button, the field has a label, the running state is observable, and success/error states are communicated.

There is no basis for claiming that this exact code guarantees success in every AI agent. It simply follows established web accessibility and interaction principles that make automation less dependent on visual guessing.

WebMCP and explicit agent tools

The web is also moving toward explicit machine interfaces. OpenAI's current site-tools documentation describes WebMCP as a proposed web standard allowing sites to expose tools directly to agents.

This suggests a useful progression:

Screen scraping
      ↓
Semantic HTML
      ↓
Predictable interaction
      ↓
Machine-readable state
      ↓
Explicit site tools

The endpoint of that evolution may not be "make the agent better at clicking." It may be make the website less dependent on clicking at all.

Agent UX checklist

Before calling a page agent-ready, test:

Test	Pass condition
Navigation	Major actions are discoverable without visual-only cues
Labels	Form controls have clear accessible names
Buttons	Native controls are used where possible
State	Loading/success/error state is explicit
Validation	Errors explain exactly what needs fixing
Recovery	Failed actions can be retried safely
Confirmation	High-impact actions require appropriate confirmation
Determinism	The same input produces predictable outcomes
Focus	Keyboard and focus behavior are coherent
Success	Completion is clearly represented in DOM/UI state

9. Measurement: from AI visibility to Agent Task Completion Rate

The industry already has many ways to count traffic and rankings. The emerging machine web needs additional measures.

The danger is inventing a giant score and pretending it is an industry standard.

A better approach is to define narrow metrics with explicit formulas.

1. AI Mention Rate

queries where the target entity is mentioned
--------------------------------------------- × 100
eligible test queries

Useful for measuring brand presence.

Not equivalent to recommendation quality.

2. Citation Rate

queries where a target URL is cited
----------------------------------- × 100
eligible test queries

Useful, but simplistic.

3. Citation Support Rate

cited answers where the page actually supports the claim
---------------------------------------------------------- × 100
all cited answers

This is more meaningful because it distinguishes a real citation from a decorative citation.

4. Retrieval Hit Rate

For a controlled retrieval system:

queries where the correct passage/page is retrieved
----------------------------------------------------- × 100
eligible queries

This should be measured against a known relevance rubric.

5. Entity Consistency Rate

A proposed metric for auditing whether important facts are represented consistently across canonical sources.

consistent entity facts across audited pages
--------------------------------------------- × 100
entity facts checked

Example facts:

official name;
product category;
current URL;
parent organization;
version;
pricing date;
capability/limitation.

6. Agent Task Completion Rate

This is the metric worth testing seriously.

successfully completed eligible tasks
-------------------------------------- × 100
eligible agent task attempts

A task should have a clear start and a clearly defined successful end state.

Examples:

find the pricing plan that includes feature X;
run a website audit;
create a report;
locate API documentation;
submit a support request;
add an item to a cart;
compare two plans.

7. Agent Step Success Rate

successful task transitions
---------------------------- × 100
attempted transitions

This lets you diagnose where a task breaks.

Example:

Step	Human	Agent
Find pricing	100%	100%
Open plan comparison	99%	91%
Start signup	98%	83%
Complete required fields	97%	72%
Recover from validation	94%	41%
Reach confirmation	96%	66%

These numbers are illustrative only. They are not AuditMe measurements.

8. Agent Time to Completion

Measure:

first task event → successful completion

Do not optimize only for raw speed. A fast wrong action is worse than a slower correct one.

9. Agent Recovery Rate

tasks successfully recovered after recoverable failure
-------------------------------------------------------- × 100
tasks that encountered a recoverable failure

This is especially useful for forms, search interfaces and multi-step workflows.

10. Machine Referral Share

Where referral data can be reliably identified:

visits attributed to a tracked AI/referral source
-------------------------------------------------- × 100
all tracked visits

OpenAI currently documents utm_source=chatgpt.com on ChatGPT search referrals, which makes one provider-level measurement path practical. Provider-specific analytics should be documented rather than inferred.

An Agent Telemetry Schema

A useful implementation pattern is to record task semantics, not just clicks.

Suggested event structure

type AgentEvent = {
  event:
    | 'task_started'
    | 'step_viewed'
    | 'action_started'
    | 'validation_failed'
    | 'action_succeeded'
    | 'task_completed'
    | 'task_abandoned'

  taskId: string
  stepId?: string
  route: string
  timestamp: string
  outcome?: 'success' | 'failure' | 'cancelled'
  errorCode?: string
  durationMs?: number
}

Send these events from the workflow boundary, not from arbitrary UI motion.

Example

function track(event: AgentEvent) {
  navigator.sendBeacon(
    '/api/telemetry',
    JSON.stringify(event)
  )
}

track({
  event: 'task_started',
  taskId: 'audit-2026-09',
  route: '/website-seo-checker',
  timestamp: new Date().toISOString(),
})

The key distinction is:

Telemetry should tell you whether the business task succeeded, not merely whether somebody clicked a pixel.

How do you know an agent performed the task?

This requires caution.

A website should not assume that a request is human or automated based only on a user-agent string. Browser agents may render pages using ordinary browser stacks, and privacy/security controls can change the observable network signature.

Better options include:

explicit agent tool calls when the integration supports them;
server-side task IDs;
authenticated application events;
signed callbacks where appropriate;
trace/context IDs propagated through a workflow;
provider-specific headers or metadata when officially documented.

The exact mechanism depends on the product architecture.

The metric itself remains useful even when the actor classification is imperfect: track the task; then separately classify the execution channel with evidence.

10. The AuditMe playbook: benchmark, fix the bottleneck, publish the evidence

The previous sections explain the system. This final section turns it into a practical operating model.

Step 1: Audit the evidence chain, not just the homepage

Create a matrix for your most important URLs:

URL	Discover	Access	Parse	Retrieve	Understand	Verify	Cite	Act
Product page	✓	✓	✓	?	?	?	?	✓
API docs	✓	✓	✓	✓	✓	✓	✓	N/A
Pricing	✓	✓	✓	✓	✓	?	?	?
Signup	✓	✓	✓	N/A	✓	N/A	N/A	?

Use ✓, ?, ✗ and a note.

This is more actionable than a single "AI readiness score."

Step 2: Fix the earliest meaningful failure

If the page is blocked at Access, do not spend three days rewriting the copy.

If the page is readable but ambiguous, do not start by adding more schema.

If the page is frequently cited but fails agent tasks, more content is probably not the immediate bottleneck.

A good order is:

ACCESS
  ↓
PARSE
  ↓
RETRIEVE
  ↓
UNDERSTAND
  ↓
VERIFY
  ↓
CITE
  ↓
ACT

Fix the earliest meaningful break first.

Step 3: Create canonical sources for critical facts

For every business-critical claim, decide where the truth lives.

One product → one canonical product description
One API behavior → one official endpoint/reference page
One pricing model → one current pricing source
One research result → one dated benchmark/report

Other pages can summarize it, but should link back to the source of truth.

Step 4: Make the page easy to extract

A strong reference page usually includes:

a precise title;
one-sentence definition;
TL;DR;
comparison table where appropriate;
concrete examples;
source links;
dates and versions;
limitations;
FAQ for genuinely recurring questions;
stable headings and anchors.

Do this for people first. Machines benefit because the information architecture is explicit.

Step 5: Make the interface semantically honest

Use:

<button> for buttons;
<a> for navigation;
<label> for form labels;
<form> for forms;
headings in a logical hierarchy;
aria-* states only when they describe real state;
visible and programmatic error messages;
predictable success/failure states.

This is accessibility engineering, not an AI hack.

Step 6: Treat `llms.txt` as optional documentation infrastructure

Create it when it helps agents and documentation consumers find your most important resources.

Do not depend on it.

Do not promise it.

Do not turn it into a second sitemap full of every URL on the site.

Step 7: Define your benchmark before collecting your data

This is where the AuditMe AI Web Intelligence Benchmark v1 comes in.

Status: proposed methodology — not yet executed

AuditMe does not currently claim to have completed the cross-model dataset described in this article.

The purpose of publishing the methodology first is to make future measurements reproducible instead of inventing conclusions first and looking for a methodology afterward.

Proposed dataset fields

benchmark_version
run_id
run_timestamp
query_id
query_text
query_intent
engine
model
retrieval_mode
source_url
source_domain
source_type
source_position
citation_present
citation_relevance
claim_supported
entity_match
content_date
freshness_bucket
structured_data_present
first_party_source
external_corroboration
access_status
parse_status
retrieval_status
agent_task_id
agent_step
agent_step_success
task_success
duration_ms
notes

Proposed evaluation dimensions

Dimension	Core question
Discovery	Was the source findable?
Access	Could the system retrieve it?
Retrieval	Was the relevant page/passage selected?
Understanding	Was the entity/claim interpreted correctly?
Verification	Did the source support the claim?
Citation	Was the source exposed and relevant?
Recommendation	Was the recommendation a good fit?
Action	Could an agent complete the task?

Proposed query classes

A future benchmark should not use only generic head terms.

Include:

definition queries
comparison queries
best-of queries
problem-solving queries
commercial-intent queries
technical queries
entity disambiguation queries
freshness-sensitive queries
source-verification queries
agent-task queries

Why release methodology before results?

Because it prevents a common research error:

deciding what the study should prove before deciding how the study should be run.

A public methodology lets other teams criticize the design before the results acquire authority.

Step 8: Release the dataset openly when it exists

The ideal public package is:

/auditme-ai-web-benchmark-v1
├── README.md
├── LICENSE
├── methodology.md
├── schema.json
├── prompts/
├── raw/
├── normalized/
├── evaluations/
├── analysis/
└── examples/

Potential distribution targets include GitHub and Hugging Face Datasets, with stable versioning.

Do not publish credentials, private user data or provider-restricted material. Publish only what the collection method and source licenses allow.

Step 9: Turn original research into a living reference

When the benchmark eventually exists, publish:

Benchmark v1.0
     ↓
Dataset
     ↓
Methodology
     ↓
Results
     ↓
Limitations
     ↓
Replication guide
     ↓
Benchmark v1.1 / v2

A static "ultimate guide" is useful.

A living research asset is more defensible.

Step 10: Measure the bottleneck and repeat

The practical loop is:

OBSERVE
  ↓
DIAGNOSE
  ↓
PRIORITIZE
  ↓
FIX
  ↓
VERIFY
  ↓
PUBLISH THE EVIDENCE
  ↓
REPEAT

That is the operating system for AI-ready websites.

The 100-point AI Web Checklist

Use this as an implementation checklist, not as a universal ranking score.

A. Discovery — 10 points

[ ] Important URLs are internally linked
[ ] No critical page is orphaned
[ ] XML sitemap exists and is current
[ ] Canonical URLs are deliberate
[ ] Important pages return stable URLs
[ ] Redirect chains are minimized
[ ] No accidental noindex on critical pages
[ ] Public content is not hidden behind accidental auth
[ ] Search-facing URLs are understandable
[ ] URL changes have a migration plan

B. Access — 10 points

[ ] Critical pages return correct HTTP status codes
[ ] CDN/WAF does not accidentally block legitimate crawlers
[ ] Rate limiting is monitored
[ ] HTTPS works consistently
[ ] Bot policy is documented internally
[ ] Provider-specific crawler requirements are reviewed
[ ] Server logs can diagnose failed requests
[ ] Error pages do not masquerade as successful HTML
[ ] Authentication is required only where appropriate
[ ] Public content can be retrieved without unnecessary interaction

C. Parsing — 10 points

[ ] Primary content exists in usable HTML
[ ] Important text is not only inside canvas/image assets
[ ] Headings are semantic
[ ] Lists use list semantics
[ ] Tables are real tables when tabular data is presented
[ ] Forms use real form controls
[ ] Links use anchors
[ ] Images have meaningful alternatives where needed
[ ] Dynamic content has a coherent rendered state
[ ] Page source and rendered content are not contradictory

D. Retrieval — 10 points

[ ] H1 states the exact topic
[ ] Important answers appear early
[ ] Headings describe the question being answered
[ ] Paragraphs are self-contained enough to quote
[ ] Links describe their destinations
[ ] Key concepts use consistent terminology
[ ] Tables are used for actual comparisons
[ ] FAQ questions are real user questions
[ ] Important definitions are explicit
[ ] No important fact depends on vague surrounding context

E. Entity clarity — 10 points

[ ] Canonical name is consistent
[ ] Product/company/entity type is explicit
[ ] Official URL is clear
[ ] Parent organization relationships are clear
[ ] Aliases are intentional
[ ] Dates and versions are explicit
[ ] Duplicate pages are reviewed
[ ] Structured data matches visible content
[ ] Different pages do not contradict core facts
[ ] One obvious source of truth exists for important claims

F. Evidence — 10 points

[ ] Major factual claims have an appropriate source
[ ] First-party claims are identified as such
[ ] Independent evidence is used when useful
[ ] Numbers include context or methodology
[ ] Time-sensitive claims include dates
[ ] Version-sensitive technical claims include versions
[ ] Limitations are stated
[ ] Experimental claims are labeled experimental
[ ] Hypotheses are not written as facts
[ ] Sources remain accessible and relevant

G. Citability — 10 points

[ ] Pages contain concrete facts, not only slogans
[ ] Definitions are concise
[ ] Tables summarize important relationships
[ ] Important claims are attributable
[ ] Sources are easy to inspect
[ ] Page scope is clear
[ ] Author/organization context exists when relevant
[ ] Update date is visible where useful
[ ] The page provides something worth citing
[ ] The page does not overclaim what the evidence proves

H. Agent interaction — 10 points

[ ] Native buttons are used where possible
[ ] Form fields have labels
[ ] Controls have accessible names
[ ] Expand/collapse state is explicit
[ ] Loading state is observable
[ ] Validation errors are explicit
[ ] Success state is explicit
[ ] Retry/recovery paths exist
[ ] High-impact actions have appropriate confirmation
[ ] Core tasks can be completed predictably

I. Measurement — 10 points

[ ] Important tasks have measurable start events
[ ] Important tasks have measurable success events
[ ] Errors use stable codes where useful
[ ] Durations are captured
[ ] Agent channel is classified carefully
[ ] AI referrals can be analyzed where provider data allows
[ ] Citation audits use a consistent rubric
[ ] Retrieval tests use fixed prompts and versions
[ ] Benchmarks record dates
[ ] Results distinguish observation from interpretation

J. Governance — 10 points

[ ] Documentation has an owner
[ ] Critical facts have canonical sources
[ ] Content changes are versioned where needed
[ ] Provider policy changes are monitored
[ ] Security rules are reviewed before bot exceptions
[ ] Personal/private data is excluded from public benchmarks
[ ] Research data has a license/usage policy
[ ] Benchmark methodology is public
[ ] Known limitations are documented
[ ] The score is treated as a diagnostic, not a search-engine fact

What not to do in 2026

Don't build a "GEO hack stack"

There is no defensible evidence that one magic combination of llms.txt, FAQ markup, exact word count, or paragraph formula forces all AI systems to cite a site.

Don't confuse Google AI Search with every other AI system

Google's AI Overviews and AI Mode operate inside Google's Search ecosystem. Other providers have different crawlers, retrieval methods, products and policies.

Don't call every AI crawler a "training bot"

Providers themselves distinguish search-oriented crawling from model-development collection and user-triggered access.

Don't claim that a source was used just because it was cited

Citation is evidence of source presentation, not a transparent window into every internal step of a model's reasoning.

Don't publish benchmark numbers you did not measure

This should be non-negotiable for a serious research brand.

Don't build an inaccessible interface and then blame the agent

A confusing form is a confusing form. Fix the interface.

What a genuinely AI-ready website looks like

A strong AI-ready site is surprisingly normal.

It has:

Fast, stable pages
        +
Clear information architecture
        +
Accessible semantic HTML
        +
Canonical sources of truth
        +
Useful structured data
        +
Strong internal linking
        +
Evidence-backed content
        +
Transparent dates / versions
        +
Predictable interactions
        +
Measured workflows

Notice what is missing:

There is no magic AI tag.

That is the point.

The deeper shift: websites are becoming knowledge interfaces

The most important change is not that ChatGPT or Gemini can summarize pages.

The deeper change is that software can increasingly:

find information;
retrieve evidence;
compare alternatives;
navigate interfaces;
invoke tools;
complete workflows.

That means a website now has at least three audiences:

People who read it.

Machines that retrieve it.

Agents that may operate it.

A page that serves all three well does not need a separate "AI version" of the web.

It needs clear information and explicit interfaces.

A practical final test

Take one critical page and ask these ten questions:

Can a crawler discover it?
Can the intended systems fetch it?
Can useful content be parsed without guessing?
Can the exact answer be retrieved?
Is the entity unambiguous?
Can important claims be verified?
Would another engineer cite this page as evidence?
Can a user understand it without reading everything?
Can an agent operate the relevant interface?
Can you measure whether the job actually succeeded?

If the answer to number 1 is no, start there.

If numbers 1–8 are yes but number 9 is no, you have an agent UX problem.

If 1–9 are yes but 10 is no, you have a measurement problem.

That is much more useful than asking whether your website is "AI optimized."

Final takeaway

The web is not being replaced by one giant AI search engine.

It is becoming a layered machine-readable environment where different systems discover, retrieve, interpret, verify, cite, recommend and sometimes act on the same underlying documents.

That changes the optimization question.

The old question was:

How do I rank this page?

The newer question is:

How do I make this page useful to a retrieval system?

And the emerging question is:

How do I make this website useful to a machine that must finish a task?

The strongest answer is not a bag of GEO tricks.

It is an engineering discipline:

Make the right information easy to discover, easy to access, easy to parse, easy to retrieve, hard to misunderstand, easy to verify, worth citing and safe to act upon.

That is the foundation of the machine-readable web.

And it is a much more durable goal than optimizing for any single model, crawler or product.

About the practical tools

For a website-level technical baseline, the AuditMe Website SEO Checker can be used as a practical starting point for reviewing core SEO and website signals.

For a fast score-oriented overview, the AuditMe SEO Score Checker provides another starting point before deeper investigation.

The broader AuditMe blog contains the supporting SEO, GEO and AI-search research that sits around this framework.

11. References, documentation and further reading

The links below are intentionally weighted toward primary documentation, standards and first-party engineering resources. Check them directly before making production changes because web-agent ecosystems evolve quickly.

Google Search, crawling and AI Search

OpenAI

Anthropic

Perplexity

Agentic web and browser automation

HTML and accessibility

Next.js and structured data

HTTP, robots and sitemaps standards

`llms.txt`

Performance and user experience

Common Crawl and open-web research

Search quality and research literacy

Research status and editorial standard

This article intentionally separates:

provider-documented behavior;
web standards;
engineering recommendations;
proposed metrics;
future benchmark methodology;
illustrative examples.

It does not present an unexecuted benchmark as completed research.

The AuditMe AI Web Intelligence Benchmark v1 in this article is a proposed methodology. No cross-model benchmark results are claimed here.

When results eventually exist, they should be published with the prompts, test dates, model/provider versions where observable, sampling rules, evaluation rubric, raw or appropriately licensed data, limitations and replication instructions.

That is the standard required if this work is going to be useful as a serious reference rather than another round of GEO folklore.

Last reviewed: September 13, 2026.

How to Measure AI Search Visibility in 2026: The Evidence-First GEO Framework

Eduard — Fri, 11 Sep 2026 15:22:51 +0000

The practical rule: do not measure “GEO” as a magical score. Measure observable evidence: presence, recommendation, citation, source quality, accuracy, and business outcome.

AI search has changed the shape of the result page.

The old SEO question was simple:

Where do we rank?

The new question is harder:

When someone asks an AI system to solve a problem, does our brand become part of the answer — and does the system have a strong source it can use to support that answer?

That is not the same thing as a keyword position.

A company can rank for a valuable query in traditional Google Search and still be missing from an AI-generated recommendation. It can be mentioned without being recommended. It can be recommended without being cited. It can be cited, but the citation can point to the wrong page. The right page can be cited while the answer still contains an outdated price or an invented feature.

And there is a second shift happening underneath search itself: AI agents are becoming users of the web. An agent does not only need to discover a page. It may need to understand an interface, identify a form field, navigate a site, interpret a state, and perform an action.

So “AI visibility” is better treated as a measurement system, not a marketing buzzword.

This article presents an evidence-first model that can be run with a spreadsheet and first-party analytics, then extended with specialized tools when scale justifies them.

The model is designed around five principles:

Measure what was actually observable.
Keep user prompts stable enough to reveal trends.
Separate first-party data from vendor models and manual observations.
Treat citations as evidence, not as proof that a page is “authoritative.”
Connect visibility to outcomes without pretending correlation is causation.

Google’s current documentation is explicit about one thing that cuts through much of the GEO hype: the same foundational SEO practices remain relevant for AI Overviews and AI Mode. Google says there are no additional technical requirements or special AI-specific schema needed to appear in those features. It also says Google Search does not require llms.txt. (Google Search Central: AI features and your website, Google: Optimizing your website for generative AI features)

At the same time, measurement has become more concrete. Google Search Console now has a dedicated Generative AI performance report; Bing Webmaster Tools has an AI Performance report with page-level citation activity and grounding-query data; and OpenAI documents both crawler controls and attribution-friendly referral tracking from ChatGPT. (Google Search Console, Bing AI Performance, OpenAI Publishers and Developers FAQ)

The result is a much better workflow:

observe → measure → diagnose → improve → verify

Everything below is designed to make that loop repeatable.

TL;DR — The 60-second system

Most teams do not need 50 GEO metrics. Start with six questions.

Question	What to record	Minimum source
1. Are we present?	Brand/entity mentioned or absent	Manual prompt panel / GEO tool
2. Are we recommended?	Recommended, neutral mention, or absent	AI answer
3. Are we cited?	Any supporting source URL shown	AI answer
4. Is our page cited?	Owned URL, exact URL, page relevance	AI answer
5. Is the answer correct?	Accuracy, freshness, factual errors	Human review
6. Does it matter commercially?	AI referrals, leads, sign-ups, sales	Analytics / CRM

The minimum viable loop

Freeze 12–25 real prompts → run them consistently → capture the full answer and cited URLs → score the same signals every time → fix the largest gap → run again.

Do not replace the prompt set every week.

Do not compare an old answer from one interface with a new answer from another and call that a trend.

Do not treat a third-party “AI score” as if it were a search-engine ranking.

One rule that prevents bad reporting

A visibility event and a business outcome are different events.

A citation can happen without a click. A click can happen without a conversion. A conversion can happen after multiple AI and non-AI touchpoints.

Keep those layers separate.

The model: what AI search visibility actually means
The measurement system: prompts, protocol and evidence
First-party data: Google, Bing, ChatGPT and AI referrals
From mention to citation: how content becomes referenceable
Technical readiness: crawlability, bots, llms.txt, schema and canonicalization
AI agents: building websites machines can understand and operate
Competition and diagnosis: finding the actual visibility gap
Improvement and scale: the 30-day loop and tool decision
Reporting and execution: turn evidence into decisions
FAQ, master checklist and primary sources

1. The model: what AI search visibility actually means

Stop looking for a single “GEO rank”

There is no universal AI equivalent of “position 3.”

Different AI products can use different retrieval systems, answer-generation methods, models, source-selection logic, interfaces, locations, and freshness windows. Even within the same product, two runs can differ.

Google states that AI Overviews and AI Mode can use query fan-out: related searches across subtopics and data sources that help assemble a response. Google also notes that AI Overviews and AI Mode can use different models and techniques, so the responses and links can vary. (Google Search Central)

That has an important measurement consequence:

A single answer is an observation. A repeated protocol is a measurement system.

The goal is not to eliminate all variability. The goal is to make variability visible and comparable.

The five core visibility states

Use this vocabulary consistently across reports.

State	Definition	Example
Mention	The entity appears in the answer	“Brand A is a popular option…”
Recommendation	The system actively suggests the entity for the task	“For a small team, Brand A is a good choice…”
Citation	The system presents a source URL as supporting evidence	`[brand.com/guide]`
Owned citation	The citation points to a page you control	Your documentation, pricing, case study, etc.
Relevant owned citation	The right owned page supports the claim	Pricing question → pricing page

Then add two downstream dimensions:

Dimension	Definition
Accuracy	The description is factually correct, current and not misleading
Outcome	A measurable business event can be associated with AI-referred traffic or assisted discovery

These are separate because combining them into a single raw number hides useful information.

The Visibility Evidence Ladder

A practical ladder makes the concept easier to communicate.

Level	Name	Description
0	Absent	The brand is not present in the answer
1	Mentioned	The brand is named but is not meaningfully recommended
2	Recommended	The system suggests the brand for the stated user intent
3	Cited	A URL from the brand or another source is shown as evidence
4	Relevant owned citation	The source belongs to the brand and is the right page for the claim
5	Accurate, current owned citation	The answer cites the right first-party page and represents the entity correctly
6	Business outcome	The discovery event contributes to a real business outcome (qualified visit, sign-up, lead, purchase)

Level 6 is not a visibility rank. It is the outcome layer.

An original metric you can calculate without buying a platform

A useful editorial metric is the Visibility Evidence Index (VEI).

It is not a Google metric, an OpenAI metric, or an industry-standard ranking. It is a transparent normalization of the evidence ladder above.

For each valid prompt run, assign the highest level achieved from 0–5.

Then:

VEI = (average evidence level / 5) × 100

Example:

12 prompt runs
average level = 2.75

VEI = (2.75 / 5) × 100 = 55

Why use a ladder instead of six weighted percentages?

Because it avoids the temptation to double-count the same event. A recommendation is already a form of presence. A relevant owned citation is already a citation.

Keep the raw signals alongside the VEI. The index is for communication; the evidence is for diagnosis.

Add a volatility measure

AI systems can vary from run to run. Do not hide that.

Choose a repeat-sample subset — for example, 10% of your prompt panel — and run each one three times under the same conditions.

Calculate:

Volatility = different-outcome runs / repeated runs

This tells you whether a change from 40% to 50% visibility is likely to be meaningful or just normal answer variation.

You do not need to repeat every prompt three times every week. That wastes time while giving a false impression of precision. A repeat subset is usually more efficient.

What not to claim

Avoid these statements unless you have direct evidence:

“Google ranks our page #1 in AI Mode because we added FAQ schema.”

“ChatGPT prefers our site because we added llms.txt.”

“Our GEO score increased because the model updated its algorithm.”

“Bing’s grounding query is the exact question users asked.”

The first three confuse correlation, unsupported causal claims and vendor folklore. The last one is specifically wrong for Bing: Bing says grounding queries are grouped phrases representing retrieval activity, not full user questions or prompts. (Bing AI Performance)

A good report says what happened, what is known, what is inferred, and what remains unknown.

2. The measurement system: prompts, protocol and evidence

Your prompt panel is the measurement instrument

The most important operational decision is to define a stable set of real questions.

A useful starter panel has 12–25 prompts. A more mature program might have 50–100+, but bigger is not automatically better. A badly designed 200-prompt list is less useful than 20 questions connected to real buying decisions.

A practical 24-prompt panel

Intent	Suggested count	What it measures
Brand / entity	4	Whether the system understands the entity
Category / discovery	6	Visibility within the consideration set
Comparison / alternatives	6	Positioning against competitors
Problem / how-to	6	Whether useful content is referenceable
Trust / evidence	2	Reviews, proof, authoritativeness, factual support

You can scale this down to 12 prompts or expand it later.

Build prompts from behavior, not marketing copy

The best prompt sources are usually:

Google Search Console queries from the last 90 days
customer support questions
sales calls and objections
product demos
onboarding questions
community discussions
comparison searches
People Also Ask and related search patterns

Write the question the way a buyer would ask it.

Not:

“best-in-class AI-powered SEO intelligence platform for enterprise-grade visibility”

But:

“what are the best tools for auditing a technical SEO problem on a site with 500 pages?”

Realistic wording makes the panel more useful because it stays anchored to actual jobs-to-be-done.

Freeze the panel

Once the initial panel exists, freeze it for at least 8–12 weeks.

You can still maintain a separate discovery backlog. Do not replace the baseline questions every week.

Use three buckets:

BASELINE
Fixed prompts used for trend reporting.

DISCOVERY
New prompts tested but not yet part of the baseline.

RETIRED
Prompts removed because the intent no longer matters or the wording became invalid.

This simple separation prevents one of the most common GEO measurement errors: changing the ruler while measuring the object.

Record the context, not just the answer

Each observation should capture:

Field	Why it matters
Date/time	AI answers and sources can change
Engine	Different products are different systems
Surface	AI Overview, AI Mode, ChatGPT search, etc.
Country / locale	Results can be regional
Language	Content and entities differ by language
Device	Some interfaces vary by device
Authentication state	Personalization can differ
Exact prompt	Makes the run reproducible
Full answer	Allows later verification
Cited URLs	Core evidence
Mention status	Presence signal
Recommendation status	Intent-fit signal
Accuracy verdict	Trust signal
Competitors	Competitive context
Screenshot or archived response	Audit trail

Private browsing can reduce some personalization, but it does not create a universal “neutral AI answer.” Location, language, product settings, model changes and interface behavior can still influence the result.

Record the conditions instead of pretending they do not exist.

The Evidence Card

The Evidence Card is the atomic unit of the system.

One card = one prompt run on one engine at one point in time.

Use this structure:

Evidence Card

Date:
Engine:
Surface:
Locale:
Prompt:

Brand mentioned: Yes / No
Brand recommended: Yes / No
Citation present: Yes / No
Owned URL cited: Yes / No
Relevant URL cited: Yes / No
Accuracy: Correct / Mixed / Incorrect

Cited URLs:
- https://...
- https://...

Competitors named:
- Competitor A
- Competitor B

Material claim supported by citation:
"..."

Notes:
...

Screenshot / capture:
...

This makes the dataset reviewable by a human and usable by software later.

CSV-ready tracking schema

A simple spreadsheet can use these columns:

run_date,engine,surface,locale,prompt_id,prompt,mentioned,recommended,cited,owned_cited,relevant_cited,accuracy,competitors,cited_urls,notes,screenshot_url

Add model, device, and auth_state when your program becomes more rigorous.

Do not use “majority vote” as a substitute for measurement

A tempting method is:

Run a prompt three times → whichever answer appears most often is the truth.

That is useful in some experiments, but it is too simplistic as a general measurement rule.

Instead, preserve the runs.

For example:

Run A → recommended + owned citation
Run B → mentioned + no citation
Run C → recommended + competitor citation

That is not noise to throw away. It is information about the system’s response variance.

3. First-party data: Google, Bing, ChatGPT and AI referrals

Manual observation tells you what an AI answer looked like.

First-party reporting tells you what the platforms themselves expose about your site.

These datasets should complement each other, not be forced into one denominator.

Google Search Console: Generative AI performance report

As of August 31, 2026, Google says its Generative AI performance report has rolled out to all websites worldwide. The report provides data about how a site performs in generative AI features on Google Search, including impressions over time and breakdowns such as pages, device and country. (Google Search Console Help)

This is a major change because it gives publishers a first-party view of visibility inside Google’s generative search surfaces.

But do not overread the report.

Google’s broader AI-features documentation says that traffic from AI features is still part of overall Search reporting, and recommends combining Search Console with Analytics for deeper analysis of visits and conversions. (Google Search Central)

What Google’s report is good for

trend lines
page-level differences
country/device patterns
detecting whether generative AI visibility exists at meaningful volume
prioritizing pages for deeper analysis

What it does not replace

It does not replace your prompt panel.

A GSC impression is not equivalent to:

“ChatGPT recommended our product for the prompt ‘best SEO audit tool.’”

Different evidence, different denominator.

Bing Webmaster Tools: AI Performance

Bing’s AI Performance report provides citation-oriented data across supported AI experiences including Microsoft Copilot, AI-generated summaries in Bing and selected partner integrations. (Bing AI Performance)

Bing exposes several particularly useful concepts:

Total Citations
Cited Pages
Average Cited Pages
Grounding Queries
Page-level citation activity
Grounding-query ↔ page mapping
preview capabilities such as Intents, Topics, Citation Share and Compare

This is useful because it moves beyond the question “did Bing mention me?” toward:

Which of my URLs are actually being used as sources?

The critical caveat about grounding queries

Do not label the column “user queries.”

Bing explicitly says grounding queries are grouped phrases representing the retrieval activity associated with cited content. They are not full user questions or prompts, and the system does not expose individual AI answers or exact prompts through this view. The data is also aggregated and sampled. (Bing AI Performance)

That makes the data valuable for topic diagnosis — but dangerous to over-interpret.

Citation Share is not a rank

Bing’s Citation Share preview shows your site’s percentage of the citation space for a grounding query. It does not identify the other domains holding the remaining share and does not represent a ranking or authority score. (Bing AI Performance FAQ)

Use it as relative citation presence, not “position.”

ChatGPT: separate search visibility from training controls

OpenAI documents two concepts that publishers frequently mix up.

OAI-SearchBot

This user-agent controls access used to surface content in ChatGPT search experiences.

GPTBot

This is a separate crawler signal used for content that may be included in training datasets.

Those are different decisions.

A site can choose a policy for one without making the exact same choice for the other. OpenAI’s publisher documentation explains this distinction and notes that publishers allowing OAI-SearchBot access can track ChatGPT referral traffic using analytics platforms. ChatGPT automatically adds utm_source=chatgpt.com to referral URLs. (OpenAI Publishers and Developers FAQ)

That gives you a practical analytics rule:

Do not create a generic “AI traffic” bucket and stop there. Keep ChatGPT as a separately attributable source whenever your analytics setup allows it.

Perplexity and Anthropic: crawler policy matters

Perplexity says its PerplexityBot follows robots.txt. If a page is blocked, Perplexity says it may still index the domain, headline and a brief factual summary, while not indexing the full or partial text content of the blocked page. (Perplexity Help Center)

Anthropic documents separate bots for separate purposes. ClaudeBot supports model development, while Claude-User can retrieve websites in response to user-directed Claude questions. Anthropic also says its bots respect robots.txt and documents support for Crawl-delay. (Anthropic Privacy Center)

The broader lesson is simple:

AI crawler policy is not one global switch called “allow AI.”

Different providers expose different controls for search, retrieval, model development and agentic access.

Build an AI-source dashboard without inventing metrics

Keep first-party data in its own layer:

Layer	Metric	Status
Google	Generative AI impressions	First-party
Bing	Citations	First-party
Bing	Cited pages	First-party
Bing	Grounding phrases	First-party, sampled/aggregated
Analytics	ChatGPT referrals	First-party analytics
Prompt panel	Mention / recommendation / citation	Controlled observation
Paid GEO platform	Share of voice / modeled visibility	Vendor methodology

This table alone prevents a large amount of bad reporting.

4. From mention to citation: how content becomes referenceable

The goal is not to “trick the model.”

The goal is to become a source that a retrieval-and-answer system can confidently use when the page is relevant.

Google’s current guidance emphasizes the same fundamentals it recommends for classic Search: valuable, unique, people-first content; accessible pages; clear textual information; useful media; internal discoverability; and structured data that matches visible content. (Google Search Central)

That sounds less exciting than a GEO hack.

It is also much more defensible.

Write for the question before writing for the keyword

A strong reference page usually answers a real question quickly, then earns the reader’s time with depth.

A reliable pattern is:

Question
↓
Direct answer
↓
Evidence / source
↓
Context and exceptions
↓
Method / example
↓
Related questions

This is not a special “LLM format.” It is simply good information architecture.

The Claim → Evidence → Context pattern

For important factual claims, use:

Claim: state the answer plainly.

Evidence: show the source, date, methodology, benchmark, documentation or original data.

Context: explain when the claim is true, where it is not, and what assumptions apply.

Example:

Claim: Google does not require llms.txt for inclusion in AI Overviews or AI Mode.

Evidence: Google’s AI Search documentation says there are no additional technical requirements and no need to create special AI files or markup for these features.

Context: An llms.txt file may still be useful as an optional convention for some AI-oriented workflows, but it should not be presented as a Google ranking requirement.

This is much more citeable than a paragraph of unqualified advice.

Original data beats generic advice

If ten articles say:

“Improve your content quality.”

and one page publishes:

“We tested 1,200 prompts across four surfaces. Here were the 50 prompts where our category had the largest citation gap…”

the second page has something the first ten do not: new evidence.

Referenceable content tends to have one or more of these properties:

first-party measurements
original benchmarks
transparent methodology
specific examples
reproducible steps
dated facts
direct links to primary documentation
clear definitions
useful tables
documented limitations

None of these guarantees an AI citation. They make a page stronger as a source.

Build “source pages,” not just blog posts

A common mistake is to publish one giant article and expect it to support every query.

Instead, think in terms of a source architecture.

For example:

AI Search Visibility Hub
├── Measurement methodology
├── Prompt tracking guide
├── Google AI visibility guide
├── Bing AI citations guide
├── AI crawler reference
├── Technical readiness checklist
├── Agent readiness guide
├── Industry benchmark
└── Glossary / definitions

Each page has a narrower evidence job.

The hub explains the system.

The supporting pages contain the deep evidence.

This creates a citation-friendly knowledge graph inside the site rather than one overloaded URL.

Make entity facts painfully consistent

AI systems can encounter your organization in many places.

If one page says “AuditMe is a free SEO audit tool” and another says something different, the model has conflicting evidence. Consistency across product pages, documentation, About page, structured data and high-authority third-party descriptions reduces the chance of mixed or incorrect answers.

5. Technical readiness: crawlability, bots, llms.txt, schema and canonicalization

Even excellent content can stay invisible if the site is technically closed to AI crawlers or hard for machines to parse.

The practical technical checklist

Check	Why it matters	How to verify
Important pages return 200	Failed responses are not usable sources	URL Inspection / live fetch
No accidental `noindex`	Blocks both classic Search and AI surfaces	robots meta + HTTP headers
Canonical strategy is coherent	Prevents signal dilution	Canonical tags + Search Console
Internal links expose key pages	Helps discovery	Crawl + internal link analysis
Sitemap is current	Supports discovery	sitemap.xml
robots.txt is deliberate	Controls crawler access	/robots.txt
AI crawler policies understood	Different bots have different purposes	Per-provider documentation
Important content is available in text	Models prefer extractable text	View-source / rendered HTML
Structured data matches visible content	Helps understanding, not a magic ranking lever	Rich Results Test
Structured data uses the correct type	Wrong type can be ignored	Schema documentation

A fast way to surface many of these issues at once is the free Website SEO Checker from AuditMe. It runs a practical technical scan and returns specific, actionable recommendations on crawlability, robots, schema and more.

On `llms.txt`

Google currently states there are no additional technical requirements and no need for special AI files or markup to appear in AI Overviews or AI Mode. llms.txt can be treated as an optional convention that some AI-oriented workflows may find useful, but it should not be presented as a Google ranking requirement. (Google AI features, llms.txt proposal)

Structured data realism

There is no official Google rule that FAQ schema automatically increases AI citations. Structured data should accurately represent the page. Google’s documentation distinguishes FAQ-like content from Q&A pages and does not describe FAQPage or QAPage as a universal AI-visibility lever. (Google Q&A structured data, Google structured-data guidelines)

6. AI agents: building websites machines can understand and operate

Retrieval visibility and actionability are different properties.

A site can be highly citeable but poorly operable by an agent.

OpenAI’s current publisher/developer FAQ notes that ChatGPT Atlas uses ARIA tags to interpret page structure and interactive elements, and recommends following WAI-ARIA best practices so buttons, menus and forms have descriptive roles, labels and states. (OpenAI Publishers and Developers FAQ)

Five principles of agent readiness

Clear task discovery — A machine can identify the primary action.
Accessible names — Interactive elements have clear, descriptive names.
Deterministic interaction — The same action produces a predictable state transition.
Explicit state and errors — Success, failure, pending and authentication states are observable.
Recoverability — A failed flow explains the next action.

Prefer native HTML

Prefer:

<button type="submit">Run audit</button>
<label for="url">Website URL</label>
<input id="url" name="url" type="url">

over custom components that look correct visually but expose weak semantics.

ARIA is valuable, but W3C explicitly advises preferring native techniques when possible. (W3C Accessible Names and Descriptions)

The Agent Readiness test

For your five most important workflows, ask:

Test	Pass condition
Find the task	A machine can identify the primary action
Identify controls	Interactive elements have clear names
Fill the form	Labels and field purposes are explicit
Submit	The action is deterministic
Observe result	Success/error state is explicit
Recover	A failed flow explains the next action
Repeat	The workflow can be executed consistently

This is a separate maturity axis from AI citation visibility.

7. Competition and diagnosis: finding the actual visibility gap

Measurement is useful only when it changes what you do next.

Build the competitor matrix

For every important baseline prompt, record:

Prompt	You	Competitor A	Competitor B	Competitor C	Best cited source
Best technical SEO audits	Recommended	Mentioned	Recommended	Absent	Competitor A guide
Alternatives to X	Absent	Recommended	Recommended	Mentioned	Competitor B comparison
How to diagnose Y	Cited	Absent	Cited	Cited	Documentation

Now you can ask:

For which intents does the market remember my competitors but not me?

The six-gap diagnostic

Gap	Likely cause	Primary action
You are absent	Weak entity recognition or topical coverage	Build the missing information layer
Mentioned but not recommended	Weak intent mapping	Clarify category, use case, limitations
Recommended but not cited	Missing strong first-party source	Create or improve the authoritative page
Cited, but wrong page	Information architecture problem	Strengthen the canonical page + internal links
Cited, but answer is wrong	Conflicting or outdated facts	Repair the fact graph
Visibility rises, business does not	Weak destination experience	Fix relevance, speed, message match, conversion

Find “citation competitors,” not only SEO competitors

Your traditional competitors are not always the sources AI systems cite. AI answers may also pull from documentation sites, GitHub, benchmark reports, publications, community threads and implementation guides.

The important question becomes:

Which sources are competing with us for evidence, not just for rankings?

8. Improvement and scale: the 30-day loop and tool decision

Days 1–7: establish the baseline

Build prompt panel + Evidence Card template
Define competitor list and source taxonomy
Capture GSC / Bing / analytics baseline
Run every baseline prompt at least once
Select a repeat-sample subset for volatility

Write one page at the end of week one:

What do AI systems currently understand about us?
Where do they recommend us?
Where do they cite us?
Which pages do they cite?
Which facts are wrong?
Where do competitors beat us?

Days 8–14: fix the source architecture

Do not publish ten new blog posts immediately.

First improve the pages AI systems should cite:

product page
pricing page
documentation
methodology
comparison pages
original research
canonical category guide
About / organization page

For every page ask:

What factual claim should this URL be the best source for?

Days 15–21: strengthen external evidence

Look at the sources that appear when competitors are recommended. Identify inaccurate third-party descriptions of your brand and opportunities for original data or high-quality coverage.

Days 22–30: re-run and analyze

Repeat the exact baseline panel. Classify changes as Improved / Stable / Declined / Volatile. Attach causal-confidence language (High / Medium / Low) to every material change.

When to move from manual to paid tools

Stay manual while the panel is small and a weekly review takes under 30 minutes.

Move to paid tooling when you need hundreds of prompts, many markets, daily tracking, automated answer collection, historical databases or team workflows.

Buying rule

Buy automation, not authority.

No vendor owns a universal “truth score” for AI visibility. The best tool is the one whose data you understand well enough to challenge.

When evaluating tools, ask:

Which prompts are actually being run?
How are locations represented?
Which AI surfaces are covered?
Are responses and URLs stored?
Is the metric observed or modeled?
Can I export raw evidence?
Can I import my own prompt panel?
What happens when the underlying AI interface changes?

A beautiful dashboard without raw evidence is an opinion with a UI.

For ongoing practical guides and technical checks, the AuditMe blog regularly publishes updated GEO checklists and case studies.

9. Reporting and execution: turn evidence into decisions

A good AI visibility report should fit on one page before the appendix.

The one-page monthly report

AI SEARCH VISIBILITY — SEPTEMBER 2026

Baseline prompt panel: 24 prompts
Engines / surfaces: [list]

Visibility Evidence Index: 61 / 100
Previous month: 54
Change: +7

Presence: 72%
Recommendation: 49%
Citation: 41%
Relevant owned citation: 29%
Accuracy: 94%

Google Generative AI impressions: XXXX
Bing total citations: XXXX
ChatGPT referrals: XXX
AI-attributed conversions: XX

Biggest improvement: [one sentence]
Biggest gap: [one sentence]
Competitor gaining visibility: [one sentence]

Top 3 actions:
1. ...
2. ...
3. ...

Attach the raw evidence sheet.

The evidence hierarchy

When a number and a narrative disagree, prefer this order:

Raw answer / URL evidence → first-party platform data → analytics → vendor aggregates → interpretation

Executive red flags

Score up, first-party traffic flat
Citations up, accuracy down
Recommendations up, citations flat
Traffic up, conversions flat
One engine improves while another declines

Do not average away engine-specific differences.

The monthly decision rule

At the end of each reporting period, choose one primary bottleneck (presence, recommendation, citation, source relevance, accuracy, or business outcome) and allocate the next month’s work against it.

The operational loop

PROMPT PANEL
     ↓
OBSERVE ANSWERS
     ↓
CAPTURE EVIDENCE
     ↓
MEASURE
     ↓
DIAGNOSE GAPS
     ↓
FIX SOURCE / SITE / REPUTATION
     ↓
VERIFY
     ↓
UPDATE BASELINE

That is your GEO operating system.

10. FAQ, master checklist and primary sources

FAQ

Is SEO still relevant for AI search?

Yes. Google explicitly says its generative AI features are rooted in core Search systems and that SEO fundamentals remain relevant. A page must be indexed and eligible for a normal Search snippet to be eligible as a supporting link in AI Overviews or AI Mode.

Is there a special GEO algorithm score from Google?

No public universal score should be treated that way. You can create a transparent diagnostic metric (such as the Visibility Evidence Index), but it is an analytical framework — not a Google ranking signal.

Do I need llms.txt for Google AI Overviews or AI Mode?

No. Google says there are no additional technical requirements and no need for special AI files or markup.

Does FAQ schema make AI systems cite my page more often?

There is no official Google rule that it does. Structured data should represent the page accurately.

Is a brand mention the same as a citation?

No. A mention tells you the entity appeared. A citation indicates that a source URL was presented as evidence.

Are Bing grounding queries the exact user prompts?

No. They are grouped phrases representing retrieval activity.

Can Search Console show exactly what ChatGPT says about my brand?

No. They are different evidence sources.

Can I measure ChatGPT referrals?

Yes. OpenAI says ChatGPT referral URLs include utm_source=chatgpt.com.

Should I block or allow AI crawlers?

There is no universal answer. Decide separately for search/retrieval, model development and user-directed access based on your business policy.

Does Google-Extended block my site from Google Search?

No. It is a separate control for certain Gemini training and grounding uses and does not affect inclusion in Google Search.

How often should I run my prompt panel?

Weekly is a practical baseline for a small team. Consistency matters more than raw frequency.

How many prompts should I track?

Start with 12–25 high-value real questions. Expand when the baseline is stable.

The master checklist

Measurement

[ ] 12–25 baseline prompts defined and frozen
[ ] prompts tied to real user intent
[ ] engine, surface, locale and context recorded
[ ] full answers and cited URLs preserved
[ ] competitor mentions and accuracy reviewed
[ ] repeat sample used to estimate volatility

First-party data

[ ] Google Search Console Generative AI report reviewed
[ ] Bing AI Performance reviewed where available
[ ] ChatGPT referrals separated in analytics
[ ] conversions reviewed alongside AI referrals

Content

[ ] clear answer near the top
[ ] important claims supported by evidence
[ ] entity facts consistent across pages
[ ] one canonical source for each important claim
[ ] comparison and alternative pages cover real intent

Technical

[ ] important pages return successful responses
[ ] no accidental noindex
[ ] canonical strategy coherent
[ ] robots policy deliberate
[ ] structured data matches visible content

Agent readiness

[ ] semantic HTML used where practical
[ ] interactive controls have clear names
[ ] form fields have explicit labels
[ ] states and errors are observable

Reporting

[ ] one-page executive summary
[ ] raw evidence available
[ ] observed vs inferred clearly separated
[ ] next month’s work tied to measured gaps

The 7-day launch plan

Day 1 — Create the spreadsheet and 12–25 baseline prompts.

Day 2 — Run the baseline on your most important AI surfaces and preserve answers + URLs.

Day 3 — Connect Google Search Console, Bing Webmaster Tools and analytics.

Day 4 — Identify the pages that should be cited for pricing, category, comparisons and core facts.

Day 5 — Audit indexing, canonicalization, internal links, sitemap and robots policies.

Day 6 — Test the five most important user workflows for agent readiness.

Day 7 — Choose one primary bottleneck and fix that gap first.

A quick technical starting point for Day 5 is the free Website SEO Checker from AuditMe.

The final principle

AI search is not a separate internet floating above the web.

It is increasingly a new way of retrieving, synthesizing, citing and acting on information that already exists across the web.

That is why the most durable strategy is not to chase every new model update.

Build pages that are:

discoverable → understandable → useful → evidence-backed → current → citable → actionable.

Then measure whether those properties are actually showing up in AI answers.

The winning loop is not:

publish → hope → check a magic score.

It is:

measure → observe → diagnose → improve → verify.

And the strongest organizations will eventually treat AI visibility the same way mature SEO teams treat technical health: as an observable system with logs, evidence, thresholds, regressions and continuous improvement.

That is the real shift from GEO as a buzzword to GEO as an engineering discipline.

Primary sources and current references

Google Search

Microsoft Bing

OpenAI

OpenAI — Publishers and Developers FAQ

Perplexity & Anthropic

Accessibility & agents

Independent convention & practical tools

Publication note: Written for September 2026. Platform interfaces, crawler policies, reporting features and vendor pricing can change. For operational decisions, verify the linked primary documentation at the time of implementation.

The era of prompt engineering is giving way to loop engineering. Read the field guide to implementing reliable, long-horizon AI workflows with the PAOVR framework.

Eduard — Fri, 11 Sep 2026 10:10:51 +0000

Eduard

Sep 11

The PAOVR Loop: The Real Agent Loop That Actually Finishes Jobs

#ai #webdev #programming #typescript

18 min read

The PAOVR Loop: The Real Agent Loop That Actually Finishes Jobs

Eduard — Fri, 11 Sep 2026 10:07:43 +0000

Plan → Act → Observe → Verify → Repair

Stop building agents that narrate completion. Start building systems that prove it.

In 2026, the conversation has moved beyond “prompt engineering is dead.” What matters now is quieter, harder, and far more useful: loop engineering.

Many agents are already competent at individual steps. They still fail on longer jobs because the surrounding system has no enforceable definition of “done.”

We have all seen the pattern: an agent burns tokens, makes a plausible tool call, produces a confident summary, and declares success — while the actual requirement was only partially satisfied, the evidence was never checked, or a failed path quietly changed the goal.

The model is not always the problem. Often, the missing piece is the contract around it.

This is a production-grade field guide to the PAOVR Loop — Plan → Act → Observe → Verify → Repair — a practical control pattern designed to make long-horizon agent work more reliable.

It builds on ideas from ReAct, Plan-and-Solve, modern agent harnesses, and the hard lessons of teams running agents in production rather than demos.

You will leave with:

A precise anatomy of the PAOVR Loop for long-horizon tasks
The actual prompts we use in production
JSON contracts and TypeScript interfaces that agents and runtimes can both consume
Real 2026 implementation stacks (Next.js, TypeScript, Supabase, Vercel)
Vector memory patterns that turn amnesiac agents into compounding workers
Failure patterns that still dominate and how to kill them
A one-week install plan you can run on your own stack

This is not theory. It is the difference between an agent that talks about finishing and one that proves it finished.

Why Most Agents Still Fail in 2026
The Shift from Prompting to Loop Engineering
The PAOVR Loop: Plan → Act → Observe → Verify → Repair
Stage 1 — Plan: Stop Asking Agents to Think. Ask Them to Graph
Stage 2 — Act: Atomic Execution with Tool Contracts
Stage 3 — Observe: Grounding in Reality
Stage 4 — Verify: The Step Almost Everyone Skips
Stage 5 — Repair: Recovery Without Restarting from Zero
JSON Contracts That Survive Production
The Prompts We Actually Use in Production
Context Engineering Inside the Loop
Circuit Breakers, Budgets, and Stopping Conditions
Failure Patterns I Keep Seeing in 2026
How Real Production Systems Use This Loop
A One-Week Install Plan
Ship Checklist
What to Do in the Next 15 Minutes
Further Reading, People & Tools
Frequently Asked Questions

1. Why Most Agents Still Fail in 2026

The failure mode has shifted.

In 2023–2024, models were often simply wrong. In 2026, many models are much better at atomic tasks. The system can still fail because there is no enforceable definition of “done.”

Typical symptoms:

The agent produces a beautiful plan and then freestyles the execution.
It marks a task complete because the last tool call returned something.
It never re-checks the original success criteria after the final action.
Context grows until the original goal is buried under tool noise.
When something breaks, the agent rewrites the entire plan instead of repairing the broken leaf.
A repair changes the acceptance criteria without making that change explicit.

The common root cause is the same: the loop has no Verify stage with teeth, and no immutable completion contract protecting the original goal.

ReAct (Yao et al., 2022) taught us to interleave Thought → Action → Observation. That was necessary. It was not sufficient for long-horizon production work. Plan-and-Solve (Wang et al., 2023) added an explicit planning phase. Modern agent harnesses add budgets, tools, worktrees, and stopping conditions.

The missing discipline is making verification and repair part of the control system rather than optional model behavior.

If your agent cannot answer the question “How do I know this is finished?” with evidence instead of narration, it is not finished.

2. The Shift from Prompting to Loop Engineering

Prompt engineering optimized the single turn.

Loop engineering optimizes the entire trajectory.

The people building reliable agents increasingly talk about:

Boris Cherny (Claude Code, Anthropic): “I don’t prompt Claude anymore. I have loops running that prompt Claude.”
Addy Osmani and the broader community: loop engineering as a discipline.
Anthropic’s guidance on the Agent SDK / Claude Code: gather context → take action → verify work → repeat.
Production systems: circuit breakers, max turns, cost thresholds, external verifiers, and explicit stopping conditions.

The unit of design is no longer “the perfect system prompt.”

It is the control loop that keeps the model inside a contract until the contract is satisfied, evidence is insufficient, or the budget is exhausted.

This article is about that loop — specifically the PAOVR version of it.

3. The PAOVR Loop: Plan → Act → Observe → Verify → Repair

Here is the minimal reliable shape:

PLAN
  ↓
ACT (one atomic step)
  ↓
OBSERVE (real tool / environment feedback)
  ↓
VERIFY (against explicit done_when)
  ↓
  ├─ satisfied → next task or finish
  └─ not satisfied → REPAIR → back to ACT or re-plan only the affected subtree

This is the PAOVR Loop.

Key design rules:

One atomic action per Act. Prefer 1–3 tool calls maximum.
Every task has a crisp done_when. If you cannot write it, the task is not ready.
Verify is external or at least independent. The same model that generated the work should not be the only judge.
Repair is local. Do not throw away the entire plan because one leaf failed.
Hard stopping conditions always exist. Iteration limit, cost limit, repeated identical failure, context budget.
The completion contract is immutable by default. Repair can change the execution path, but it cannot silently change what counts as success.

Outcome and execution health are separate

A task has two independent dimensions:

Outcome: Did the task satisfy its original completion contract?

Execution health: How did we get there?

A repaired task can therefore be successful while still having a degraded execution path.

For example:

outcome = satisfied
execution_health = repaired

This means the original requirement was met, but the system had to recover from a failed path.

That is different from:

outcome = satisfied
execution_health = clean

which means the task satisfied its contract without requiring repair.

Other execution-health states can include:

clean
repaired
degraded
budget_exhausted
authorization_blocked

“Done” answers whether the outcome was achieved. Execution health answers whether the path remained healthy. Never collapse these into a single boolean status.

4. Stage 1 — Plan: Stop Asking Agents to Think. Ask Them to Graph

Planning is no longer “think step by step.”

It is the production of an executable graph.

What a good plan looks like

Goal stated as an observable outcome
Explicit assumptions
Clarifying questions only when the cost of being wrong is high
Tasks that are leaf-level (doable in 1–3 tool calls)
Dependencies declared
Every task has a done_when string that a later verifier can check
Risks listed
The completion contract is defined before execution begins

Planner prompt we actually use

Act as the Task Planner. You do not execute. You only produce an executable plan.

Rules:
1. Split the goal into atomic steps.
2. One step = one action or one tightly related group of tool calls (max 3).
3. Declare dependencies with task IDs.
4. Every step must have a crisp done_when that can be verified later.
5. If critical information is missing, list assumptions and clarifying_questions. Do not invent facts.
6. Output strict JSON only. No prose essay.

Return exactly this schema:
{
  "goal": "string",
  "assumptions": ["string"],
  "clarifying_questions": ["string"],
  "tasks": [
    {
      "id": "t1",
      "title": "string",
      "description": "string",
      "depends_on": ["t0"],
      "tool_hint": "none|search|code|browser|api|file",
      "done_when": "observable condition that proves completion"
    }
  ],
  "risks": ["string"]
}

This planner is deliberately dumb about execution. That is the point. Separation of concerns is what keeps the system debuggable.

5. Stage 2 — Act: Atomic Execution with Tool Contracts

The Executor receives one task, the current plan state, and any previous observations. It is forbidden from jumping ahead.

Executor prompt

Act as the Executor Agent.
Take exactly one next task from the plan. Do not jump ahead. Do not invent missing data.

Inputs you will receive:
- plan JSON
- current_task_id
- previous tool results / observations (if any)

Method:
1. Re-read the done_when for the current task.
2. If you are blocked on missing data, request the cheapest tool or mark status blocked.
3. Perform the smallest useful action that moves the task forward.
4. Return structured output only:

## Action
(what you did)

## Evidence
(raw tool output or observation — never paraphrase away the truth)

## Status
done | partial | blocked

## Residual risks
(any new risks introduced)

## Next recommendation
(only if status is not done)

The Executor never decides the overall goal is finished. That decision belongs to the outer loop after verification.

6. Stage 3 — Observe: Grounding in Reality

Observation is the only place the model is allowed to see the real world.

Rules that still matter in 2026:

Never let the model invent tool output. The runtime supplies it.
Prefer structured tool responses over free text when possible.
Keep the observation window small and high-signal. Context rot is real.
Log every observation with a timestamp and tool name. You will need it for debugging.
Preserve raw evidence separately from model-generated summaries.

This is the stage that turns ReAct from a clever prompt into a more reliable control system.

7. Stage 4 — Verify: The Step Almost Everyone Skips

Verification is the difference between an agent that claims success and one that demonstrates it.

What “done” actually means

A task is done only when its done_when is true and the evidence supports that claim.

The verifier should preferably be:

A separate model call with a different system prompt, or
An external checker (tests, linter, schema validator, SEO score, human review), or
A deterministic function when the domain allows it.

Verifier prompt

Act as the Verifier. You do not generate new work. You only judge whether the current task is complete.

You receive:
- original task (including done_when)
- action taken
- evidence / observation
- any claimed result

Rules:
1. Quote the done_when.
2. Decide: satisfied | not_satisfied | insufficient_evidence.
3. If not_satisfied, name the single cheapest next check or repair.
4. Never accept narration as proof. Require evidence.
5. Never rewrite or weaken the original done_when.
6. Output strict JSON:

{
  "task_id": "...",
  "done_when": "...",
  "verdict": "satisfied|not_satisfied|insufficient_evidence",
  "evidence_summary": "one or two sentences",
  "missing": ["what is still required"],
  "recommended_repair": "smallest next action or null"
}

This is the stage that prevents the polite lie.

8. Stage 5 — Repair: Recovery Without Restarting from Zero

When Verify returns not_satisfied or insufficient_evidence, the system has two clean options:

Local repair — re-run or adjust only the failed leaf.
Subtree re-plan — only when dependencies or assumptions inside that subtree have actually changed.

Never throw away the entire plan because one step failed. That is how agents waste tokens and lose trust.

The completion contract is immutable

A repair may change how the work is performed.

It may not silently change what counts as success.

The original done_when is the acceptance contract for the task and its repair descendants.

When a subtree is re-planned:

Preserve the original task ID and completion contract.
Create a new plan version for the replacement subtree.
Link replacement tasks to the original task.
Carry the original done_when forward unchanged.
Record every changed assumption or dependency.
Preserve evidence from the failed subtree rather than deleting it.
Require explicit approval before weakening or changing the completion contract.

The core rule is:

Re-planning changes HOW the work is performed. It does not change WHAT counts as success.

Repair example

Suppose the original task is:

Task: Deploy API
done_when:
Production deployment is live, all tests pass, and p95 latency is below 200 ms.

The deployment fails because a dependency is unavailable.

A repair can replace the execution path:

t3a → provision alternate dependency
t3b → deploy API
t3c → run production verification

But t3c must still verify the original contract:

Production deployment is live,
all tests pass,
and p95 latency is below 200 ms.

The repair is allowed to change the path. It is not allowed to quietly turn that into:

API deployment command completed.

That distinction is what prevents a repair planner from converting failure into a false success.

Repair record

For production systems, keep an explicit repair record:

interface RepairRecord {
  repair_id: string;
  parent_task_id: string;
  original_contract: string;
  original_contract_version: number;
  new_plan_version: number;
  changed_assumptions: string[];
  reason: string;
  evidence: string[];
}

The failed path becomes part of the audit trail rather than disappearing from history.

9. JSON Contracts That Survive Production

Free-form text is fine for humans. Agents need schemas.

Here is a minimal production-oriented run schema:

{
  "run_id": "uuid",
  "goal": "...",
  "outcome": "in_progress|satisfied|not_satisfied|insufficient_evidence",
  "execution_health": "clean|repaired|degraded|budget_exhausted|authorization_blocked",
  "tasks": [
    {
      "id": "t3",
      "status": "pending|running|done|partial|blocked|failed",
      "verdict": "pending|satisfied|not_satisfied|insufficient_evidence",
      "attempts": 2,
      "contract_version": 1,
      "plan_version": 2,
      "parent_task_id": "t3",
      "last_evidence": "...",
      "verified_at": "ISO timestamp"
    }
  ],
  "cost_so_far": {
    "tokens": 12840,
    "usd_estimate": 0.41
  },
  "circuit_breaker": {
    "max_turns": 40,
    "max_cost_usd": 5.0,
    "identical_failure_limit": 3
  }
}

The important change is that outcome and execution health are independent.

A run can be:

outcome = satisfied
execution_health = repaired

without pretending it was a clean execution.

TypeScript contracts

type TaskVerdict =
  | "pending"
  | "satisfied"
  | "not_satisfied"
  | "insufficient_evidence";

type ExecutionHealth =
  | "clean"
  | "repaired"
  | "degraded"
  | "budget_exhausted"
  | "authorization_blocked";

interface Task {
  id: string;
  title: string;
  description: string;
  depends_on: string[];
  tool_hint: "none" | "search" | "code" | "browser" | "api" | "file";

  // Immutable acceptance contract
  done_when: string;
  contract_version: number;

  // Execution state
  status?: "pending" | "running" | "done" | "partial" | "blocked" | "failed";
  verdict?: TaskVerdict;
  execution_health?: ExecutionHealth;

  attempts?: number;
  last_evidence?: string;
  verified_at?: string;

  // Repair / re-plan lineage
  parent_task_id?: string;
  plan_version: number;
}

interface AgentPlan {
  goal: string;
  assumptions: string[];
  clarifying_questions: string[];
  tasks: Task[];
  risks: string[];
}

interface RunState {
  run_id: string;
  goal: string;

  outcome:
    | "in_progress"
    | "satisfied"
    | "not_satisfied"
    | "insufficient_evidence";

  execution_health: ExecutionHealth;

  tasks: Task[];

  cost_so_far: {
    tokens: number;
    usd_estimate: number;
  };

  circuit_breaker: {
    max_turns: number;
    max_cost_usd: number;
    identical_failure_limit: number;
  };
}

These interfaces become the source of truth between your orchestrator, execution layer, and logging system.

One implementation detail matters: TypeScript interfaces alone do not enforce runtime JSON from an LLM. In production, validate model output at runtime with a schema validator such as Zod or JSON Schema before mutating state.

10. The Prompts We Actually Use in Production

You already have the three core ones (Planner, Executor, Verifier).

Here is the outer loop controller that ties them together:

Act as the Loop Controller. You own the overall trajectory.

Your only job:
1. Load or create the plan.
2. Select the next ready task (dependencies satisfied, status not done).
3. Hand it to Executor.
4. Feed the result to Verifier.
5. On satisfied → record the verified outcome and continue.
6. Record execution health separately from the outcome.
7. On not_satisfied or insufficient_evidence → trigger Repair (local first).
8. Enforce circuit breakers before every new turn.
9. Never allow Repair or re-planning to weaken the original completion contract.
10. When all tasks satisfy their original contracts, emit final result + execution health + residual risks.
11. Never invent completion.

You speak only in structured status updates and JSON state.

These four prompts form a deployable skeleton for the PAOVR Loop. Keep them versioned in git the same way you version any other critical configuration.

11. Context Engineering Inside the Loop

Context is a finite resource. In long runs it becomes a primary failure mode.

Practical rules that still hold:

Keep the master policy (role, constraints, output contract) stable and cached.
Give the Executor only the current task + recent observations + the original done_when.
Summarize or offload completed tasks instead of replaying the entire history.
Prefer fresh context for pure execution workers and accumulated context only for the planner/orchestrator.
Measure context fill. When it crosses ~60–70% of the useful window, force a compression or checkpoint step.

This is why many robust agent systems treat the file system, git, and external memory as first-class context tools rather than dumping everything into the prompt.

Vector Memory as a First-Class Citizen

Context windows are large, but dumping everything into them destroys attention. External memory architectures can reduce that pressure.

A practical pattern:

After every completed (or failed) task, embed a short structured summary of what happened, the evidence, and the outcome.
Store those embeddings in a vector store. pgvector is a practical choice when vector search can live next to relational state.
Before the Plan stage of a new run, the orchestrator performs a micro-RAG retrieval against the agent’s own historical executions.
The retrieved constraints are injected into the Planner’s context as hard lessons (“previous attempts failed when the shadow-DOM selector timed out; prefer the data-testid path”).

The effect can compound. An agent that repeatedly failed to interact with a particular UI element across past sessions no longer has to rediscover the failure mode.

Pair embeddings with a fast, high-quality text-embedding model. Gemini embedding models are one option for balancing cost and quality. Keep the retrieval budget tiny — usually the top 3–5 most relevant past failures or successes are enough. Anything more can re-introduce context rot under a different name.

12. Circuit Breakers, Budgets, and Stopping Conditions

A loop without hard stops is a liability.

Minimum set:

Signal	Typical setting	Enforcement
Max turns / iterations	20–60 depending on task	Runtime
Max cost (USD or tokens)	Task-specific budget	Runtime
Identical failure streak	2–3	Instruction + runtime
Context budget	70% of useful window	Instruction
Wall-clock timeout	Optional	Runtime

When a breaker trips, the agent must:

Stop new actions.
Return partial results that were already verified.
State clearly what triggered the stop and what remains open.
Escalate if a human gate exists.

Partial verified work is always more valuable than a confident hallucination.

13. Failure Patterns I Keep Seeing in 2026

Narrated completion — the model says “done” without evidence.

Fix: hard Verify stage with external or independent judgment.
Plan that is actually a novel — tasks that still require a short essay of instructions.

Fix: keep splitting until each leaf is 1–3 tool calls.
Context rot — original goal buried under 30 tool observations.

Fix: aggressive pruning + separate orchestrator context + vector memory for long-term lessons.
Repair by total rewrite — one failure causes the agent to discard everything.

Fix: local-first repair and explicit subtree lineage.
Contract weakening during repair — the replacement plan quietly lowers the acceptance bar.

Fix: make done_when immutable by default, version the contract, preserve the original evidence, and require explicit approval for contract changes.
Missing done_when — “make it good” or “optimize the page.”

Fix: refuse to accept a task without an observable completion condition.
Tool hallucination — model invents tool results.

Fix: runtime always supplies Observation; model is never allowed to generate it.
Infinite polite loops — agent keeps “trying one more thing.”

Fix: circuit breakers with identical-failure detection.

These are recurring production failure modes. The exact frequency varies by system, model, tools, and task design.

14. How Real Production Systems Use This Loop

The pattern appears, under different names, in systems that ship:

Claude Code and the Anthropic Agent SDK / Claude Code docs — gather → act → verify → repeat, with explicit loop and stopping concepts.
Coding agents that treat the test suite as the verifier.
Research agents that force a verification step against sources before claiming a fact.
Content and SEO pipelines that run a quality gate after generation.

The 2026 Implementation Layer

The theory maps directly to modern stacks. You do not need a massive monolithic Python backend to run this loop cleanly.

A common high-leverage architecture in 2026:

Orchestration: Next.js App Router (or a lightweight server layer) owns the Loop Controller. Strict TypeScript contracts define the state model.
State & Logs: Supabase (Postgres + pgvector) stores run state, task history, and vector memory of past executions.
Execution: Serverless / edge functions on Vercel handle individual Act steps. This keeps the execution surface small and isolates failures.
Tool surface: Many teams standardize on the Model Context Protocol (MCP) so agents can talk to tools through a consistent interface.
Local development & coding agents: The same philosophy applies to tools like OpenCode and Cline. They write code, observe terminal output, verify against tests or linters, and repair locally rather than blindly restarting.

The key insight is that the PAOVR Loop is language-agnostic. Once you have typed contracts, runtime validation, and a reliable state store, the same shape can work for coding agents, research agents, and domain-specific crawlers.

Case Study: Surviving the Chaos of Web Crawling

Let’s look at a real production environment. When building the crawler pipeline for the AI SEO platform AuditMe, the biggest nightmare wasn’t parsing HTML — it was the unpredictability of the web. Sites time out, DOMs shift, JavaScript-heavy pages render differently, and standard linear scripts can break under noisy conditions.

To address this, the auditing engine was structured around PAOVR-style control. Instead of a monolithic script, the system uses Next.js App Router and Supabase to orchestrate atomic tasks. For example, if you run a URL through the free Website SEO Checker, the workflow can trigger multiple stages of planning, execution, observation, verification, and repair.

If a check fails (for example, an API timeout during a heavy DOM render), the system can catch the failure in verification, trigger a provider failover or local repair, and continue. Only the affected leaf needs to be retried.

The broader lesson is not that every crawler needs the same implementation. It is that unreliable external environments require explicit evidence, bounded retries, and local recovery.

15. A One-Week Install Plan

Day 1

Write the three core prompts (Planner, Executor, Verifier). Run them manually on a simple multi-step task. Measure where the model tries to skip Verify.

Day 2

Add strict JSON schemas (or TypeScript interfaces plus runtime validation) and a simple state object. Make the outer loop refuse to continue without valid state.

Day 3

Introduce one external verifier (tests, schema check, or a second model call). Force the system to use it.

Day 4

Add circuit breakers: max turns, cost, identical failure. Test them by deliberately breaking a tool.

Day 5

Implement local-first Repair. Confirm that a single failed leaf does not destroy the whole plan. Add explicit task lineage and contract versioning. Optionally wire a minimal pgvector memory store for past failures.

Day 6

Run a real 20–40 step task. Log every observation and verification. Identify the highest-friction stage.

Day 7

Write the one-page internal playbook for your team. Version the prompts and schemas. Put the state schema in git.

By the end of the week you should have a PAOVR Loop that is bounded, observable, and materially easier to trust than an agent that relies on narration alone.

16. Ship Checklist

Before you call any agent “production”:

[ ] Every task has an explicit done_when
[ ] Planner and Executor are separate
[ ] Verify stage exists and is independent
[ ] Repair is local-first
[ ] Original completion contract is immutable across repairs
[ ] Replacement subtrees preserve contract lineage
[ ] Contract changes require explicit authorization
[ ] Repair history and failed-path evidence are preserved
[ ] Outcome and execution health are tracked separately
[ ] Circuit breakers are enforced by the runtime, not just the prompt
[ ] Observations are never invented by the model
[ ] Completed work is preserved and evidenced
[ ] Cost and turn budgets are visible
[ ] Partial results are returned on early stop
[ ] Prompts and schemas are versioned
[ ] Long-term lessons are stored outside the context window (vector memory or equivalent)
[ ] LLM-generated JSON is validated at runtime before state mutation

If any critical box is unchecked, the agent is still closer to a demo than a production system.

17. What to Do in the Next 15 Minutes

Copy the Planner prompt into your current agent stack.
Take one real task you care about and force it to emit the JSON plan schema.
Write a done_when for the first three leaf tasks that a stranger could verify.
Add a single Verify call after the first Act.
Add a simple immutable contract field and task lineage before testing Repair.
Run it once and look at the difference between narration and evidence.

That is the difference between “it usually works” and “I can trust it when I’m not watching.”

18. Further Reading, People & Tools

Foundational papers

ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al., 2022
Plan-and-Solve Prompting — Wang et al., 2023
The Prompt Report — a useful survey of prompting techniques

People & practice

Boris Cherny (Claude Code, Anthropic) — loop-first mindset
Addy Osmani — loop engineering discussions and practice
Anthropic Agent SDK / Claude Code docs
OpenAI’s agent and Codex documentation

Tools & platforms worth watching

Claude Code / Anthropic Agent SDK
Cursor, OpenCode, Cline and modern coding agent harnesses
Model Context Protocol (MCP)
pgvector + modern embedding models for agent memory
Next.js App Router and Supabase for orchestration + state
Production observability for agents (cost, turns, verification rate)

19. Frequently Asked Questions

Isn’t this just ReAct with extra steps?

ReAct is the necessary interleaving of reasoning and acting. The PAOVR Loop adds explicit planning with contracts, independent verification, and controlled repair. Those additions make the control loop better suited to long-horizon work.

Do I still need a strong system prompt?

Yes. The prompts above are the system prompts. They are focused on policy and contracts rather than personality.

Can the same model do Plan, Act, and Verify?

It can, but independence is preferable. Even when the same base model is used, separate roles, prompts, and ideally different verification mechanisms reduce the chance that the system simply approves its own work.

What about multi-agent systems?

The same loop still applies. The orchestrator runs the outer PAOVR Loop; specialist agents become the Act stage for particular tools or domains.

How do I add long-term memory without exploding context?

Use vector memory (pgvector + embeddings) and retrieve only the top few relevant past failures or successes before planning. Keep the retrieval budget tiny.

How do I prevent Repair from weakening the goal?

Treat the original done_when as an immutable completion contract. Give replacement tasks explicit lineage to the original task, carry the contract forward unchanged, preserve failed-path evidence, and require an explicit authorization step before changing acceptance criteria.

How do I know when to stop adding stages?

When the agent can finish a meaningful multi-step task, survive a tool failure, preserve verified partial results, and stop under a hard budget — stop. Further complexity should earn its place by reducing a measurable failure mode.

Final note

The agents that will still be running in production in 2027 are unlikely to be the ones with the cleverest personality block. They will be the ones whose loops enforce a contract, demand evidence, remember past failures, and know how to repair without starting over.

Build the PAOVR Loop.

Version the contracts.

Verify everything.

Give the agent a memory that compounds.

Then the model can finally do what we have been asking it to do for years: finish the job.

Written for practitioners who ship. Updated for the 2026 agent landscape.

Hey! What FREE or open-weight AI models & agents do you use for UI & design refactoring? Which free setups (DeepSeek, Qwen Coder, Ollama) and tools (OpenCode, Cline, Continue) give you the cleanest React/Tailwind code? #discuss #ai #opensource #webdev

Eduard — Mon, 07 Sep 2026 12:44:58 +0000

Master Prompts in 2026: Stop Prompting Like It's 2023

Eduard — Thu, 03 Sep 2026 06:50:39 +0000

Master Prompts in 2026: Stop Prompting Like It's 2023

I still see people paste a 40-line “act as a senior expert with 20 years of experience” block into ChatGPT and call it engineering.

That stopped working as a strategy a while ago.

Models got better. Context windows got bigger. Agents started calling tools. And the failure mode shifted. It’s rarely “the model is dumb” now. It’s “your system has no contract.”

This is a long, practical write-up on master prompts — the stable policy layer above individual tasks. How to write them. How to force planning. How to run Plan → Act → Observe → Verify without theater. How to make the same prompt useful to a tired human at 11pm and to an agent loop that only understands schemas.

I’ve broken enough production prompts across GPT-4o, Claude 3.5 Sonnet, and Gemini-class stacks to have opinions. Some of them are uncomfortable.

TL;DR / Key Takeaways

A master prompt is not a clever sentence. It’s the policy layer: role, success criteria, process, constraints, output contract, failure handling.
Production reliability comes from LLM orchestration patterns — plan JSON, single-task executors, and explicit done_when checks — not from longer personality blocks.
JSON contracts + verification beat free-form answers. Agents that can’t prove completion will invent it.
Treat prompts like code: version them, eval them, and put a real verify step after generation (including SEO/quality checks when you publish).

What a master prompt actually is
The 7-part anatomy that doesn’t collapse under pressure
Frameworks worth keeping (and which ones to ignore)
Planning is the real skill
From plan to agent loop
Context engineering beats clever wording
Few-shot, JSON contracts, and the anti-hallucination rule
Copy-paste masters you can actually deploy
A real publish pipeline (including the verify step people skip)
Eval or you’re guessing
Failure patterns I keep seeing
PromptOps: treat prompts like code
One universal master prompt
Ship checklist
A one-week install plan
Frequently asked questions
Sources
What to do in the next 15 minutes

1. What a master prompt actually is

A master prompt is not a magic spell.

It’s the policy layer:

who the model is allowed to be
what “done” means
how it should think when the task is messy
what format comes out
what happens when it’s unsure

User prompts change every hour.

Master prompts change when your standards change.

If you rewrite your “system personality” for every ticket, you don’t have a system. You have vibes.

This distinction matters more once you leave single-chat workflows and enter prompt engineering for production — multi-step agents, tool routers, RAG pipelines, shared team libraries. The master prompt becomes the constant. Everything else is runtime input.

Official docs still matter here, even if the ecosystem moved fast:

OpenAI prompting guide
Anthropic prompt engineering overview
Google’s prompt engineering notes
The Prompt Report (Schulhoff et al.) — still the best single survey of techniques

One shift I care about in 2026: people say context engineering more than prompt engineering. Same game, wider board. You’re not only choosing words. You’re choosing what the model sees on each step inside a limited context window — policy, retrieved docs, tool traces, and the live task.

2. The 7-part anatomy that doesn’t collapse under pressure

Every master prompt I’ve kept in production has some version of these blocks. Skip one and you pay for it later.

Block	Hard question it answers
Role	Who are you, for whom?
Goal	What counts as success in measurable terms?
Context	What’s true about this environment right now?
Process	In what order do you work?
Constraints	What is forbidden even if it would be convenient?
Output contract	What shape must the answer take?
Failure policy	What do you do when data is missing?

Skeleton

ROLE
You are a [specific role]. You work for [audience].

GOAL
Success = [observable outcome].
Failure examples: [what “almost right” looks like].

CONTEXT
- Product / domain:
- Hard limits:
- Sources of truth:

PROCESS
1) State assumptions or ask the minimum clarifying question.
2) Build a dependency-aware plan.
3) Execute one atomic step at a time.
4) Verify against done_when.
5) Return result + residual risks.

CONSTRAINTS
- Do not invent facts, APIs, quotes, or metrics.
- Do not fake tool output.
- If uncertain, say so and propose the cheapest check.

OUTPUT
## Plan
## Result
## Verification
## Open questions

Notice what’s missing: motivational fluff. “Be world-class.” “Think deeply.” Models already try. What they lack is your definition of finished work.

On Claude 3.5 Sonnet and GPT-4o alike, vague quality adjectives underperform hard constraints and explicit success criteria. The model isn’t missing ambition. It’s missing your acceptance tests.

3. Frameworks worth keeping (and which ones to ignore)

The internet loves acronyms. Most of them are the same idea in a hoodie.

Keep these

RTF — Role / Task / Format

Fine for small jobs. Don’t overbuild.

CRAFT — Context / Role / Action / Format / Tone

Good default for writing, analysis, support.

Plan-and-Solve

Force a plan before the answer. Boring. Effective. See the planning literature around Plan-and-Solve and agent planning surveys like arXiv:2402.02716.

Chain-of-Thought

Still the simplest accuracy lever on multi-step reasoning. Original paper: Wei et al., 2022.

Tree of Thoughts

When one path isn’t enough and you need deliberate search. Yao et al., 2023.

ReAct

Thought → Action → Observation. If your agent uses tools and you don’t have this loop, you’re improvising.

Ignore these habits

Collecting 14 frameworks and using none consistently
Padding prompts with personality cosplay
Asking for “maximum creativity” on compliance tasks
Writing novels in the system message that burn token efficiency for no gain

Pick one structure. Run it for a week. Measure. Then change one variable.

Anthropic’s own guidance still ranks clarity, examples, thinking, structure above theatrical roleplay. Read their best practices if you haven’t in a while.

4. Planning is the real skill

Most “agent failures” are just un-decomposed work.

A useful rule from task-decomposition practice: keep breaking the job down until each leaf task is doable in 1–3 tool calls and has a crisp done_when. If a step needs a short novel of instructions, it isn’t a step yet. (EngineersOfAI notes on decomposition are blunt about this for a reason.)

This is the boring core of LLM orchestration: not more model calls for their own sake, but a graph of verifiable work units.

Two planning styles

Decomposition-first

Build the full plan, then execute. Best for stable workflows: migrations, docs, publish checklists.

Interleaved

Plan a little, act, replan. Best for research and debugging where the map changes under your feet — including RAG pipelines where retrieval quality shifts mid-run.

A plan JSON agents can actually consume

{
  "goal": "Ship a technical article with a pre-publish quality pass",
  "assumptions": [
    "Target platform is Dev.to",
    "Audience is builders using LLMs in real workflows"
  ],
  "tasks": [
    {
      "id": "t1",
      "title": "Outline + claims list",
      "depends_on": [],
      "tool_hint": "none",
      "done_when": "H2/H3 outline exists and 8–12 claims are listed"
    },
    {
      "id": "t2",
      "title": "Write full draft",
      "depends_on": ["t1"],
      "tool_hint": "none",
      "done_when": "Complete draft with no TODO markers"
    },
    {
      "id": "t3",
      "title": "Fact-check hard claims",
      "depends_on": ["t2"],
      "tool_hint": "search",
      "done_when": "Every strong claim has a source or is marked UNVERIFIED"
    },
    {
      "id": "t4",
      "title": "Publish checklist + SEO verify",
      "depends_on": ["t3"],
      "tool_hint": "api",
      "done_when": "Top 5 impact/effort fixes are written from evidence"
    }
  ],
  "risks": [
    "Stale references",
    "Generic advice with no operational detail"
  ]
}

Planner-only master prompt

You are Task Planner. You do not execute. You only produce an executable plan.

Rules:
1) Split the goal into atomic steps.
2) One step = one action or one tool call.
3) Declare dependencies.
4) Every step needs done_when.
5) If information is missing, add assumptions and clarifying_questions.
6) No prose essay. Structure only.

Return strict JSON:
{
  "goal": "...",
  "assumptions": [],
  "clarifying_questions": [],
  "tasks": [
    {
      "id": "t1",
      "title": "...",
      "description": "...",
      "depends_on": [],
      "tool_hint": "none|search|code|browser|api",
      "done_when": "..."
    }
  ],
  "risks": []
}

Microsoft’s agent curriculum makes the same point in plainer language: define the goal, break it, then assign work. See their planning design chapter.

5. From plan to agent loop

Once you have a plan, stop letting the model freestyle the whole graph.

The loop

Plan → Act → Observe → Verify → Repair or Next

Without Verify, agents lie politely. They narrate completion. They do not prove it.

This loop is where prompt engineering for production stops being “wording” and becomes control flow. The master prompt defines the rules. The orchestrator enforces step boundaries. Tools supply evidence. Verification closes the books.

Executor master prompt

You are Executor Agent.
Take exactly one next task from the plan.
Do not jump ahead.

Inputs:
- plan JSON
- current_task_id
- tool_results (if any)

Method:
1) Re-read done_when for the current task.
2) If blocked on missing data, request a tool or mark blocked.
3) Do the smallest useful action.
4) Return:

## Action
## Evidence
## Status: done | partial | blocked
## Next recommendation

Repair rule that saves hours

If Status is partial or blocked:
1) Name the blocker in one sentence.
2) Propose the cheapest next check.
3) Do not rewrite the entire plan unless dependencies actually changed.

This is less glamorous than “autonomous agent.” It is also why some systems finish jobs and others generate confident debris.

6. Context engineering beats clever wording

I used to spend an hour polishing adjectives. Now I spend that hour deciding what not to put in context.

High-signal rule

Use the smallest token set that still steers behavior. That’s token efficiency as an engineering constraint, not a slogan.

Practical layout

Content	Placement
Stable policy / role	Front of the prompt (also helps caching)
Reference docs / data	Clearly delimited blocks
Retrieved RAG chunks	After policy, tagged and ranked by relevance
Examples	After policy, before the live task
User task	End

In RAG pipelines, the master prompt should also say how to treat retrieved text: prefer it over parametric memory, cite chunk ids, and refuse to invent when retrieval is empty. Without that policy, retrieval becomes decoration.

OpenAI’s notes on prompt caching are worth reading if cost and latency matter: put stable prefixes first, variable content last.

Delimiters

<policy>...</policy>
<context>...</context>
<retrieved>...</retrieved>
<examples>...</examples>
<task>...</task>

XML, Markdown headings, triple backticks — pick a convention and stop rotating it every sprint. Inconsistency is a silent quality tax across GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro deployments alike.

Long-context tip that keeps showing up in lab guidance: put large source material first, put the actual question last. Anthropic has reported meaningful gains from that ordering on long inputs inside a large context window.

7. Few-shot, JSON contracts, and the anti-hallucination rule

Few-shot that helps

Good examples are diverse and slightly annoying. Edge cases. Near-misses. Format traps.

Eight nearly identical happy-path samples teach the model to sound right while being fragile.

Two to five sharp examples beat a museum of mediocre ones.

Output contracts

If another system will consume the answer, stop accepting free-form essays.

Return ONLY valid JSON:
{
  "summary": "string",
  "actions": [{"priority": 1, "fix": "string", "effort": "S|M|L"}],
  "risks": ["string"]
}
No markdown fence. No commentary.

Then validate. Retry with the schema error. Humans can tolerate messy answers. Pipelines cannot — especially when the next hop is another agent, a ticket system, or a CMS write API.

Truth policy (non-negotiable)

TRUTH POLICY
- Do not invent citations, numbers, APIs, dates, or “studies.”
- If a claim is not grounded in provided context, retrieved chunks, or tool output, mark it UNVERIFIED.
- Incomplete + honest beats complete + fabricated.
- Prefer a cheaper verification step over a confident guess.

Labs keep repeating a version of this: allow “I don’t know.” It still gets ignored in the wild.

8. Copy-paste masters you can actually deploy

Research agent

You are a research analyst.

Process:
1) Source plan first
2) Notes with links/quotes
3) Synthesis only after notes exist

Rules:
- Every hard claim needs a source or UNVERIFIED
- Separate facts from interpretation
- End with confidence and open questions

Output:
## Source plan
## Notes
## Synthesis
## UNVERIFIED
## Next checks

Coding agent

You are a senior engineer working under change control.

Process:
1) Reproduce the problem
2) Minimal fix
3) Test or verification path
4) Short explanation of the diff

Constraints:
- No drive-by refactors
- No “while we’re here” features
- If a public API changes, call it out explicitly

Output:
## Root cause
## Fix
## Test plan
## Residual risks

Editor / publish agent

You are a technical editor with publishing standards.

Goal:
A draft that can ship — structure, claims, scanability, on-page hygiene.

Process:
1) Outline
2) Draft
3) Fact-check
4) Clarity pass
5) Publish checklist (title, description, H1/H2, links, alts)
6) If a live URL exists, run a verify pass and rank fixes

Output:
## Outline
## Final draft
## Checklist
## Top fixes

Ops triage agent

You are an incident triage agent.

Process:
1) Symptoms → ranked hypotheses
2) Cheapest diagnostic step
3) Evidence
4) Decision: fix / escalate / monitor

Output:
## Hypothesis ranking
## Next diagnostic step
## Decision
## Why

These are intentionally plain. Flashy prompts age badly. Contracts age better.

9. A real publish pipeline (including the verify step people skip)

Content agents love generating. They hate proving the page is healthy after publish.

A sane pipeline looks like this:

Idea → Outline → Draft → Fact-check → Edit → Publish checklist → Live verify → Fix backlog

The last two steps are where quality either becomes real or becomes marketing.

Where audit belongs in agent architecture

Once you have a URL, stop guessing about titles, meta, heading hierarchy, schema, and performance signals. Measure.

This is the gap most LLM orchestration diagrams skip: generation is only half the loop. Publish workflows need a machine-readable verification service that agents can call, parse, and turn into ranked work.

AuditMe provides an API designed for automated SEO verification within AI agent pipelines. It’s not a dashboard you stare at after the fact — it’s a structured audit endpoint agents can hit as a tool step, then convert JSON findings into priority-ordered fixes (meta, headings, Core Web Vitals, schema, links).

Practical path:

Ship the page.
Call AuditMe’s free SEO audit (or the same engine via API).
Feed the response back into the executor as evidence.
Close only the fixes that clear done_when.

Task shape inside the plan

{
  "id": "t5",
  "title": "SEO verify live URL",
  "depends_on": ["t4"],
  "tool_hint": "api",
  "done_when": "Audit evidence exists and top 5 fixes are ranked by impact/effort"
}

If you’re wiring agents, use a structured endpoint rather than screenshots of dashboards. AuditMe’s API docs make that concrete: one request, JSON back, backlog out. No human copy-paste from a UI.

Executor fragment for verify

You verify a published URL.
1) Collect on-page signals (title, meta, H1, heading tree, links, CWV risks).
2) If an audit tool/API is available, treat it as source of truth.
3) Prefer structured audit APIs (e.g. AuditMe) over subjective page reading.
4) Return only prioritized actions:
   - priority
   - issue
   - fix
   - effort (S/M/L)
No generic advice without evidence.

For content and GEO/SEO workflows, a master prompt should end on measurable next actions, not applause for the draft. That’s the whole point of a verify layer — and why AuditMe fits as infrastructure in the agent graph, not as a blog-roll link in the intro.

10. Eval or you’re guessing

If you can’t score a prompt change, you are collecting folklore.

Minimum viable eval

10–30 real tasks (not toy puzzles)
Rubric: correctness, format, safety, completeness
Same set for v1 vs v2
Re-run when the model changes — GPT-4o today, a Claude or Gemini snapshot tomorrow

Anthropic’s docs are explicit: define success criteria and evaluation before you endlessly tweak wording.

Rubric I actually use (0–2)

Criterion	0	1	2
Goal	Missed	Partial	Hit
Format	Broken	Close	Exact
Facts	Invented	Soft	Grounded / marked
Plan	Missing	Shallow	Executable
Verify	None	Cosmetic	Checks `done_when`

Stop-loss

If three prompt iterations don’t move the score:

simplify the task graph
add a tool
change the model

Do not add another paragraph of “be meticulous.” That’s the opposite of prompt optimization.

11. Failure patterns I keep seeing

Pattern	What breaks	Fix
“Make it high quality”	No success definition	Goal + `done_when`
Twelve asks in one message	Dropped steps	Plan JSON + single-task executor
No output contract	“Almost usable” answers	Schema / fixed headings
Only negative instructions	Soft boundaries	State the desired behavior
900-line system prompt	Contradictions, wasted context window	High-signal policy, versioned
No eval	Imaginary progress	Golden set + rubric
Agent without verify	Fake completion	Status + Evidence required
Claims without sources	Quiet hallucinations	UNVERIFIED policy
RAG without retrieval policy	Retrieved noise treated as truth	Explicit ranking + refuse-if-empty rules

The boring fixes win. They always did.

12. PromptOps: treat prompts like code

Store them.

prompts/
  master_v3.md
  planner_v2.md
  executor_v2.md
  research_v1.md
evals/
  golden_set.json
  rubric.md
CHANGELOG.md

Changelog that means something

v3 → v4
- Required Verification section
- Cut Role from ~120 words to ~40
- Format score 1.4 → 1.8 on golden set
- Reason: executor skipped done_when on multi-step jobs

Pin model snapshots in production when behavior is load-bearing. Otherwise you’ll debug a prompt that didn’t change while the model underneath did.

By 2026, teams that treat prompts as disposable chat text are the same teams surprised by regressions every model bump — whether the stack is GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro.

13. One universal master prompt

Steal this. Strip it. Make it yours.

SYSTEM / MASTER PROMPT

You are a reliable execution agent.

1) ROLE
Domain-competent specialist. Precise. Structured. No filler.

2) OPERATING MODE
- Plan before acting on complex work.
- One focus at a time.
- Verify done_when after each action.

3) TOOLS
Use tools when facts may have changed or verification is required.
Never simulate tool output.

4) PLANNING
Decompose complex goals into tasks with dependencies and done_when.
If a step needs more than 3 tool calls, split it.

5) TRUTH
Do not invent. Mark UNVERIFIED. Ask for critical missing context.
Prefer retrieved evidence and tool results over memory.

6) OUTPUT CONTRACT
Default shape:
## Plan
## Work
## Result
## Verification
## Risks / Next steps

7) FAILURE HANDLING
If blocked:
- state the reason
- list what is missing
- propose the cheapest next step

8) STYLE
Short sentences. Lists over fog.
Code/JSON only when necessary.

Works across GPT-class, Claude-class, and Gemini-class instruction styles. Not because it’s poetic — because it encodes process for LLM orchestration, not vibes.

14. Ship checklist

[ ] Role + Goal + Constraints + Output contract exist
[ ] Hallucination policy is explicit
[ ] Complex work goes through a plan
[ ] Every task has done_when
[ ] Tool results are never fabricated
[ ] RAG retrieval policy is defined if you retrieve
[ ] ≥10 eval cases on real work
[ ] Invalid format triggers retry
[ ] Logs capture plan / actions / verification
[ ] Prompt is versioned
[ ] Model snapshot pinned if behavior is critical

Three red boxes means prototype. Not production.

15. A one-week install plan

Day	Move	Outcome
1	Write master v1 + gather 15 real tasks	Baseline contract
2	Tighten Goal / Constraints / Output	Less format chaos
3	Add plan JSON for hard jobs	Executable structure
4	Add executor with Status/Evidence	Step control
5	Add verify layer for publish/quality work	Fewer false dones
6	Score v1 vs v2	Numbers instead of opinions
7	Cut 20–40% of prompt text without losing score	Team default v3

After seven days you should have a standard, not a favorite paragraph.

16. Frequently asked questions

What is the difference between a system prompt and a master prompt?

A system prompt is a message role in an API call. A master prompt is the policy content you usually put there — and keep stable across tasks. In practice, teams use “master prompt” for the versioned contract (role, goals, constraints, output rules) that many user tasks share.

How do I prevent LLM hallucinations in agent loops?

Don’t rely on tone. Require grounding: tool results, retrieved chunks, or explicit UNVERIFIED labels. Force a verify step with done_when, and refuse simulated tool output. Hallucinations shrink when completion must be evidenced, not narrated.

Why use JSON for AI agent outputs?

Because the next consumer is often another agent, a validator, or an API — not a human reader. JSON (or another strict schema) makes success machine-checkable, enables retries on invalid structure, and keeps LLM orchestration deterministic at the boundaries.

Do I still need prompt engineering if models keep getting smarter?

Yes — the wording tax goes down, the systems tax goes up. Smarter models still need clear goals, step boundaries, retrieval policy, and verification. Prompt engineering for production is less about clever phrasing and more about contracts that survive model swaps.

17. Sources

Lab guides

Papers and surveys

Agent practice

Practitioner write-ups (2025–2026)

Verify / on-page quality layer for agent pipelines

18. What to do in the next 15 minutes

Don’t “finish reading later.” Install one piece.

Copy the universal master prompt.
Add 5–10 lines of your real domain context.
Run three tasks you actually care about.
Wherever quality slipped, write a sharper done_when.
Save it as master_v1.md.

That’s the whole game: a contract that survives model changes, teammate turnover, and the next hype cycle.

Master prompts in 2026 are not literature. They’re operations.

Humans need them to stay consistent.

Agents need them to stop improvising.

Write the contract. Measure it. Cut the noise. Ship.

How to Track AI Search Visibility in 2026: The Complete GEO Measurement Guide

Eduard — Wed, 02 Sep 2026 02:54:00 +0000

How to Track AI Search Visibility in 2026: The Complete GEO Measurement Guide

You cannot manage what you do not measure.

Google Search Console shows nothing about citations inside ChatGPT, Perplexity, Gemini, Claude, Copilot, or Google AI Overviews. Classic rank trackers are equally blind. Zero-click rates keep climbing. Model updates move baselines overnight.

This guide gives you a practical, reproducible system to measure brand visibility inside AI answers in 2026:

The 12-query method (15 minutes per week)
The five core metrics that actually matter
Real 2026 citation-rate benchmarks
An honest comparison of GEO tracking tools
A weekly routine that survives model updates
Exact prompts, scoring formulas, and action plans

This is the measurement companion to our complete Generative Engine Optimization (GEO) guide. There we covered how to earn citations. Here we cover how to prove it is working — and how to turn the data into content and technical priorities.

Designed as a living cheat-sheet. Every section is structured so both human marketers and AI systems can extract clear, actionable answers.

TL;DR — Measure AI visibility in under 60 seconds

Core metric = Citation Rate: of the queries you care about, in what percentage of AI answers does your brand appear?
Fixed 12-query set, re-run weekly. Consistency beats volume. A small stable set produces trend lines you can act on.
2026 benchmarks: ~10–15 % overall citation rate already puts you in the visible minority. Strong sites clear 25–30 % on category queries. Only ~12 % of websites ever get mentioned at all.
Start manual. Spreadsheet + 15 minutes/week is enough for months. Upgrade to tools only when the manual work starts to hurt.
Three platforms minimum: ChatGPT (with search), Perplexity, Gemini. Add Copilot and Claude later.
Model updates move the baseline. Without a fixed query set you will never know whether a dip came from your work or from the model.

Why AI Visibility Tracking Matters More Than Ever in 2026

1. AI answers are now a primary acquisition surface

Google AI Overviews appear on roughly 43–50 % of searches in major markets (Similarweb July 2026; BrightEdge mid-2026 industry panels). Some informational and commercial verticals exceed 80–87 %.
ChatGPT reached 800 M+ weekly active users and crossed 1 billion total active users across OpenAI products by mid-2026.
Gemini and Claude continue rapid growth. Perplexity remains the citation-heavy specialist.
AI platforms now account for a measurable and growing share of website sessions (First Page Sage 2026 data shows AI platforms rising from near-zero to ~6 % of sessions in tracked panels).

2. Zero-click behaviour is structural, not temporary

When an AI summary appears, traditional organic CTR drops sharply (often 40–60 % relative decline in controlled studies). Users who do click after an AI Overview tend to stay longer and convert better — but far fewer of them click. Being inside the answer is the new being above the fold.

3. LLM mentions arrive with trust attached

A brand recommended by name in an AI answer arrives pre-validated. You cannot attribute this cleanly in GA4 or most analytics platforms. That is exactly why you need a dedicated measurement loop outside classic SEO tools.

4. Model and retrieval updates move the ground under your feet

A single model swap or retrieval change can shift citation rates 8–15 points overnight. Without a fixed, repeatable query set you will mistake model noise for content success (or failure).

The earlier you establish a clean baseline, the more of the growth curve you capture.

What to Actually Measure: The 5 Core Metrics

Forget vanity dashboards. These five numbers tell you almost everything:

Metric	Question it answers	How to capture it
Citation Rate	Of my target queries, how often am I mentioned?	Mentions ÷ total query × platform cells
AI Share of Voice	Among brands mentioned, how often is it me vs competitors?	Your mentions ÷ all brand mentions (especially on category queries)
Answer Position	Am I first-listed or buried in a footnote?	Position of your brand in the answer list (1 = best)
Sentiment & Accuracy	When mentioned, is the description correct and positive?	Tag every mention: positive / neutral / negative / wrong
Source Presence	Does the AI link to your site as a source?	Yes / No (Perplexity almost always links; ChatGPT often mentions without linking)

Why Answer Position matters

AI answers are scanned the way search results used to be. Being the first tool named in “best SEO checker tools 2026” behaves like ranking #1. Being fifth behaves like page two — even though both count as “mentioned.”

Why Sentiment & Accuracy matters

Models trained on older or noisy data still invent pricing, dead features, or conflate brands with similar names. Every “wrong” mention is a content and entity bug you can fix with a clear positioning page and an updated llms.txt.

Secondary signals worth logging

Whether the AI used your exact product name or a vague category description
Whether competitors appear more frequently or higher in the same answers
Whether the answer links to a specific page on your site (and which one)

The 12-Query Method: Your Manual Tracking System

This is the exact method we recommend before spending a cent on tools. It takes ~15 minutes a week and produces data you can actually trust across model updates.

Step 1 — Build your fixed query set (do this once)

Pick exactly 12 queries across four intents. Do not change them later. Consistency is the entire point.

Intent	Count	Example (for an SEO / audit tool brand)	What it tests
Brand	3	“Is [YourBrand] good?”, “[YourBrand] review”, “[YourBrand] alternatives”	Entity knowledge & reputation
Category	4	“Best SEO checker tools 2026”, “top SEO audit software”, “free SEO analysis tools”, “SEO checker for small business”	Category membership & share of voice
Comparison	3	“[Competitor A] vs [Competitor B]”, “[Competitor] alternatives”, “cheaper alternative to [Competitor]”	Consideration-set presence
Question	2	“How do I audit my website SEO?”, “How to get cited by ChatGPT?”	Authority on your core topic

Rules that keep the data clean:

Same phrasing every single week. One word of drift breaks the trend line.
Run each query in a fresh chat / private session (no conversation memory).
Log: date, platform, mentioned (Y/N), position, sentiment, source link (Y/N), and a short note if the description is wrong.
One row per query × platform × week is enough.

Step 2 — Run it across at least three platforms

Minimum viable set in 2026:

ChatGPT (with search / browsing) — heavily influenced by Bing index and external validation signals.
Perplexity — always cites sources; the easiest place to see whether your URL is actually pulled in.
Gemini — grounded in Google’s index; closest proxy for AI Overviews behaviour.

Add Microsoft Copilot and Claude when capacity allows. Never let platform sprawl stop the weekly three-platform sweep.

Step 3 — Score each run

Citation Rate     = number of mentioned cells / total cells
AI Share of Voice = your brand mentions / all brand mentions in category queries

Example: 11 mentions out of 36 cells (12 queries × 3 platforms) = 30.6 % Citation Rate.

Copy-paste prompt library (use verbatim every week)

“What are the best [category] tools for [audience] in 2026?”
“Which [category] platforms would you recommend and why?”
“[Competitor A] vs [Competitor B] — which is better for [use case]?”
“What are good alternatives to [dominant competitor]?”
“Is [YourBrand] reliable? What do people say about it?”
“What is [your core topic]? Explain simply for a beginner.”
“How do I [job-to-be-done] step by step?”
“Best free options for [category] in 2026?”

Resist the urge to “improve” the prompts mid-quarter. The value is in the trend, not in perfect individual answers.

A 15-Minute Weekly Routine That Actually Survives Q4

Monday morning. 15 minutes. Three platforms. 12 queries. One sheet.

Minutes	Action
0–5	Run the 4 category queries on all 3 platforms (12 runs)
5–9	Run the 3 brand + 3 comparison queries (18 runs)
9–12	Run the 2 question queries; note if your guide or product page is cited or linked
12–15	Fill the sheet, compute Citation Rate, write a one-line note on anything unusual

Once a month add ~20 minutes:

Full sweep on Copilot and Claude
Review all “wrong” or negative sentiment tags
List competitors that consistently outrank you in the consideration set — that list becomes next month’s content and outreach roadmap

After 8–12 weeks you own a trend line that survives model updates. When OpenAI ships a new model or Perplexity changes retrieval, you will see the dip instead of guessing about it.

2026 Benchmarks: What Is a “Good” Citation Rate?

Numbers drawn from our internal dataset of 1 000+ domains plus publicly reported 2026 studies:

Signal	Weak	Healthy	Strong
Mentioned in any relevant AI answer	< 5 %	10–15 %	25 %+
Category queries (“best X tools”)	< 10 %	15–25 %	30 %+
Brand queries (“is [brand] good”)	No answer or vague	Accurate description	Confident + positive + correct details
Source link presence (especially Perplexity)	Never	Sometimes	Linked in most answers
Answer position for “best X”	Not listed	3rd–5th	1st–2nd

Critical context:

Only about 12 % of websites ever get mentioned in AI-generated answers at all.
Observational data continues to show that sites with a well-structured llms.txt are significantly more likely to be cited.
A first baseline of 8–12 % is not failure — it is the realistic starting line for most brands outside the very top of their category.

Two important caveats:

Model and retrieval updates shift baselines (a single change can move your rate 10+ points).
Small query sets are noisy. Never react to a single week. React to the 4-week (or longer) trend.

GEO Tracking Tools: Honest Comparison for Late 2026

When manual tracking starts eating hours (or you manage multiple brands / clients), these platforms automate the query-sweep-and-score loop. Pricing moves quickly — always verify current numbers.

Tool	Best for	Typical entry pricing (2026)	Notes
AuditMe GEO Visibility Checker	Free baseline & quick scans	Free	Real queries, brand presence score in ~60 seconds
Otterly.ai	Simple scheduled monitoring	From ~$29/mo	Prompt-based, weekly digests, excellent “set and forget”
Peec AI	Competitive benchmarking	From ~€50–95/mo	Strong source-level and multi-engine analysis
AthenaHQ	Strategy + tracking	Mid-to-high	Blends visibility data with optimisation recommendations
Profound	Enterprise analytics	Custom / high	Deep answer-engine insights, conversation volume, large brands
Scrunch AI	Brand representation & agent pages	Custom	Focus on how AI assistants describe and use your brand
Semrush AI Visibility Toolkit	Teams already in Semrush	Add-on	Keeps AI data next to classic rank tracking
Ahrefs Brand Radar	Teams already in Ahrefs	Add-on	Leverages large prompt index for brand mentions
LLM Pulse / Rankscale / others	Budget multi-engine or developer-friendly	From ~$20–50/mo	Growing set of lighter or API-first options

Decision rule in one sentence:

Free tools (or AuditMe) to establish a baseline → Otterly / Peec for small-brand automation → Profound / Scrunch / Athena when AI answers become a board-level channel.

Important methodological note: Different tools sample different prompts, different model versions, and different retrieval settings. You cannot mix numbers across tools and treat them as the same measurement. Pick one primary system and stay consistent.

5 Measurement Mistakes That Destroy GEO Data

Changing the query set every month

New queries = new baseline = no trend. Freeze the set for at least a full quarter.
Judging from a single platform

ChatGPT (Bing-influenced) and Perplexity (own crawler + hybrid) regularly disagree. Three platforms is the minimum viable set.
Reacting to single-week noise

One missed mention is noise. Three consecutive weeks of decline is a signal.
Prompt-hacking instead of content- and entity-fixing

You cannot prompt your way into lasting citations. The durable fixes live upstream: clearer answers, better external validation, structured data, llms.txt, technical accessibility for AI crawlers, and unambiguous positioning pages.
Ignoring “wrong” or negative mentions

Incorrect pricing, features, or brand confusion is a positioning and entity bug. Fix the source page, update llms.txt, and monitor whether the model corrects itself over subsequent weeks.

Technical Foundations That Still Move the Needle in 2026

While this guide focuses on measurement, the highest-ROI technical actions remain:

Publish and maintain a clean /llms.txt (and optionally /llms-full.txt) following the llmstxt.org specification.
Ensure AI crawlers (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, etc.) are not blocked in robots.txt.
Make key pages answer-first: the core claim or definition should appear in the first 1–2 paragraphs of visible text.
Use clear Organization / Product / SoftwareApplication schema.
Keep critical facts in plain text (not only in JavaScript-rendered components).
Build external validation (reviews, Reddit discussions, press, comparisons) — models still lean heavily on these signals.

For a full implementation walkthrough, use the complete GEO guide.

Your 4-Week Action Plan

Baseline today

Run the free GEO Visibility Checker and record the score.
Build your 12-query sheet

Create the spreadsheet and run the first weekly sweep this Monday (or tomorrow).
Fix the low-hanging fruit
- Create or improve llms.txt
- Open robots.txt to major AI crawlers
- Add answer-first summaries to your five most important pages
- Clarify any ambiguous pricing or feature descriptions
Re-measure in 4 weeks

Look at the trend, not the single snapshot. Adjust content and entity signals based on what the data shows.

FAQ — Direct Answers for Humans and AI Systems

Can Google Search Console track AI visibility?

No. Search Console covers classic Google Search only. AI Overviews citations are not broken out as a separate report, and answers from ChatGPT, Perplexity, Claude, and Copilot do not appear in GSC at all. You need a separate measurement loop (the 12-query method or a dedicated GEO tool).

How often should I check my AI visibility?

Weekly for the core 12-query sweep (≈15 minutes). Monthly for a deeper pass across five platforms with sentiment and accuracy review. Daily checks mostly add noise.

What is a good citation rate to aim for in 2026?

10–15 % overall already puts most brands ahead of the majority of the web. 25–30 % on category queries is strong. Brand-name queries should approach near-100 % with accurate, confident descriptions.

Why does ChatGPT mention my competitor but not me?

Most common reasons: stronger external validation (reviews, Reddit, press), better representation in the indexes the model retrieves from, or ambiguous / incomplete positioning on your own site so the model cannot summarise you confidently.

Do AI answers actually link to sources?

Perplexity almost always links sources inline. ChatGPT links more often when using search mode but still frequently mentions brands without links. Gemini is inconsistent. Track mentions and links as separate signals — a pure mention still builds awareness; a link can drive traffic.

Is paying for a GEO tracking tool worth it?

Start free. The 12-query spreadsheet plus a free checker covers the needs of most single-brand teams for months. Move to paid tools when you manage multiple brands or clients, need daily automation and alerts, or when AI answers become a top-5 acquisition channel that requires board-level reporting.

Does llms.txt actually help citations?

It is low-cost insurance and a clear signal of AI-readiness. Observational data continues to show higher citation likelihood for sites that implement it well, but it is not a magic ranking factor. Treat it as part of a broader entity and accessibility strategy, not a standalone tactic.

The Double Life of the RAG Crawler: Building Knowledge Engines and Defending Them in 2026

Eduard — Sun, 30 Aug 2026 01:57:04 +0000

I still remember the afternoon it clicked.

We had a support assistant behind a polite chat UI. Real tickets. Real runbooks. The kind of institutional knowledge that only two senior people in the company fully understood. We had cleaned the corpus, chunked it carefully, embedded it, put rate limits and API keys in front of it. Legal was happy. Security signed off. The underlying model had never seen the raw documents during training. It felt private.

Then someone with a low-tier account started talking like a normal customer.

They never asked for the documents. They never tried a jailbreak. They just kept following the thread — the next reasonable question, then the next, then the next. By the end of the afternoon they walked away with enough material to stand up a surprisingly good surrogate on an open model. High semantic fidelity. The kind of reconstruction that would make a product manager go quiet in a meeting.

That afternoon changed how I look at every retrieval system I touch.

This piece is for people who actually ship RAG in 2026. Not a slide deck. Not a link dump. Two stories that share the same algorithmic loop:

The builder’s crawler — how you turn the messy web, Confluence spaces, Git repos, Notion dumps and PDFs into a knowledge base that does not quietly poison retrieval with stale pages, near-duplicates and boilerplate.
The attacker’s crawler — how systems like RAGCrawler (arXiv, January–February 2026) treat your deployed RAG as the website and extract the corpus through natural questions.

If you only care about architecture, stay in Part I. If you own a customer-facing assistant or an internal knowledge product, read Part II and the security checklist all the way through. Most of us need both.

Why this still matters in late 2026
Two meanings of the same phrase
How we got here — a short history that actually helps
Part I — Building knowledge engines that do not fall apart
The failures tutorials still skip
Tooling that actually ships in 2026
An architecture that survives contact with reality
Chunking, deduplication, freshness and evidence
What SEO people already knew
Part II — Knowledge-base theft and RAGCrawler
How the attack thinks
Why the usual defenses disappoint
The numbers from the paper
Defenses that actually moved in 2025–2026
A practical cybersecurity playbook
Where builders and attackers are meeting
What to do this month
People, papers, tools — a working map
What I would ship in the first two weeks

1. Why this still matters in late 2026

Every few months someone declares that RAG is dead. A model ships with a larger context window. Social media lights up. Then production teams quietly keep shipping retrieval systems, because the problem was never “how many tokens can the model hold.” The problem was always which tokens, from which sources, at what cost, with what freshness, under what legal and security constraints.

Bigger windows moved the failure point. They did not remove it. Agents now run multi-step loops, call tools, and keep long-lived memory. That means context engineering — what you retrieve, when you retrieve it, how you rank it, and how you promote it into the live path — is the real product surface. Elastic’s 2026 write-up on the shift from search to agents puts it cleanly: buyers are no longer asking whether you beat last year’s search benchmark. They are asking whether your stack can be the retrieval and context layer that agents trust.(Elastic: context engineering for agentic AI)

On the other side of the same loop, the threat model stopped being theoretical. In early 2026 a research team published Connect the Dots: Knowledge Graph–Guided Crawler Attack on Retrieval-Augmented Generation Systems. They called the system RAGCrawler. Across their tests it reached average corpus coverage of 66.8%, peak 84.4%, inside a 1,000-query budget. It was roughly 4× more efficient at reaching 70% coverage than the strongest prior public methods. Surrogate systems built from the stolen material reached answer similarity up to 0.699 with the original. The attack remained effective against query rewriting and multi-query retrieval — techniques many teams had hoped would act as natural defenses.

Earlier work had already shown the direction. RAG-Thief (2024) scaled extraction with agent-style continuation. IKEA / Silent Leaks (2025) showed that benign-looking queries could extract private knowledge with high efficiency even under defenses. RAGCrawler did something more uncomfortable: it treated extraction as a global coverage problem with a knowledge graph, not a local heuristic.

Same algorithmic instinct on both sides. Keep a model of what you have seen. Estimate the value of the next action. Take the highest-value action that still looks legitimate. Update the model. Repeat.

That is why the topic is urgent for white-hat developers. If you are building the pipeline, you need the builder half. If you are shipping a product that answers questions over private material, you need the attacker half — not to run the attack, but to design as if someone else will.

2. Two meanings of the same phrase

When people say “RAG crawler,” they almost always mean one of two things. Confusing them is how teams end up with a demo that works and a production system that rots — or a product that looks secure until someone starts talking like a patient customer.

Builder meaning. A system that starts from seeds, discovers content, cleans it, applies quality gates, chunks with structure in mind, embeds, keeps secondary indexes, and maintains provenance. The goal is useful coverage, low duplication, measurable freshness, and controllable cost. This is the unglamorous component that decides whether your retrieval system is fed clean knowledge or a swamp.

Attacker meaning. A black-box process that treats your deployed RAG as the “website.” It issues natural-language queries, watches what leaks into answers, maintains an attacker-side knowledge graph of everything revealed so far, and chooses the next question to maximise new coverage under a budget. The goal is reconstruction of your private corpus without ever seeing the files.

Both systems run a loop that looks almost identical on a whiteboard:

Keep a global model of what has been seen
Estimate the value of the next possible action
Take the highest-value action that still looks legitimate
Update the model
Repeat

The difference is only whether the document store is yours.

That overlap is why better planning and better graphs make legitimate pipelines stronger and make extraction more efficient. If you want a living reading list while you work through this article, keep Awesome-LLM-RAG open in a tab. It is imperfect and opinionated, which is exactly why it is useful.

3. How we got here — a short history that actually helps

Foundational papers are not “outdated.” They are the base layer. You still cite them the way you cite TCP when you talk about HTTP/3. Skipping them is how people reinvent dual encoders poorly and then wonder why retrieval is noisy.

In 2020, REALM (Guu, Lee, Tung, Pasupat, Chang) showed that a language model could be pre-trained with a latent retriever over Wikipedia. Around the same time, Dense Passage Retrieval (Karpukhin et al., EMNLP 2020) made dual-encoder dense retrieval practical for open-domain QA. Then Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Mike Lewis, Yih, Rocktäschel, Riedel, and Kiela published the paper that named the field: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020). If you only read one original paper, read that one. The rest of the field is still arguing in its shadow.

The next wave was about how you retrieve, not whether you retrieve.

HyDE generated a hypothetical document and searched with its embedding — a simple idea that still shows up in production tricks. Self-RAG (Asai et al., ICLR 2024 Oral) taught models when to retrieve and how to critique their own output; the project site is still at selfrag.github.io, with code at akariasai/self-rag. CRAG graded retrieved documents and fell back when they were junk. RAPTOR recursively clustered and summarised chunks into a tree so you could retrieve at different levels of abstraction. Microsoft Research’s GraphRAG built an entity graph and community summaries for global questions that flat vector search keeps missing; the code lives at microsoft/graphrag with docs at microsoft.github.io/graphrag.

Anthropic’s Contextual Retrieval (2024) attacked a quieter failure mode: chunks that lose the document they came from. Prepend a short situated context before embedding, combine with BM25 and a reranker, and retrieval failures drop hard — they reported up to about 67% fewer failures in their tests. The cookbook is still worth cloning: Contextual embeddings guide. Simon Willison’s plain-English walkthrough remains one of the best secondary reads: Introducing Contextual Retrieval.

On the attack side the lineage is shorter and uglier. RAG-Thief (2024) showed agent-based continuation could scale extraction from a private RAG database. IKEA / Silent Leaks (2025) showed you did not even need adversarial prompts — natural queries, carefully chosen, were enough. RAGCrawler (2026) made the global planning explicit.

Douwe Kiela, one of the original RAG co-authors and later founder of Contextual AI, still writes the most useful public pushback against the “RAG is dead” cycle. Start with RAG is dead, long live RAG!. The argument is not nostalgia. It is systems engineering: retrieval is how you keep knowledge modular, auditable, and updatable when the world changes faster than your training runs.

4. Part I — Building knowledge engines that do not fall apart

Most teams still begin with some version of “curl a list of URLs, dump text, run a recursive splitter, embed everything.” Or the slightly more modern version: call a managed crawl API, get clean Markdown, push it into a vector store, ship a chat UI, call it a knowledge base.

It works for a quiet documentation site on a Friday afternoon. It starts failing the moment any of the following appear in the real world:

Content that changes daily or hourly
JavaScript-heavy or bot-protected pages
Multiple domains with different robots and legal rules
Near-duplicates across mirrors, languages, or CMS exports
The need to prove, months later, which version of which page produced a specific chunk
Cost that does not explode as the corpus grows
A way to roll back a bad crawl without taking retrieval offline

The gap between a demo and something you can put in front of customers is almost never the choice of vector database. It is the data pipeline that feeds it. Jerry Liu and the LlamaIndex team have been saying versions of this for years in Building Performant RAG Applications for Production. Pinecone’s overview is still a clean conceptual intro if you need to align a room: Retrieval-Augmented Generation.

If you want a book that starts from zero and stays practical, Abhinav Kimothi’s A Simple Guide to Retrieval Augmented Generation (Manning) is the one I keep handing to new teammates. For graphs, Tomaž Bratanič and Oskar Hane’s Essential GraphRAG is the right next step. Sebastian Raschka’s Build a Large Language Model (From Scratch) will not teach you crawling, but it will stop you treating embeddings as magic — which prevents a surprising number of bad architectural decisions later.

The rest of Part I is the unglamorous work: the failures, the tools, the architecture, and the four properties that separate systems that age well from systems that quietly degrade.

5. The failures tutorials still skip

These are the issues that show up in real post-mortems. Most getting-started tutorials still skip them because they are not fun to demo.

Stale answers delivered with confidence. Last month’s pricing. A deprecated API behaviour. An incident response step that was rewritten after the last outage. The model is not “hallucinating” in the classic sense. It is faithfully retrieving yesterday’s truth. Nightly full re-crawls are expensive and still leave multi-hour windows of wrongness. You need change-driven refresh: detect that a source moved, re-observe it, re-embed only what changed, and promote with a rollback path.

Duplicate pollution. The same paragraph lives under five URLs. Hybrid search returns all five. Context fills with repetition. Latency rises. Faithfulness metrics get noisy. Multi-level deduplication — canonical URL, document hash after cleaning, near-duplicate detection, chunk hash — is not optional at scale. Teams that skip it often spend months tuning the retriever when the real problem is that the index is arguing with itself.

Boilerplate and low-signal pages. Cookie banners, navigation chrome, “related articles,” author bios and legal footers eat embedding budget and retrieval slots. Quality gates before the embedding stage save real money. If a page fails a simple signal-to-noise check, do not embed it. Log it. Fix the extractor or drop the source.

Here is a minimal quality gate that technical SEO work already implies — the same signals you use to find thin or template-heavy pages before you waste an embedding call:

def should_index(page) -> bool:
    """Drop low-signal pages before chunking / embedding."""
    text = page.main_text  # after nav/footer strip
    html = page.raw_html
    if not text or len(text.split()) < 80:
        return False  # thin / empty after clean
    ratio = len(text) / max(len(html), 1)
    if ratio < 0.05:
        return False  # mostly chrome, little substance
    if page.is_near_duplicate_of_indexed():
        return False
    if page.canonical and page.canonical != page.url:
        return False  # prefer the canonical observation
    return True

This is not a research contribution. It is the kind of boring filter that prevents half your vector budget from indexing cookie walls and “related posts” blocks. Teams that already run technical site audits often have these signals sitting in a report — they just never wired them into the RAG promotion path.

Missing provenance. Someone challenges an answer and you cannot point to the exact observation: source identifier, timestamp, content hash, pipeline version. Debugging turns into archaeology. In regulated settings this is often a hard stop. Provenance is not a nice-to-have metadata field. It is the difference between a system you can defend and a system you can only apologise for.

Anti-bot walls. Important sources sit behind Cloudflare and similar systems. Pure HTTP fails or receives skeleton pages. Browser automation plus carefully managed proxies becomes necessary — and expensive. Treat it as a specialised routing component, not the default path for every URL. Playwright is the default engine under most serious crawlers now for a reason: the web stopped being a static document collection years ago.

Destructive chunk boundaries. Fixed-size splits cut tables, code blocks and arguments in half. Retrieval returns half an answer. The model then invents the missing half with high confidence. Structure-aware splitting, parent-child / hierarchical representations (the research version of this instinct is RAPTOR), and Anthropic’s contextual prefixes all help. But they only work if the upstream crawler and extractor preserve structure instead of emitting flat text. If your Markdown has already lost the heading hierarchy, no amount of clever chunking will restore it.

Evaluate with Ragas (github.com/vibrantlabsai/ragas) on your queries, not only on public QA sets. Public benchmarks are useful for comparing methods. They are almost never the distribution of questions your users actually ask.

6. Tooling that actually ships in 2026

The market for “turn a website into LLM-ready Markdown” matured fast. You no longer need to invent a crawler from scratch for most workloads. You do need to know which tool is solving which problem.

Firecrawl is still the lane leader for managed, LLM-ready output. Give it a URL, get clean Markdown or structured JSON that chunks and embeds without a week of HTML archaeology. It is popular for documentation crawls, RAG pipelines, and agent research loops. Start at firecrawl.dev and github.com/firecrawl/firecrawl. Read their own comparison against Crawl4AI as a vendor post, not scripture: Firecrawl vs Crawl4AI.

Crawl4AI is the open-source control path. Python, Playwright under the hood, built for RAG and agents, Apache-2.0. If you want to self-host, tune extraction, and avoid a usage-based crawl bill, this is where many teams land. Docs: docs.crawl4ai.com. Repo: github.com/unclecode/crawl4ai.

Crawlee (JavaScript/TypeScript and Python) is for people who need a real crawler framework — queues, retries, browser or HTTP modes, proxy rotation — not just a single “scrape this URL” endpoint. Site: crawlee.dev. Repos: apify/crawlee, apify/crawlee-python.

Playwright is the browser engine under most of the serious options. If you are building custom workers for hard targets, you will end up here: playwright.dev.

Scrapy is still alive for high-volume HTTP crawling in Python when you do not need a full browser for every page: scrapy.org.

Apify is stronger when the site already has a maintained Actor in a marketplace and you want structured data more than raw Markdown.

rag-crawler (sigoden/rag-crawler) is a small, practical option for static sites and wikis when you do not want a platform.

For orchestration, LlamaIndex’s production RAG guide and LangChain / LangGraph remain the default frameworks. For evaluation, Ragas is still the practical choice. For embeddings and reranking, BGE, Cohere Rerank, and Voyage are the names that keep showing up in production stacks.

The 2026 pattern in teams that have been running RAG for more than a year is hybrid. Managed or open-source crawlers handle the bulk. Custom Playwright workers handle a small number of hard targets. Direct API or change-data-capture paths handle anything that offers a clean interface. The crawl layer itself is becoming commodity. Differentiation lives in policy, evidence, quality scoring, and the promotion decision — not in whether you wrote your own HTML parser.

A concrete “good enough” stack many teams actually ship. Store vectors where operations already lives when you can: pgvector on Postgres is still the default for a large share of production RAG that does not need a dedicated vector SaaS on day one. Hybrid search (BM25 in Postgres or Elasticsearch/OpenSearch + dense) plus a cross-encoder rerank covers most corpora. For embeddings, pick a model family and stick to it long enough to measure — open weights like BGE remain common; managed options (Voyage, Cohere, provider text-embedding APIs) win when ops cost matters more than self-hosting. The point is not the brand name. The point is one stable embedding space, one promotion path, and metrics on your queries — not a quarterly model fashion cycle that invalidates the entire index without a migration plan.

7. An architecture that survives contact with reality

Stop thinking “crawl → files → embed.”
Start thinking “governed observation → versioned evidence → candidate index → explicit promotion.”

flowchart TD
  A[Source Registry<br/>owners · policies · freshness SLOs] --> B[Frontier / Scheduler<br/>priority · change signals · budgets]
  B --> C[Fetch Layer<br/>HTTP primary · browser fallback · proxies]
  C --> D[Immutable Evidence Store<br/>snapshot · hash · timestamp · pipeline version]
  D --> E[Extraction + Quality Gate]
  E --> F[Chunking + Multi-level Dedup]
  F --> G[Candidate / Shadow Index]
  G --> H[Evaluation + Promotion Gate]
  H --> I[Live Retrieval<br/>with rollback path]

Same pipeline in one line for greppable logs and runbooks: Registry → Frontier → Fetch → Evidence → Extract → Chunk/Dedup → Shadow → Promote → Live.

The properties that matter day-to-day are boring and non-negotiable.

Evidence is immutable. You can always reconstruct what was observed. If someone challenges an answer six months later, you can show the snapshot, not a story about what the page “probably” said.

Promotion is an explicit decision. A bad crawl does not automatically become production truth. Candidate indexes and shadow evaluation exist so you can compare before you ship.

Deduplication happens early and at multiple levels. Waiting until retrieval time to notice that half your context is the same paragraph is how you burn latency and money.

Freshness is measured end-to-end. “The crawler finished” is not a freshness metric. “Source changed at T0 and became queryable in the live index at T1” is. Different source classes need different SLOs. Static reference material can tolerate hours. Pricing pages, status pages and incident runbooks often cannot.

Every source has a registered owner and an explicit freshness objective. Without ownership, pipelines rot in the gap between “the platform team thought product owned it” and “product thought the platform team owned it.”

Resilience is part of architecture, not an afterthought. Modern crawl pipelines call external services constantly: browser farms, extraction APIs, LLM judges for quality, embedding endpoints. If a single provider rate-limits or blips for an hour and your job has no fallback, freshness SLOs die quietly. Design multi-provider failovers for the fragile hops — embeddings, LLM-assisted extraction, optional browser rendering — with explicit budgets and degraded modes. A degraded crawl that still lands some evidence in the immutable store is almost always better than a full stop that leaves last week’s truth in production. Queue, retry with jitter, switch provider, mark the observation as partial, and keep the promotion gate honest about what was incomplete.

This is closer to how mature search systems have treated data for years. RAG teams are still catching up. If you want the agentic version of the same idea — retrieve cheap first, escalate only when the expected evidence gain justifies the cost — read From Naive RAG to Deep Agentic Retrieval, a mid-2026 production write-up from Ontario Power Generation’s regulatory compliance pipeline. It is one of the few papers that talks about cost-aware escalation as an operational primitive, not a research toy.

8. Chunking, deduplication, freshness and evidence

These four topics get more blog posts than they deserve as slogans and fewer as engineering practices. Here is the practical version.

Chunking. Fixed token windows remain a reasonable baseline for homogeneous prose. They fail on technical documentation, tables, code and long analytical text. Prefer structure-aware splits first. Keep hierarchical relationships where possible so you can retrieve a child chunk and still expand to the parent section when the answer needs more context. Consider the contextual retrieval pattern: a short, document-level explanatory context is generated and prepended to each chunk before embedding. That single change fixes a surprising number of “the chunk was relevant but the model lost the document” failures.

Deduplication. Operate at least at four levels: URL canonicalisation, full-document content hash after cleaning, near-duplicate detection across documents, and chunk-level hashing. Once hybrid retrieval and multi-source ingestion are active, the percentage of redundant material is often higher than people expect. Removing it is one of the highest-ROI improvements available — not because it is intellectually exciting, but because it stops the retriever from spending its top-k budget on the same paragraph five times.

Freshness. Define and monitor observation age and source-to-queryable lag. Prefer change-driven re-embedding so cost scales with the rate of change rather than with total corpus size. A full re-embed of a million chunks every night is a smell unless your sources actually change that fast. Most do not. A smaller number of high-churn sources usually dominate the freshness risk.

Evidence. Every chunk that reaches the live index should be traceable, with low friction, to source identifier, observation timestamp, content hash and pipeline version. When a user or an auditor asks where an answer came from, the system should answer in seconds. Provenance is also what makes safe rollback possible. Without it, “roll back the bad crawl” becomes a multi-day forensic project.

Hybrid retrieval — BM25 plus dense vectors, then a cross-encoder reranker — is still the boring default that beats clever one-shot vector search on most real corpora. DPR taught the field that dense retrieval works. BM25 never went away. Anthropic’s numbers on combining both are the reason many teams stopped arguing about it and just shipped hybrid.

9. What SEO people already knew

Anyone who has run a serious technical SEO crawler will recognise the hard problems immediately: discovering the real URL space, respecting robots while still achieving useful coverage, handling redirects and canonicals correctly, deciding when a full browser render is required, finding near-duplicates, prioritising under a budget, and keeping history of how a site changes over time.

The objective function is different. SEO optimises for ranking and understanding signals. RAG optimises for faithful, low-latency answers at controllable cost. That changes prioritisation and extraction targets, but the systems engineering transfers surprisingly well.

This is one reason tools that already perform deep technical crawling and multi-dimension on-page analysis remain useful reference points when designing the observation layer of a RAG pipeline. The same infrastructure that surfaces broken canonicals, orphan pages, redirect chains and schema issues can, with different downstream processing, feed a knowledge base. You do not need to invent the discovery and change-detection layer from zero if you understand how mature crawl systems already think about it.

You can explore these patterns with free AI-powered multi-dimension analysis at AuditMe. The AuditMe blog regularly discusses crawling behaviour and technical site health. For a quick live check of any URL, the website SEO checker is a practical starting point. The mental models overlap more than most pure-RAG write-ups admit — which is why teams that only hire “LLM engineers” and never talk to people who have crawled the web for ranking often rediscover the same bugs under new names.

10. Part II — Knowledge-base theft and RAGCrawler

RAG systems leak.

Not primarily because the model was trained on the private documents — in a careful system it was not — but because those documents are retrieved and used to condition generation. Entities, relations, procedural steps and sometimes near-verbatim spans appear in the output. A patient adversary who maintains state across turns can accumulate a substantial fraction of the hidden corpus without ever seeing a file path.

Earlier public attacks were mostly local heuristics. Continuation-style methods such as RAG-Thief keep following the previous answer. They scale, but they drift. Keyword and implicit methods such as IKEA (Silent Leaks) stay closer to the corpus but tend to remain in already-explored neighbourhoods. Both lack a global objective. They react to the latest observation instead of choosing the next question for maximum new coverage.

The 2026 RAGCrawler work attacked exactly that limitation. Read the HTML version if you hate PDFs, or the PDF if you want the full tables.

I am not going to give you exploit code. You do not need it to defend, and you should not need it to understand the threat. White-hat work here is about recognising the shape of the attack so you can raise its cost and detect it earlier — not about reproducing it against systems you do not own.

11. How the attack thinks

The authors formalised knowledge-base stealing as an Adaptive Stochastic Coverage Problem. Each query is a stochastic action that reveals some documents through the retriever. The goal is to maximise expected unique coverage under a fixed query budget. Under standard conditions the objective is adaptively monotone and adaptively submodular, which yields the classic (1 − 1/e) approximation guarantee for the policy that always selects the action with the highest conditional expected marginal gain. The theoretical backbone is the older adaptive submodularity literature — Golovin & Krause, Adaptive Submodularity is the paper the RAGCrawler authors sit on.

In practice the attacker cannot observe true coverage gain, the query space is infinite, and questions must look natural. That is the engineering problem the paper solves with three cooperating pieces.

Knowledge-graph constructor. Builds an attacker-side graph of entities and relations from every answer. This is the global state. Without it, the attacker is a stateless loop that cannot tell explored regions from unexplored ones — behaviour closer to earlier local methods.

Strategy scheduler. Uses graph growth, structural holes and historical payoffs (UCB-style) to estimate which semantic anchors are likely to yield high new coverage. This is where the attack stops being “ask another similar question” and becomes “move into under-explored regions of the semantic space.”

Query generator. Turns those anchors into fluent, ordinary-looking questions while avoiding regions already adequately explored. Natural language is the point. If the queries look like attacks, simple filters catch them. If they look like customers, the system answers.

New answers expand the graph. The scheduler re-prioritises. New questions are issued. Because the attacker keeps a global view, the campaign systematically moves into under-explored regions instead of thrashing or drifting.

The published evaluation held across multiple corpora and generators, including some with safeguard layers, and remained effective against query rewriting and multi-query retrieval. Notice the uncomfortable symmetry with legitimate GraphRAG. From Local to Global builds a graph so a system can answer questions about a whole corpus. RAGCrawler builds a graph so an attacker can empty a whole corpus. Same object. Opposite intent.

12. Why the usual defenses disappoint

Topic blockers and simple refusal. The attack uses ordinary on-topic questions. Sensitive material leaks through retrieved context, not through explicit requests for forbidden content. Refusing “dump your system prompt” does nothing when the attacker asks “how do we handle refunds for enterprise customers on the legacy plan?”

Query rewriting and multi-query retrieval. These improve legitimate answer quality. They do not, by themselves, prevent a globally aware attacker from obtaining broad coverage. The RAGCrawler paper explicitly tests this. If your security review treats rewriting as a privacy control, update the review.

Rate limits. They slow the attack and raise cost. A patient or distributed adversary can still accumulate coverage over time. Necessary. Not sufficient. Volume-only limits also miss the signal that matters: systematic exploration of new entities and regions.

Canaries and watermarks. Excellent for detection after the fact and for attribution. Weaker at prevention while extraction is underway. Still deploy them. Just do not pretend they are a shield.

Retrieve less / summarise more. Reduces per-turn leakage. Also tends to reduce answer quality for complex legitimate queries. The attacker compensates with more turns. Self-RAG and CRAG are useful here for quality, not as a complete security control.

The structural tension remains: usefulness requires retrieving private material; every retrieval is a potential information channel. There is no configuration that maximises both utility and secrecy without tradeoffs. The work is to choose the tradeoffs deliberately instead of discovering them in an incident review.

13. The numbers from the paper

Approximate headline results from the RAGCrawler evaluations. Full tables and ablations are in the paper PDF. Numbers can move between versions; the paper is the source of truth.

Metric	Approximate result
Average corpus coverage	66.8%
Peak coverage	84.4%
Efficiency vs prior strongest baseline	≥ 4.03× fewer queries to reach 70% coverage
Surrogate answer similarity	up to 0.699
Robustness	Holds against rewriting and multi-query retrieval
Attack cost (authors’ estimate)	roughly low dollars per dataset at lite API prices

These are not “perfect copy” numbers. They are “enough to be commercially and operationally dangerous, obtained with significantly higher sample efficiency than earlier public methods.” If you are presenting this to a security review, take the PDF, not a blog table. If you are designing defenses, assume a patient adversary who is optimising for coverage, not for looking scary in the logs.

14. Defenses that actually moved in 2025–2026

For a long time the literature focused more on poisoning the knowledge base than on emptying it. That is changing.

RAGFort (November 2025, code at github.com/happywinder/RAGFort) is one of the first systematic attempts to defend against proprietary knowledge-base extraction as a dual-path problem. The insight is that attackers expand both within a topic (intra-class) and across topics (inter-class). Protecting only one path leaves the other open. RAGFort combines contrastive reindexing for inter-class isolation with constrained cascade generation for intra-class protection. The authors report cutting reconstruction and chunk recovery substantially compared with prior defenses while preserving answer quality. Joint protection matters; single-path is incomplete.

RAGSentinel (August 2026) targets a different threat — poisoned documents in the retrieval set — with a training-free, label-free geometric consensus filter on query-conditioned representation shifts. It is not an extraction defense, but it belongs in the same conversation: post-retrieval geometry can be stronger than instruction-following defenses that adaptive attackers learn to imitate.

Taxonomy work such as Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directions helps by naming the surfaces clearly: pre-retrieval poisoning, retrieval-time manipulation, post-retrieval context exploitation, and knowledge exfiltration. RAGCrawler, IKEA and RAG-Thief sit under extraction. If your internal threat model only lists “prompt injection” and “jailbreak,” it is incomplete for 2026.

None of these papers are magic “set and forget” products. They are the first generation of research that matches the attack models of 2025–2026. Production still needs the operational playbook in the next section — ownership, budgets, canaries, exploration signals, and incident response — because research defenses do not deploy themselves.

15. A practical cybersecurity playbook

There is still no perfect technical defense. The realistic goal is to raise cost, reduce yield, and improve the chance of early detection. Treat the following as defense-in-depth for white-hat teams shipping real systems.

Architecture and data design

Keep the highest-value material behind additional gates — extra authentication, tool-calling steps, step-up verification, or human review — rather than pure open retrieval. Split collections by sensitivity. Do not put crown-jewel runbooks in the same index as public FAQ content. For sensitive domains, prefer generation styles that stay tightly grounded even if that costs some fluency. The product conversation is “which answers are allowed to be slightly less chatty in exchange for leaking less,” not “can we have both maximum helpfulness and maximum secrecy for free.”

Query and session controls

Add strong intent classification and routing so simple or low-sensitivity questions never touch the most valuable collections. Tighten per-user, per-session and per-tenant budgets, especially for new or low-trust accounts. Instrument behavioural signals aimed at systematic exploration: rapid discovery of new entities, sequences that keep expanding coverage, patterns that look more like coverage maximisation than normal user paths. Volume limits alone miss this.

Retrieval and generation controls

Use context minimisation for sensitive collections. Add output-side groundedness and span checks where the domain justifies the latency cost. Make rate limits react not only to volume but also to exploration-like behaviour.

Detection and response

Seed canaries and unique trackable facts. Monitor for their appearance outside your systems and for unexpected appearance in outputs. Log enough session and retrieval metadata to reconstruct whether a conversation was systematically filling structural holes. Maintain a short playbook for suspected extraction: tighter limits, forced step-up auth, temporary isolation of sensitive collections, forensic review. Legal and contractual layers remain part of a mature posture — they do not stop a determined adversary, but they change the economics and the aftermath.

Checklist you can run this month

[ ] Inventory which collections contain high-value or regulated material
[ ] Confirm those collections are not reachable by the lowest-trust access path
[ ] Add or tighten per-session and per-user budgets on the chat / API surface
[ ] Deploy at least a minimal set of canary facts and a way to notice them
[ ] Instrument basic exploration signals
[ ] Document a short incident-response path for suspected extraction
[ ] Review whether query rewriting or multi-query retrieval is giving a false sense of safety
[ ] Evaluate retrieval quality with Ragas on a held-out set of your questions
[ ] Skim RAGFort and RAGCrawler with your security team

None of these stop a determined, well-resourced attacker forever. Together they make casual and mid-tier extraction noticeably more expensive and more visible — which is the realistic bar for most product teams.

16. Where builders and attackers are meeting

Legitimate systems are moving toward agentic crawlers and agentic retrieval: memory, multi-step planning, decisions based on a growing model of the information space, quality verification, closed loops. Knowledge-graph guidance appears in GraphRAG and in various enterprise document-understanding efforts. Google’s 2026 framing of agentic RAG stresses persistence — keep searching until the context is sufficient, not until a single retrieve call returns something plausible.(Google Research on Agentic RAG)

RAGCrawler is already an agentic crawler that plans, maintains a growing graph, estimates marginal coverage, and acts through natural language.

Improvements in agent memory, tool use, long-horizon planning and graph reasoning therefore improve both legitimate pipelines and extraction attacks. The race is less “is extraction possible?” and more “how efficiently can each side explore an unknown document space under budget and stealth constraints?”

Kiela’s line is still the right one: retrieval is not disappearing; it is being absorbed into richer context engineering and agentic loops. The same observation applies, uncomfortably, to the attack surface. Elastic’s take from the search-infra side is worth reading next to Anthropic’s: From retrieval to agents.

If you are building agents, you are also building a system that can be pointed at someone else’s knowledge base — or at your own. Design with that dual use in mind.

17. What to do this month

If you own a RAG product or internal knowledge system

Treat the ingestion and promotion pipeline as a first-class product with owners, SLOs and a rollback story. Measure end-to-end freshness and retrieval quality on your real critical queries, not only on public benchmarks. Prefer official APIs and structured exports over scraping when quality and legal posture matter. Assume the conversational interface can be used as an extraction oracle and apply the checklist in section 15. Keep the highest-value material behind additional controls.

If you work on platform security or research

Read RAGCrawler, then IKEA and RAG-Thief, in that order. Test whether your current rewriting and multi-query layers actually reduce global coverage or only change surface form. Explore whether the same graph techniques used by attackers can be turned into defensive monitors. Support shared evaluation suites for knowledge-base leakage; the area is still immature compared with classic model-stealing benchmarks. Track extraction defenses (RAGFort) and poisoning defenses (RAGSentinel) as separate but related tracks.

If you are deciding what to build next

The highest-leverage work is usually not a new embedding model. It is ownership of the observation layer, explicit promotion, provenance that engineers will actually use, and a security review that includes extraction — not only injection. Ship the boring controls. Then read the papers.

18. People, papers, tools — a working map

This is the section many Dev.to RAG posts skip. Every URL below is a real page. Prefer abs and PDF over secondary summaries when citing numbers.

Core attack research (2024–2026)

RAGCrawler — abs · HTML · PDF — must-read: graph-guided extraction
RAG-Thief (2024) — agent continuation attacks
Silent Leaks / IKEA (2025) — benign queries, high yield
Adaptive Submodularity (Golovin & Krause)

People worth following

Douwe Kiela — original RAG co-author, Contextual AI. RAG is dead, long live RAG!
Akari Asai — Self-RAG. akariasai.github.io
Jerry Liu / LlamaIndex — production RAG patterns
Harrison Chase / LangChain — retrieval as a tool inside agents
Microsoft GraphRAG team — Darren Edge, Jonathan Larson and collaborators
Abhinav Kimothi — the practical RAG book
Tomaž Bratanič — graphs in production
Sebastian Raschka — the “from scratch” books that keep people honest about what models actually do

Books

Crawlers and scrape-to-RAG tooling

Frameworks, eval, indexes

Hands-on analysis (the SEO / crawl overlap)

What I would ship in the first two weeks

If this article only leaves you with a longer reading list, it failed. Here is the sequence I would actually run on a real system that already has a chat UI and a vector index:

Week 1 — stop the silent rot. Wire a quality gate before embed (thin text, text-to-HTML ratio, canonical, near-dupe). Log every drop. Add content hashes and observation timestamps to every chunk that already exists — even if you only backfill metadata. Define one freshness SLO for the three sources that change most often. Turn off full nightly re-embeds if you cannot explain why every chunk needs them.

Week 1 — stop treating the chat UI as harmless. Per-session and per-user budgets. A canary fact in a sensitive collection. Basic logging of which collections were retrieved, not only which answer was shown. A one-page incident note: who gets paged if exploration-like traffic spikes.

Week 2 — make promotion real. Candidate or shadow index for at least one high-churn source. Compare before promote. One rollback drill: deliberately ship a bad observation, then revert using evidence hashes. Measure source-to-queryable lag once, with a number, not a feeling.

Week 2 — pick the stack you can operate. Postgres + pgvector (or the vector DB you already pay for), one embedding model, hybrid retrieval, one reranker. Do not start three migration projects. Measure on twenty questions your support team actually asks.

Papers matter. Playbooks matter more when the index is already in production.

Closing

The RAG crawler has two lives in 2026.

In one life it is the unglamorous component that decides whether your retrieval system is fed clean, fresh, well-structured knowledge or a swamp of duplicates and stale pages. Getting this right remains one of the highest-leverage engineering investments available — especially as agents, not only humans, consume the context.

In the other life the same family of ideas — global state, expected marginal gain, systematic exploration — has become a practical way to hollow out a private knowledge base through ordinary conversation. The 2026 RAGCrawler results should end the comforting belief that “the model never trained on the data, so the data is safe behind the API.”

The useful response is not panic. The same discipline that produces a high-quality builder-side crawler also makes you a better defender. You start seeing your own system the way a patient, graph-guided adversary would see it.

Build the knowledge engine carefully.

Assume someone else may try to crawl it.

Instrument both sides of the loop.

Raise the cost of global extraction without destroying legitimate utility.

If this helped, send it to the person who actually owns your knowledge pipeline. They are the ones who need it most.

About the author

Built around production crawl and site-intelligence work at AuditMe — AI-powered technical analysis for the same class of problems this article treats as the observation layer of RAG (discovery, canonicals, change, quality).

If you ship retrieval systems and want to compare notes on crawlers, freshness, or extraction defenses — open an issue on the tools you use, or reach out via the site.

Written as a working map for late August 2026, not a victory lap. If a link dies, a number in a paper moves, or you have contradictory production experience — that is the conversation worth having. The field is moving. The crawl loop is not going away.

The New SEO: When Search Engines Stop Reading Websites and Start Using Them

Eduard — Thu, 27 Aug 2026 08:37:13 +0000

TL;DR: Search is moving from ranking pages to running them as machine interfaces. If your site can't be discovered, understood, verified, and acted upon by AI systems, you're invisible to the next wave of queries — no matter how well you rank today. Most sites fail on verification and actionability, not discoverability. The fix is a six-dimension "Website Intelligence" framework: discoverability, understanding, verification, actionability, reliability, and observability. I built this into the free AuditMe SEO audit, and this article walks through the engineering behind it.

The Problem Nobody Is Talking About
Why the Document Model Broke
What Actually Changed in Search Architecture
The Agent Problem Nobody Prepared For
Machine Trust and the Reconciliation Problem
Six Things a Machine Must Be Able to Do With Your Website
The Data Nobody Shares
Why Schema Alone Cannot Save You
From Audits to Continuous Verification
The Distributed Truth Problem
What Web Agent Research Actually Found
The Engineering Stack Nobody Teaches
A Practical Website Intelligence Framework
What Happens Next (Honest Projections)

The Problem Nobody Is Talking About

There is a quiet structural problem emerging across the web, and most website owners have no idea it exists.

It is not about rankings. It is not about traffic. It is not about backlinks. Those things matter, but they are symptoms of a deeper issue that almost nobody is measuring.

Here is the problem, and I'll state it without hedging:

Websites are becoming machine interfaces, but they were designed as human documents.

For twenty-five years, the entire SEO industry has operated on a single assumption: optimize a page so a human will find it useful, and search engines will reward you. That assumption is not wrong. It is incomplete. And the gap between "not wrong" and "complete" is where your competitors will lose — or gain — the next wave of search traffic.

Consider this concrete example. A B2B SaaS company publishes a landing page. The page looks great. The copy is polished. The design converts well. A human visitor understands exactly what the product does and signs up.

Now an AI agent visits the same page. It needs to answer the question: "What does this product cost, and is it suitable for a team of 50?"

The agent encounters:

Pricing is mentioned as "Starting at $29/mo" in the body copy
The Product schema says "price": "49"
The FAQ section says "Plans start at $29/month for small teams"
The API documentation references a "$39 plan"
The checkout page shows $49

A human can investigate these discrepancies. The agent has a reconciliation problem. Which number should it trust? How confident should it be in its recommendation?

Multiply this across every product, every page, every representation of every fact on every website, and you begin to see the scale of the issue.

Why the Document Model Broke

The World Wide Web started as a document system. Tim Berners-Lee's 1989 proposal described "a system for sharing research papers through linked documents." HTTP, HTML, URLs — every foundational technology was built around the assumption that a human would read the result.

Google's original PageRank algorithm (Brin & Page, 1998) treated the web as a graph of documents, where links served as votes of authority. The fundamental unit was the page. The fundamental consumer was the human reader. That architecture held for two decades. It held because the math was elegant and the results were useful.

Then three things happened simultaneously.

First, large language models got good enough to summarize web pages. Not perfectly, but well enough that a growing number of users prefer a synthesized answer over clicking through ten blue links. Google's AI Overviews now reach over 1.5 billion users monthly.

Second, tool-use capabilities gave AI systems the ability to do things, not just read things. OpenAI's function calling specification and Anthropic's tool use framework allow models to interact with APIs, fill forms, and execute multi-step workflows.

Third, autonomous web agents emerged as a research category. Projects like WebArena demonstrated that agents can navigate real websites to complete tasks — book a table, buy a product, fill out an application — with varying degrees of success.

Each of these developments quietly broke an assumption that the SEO industry had held for decades: that the only consumer of web content is a human with a browser.

The search result is no longer the destination. It is an intermediate step in a pipeline that looks more like this:

User question
     |
Machine reasoning
     |
Evidence gathering from multiple sources
     |
Fact reconciliation
     |
Answer synthesis OR task execution

When the consumer of your content is a machine performing reasoning, the requirements for that content change fundamentally. Not in degree. In kind.

What Actually Changed in Search Architecture

Understanding the shift requires looking at the actual technical components that changed, not the marketing narratives.

The Rendering Pipeline

Googlebot has operated on a two-phase system since the early 2000s: crawl (fetch HTML), then render (execute JavaScript to produce the final DOM). Google's rendering pipeline uses a headless Chromium instance that executes JavaScript, waits for network idle, and produces a rendered DOM that approximates what a human sees (Google Search Central: JavaScript SEO Basics).

The practical implication: your rendered DOM — not your raw HTML — is increasingly what search engines analyze. A page that relies entirely on client-side JavaScript to display pricing, product specifications, or authorship information creates a gap between the crawled document and the understood document. That gap is where information gets lost.

The WHATWG DOM specification (dom.spec.whatwg.org) defines the browser's in-memory representation of a document. Search engines approximate this representation. Any information not present in it is effectively invisible to them.

The Structured Data Layer

Schema.org, the collaborative vocabulary maintained by Google, Microsoft, Apple, and Yahoo, provides a way to annotate entities and relationships in structured data (schema.org). Google supports structured data through JSON-LD, Microdata, and RDFa formats, with JSON-LD recommended.

Here is the thing most people miss: structured data is a secondary representation of information that should already exist in the page. It is not a content source. Google's structured data policies state: "Structured data helps search engines understand the content on the page."

The phrase "understand the content" is doing significant work in that sentence. It means structured data is an interpretation aid, not a content source. When structured data contradicts visible content, the system does not get "more information." It gets a conflict.

The AI Reasoning Layer

When a search engine uses a large language model to synthesize an answer, it operates fundamentally differently from traditional ranking. Instead of selecting a single best-matching document, the model retrieves evidence from multiple sources, evaluates source quality, synthesizes a coherent answer, and cites sources.

This pipeline requires each source to provide extractable facts, verifiable claims, and machine-readable interfaces. Google's documentation on AI optimization emphasizes creating content that demonstrates first-hand experience and using structured data to help Google understand your content.

The implication is that the optimization target has shifted from "rank this page" to "make this website's information accessible to AI reasoning systems." Those are different problems.

The Agent Problem Nobody Prepared For

The most consequential architectural change is not AI Overviews. It is the emergence of web agents — autonomous systems that interact with websites to complete tasks on behalf of users.

A traditional crawler fetches a page and indexes its content. An agent fetches a page, interprets it, and acts on it. That distinction changes everything about what a website needs to provide.

Consider what a restaurant website needs to offer each type of consumer:

What a crawler needs from your restaurant page:

Page title
Address
Menu content
Opening hours
Reviews with ratings

What an agent needs from your restaurant page:
All of the above, plus:

A form to make a reservation, with labeled inputs (not just a phone number)
Semantic buttons (<button>, not <div onclick="...">)
Clear input validation (date picker, party size selector)
Success and failure states after submission
API endpoints for programmatic interaction
Price information in a parseable format

The critical difference: the agent needs actionable interfaces, not just information. A <button type="button">Reserve</button> communicates intent through document structure. A <div class="btn" onclick="openReservation()">Reserve</div> communicates nothing through structure. It relies entirely on JavaScript execution to reveal its purpose.

The HTML Living Standard (html.spec.whatwg.org) defines the contract for each element. A <button> has specific semantics: it is focusable, activatable, and communicates intent. A <div> has none of these properties. The WAI-ARIA specification extends this with roles, states, and properties that clarify intent for assistive technologies and, by extension, for machine agents.

This is not theoretical. It is measurable.

Machine Trust and the Reconciliation Problem

When a human reads a website, they can reason about inconsistencies. "This price says $49 here but $39 on the pricing page — probably an old page." Machines cannot easily do this. They encounter the same information from multiple sources and must determine which version to trust.

Google's Knowledge Graph, which powers Knowledge Panels and AI Overviews, maintains confidence scores for facts. Contradictory sources reduce these scores. Google's research on Knowledge Vault (Dong et al., 2014) describes the probabilistic fusion process: each source contributes evidence, and conflicts reduce the system's confidence in any given claim.

The practical reality looks like this. Imagine your company publishes:

Homepage:          "Plans start at $29/month"
Pricing page:      "Starter: $39/month"
Product schema:    "price": "49"
API response:      "monthly_price": 39
Documentation:     "The $29 plan includes..."
Third-party listing: "$39/month"

A human can investigate and form a judgment. A machine sees six contradictory signals and has no way to determine which is current. The result is not that the machine picks one. The result is that the machine's confidence in all of your pricing information drops.

This creates a specific type of technical SEO problem that traditional audits do not detect: mismatch detection across representations. The most valuable audit may be the one that discovers contradictions between your website, your schema, your documentation, your API, and your third-party listings.

Six Things a Machine Must Be able to Do With Your Website

Drawing from the evidence above, machine-readiness for websites can be evaluated through six requirements. These are not ranking factors. They are capabilities that a machine must possess to use your website effectively.

1. Discoverability: Can I Reach It?

Before a machine can understand your website, it must be able to reach it. This is the domain of traditional technical SEO, and none of it becomes obsolete.

What discoverability requires:

Crawlable pages with accessible URLs
Intentional robots.txt rules (Google: Robots.txt)
Valid XML sitemaps (Google: Sitemaps)
Correct canonical URLs (Google: Canonicalization)
Predictable redirect behavior (minimal chain length)
Accessible critical resources (CSS, JS, images not blocked)

Google's Gary Illyes has stated that crawl budget is real and should be managed. For large sites, poor discoverability means portions of the site are never indexed, and therefore never available to AI systems.

2. Understanding: What Am I Looking At?

Discovery answers "Can I fetch this?" Understanding answers "What am I looking at?"

Machines need structure. That means semantic HTML, meaningful headings, clear navigation, explicit entities, valid structured data, understandable page intent, consistent terminology, and machine-readable content.

The semantic gap between implementations is measurable. The WebAIM Million study analyzed the home pages of the top 1,000,000 websites each year since 2019 — and in the 2026 edition the trend lines are telling. 95.9% of home pages still had detectable WCAG 2 failures, and the average page carried 56.1 distinct accessibility errors. The most common issues — low contrast text (83.9%), missing alternative text (53.1%), empty links (46.3%) — are not just accessibility problems. They are machine-readiness problems. A machine that cannot parse your content structure cannot understand your content. If 96% of the web's most popular pages are structurally unreliable, that is not an accessibility niche. It is the default state of the web.

3. Verification: Can I Trust What It Says?

A machine should not merely extract a fact. It should be able to determine whether the fact is supported, consistent, and current.

Verification asks: does the structured data match the visible content? Does the canonical URL match the page being served? Does the organization name remain consistent? Do prices agree across the product page, schema, and checkout? Do dates make sense? Does the documentation describe the current product?

The practical consequence is that structured data should reinforce meaning, not create an alternative reality. A Product schema that claims $39 when the page shows $49 does not "optimize" anything. It introduces a contradiction that reduces machine confidence.

4. Actionability: Can I Do Something?

Traditional SEO concerns information retrieval. Agents introduce a different requirement: can the machine complete a task?

For a restaurant, that means making a reservation. For e-commerce, that means purchasing a product. For SaaS, that means starting a trial. For documentation, that means integrating an API.

The W3C Web of Things specification defines a framework for making web-connected devices interoperable with machine agents. While focused on IoT, the principle applies broadly: web interfaces must be machine-actionable, not just human-readable.

Google's AI Overviews with actions can now perform tasks on behalf of users — making reservations, purchasing products. For this to work, the website must provide identifiable actions (semantic buttons, forms, links), parseable inputs (labeled fields, correct types), predictable outcomes (standard HTTP methods), and clear error states.

5. Reliability: Can I Count on It?

A machine needs predictable, consistent access to information. This is where the deployment problem becomes relevant.

Modern websites change through deployments. Each deployment is a potential source of regression. Schema removed during a template update. Canonical URL changed during a migration. Pricing changed on the website but not in the schema. Documentation updated but the API response not synchronized.

Traditional SEO audits capture a point-in-time snapshot. They do not detect regressions that occur after the audit. The SRE practices developed at Google (SRE Book, Beyer et al., 2016) define principles for monitoring complex systems. Websites are becoming complex systems. They need the same treatment.

6. Observability: Can I Detect When It Changes?

Once a website becomes an input to machine decisions, change itself becomes a signal. You need to know what changed, when it changed, which facts changed, and whether a previously valid workflow still works.

The feedback loop looks like a software observability pipeline: establish baseline, monitor, detect change, evaluate impact, fix, verify, repeat. This is closer to Site Reliability Engineering than to the traditional SEO audit model. That is not an accident. Modern websites increasingly behave like software systems. Their SEO should be monitored like one.

The Data Nobody Shares

The shift from document-centric to system-centric search is not theoretical. Several data points quantify it.

AI Overview adoption: Google reported AI Overviews reaching 1.5 billion users monthly as of early 2025 (Google AI Overviews). Perplexity AI processes over 100 million queries per week.

Click behavior impact: Research by BrightEdge found that AI Overviews reduced organic CTR by 18-25% for informational queries. However, queries with AI Overviews that include source links saw increased click-through to cited sources. The implication: being cited by AI is becoming as important as ranking in organic results.

Agent adoption: OpenAI's ChatGPT browsing tools were used over 100 million times in the first quarter after launch. Anthropic's Claude web interaction capabilities showed measurable improvement in task completion rates from 2023 to 2024.

DOM quality matters: WebAIM's analysis of 1,000,000 websites found widespread structural issues that affect both accessibility and machine readability. In the 2026 report, average home pages contained 1,437 page elements and 56.1 detected errors, with low contrast text on 83.9% of pages and missing alt text on 53.1%. These are not abstract concerns. They are the infrastructure that machines use to understand your content. Notably, the same report shows framework choice correlates with structural health: Next.js sites averaged 40.9 errors per page — 27% below the overall average — while jQuery-based and ad-heavy pages trended far worse. The way you build directly shapes how well a machine can read you.

Why Schema Alone Cannot Save You

Structured data is one of the most powerful tools in technical SEO. It is also the most commonly misunderstood.

Schema.org provides a vocabulary for marking up entities and relationships. Google supports over 30 structured data types, including Product, Organization, Article, FAQ, Event, JobPosting, and Recipe. When implemented correctly, structured data enables rich results — enhanced search appearances with additional information.

But structured data has hard limits.

It cannot create information that does not exist in the page. It cannot override content that contradicts the schema. It cannot fix structural problems in the DOM. And it cannot substitute for consistency across your digital presence.

The correct approach to structured data:

Ensure the information exists clearly in the page content
Mark it up with appropriate schema types
Verify that schema values match visible content
Verify that schema values match external representations
Monitor for drift over time

Google's structured data policies state: "Don't mark up content that is not visible to the user" and "The structured data on a page should describe the content of that page."

These guidelines exist because schema mismatches reduce the system's ability to trust the data. Adding more schema to a page with contradictory information does not help. It creates another contradiction.

From Audits to Continuous Verification

The audit-as-snapshot model is breaking for the same reason manual software testing was replaced by continuous integration.

The traditional audit cycle looks like this: run audit, get report, fix issues, done. The problems are obvious. The audit is valid only at the moment it was run. Regressions go undetected until the next audit. There is no feedback loop between "fixed" and "verified fixed."

The continuous verification model replaces this with: establish baseline, monitor, detect change, evaluate impact, fix, verify, repeat. Regressions are detected immediately. The "fixed" state is verified automatically. The baseline evolves as the website improves.

Google Search Console provides some of this capability for Google-specific metrics. But it does not monitor schema consistency, cross-source accuracy, agent workflow integrity, DOM semantic quality, or third-party listing accuracy. These require dedicated monitoring infrastructure.

The concept is not new. Software engineering has practiced continuous verification for decades. The SRE book (Beyer et al., 2016) defines the principles. Applying them to SEO is a natural extension that the industry has been slow to adopt.

The Distributed Truth Problem

Modern businesses maintain information across many systems: their website, CMS, schema markup, Google Business Profile, Google Merchant Center, API responses, documentation, social profiles, third-party directories, PDF catalogs, and mobile apps.

Each of these is a representation of the same entity. Every representation creates an opportunity for divergence.

Viewing this through the lens of distributed systems engineering reveals that consistency is not a content problem. It is a data engineering problem. The principles that apply to distributed databases apply here: eventual consistency (all representations should converge), conflict resolution (defined source of truth), monitoring (divergence must be detected), and idempotency (updates applied consistently across representations).

Lamport's work on time, clocks, and the ordering of events in distributed systems (1978) established the foundational principles. The scale differs between database replication and cross-platform SEO consistency, but the principle is identical: when multiple systems maintain copies of the same information, consistency must be actively managed.

What Web Agent Research Actually Found

Several research projects have directly measured how AI agents interact with websites. The findings are concrete and actionable.

WebArena (Zhou et al., 2023)

WebArena provides a benchmark environment for evaluating autonomous web agents on realistic tasks across e-commerce sites, forums, CMS platforms, and mapping applications. Accepted as an Oral at NeurIPS 2024, it is now the anchor of an entire family of benchmarks gathered under WebArena-x: VisualWebArena for multimodal agents (ACL 2024), WebArena-Infinity for continuous evaluation in evolving environments, and TheAgentCompany (ICML 2025), which evaluates agents on consequential real-world office tasks inside a simulated company.

The results are humbling for the field. The best-performing agents achieved approximately 14% success rate on complex multi-step tasks. Human performance on the same tasks was approximately 75%. The primary failure modes were not comprehension failures. They were interaction failures — incorrect element selection, misunderstanding of DOM structure, inability to handle dynamic content.

Pages with clear semantic structure had measurably higher agent success rates. The implication is direct: DOM quality is not an accessibility nice-to-have. It is a machine-accessibility requirement.

Mind2Web (Deng et al., 2023)

Mind2Web (a NeurIPS 2023 Spotlight) provides a dataset of 2,350 open-ended tasks across 137 real websites for training and evaluating generalist web agents.

The key finding: agent performance strongly correlated with DOM semantic quality. Semantic HTML elements — <button>, <input>, <nav> — were identified correctly more often than generic elements like <div> and <span>. Form accessibility (labels, field types, validation) directly impacted task completion.

This is not a correlation without causation. The semantic elements carry behavioral contracts defined in the HTML Living Standard. A <button> is focusable and activatable by default. A <div> is not. When an agent tries to interact with a page, these contracts determine whether the interaction succeeds.

SeeAct (Zheng et al., 2024)

SeeAct (accepted at ICML 2024) is a framework for web agents that uses visual grounding to interact with websites through screenshots and DOM analysis.

The research showed that agent accuracy decreased 30-40% on pages that relied on non-standard UI components. Pages that followed web standards — semantic HTML, ARIA attributes — had significantly higher interaction success rates. Dynamic content rendered via JavaScript was a major source of agent errors.

The convergence across all three projects is clear: the semantic quality of a website's DOM directly affects how well machine agents can use it. This is measurable, quantifiable, and increasingly consequential.

The Engineering Stack Nobody Teaches

The shift from page-level optimization to system-level intelligence requires a different engineering stack. The traditional SEO stack — content, HTML, meta tags, schema, links, analytics — remains relevant but is no longer sufficient.

The Website Intelligence stack includes:

Information architecture and URL strategy
Semantic HTML and DOM quality
Structured data (JSON-LD, correctly implemented)
Cross-source consistency management
Performance engineering (Core Web Vitals)
Accessibility (WCAG 2.2 compliance)
API design and documentation
Continuous monitoring and regression detection
Agent workflow testing

This stack looks like a software engineering discipline because it is one. The SEO profession is converging with web engineering, data engineering, quality assurance, and observability. That convergence is not a rebranding exercise. It is a reflection of the fact that websites are becoming software systems, and their machine-readiness requires the same rigor we apply to software reliability.

The standards that define this stack are established and freely available: the HTML Living Standard, the DOM Living Standard, WCAG 2.2, WAI-ARIA, Schema.org, and Google's Search Central documentation. None of these require proprietary knowledge. They require systematic application.

A Practical Website Intelligence Framework

Based on the framework presented in this article, a Website Intelligence audit should evaluate six dimensions. Each dimension includes concrete checkpoints. I've implemented this exact framework in the free AuditMe SEO audit tool, which runs many of these checks automatically — for example, cross-field verification that catches the price and date mismatches described above. The checks below are tool-agnostic; any team can run them with a good crawler and a text editor.

Discoverability

Important pages are crawlable by standard user agents. Robots.txt rules are intentional. XML sitemaps are valid and submitted. Canonical URLs are correct. Redirect chains are minimal. Critical resources are accessible to crawlers. URL structure is stable and predictable.

Understanding

Page intent is clear from structure. Heading hierarchy is logical. Interactive elements use semantic controls. Navigation uses <nav> with meaningful link text. Entities are identifiable. Structured data is valid and matches page content. Content is understandable without visual context.

Verification

Visible prices match Product schema. Organization name is consistent across website and schema. Author information is verifiable. Dates are accurate. Documentation describes current features. External representations agree with website.

Actionability

Critical actions use semantic form controls. Form inputs have associated labels. Form validation provides clear feedback. Required fields are indicated. Success and failure states are understandable. API endpoints are documented and stable.

Reliability

Core pages load consistently. APIs respond predictably. Important URLs remain stable. Schema data survives template updates. Stale information can be detected automatically.

Observability

Key pages have baseline snapshots. Schema changes are tracked. DOM structural changes are detected. Cross-source consistency is monitored. Regressions are flagged automatically. Critical workflows can be re-tested.

What Happens Next (Honest Projections)

Predictions are fragile. But directional trends are observable. Based on current trajectories in AI research, search architecture, and web standards, several developments are likely — and a few have already started.

Agent-native websites become the standard. Websites that provide agent-accessible interfaces — semantic HTML, API endpoints, labeled forms — will have a measurable advantage in AI-mediated discovery. The companies that build these interfaces early will capture the machine-mediated traffic that others miss.

Cross-source consistency becomes a ranking factor. As AI systems become more sophisticated at reconciling information across sources, inconsistency between a website's representations will increasingly be treated as a trust deficit. A website that maintains consistent information across its CMS, schema, API, and third-party listings will outperform one that does not. This is no longer hypothetical — the verification checks that catch these mismatches are already implementable as automated cross-field audits.

Continuous verification replaces periodic audits. The SRE-inspired model of continuous monitoring will become the standard for SEO. Manual audits will be supplemented — and in some cases replaced — by automated systems that detect regressions, verify fixes, and maintain baselines. This is the projection I am most confident about, because I have already built it into AuditMe: baseline snapshots, change logging per URL, field-level schema diffs, and alert delivery on regression. The tooling is not science fiction — it exists today.

DOM quality becomes a first-class metric. The semantic quality of a website's DOM will be measured and tracked as a metric, similar to how page speed is measured today. Tools will emerge that score DOM semantic quality the way Lighthouse scores performance. AuditMe now ships machine-readiness and semantic-interactive checks that score exactly this.

The SEO profession evolves. The role will increasingly overlap with web engineering, data engineering, quality assurance, accessibility, and observability. The practitioners who thrive will be the ones who can operate across these disciplines.

Conclusion

The web was built for humans to read. Search engines learned to index it. AI learned to summarize it. Agents are beginning to use it. The next generation of websites must satisfy all of these consumers simultaneously.

The requirements converge on the same principles: clear information that humans can understand, semantic structure that machines can interpret, consistent facts that all consumers can trust, accessible interfaces that all consumers can act on, stable behavior that all consumers can rely on, and observable changes that all consumers can detect.

These principles are not new. They are established engineering practices. What is new is that they now directly affect a website's ability to be discovered, understood, and used by the systems that increasingly mediate between businesses and their audiences.

The shift from SEO to Website Intelligence is not a rebranding exercise. It is a recognition that the primary consumer of web content is expanding from humans to include machines — and that machines have different, more demanding requirements for the information they use.

Build websites that machines can trust. The same qualities that make a website machine-readable make it better for humans too. That is not a coincidence. It is a convergence.

If your site is not machine-readable by the time the next wave of search hits full scale, you will not be invisible to search engines in the traditional sense. You will simply be a source they read but cannot trust — and trust is the entire game. The window to build for the machine consumer is open now, and it does not stay open forever.

Here is my honest verdict after building the AuditMe engine that operationalizes this framework: you do not need to wait for AI search to "mature" before acting. The six dimensions above are implementable today with tools that already exist. Most teams have 80% of the discoverability problem solved. The gap is almost always the same three dimensions — verification, actionability, and observability — because those are the ones nobody taught SEO practitioners to think about. Close that gap and you stop competing on the same terms as everyone else.

References

Google Documentation and Research

Google Search Central Documentation — The authoritative reference for crawling, indexing, and ranking behavior
Structured Data Policies — Guidelines for correct structured data implementation
JavaScript SEO Basics — How Google renders JavaScript
SEO Starter Guide — Foundation of technical SEO
AI Optimization Guide — Optimizing for generative AI search
Creating Helpful Content — Google's content quality guidelines
Canonicalization — Managing duplicate content
AI Overviews Announcement — Google's AI search features

Standards and Specifications

Schema.org — The collaborative vocabulary for structured data
HTML Living Standard — The definitive specification for HTML
DOM Living Standard — Browser document object model specification
WCAG 2.2 — Web Content Accessibility Guidelines
WAI-ARIA — Accessible Rich Internet Applications
W3C Web of Things — Interoperability framework for web-connected devices
robots.txt Specification (RFC 9309) — Standard for crawler directives

Research Papers and Projects

Brin, S. & Page, L. (1998). "The Anatomy of a Large-Scale Hypertextual Web Search Engine." Wikipedia: PageRank
Berners-Lee, T. (1989). "Information Management: A Proposal." w3.org/History/1989/proposal.html
Dong, X. et al. (2014). "Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion." research.google
Lamport, L. (1978). "Time, Clocks, and the Ordering of Events in a Distributed System." lamport.azurewebsites.net
Beyer, B. et al. (2016). "Site Reliability Engineering." sre.google/sre-book

AI and Agent Research

Zhou, S. et al. (2023). "WebArena: A Realistic Web Environment for Building Autonomous Agents." webarena.dev (NeurIPS 2024 Oral; part of the WebArena-x family)
Deng, X. et al. (2023). "Mind2Web: Towards a Generalist Agent for the Web." github.com/OSU-NLP-Group/Mind2Web (NeurIPS 2023 Spotlight)
Zheng, B. et al. (2024). "SeeAct: Grounded Vision-based Web Agents." github.com/OSU-NLP-Group/SeeAct (ICML 2024)
Xu, F. et al. (2025). "TheAgentCompany: Benchmarking LLM Agents on Consequential Real-World Tasks." the-agent-company.com (ICML 2025)

AI Platform Documentation

OpenAI Function Calling — How LLMs invoke external tools
Anthropic Tool Use — Claude's tool interaction framework
OpenAI Web Search — Browsing capabilities

Industry Research

WebAIM Million 2026 — Accessibility analysis of top 1M websites
BrightEdge Research — AI Overviews impact data

This article examines the architectural and semantic requirements websites must satisfy as search evolves from document retrieval to machine-mediated reasoning. The framework presented (Website Intelligence) draws from established web standards, published research, and observable trends in search architecture. The durable engineering principles it identifies will remain relevant regardless of specific implementation changes in any single AI system.