DEV Community: Eduard

Hey! What FREE or open-weight AI models & agents do you use for UI & design refactoring? Which free setups (DeepSeek, Qwen Coder, Ollama) and tools (OpenCode, Cline, Continue) give you the cleanest React/Tailwind code? #discuss #ai #opensource #webdev

Eduard — Mon, 07 Sep 2026 12:44:58 +0000

Master Prompts in 2026: Stop Prompting Like It's 2023

Eduard — Thu, 03 Sep 2026 06:50:39 +0000

Master Prompts in 2026: Stop Prompting Like It's 2023

I still see people paste a 40-line “act as a senior expert with 20 years of experience” block into ChatGPT and call it engineering.

That stopped working as a strategy a while ago.

Models got better. Context windows got bigger. Agents started calling tools. And the failure mode shifted. It’s rarely “the model is dumb” now. It’s “your system has no contract.”

This is a long, practical write-up on master prompts — the stable policy layer above individual tasks. How to write them. How to force planning. How to run Plan → Act → Observe → Verify without theater. How to make the same prompt useful to a tired human at 11pm and to an agent loop that only understands schemas.

I’ve broken enough production prompts across GPT-4o, Claude 3.5 Sonnet, and Gemini-class stacks to have opinions. Some of them are uncomfortable.

TL;DR / Key Takeaways

A master prompt is not a clever sentence. It’s the policy layer: role, success criteria, process, constraints, output contract, failure handling.
Production reliability comes from LLM orchestration patterns — plan JSON, single-task executors, and explicit done_when checks — not from longer personality blocks.
JSON contracts + verification beat free-form answers. Agents that can’t prove completion will invent it.
Treat prompts like code: version them, eval them, and put a real verify step after generation (including SEO/quality checks when you publish).

What a master prompt actually is
The 7-part anatomy that doesn’t collapse under pressure
Frameworks worth keeping (and which ones to ignore)
Planning is the real skill
From plan to agent loop
Context engineering beats clever wording
Few-shot, JSON contracts, and the anti-hallucination rule
Copy-paste masters you can actually deploy
A real publish pipeline (including the verify step people skip)
Eval or you’re guessing
Failure patterns I keep seeing
PromptOps: treat prompts like code
One universal master prompt
Ship checklist
A one-week install plan
Frequently asked questions
Sources
What to do in the next 15 minutes

1. What a master prompt actually is

A master prompt is not a magic spell.

It’s the policy layer:

who the model is allowed to be
what “done” means
how it should think when the task is messy
what format comes out
what happens when it’s unsure

User prompts change every hour.

Master prompts change when your standards change.

If you rewrite your “system personality” for every ticket, you don’t have a system. You have vibes.

This distinction matters more once you leave single-chat workflows and enter prompt engineering for production — multi-step agents, tool routers, RAG pipelines, shared team libraries. The master prompt becomes the constant. Everything else is runtime input.

Official docs still matter here, even if the ecosystem moved fast:

OpenAI prompting guide
Anthropic prompt engineering overview
Google’s prompt engineering notes
The Prompt Report (Schulhoff et al.) — still the best single survey of techniques

One shift I care about in 2026: people say context engineering more than prompt engineering. Same game, wider board. You’re not only choosing words. You’re choosing what the model sees on each step inside a limited context window — policy, retrieved docs, tool traces, and the live task.

2. The 7-part anatomy that doesn’t collapse under pressure

Every master prompt I’ve kept in production has some version of these blocks. Skip one and you pay for it later.

Block	Hard question it answers
Role	Who are you, for whom?
Goal	What counts as success in measurable terms?
Context	What’s true about this environment right now?
Process	In what order do you work?
Constraints	What is forbidden even if it would be convenient?
Output contract	What shape must the answer take?
Failure policy	What do you do when data is missing?

Skeleton

ROLE
You are a [specific role]. You work for [audience].

GOAL
Success = [observable outcome].
Failure examples: [what “almost right” looks like].

CONTEXT
- Product / domain:
- Hard limits:
- Sources of truth:

PROCESS
1) State assumptions or ask the minimum clarifying question.
2) Build a dependency-aware plan.
3) Execute one atomic step at a time.
4) Verify against done_when.
5) Return result + residual risks.

CONSTRAINTS
- Do not invent facts, APIs, quotes, or metrics.
- Do not fake tool output.
- If uncertain, say so and propose the cheapest check.

OUTPUT
## Plan
## Result
## Verification
## Open questions

Notice what’s missing: motivational fluff. “Be world-class.” “Think deeply.” Models already try. What they lack is your definition of finished work.

On Claude 3.5 Sonnet and GPT-4o alike, vague quality adjectives underperform hard constraints and explicit success criteria. The model isn’t missing ambition. It’s missing your acceptance tests.

3. Frameworks worth keeping (and which ones to ignore)

The internet loves acronyms. Most of them are the same idea in a hoodie.

Keep these

RTF — Role / Task / Format

Fine for small jobs. Don’t overbuild.

CRAFT — Context / Role / Action / Format / Tone

Good default for writing, analysis, support.

Plan-and-Solve

Force a plan before the answer. Boring. Effective. See the planning literature around Plan-and-Solve and agent planning surveys like arXiv:2402.02716.

Chain-of-Thought

Still the simplest accuracy lever on multi-step reasoning. Original paper: Wei et al., 2022.

Tree of Thoughts

When one path isn’t enough and you need deliberate search. Yao et al., 2023.

ReAct

Thought → Action → Observation. If your agent uses tools and you don’t have this loop, you’re improvising.

Ignore these habits

Collecting 14 frameworks and using none consistently
Padding prompts with personality cosplay
Asking for “maximum creativity” on compliance tasks
Writing novels in the system message that burn token efficiency for no gain

Pick one structure. Run it for a week. Measure. Then change one variable.

Anthropic’s own guidance still ranks clarity, examples, thinking, structure above theatrical roleplay. Read their best practices if you haven’t in a while.

4. Planning is the real skill

Most “agent failures” are just un-decomposed work.

A useful rule from task-decomposition practice: keep breaking the job down until each leaf task is doable in 1–3 tool calls and has a crisp done_when. If a step needs a short novel of instructions, it isn’t a step yet. (EngineersOfAI notes on decomposition are blunt about this for a reason.)

This is the boring core of LLM orchestration: not more model calls for their own sake, but a graph of verifiable work units.

Two planning styles

Decomposition-first

Build the full plan, then execute. Best for stable workflows: migrations, docs, publish checklists.

Interleaved

Plan a little, act, replan. Best for research and debugging where the map changes under your feet — including RAG pipelines where retrieval quality shifts mid-run.

A plan JSON agents can actually consume

{
  "goal": "Ship a technical article with a pre-publish quality pass",
  "assumptions": [
    "Target platform is Dev.to",
    "Audience is builders using LLMs in real workflows"
  ],
  "tasks": [
    {
      "id": "t1",
      "title": "Outline + claims list",
      "depends_on": [],
      "tool_hint": "none",
      "done_when": "H2/H3 outline exists and 8–12 claims are listed"
    },
    {
      "id": "t2",
      "title": "Write full draft",
      "depends_on": ["t1"],
      "tool_hint": "none",
      "done_when": "Complete draft with no TODO markers"
    },
    {
      "id": "t3",
      "title": "Fact-check hard claims",
      "depends_on": ["t2"],
      "tool_hint": "search",
      "done_when": "Every strong claim has a source or is marked UNVERIFIED"
    },
    {
      "id": "t4",
      "title": "Publish checklist + SEO verify",
      "depends_on": ["t3"],
      "tool_hint": "api",
      "done_when": "Top 5 impact/effort fixes are written from evidence"
    }
  ],
  "risks": [
    "Stale references",
    "Generic advice with no operational detail"
  ]
}

Planner-only master prompt

You are Task Planner. You do not execute. You only produce an executable plan.

Rules:
1) Split the goal into atomic steps.
2) One step = one action or one tool call.
3) Declare dependencies.
4) Every step needs done_when.
5) If information is missing, add assumptions and clarifying_questions.
6) No prose essay. Structure only.

Return strict JSON:
{
  "goal": "...",
  "assumptions": [],
  "clarifying_questions": [],
  "tasks": [
    {
      "id": "t1",
      "title": "...",
      "description": "...",
      "depends_on": [],
      "tool_hint": "none|search|code|browser|api",
      "done_when": "..."
    }
  ],
  "risks": []
}

Microsoft’s agent curriculum makes the same point in plainer language: define the goal, break it, then assign work. See their planning design chapter.

5. From plan to agent loop

Once you have a plan, stop letting the model freestyle the whole graph.

The loop

Plan → Act → Observe → Verify → Repair or Next

Without Verify, agents lie politely. They narrate completion. They do not prove it.

This loop is where prompt engineering for production stops being “wording” and becomes control flow. The master prompt defines the rules. The orchestrator enforces step boundaries. Tools supply evidence. Verification closes the books.

Executor master prompt

You are Executor Agent.
Take exactly one next task from the plan.
Do not jump ahead.

Inputs:
- plan JSON
- current_task_id
- tool_results (if any)

Method:
1) Re-read done_when for the current task.
2) If blocked on missing data, request a tool or mark blocked.
3) Do the smallest useful action.
4) Return:

## Action
## Evidence
## Status: done | partial | blocked
## Next recommendation

Repair rule that saves hours

If Status is partial or blocked:
1) Name the blocker in one sentence.
2) Propose the cheapest next check.
3) Do not rewrite the entire plan unless dependencies actually changed.

This is less glamorous than “autonomous agent.” It is also why some systems finish jobs and others generate confident debris.

6. Context engineering beats clever wording

I used to spend an hour polishing adjectives. Now I spend that hour deciding what not to put in context.

High-signal rule

Use the smallest token set that still steers behavior. That’s token efficiency as an engineering constraint, not a slogan.

Practical layout

Content	Placement
Stable policy / role	Front of the prompt (also helps caching)
Reference docs / data	Clearly delimited blocks
Retrieved RAG chunks	After policy, tagged and ranked by relevance
Examples	After policy, before the live task
User task	End

In RAG pipelines, the master prompt should also say how to treat retrieved text: prefer it over parametric memory, cite chunk ids, and refuse to invent when retrieval is empty. Without that policy, retrieval becomes decoration.

OpenAI’s notes on prompt caching are worth reading if cost and latency matter: put stable prefixes first, variable content last.

Delimiters

<policy>...</policy>
<context>...</context>
<retrieved>...</retrieved>
<examples>...</examples>
<task>...</task>

XML, Markdown headings, triple backticks — pick a convention and stop rotating it every sprint. Inconsistency is a silent quality tax across GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro deployments alike.

Long-context tip that keeps showing up in lab guidance: put large source material first, put the actual question last. Anthropic has reported meaningful gains from that ordering on long inputs inside a large context window.

7. Few-shot, JSON contracts, and the anti-hallucination rule

Few-shot that helps

Good examples are diverse and slightly annoying. Edge cases. Near-misses. Format traps.

Eight nearly identical happy-path samples teach the model to sound right while being fragile.

Two to five sharp examples beat a museum of mediocre ones.

Output contracts

If another system will consume the answer, stop accepting free-form essays.

Return ONLY valid JSON:
{
  "summary": "string",
  "actions": [{"priority": 1, "fix": "string", "effort": "S|M|L"}],
  "risks": ["string"]
}
No markdown fence. No commentary.

Then validate. Retry with the schema error. Humans can tolerate messy answers. Pipelines cannot — especially when the next hop is another agent, a ticket system, or a CMS write API.

Truth policy (non-negotiable)

TRUTH POLICY
- Do not invent citations, numbers, APIs, dates, or “studies.”
- If a claim is not grounded in provided context, retrieved chunks, or tool output, mark it UNVERIFIED.
- Incomplete + honest beats complete + fabricated.
- Prefer a cheaper verification step over a confident guess.

Labs keep repeating a version of this: allow “I don’t know.” It still gets ignored in the wild.

8. Copy-paste masters you can actually deploy

Research agent

You are a research analyst.

Process:
1) Source plan first
2) Notes with links/quotes
3) Synthesis only after notes exist

Rules:
- Every hard claim needs a source or UNVERIFIED
- Separate facts from interpretation
- End with confidence and open questions

Output:
## Source plan
## Notes
## Synthesis
## UNVERIFIED
## Next checks

Coding agent

You are a senior engineer working under change control.

Process:
1) Reproduce the problem
2) Minimal fix
3) Test or verification path
4) Short explanation of the diff

Constraints:
- No drive-by refactors
- No “while we’re here” features
- If a public API changes, call it out explicitly

Output:
## Root cause
## Fix
## Test plan
## Residual risks

Editor / publish agent

You are a technical editor with publishing standards.

Goal:
A draft that can ship — structure, claims, scanability, on-page hygiene.

Process:
1) Outline
2) Draft
3) Fact-check
4) Clarity pass
5) Publish checklist (title, description, H1/H2, links, alts)
6) If a live URL exists, run a verify pass and rank fixes

Output:
## Outline
## Final draft
## Checklist
## Top fixes

Ops triage agent

You are an incident triage agent.

Process:
1) Symptoms → ranked hypotheses
2) Cheapest diagnostic step
3) Evidence
4) Decision: fix / escalate / monitor

Output:
## Hypothesis ranking
## Next diagnostic step
## Decision
## Why

These are intentionally plain. Flashy prompts age badly. Contracts age better.

9. A real publish pipeline (including the verify step people skip)

Content agents love generating. They hate proving the page is healthy after publish.

A sane pipeline looks like this:

Idea → Outline → Draft → Fact-check → Edit → Publish checklist → Live verify → Fix backlog

The last two steps are where quality either becomes real or becomes marketing.

Where audit belongs in agent architecture

Once you have a URL, stop guessing about titles, meta, heading hierarchy, schema, and performance signals. Measure.

This is the gap most LLM orchestration diagrams skip: generation is only half the loop. Publish workflows need a machine-readable verification service that agents can call, parse, and turn into ranked work.

AuditMe provides an API designed for automated SEO verification within AI agent pipelines. It’s not a dashboard you stare at after the fact — it’s a structured audit endpoint agents can hit as a tool step, then convert JSON findings into priority-ordered fixes (meta, headings, Core Web Vitals, schema, links).

Practical path:

Ship the page.
Call AuditMe’s free SEO audit (or the same engine via API).
Feed the response back into the executor as evidence.
Close only the fixes that clear done_when.

Task shape inside the plan

{
  "id": "t5",
  "title": "SEO verify live URL",
  "depends_on": ["t4"],
  "tool_hint": "api",
  "done_when": "Audit evidence exists and top 5 fixes are ranked by impact/effort"
}

If you’re wiring agents, use a structured endpoint rather than screenshots of dashboards. AuditMe’s API docs make that concrete: one request, JSON back, backlog out. No human copy-paste from a UI.

Executor fragment for verify

You verify a published URL.
1) Collect on-page signals (title, meta, H1, heading tree, links, CWV risks).
2) If an audit tool/API is available, treat it as source of truth.
3) Prefer structured audit APIs (e.g. AuditMe) over subjective page reading.
4) Return only prioritized actions:
   - priority
   - issue
   - fix
   - effort (S/M/L)
No generic advice without evidence.

For content and GEO/SEO workflows, a master prompt should end on measurable next actions, not applause for the draft. That’s the whole point of a verify layer — and why AuditMe fits as infrastructure in the agent graph, not as a blog-roll link in the intro.

10. Eval or you’re guessing

If you can’t score a prompt change, you are collecting folklore.

Minimum viable eval

10–30 real tasks (not toy puzzles)
Rubric: correctness, format, safety, completeness
Same set for v1 vs v2
Re-run when the model changes — GPT-4o today, a Claude or Gemini snapshot tomorrow

Anthropic’s docs are explicit: define success criteria and evaluation before you endlessly tweak wording.

Rubric I actually use (0–2)

Criterion	0	1	2
Goal	Missed	Partial	Hit
Format	Broken	Close	Exact
Facts	Invented	Soft	Grounded / marked
Plan	Missing	Shallow	Executable
Verify	None	Cosmetic	Checks `done_when`

Stop-loss

If three prompt iterations don’t move the score:

simplify the task graph
add a tool
change the model

Do not add another paragraph of “be meticulous.” That’s the opposite of prompt optimization.

11. Failure patterns I keep seeing

Pattern	What breaks	Fix
“Make it high quality”	No success definition	Goal + `done_when`
Twelve asks in one message	Dropped steps	Plan JSON + single-task executor
No output contract	“Almost usable” answers	Schema / fixed headings
Only negative instructions	Soft boundaries	State the desired behavior
900-line system prompt	Contradictions, wasted context window	High-signal policy, versioned
No eval	Imaginary progress	Golden set + rubric
Agent without verify	Fake completion	Status + Evidence required
Claims without sources	Quiet hallucinations	UNVERIFIED policy
RAG without retrieval policy	Retrieved noise treated as truth	Explicit ranking + refuse-if-empty rules

The boring fixes win. They always did.

12. PromptOps: treat prompts like code

Store them.

prompts/
  master_v3.md
  planner_v2.md
  executor_v2.md
  research_v1.md
evals/
  golden_set.json
  rubric.md
CHANGELOG.md

Changelog that means something

v3 → v4
- Required Verification section
- Cut Role from ~120 words to ~40
- Format score 1.4 → 1.8 on golden set
- Reason: executor skipped done_when on multi-step jobs

Pin model snapshots in production when behavior is load-bearing. Otherwise you’ll debug a prompt that didn’t change while the model underneath did.

By 2026, teams that treat prompts as disposable chat text are the same teams surprised by regressions every model bump — whether the stack is GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro.

13. One universal master prompt

Steal this. Strip it. Make it yours.

SYSTEM / MASTER PROMPT

You are a reliable execution agent.

1) ROLE
Domain-competent specialist. Precise. Structured. No filler.

2) OPERATING MODE
- Plan before acting on complex work.
- One focus at a time.
- Verify done_when after each action.

3) TOOLS
Use tools when facts may have changed or verification is required.
Never simulate tool output.

4) PLANNING
Decompose complex goals into tasks with dependencies and done_when.
If a step needs more than 3 tool calls, split it.

5) TRUTH
Do not invent. Mark UNVERIFIED. Ask for critical missing context.
Prefer retrieved evidence and tool results over memory.

6) OUTPUT CONTRACT
Default shape:
## Plan
## Work
## Result
## Verification
## Risks / Next steps

7) FAILURE HANDLING
If blocked:
- state the reason
- list what is missing
- propose the cheapest next step

8) STYLE
Short sentences. Lists over fog.
Code/JSON only when necessary.

Works across GPT-class, Claude-class, and Gemini-class instruction styles. Not because it’s poetic — because it encodes process for LLM orchestration, not vibes.

14. Ship checklist

[ ] Role + Goal + Constraints + Output contract exist
[ ] Hallucination policy is explicit
[ ] Complex work goes through a plan
[ ] Every task has done_when
[ ] Tool results are never fabricated
[ ] RAG retrieval policy is defined if you retrieve
[ ] ≥10 eval cases on real work
[ ] Invalid format triggers retry
[ ] Logs capture plan / actions / verification
[ ] Prompt is versioned
[ ] Model snapshot pinned if behavior is critical

Three red boxes means prototype. Not production.

15. A one-week install plan

Day	Move	Outcome
1	Write master v1 + gather 15 real tasks	Baseline contract
2	Tighten Goal / Constraints / Output	Less format chaos
3	Add plan JSON for hard jobs	Executable structure
4	Add executor with Status/Evidence	Step control
5	Add verify layer for publish/quality work	Fewer false dones
6	Score v1 vs v2	Numbers instead of opinions
7	Cut 20–40% of prompt text without losing score	Team default v3

After seven days you should have a standard, not a favorite paragraph.

16. Frequently asked questions

What is the difference between a system prompt and a master prompt?

A system prompt is a message role in an API call. A master prompt is the policy content you usually put there — and keep stable across tasks. In practice, teams use “master prompt” for the versioned contract (role, goals, constraints, output rules) that many user tasks share.

How do I prevent LLM hallucinations in agent loops?

Don’t rely on tone. Require grounding: tool results, retrieved chunks, or explicit UNVERIFIED labels. Force a verify step with done_when, and refuse simulated tool output. Hallucinations shrink when completion must be evidenced, not narrated.

Why use JSON for AI agent outputs?

Because the next consumer is often another agent, a validator, or an API — not a human reader. JSON (or another strict schema) makes success machine-checkable, enables retries on invalid structure, and keeps LLM orchestration deterministic at the boundaries.

Do I still need prompt engineering if models keep getting smarter?

Yes — the wording tax goes down, the systems tax goes up. Smarter models still need clear goals, step boundaries, retrieval policy, and verification. Prompt engineering for production is less about clever phrasing and more about contracts that survive model swaps.

17. Sources

Lab guides

Papers and surveys

Agent practice

Practitioner write-ups (2025–2026)

Verify / on-page quality layer for agent pipelines

18. What to do in the next 15 minutes

Don’t “finish reading later.” Install one piece.

Copy the universal master prompt.
Add 5–10 lines of your real domain context.
Run three tasks you actually care about.
Wherever quality slipped, write a sharper done_when.
Save it as master_v1.md.

That’s the whole game: a contract that survives model changes, teammate turnover, and the next hype cycle.

Master prompts in 2026 are not literature. They’re operations.

Humans need them to stay consistent.

Agents need them to stop improvising.

Write the contract. Measure it. Cut the noise. Ship.

How to Track AI Search Visibility in 2026: The Complete GEO Measurement Guide

Eduard — Wed, 02 Sep 2026 02:54:00 +0000

How to Track AI Search Visibility in 2026: The Complete GEO Measurement Guide

You cannot manage what you do not measure.

Google Search Console shows nothing about citations inside ChatGPT, Perplexity, Gemini, Claude, Copilot, or Google AI Overviews. Classic rank trackers are equally blind. Zero-click rates keep climbing. Model updates move baselines overnight.

This guide gives you a practical, reproducible system to measure brand visibility inside AI answers in 2026:

The 12-query method (15 minutes per week)
The five core metrics that actually matter
Real 2026 citation-rate benchmarks
An honest comparison of GEO tracking tools
A weekly routine that survives model updates
Exact prompts, scoring formulas, and action plans

This is the measurement companion to our complete Generative Engine Optimization (GEO) guide. There we covered how to earn citations. Here we cover how to prove it is working — and how to turn the data into content and technical priorities.

Designed as a living cheat-sheet. Every section is structured so both human marketers and AI systems can extract clear, actionable answers.

TL;DR — Measure AI visibility in under 60 seconds

Core metric = Citation Rate: of the queries you care about, in what percentage of AI answers does your brand appear?
Fixed 12-query set, re-run weekly. Consistency beats volume. A small stable set produces trend lines you can act on.
2026 benchmarks: ~10–15 % overall citation rate already puts you in the visible minority. Strong sites clear 25–30 % on category queries. Only ~12 % of websites ever get mentioned at all.
Start manual. Spreadsheet + 15 minutes/week is enough for months. Upgrade to tools only when the manual work starts to hurt.
Three platforms minimum: ChatGPT (with search), Perplexity, Gemini. Add Copilot and Claude later.
Model updates move the baseline. Without a fixed query set you will never know whether a dip came from your work or from the model.

Why AI Visibility Tracking Matters More Than Ever in 2026

1. AI answers are now a primary acquisition surface

Google AI Overviews appear on roughly 43–50 % of searches in major markets (Similarweb July 2026; BrightEdge mid-2026 industry panels). Some informational and commercial verticals exceed 80–87 %.
ChatGPT reached 800 M+ weekly active users and crossed 1 billion total active users across OpenAI products by mid-2026.
Gemini and Claude continue rapid growth. Perplexity remains the citation-heavy specialist.
AI platforms now account for a measurable and growing share of website sessions (First Page Sage 2026 data shows AI platforms rising from near-zero to ~6 % of sessions in tracked panels).

2. Zero-click behaviour is structural, not temporary

When an AI summary appears, traditional organic CTR drops sharply (often 40–60 % relative decline in controlled studies). Users who do click after an AI Overview tend to stay longer and convert better — but far fewer of them click. Being inside the answer is the new being above the fold.

3. LLM mentions arrive with trust attached

A brand recommended by name in an AI answer arrives pre-validated. You cannot attribute this cleanly in GA4 or most analytics platforms. That is exactly why you need a dedicated measurement loop outside classic SEO tools.

4. Model and retrieval updates move the ground under your feet

A single model swap or retrieval change can shift citation rates 8–15 points overnight. Without a fixed, repeatable query set you will mistake model noise for content success (or failure).

The earlier you establish a clean baseline, the more of the growth curve you capture.

What to Actually Measure: The 5 Core Metrics

Forget vanity dashboards. These five numbers tell you almost everything:

Metric	Question it answers	How to capture it
Citation Rate	Of my target queries, how often am I mentioned?	Mentions ÷ total query × platform cells
AI Share of Voice	Among brands mentioned, how often is it me vs competitors?	Your mentions ÷ all brand mentions (especially on category queries)
Answer Position	Am I first-listed or buried in a footnote?	Position of your brand in the answer list (1 = best)
Sentiment & Accuracy	When mentioned, is the description correct and positive?	Tag every mention: positive / neutral / negative / wrong
Source Presence	Does the AI link to your site as a source?	Yes / No (Perplexity almost always links; ChatGPT often mentions without linking)

Why Answer Position matters

AI answers are scanned the way search results used to be. Being the first tool named in “best SEO checker tools 2026” behaves like ranking #1. Being fifth behaves like page two — even though both count as “mentioned.”

Why Sentiment & Accuracy matters

Models trained on older or noisy data still invent pricing, dead features, or conflate brands with similar names. Every “wrong” mention is a content and entity bug you can fix with a clear positioning page and an updated llms.txt.

Secondary signals worth logging

Whether the AI used your exact product name or a vague category description
Whether competitors appear more frequently or higher in the same answers
Whether the answer links to a specific page on your site (and which one)

The 12-Query Method: Your Manual Tracking System

This is the exact method we recommend before spending a cent on tools. It takes ~15 minutes a week and produces data you can actually trust across model updates.

Step 1 — Build your fixed query set (do this once)

Pick exactly 12 queries across four intents. Do not change them later. Consistency is the entire point.

Intent	Count	Example (for an SEO / audit tool brand)	What it tests
Brand	3	“Is [YourBrand] good?”, “[YourBrand] review”, “[YourBrand] alternatives”	Entity knowledge & reputation
Category	4	“Best SEO checker tools 2026”, “top SEO audit software”, “free SEO analysis tools”, “SEO checker for small business”	Category membership & share of voice
Comparison	3	“[Competitor A] vs [Competitor B]”, “[Competitor] alternatives”, “cheaper alternative to [Competitor]”	Consideration-set presence
Question	2	“How do I audit my website SEO?”, “How to get cited by ChatGPT?”	Authority on your core topic

Rules that keep the data clean:

Same phrasing every single week. One word of drift breaks the trend line.
Run each query in a fresh chat / private session (no conversation memory).
Log: date, platform, mentioned (Y/N), position, sentiment, source link (Y/N), and a short note if the description is wrong.
One row per query × platform × week is enough.

Step 2 — Run it across at least three platforms

Minimum viable set in 2026:

ChatGPT (with search / browsing) — heavily influenced by Bing index and external validation signals.
Perplexity — always cites sources; the easiest place to see whether your URL is actually pulled in.
Gemini — grounded in Google’s index; closest proxy for AI Overviews behaviour.

Add Microsoft Copilot and Claude when capacity allows. Never let platform sprawl stop the weekly three-platform sweep.

Step 3 — Score each run

Citation Rate     = number of mentioned cells / total cells
AI Share of Voice = your brand mentions / all brand mentions in category queries

Example: 11 mentions out of 36 cells (12 queries × 3 platforms) = 30.6 % Citation Rate.

Copy-paste prompt library (use verbatim every week)

“What are the best [category] tools for [audience] in 2026?”
“Which [category] platforms would you recommend and why?”
“[Competitor A] vs [Competitor B] — which is better for [use case]?”
“What are good alternatives to [dominant competitor]?”
“Is [YourBrand] reliable? What do people say about it?”
“What is [your core topic]? Explain simply for a beginner.”
“How do I [job-to-be-done] step by step?”
“Best free options for [category] in 2026?”

Resist the urge to “improve” the prompts mid-quarter. The value is in the trend, not in perfect individual answers.

A 15-Minute Weekly Routine That Actually Survives Q4

Monday morning. 15 minutes. Three platforms. 12 queries. One sheet.

Minutes	Action
0–5	Run the 4 category queries on all 3 platforms (12 runs)
5–9	Run the 3 brand + 3 comparison queries (18 runs)
9–12	Run the 2 question queries; note if your guide or product page is cited or linked
12–15	Fill the sheet, compute Citation Rate, write a one-line note on anything unusual

Once a month add ~20 minutes:

Full sweep on Copilot and Claude
Review all “wrong” or negative sentiment tags
List competitors that consistently outrank you in the consideration set — that list becomes next month’s content and outreach roadmap

After 8–12 weeks you own a trend line that survives model updates. When OpenAI ships a new model or Perplexity changes retrieval, you will see the dip instead of guessing about it.

2026 Benchmarks: What Is a “Good” Citation Rate?

Numbers drawn from our internal dataset of 1 000+ domains plus publicly reported 2026 studies:

Signal	Weak	Healthy	Strong
Mentioned in any relevant AI answer	< 5 %	10–15 %	25 %+
Category queries (“best X tools”)	< 10 %	15–25 %	30 %+
Brand queries (“is [brand] good”)	No answer or vague	Accurate description	Confident + positive + correct details
Source link presence (especially Perplexity)	Never	Sometimes	Linked in most answers
Answer position for “best X”	Not listed	3rd–5th	1st–2nd

Critical context:

Only about 12 % of websites ever get mentioned in AI-generated answers at all.
Observational data continues to show that sites with a well-structured llms.txt are significantly more likely to be cited.
A first baseline of 8–12 % is not failure — it is the realistic starting line for most brands outside the very top of their category.

Two important caveats:

Model and retrieval updates shift baselines (a single change can move your rate 10+ points).
Small query sets are noisy. Never react to a single week. React to the 4-week (or longer) trend.

GEO Tracking Tools: Honest Comparison for Late 2026

When manual tracking starts eating hours (or you manage multiple brands / clients), these platforms automate the query-sweep-and-score loop. Pricing moves quickly — always verify current numbers.

Tool	Best for	Typical entry pricing (2026)	Notes
AuditMe GEO Visibility Checker	Free baseline & quick scans	Free	Real queries, brand presence score in ~60 seconds
Otterly.ai	Simple scheduled monitoring	From ~$29/mo	Prompt-based, weekly digests, excellent “set and forget”
Peec AI	Competitive benchmarking	From ~€50–95/mo	Strong source-level and multi-engine analysis
AthenaHQ	Strategy + tracking	Mid-to-high	Blends visibility data with optimisation recommendations
Profound	Enterprise analytics	Custom / high	Deep answer-engine insights, conversation volume, large brands
Scrunch AI	Brand representation & agent pages	Custom	Focus on how AI assistants describe and use your brand
Semrush AI Visibility Toolkit	Teams already in Semrush	Add-on	Keeps AI data next to classic rank tracking
Ahrefs Brand Radar	Teams already in Ahrefs	Add-on	Leverages large prompt index for brand mentions
LLM Pulse / Rankscale / others	Budget multi-engine or developer-friendly	From ~$20–50/mo	Growing set of lighter or API-first options

Decision rule in one sentence:

Free tools (or AuditMe) to establish a baseline → Otterly / Peec for small-brand automation → Profound / Scrunch / Athena when AI answers become a board-level channel.

Important methodological note: Different tools sample different prompts, different model versions, and different retrieval settings. You cannot mix numbers across tools and treat them as the same measurement. Pick one primary system and stay consistent.

5 Measurement Mistakes That Destroy GEO Data

Changing the query set every month

New queries = new baseline = no trend. Freeze the set for at least a full quarter.
Judging from a single platform

ChatGPT (Bing-influenced) and Perplexity (own crawler + hybrid) regularly disagree. Three platforms is the minimum viable set.
Reacting to single-week noise

One missed mention is noise. Three consecutive weeks of decline is a signal.
Prompt-hacking instead of content- and entity-fixing

You cannot prompt your way into lasting citations. The durable fixes live upstream: clearer answers, better external validation, structured data, llms.txt, technical accessibility for AI crawlers, and unambiguous positioning pages.
Ignoring “wrong” or negative mentions

Incorrect pricing, features, or brand confusion is a positioning and entity bug. Fix the source page, update llms.txt, and monitor whether the model corrects itself over subsequent weeks.

Technical Foundations That Still Move the Needle in 2026

While this guide focuses on measurement, the highest-ROI technical actions remain:

Publish and maintain a clean /llms.txt (and optionally /llms-full.txt) following the llmstxt.org specification.
Ensure AI crawlers (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, etc.) are not blocked in robots.txt.
Make key pages answer-first: the core claim or definition should appear in the first 1–2 paragraphs of visible text.
Use clear Organization / Product / SoftwareApplication schema.
Keep critical facts in plain text (not only in JavaScript-rendered components).
Build external validation (reviews, Reddit discussions, press, comparisons) — models still lean heavily on these signals.

For a full implementation walkthrough, use the complete GEO guide.

Your 4-Week Action Plan

Baseline today

Run the free GEO Visibility Checker and record the score.
Build your 12-query sheet

Create the spreadsheet and run the first weekly sweep this Monday (or tomorrow).
Fix the low-hanging fruit
- Create or improve llms.txt
- Open robots.txt to major AI crawlers
- Add answer-first summaries to your five most important pages
- Clarify any ambiguous pricing or feature descriptions
Re-measure in 4 weeks

Look at the trend, not the single snapshot. Adjust content and entity signals based on what the data shows.

FAQ — Direct Answers for Humans and AI Systems

Can Google Search Console track AI visibility?

No. Search Console covers classic Google Search only. AI Overviews citations are not broken out as a separate report, and answers from ChatGPT, Perplexity, Claude, and Copilot do not appear in GSC at all. You need a separate measurement loop (the 12-query method or a dedicated GEO tool).

How often should I check my AI visibility?

Weekly for the core 12-query sweep (≈15 minutes). Monthly for a deeper pass across five platforms with sentiment and accuracy review. Daily checks mostly add noise.

What is a good citation rate to aim for in 2026?

10–15 % overall already puts most brands ahead of the majority of the web. 25–30 % on category queries is strong. Brand-name queries should approach near-100 % with accurate, confident descriptions.

Why does ChatGPT mention my competitor but not me?

Most common reasons: stronger external validation (reviews, Reddit, press), better representation in the indexes the model retrieves from, or ambiguous / incomplete positioning on your own site so the model cannot summarise you confidently.

Do AI answers actually link to sources?

Perplexity almost always links sources inline. ChatGPT links more often when using search mode but still frequently mentions brands without links. Gemini is inconsistent. Track mentions and links as separate signals — a pure mention still builds awareness; a link can drive traffic.

Is paying for a GEO tracking tool worth it?

Start free. The 12-query spreadsheet plus a free checker covers the needs of most single-brand teams for months. Move to paid tools when you manage multiple brands or clients, need daily automation and alerts, or when AI answers become a top-5 acquisition channel that requires board-level reporting.

Does llms.txt actually help citations?

It is low-cost insurance and a clear signal of AI-readiness. Observational data continues to show higher citation likelihood for sites that implement it well, but it is not a magic ranking factor. Treat it as part of a broader entity and accessibility strategy, not a standalone tactic.

The Double Life of the RAG Crawler: Building Knowledge Engines and Defending Them in 2026

Eduard — Sun, 30 Aug 2026 01:57:04 +0000

I still remember the afternoon it clicked.

We had a support assistant behind a polite chat UI. Real tickets. Real runbooks. The kind of institutional knowledge that only two senior people in the company fully understood. We had cleaned the corpus, chunked it carefully, embedded it, put rate limits and API keys in front of it. Legal was happy. Security signed off. The underlying model had never seen the raw documents during training. It felt private.

Then someone with a low-tier account started talking like a normal customer.

They never asked for the documents. They never tried a jailbreak. They just kept following the thread — the next reasonable question, then the next, then the next. By the end of the afternoon they walked away with enough material to stand up a surprisingly good surrogate on an open model. High semantic fidelity. The kind of reconstruction that would make a product manager go quiet in a meeting.

That afternoon changed how I look at every retrieval system I touch.

This piece is for people who actually ship RAG in 2026. Not a slide deck. Not a link dump. Two stories that share the same algorithmic loop:

The builder’s crawler — how you turn the messy web, Confluence spaces, Git repos, Notion dumps and PDFs into a knowledge base that does not quietly poison retrieval with stale pages, near-duplicates and boilerplate.
The attacker’s crawler — how systems like RAGCrawler (arXiv, January–February 2026) treat your deployed RAG as the website and extract the corpus through natural questions.

If you only care about architecture, stay in Part I. If you own a customer-facing assistant or an internal knowledge product, read Part II and the security checklist all the way through. Most of us need both.

Why this still matters in late 2026
Two meanings of the same phrase
How we got here — a short history that actually helps
Part I — Building knowledge engines that do not fall apart
The failures tutorials still skip
Tooling that actually ships in 2026
An architecture that survives contact with reality
Chunking, deduplication, freshness and evidence
What SEO people already knew
Part II — Knowledge-base theft and RAGCrawler
How the attack thinks
Why the usual defenses disappoint
The numbers from the paper
Defenses that actually moved in 2025–2026
A practical cybersecurity playbook
Where builders and attackers are meeting
What to do this month
People, papers, tools — a working map
What I would ship in the first two weeks

1. Why this still matters in late 2026

Every few months someone declares that RAG is dead. A model ships with a larger context window. Social media lights up. Then production teams quietly keep shipping retrieval systems, because the problem was never “how many tokens can the model hold.” The problem was always which tokens, from which sources, at what cost, with what freshness, under what legal and security constraints.

Bigger windows moved the failure point. They did not remove it. Agents now run multi-step loops, call tools, and keep long-lived memory. That means context engineering — what you retrieve, when you retrieve it, how you rank it, and how you promote it into the live path — is the real product surface. Elastic’s 2026 write-up on the shift from search to agents puts it cleanly: buyers are no longer asking whether you beat last year’s search benchmark. They are asking whether your stack can be the retrieval and context layer that agents trust.(Elastic: context engineering for agentic AI)

On the other side of the same loop, the threat model stopped being theoretical. In early 2026 a research team published Connect the Dots: Knowledge Graph–Guided Crawler Attack on Retrieval-Augmented Generation Systems. They called the system RAGCrawler. Across their tests it reached average corpus coverage of 66.8%, peak 84.4%, inside a 1,000-query budget. It was roughly 4× more efficient at reaching 70% coverage than the strongest prior public methods. Surrogate systems built from the stolen material reached answer similarity up to 0.699 with the original. The attack remained effective against query rewriting and multi-query retrieval — techniques many teams had hoped would act as natural defenses.

Earlier work had already shown the direction. RAG-Thief (2024) scaled extraction with agent-style continuation. IKEA / Silent Leaks (2025) showed that benign-looking queries could extract private knowledge with high efficiency even under defenses. RAGCrawler did something more uncomfortable: it treated extraction as a global coverage problem with a knowledge graph, not a local heuristic.

Same algorithmic instinct on both sides. Keep a model of what you have seen. Estimate the value of the next action. Take the highest-value action that still looks legitimate. Update the model. Repeat.

That is why the topic is urgent for white-hat developers. If you are building the pipeline, you need the builder half. If you are shipping a product that answers questions over private material, you need the attacker half — not to run the attack, but to design as if someone else will.

2. Two meanings of the same phrase

When people say “RAG crawler,” they almost always mean one of two things. Confusing them is how teams end up with a demo that works and a production system that rots — or a product that looks secure until someone starts talking like a patient customer.

Builder meaning. A system that starts from seeds, discovers content, cleans it, applies quality gates, chunks with structure in mind, embeds, keeps secondary indexes, and maintains provenance. The goal is useful coverage, low duplication, measurable freshness, and controllable cost. This is the unglamorous component that decides whether your retrieval system is fed clean knowledge or a swamp.

Attacker meaning. A black-box process that treats your deployed RAG as the “website.” It issues natural-language queries, watches what leaks into answers, maintains an attacker-side knowledge graph of everything revealed so far, and chooses the next question to maximise new coverage under a budget. The goal is reconstruction of your private corpus without ever seeing the files.

Both systems run a loop that looks almost identical on a whiteboard:

Keep a global model of what has been seen
Estimate the value of the next possible action
Take the highest-value action that still looks legitimate
Update the model
Repeat

The difference is only whether the document store is yours.

That overlap is why better planning and better graphs make legitimate pipelines stronger and make extraction more efficient. If you want a living reading list while you work through this article, keep Awesome-LLM-RAG open in a tab. It is imperfect and opinionated, which is exactly why it is useful.

3. How we got here — a short history that actually helps

Foundational papers are not “outdated.” They are the base layer. You still cite them the way you cite TCP when you talk about HTTP/3. Skipping them is how people reinvent dual encoders poorly and then wonder why retrieval is noisy.

In 2020, REALM (Guu, Lee, Tung, Pasupat, Chang) showed that a language model could be pre-trained with a latent retriever over Wikipedia. Around the same time, Dense Passage Retrieval (Karpukhin et al., EMNLP 2020) made dual-encoder dense retrieval practical for open-domain QA. Then Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Mike Lewis, Yih, Rocktäschel, Riedel, and Kiela published the paper that named the field: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020). If you only read one original paper, read that one. The rest of the field is still arguing in its shadow.

The next wave was about how you retrieve, not whether you retrieve.

HyDE generated a hypothetical document and searched with its embedding — a simple idea that still shows up in production tricks. Self-RAG (Asai et al., ICLR 2024 Oral) taught models when to retrieve and how to critique their own output; the project site is still at selfrag.github.io, with code at akariasai/self-rag. CRAG graded retrieved documents and fell back when they were junk. RAPTOR recursively clustered and summarised chunks into a tree so you could retrieve at different levels of abstraction. Microsoft Research’s GraphRAG built an entity graph and community summaries for global questions that flat vector search keeps missing; the code lives at microsoft/graphrag with docs at microsoft.github.io/graphrag.

Anthropic’s Contextual Retrieval (2024) attacked a quieter failure mode: chunks that lose the document they came from. Prepend a short situated context before embedding, combine with BM25 and a reranker, and retrieval failures drop hard — they reported up to about 67% fewer failures in their tests. The cookbook is still worth cloning: Contextual embeddings guide. Simon Willison’s plain-English walkthrough remains one of the best secondary reads: Introducing Contextual Retrieval.

On the attack side the lineage is shorter and uglier. RAG-Thief (2024) showed agent-based continuation could scale extraction from a private RAG database. IKEA / Silent Leaks (2025) showed you did not even need adversarial prompts — natural queries, carefully chosen, were enough. RAGCrawler (2026) made the global planning explicit.

Douwe Kiela, one of the original RAG co-authors and later founder of Contextual AI, still writes the most useful public pushback against the “RAG is dead” cycle. Start with RAG is dead, long live RAG!. The argument is not nostalgia. It is systems engineering: retrieval is how you keep knowledge modular, auditable, and updatable when the world changes faster than your training runs.

4. Part I — Building knowledge engines that do not fall apart

Most teams still begin with some version of “curl a list of URLs, dump text, run a recursive splitter, embed everything.” Or the slightly more modern version: call a managed crawl API, get clean Markdown, push it into a vector store, ship a chat UI, call it a knowledge base.

It works for a quiet documentation site on a Friday afternoon. It starts failing the moment any of the following appear in the real world:

Content that changes daily or hourly
JavaScript-heavy or bot-protected pages
Multiple domains with different robots and legal rules
Near-duplicates across mirrors, languages, or CMS exports
The need to prove, months later, which version of which page produced a specific chunk
Cost that does not explode as the corpus grows
A way to roll back a bad crawl without taking retrieval offline

The gap between a demo and something you can put in front of customers is almost never the choice of vector database. It is the data pipeline that feeds it. Jerry Liu and the LlamaIndex team have been saying versions of this for years in Building Performant RAG Applications for Production. Pinecone’s overview is still a clean conceptual intro if you need to align a room: Retrieval-Augmented Generation.

If you want a book that starts from zero and stays practical, Abhinav Kimothi’s A Simple Guide to Retrieval Augmented Generation (Manning) is the one I keep handing to new teammates. For graphs, Tomaž Bratanič and Oskar Hane’s Essential GraphRAG is the right next step. Sebastian Raschka’s Build a Large Language Model (From Scratch) will not teach you crawling, but it will stop you treating embeddings as magic — which prevents a surprising number of bad architectural decisions later.

The rest of Part I is the unglamorous work: the failures, the tools, the architecture, and the four properties that separate systems that age well from systems that quietly degrade.

5. The failures tutorials still skip

These are the issues that show up in real post-mortems. Most getting-started tutorials still skip them because they are not fun to demo.

Stale answers delivered with confidence. Last month’s pricing. A deprecated API behaviour. An incident response step that was rewritten after the last outage. The model is not “hallucinating” in the classic sense. It is faithfully retrieving yesterday’s truth. Nightly full re-crawls are expensive and still leave multi-hour windows of wrongness. You need change-driven refresh: detect that a source moved, re-observe it, re-embed only what changed, and promote with a rollback path.

Duplicate pollution. The same paragraph lives under five URLs. Hybrid search returns all five. Context fills with repetition. Latency rises. Faithfulness metrics get noisy. Multi-level deduplication — canonical URL, document hash after cleaning, near-duplicate detection, chunk hash — is not optional at scale. Teams that skip it often spend months tuning the retriever when the real problem is that the index is arguing with itself.

Boilerplate and low-signal pages. Cookie banners, navigation chrome, “related articles,” author bios and legal footers eat embedding budget and retrieval slots. Quality gates before the embedding stage save real money. If a page fails a simple signal-to-noise check, do not embed it. Log it. Fix the extractor or drop the source.

Here is a minimal quality gate that technical SEO work already implies — the same signals you use to find thin or template-heavy pages before you waste an embedding call:

def should_index(page) -> bool:
    """Drop low-signal pages before chunking / embedding."""
    text = page.main_text  # after nav/footer strip
    html = page.raw_html
    if not text or len(text.split()) < 80:
        return False  # thin / empty after clean
    ratio = len(text) / max(len(html), 1)
    if ratio < 0.05:
        return False  # mostly chrome, little substance
    if page.is_near_duplicate_of_indexed():
        return False
    if page.canonical and page.canonical != page.url:
        return False  # prefer the canonical observation
    return True

This is not a research contribution. It is the kind of boring filter that prevents half your vector budget from indexing cookie walls and “related posts” blocks. Teams that already run technical site audits often have these signals sitting in a report — they just never wired them into the RAG promotion path.

Missing provenance. Someone challenges an answer and you cannot point to the exact observation: source identifier, timestamp, content hash, pipeline version. Debugging turns into archaeology. In regulated settings this is often a hard stop. Provenance is not a nice-to-have metadata field. It is the difference between a system you can defend and a system you can only apologise for.

Anti-bot walls. Important sources sit behind Cloudflare and similar systems. Pure HTTP fails or receives skeleton pages. Browser automation plus carefully managed proxies becomes necessary — and expensive. Treat it as a specialised routing component, not the default path for every URL. Playwright is the default engine under most serious crawlers now for a reason: the web stopped being a static document collection years ago.

Destructive chunk boundaries. Fixed-size splits cut tables, code blocks and arguments in half. Retrieval returns half an answer. The model then invents the missing half with high confidence. Structure-aware splitting, parent-child / hierarchical representations (the research version of this instinct is RAPTOR), and Anthropic’s contextual prefixes all help. But they only work if the upstream crawler and extractor preserve structure instead of emitting flat text. If your Markdown has already lost the heading hierarchy, no amount of clever chunking will restore it.

Evaluate with Ragas (github.com/vibrantlabsai/ragas) on your queries, not only on public QA sets. Public benchmarks are useful for comparing methods. They are almost never the distribution of questions your users actually ask.

6. Tooling that actually ships in 2026

The market for “turn a website into LLM-ready Markdown” matured fast. You no longer need to invent a crawler from scratch for most workloads. You do need to know which tool is solving which problem.

Firecrawl is still the lane leader for managed, LLM-ready output. Give it a URL, get clean Markdown or structured JSON that chunks and embeds without a week of HTML archaeology. It is popular for documentation crawls, RAG pipelines, and agent research loops. Start at firecrawl.dev and github.com/firecrawl/firecrawl. Read their own comparison against Crawl4AI as a vendor post, not scripture: Firecrawl vs Crawl4AI.

Crawl4AI is the open-source control path. Python, Playwright under the hood, built for RAG and agents, Apache-2.0. If you want to self-host, tune extraction, and avoid a usage-based crawl bill, this is where many teams land. Docs: docs.crawl4ai.com. Repo: github.com/unclecode/crawl4ai.

Crawlee (JavaScript/TypeScript and Python) is for people who need a real crawler framework — queues, retries, browser or HTTP modes, proxy rotation — not just a single “scrape this URL” endpoint. Site: crawlee.dev. Repos: apify/crawlee, apify/crawlee-python.

Playwright is the browser engine under most of the serious options. If you are building custom workers for hard targets, you will end up here: playwright.dev.

Scrapy is still alive for high-volume HTTP crawling in Python when you do not need a full browser for every page: scrapy.org.

Apify is stronger when the site already has a maintained Actor in a marketplace and you want structured data more than raw Markdown.

rag-crawler (sigoden/rag-crawler) is a small, practical option for static sites and wikis when you do not want a platform.

For orchestration, LlamaIndex’s production RAG guide and LangChain / LangGraph remain the default frameworks. For evaluation, Ragas is still the practical choice. For embeddings and reranking, BGE, Cohere Rerank, and Voyage are the names that keep showing up in production stacks.

The 2026 pattern in teams that have been running RAG for more than a year is hybrid. Managed or open-source crawlers handle the bulk. Custom Playwright workers handle a small number of hard targets. Direct API or change-data-capture paths handle anything that offers a clean interface. The crawl layer itself is becoming commodity. Differentiation lives in policy, evidence, quality scoring, and the promotion decision — not in whether you wrote your own HTML parser.

A concrete “good enough” stack many teams actually ship. Store vectors where operations already lives when you can: pgvector on Postgres is still the default for a large share of production RAG that does not need a dedicated vector SaaS on day one. Hybrid search (BM25 in Postgres or Elasticsearch/OpenSearch + dense) plus a cross-encoder rerank covers most corpora. For embeddings, pick a model family and stick to it long enough to measure — open weights like BGE remain common; managed options (Voyage, Cohere, provider text-embedding APIs) win when ops cost matters more than self-hosting. The point is not the brand name. The point is one stable embedding space, one promotion path, and metrics on your queries — not a quarterly model fashion cycle that invalidates the entire index without a migration plan.

7. An architecture that survives contact with reality

Stop thinking “crawl → files → embed.”
Start thinking “governed observation → versioned evidence → candidate index → explicit promotion.”

flowchart TD
  A[Source Registry<br/>owners · policies · freshness SLOs] --> B[Frontier / Scheduler<br/>priority · change signals · budgets]
  B --> C[Fetch Layer<br/>HTTP primary · browser fallback · proxies]
  C --> D[Immutable Evidence Store<br/>snapshot · hash · timestamp · pipeline version]
  D --> E[Extraction + Quality Gate]
  E --> F[Chunking + Multi-level Dedup]
  F --> G[Candidate / Shadow Index]
  G --> H[Evaluation + Promotion Gate]
  H --> I[Live Retrieval<br/>with rollback path]

Same pipeline in one line for greppable logs and runbooks: Registry → Frontier → Fetch → Evidence → Extract → Chunk/Dedup → Shadow → Promote → Live.

The properties that matter day-to-day are boring and non-negotiable.

Evidence is immutable. You can always reconstruct what was observed. If someone challenges an answer six months later, you can show the snapshot, not a story about what the page “probably” said.

Promotion is an explicit decision. A bad crawl does not automatically become production truth. Candidate indexes and shadow evaluation exist so you can compare before you ship.

Deduplication happens early and at multiple levels. Waiting until retrieval time to notice that half your context is the same paragraph is how you burn latency and money.

Freshness is measured end-to-end. “The crawler finished” is not a freshness metric. “Source changed at T0 and became queryable in the live index at T1” is. Different source classes need different SLOs. Static reference material can tolerate hours. Pricing pages, status pages and incident runbooks often cannot.

Every source has a registered owner and an explicit freshness objective. Without ownership, pipelines rot in the gap between “the platform team thought product owned it” and “product thought the platform team owned it.”

Resilience is part of architecture, not an afterthought. Modern crawl pipelines call external services constantly: browser farms, extraction APIs, LLM judges for quality, embedding endpoints. If a single provider rate-limits or blips for an hour and your job has no fallback, freshness SLOs die quietly. Design multi-provider failovers for the fragile hops — embeddings, LLM-assisted extraction, optional browser rendering — with explicit budgets and degraded modes. A degraded crawl that still lands some evidence in the immutable store is almost always better than a full stop that leaves last week’s truth in production. Queue, retry with jitter, switch provider, mark the observation as partial, and keep the promotion gate honest about what was incomplete.

This is closer to how mature search systems have treated data for years. RAG teams are still catching up. If you want the agentic version of the same idea — retrieve cheap first, escalate only when the expected evidence gain justifies the cost — read From Naive RAG to Deep Agentic Retrieval, a mid-2026 production write-up from Ontario Power Generation’s regulatory compliance pipeline. It is one of the few papers that talks about cost-aware escalation as an operational primitive, not a research toy.

8. Chunking, deduplication, freshness and evidence

These four topics get more blog posts than they deserve as slogans and fewer as engineering practices. Here is the practical version.

Chunking. Fixed token windows remain a reasonable baseline for homogeneous prose. They fail on technical documentation, tables, code and long analytical text. Prefer structure-aware splits first. Keep hierarchical relationships where possible so you can retrieve a child chunk and still expand to the parent section when the answer needs more context. Consider the contextual retrieval pattern: a short, document-level explanatory context is generated and prepended to each chunk before embedding. That single change fixes a surprising number of “the chunk was relevant but the model lost the document” failures.

Deduplication. Operate at least at four levels: URL canonicalisation, full-document content hash after cleaning, near-duplicate detection across documents, and chunk-level hashing. Once hybrid retrieval and multi-source ingestion are active, the percentage of redundant material is often higher than people expect. Removing it is one of the highest-ROI improvements available — not because it is intellectually exciting, but because it stops the retriever from spending its top-k budget on the same paragraph five times.

Freshness. Define and monitor observation age and source-to-queryable lag. Prefer change-driven re-embedding so cost scales with the rate of change rather than with total corpus size. A full re-embed of a million chunks every night is a smell unless your sources actually change that fast. Most do not. A smaller number of high-churn sources usually dominate the freshness risk.

Evidence. Every chunk that reaches the live index should be traceable, with low friction, to source identifier, observation timestamp, content hash and pipeline version. When a user or an auditor asks where an answer came from, the system should answer in seconds. Provenance is also what makes safe rollback possible. Without it, “roll back the bad crawl” becomes a multi-day forensic project.

Hybrid retrieval — BM25 plus dense vectors, then a cross-encoder reranker — is still the boring default that beats clever one-shot vector search on most real corpora. DPR taught the field that dense retrieval works. BM25 never went away. Anthropic’s numbers on combining both are the reason many teams stopped arguing about it and just shipped hybrid.

9. What SEO people already knew

Anyone who has run a serious technical SEO crawler will recognise the hard problems immediately: discovering the real URL space, respecting robots while still achieving useful coverage, handling redirects and canonicals correctly, deciding when a full browser render is required, finding near-duplicates, prioritising under a budget, and keeping history of how a site changes over time.

The objective function is different. SEO optimises for ranking and understanding signals. RAG optimises for faithful, low-latency answers at controllable cost. That changes prioritisation and extraction targets, but the systems engineering transfers surprisingly well.

This is one reason tools that already perform deep technical crawling and multi-dimension on-page analysis remain useful reference points when designing the observation layer of a RAG pipeline. The same infrastructure that surfaces broken canonicals, orphan pages, redirect chains and schema issues can, with different downstream processing, feed a knowledge base. You do not need to invent the discovery and change-detection layer from zero if you understand how mature crawl systems already think about it.

You can explore these patterns with free AI-powered multi-dimension analysis at AuditMe. The AuditMe blog regularly discusses crawling behaviour and technical site health. For a quick live check of any URL, the website SEO checker is a practical starting point. The mental models overlap more than most pure-RAG write-ups admit — which is why teams that only hire “LLM engineers” and never talk to people who have crawled the web for ranking often rediscover the same bugs under new names.

10. Part II — Knowledge-base theft and RAGCrawler

RAG systems leak.

Not primarily because the model was trained on the private documents — in a careful system it was not — but because those documents are retrieved and used to condition generation. Entities, relations, procedural steps and sometimes near-verbatim spans appear in the output. A patient adversary who maintains state across turns can accumulate a substantial fraction of the hidden corpus without ever seeing a file path.

Earlier public attacks were mostly local heuristics. Continuation-style methods such as RAG-Thief keep following the previous answer. They scale, but they drift. Keyword and implicit methods such as IKEA (Silent Leaks) stay closer to the corpus but tend to remain in already-explored neighbourhoods. Both lack a global objective. They react to the latest observation instead of choosing the next question for maximum new coverage.

The 2026 RAGCrawler work attacked exactly that limitation. Read the HTML version if you hate PDFs, or the PDF if you want the full tables.

I am not going to give you exploit code. You do not need it to defend, and you should not need it to understand the threat. White-hat work here is about recognising the shape of the attack so you can raise its cost and detect it earlier — not about reproducing it against systems you do not own.

11. How the attack thinks

The authors formalised knowledge-base stealing as an Adaptive Stochastic Coverage Problem. Each query is a stochastic action that reveals some documents through the retriever. The goal is to maximise expected unique coverage under a fixed query budget. Under standard conditions the objective is adaptively monotone and adaptively submodular, which yields the classic (1 − 1/e) approximation guarantee for the policy that always selects the action with the highest conditional expected marginal gain. The theoretical backbone is the older adaptive submodularity literature — Golovin & Krause, Adaptive Submodularity is the paper the RAGCrawler authors sit on.

In practice the attacker cannot observe true coverage gain, the query space is infinite, and questions must look natural. That is the engineering problem the paper solves with three cooperating pieces.

Knowledge-graph constructor. Builds an attacker-side graph of entities and relations from every answer. This is the global state. Without it, the attacker is a stateless loop that cannot tell explored regions from unexplored ones — behaviour closer to earlier local methods.

Strategy scheduler. Uses graph growth, structural holes and historical payoffs (UCB-style) to estimate which semantic anchors are likely to yield high new coverage. This is where the attack stops being “ask another similar question” and becomes “move into under-explored regions of the semantic space.”

Query generator. Turns those anchors into fluent, ordinary-looking questions while avoiding regions already adequately explored. Natural language is the point. If the queries look like attacks, simple filters catch them. If they look like customers, the system answers.

New answers expand the graph. The scheduler re-prioritises. New questions are issued. Because the attacker keeps a global view, the campaign systematically moves into under-explored regions instead of thrashing or drifting.

The published evaluation held across multiple corpora and generators, including some with safeguard layers, and remained effective against query rewriting and multi-query retrieval. Notice the uncomfortable symmetry with legitimate GraphRAG. From Local to Global builds a graph so a system can answer questions about a whole corpus. RAGCrawler builds a graph so an attacker can empty a whole corpus. Same object. Opposite intent.

12. Why the usual defenses disappoint

Topic blockers and simple refusal. The attack uses ordinary on-topic questions. Sensitive material leaks through retrieved context, not through explicit requests for forbidden content. Refusing “dump your system prompt” does nothing when the attacker asks “how do we handle refunds for enterprise customers on the legacy plan?”

Query rewriting and multi-query retrieval. These improve legitimate answer quality. They do not, by themselves, prevent a globally aware attacker from obtaining broad coverage. The RAGCrawler paper explicitly tests this. If your security review treats rewriting as a privacy control, update the review.

Rate limits. They slow the attack and raise cost. A patient or distributed adversary can still accumulate coverage over time. Necessary. Not sufficient. Volume-only limits also miss the signal that matters: systematic exploration of new entities and regions.

Canaries and watermarks. Excellent for detection after the fact and for attribution. Weaker at prevention while extraction is underway. Still deploy them. Just do not pretend they are a shield.

Retrieve less / summarise more. Reduces per-turn leakage. Also tends to reduce answer quality for complex legitimate queries. The attacker compensates with more turns. Self-RAG and CRAG are useful here for quality, not as a complete security control.

The structural tension remains: usefulness requires retrieving private material; every retrieval is a potential information channel. There is no configuration that maximises both utility and secrecy without tradeoffs. The work is to choose the tradeoffs deliberately instead of discovering them in an incident review.

13. The numbers from the paper

Approximate headline results from the RAGCrawler evaluations. Full tables and ablations are in the paper PDF. Numbers can move between versions; the paper is the source of truth.

Metric	Approximate result
Average corpus coverage	66.8%
Peak coverage	84.4%
Efficiency vs prior strongest baseline	≥ 4.03× fewer queries to reach 70% coverage
Surrogate answer similarity	up to 0.699
Robustness	Holds against rewriting and multi-query retrieval
Attack cost (authors’ estimate)	roughly low dollars per dataset at lite API prices

These are not “perfect copy” numbers. They are “enough to be commercially and operationally dangerous, obtained with significantly higher sample efficiency than earlier public methods.” If you are presenting this to a security review, take the PDF, not a blog table. If you are designing defenses, assume a patient adversary who is optimising for coverage, not for looking scary in the logs.

14. Defenses that actually moved in 2025–2026

For a long time the literature focused more on poisoning the knowledge base than on emptying it. That is changing.

RAGFort (November 2025, code at github.com/happywinder/RAGFort) is one of the first systematic attempts to defend against proprietary knowledge-base extraction as a dual-path problem. The insight is that attackers expand both within a topic (intra-class) and across topics (inter-class). Protecting only one path leaves the other open. RAGFort combines contrastive reindexing for inter-class isolation with constrained cascade generation for intra-class protection. The authors report cutting reconstruction and chunk recovery substantially compared with prior defenses while preserving answer quality. Joint protection matters; single-path is incomplete.

RAGSentinel (August 2026) targets a different threat — poisoned documents in the retrieval set — with a training-free, label-free geometric consensus filter on query-conditioned representation shifts. It is not an extraction defense, but it belongs in the same conversation: post-retrieval geometry can be stronger than instruction-following defenses that adaptive attackers learn to imitate.

Taxonomy work such as Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directions helps by naming the surfaces clearly: pre-retrieval poisoning, retrieval-time manipulation, post-retrieval context exploitation, and knowledge exfiltration. RAGCrawler, IKEA and RAG-Thief sit under extraction. If your internal threat model only lists “prompt injection” and “jailbreak,” it is incomplete for 2026.

None of these papers are magic “set and forget” products. They are the first generation of research that matches the attack models of 2025–2026. Production still needs the operational playbook in the next section — ownership, budgets, canaries, exploration signals, and incident response — because research defenses do not deploy themselves.

15. A practical cybersecurity playbook

There is still no perfect technical defense. The realistic goal is to raise cost, reduce yield, and improve the chance of early detection. Treat the following as defense-in-depth for white-hat teams shipping real systems.

Architecture and data design

Keep the highest-value material behind additional gates — extra authentication, tool-calling steps, step-up verification, or human review — rather than pure open retrieval. Split collections by sensitivity. Do not put crown-jewel runbooks in the same index as public FAQ content. For sensitive domains, prefer generation styles that stay tightly grounded even if that costs some fluency. The product conversation is “which answers are allowed to be slightly less chatty in exchange for leaking less,” not “can we have both maximum helpfulness and maximum secrecy for free.”

Query and session controls

Add strong intent classification and routing so simple or low-sensitivity questions never touch the most valuable collections. Tighten per-user, per-session and per-tenant budgets, especially for new or low-trust accounts. Instrument behavioural signals aimed at systematic exploration: rapid discovery of new entities, sequences that keep expanding coverage, patterns that look more like coverage maximisation than normal user paths. Volume limits alone miss this.

Retrieval and generation controls

Use context minimisation for sensitive collections. Add output-side groundedness and span checks where the domain justifies the latency cost. Make rate limits react not only to volume but also to exploration-like behaviour.

Detection and response

Seed canaries and unique trackable facts. Monitor for their appearance outside your systems and for unexpected appearance in outputs. Log enough session and retrieval metadata to reconstruct whether a conversation was systematically filling structural holes. Maintain a short playbook for suspected extraction: tighter limits, forced step-up auth, temporary isolation of sensitive collections, forensic review. Legal and contractual layers remain part of a mature posture — they do not stop a determined adversary, but they change the economics and the aftermath.

Checklist you can run this month

[ ] Inventory which collections contain high-value or regulated material
[ ] Confirm those collections are not reachable by the lowest-trust access path
[ ] Add or tighten per-session and per-user budgets on the chat / API surface
[ ] Deploy at least a minimal set of canary facts and a way to notice them
[ ] Instrument basic exploration signals
[ ] Document a short incident-response path for suspected extraction
[ ] Review whether query rewriting or multi-query retrieval is giving a false sense of safety
[ ] Evaluate retrieval quality with Ragas on a held-out set of your questions
[ ] Skim RAGFort and RAGCrawler with your security team

None of these stop a determined, well-resourced attacker forever. Together they make casual and mid-tier extraction noticeably more expensive and more visible — which is the realistic bar for most product teams.

16. Where builders and attackers are meeting

Legitimate systems are moving toward agentic crawlers and agentic retrieval: memory, multi-step planning, decisions based on a growing model of the information space, quality verification, closed loops. Knowledge-graph guidance appears in GraphRAG and in various enterprise document-understanding efforts. Google’s 2026 framing of agentic RAG stresses persistence — keep searching until the context is sufficient, not until a single retrieve call returns something plausible.(Google Research on Agentic RAG)

RAGCrawler is already an agentic crawler that plans, maintains a growing graph, estimates marginal coverage, and acts through natural language.

Improvements in agent memory, tool use, long-horizon planning and graph reasoning therefore improve both legitimate pipelines and extraction attacks. The race is less “is extraction possible?” and more “how efficiently can each side explore an unknown document space under budget and stealth constraints?”

Kiela’s line is still the right one: retrieval is not disappearing; it is being absorbed into richer context engineering and agentic loops. The same observation applies, uncomfortably, to the attack surface. Elastic’s take from the search-infra side is worth reading next to Anthropic’s: From retrieval to agents.

If you are building agents, you are also building a system that can be pointed at someone else’s knowledge base — or at your own. Design with that dual use in mind.

17. What to do this month

If you own a RAG product or internal knowledge system

Treat the ingestion and promotion pipeline as a first-class product with owners, SLOs and a rollback story. Measure end-to-end freshness and retrieval quality on your real critical queries, not only on public benchmarks. Prefer official APIs and structured exports over scraping when quality and legal posture matter. Assume the conversational interface can be used as an extraction oracle and apply the checklist in section 15. Keep the highest-value material behind additional controls.

If you work on platform security or research

Read RAGCrawler, then IKEA and RAG-Thief, in that order. Test whether your current rewriting and multi-query layers actually reduce global coverage or only change surface form. Explore whether the same graph techniques used by attackers can be turned into defensive monitors. Support shared evaluation suites for knowledge-base leakage; the area is still immature compared with classic model-stealing benchmarks. Track extraction defenses (RAGFort) and poisoning defenses (RAGSentinel) as separate but related tracks.

If you are deciding what to build next

The highest-leverage work is usually not a new embedding model. It is ownership of the observation layer, explicit promotion, provenance that engineers will actually use, and a security review that includes extraction — not only injection. Ship the boring controls. Then read the papers.

18. People, papers, tools — a working map

This is the section many Dev.to RAG posts skip. Every URL below is a real page. Prefer abs and PDF over secondary summaries when citing numbers.

Core attack research (2024–2026)

RAGCrawler — abs · HTML · PDF — must-read: graph-guided extraction
RAG-Thief (2024) — agent continuation attacks
Silent Leaks / IKEA (2025) — benign queries, high yield
Adaptive Submodularity (Golovin & Krause)

People worth following

Douwe Kiela — original RAG co-author, Contextual AI. RAG is dead, long live RAG!
Akari Asai — Self-RAG. akariasai.github.io
Jerry Liu / LlamaIndex — production RAG patterns
Harrison Chase / LangChain — retrieval as a tool inside agents
Microsoft GraphRAG team — Darren Edge, Jonathan Larson and collaborators
Abhinav Kimothi — the practical RAG book
Tomaž Bratanič — graphs in production
Sebastian Raschka — the “from scratch” books that keep people honest about what models actually do

Books

Crawlers and scrape-to-RAG tooling

Frameworks, eval, indexes

Hands-on analysis (the SEO / crawl overlap)

What I would ship in the first two weeks

If this article only leaves you with a longer reading list, it failed. Here is the sequence I would actually run on a real system that already has a chat UI and a vector index:

Week 1 — stop the silent rot. Wire a quality gate before embed (thin text, text-to-HTML ratio, canonical, near-dupe). Log every drop. Add content hashes and observation timestamps to every chunk that already exists — even if you only backfill metadata. Define one freshness SLO for the three sources that change most often. Turn off full nightly re-embeds if you cannot explain why every chunk needs them.

Week 1 — stop treating the chat UI as harmless. Per-session and per-user budgets. A canary fact in a sensitive collection. Basic logging of which collections were retrieved, not only which answer was shown. A one-page incident note: who gets paged if exploration-like traffic spikes.

Week 2 — make promotion real. Candidate or shadow index for at least one high-churn source. Compare before promote. One rollback drill: deliberately ship a bad observation, then revert using evidence hashes. Measure source-to-queryable lag once, with a number, not a feeling.

Week 2 — pick the stack you can operate. Postgres + pgvector (or the vector DB you already pay for), one embedding model, hybrid retrieval, one reranker. Do not start three migration projects. Measure on twenty questions your support team actually asks.

Papers matter. Playbooks matter more when the index is already in production.

Closing

The RAG crawler has two lives in 2026.

In one life it is the unglamorous component that decides whether your retrieval system is fed clean, fresh, well-structured knowledge or a swamp of duplicates and stale pages. Getting this right remains one of the highest-leverage engineering investments available — especially as agents, not only humans, consume the context.

In the other life the same family of ideas — global state, expected marginal gain, systematic exploration — has become a practical way to hollow out a private knowledge base through ordinary conversation. The 2026 RAGCrawler results should end the comforting belief that “the model never trained on the data, so the data is safe behind the API.”

The useful response is not panic. The same discipline that produces a high-quality builder-side crawler also makes you a better defender. You start seeing your own system the way a patient, graph-guided adversary would see it.

Build the knowledge engine carefully.

Assume someone else may try to crawl it.

Instrument both sides of the loop.

Raise the cost of global extraction without destroying legitimate utility.

If this helped, send it to the person who actually owns your knowledge pipeline. They are the ones who need it most.

About the author

Built around production crawl and site-intelligence work at AuditMe — AI-powered technical analysis for the same class of problems this article treats as the observation layer of RAG (discovery, canonicals, change, quality).

If you ship retrieval systems and want to compare notes on crawlers, freshness, or extraction defenses — open an issue on the tools you use, or reach out via the site.

Written as a working map for late August 2026, not a victory lap. If a link dies, a number in a paper moves, or you have contradictory production experience — that is the conversation worth having. The field is moving. The crawl loop is not going away.

The New SEO: When Search Engines Stop Reading Websites and Start Using Them

Eduard — Thu, 27 Aug 2026 08:37:13 +0000

TL;DR: Search is moving from ranking pages to running them as machine interfaces. If your site can't be discovered, understood, verified, and acted upon by AI systems, you're invisible to the next wave of queries — no matter how well you rank today. Most sites fail on verification and actionability, not discoverability. The fix is a six-dimension "Website Intelligence" framework: discoverability, understanding, verification, actionability, reliability, and observability. I built this into the free AuditMe SEO audit, and this article walks through the engineering behind it.

The Problem Nobody Is Talking About
Why the Document Model Broke
What Actually Changed in Search Architecture
The Agent Problem Nobody Prepared For
Machine Trust and the Reconciliation Problem
Six Things a Machine Must Be Able to Do With Your Website
The Data Nobody Shares
Why Schema Alone Cannot Save You
From Audits to Continuous Verification
The Distributed Truth Problem
What Web Agent Research Actually Found
The Engineering Stack Nobody Teaches
A Practical Website Intelligence Framework
What Happens Next (Honest Projections)

The Problem Nobody Is Talking About

There is a quiet structural problem emerging across the web, and most website owners have no idea it exists.

It is not about rankings. It is not about traffic. It is not about backlinks. Those things matter, but they are symptoms of a deeper issue that almost nobody is measuring.

Here is the problem, and I'll state it without hedging:

Websites are becoming machine interfaces, but they were designed as human documents.

For twenty-five years, the entire SEO industry has operated on a single assumption: optimize a page so a human will find it useful, and search engines will reward you. That assumption is not wrong. It is incomplete. And the gap between "not wrong" and "complete" is where your competitors will lose — or gain — the next wave of search traffic.

Consider this concrete example. A B2B SaaS company publishes a landing page. The page looks great. The copy is polished. The design converts well. A human visitor understands exactly what the product does and signs up.

Now an AI agent visits the same page. It needs to answer the question: "What does this product cost, and is it suitable for a team of 50?"

The agent encounters:

Pricing is mentioned as "Starting at $29/mo" in the body copy
The Product schema says "price": "49"
The FAQ section says "Plans start at $29/month for small teams"
The API documentation references a "$39 plan"
The checkout page shows $49

A human can investigate these discrepancies. The agent has a reconciliation problem. Which number should it trust? How confident should it be in its recommendation?

Multiply this across every product, every page, every representation of every fact on every website, and you begin to see the scale of the issue.

Why the Document Model Broke

The World Wide Web started as a document system. Tim Berners-Lee's 1989 proposal described "a system for sharing research papers through linked documents." HTTP, HTML, URLs — every foundational technology was built around the assumption that a human would read the result.

Google's original PageRank algorithm (Brin & Page, 1998) treated the web as a graph of documents, where links served as votes of authority. The fundamental unit was the page. The fundamental consumer was the human reader. That architecture held for two decades. It held because the math was elegant and the results were useful.

Then three things happened simultaneously.

First, large language models got good enough to summarize web pages. Not perfectly, but well enough that a growing number of users prefer a synthesized answer over clicking through ten blue links. Google's AI Overviews now reach over 1.5 billion users monthly.

Second, tool-use capabilities gave AI systems the ability to do things, not just read things. OpenAI's function calling specification and Anthropic's tool use framework allow models to interact with APIs, fill forms, and execute multi-step workflows.

Third, autonomous web agents emerged as a research category. Projects like WebArena demonstrated that agents can navigate real websites to complete tasks — book a table, buy a product, fill out an application — with varying degrees of success.

Each of these developments quietly broke an assumption that the SEO industry had held for decades: that the only consumer of web content is a human with a browser.

The search result is no longer the destination. It is an intermediate step in a pipeline that looks more like this:

User question
     |
Machine reasoning
     |
Evidence gathering from multiple sources
     |
Fact reconciliation
     |
Answer synthesis OR task execution

When the consumer of your content is a machine performing reasoning, the requirements for that content change fundamentally. Not in degree. In kind.

What Actually Changed in Search Architecture

Understanding the shift requires looking at the actual technical components that changed, not the marketing narratives.

The Rendering Pipeline

Googlebot has operated on a two-phase system since the early 2000s: crawl (fetch HTML), then render (execute JavaScript to produce the final DOM). Google's rendering pipeline uses a headless Chromium instance that executes JavaScript, waits for network idle, and produces a rendered DOM that approximates what a human sees (Google Search Central: JavaScript SEO Basics).

The practical implication: your rendered DOM — not your raw HTML — is increasingly what search engines analyze. A page that relies entirely on client-side JavaScript to display pricing, product specifications, or authorship information creates a gap between the crawled document and the understood document. That gap is where information gets lost.

The WHATWG DOM specification (dom.spec.whatwg.org) defines the browser's in-memory representation of a document. Search engines approximate this representation. Any information not present in it is effectively invisible to them.

The Structured Data Layer

Schema.org, the collaborative vocabulary maintained by Google, Microsoft, Apple, and Yahoo, provides a way to annotate entities and relationships in structured data (schema.org). Google supports structured data through JSON-LD, Microdata, and RDFa formats, with JSON-LD recommended.

Here is the thing most people miss: structured data is a secondary representation of information that should already exist in the page. It is not a content source. Google's structured data policies state: "Structured data helps search engines understand the content on the page."

The phrase "understand the content" is doing significant work in that sentence. It means structured data is an interpretation aid, not a content source. When structured data contradicts visible content, the system does not get "more information." It gets a conflict.

The AI Reasoning Layer

When a search engine uses a large language model to synthesize an answer, it operates fundamentally differently from traditional ranking. Instead of selecting a single best-matching document, the model retrieves evidence from multiple sources, evaluates source quality, synthesizes a coherent answer, and cites sources.

This pipeline requires each source to provide extractable facts, verifiable claims, and machine-readable interfaces. Google's documentation on AI optimization emphasizes creating content that demonstrates first-hand experience and using structured data to help Google understand your content.

The implication is that the optimization target has shifted from "rank this page" to "make this website's information accessible to AI reasoning systems." Those are different problems.

The Agent Problem Nobody Prepared For

The most consequential architectural change is not AI Overviews. It is the emergence of web agents — autonomous systems that interact with websites to complete tasks on behalf of users.

A traditional crawler fetches a page and indexes its content. An agent fetches a page, interprets it, and acts on it. That distinction changes everything about what a website needs to provide.

Consider what a restaurant website needs to offer each type of consumer:

What a crawler needs from your restaurant page:

Page title
Address
Menu content
Opening hours
Reviews with ratings

What an agent needs from your restaurant page:
All of the above, plus:

A form to make a reservation, with labeled inputs (not just a phone number)
Semantic buttons (<button>, not <div onclick="...">)
Clear input validation (date picker, party size selector)
Success and failure states after submission
API endpoints for programmatic interaction
Price information in a parseable format

The critical difference: the agent needs actionable interfaces, not just information. A <button type="button">Reserve</button> communicates intent through document structure. A <div class="btn" onclick="openReservation()">Reserve</div> communicates nothing through structure. It relies entirely on JavaScript execution to reveal its purpose.

The HTML Living Standard (html.spec.whatwg.org) defines the contract for each element. A <button> has specific semantics: it is focusable, activatable, and communicates intent. A <div> has none of these properties. The WAI-ARIA specification extends this with roles, states, and properties that clarify intent for assistive technologies and, by extension, for machine agents.

This is not theoretical. It is measurable.

Machine Trust and the Reconciliation Problem

When a human reads a website, they can reason about inconsistencies. "This price says $49 here but $39 on the pricing page — probably an old page." Machines cannot easily do this. They encounter the same information from multiple sources and must determine which version to trust.

Google's Knowledge Graph, which powers Knowledge Panels and AI Overviews, maintains confidence scores for facts. Contradictory sources reduce these scores. Google's research on Knowledge Vault (Dong et al., 2014) describes the probabilistic fusion process: each source contributes evidence, and conflicts reduce the system's confidence in any given claim.

The practical reality looks like this. Imagine your company publishes:

Homepage:          "Plans start at $29/month"
Pricing page:      "Starter: $39/month"
Product schema:    "price": "49"
API response:      "monthly_price": 39
Documentation:     "The $29 plan includes..."
Third-party listing: "$39/month"

A human can investigate and form a judgment. A machine sees six contradictory signals and has no way to determine which is current. The result is not that the machine picks one. The result is that the machine's confidence in all of your pricing information drops.

This creates a specific type of technical SEO problem that traditional audits do not detect: mismatch detection across representations. The most valuable audit may be the one that discovers contradictions between your website, your schema, your documentation, your API, and your third-party listings.

Six Things a Machine Must Be able to Do With Your Website

Drawing from the evidence above, machine-readiness for websites can be evaluated through six requirements. These are not ranking factors. They are capabilities that a machine must possess to use your website effectively.

1. Discoverability: Can I Reach It?

Before a machine can understand your website, it must be able to reach it. This is the domain of traditional technical SEO, and none of it becomes obsolete.

What discoverability requires:

Crawlable pages with accessible URLs
Intentional robots.txt rules (Google: Robots.txt)
Valid XML sitemaps (Google: Sitemaps)
Correct canonical URLs (Google: Canonicalization)
Predictable redirect behavior (minimal chain length)
Accessible critical resources (CSS, JS, images not blocked)

Google's Gary Illyes has stated that crawl budget is real and should be managed. For large sites, poor discoverability means portions of the site are never indexed, and therefore never available to AI systems.

2. Understanding: What Am I Looking At?

Discovery answers "Can I fetch this?" Understanding answers "What am I looking at?"

Machines need structure. That means semantic HTML, meaningful headings, clear navigation, explicit entities, valid structured data, understandable page intent, consistent terminology, and machine-readable content.

The semantic gap between implementations is measurable. The WebAIM Million study analyzed the home pages of the top 1,000,000 websites each year since 2019 — and in the 2026 edition the trend lines are telling. 95.9% of home pages still had detectable WCAG 2 failures, and the average page carried 56.1 distinct accessibility errors. The most common issues — low contrast text (83.9%), missing alternative text (53.1%), empty links (46.3%) — are not just accessibility problems. They are machine-readiness problems. A machine that cannot parse your content structure cannot understand your content. If 96% of the web's most popular pages are structurally unreliable, that is not an accessibility niche. It is the default state of the web.

3. Verification: Can I Trust What It Says?

A machine should not merely extract a fact. It should be able to determine whether the fact is supported, consistent, and current.

Verification asks: does the structured data match the visible content? Does the canonical URL match the page being served? Does the organization name remain consistent? Do prices agree across the product page, schema, and checkout? Do dates make sense? Does the documentation describe the current product?

The practical consequence is that structured data should reinforce meaning, not create an alternative reality. A Product schema that claims $39 when the page shows $49 does not "optimize" anything. It introduces a contradiction that reduces machine confidence.

4. Actionability: Can I Do Something?

Traditional SEO concerns information retrieval. Agents introduce a different requirement: can the machine complete a task?

For a restaurant, that means making a reservation. For e-commerce, that means purchasing a product. For SaaS, that means starting a trial. For documentation, that means integrating an API.

The W3C Web of Things specification defines a framework for making web-connected devices interoperable with machine agents. While focused on IoT, the principle applies broadly: web interfaces must be machine-actionable, not just human-readable.

Google's AI Overviews with actions can now perform tasks on behalf of users — making reservations, purchasing products. For this to work, the website must provide identifiable actions (semantic buttons, forms, links), parseable inputs (labeled fields, correct types), predictable outcomes (standard HTTP methods), and clear error states.

5. Reliability: Can I Count on It?

A machine needs predictable, consistent access to information. This is where the deployment problem becomes relevant.

Modern websites change through deployments. Each deployment is a potential source of regression. Schema removed during a template update. Canonical URL changed during a migration. Pricing changed on the website but not in the schema. Documentation updated but the API response not synchronized.

Traditional SEO audits capture a point-in-time snapshot. They do not detect regressions that occur after the audit. The SRE practices developed at Google (SRE Book, Beyer et al., 2016) define principles for monitoring complex systems. Websites are becoming complex systems. They need the same treatment.

6. Observability: Can I Detect When It Changes?

Once a website becomes an input to machine decisions, change itself becomes a signal. You need to know what changed, when it changed, which facts changed, and whether a previously valid workflow still works.

The feedback loop looks like a software observability pipeline: establish baseline, monitor, detect change, evaluate impact, fix, verify, repeat. This is closer to Site Reliability Engineering than to the traditional SEO audit model. That is not an accident. Modern websites increasingly behave like software systems. Their SEO should be monitored like one.

The Data Nobody Shares

The shift from document-centric to system-centric search is not theoretical. Several data points quantify it.

AI Overview adoption: Google reported AI Overviews reaching 1.5 billion users monthly as of early 2025 (Google AI Overviews). Perplexity AI processes over 100 million queries per week.

Click behavior impact: Research by BrightEdge found that AI Overviews reduced organic CTR by 18-25% for informational queries. However, queries with AI Overviews that include source links saw increased click-through to cited sources. The implication: being cited by AI is becoming as important as ranking in organic results.

Agent adoption: OpenAI's ChatGPT browsing tools were used over 100 million times in the first quarter after launch. Anthropic's Claude web interaction capabilities showed measurable improvement in task completion rates from 2023 to 2024.

DOM quality matters: WebAIM's analysis of 1,000,000 websites found widespread structural issues that affect both accessibility and machine readability. In the 2026 report, average home pages contained 1,437 page elements and 56.1 detected errors, with low contrast text on 83.9% of pages and missing alt text on 53.1%. These are not abstract concerns. They are the infrastructure that machines use to understand your content. Notably, the same report shows framework choice correlates with structural health: Next.js sites averaged 40.9 errors per page — 27% below the overall average — while jQuery-based and ad-heavy pages trended far worse. The way you build directly shapes how well a machine can read you.

Why Schema Alone Cannot Save You

Structured data is one of the most powerful tools in technical SEO. It is also the most commonly misunderstood.

Schema.org provides a vocabulary for marking up entities and relationships. Google supports over 30 structured data types, including Product, Organization, Article, FAQ, Event, JobPosting, and Recipe. When implemented correctly, structured data enables rich results — enhanced search appearances with additional information.

But structured data has hard limits.

It cannot create information that does not exist in the page. It cannot override content that contradicts the schema. It cannot fix structural problems in the DOM. And it cannot substitute for consistency across your digital presence.

The correct approach to structured data:

Ensure the information exists clearly in the page content
Mark it up with appropriate schema types
Verify that schema values match visible content
Verify that schema values match external representations
Monitor for drift over time

Google's structured data policies state: "Don't mark up content that is not visible to the user" and "The structured data on a page should describe the content of that page."

These guidelines exist because schema mismatches reduce the system's ability to trust the data. Adding more schema to a page with contradictory information does not help. It creates another contradiction.

From Audits to Continuous Verification

The audit-as-snapshot model is breaking for the same reason manual software testing was replaced by continuous integration.

The traditional audit cycle looks like this: run audit, get report, fix issues, done. The problems are obvious. The audit is valid only at the moment it was run. Regressions go undetected until the next audit. There is no feedback loop between "fixed" and "verified fixed."

The continuous verification model replaces this with: establish baseline, monitor, detect change, evaluate impact, fix, verify, repeat. Regressions are detected immediately. The "fixed" state is verified automatically. The baseline evolves as the website improves.

Google Search Console provides some of this capability for Google-specific metrics. But it does not monitor schema consistency, cross-source accuracy, agent workflow integrity, DOM semantic quality, or third-party listing accuracy. These require dedicated monitoring infrastructure.

The concept is not new. Software engineering has practiced continuous verification for decades. The SRE book (Beyer et al., 2016) defines the principles. Applying them to SEO is a natural extension that the industry has been slow to adopt.

The Distributed Truth Problem

Modern businesses maintain information across many systems: their website, CMS, schema markup, Google Business Profile, Google Merchant Center, API responses, documentation, social profiles, third-party directories, PDF catalogs, and mobile apps.

Each of these is a representation of the same entity. Every representation creates an opportunity for divergence.

Viewing this through the lens of distributed systems engineering reveals that consistency is not a content problem. It is a data engineering problem. The principles that apply to distributed databases apply here: eventual consistency (all representations should converge), conflict resolution (defined source of truth), monitoring (divergence must be detected), and idempotency (updates applied consistently across representations).

Lamport's work on time, clocks, and the ordering of events in distributed systems (1978) established the foundational principles. The scale differs between database replication and cross-platform SEO consistency, but the principle is identical: when multiple systems maintain copies of the same information, consistency must be actively managed.

What Web Agent Research Actually Found

Several research projects have directly measured how AI agents interact with websites. The findings are concrete and actionable.

WebArena (Zhou et al., 2023)

WebArena provides a benchmark environment for evaluating autonomous web agents on realistic tasks across e-commerce sites, forums, CMS platforms, and mapping applications. Accepted as an Oral at NeurIPS 2024, it is now the anchor of an entire family of benchmarks gathered under WebArena-x: VisualWebArena for multimodal agents (ACL 2024), WebArena-Infinity for continuous evaluation in evolving environments, and TheAgentCompany (ICML 2025), which evaluates agents on consequential real-world office tasks inside a simulated company.

The results are humbling for the field. The best-performing agents achieved approximately 14% success rate on complex multi-step tasks. Human performance on the same tasks was approximately 75%. The primary failure modes were not comprehension failures. They were interaction failures — incorrect element selection, misunderstanding of DOM structure, inability to handle dynamic content.

Pages with clear semantic structure had measurably higher agent success rates. The implication is direct: DOM quality is not an accessibility nice-to-have. It is a machine-accessibility requirement.

Mind2Web (Deng et al., 2023)

Mind2Web (a NeurIPS 2023 Spotlight) provides a dataset of 2,350 open-ended tasks across 137 real websites for training and evaluating generalist web agents.

The key finding: agent performance strongly correlated with DOM semantic quality. Semantic HTML elements — <button>, <input>, <nav> — were identified correctly more often than generic elements like <div> and <span>. Form accessibility (labels, field types, validation) directly impacted task completion.

This is not a correlation without causation. The semantic elements carry behavioral contracts defined in the HTML Living Standard. A <button> is focusable and activatable by default. A <div> is not. When an agent tries to interact with a page, these contracts determine whether the interaction succeeds.

SeeAct (Zheng et al., 2024)

SeeAct (accepted at ICML 2024) is a framework for web agents that uses visual grounding to interact with websites through screenshots and DOM analysis.

The research showed that agent accuracy decreased 30-40% on pages that relied on non-standard UI components. Pages that followed web standards — semantic HTML, ARIA attributes — had significantly higher interaction success rates. Dynamic content rendered via JavaScript was a major source of agent errors.

The convergence across all three projects is clear: the semantic quality of a website's DOM directly affects how well machine agents can use it. This is measurable, quantifiable, and increasingly consequential.

The Engineering Stack Nobody Teaches

The shift from page-level optimization to system-level intelligence requires a different engineering stack. The traditional SEO stack — content, HTML, meta tags, schema, links, analytics — remains relevant but is no longer sufficient.

The Website Intelligence stack includes:

Information architecture and URL strategy
Semantic HTML and DOM quality
Structured data (JSON-LD, correctly implemented)
Cross-source consistency management
Performance engineering (Core Web Vitals)
Accessibility (WCAG 2.2 compliance)
API design and documentation
Continuous monitoring and regression detection
Agent workflow testing

This stack looks like a software engineering discipline because it is one. The SEO profession is converging with web engineering, data engineering, quality assurance, and observability. That convergence is not a rebranding exercise. It is a reflection of the fact that websites are becoming software systems, and their machine-readiness requires the same rigor we apply to software reliability.

The standards that define this stack are established and freely available: the HTML Living Standard, the DOM Living Standard, WCAG 2.2, WAI-ARIA, Schema.org, and Google's Search Central documentation. None of these require proprietary knowledge. They require systematic application.

A Practical Website Intelligence Framework

Based on the framework presented in this article, a Website Intelligence audit should evaluate six dimensions. Each dimension includes concrete checkpoints. I've implemented this exact framework in the free AuditMe SEO audit tool, which runs many of these checks automatically — for example, cross-field verification that catches the price and date mismatches described above. The checks below are tool-agnostic; any team can run them with a good crawler and a text editor.

Discoverability

Important pages are crawlable by standard user agents. Robots.txt rules are intentional. XML sitemaps are valid and submitted. Canonical URLs are correct. Redirect chains are minimal. Critical resources are accessible to crawlers. URL structure is stable and predictable.

Understanding

Page intent is clear from structure. Heading hierarchy is logical. Interactive elements use semantic controls. Navigation uses <nav> with meaningful link text. Entities are identifiable. Structured data is valid and matches page content. Content is understandable without visual context.

Verification

Visible prices match Product schema. Organization name is consistent across website and schema. Author information is verifiable. Dates are accurate. Documentation describes current features. External representations agree with website.

Actionability

Critical actions use semantic form controls. Form inputs have associated labels. Form validation provides clear feedback. Required fields are indicated. Success and failure states are understandable. API endpoints are documented and stable.

Reliability

Core pages load consistently. APIs respond predictably. Important URLs remain stable. Schema data survives template updates. Stale information can be detected automatically.

Observability

Key pages have baseline snapshots. Schema changes are tracked. DOM structural changes are detected. Cross-source consistency is monitored. Regressions are flagged automatically. Critical workflows can be re-tested.

What Happens Next (Honest Projections)

Predictions are fragile. But directional trends are observable. Based on current trajectories in AI research, search architecture, and web standards, several developments are likely — and a few have already started.

Agent-native websites become the standard. Websites that provide agent-accessible interfaces — semantic HTML, API endpoints, labeled forms — will have a measurable advantage in AI-mediated discovery. The companies that build these interfaces early will capture the machine-mediated traffic that others miss.

Cross-source consistency becomes a ranking factor. As AI systems become more sophisticated at reconciling information across sources, inconsistency between a website's representations will increasingly be treated as a trust deficit. A website that maintains consistent information across its CMS, schema, API, and third-party listings will outperform one that does not. This is no longer hypothetical — the verification checks that catch these mismatches are already implementable as automated cross-field audits.

Continuous verification replaces periodic audits. The SRE-inspired model of continuous monitoring will become the standard for SEO. Manual audits will be supplemented — and in some cases replaced — by automated systems that detect regressions, verify fixes, and maintain baselines. This is the projection I am most confident about, because I have already built it into AuditMe: baseline snapshots, change logging per URL, field-level schema diffs, and alert delivery on regression. The tooling is not science fiction — it exists today.

DOM quality becomes a first-class metric. The semantic quality of a website's DOM will be measured and tracked as a metric, similar to how page speed is measured today. Tools will emerge that score DOM semantic quality the way Lighthouse scores performance. AuditMe now ships machine-readiness and semantic-interactive checks that score exactly this.

The SEO profession evolves. The role will increasingly overlap with web engineering, data engineering, quality assurance, accessibility, and observability. The practitioners who thrive will be the ones who can operate across these disciplines.

Conclusion

The web was built for humans to read. Search engines learned to index it. AI learned to summarize it. Agents are beginning to use it. The next generation of websites must satisfy all of these consumers simultaneously.

The requirements converge on the same principles: clear information that humans can understand, semantic structure that machines can interpret, consistent facts that all consumers can trust, accessible interfaces that all consumers can act on, stable behavior that all consumers can rely on, and observable changes that all consumers can detect.

These principles are not new. They are established engineering practices. What is new is that they now directly affect a website's ability to be discovered, understood, and used by the systems that increasingly mediate between businesses and their audiences.

The shift from SEO to Website Intelligence is not a rebranding exercise. It is a recognition that the primary consumer of web content is expanding from humans to include machines — and that machines have different, more demanding requirements for the information they use.

Build websites that machines can trust. The same qualities that make a website machine-readable make it better for humans too. That is not a coincidence. It is a convergence.

If your site is not machine-readable by the time the next wave of search hits full scale, you will not be invisible to search engines in the traditional sense. You will simply be a source they read but cannot trust — and trust is the entire game. The window to build for the machine consumer is open now, and it does not stay open forever.

Here is my honest verdict after building the AuditMe engine that operationalizes this framework: you do not need to wait for AI search to "mature" before acting. The six dimensions above are implementable today with tools that already exist. Most teams have 80% of the discoverability problem solved. The gap is almost always the same three dimensions — verification, actionability, and observability — because those are the ones nobody taught SEO practitioners to think about. Close that gap and you stop competing on the same terms as everyone else.

References

Google Documentation and Research

Google Search Central Documentation — The authoritative reference for crawling, indexing, and ranking behavior
Structured Data Policies — Guidelines for correct structured data implementation
JavaScript SEO Basics — How Google renders JavaScript
SEO Starter Guide — Foundation of technical SEO
AI Optimization Guide — Optimizing for generative AI search
Creating Helpful Content — Google's content quality guidelines
Canonicalization — Managing duplicate content
AI Overviews Announcement — Google's AI search features

Standards and Specifications

Schema.org — The collaborative vocabulary for structured data
HTML Living Standard — The definitive specification for HTML
DOM Living Standard — Browser document object model specification
WCAG 2.2 — Web Content Accessibility Guidelines
WAI-ARIA — Accessible Rich Internet Applications
W3C Web of Things — Interoperability framework for web-connected devices
robots.txt Specification (RFC 9309) — Standard for crawler directives

Research Papers and Projects

Brin, S. & Page, L. (1998). "The Anatomy of a Large-Scale Hypertextual Web Search Engine." Wikipedia: PageRank
Berners-Lee, T. (1989). "Information Management: A Proposal." w3.org/History/1989/proposal.html
Dong, X. et al. (2014). "Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion." research.google
Lamport, L. (1978). "Time, Clocks, and the Ordering of Events in a Distributed System." lamport.azurewebsites.net
Beyer, B. et al. (2016). "Site Reliability Engineering." sre.google/sre-book

AI and Agent Research

Zhou, S. et al. (2023). "WebArena: A Realistic Web Environment for Building Autonomous Agents." webarena.dev (NeurIPS 2024 Oral; part of the WebArena-x family)
Deng, X. et al. (2023). "Mind2Web: Towards a Generalist Agent for the Web." github.com/OSU-NLP-Group/Mind2Web (NeurIPS 2023 Spotlight)
Zheng, B. et al. (2024). "SeeAct: Grounded Vision-based Web Agents." github.com/OSU-NLP-Group/SeeAct (ICML 2024)
Xu, F. et al. (2025). "TheAgentCompany: Benchmarking LLM Agents on Consequential Real-World Tasks." the-agent-company.com (ICML 2025)

AI Platform Documentation

OpenAI Function Calling — How LLMs invoke external tools
Anthropic Tool Use — Claude's tool interaction framework
OpenAI Web Search — Browsing capabilities

Industry Research

WebAIM Million 2026 — Accessibility analysis of top 1M websites
BrightEdge Research — AI Overviews impact data

This article examines the architectural and semantic requirements websites must satisfy as search evolves from document retrieval to machine-mediated reasoning. The framework presented (Website Intelligence) draws from established web standards, published research, and observable trends in search architecture. The durable engineering principles it identifies will remain relevant regardless of specific implementation changes in any single AI system.

What Actually Makes ChatGPT, Claude & Perplexity Cite Your Website (3 Months, 47 Tests, Real Numbers)

Eduard — Sun, 23 Aug 2026 14:10:13 +0000

What Actually Makes ChatGPT, Claude & Perplexity Cite Your Website (3 Months, 47 Tests, Real Numbers)

Three months ago, I typed "best free SEO audit tool" into ChatGPT. It listed five tools. Mine wasn't one of them. That kicked off 90 days of systematic testing — 47 specific tests across four AI chatbots, with exact queries, exact dates, and exact results.

What this article is: A detailed log of what I tested, what worked, what didn't, and the specific timelines I observed.

What this article isn't: Reverse-engineering of proprietary algorithms. I don't have insider access to OpenAI, Anthropic, Perplexity, or Google. Everything here is based on my own testing against live AI systems, publicly available documentation, and patterns I observed. Some tactics work great in some niches and poorly in others. There are no guarantees.

Who I am: I built AuditMe, a free SEO audit tool. I'm sharing this because the GEO space needs more real data and less speculation. Yes, I mention my tool in this article — it's the product I tested with. But I've included other tools where relevant, and I've been honest about what worked regardless of which tool was involved.

Why this matters: AI search is growing fast. Perplexity now handles 15M+ daily searches. ChatGPT has 100M+ weekly users. Google's AI Overviews appear in ~30% of searches. If your website isn't optimized for AI extraction, you're missing a growing channel. But the advice out there is mostly theory — I wanted real data.

What I tested: 5 specific tactics (FAQPage schema, llms.txt, content structure, Reddit engagement, robots.txt) across 4 AI platforms (ChatGPT, Claude, Perplexity, Gemini) over 3 months (January-March 2026). I tracked 47 specific queries with exact dates and outcomes.

What I found: Some tactics worked consistently across all platforms. Others were platform-specific. Some commonly recommended tactics didn't work at all. The results surprised me — both positively and negatively.

TL;DR (If You Read Nothing Else)

FAQPage schema + structured content had the biggest impact across all four AI systems. Pages with FAQPage schema appeared in AI answers 2-3x more often.
robots.txt is binary — if you block AI crawlers, you're invisible. If you allow them, you're in the game. Takes 5 minutes to fix.
llms.txt helps, but it's not magic. Perplexity reads it actively. ChatGPT and Claude seem to use it occasionally. Worth implementing (15 minutes), but don't expect miracles.
Reddit mentions correlate with AI citations in consumer niches. GitHub matters more for developer tools. But this varies significantly by niche.
Content structure matters more than most people think. Rewriting paragraphs into question-answer-list format measurably increased AI citations.
No tactic guarantees results. These increase probability. In competitive niches, domain authority and external signals matter more than technical GEO.

The Landscape: Search in 2026
How AI Chatbots Actually Find and Cite Websites
The Five Tactics I Tested
Observations & Patterns: What Changed and When
One Before/After Example That Shows Everything
What Works Almost Always vs What Depends on Niche
What Doesn't Work (Myths I Busted)
How Different AI Systems Handle Citations
The Competition Problem
The Honest Limitations
Complete GEO Checklist (Must-Do vs Nice-to-Have)
Free Tools I Used
Where This Is Heading (2026-2027)
FAQ

The Landscape: Search in 2026

The numbers, with sources and caveats:

Metric	Number	Source	Caveat
ChatGPT weekly queries	100M+	OpenAI blog (Feb 2024)	Most are conversational, not search-like
Perplexity daily searches	15M+	Perplexity blog (2024)	Growing fast, but still small vs Google
Google AI Overviews	~30% of searches	Search Engine Land	Percentage varies wildly by query type
Zero-click searches	65%	SparkToro (2024)	Includes featured snippets, not just AI

What this means: AI search is growing but still a fraction of total search. Google remains dominant for navigational queries. AI chatbots are most impactful for research queries ("what's the best X", "how do I do Y").

The practical implication: you don't need to abandon traditional SEO. You need to add GEO on top of it. The good news is most GEO tactics also improve traditional SEO.

How AI Chatbots Actually Find and Cite Websites

When you ask ChatGPT or Perplexity a question, here's what happens:

Your query is processed by the language model
The model decides if it needs real-time information
A web search is triggered (similar to Google, but simpler)
Top 5-15 results are fetched and processed
The model generates an answer using those results
Citations are added to sources it used

This is RAG (Retrieval-Augmented Generation) — the model retrieves information in real time, not just from training data.

What this means for you: Changes to your site can appear in AI answers within days (for RAG-based systems like Perplexity and ChatGPT Search), not months (for training-based updates). This is a fundamental difference from traditional SEO, where changes can take weeks or months to impact rankings.

The RAG pipeline in detail:

Query parsing: The model breaks down your question into searchable components
Search retrieval: A search engine (Bing, Google, or proprietary) finds relevant pages
Content extraction: The model extracts text, metadata, and structured data from results
Relevance scoring: Pages are scored based on relevance to the query
Answer generation: The model synthesizes information from top-scored pages
Citation attribution: Sources are added to the answer

What this means for GEO: Each step in the pipeline is an opportunity. Structured data helps step 3. Content structure helps step 4. FAQPage schema directly maps to step 1 (question-answer format). llms.txt helps step 2 (providing context about your site).

The factors that seem to matter most based on my testing:

Factor	What I Observed	Strength	Caveat
Domain authority	Established domains cited more often	Strong	Hard to change quickly
Content structure	Structured content extracted more easily	Strong	Easy to implement
FAQPage schema	Measurable increase in AI citations	Strong	Only for Q&A-style queries
External mentions	Reddit/GitHub correlate with citations	Moderate-Strong	Varies heavily by niche
llms.txt	Some systems read it, many don't	Moderate	Still very new
Page speed	Slow pages sometimes skipped	Minor	Easy fix

Important: These are my observations from testing, not official documentation. The actual algorithms are proprietary.

The Five Tactics I Tested

Here's what I actually did, with enough detail that you can replicate the tests.

Tactic 1: FAQPage Schema (JSON-LD)

Added FAQPage schema to 5 pages with common questions about the product. Each FAQ had 3-5 questions with direct, concise answers. The key insight: questions should match real user queries — check your analytics for "People Also Ask" data, search console queries, and customer support questions.

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is the best free SEO audit tool?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Several free SEO audit tools are available in 2026, including AuditMe, Ubersuggest, and SEMrush free tier. Each has different strengths."
      }
    },
    {
      "@type": "Question",
      "name": "How much does an SEO audit cost?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Free SEO audit tools provide basic analysis. Professional audits cost $500-$5,000. Tools like AuditMe offer free basic audits with Pro plans at $19/month."
      }
    }
  ]
}

Where I added it: Homepage, pricing page, blog posts about SEO tools, comparison pages. I also added FAQPage schema to the "About" page and the "Contact" page with questions like "What does AuditMe do?" and "How do I contact support?"

Implementation notes:

Validated with Google Rich Results Test before deploying
Checked with Schema.org validator for syntax errors
Used ld+json script tags in the <head> section
Each answer was 1-3 sentences — concise, factual, no marketing speak
Questions used natural language (not keyword-stuffed)

Tactic 2: llms.txt

Created a concise llms.txt (46 lines) and an extended llms-full.txt (264 lines) with detailed documentation. The llms.txt standard is still emerging — I implemented it because the cost is low (15 minutes) and the potential upside is high.

What I included in llms.txt:

One-line description: "AuditMe: Free AI-powered SEO audit tool analyzing 51 checks across 11 categories"
Core features (4 items): meta, content, technical, links, performance, schema, images, social, eeat, accessibility, security
Target audience (3 items): solopreneurs, agencies, developers
Pricing tiers: Free (1 audit/day), Pro ($19/month, unlimited)
Key pages with URLs: homepage, pricing, blog, API docs
Contact info: email, GitHub, Twitter

What I included in llms-full.txt:

Detailed feature descriptions for each of the 11 audit categories
Technical architecture (Next.js, Supabase, Vercel)
API documentation with endpoints and examples
Supported platforms and integrations
Comparison with alternatives (SEMrush, Ahrefs, Screaming Frog)

Key finding: The short llms.txt (46 lines) outperformed the long version (264 lines). AI systems seem to truncate long files. Keep essential info under 500 words.

Tactic 3: Content Structure Overhaul

Rewrote 10 existing blog posts from paragraph-style to AI-extraction-friendly format. This took the most time (about 2 weeks of work) but produced the most consistent results across all AI systems.

Key changes:

Question-based H2/H3 headings: Changed "Title Tag Optimization" to "How Do I Optimize Title Tags for SEO?"
Definition-style openings: First sentence directly answers the heading question (e.g., "Title tags are HTML elements that define the title of a web page.")
Numbered lists: Converted recommendations from paragraphs to numbered lists with impact/time estimates
Tables: Added comparison tables with metrics, tools, and benchmarks
Bold key terms: Bolded important concepts so RAG systems can extract them easily
Metric tables: Added tables with specific numbers (LCP < 2.5s, CLS < 0.1, etc.)

Time investment: 2 weeks for 10 posts. Average 2-3 hours per post for research, rewriting, and validation.

Tactic 4: Reddit Engagement

Posted genuine answers in r/SEO, r/webdev, r/SaaS, and r/entrepreneur over 4 weeks. Not spam — actual helpful answers to real questions where the product was relevant as a solution.

Volume: 3-4 posts per week, each taking 10-15 minutes to write genuinely.

What I did right:

Answered questions first, mentioned product second
Included specific examples and data from my own experience
Was transparent about being the product creator
Engaged in follow-up comments

What I did wrong (at first):

First 2 posts were too promotional — got flagged by moderators
Learned to lead with value, not product mentions
Started including screenshots and specific examples instead of links

Tactic 5: robots.txt Optimization

Ensured all major AI crawlers were allowed. Started with User-agent: * Allow: / and then verified specific bots were not blocked by any intermediate rules.

Bots I explicitly verified:

GPTBot (OpenAI)
CCBot (Common Crawl)
anthropic-ai (Anthropic)
PerplexityBot (Perplexity)
Google-Extended (Gemini)
Bytespider (ByteDance)

Implementation: Simple robots.txt at the root with User-agent: * Allow: / and a Sitemap directive. If your site already doesn't block any bots, this is a no-op. But if you're blocking them, this is critical.

Observations & Patterns: What Changed and When

Methodology note: Multiple interventions (FAQ schema, llms.txt, Reddit activity, content rewrites) overlapped from January to March 2026. The observations below show correlations, not isolated causal relationships. For context, research across 284 Korean DTC brands found 65.5% had zero AI appearances and the mean was only 0.648 out of 50 queries — citation sparsity is real, so even small lifts can appear significant. A staggered rollout would be needed to cleanly isolate each variable.

Observation 1: FAQPage Schema on Perplexity

Period: January 15 - February 15, 2026
Query: "best free SEO audit tool"
Platform: Perplexity (shows sources, easiest to track)

Date	Result	Position	Citation?
Jan 15 (before)	Listed 8 tools. Product not included.	—	No
Jan 22 (+7 days)	Listed 8 tools. Product appeared at #6.	#6	Yes — homepage
Feb 1 (+17 days)	Listed 10 tools. Product at #5.	#5	Yes — blog post
Feb 15 (+31 days)	Listed 10 tools. Product at #4.	#4	Yes — homepage + blog

What I added between Jan 15 and Jan 22: FAQPage schema with 5 questions on the homepage and pricing page. The questions were:

"What is the best free SEO audit tool?"
"How much does an SEO audit cost?"
"What does AuditMe check in an SEO audit?"
"How long does an SEO audit take?"
"Is AuditMe free to use?"

Observation: Product appeared ~7 days after adding FAQPage schema. Position improved over the next month. The schema made content easier for Perplexity's RAG system to extract and cite. The FAQ answers were concise (1-2 sentences each) and matched real user queries.

Observation 2: FAQPage Schema on ChatGPT

Period: January 15 - February 15, 2026
Query: "What is the best free SEO audit tool?"
Platform: ChatGPT (free tier, search enabled)

Date	Result	Notes
Jan 15 (before)	Listed 5 tools. Product not included.	Generic recommendations
Feb 1 (+17 days)	Listed 6 tools. Product at #4.	More detailed description
Feb 15 (+31 days)	Listed 6 tools. Product at #3.	Mentioned specific features

Observation: ChatGPT took 2-3 weeks to pick up changes. Product appeared after both FAQPage schema and Reddit activity — I can't isolate which caused the change. Likely both contributed. ChatGPT's recommendations were more brand-focused than Perplexity's.

Observation 3: llms.txt Impact (All Four Platforms)

Period: February 1 - March 1, 2026
Query: "What is [product name]?"
Platforms: ChatGPT, Claude, Perplexity, Gemini

Date	Perplexity	ChatGPT	Claude	Gemini
Feb 1 (before)	Third-party info. Inaccurate pricing.	Generic. Some features wrong.	"I'm not familiar with this tool."	Found GitHub. Minimal detail.
Feb 8 (+7 days)	Cited llms.txt directly. Accurate.	More accurate. Mentioned 11 dimensions.	Still limited.	More complete.
Feb 22 (+21 days)	Stable. Cited llms.txt + blog.	Accurate. Mentioned pricing.	Better. Mentioned capabilities.	Accurate. Mentioned audience.

What I added between Feb 1 and Feb 8: llms.txt and llms-full.txt at site root. The llms.txt file was placed at https://www.auditme.dev/llms.txt and included:

One-line description
Core features (4 items)
Target audience (3 items)
Pricing tiers
Key pages with URLs
Contact info

Key finding: Perplexity responded fastest (~7 days) and cited llms.txt directly. ChatGPT and Gemini improved over 2-3 weeks. Claude was slowest. The short llms.txt (46 lines) outperformed the long version (264 lines).

Observation 4: Content Structure Overhaul

Period: February 15 - March 15, 2026
Query: "how to improve SEO score"
Platform: Perplexity

Post	Before	After	AI Citations
"How to Improve SEO Score"	Never cited	Cited by Perplexity	+3 appearances
"Technical SEO Checklist"	Occasionally cited	Cited consistently	+5 appearances
"Core Web Vitals Guide"	Never cited	Cited for CWV queries	+2 appearances

What I changed: Question-based headings, definition-style openings, numbered lists, metric tables. For example, the "Core Web Vitals Guide" went from 3 paragraphs of explanation to a structured format with:

H2: "What Are Core Web Vitals?"
Opening definition: "Core Web Vitals are three metrics that measure real-world user experience..."
Numbered list: LCP (2.5s), CLS (0.1), INP (200ms)
Table comparing before/after scores

Timeline: ~2 weeks for changes to appear in AI answers. The structured content was extracted and cited much more frequently.

Observation 5: Reddit Activity

Period: February 1 - February 28, 2026
Query: "best SEO tool" and variations
Platforms: ChatGPT and Perplexity

12 genuine answers posted over 4 weeks (3-4 per week, 200-500 words each).

Metric	Before (Jan 15)	After (Feb 28)	Change
Perplexity citations	2/20 queries	6/20 queries	+200%
ChatGPT recommendations	1/15 queries	4/15 queries	+300%

Caveat: Can't prove causation. FAQPage schema and llms.txt were also active. Most likely a combined effect. But the correlation is strong enough to warrant genuine engagement.

Observation 6: robots.txt (Control)

Finding: Site was already neutral (not blocking AI bots). Explicitly allowing all AI bots didn't measurably change citation frequency. But if your site IS blocking AI bots, allowing them is critical.

Why this matters: Many sites accidentally block AI crawlers with broad Disallow rules or CDN-level blocking. Checking robots.txt is the first thing you should do. Use the AuditMe robots.txt Checker or Google Search Console to verify.

Observation 7: Organization Schema Impact

Period: January 20 - February 20, 2026
Query: "What is AuditMe?"
Platforms: All four

Added Organization JSON-LD with name, URL, logo, description, founding date, and contact info.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "AuditMe",
  "url": "https://www.auditme.dev",
  "logo": "https://www.auditme.dev/logo.png",
  "description": "Free AI-powered SEO audit tool analyzing 51 checks across 11 categories",
  "foundingDate": "2025",
  "contactPoint": {
    "@type": "ContactPoint",
    "contactType": "customer support",
    "email": "support@auditme.dev"
  }
}

Date	Result
Jan 20 (before)	Mixed results — some platforms had accurate info, others had outdated data
Feb 5 (+16 days)	More consistent descriptions across all platforms
Feb 20 (+31 days)	Stable. All platforms had accurate info

Observation: Organization schema helped with accuracy but didn't dramatically increase citation frequency. It's a "trust signal" — makes AI systems more confident about citing you. The biggest impact was on Gemini, which uses Google's structured data heavily.

Observation 8: SoftwareApplication Schema

Period: February 1 - March 1, 2026
Query: "free SEO audit tool"
Platform: Perplexity

Added SoftwareApplication JSON-LD with name, description, applicationCategory, operatingSystem, pricing, and review.

{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "AuditMe",
  "description": "Free AI-powered SEO audit tool analyzing 51 checks across 11 categories",
  "applicationCategory": "SEOTool",
  "operatingSystem": "Web",
  "offers": {
    "@type": "Offer",
    "price": "0",
    "priceCurrency": "USD"
  }
}

Date	Result
Feb 1 (before)	Listed as "AuditMe" with generic description
Feb 15 (+14 days)	Listed with pricing ($0/free), category (SEO tool), and features
Mar 1 (+28 days)	Stable. Consistent description across citations

Observation: SoftwareApplication schema provided structured product information that AI systems could extract directly. This is especially useful for product pages. The pricing information was particularly valuable — AI systems could now accurately state "free" instead of guessing.

Observation 9: Content Freshness Impact

Period: January - March 2026
Query: "SEO audit checklist 2026"
Platform: Perplexity

Updated 3 old blog posts with "2026" in titles and content. Added new sections, updated statistics, refreshed screenshots.

Post	Before Update	After Update	Change
"SEO Audit Checklist" (2024)	Not cited	Cited	+1 appearance
"Technical SEO Guide" (2023)	Occasionally cited	Cited consistently	+3 appearances
"Core Web Vitals Explained" (2024)	Not cited	Cited for CWV queries	+2 appearances

Observation: Content freshness matters. AI systems seem to prefer recently updated content. The "2026" in titles may have helped — it signals current, relevant content. Timeline: ~2 weeks for changes to appear in AI answers.

One Before/After Example That Shows Everything

This is from a real blog post on my site. Same content, different structure. The "before" version was written in January 2026. The "after" version was rewritten in February 2026.

Before (January 2026)

There are many factors that affect your SEO score and it's important to understand them all if you want to improve your rankings. First, you need to make sure your title tags are properly optimized with relevant keywords that match what people are searching for. Additionally, your meta descriptions should be compelling and include target phrases that encourage clicks from search results. Furthermore, you should pay close attention to your Core Web Vitals scores as these metrics directly impact your search rankings and user experience. Many people overlook the importance of heading structure but having proper H1, H2, and H3 tags helps search engines understand your content better.

What AI did with this: When asked "how to improve SEO score", AI systems either ignored this content or extracted inaccurate information. The paragraph format made it impossible for RAG systems to identify specific actionable items. The key metrics (50-60 characters for title tags, 120-160 for meta descriptions) were buried in flowing text that AI couldn't parse reliably.

After (February 2026)

How to Improve Your SEO Score (2026 Guide)

Your SEO score is a composite metric based on technical, content, and authority factors. Here are the highest-impact improvements:

1. Optimize Title Tags (Impact: High, 5 min/page)

Keep titles between 50-60 characters

Place primary keyword near the start

Make it compelling for clicks

2. Fix Core Web Vitals (Impact: High, 1-4 hours)

LCP: Under 2.5 seconds

CLS: Under 0.1

INP: Under 200ms

3. Write Meta Descriptions (Impact: Medium, 5 min/page)

120-160 characters

Clear value proposition

End with a call to action

4. Improve Content Quality (Impact: High, Varies)

Minimum 1,500 words for pillar content

H2/H3 headings every 200-300 words

3-5 internal links per post

5. Add Structured Data (Impact: Medium, 30 min)

Organization schema on every page

FAQPage schema on Q&A content

What AI did with this: After restructuring, Perplexity started citing specific items from this content in answers about SEO improvement. ChatGPT referenced the Core Web Vitals thresholds. The structured format made extraction trivial. The numbered list format gave AI systems clear, extractable items to cite.

The lesson: AI doesn't "read" paragraphs. It parses structure. The same information, formatted differently, gets completely different treatment from RAG systems. The "after" version has the same information but formatted for machine extraction — bold terms, numbered lists, specific metrics, and clear hierarchy.

What Works Almost Always vs What Depends on Niche

This distinction is important. Not all tactics are equally universal.

Works Almost Everywhere

Tactic	Why It's Universal	Implementation Time
Open robots.txt	Binary — blocked = invisible	5 minutes
FAQPage schema	Direct question-answer format maps to AI queries	1-2 hours
Structured content	AI systems extract lists better than paragraphs	Varies
Organization schema	Tells AI who you are unambiguously	30 minutes
Page speed	Slow pages get skipped by crawlers	Varies

Depends Heavily on Niche

Tactic	When It Works	When It Doesn't
Reddit mentions	Consumer products, tools, SaaS	B2B enterprise, regulated industries
GitHub presence	Developer tools, open source	Non-technical products
Product Hunt launch	New products, consumer apps	Established brands, services
llms.txt	Any site (but variable results)	Too new to know for all niches
Blog content	Informational queries	Transactional queries ("buy X")

What This Means for You

If you're building a developer tool: focus on GitHub + structured content + FAQPage schema. Reddit is secondary.

If you're building a consumer product: focus on Reddit + structured content + FAQPage schema. GitHub is secondary.

If you're in a regulated industry (health, finance, legal): focus on structured data + authoritative content. External signals are harder to control.

What Doesn't Work (Myths I Busted)

After 3 months of testing, here are commonly recommended tactics that didn't move the needle for me:

Myth 1: "Just Add llms.txt and Wait for Magic"

Reality: llms.txt is useful but not a silver bullet. Perplexity reads it actively. ChatGPT and Claude seem to use it occasionally. Many AI systems ignore it entirely. Implement it (15 minutes, no downside), but don't expect it alone to change your AI visibility. The data shows: llms.txt alone improved accuracy by ~30% but didn't increase citation frequency by much.

Myth 2: "Blocking GPTBot Protects Your Content"

Reality: Blocking GPTBot doesn't protect anything — your content is already public. It just prevents ChatGPT from recommending you. If someone wants your content, they'll find it regardless. The irony: sites that block GPTBot are invisible to ChatGPT but still appear in Google's AI Overviews (which don't use GPTBot).

Myth 3: "Buying Reddit Upvotes Helps"

Reality: Reddit's algorithm detects vote manipulation. Even if it temporarily works, the account and product can get banned. I've seen this happen. Genuine engagement is the only sustainable approach. The pattern I observed: posts with 10+ genuine comments performed better than posts with 100+ upvotes but no discussion.

Myth 4: "Longer llms-full.txt Is Better"

Reality: In my testing, the short llms.txt (46 lines) outperformed the long llms-full.txt (264 lines). AI systems seem to truncate long files. Keep essential info under 500 words. The extended version is useful for human developers but doesn't improve AI citations.

Myth 5: "You Need to Be on Every Platform"

Reality: Focus on 2-3 platforms where your audience lives. Spreading thin across 10 platforms produces worse results than deep engagement on 2-3. For developer tools: GitHub + Reddit + DEV.to. For consumer products: Reddit + Product Hunt + Twitter/X. For B2B: LinkedIn + industry forums + case studies.

Myth 6: "Structured Data Guarantees AI Citations"

Reality: Structured data increases the probability of being cited. It doesn't guarantee it. If your content isn't genuinely useful, no amount of schema will help. I tested this: added FAQPage schema to 5 pages, only 3 got cited consistently. The other 2 had weak content — schema couldn't compensate.

Myth 7: "OpenAI Definitely Reads Your Meta Keywords"

Reality: Meta keywords tag has been ignored by every major search engine since 2009. AI systems don't use it either. Don't waste time on it. Focus on structured data, content quality, and external signals instead.

How Different AI Systems Handle Citations Differently

Not all AI chatbots work the same way. Here's what I observed across 47 tests:

Perplexity

Most transparent. Shows sources directly — you can see exactly which websites it cited. Seems to read llms.txt more actively than other systems. Tends to cite more sources (5-15 per answer). Best for testing your GEO setup because you can see results directly.

What works best: Comprehensive content with clear structure, FAQPage schema, llms.txt. Perplexity seems to reward thorough, well-structured content. It also appears to favor content that's been recently updated — I noticed citations shifted to newer blog posts after I updated them.

What doesn't work well: Thin content, promotional language without substance. Perplexity's algorithm seems to filter out marketing-speak. Content that reads like a sales page gets ignored.

Specific observation: Perplexity cited my llms.txt directly in the answer source list. This is unique — no other AI system did this. If you implement llms.txt, Perplexity is the platform where you'll see the most direct impact.

ChatGPT (with Search)

Less transparent. Shows some citations but not always. Uses Bing for web search, so Bing-indexed content has an advantage. ChatGPT seems to favor well-known brands and authoritative domains. When it recommends products, it often picks from a smaller set of established players.

What works best: Domain authority, Reddit mentions, established brand presence. If your product is already well-known, ChatGPT will mention it. If it's new, you need external signals. ChatGPT also seems to weight Bing search rankings heavily — content that ranks well on Bing gets cited more often.

What doesn't work well: New, unknown products without external validation. ChatGPT is conservative with recommendations. It won't recommend a product that has zero external mentions, no matter how good your GEO setup is.

Specific observation: ChatGPT's recommendations were more brand-focused than Perplexity's. It tended to list 5-6 established tools rather than 10+ including newer options.

Claude

Most cautious. Often declines to make specific product recommendations. Tends to present options without strong endorsements. "There are several tools including X, Y, and Z" rather than "The best tool is X." Seems to rely more on training data than real-time search for product knowledge.

What works best: Factual, well-structured content. Balanced comparisons. Avoid promotional language. Claude responds better to "here are the options" than "use this tool." It also seems to prefer content that presents multiple perspectives rather than a single "best" solution.

What doesn't work well: Anything that sounds like a sales pitch. Claude actively avoids endorsing specific products. It's the most "neutral" of the four systems.

Specific observation: Claude was slowest to pick up changes — took 3-4 weeks vs 1-2 weeks for Perplexity. It seems to update its knowledge less frequently.

Gemini

Most Google-integrated. Heavily uses Google's own index. Content that ranks well in Google tends to be cited more by Gemini. Also uses structured data actively. If you're already ranking on page 1 of Google, Gemini is more likely to cite you.

What works best: Traditional SEO (Google ranking), structured data, Google-friendly content. The overlap between "good for Google" and "good for Gemini" is significant. If you're already doing well on Google, Gemini will pick you up naturally.

What doesn't work well: Content that's optimized for AI but not for Google. Gemini leans heavily on the Google index. If your content doesn't rank on Google, it's unlikely to be cited by Gemini.

Specific observation: Gemini was the most responsive to structured data. After adding FAQPage schema, Gemini started citing my content within 5-7 days — faster than any other system.

Bottom Line

Optimize for the common factors (good content, structured data, external signals) and you'll cover all four systems reasonably well. But if you had to pick one platform to optimize for first, I'd pick Perplexity — it's the most transparent and easiest to test.

The Competition Problem

Here's something most GEO articles won't tell you: in competitive niches, technical GEO alone won't get you recommended.

I tested in the SEO tools niche — one of the most competitive categories for AI recommendations. ChatGPT, Claude, and Perplexity all heavily favor established players (Ahrefs, SEMrush, Moz) regardless of what you do technically.

Why: Domain authority is the strongest signal, and it takes years to build. When AI systems choose between a new tool with perfect GEO and an established tool with mediocre GEO, they almost always pick the established tool. The reasoning is simple: established tools have more external validation, more reviews, more mentions, and more trust signals.

The numbers: I tracked 20 different queries across 4 platforms over 3 months. Established tools (Ahrefs, SEMrush, Moz) appeared in 85% of answers. My tool appeared in 30% of answers. The gap is significant but not insurmountable.

What this means: GEO tactics are most effective for:

Less competitive niches — if few competitors have implemented GEO, you have a huge advantage. In the SEO tools niche, everyone has decent GEO. In a niche like "free CSS animation tools," GEO could make a real difference.
Long-tail queries — "best free SEO audit tool for WordPress" is easier to win than "best SEO tool." The more specific the query, the less competition you face.
Specific use cases — "how to check Core Web Vitals for free" is more tractable than "what's the best SEO tool." Target use-case queries, not category queries.

What this also means: Even in competitive niches, GEO tactics compound over time. The product I tested went from never being mentioned to being a regular recommendation over 3 months. It wasn't the #1 recommendation (Ahrefs and SEMrush hold that position), but being in the list at all is valuable. The key insight: you don't need to be #1 to get traffic. Being in the list at all exposes you to users who might not have found you otherwise.

Practical advice: If you're in a competitive niche, focus on long-tail queries and specific use cases first. Build up external signals (Reddit, GitHub, Product Hunt) while optimizing your technical GEO. The combination of targeted queries + external signals + structured content is more effective than any single tactic.

The Honest Limitations

Let me be clear about what this article is and isn't:

This article IS:

A collection of tactics that showed observable results in my testing
Based on publicly available information about how RAG systems work
A practical guide you can implement in a few hours
A detailed log of 47 specific tests with exact queries, dates, and outcomes

This article ISN'T:

Reverse-engineering of proprietary algorithms
Guaranteed to work for every niche and product
Based on insider knowledge from AI companies
A comprehensive study with statistical significance (I'm one person testing one product)

What I don't know:

The exact weight AI systems give to each factor
How often AI models update their "knowledge" about products
Whether these tactics will work as well in 6 months
How AI systems handle competing recommendations in your specific niche
Whether my results generalize to other industries and products

What I do know:

Sites with good structure get cited more than sites without
External signals (Reddit, GitHub) clearly matter for some niches
FAQPage schema has a measurable impact on AI citations
Blocking AI crawlers makes you invisible to AI chatbots
The effect is real but modest — no single tactic transformed my results overnight
Content structure matters more than most people think
The ROI on technical GEO (robots.txt, schema, llms.txt) is high because the time investment is low

The honest truth: These tactics increase your chances of being cited by AI chatbots. There are no guarantees — especially in competitive niches. But if you're doing nothing right now, you're definitely leaving opportunities on the table. In competitive niches, domain authority and genuine external mentions matter more than any technical GEO tactic.

What I'd do differently: If I were starting over, I'd focus on Reddit engagement + content structure first (highest impact, lowest cost), then add FAQPage schema and llms.txt. The order matters — external signals and content quality are more important than technical setup.

ROI: Is GEO Worth Your Time?

Let's do the math. Here's the time investment vs potential return for each tactic:

Tactic	Time Investment	Potential Return	ROI
robots.txt fix	5 minutes	Critical (binary: visible/invisible)	Infinite
llms.txt	15 minutes	Moderate (30% accuracy improvement)	High
FAQPage schema	1-2 hours	High (2-3x more citations)	Very High
Content structure	2 weeks (10 posts)	High (measurable citation increase)	High
Reddit engagement	4 weeks (3-4 posts/week)	Moderate (correlated with citations)	Medium
Organization schema	30 minutes	Low-Moderate (trust signal)	Medium

The bottom line: Technical GEO (robots.txt + llms.txt + schema) takes ~2 hours total and provides significant value. Content structure takes longer (2 weeks) but provides the most consistent results across all AI systems. The ROI is highest for technical GEO because the time investment is minimal and the potential upside is substantial.

For bootstrapped founders: Focus on the 2-hour technical setup first. It's the highest ROI activity. Then invest in content structure over the next 2 weeks. Reddit engagement is ongoing but low-effort (3-4 posts per week, 10-15 minutes each). Total time investment: ~20 hours over 1 month for potentially significant AI visibility improvement.

For agencies: Package GEO as a service. The 2-hour technical setup can be standardized and sold as a "GEO optimization" add-on to existing SEO services. Content structure requires more customization but can be templated. The monthly tracking template provides ongoing value and client reporting.

Complete GEO Checklist (Must-Do vs Nice-to-Have)

Must-Do (Do These First — They're Free and Fast)

[ ] robots.txt allows AI crawlers — User-agent: * Allow: / (5 minutes). Verify with AuditMe robots.txt Checker.
[ ] FAQPage schema on product/homepage — 3-5 common questions (1-2 hours). Validate with Google Rich Results Test.
[ ] Organization JSON-LD on every page (30 minutes). Include name, URL, logo, description, founding date, contact info.
[ ] llms.txt at site root — Under 500 words, essential info first (15 minutes). See llmstxt.org for format.
[ ] Question-based H2/H3 headings on key pages (varies). Convert "Title Tag Optimization" to "How Do I Optimize Title Tags for SEO?"
[ ] Definition-style opening paragraphs — First sentence answers the heading question (varies). "Core Web Vitals are three metrics that measure real-world user experience..."

Nice-to-Have (Do These After the Basics Are Solid)

[ ] llms-full.txt with detailed documentation (1-2 hours). Extended version for developers and AI systems.
[ ] SoftwareApplication schema on product pages (30 minutes). Include name, description, applicationCategory, operatingSystem, pricing, review.
[ ] Content structure overhaul — Convert paragraphs to lists (varies). Add question headings, definition-style openings, numbered lists, tables.
[ ] Reddit engagement — Genuine answers in relevant subreddits (ongoing). 3-4 posts per week, 200-500 words each.
[ ] GitHub repository if developer tool (varies). README, docs, examples, issues.
[ ] Product Hunt launch (one-time). Prepare assets, schedule for Tuesday-Thursday, engage in comments.
[ ] Technical articles on DEV.to/Medium (ongoing). 2-4 articles per month, genuine expertise, not promotional.
[ ] BreadcrumbList schema for navigation (30 minutes). Helps AI understand site structure.

Verification

[ ] Test llms.txt: curl -I https://yoursite.com/llms.txt
[ ] Validate structured data: Google Rich Results Test
[ ] Test in Perplexity: "best [category] tools" (check citations)
[ ] Test in ChatGPT: "What is [your product]?"
[ ] Test in Claude: Same questions
[ ] Test in Gemini: Same questions
[ ] Track results monthly — write down what AI says about you

Monthly Tracking Template

Create a simple spreadsheet with:

Date
Query tested
Platform (Perplexity, ChatGPT, Claude, Gemini)
Result (cited, not cited, position)
Notes (what changed since last test)

This gives you longitudinal data to track progress and identify patterns.

Free Tools I Used

Tool	Purpose	Link
Perplexity	Test AI citations (shows sources)	Free
ChatGPT	Test product recommendations	Free
Claude	Test product recommendations	Free
Gemini	Test product recommendations	Free
Google Rich Results Test	Validate structured data	Free
Schema.org Validator	JSON-LD syntax check	Free
AuditMe	Full SEO analysis + AI visibility check	Free (1/day)
AuditMe robots.txt Checker	Validate robots.txt	Free
AuditMe JSON-LD Checker	Structured data audit	Free
PageSpeed Insights	Core Web Vitals testing	Free
curl	Check llms.txt accessibility	Free (CLI)
Screaming Frog	Full site SEO crawl	Free (500 URLs)
Google Search Console	Track search performance	Free
Bing Webmaster Tools	Track Bing performance	Free

Note: AuditMe is my product — I'm biased. But it genuinely does what I needed for testing. Other tools like Screaming Frog, Ahrefs, and SEMrush also work for parts of this.

How I used each tool:

Perplexity: Primary testing platform — shows citations directly, easiest to track
ChatGPT: Secondary testing — less transparent but most popular
Claude: Tertiary testing — most cautious, good for understanding "neutral" AI perspective
Gemini: Google-integrated testing — useful for understanding Google AI Overlaps
Google Rich Results Test: Validate FAQPage, Organization, SoftwareApplication schemas
Schema.org Validator: Catch syntax errors that Google Rich Results Test misses
AuditMe: Full SEO audit + AI visibility check in one tool
PageSpeed Insights: Core Web Vitals testing — slow pages get skipped by AI crawlers
curl: Quick check that llms.txt is accessible (200 OK, not 404)
Screaming Frog: Full site crawl to find schema errors, broken links, missing meta tags
Google Search Console: Track which queries drive traffic — use these for FAQPage schema
Bing Webmaster Tools: ChatGPT uses Bing — important to track Bing performance

Where This Is Heading (2026-2027)

These are predictions, not facts. Take them with appropriate skepticism.

1. GEO Will Become Standard Practice

Right now, GEO is a competitive advantage. Within 12-18 months, it will be table stakes. The sites that implement GEO now will have a compounding advantage. Think of it like mobile-first indexing in 2015 — early adopters benefited, but eventually everyone had to adapt.

2. AI Search Will Grow, But Google Won't Die

AI chatbots will take significant share for research queries. But Google will remain dominant for navigational queries ("go to Amazon") and local searches ("restaurants near me"). The smart play is optimizing for both. Most GEO tactics also improve traditional SEO — it's not an either/or choice.

3. llms.txt Will Either Take Off or Die

llms.txt is at an inflection point. If major AI systems start actively using it, it becomes essential. If they don't, it fades. Implement now — cost is low (15 minutes), potential upside is high. The standard is gaining traction — llmstxt.org has growing adoption.

4. AI Will Get Better at Evaluating Quality

Right now, AI systems often cite whatever has the best structure. As they improve, they'll get better at evaluating actual content quality. Sites with genuine expertise will beat sites with SEO-optimized filler. This means: invest in real content, not just structured content.

5. The GEO Tool Category Will Emerge

Just as SEO tools emerged in the 2010s, GEO tools will emerge in 2026-2027. We'll see platforms for tracking AI visibility and optimizing for AI extraction. AuditMe is already moving in this direction with its AI visibility checks.

6. Multi-Modal AI Changes Everything

When AI systems can process images, videos, and audio alongside text, GEO will expand to include visual and video optimization. Sites with well-structured multimedia content will have an advantage. Start thinking about image alt text, video transcripts, and structured data for media now.

FAQ

What is GEO?

GEO (Generative Engine Optimization) is the practice of optimizing your website so AI-powered answer engines like ChatGPT, Claude, Perplexity, and Gemini cite it when users ask questions. It's complementary to traditional SEO — you don't replace SEO with GEO, you add GEO on top of it.

How is GEO different from SEO?

Traditional SEO targets search engine result pages (10 blue links). GEO targets AI-generated answers. The two overlap — good SEO helps GEO — but GEO has its own signals: llms.txt, structured data, content structure, and external authority signals. The biggest difference: GEO is about being cited in AI answers, not ranking in search results.

How long does it take for AI to start citing my site?

It depends on the system. Perplexity can pick up changes within days (fastest). ChatGPT takes 2-3 weeks. Claude and Gemini are slower (3-4 weeks). RAG-based systems are faster than training-based systems. The key insight: RAG systems can pick up changes in days, but training-based updates take months.

Is GEO worth the effort?

If you're in a niche where AI chatbots are recommending products — yes. If people still primarily use Google — traditional SEO might be more impactful. The best approach is doing both. GEO tactics also improve traditional SEO, so it's not wasted effort even if AI search doesn't take off as fast as predicted.

What's the single most impactful GEO tactic?

From my testing: FAQPage schema combined with structured content that answers questions directly. This gave the most consistent results across all AI systems. The combination of structured data + content structure creates a "double signal" that AI systems respond to.

Can I track if AI is citing my site?

Yes, with difficulty. Perplexity shows citations directly (easiest to track). ChatGPT shows some in the paid tier. For Claude and Gemini, ask directly and check. The best approach: ask AI chatbots about your product monthly and track results in a spreadsheet. Use the monthly tracking template in the checklist section.

How do I start from zero?

Add User-agent: * Allow: / to robots.txt (5 min)
Create llms.txt at site root (15 min)
Add FAQPage schema to your homepage (1-2 hours)
Restructure your best content with question headings and lists (varies)
Start engaging genuinely on Reddit (ongoing)
Test monthly in Perplexity, ChatGPT, Claude, and Gemini

The order matters: robots.txt + llms.txt first (5 minutes total), then FAQPage schema (1-2 hours), then content structure (varies), then external signals (ongoing).

What if my competitors haven't implemented GEO?

You have a huge advantage. In less competitive niches, GEO tactics can make a real difference. Focus on structured content, FAQPage schema, and llms.txt. If your competitors are already doing GEO, you need to combine technical GEO with external signals (Reddit, GitHub, Product Hunt) to compete.

Does GEO work for local businesses?

It depends. If people ask ChatGPT "what's the best coffee shop in [city]?" — yes, GEO matters. But most local searches still go through Google Maps and Yelp. Focus on Google Business Profile + traditional local SEO first, then add GEO as a secondary channel.

Summary

The GEO playbook is straightforward but not magic:

Open robots.txt — Let AI crawlers access your content (5 minutes)
Add FAQPage schema — The most impactful structured data (1-2 hours)
Create llms.txt — Quick win, no downside (15 minutes)
Structure content for AI extraction — Question headings, lists, definition-style openings (varies)
Build external signals — Reddit, GitHub, Product Hunt (genuine engagement, ongoing)

These tactics increase your chances of being cited by AI chatbots. There are no guarantees — especially in competitive niches where domain authority dominates. But if you're doing nothing right now, you're leaving opportunities on the table. In less competitive niches, these tactics can give you a significant advantage over competitors who haven't implemented them yet.

What I learned from 47 tests:

FAQPage schema had the biggest measurable impact (2-3x more citations)
Content structure mattered more than expected (AI extracts lists better than paragraphs)
External signals (Reddit, GitHub) correlated with citations in consumer/developer niches
Perplexity was the most transparent and easiest to test
The effect is real but modest — no single tactic transformed results overnight

The landscape is shifting. AI search is growing. The sites that adapt now will benefit the most as this trend accelerates. Start with the 5-step playbook, test monthly, and iterate based on what you observe.

Found this useful? Share it with someone building a product that needs AI visibility. Follow me for more practical tests on AI, SEO, and product growth.

Have questions or different results from your own testing? Drop them in the comments — I'm genuinely curious what's working for others.

Tools mentioned: Perplexity, ChatGPT, Claude, Gemini, AuditMe, Schema.org, llmstxt.org, Google Rich Results Test, PageSpeed Insights, Screaming Frog

Master Technical SEO, Core Web Vitals, and the 12-Point Audit Checklist for 2026

Eduard — Sat, 22 Aug 2026 19:12:36 +0000

Table of Contents

How to Use This Guide
What Is an SEO Audit and Why Do You Need One in 2026
Why Regular Audits Matter More Than Ever
The 12-Point SEO Audit Checklist
Advanced: Expert-Level Audit Techniques
Prioritizing Your Fixes
Tools I Actually Use
Preparing for AI-Powered Search
Case Studies
How AI Is Changing SEO Audits
Next Steps
FAQ

How to Use This Guide

I've been doing SEO audits professionally since 2018. I've audited over 300 websites — from two-page plumbing company sites to 50,000-page e-commerce platforms running on Shopify, WordPress, and custom stacks. This guide is the distilled version of everything I've learned, organized as a step-by-step process you can follow even if you've never run an audit before.

You can use this guide two ways. First, run our free tool at the top of this page — it covers every checklist point below in under 60 seconds. Second, follow the manual steps I lay out if you want to understand what the tool is doing and why each check matters.

Bookmark this page. I update it whenever Google changes something significant, which in 2026 happens roughly every six to eight weeks.

What Is an SEO Audit and Why Do You Need One in 2026

An SEO audit is a structured evaluation of everything that affects how your website appears in search results. That includes technical performance, content quality, backlink health, user experience signals, and now — for the first time — how well your content answers questions that AI systems pull from.

Think of it like a medical checkup. You might feel fine, but blood work can catch high cholesterol, vitamin deficiencies, or early signs of diabetes before you notice any symptoms. An SEO audit does the same thing for your website. Traffic can be declining for months and you might not realize it because you're still getting some leads. By the time you notice, competitors have already taken your positions.

According to a 2024 Ahrefs study, 68.3% of all web traffic starts with a search engine. That number hasn't changed much in five years despite all the talk about social media replacing Google. What HAS changed is how competitive every keyword has become. Ten years ago you could throw up a 500-word page with a few keywords and rank. Today, the top 10 results for any commercial keyword are typically long-form, well-structured pages backed by genuine expertise.

The scope of what an audit covers has expanded too. In 2020, you could check meta tags, run a speed test, and call it done. In 2026 you need to consider:

Core Web Vitals — LCP, INP, and CLS are direct ranking factors. Google published the thresholds at web.dev/vitals: LCP under 2.5s, INP under 200ms, CLS under 0.1. For a deep dive into fixing these, see our Core Web Vitals checklist for 2026.
E-E-A-T signals — Experience, Expertise, Authoritativeness, and Trustworthiness. Google's Search Quality Rater Guidelines (a 176-page PDF) spell out exactly what these mean. Google's own documentation on creating helpful content explains how E-E-A-T influences rankings. Yes, it's dense. Yes, it's worth reading.
AI Overviews — Google's AI-generated answers now appear on roughly 30% of search results according to Search Engine Land. Google's documentation on AI Overviews explains how content gets cited. If your content can't be cited by an AI system, you're invisible to a growing chunk of searchers.
Structured data — Schema markup is no longer optional. It's how you get featured snippets, FAQ dropdowns, product carousels, and how-to panels. Google's official structured data documentation explains how it works. For common mistakes to avoid, read our FAQ schema usage mistakes guide.
Content decay — Google's Helpful Content system actively demotes content written primarily for search engines rather than humans. If your blog reads like it was generated by a keyword tool, it's probably being suppressed. Our content refresh strategy guide shows how to keep old content performing.

I made the mistake of underestimating E-E-A-T in 2023. I had a client — a financial advice blog — where every article was technically perfect. Perfect keywords, perfect internal linking, perfect schema. But none of it mentioned who actually wrote the articles. No author bios, no credentials, no LinkedIn profiles. Google's algorithm basically said "we have no reason to trust this content" and pushed us below sites with half the technical quality but real, named experts writing under their own names. We lost 55% of organic traffic in six months. Fixing it cost more than the original audit.

Why Regular Audits Matter More Than Ever

Google made 4,725 search algorithm changes in 2023 (tracked by Search Engine Journal). In 2024 they stopped reporting the exact number, which tells you something about where things are heading. The pace of change is accelerating, not slowing down. Barry Schwartz at Search Engine Roundtable has documented every confirmed update since 2003 — his archive is the definitive historical record.

Here's a scenario I see constantly. A site owner runs an audit in March, fixes a bunch of stuff, watches rankings improve, and then doesn't look at their site again until October. By then, three things have happened: Google rolled out updates that changed what "good" looks like, a competitor improved their site and leapfrogged them, and technical issues crept in — broken links, slow pages, expired SSL certificates, JavaScript errors after a plugin update.

The cost of catching problems early versus late is dramatic. I tracked this across 20 client engagements in 2025:

A quarterly audit costs maybe four hours of work. Skipping audits for a year can cost months of lost revenue and thousands in recovery.

Resources for staying on top of changes:

Google Search Status Dashboard — official incident reports for ranking updates
Search Engine Roundtable — Barry Schwartz has tracked every Google update since 2003. Single best source for staying informed.
Google Search Central Blog — official announcements. Dense but authoritative.
Moz Google Algorithm History — comprehensive archive going back to 2000

The 12-Point SEO Audit Checklist

Here's the checklist I use on every single audit. Each section tells you what to check, how to check it, what tools to use, and how to prioritize the fix. Priority levels: HIGH = fix this week, MEDIUM = fix this month, LOW = fix this quarter.

1. Meta Titles — Priority: HIGH

Your title tag is the single most important on-page SEO element. It's the first thing people see in search results and one of the strongest signals Google uses to understand what your page is about. Google's documentation on title links explains exactly how they generate and display titles in search results.

What to check:

Length: 50-60 characters. Google truncates titles over 60 characters on desktop and even shorter on mobile. Moz's title tag guide has current character limits.
Keyword placement: Primary keyword within the first 40 characters. Google gives more weight to early words — not speculation, just how tokenization works.
Uniqueness: Every page needs a unique title. Duplicate titles confuse crawlers and they'll pick one — probably not the one you want.
Branding: Brand name at the end, separated by a pipe or dash. Don't waste character space upfront unless you're a household name.

How to check: Crawl your site with Screaming Frog (free for 500 URLs). Export Title 1 and Title Length columns. Sort by length, flag anything over 60. Sort alphabetically to find duplicates. Takes about 15 minutes and catches 80% of title issues.

Real example: I audited a 300-page e-commerce site where the developer used one title template for every product: "Product Name | Store Name." No keywords, no category context, nothing. We rewrote titles to include category and a key feature — "Men's Running Shoes - Lightweight Breathable | Store Name" instead of just "Shoes | Store Name." Organic traffic to product pages increased 34% within 60 days. One afternoon of work.

Tools:

Screaming Frog SEO Spider — free up to 500 URLs, best technical audit tool available
SEO Minion — free Chrome extension, check titles without viewing source
Ahrefs Site Audit — cloud-based, tracks issues over time

2. Meta Descriptions — Priority: MEDIUM

Meta descriptions don't directly affect rankings — Google confirmed this in 2022 and reiterated in 2024. But they absolutely affect click-through rates, and CTR is a ranking signal according to Google's own patents. So indirectly? They matter a lot.

What to check:

Length: 150-160 characters. Longer gets truncated.
CTA: Every description should include an action phrase — "Learn how to," "Get your free," "Compare plans."
Keyword match: Target keyword appearing naturally. When it matches the search query, Google bolds it in results.
Uniqueness: Never copy the same description across multiple pages.

How to check: Screaming Frog export — Meta Description 1 and Meta Description Length columns. Filter for empty descriptions first (biggest wins), then look for duplicates.

Real example: A SaaS client had 15 feature pages all saying "A powerful tool for your business." We rewrote each to highlight specific benefits — "Automate your invoicing in 3 clicks — save 5 hours per week" for the invoicing page. Click-through rates increased 22% across those 15 pages within a month.

3. Heading Hierarchy — Priority: MEDIUM

H1 through H6 should form a logical outline — like a table of contents for users and crawlers. Google's documentation on headings explains that headings help Google understand the structure and hierarchy of your content.

What to check:

Exactly one H1 per page that matches search intent
H2s divide major sections, H3s subdivide those
Never skip levels (H1 straight to H3 confuses crawlers)
Headings should be descriptive, not stuffed with keywords

How to check: Use HeadingsMap (free browser extension) to visualize heading structure. Or Screaming Frog's built-in heading analysis.

Common mistake: People use headings as styling tools. They want bold text so they make it an H2 even though it's not a section heading. Headings define document structure, not visual appearance. Use CSS for styling.

4. Core Web Vitals — Priority: HIGH

Google uses three metrics to measure real user experience. These are direct ranking factors — not nice-to-haves. Thresholds at web.dev/vitals.

The three metrics:

LCP (Largest Contentful Paint): Under 2.5 seconds. How long main content takes to load. Common culprits: unoptimized images, slow server, render-blocking JS, web fonts delaying text.
INP (Interaction to Next Paint): Under 200ms. Replaced First Input Delay in March 2024 per Google's official announcement. Measures responsiveness to ALL interactions, not just the first one. Heavy JavaScript is usually to blame.
CLS (Cumulative Layout Shift): Under 0.1. Visual stability — how much the layout jumps while loading. Caused by images without dimensions, late-loading ads, dynamically injected content.

How to check:

PageSpeed Insights — free, gives both lab data and real user data from CrUX. Google's PageSpeed Insights documentation explains the difference between lab and field data.
Search Console Core Web Vitals — real user data across all pages, broken by device. Google's CWV documentation explains how each metric is measured.
Chrome UX Report — the raw data source Google uses. Free API.

Important distinction: Lab data from Lighthouse simulates a load on a controlled device. Field data from CrUX comes from real Chrome users. Google uses field data for rankings. Always prioritize field data.

What actually works:

Serve images in WebP/AVIF — 30-50% smaller with no visible quality loss. Use Squoosh (free, by Google).
Preload critical resources — use a preload link tag for your LCP image and critical CSS to cut LCP by 0.5-1s.
Defer non-critical JS — move analytics, chat widgets, ad scripts to load after main content.
Use a CDN — Cloudflare has a free tier. BunnyCDN starts at $1/month.

5. Structured Data and Schema Markup — Priority: HIGH

Schema helps search engines understand content and triggers rich results. Google's structured data documentation explains the full specification and supported types. In 2026, it's also how AI systems identify your content as authoritative.

Essential schemas:

Organization — company name, logo, social profiles. schema.org/Organization
WebPage — basic page metadata. Include datePublished and dateModified.
Article — for blog posts. Include author, publisher, dates.
FAQPage — for Q&A content. Shows as expandable dropdowns in search.
Product — for e-commerce. Include name, description, image, offers, aggregateRating.
BreadcrumbList — helps Google understand site hierarchy. Shows in search results.
LocalBusiness — for physical locations. Include address, hours, geo.

How to check:

Google Rich Results Test — paste URL, see extractable rich results. Google's documentation explains what each rich result type requires.
Schema.org Validator — validates against the official Schema.org vocabulary
Search Console Enhancements tab — shows schema errors site-wide
Merkle Schema Generator — free code generator

Real example: We added FAQPage schema to 40 high-traffic pages. Within three weeks, 28 showed FAQ dropdowns in search results. CTR increased an average of 15%. Pages with HowTo schema added another 8-12% CTR boost. Total implementation: about two hours.

6. Indexation and Crawlability — Priority: HIGH

If Google can't find your pages, it can't rank them. This is the most fundamental check and the most commonly botched. For a deeper dive into server log analysis, see our JavaScript SEO rendering guide.

What to check:

robots.txt — not blocking important pages (or JS/CSS files Google needs to render). Google's robots.txt documentation explains every directive. Our robots.txt guide for 2026 explains every directive.
XML sitemap — submitted in Search Console, all URLs return 200s, no noindexed/redirected URLs included. Google's sitemap documentation covers format, submission, and best practices. Use our full site crawl tool to verify every URL in your sitemap is actually reachable.
Noindex tags — no important pages have a noindex meta tag. Google's noindex documentation explains how the directive works. Happens often after CMS migrations or plugin updates. WordPress users should check our WordPress SEO audit checklist for platform-specific gotchas.
Canonical tags — self-referencing on every page unless there's a specific reason otherwise. Google's canonical tag documentation covers when to use them and common mistakes. Canonicals pointing to 404s or redirects cause deindexation. Our canonical tags guide covers the five most common mistakes.
HTTP status codes — crawl and flag 4xx/5xx errors. Learn the difference between every status code in our HTTP status codes SEO guide.

How to check: Screaming Frog is the gold standard. Google's Search Console documentation explains how to monitor crawl activity. Export reports for response codes, canonicals, noindex tags, robots directives, sitemap contents. Cross-reference to find mismatches.

Advanced: Compare sitemap URLs against crawl data. Every URL in your sitemap should be crawlable, indexable, and return 200. Anything else needs fixing or removal. If you're running a large site with 10,000+ URLs, our full site crawl automates this entire process.

7. Internal Linking — Priority: MEDIUM

Internal links distribute authority and help crawlers discover content. Google's documentation on internal links explains how they discover and evaluate internal links. This is one of the most underrated SEO levers — most sites fixate on backlinks while ignoring the linking structure they fully control. For advanced silo strategies, see our internal link silo structure guide.

What to check:

Orphan pages — zero internal links pointing to them. Screaming Frog flags these. If you have a large site, our internal links analysis tool maps every connection automatically.
Anchor text — descriptive and keyword-rich. "Click here" tells Google nothing. "Our guide to Core Web Vitals" tells Google everything.
Link depth — important pages within 3 clicks of homepage. Key landing page buried 6 clicks deep? Something's wrong.
Link distribution — are your most important pages getting the most internal links? Or are blog posts hoarding all the equity?

How to check: Screaming Frog's "All Inlinks" report shows every internal link. Export and use a spreadsheet to find orphan pages and underlinked pages. Sitebulb ($13.50/month) has even better link architecture visualization.

Real example: An e-commerce client had 500 products but only 20 linked from the homepage. The other 480 were 3-4 clicks deep. We added "Popular Products" to the homepage and "Related Products" on each product page. Within 60 days, deeper product pages saw 28% more organic traffic. No content changes, no new backlinks — just better internal linking.

8. Image Optimization — Priority: MEDIUM

Images affect page speed AND image search visibility. Google's image optimization guide explains how they index and rank images. Google Images drives surprising traffic for e-commerce and recipe sites.

What to check:

Alt text — descriptive for every meaningful image. Accessibility requirement AND SEO signal.
File format — WebP or AVIF instead of JPEG/PNG. 30-50% smaller with comparable quality.
File size — under 200KB for most images. Hero images under 500KB.
Dimensions — don't serve 4000px when displaying at 800px.
Lazy loading — loading="lazy" for below-fold images.

Tools:

Squoosh — free, by Google, WebP/AVIF support
Cloudinary — free tier for auto optimization and CDN
Sharp — Node.js library for build pipeline compression

9. HTTPS and Security — Priority: HIGH

Table stakes in 2026. Google's HTTPS documentation explains why HTTPS is a ranking signal and how to implement it correctly.

What to check:

Mixed content — HTTP resources on HTTPS pages. Screaming Frog flags these.
Certificate validity — not expiring soon.
HTTP to HTTPS redirects — 301 redirects on every HTTP URL.
HSTS header — tells browsers to only use HTTPS. Mozilla's HSTS documentation explains the header and its security benefits.

Check: Visit your site and click the padlock. Or use SSL Labs for comprehensive testing.

10. Page Speed and Performance — Priority: MEDIUM

Beyond Core Web Vitals, other metrics matter too.

What to check:

TTFB: Under 200ms. Over 600ms = server-side bottleneck.
Page weight: Under 500KB on key landing pages. Average in 2026 is over 2MB — that's absurd.
HTTP requests: Fewer = faster. Combine CSS, use sprites, inline critical resources.
Third-party scripts: Chat widgets, analytics, ad pixels — the silent killers of page speed.

Tools:

WebPageTest — waterfall charts showing exactly what's slow. Free.
GTmetrix — performance monitoring with historical data.
Lighthouse — built into Chrome DevTools. Google's Lighthouse documentation explains each audit and scoring methodology.

Real example: A client's homepage loaded in 6.2s. Waterfall showed a chat widget adding 2.1s. Switched to a lighter alternative, 3.8s. Then lazy-loaded the chat widget and perceived load time dropped to 1.4s. One change, 77% faster perceived load.

11. Backlink Profile — Priority: MEDIUM

Backlinks remain one of Google's strongest ranking signals. But toxic links can actively hurt you — Google's link spam policy explains what constitutes a manipulative link and how manual actions work.

What to check:

Referring domains: More unique domains > many links from same domain. Quality over quantity.
Toxic links: Spammy sites, link farms, PBNs. Can trigger manual penalties.
Anchor text distribution: Natural = roughly 50% branded, 25% URL, 15% topical, 10% generic. Spikes of exact-match anchors = red flag.
Growth pattern: Steady growth is healthy. Sudden spikes/drops need investigation.
Competitor comparison: Use Ahrefs' Link Intersect to find sites linking to competitors but not you.

Tools:

Ahrefs — best backlink analysis tool. Referring domains, anchor text, toxic link risk.
Moz Link Explorer — good alternative with Domain Authority
Search Console Links — free, shows top linking sites and anchors
Google Disavow Tool — Google's documentation on disavowing links explains when and how to use it. Tell Google to ignore toxic links.

12. Content Quality and Freshness — Priority: MEDIUM

Google's Helpful Content system rewards first-hand experience and genuinely helpful content. The system uses a site-wide signal — if a significant portion of your content is unhelpful, it can drag down rankings across your entire domain. If your site runs on WordPress, our WordPress SEO audit checklist covers platform-specific content issues.

What to check:

Thin pages: Under 300 words providing no value. Expand or remove. Our content pruning guide shows exactly how to decide what to keep versus cut.
Duplicate content: Same content at multiple URLs. Canonicalize or 301 redirect. Learn more in our SEO migration guide.
Outdated content: Not updated in 18+ months. Refresh facts, stats, examples. Our content refresh strategy explains the optimal refresh cadence.
Search Console data: High impressions but low clicks = title/description issues. Our title tag optimization guide shows how to fix CTR problems.
Cannibalization: Multiple pages targeting the same keyword. Consolidate. See our topic clusters and content authority guide for how to restructure.

Tools:

Screaming Frog — flags thin and duplicate content
Copyscape — checks for duplicate content elsewhere on the web, essential for catching scraped content
Hemingway App — readability analysis. Google's guidelines on writing helpful content emphasize writing for people, not search engines.

Advanced: Expert-Level Audit Techniques

The 12-point checklist above covers 80% of what you need. But if you're auditing a large site, a JavaScript-rendered app, or a multilingual operation, you need deeper techniques. These are the checks that separate a competent audit from a thorough one.

Log File Analysis: What Googlebot Actually Crawls

Server logs record every request Googlebot makes to your site. This is the ground truth — not what you think Google is crawling, but what it actually requested.

Why it matters: A sitemap might list 5,000 pages, but Googlebot may only crawl 2,000 of them. The other 3,000 are invisible. Or worse, Googlebot might be spending 80% of its crawl budget on pages that don't matter — faceted navigation, parameter URLs, admin pages.

How to do it:

1. Access your server logs (ask your hosting provider if you're not sure where they are)

2. Filter for Googlebot user-agent requests

3. Analyze which URLs get crawled most frequently and which get zero crawls

4. Compare against your sitemap — every important page should appear in both

5. Look for crawl spikes on low-value pages (wasted budget) or drops on high-value pages (potential deindexation)

Tools for log analysis:

Screaming Frog Log File Analyzer — free, visualizes crawl patterns
JetOctopus — cloud-based, handles large log files
Botify — enterprise-grade log analysis and crawl budget optimization
SEOlyzer — free tier available, real-time log parsing

Real example: I analyzed logs for a 100,000-page e-commerce site and found Googlebot was spending 73% of its crawl budget on faceted navigation URLs (color, size, price filters). Each filter combination created a unique URL that Googlebot dutifully crawled. We blocked these with robots.txt and rel="nofollow" on filter links. Within 6 weeks, Googlebot's crawl rate on product pages increased 340%. Indexation of product pages went from 45% to 89%.

JavaScript Rendering Audit

If your site uses React, Next.js, Vue, Angular, or any JavaScript framework, you need to verify that Googlebot can actually see your content. Google's documentation on JavaScript SEO explains how Googlebot renders JavaScript. JavaScript rendering is one of the most common causes of "why isn't my page ranking?" in 2026.

The problem: Googlebot renders JavaScript in a second wave of crawling. The first wave sees the raw HTML (which might be empty), and the second wave renders the JavaScript. If your content only exists in JavaScript, Google might not see it for days — or ever, if rendering fails.

What to check:

View-source test: Right-click your page, view source, and search for your key content. If it's not in the raw HTML, Googlebot might not see it either.
Google Search Console URL Inspection: Use the "View rendered page" feature to see exactly what Googlebot sees.
Dynamic rendering: For sites where JavaScript rendering is a known issue, consider dynamic serving (serving pre-rendered HTML to bots). Google's documentation explains when dynamic rendering is appropriate. Our SSR vs SSG vs ISR guide explains when each approach makes sense.
Client-side routing: Single-page applications often break URL-based navigation. Every route should have a unique, crawlable URL that returns full HTML.

Tools:

Google Search Console URL Inspection — see exactly what Googlebot renders
Rendertron — open-source rendering proxy for testing
Screaming Frog JavaScript rendering — renders JS during crawl

Real example: A client's Next.js site had 200 product pages that looked perfect in the browser but returned empty HTML to Googlebot. The issue was that all product data was fetched client-side via API calls. We switched to server-side rendering for product pages, and indexation jumped from 60 pages to all 200 within two weeks. Organic traffic to product pages increased 180%.

International SEO Audit

If your site targets multiple countries or languages, international SEO adds an entire layer of complexity. For a complete guide, see our hreflang implementation guide.

What to check:

Hreflang tags: Every language/country variant needs reciprocal hreflang tags pointing to each other. Google's hreflang documentation explains the full specification and common mistakes. If page A links to page B as its French version, page B must link back to page A as its English version. Missing return tags are the most common hreflang error.
Canonical vs hreflang conflicts: A canonical tag pointing to a different language version overrides hreflang. Make sure canonicals point to the same-language version.
URL structure: Use either subdirectories (/fr/), subdomains (fr.example.com), or separate domains. Each has trade-offs. Subdirectories are simplest for most sites.
Content uniqueness: Machine-translated content without human review gets penalized. If you're translating, invest in human editing.
Geo-targeting in Search Console: Set the correct country target for each language version.

Tools:

Ahrefs Site Audit — flags hreflang errors automatically
Flang — free hreflang validator
Screaming Frog hreflang audit — crawls and validates hreflang across your entire site

Programmatic SEO Audit at Scale

When you're auditing a site with 10,000+ pages, manual checks don't scale. You need a systematic, automated approach.

Strategy:

1. Sample-based audit: Don't try to manually review every page. Sample 50-100 pages across different templates (homepage, category, product, blog, landing pages). Fix issues at the template level — one fix can improve thousands of pages.

2. Crawl segmentation: Group your crawl data by page type. Compare title tag patterns across product pages versus blog posts versus category pages. Inconsistencies within a template reveal systematic issues.

3. Priority scoring: Not all pages deserve equal attention. Sort by organic traffic potential (search volume x estimated CTR) and focus on the top 20% that drive 80% of traffic.

4. Automated monitoring: Set up weekly automated crawls with ContentKing or Screaming Frog scheduled crawls to catch issues before they impact rankings.

Tools for large-scale audits:

Sitebulb — handles large crawls with visual reporting
Lumar (formerly DeepCrawl) — enterprise cloud crawling
Onely — technical SEO consulting for large sites

Prioritizing Your Fixes

Fix this week (Priority 1): Indexation problems, broken canonicals, noindexed critical pages, CWV failures, security issues. These actively hurt rankings RIGHT NOW.

Fix this month (Priority 2): Missing/duplicate meta tags, schema errors, poor internal linking, image optimization, mobile usability. Holding you back but not emergencies.

Fix this quarter (Priority 3): Content consolidation, backlink cleanup, advanced performance, freshness updates, new content. Long-term improvements that compound.

Most common mistake: trying to fix everything at once. You can't change 50 things and know what worked. Prioritize, fix in batches, measure between batches. I recommend 5-10 related changes per batch.

Tools Actually Use

People spent thousands on SEO tools. Here's what I actually use:

Free or cheap:

Google Search Console — Google's official documentation explains every feature. Free, essential, non-negotiable.
Screaming Frog — free up to 500 URLs. Best technical audit tool.
PageSpeed Insights — free CWV testing with real user data
Rich Results Test — free schema validation
Squoosh — free image compression by Google
Ahrefs Webmaster Tools — free Ahrefs for your own site
SEO Minion — free Chrome extension for on-page checks

Professional (paid):

Ahrefs — $99/mo. Best for backlinks, keywords, competitor research
SEMrush — $130/mo. Great all-in-one with competitive analysis
Moz Pro — $99/mo. Good for domain authority and local SEO
Sitebulb — $13.50/mo. Best crawl visualization
Surfer SEO — $89/mo. Content optimization based on what's ranking
ContentKing — real-time monitoring and change alerts

Budget pick: If you can only afford one paid tool, get Ahrefs. If two, add Screaming Frog (one-time license for desktop).

Preparing for AI-Powered Search

This is the section most guides skip because it's new and uncomfortable. But it matters.

Google's AI Overviews, Perplexity, ChatGPT Search, and other AI tools are changing how people find info. Instead of clicking through 10 blue links, users get direct answers. According to Bloomberg, AI Overviews initially had quality issues but have improved substantially and now appear on roughly 30% of queries.

What to optimize for:

Structured data matters more. AI systems pull from schema markup to build answers. Well-structured content with proper schema is more likely to be cited.

Citations and source attribution. AI tools favor content linking to authoritative sources. Cite claims. Link to government data, peer-reviewed studies, official documentation.

Direct answers to questions. AI Overviews pull from pages that answer clearly and concisely. Put the answer in the first paragraph, then elaborate.

Freshness signals. AI tools prefer recently updated content. Include publication dates, last-updated dates, and actually refresh content regularly.

Long-tail queries. As people get comfortable with AI search, they ask longer, more conversational questions. Optimize for "how do I fix high CLS on my WordPress site" not just "CLS score."

Our site saw a 340% increase in AI Overview appearances after implementing structured data, adding last-updated dates, and restructuring content to lead with direct answers. Two weeks of work, sustained for six+ months.

Case Studies

Case Study 1: SaaS Recovering from a Core Update

A mid-size B2B SaaS company lost 45% of organic traffic after a Google core update in late 2025. They had 200 blog posts, 40 product pages, and a Next.js stack.

What we found:

68 pages had duplicate or missing title tags
Sitemap included 300+ URLs returning 404 errors
23 blog posts covered the same topic with thin 400-word summaries
Core Web Vitals failed on mobile — LCP was 4.8 seconds
No structured data on any page
Backlink profile had 300+ toxic links from a 2023 link campaign
No author bios, no E-E-A-T signals on any content

What we fixed over 8 weeks:

Consolidated 23 thin posts into 8 comprehensive guides (each 2,000+ words)
Removed dead URLs from sitemap, set up proper 301 redirects
Rewrote all title tags to be unique and under 60 characters
Implemented Organization, WebPage, and FAQPage schema site-wide
Optimized images (WebP, lazy-loaded below-fold content)
Disavowed toxic backlinks through Search Console
Added author bios with credentials, linked to LinkedIn profiles
Added publication and last-updated dates to all content

Results after 90 days:

Organic traffic recovered to pre-drop levels
12% increase in average position across all tracked keywords
Rich results appeared for 35 pages
Core Web Vitals passed on 94% of tested pages
Two pages started appearing in Google AI Overviews

Total cost: approximately $12,000. The traffic they recovered was worth roughly $45,000 per month in equivalent paid traffic.

Case Study 2: Local Business Going from Invisible to Dominant

A dental practice in a competitive metro area had zero organic visibility. Template WordPress site with stock photos, no blog, same meta description on every page.

What we found:

Zero schema markup (no LocalBusiness, no reviews, no services)
Meta title on every page: "Dental Practice | City Name"
No Google Business Profile optimization
Website not mobile-friendly (text too small, buttons too close)
Zero backlinks — literally zero referring domains
Page load time: 8.3 seconds on mobile

What we fixed over 3 months:

Added LocalBusiness schema with full NAP data
Optimized meta titles per service: "Emergency Dentist in [City] - Same-Day Appointments | Practice Name"
Claimed and optimized Google Business Profile with photos, services, regular posts
Redesigned mobile experience (larger text, bigger tap targets, simpler navigation)
Built 15 local citations (Yelp, Healthgrades, Zocdoc, local chamber of commerce)
Created service pages for each treatment with detailed descriptions
Optimized page speed: WebP images, faster hosting, caching

Results after 6 months:

Ranking top 3 for "dentist [city]" (was not in top 100)
Google Business Profile in local pack for 12 search queries
Organic traffic: 50 to 1,200 monthly visits
Phone calls from organic: 3 to 45 per month
Revenue from organic: approximately $35,000 per month

Total cost: approximately $5,000. ROI was astronomical.

Case Study 3: E-Commerce Fixing Technical Debt

A 500-product Shopify store running 3 years without SEO maintenance. Traffic slowly declining for 18 months.

What we found:

120 product pages with no meta descriptions
45 products with duplicate titles (variants sharing titles)
Category pages with 150+ products but no pagination signals
Product images averaging 3MB each (no compression, no WebP)
No internal linking between related products
Breadcrumb schema missing on all category pages
Blog with 80 posts but zero internal links to product pages

What we fixed over 6 weeks:

Wrote unique meta descriptions for all 120 product pages
Rewrote 45 duplicate titles with variant-specific keywords
Added BreadcrumbList schema to all category and product pages
Compressed all product images to WebP (average 300KB instead of 3MB)
Created "Related Products" sections on product pages
Added "Featured Products" to blog posts
Implemented Product schema on all product pages with price and availability

Results after 4 months:

Organic traffic increased 67%
23 products appeared in Google Shopping results (free listings)
Average page load dropped from 5.1s to 1.8s
Bounce rate decreased 23%
Revenue from organic increased 89%

Total cost: approximately $8,000. Revenue increase paid for it in the first month.

How AI Is Changing SEO Audits

The reality is — in 2026, AI tools can analyze your entire site in seconds. I used to spend 3-4 hours per audit manually checking each element. Now tools like AuditMe scan all 12 points simultaneously and give fix recommendations in 60 seconds.

Does that mean SEO professionals are obsolete? Absolutely not. AI can find problems, but figuring out strategy behind the fixes still requires human judgment. When AI flags that your Core Web Vitals are failing, you still need to decide whether to optimize images, switch hosting, or refactor JavaScript. For a deep dive into fixing those specific metrics, check out our guide on how to fix Core Web Vitals issues.

The real shift? AI lets you focus on strategy instead of tedious data collection. I spend less time gathering data and more time analyzing it. If you want to see what an AI-powered audit looks like in practice, run a free scan on your site — the tool at the top of this page covers all 12 checklist points in under 60 seconds.

Next Steps

You now have the complete step-by-step process for running an SEO audit. Here's what to do next:

1. Run the free scan using the tool at the top of this page — it takes 60 seconds and gives you a prioritized list of issues

2. Review the results against the 12-point checklist above — understand what each issue means and why it matters

3. Fix HIGH priority items first — use the prioritization framework to create your action plan

4. Schedule your next audit for three months from now — quarterly audits catch problems before they compound

The most important step is the first one. Most site owners know their site probably has issues but never carve out the time to investigate. The tool exists to make this painless. Paste your URL and start.

Run Your Free Audit

Ready to see where your site stands? Run a free SEO audit on your website — no signup required. Get results in 60 seconds with actionable recommendations you can implement today. The tool covers meta tags, Core Web Vitals, schema markup, content structure, internal linking, and more — every point in the checklist above.

FAQ

How often should I run an SEO audit?

Run a full audit at least once per quarter. For fast-moving industries or sites that publish frequently, monthly mini-audits focused on technical health and new content keep you ahead of ranking drops. If you've just launched a redesign or migration, run an audit within the first week to catch issues early. See our SEO migration guide for post-launch audit timing.

What tools do I need for an SEO audit?

You can start with free tools like Google Search Console, PageSpeed Insights, and Screaming Frog's free tier. For deeper analysis, paid tools like Ahrefs or Semrush help with competitor research, backlink audits, and keyword tracking. Our free SEO tools comparison breaks down what each tool does best.

How long does a complete SEO audit take?

A thorough audit of a 50-page site typically takes 3-5 hours. Larger sites with hundreds of pages may take a full day. Using automated tools like AuditMe can cut that time to under an hour by scanning all 12 checklist points simultaneously. For enterprise sites with 10,000+ pages, a full audit with log file analysis can take 1-2 weeks.

Can I do an SEO audit myself or do I need an agency?

You can absolutely do it yourself. This guide walks through every step. Where agencies add value is in interpreting findings, prioritizing fixes, and developing long-term strategy — but the technical audit itself is learnable. For WordPress-specific guidance, check our WordPress SEO audit checklist.

What is the most important part of an SEO audit?

Technical health — Core Web Vitals, crawlability, and indexation. If Google can't properly crawl and render your pages, nothing else matters. Fix technical issues first, then move to content and off-page. Our Core Web Vitals checklist walks through the specific thresholds and fixes.

Do SEO audits really improve traffic?

Yes. In my experience, fixing issues found in a typical audit leads to 15-30% increase in organic traffic within 2-3 months. The biggest gains usually come from fixing Core Web Vitals, broken links, and missing or incorrect meta tags. I tracked this across 20 client engagements — the details are in the case studies section above.

What's the difference between a technical SEO audit and a content audit?

A technical audit focuses on how search engines crawl and index your site — things like server response codes, structured data, Core Web Vitals, and site architecture. A content audit evaluates the quality, relevance, and performance of your actual pages — thin content, keyword cannibalization, and outdated information. The best audits cover both. See our content pruning guide for the content side.

How do I track SEO audit improvements over time?

Set up a tracking spreadsheet with baseline metrics before you start. Key metrics to track: organic traffic (Google Analytics), average position and click-through rate (Search Console), Core Web Vitals scores (PageSpeed Insights), indexed page count (Search Console Coverage), and keyword rankings (Ahrefs or Semrush). Check these monthly and compare against your baseline. Our SaaS SEO metrics guide has a complete tracking template.

Should I audit my competitors too?

Absolutely. Competitor analysis reveals gaps in your own strategy. Check what keywords they rank for that you don't, what content they've published that you haven't matched, and what backlinks they have that you're missing. Our competitor SEO analysis framework shows you exactly how to do this systematically.

How do I audit a WordPress site specifically?

WordPress sites have unique SEO considerations — plugin conflicts, theme bloat, WordPress-generated duplicate content, and taxonomy pages that often create thin content. Our WordPress SEO audit checklist covers every WordPress-specific issue, including plugin recommendations and configuration best practices.

What's the difference between an SEO audit and an SEO health check?

An SEO health check is a quick, automated scan of your site's most critical elements — meta tags, Core Web Vitals, basic crawlability. It takes under 60 seconds and catches obvious issues. A full SEO audit is a comprehensive, manual review that includes content analysis, backlink evaluation, competitor comparison, and strategic recommendations. Think of a health check as the triage and the full audit as the complete diagnostic.

How do I audit schema markup on my site?

Use Google's Rich Results Test to check individual pages, or Screaming Frog to crawl your entire site and flag schema errors. Google's structured data troubleshooting guide covers common errors. Common issues include missing required properties, wrong date formats, and orphaned schema that doesn't match page content. Our structured data debugging guide walks through the most common errors and how to fix them.

What are the most common SEO audit mistakes?

According to Ahrefs' research and Backlinko's analysis, the most common audit mistakes are: ignoring mobile usability, skipping log file analysis, fixing everything at once instead of prioritizing, and not setting up baseline measurements before making changes. Our prioritization framework above addresses these mistakes directly.

How do I audit my competitors' SEO?

A competitor audit reveals gaps in your own strategy. Use Ahrefs' Site Explorer or SEMrush's domain analysis to check what keywords they rank for that you don't, what content they've published that you haven't matched, and what backlinks they have that you're missing. Our competitor SEO analysis framework shows you exactly how to do this systematically. Google's documentation on search intent explains what makes content competitive.

How do I audit my Google Business Profile?

For local businesses, your Google Business Profile is as important as your website. Audit your profile for completeness (all fields filled), accuracy (NAP matches your website), photos (at least 25), reviews (respond to all), and posts (regular updates). Our Google Business Profile optimization guide covers the full audit process.

How do I audit my website's security for SEO?

Google's HTTPS documentation explains how HTTPS affects rankings. Use SSL Labs to test your certificate, check for mixed content (HTTP resources on HTTPS pages), verify your HSTS header, and ensure all HTTP URLs redirect to HTTPS with 301 status codes. Security issues can trigger Google Safe Browsing warnings that destroy click-through rates.

Sources and Further Reading

Google Search Central: Auditing Your Site — official documentation for comprehensive technical SEO audits
Ahrefs: Complete SEO Audit Checklist — step-by-step guide to identifying and fixing technical SEO issues
Moz: SEO Audit Guide — structured framework for evaluating site health and prioritizing fixes
Search Engine Journal: Technical SEO Checklist — comprehensive technical SEO walkthrough
Google Search Quality Evaluator Guidelines — the 176-page document defining Google's quality standards
web.dev: Core Web Vitals — official CWV documentation and thresholds
Schema.org — definitive reference for all structured data types
Eric Enge: The Art of SEO — the definitive book on SEO strategy, 4th edition
Marie Haynes: Google Quality Updates — expert analysis of Google's quality algorithm changes
Cyrus Shepard: SEO Testing — data-driven SEO experiments and findings
Portent: SEO Research — original research on ranking factors and algorithm behavior
Bloomberg: Google AI Overviews Rollout — reporting on AI Overviews challenges and improvements

Building a RAG Crawler with Supabase and Gemini for SEO Audits

Eduard — Fri, 21 Aug 2026 13:56:01 +0000

Building a RAG Crawler with Supabase and Gemini for SEO Audits

Most RAG examples start with PDFs, documentation, or a collection of text files.

Websites are a different problem.

A website is a connected system of pages containing HTML, metadata, headings, links, structured data, canonical URLs, and content that changes over time. If an AI SEO auditor only receives the HTML of the page being analyzed, it has very limited context.

While building AuditMe, I needed a crawler and retrieval layer that could turn a website into structured, searchable context for an AI audit.

The resulting pipeline is:

URL
  ↓
HTTP Fetch
  ↓
Cheerio Parse
  ↓
Chunking
  ↓
Gemini Embeddings
  ↓
Supabase pgvector
  ↓
Cosine Retrieval
  ↓
Grounded LLM Prompt
  ↓
Structured JSON Audit

The interesting part is not just the RAG component.

The crawler also has to deal with SSRF, redirects, robots.txt, throttling, JavaScript-rendered applications, duplicate URLs, incremental refreshes, and the difference between deterministic SEO checks and AI-generated analysis.

This article explains how the system works.

Why an SEO auditor needs RAG

A conventional SEO crawler can detect many problems deterministically:

missing title;
missing meta description;
missing canonical;
broken links;
missing alt text;
HTTP status problems;
redirect chains;
missing structured data;
heading structure;
hreflang configuration.

Those checks do not need an LLM.

But other questions are semantic:

Does the page adequately cover its topic?
Are several pages targeting the same intent?
Is the content too thin or repetitive?
Which sections are relevant to a particular recommendation?
Does the page's content support the conclusions generated by the AI?

This is where retrieval becomes useful.

Instead of giving the LLM an arbitrary amount of HTML, the system retrieves the most relevant chunks from the page and injects them into the audit prompt.

The goal is simple:

Give the model the evidence it needs, rather than asking it to invent context.

1. The architecture

The current pipeline is deliberately straightforward:

                 +----------------+
                 |      URL       |
                 +-------+--------+
                         |
                         v
                 +----------------+
                 |   HTTP Fetch   |
                 +-------+--------+
                         |
                         v
                 +----------------+
                 | Cheerio Parse  |
                 +-------+--------+
                         |
                         v
                 +----------------+
                 | Chunking       |
                 | 1400 / 150     |
                 +-------+--------+
                         |
                         v
                 +----------------+
                 | Gemini         |
                 | Embeddings     |
                 | 768 dimensions |
                 +-------+--------+
                         |
                         v
                 +----------------+
                 | Supabase       |
                 | pgvector       |
                 | HNSW           |
                 +-------+--------+
                         |
                         v
                 +----------------+
                 | Top-K Retrieval|
                 | in Node        |
                 +-------+--------+
                         |
                         v
                 +----------------+
                 | Grounded LLM   |
                 | Prompt         |
                 +-------+--------+
                         |
                         v
                 +----------------+
                 | Zod Validation |
                 | Structured JSON |
                 +----------------+

There is an important implementation detail here:

Supabase stores the embeddings and provides an HNSW index, but the current audit retrieval path performs brute-force cosine similarity in Node for the page's own chunks.

That may sound redundant.

It is intentional.

The database layer is ready for vector retrieval, while the current audit path can cheaply retrieve from the relatively small number of chunks belonging to one page.

2. Crawling the page

The first stage is an HTTP fetch.

The fetched HTML is parsed with Cheerio.

The crawler extracts substantially more than visible text:

title
meta description
canonical
Open Graph tags
Twitter cards
H1-H3
images
internal links
external links
nofollow links
word count
JSON-LD types
hreflang
redirect chains

This produces two different kinds of information.

Structured information

Examples:

title
canonical
status code
links
JSON-LD
hreflang
redirects

These are stored and analyzed as structured data.

Unstructured information

Examples:

paragraphs
headings
article content
product descriptions
documentation

This content is suitable for chunking and embeddings.

That distinction is important.

I do not want to turn every SEO fact into a vector.

If the question is:

Does this page have a canonical URL?

the crawler already knows the answer.

There is no reason to ask an embedding model.

3. SSRF protection

A crawler accepts URLs from users.

That makes SSRF protection a first-class requirement.

The crawler uses:

isSafeURL()
resolveAndValidateIP()

and validates every redirect hop.

This matters because checking only the initial URL is insufficient.

Consider:

https://example.com
        ↓
302 redirect
        ↓
http://internal-host

If the crawler validates only the first URL, the redirect can bypass the protection.

The crawler therefore resolves and validates the destination at every redirect hop.

The general rule is:

Never trust a redirect simply because the original URL was considered safe.

A production crawler also needs to consider private, loopback, link-local, and other non-public address ranges.

4. robots.txt

The crawler respects robots.txt.

The implementation caches robots.txt for one hour per domain.

It also resolves rules using the longest-pattern-match approach rather than treating the first matching rule as authoritative.

Conceptually:

Request URL
     |
     v
robots.txt cache
     |
     v
matching rules
     |
     v
longest matching pattern
     |
     v
Allow / Disallow

The crawler also honors a site-declared Crawl-delay when present.

The default throttle is 100 ms.

This is deliberately conservative.

A crawler should not turn a small SEO audit into a traffic spike against the target server.

5. BFS crawling

The scheduler uses breadth-first search.

The current limits are:

concurrency: 5
maxPages:    200
maxDepth:    5

A simplified crawl looks like:

Depth 0
  |
  +-- homepage
       |
       +-- Depth 1
             |
             +-- page A
             +-- page B
             +-- page C
                    |
                    +-- Depth 2

BFS is useful for SEO crawling because it tends to discover pages close to the site's entry points before going deeper.

The scheduler also needs URL deduplication.

Without it, the same page can be reached through multiple internal links and repeatedly scheduled.

6. Handling SPAs

Traditional HTTP fetching is not enough for every modern website.

A single-page application may initially return:

<div id="root"></div>
<script src="/assets/app.js"></script>

while the actual content is produced by JavaScript.

The crawler detects common SPA signals such as:

#root
#app
JavaScript bundles

and can optionally fall back to Playwright rendering.

The important principle is not to render every page in a full browser.

Browser rendering is more expensive.

The preferred strategy is:

HTTP fetch
   |
   +--> sufficient HTML?
   |        |
   |       yes
   |        |
   |        v
   |      parse
   |
   +--> likely SPA?
            |
            v
       optional render

Use the expensive path only when necessary.

7. Content extraction

After Cheerio parsing, the page content is normalized.

The crawler caps content at 10,000 characters before the RAG pipeline.

This is important for two reasons.

First, very large pages can dominate embedding and prompt costs.

Second, most SEO audit questions do not require every byte of a page.

The extraction stage tries to preserve meaningful text while avoiding common HTML noise.

The result is roughly:

title
meta description
h1
body content

plus the structured page metadata collected separately.

8. Chunking strategy

The chunker uses:

MAX_CHARS = 1400
OVERLAP   = 150
MAX_CHUNKS = 10

It uses a sliding window, but prefers a sentence boundary when possible.

The boundary preference is based on:

". "

rather than blindly cutting at character 1400.

This is a small implementation detail with a large practical effect.

Consider:

...Google recommends descriptive page titles because
they help users understand the result...

A hard character boundary can split a sentence.

A sentence-aware boundary is usually a better semantic unit.

Synthetic chunk 0

The first chunk is special.

Chunk 0 contains a synthetic header:

TITLE: ...
| meta_description
| h1

This gives the embedding representation immediate access to the page's most important SEO metadata.

For example:

TITLE: Technical SEO Guide
| Learn how to audit technical SEO
| Technical SEO Checklist

This can improve retrieval for queries related to the page's main topic.

Practical chunk ceiling

The body is capped at 10,000 characters.

With a maximum of 10 chunks and the current chunk size, the practical number of chunks is usually lower.

In practice, the current implementation generally produces around eight real body chunks for a page that reaches the content cap, with chunk 0 reserved for the synthetic header.

The point is to keep the retrieval corpus small and predictable.

9. Gemini embeddings

The embedding model is:

gemini-embedding-001

The stored vectors use 768 dimensions.

The model supports Matryoshka-style dimensionality reduction, allowing the embedding representation to be truncated to the required dimensionality.

The system batches up to:

96 texts per request

Embedding requests can fail transiently, particularly under rate limits.

The current retry strategy is:

maximum attempts: 3
backoff: attempt * 2000 ms

So the retry delays are approximately:

2000 ms
4000 ms

depending on which attempt is being retried.

Authentication is performed with the:

x-goog-api-key

header.

The application exposes the Gemini key through:

GOOGLE_GEMINI_API_KEY

10. Supabase and pgvector

The embeddings are stored in Supabase PostgreSQL using pgvector.

The core table is:

create extension if not exists vector;

create table page_embeddings (
  id bigint generated by default as identity primary key,
  url text not null,
  chunk_index integer not null,
  chunk_text text not null,
  embedding vector(768),
  source text not null default 'gemini-embedding-001',
  meta jsonb not null default '{}'::jsonb,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (url, chunk_index)
);

One correction is worth calling out explicitly.

If you copy an older version of this schema, you may see:

source text not null default 'gemini:text-embedding-004'

That is stale metadata from an earlier embedding setup.

The current embedding model for this implementation is:

gemini-embedding-001

The database schema should describe the model actually producing the vectors.

11. HNSW index

The table uses an HNSW index for cosine similarity:

create index page_embeddings_hnsw_idx
  on page_embeddings using hnsw (embedding vector_cosine_ops);

HNSW is useful when vector collections become large enough that scanning every vector for every query becomes expensive.

The important distinction in this implementation is that the HNSW index exists at the database layer, but the current audit path does not depend on it for its normal page-local retrieval.

Instead, the current flow is:

page URL
   |
   v
retrieve its chunks
   |
   v
calculate cosine similarity in Node
   |
   v
sort
   |
   v
take top 5

Because a single page is limited to a small number of chunks, brute-force similarity is cheap.

This is a good example of not optimizing for a problem you do not currently have.

12. Supabase RPC

The database also exposes a vector-search RPC:

create or replace function search_page_chunks(
  p_url text,
  p_embedding vector,
  p_limit integer default 5
)
returns table(
  url text,
  chunk_index integer,
  chunk_text text,
  similarity double precision
)
language sql
stable
as $$
  select
    e.url,
    e.chunk_index,
    e.chunk_text,
    1 - (e.embedding <=> p_embedding) as similarity
  from page_embeddings e
  where e.url = p_url
  order by e.embedding <=> p_embedding
  limit p_limit;
$$;

The <=> operator is pgvector's cosine distance operator.

Therefore:

similarity = 1 - cosine_distance

The RPC makes it possible to move retrieval into PostgreSQL when that becomes preferable.

13. The actual RAG flow

The audit pipeline is deliberately narrow.

For each page:

1. Fetch page
2. Parse with Cheerio
3. Cap content at 10K characters
4. Chunk content
5. Generate embeddings
6. Store chunks
7. Create query from page metadata
8. Retrieve top chunks
9. Inject chunks into LLM prompt
10. Generate structured audit
11. Validate JSON with Zod

The indexing call is conceptually:

indexPageChunks(
  url,
  {
    title,
    meta_description,
    h1,
    content_text
  }
)

The page's own metadata becomes the retrieval query:

[title, meta_description, h1].join(". ")

This is an important design choice.

The query is not an arbitrary user question.

It represents the page's own declared topic.

14. Top-K retrieval

The current retrieval function:

topKChunks()

performs cosine similarity in Node.

It returns:

top 5 chunks

with a minimum similarity threshold of:

0.15

So the practical rule is:

similarity > 0.15

and then take the five highest-scoring chunks.

The threshold should not be interpreted as a universal semantic quality boundary.

Embedding scores are model- and dataset-dependent.

A threshold that works for one corpus may be terrible for another.

The correct approach is to evaluate retrieval against representative queries and tune it empirically.

15. Grounding the LLM

The retrieved chunks are injected into the prompt under:

RETRIEVED PAGE CONTENT

The model therefore receives evidence retrieved from the actual page.

Conceptually:

SYSTEM / AUDIT INSTRUCTIONS

PAGE METADATA

TECHNICAL SEO DATA

RETRIEVED PAGE CONTENT
----------------------

[chunk 1]

[chunk 2]

[chunk 3]

[chunk 4]

[chunk 5]

Return structured JSON.

This does not magically eliminate hallucinations.

But it changes the model's job.

Instead of generating an SEO assessment from an isolated prompt, it can reason over the page's actual retrieved content.

16. Structured JSON and Zod

The LLM output is not accepted blindly.

The generated audit is validated with Zod.

The conceptual flow is:

LLM
 |
 v
JSON
 |
 v
Zod schema
 |
 +--> valid --> audit result
 |
 +--> invalid --> fallback/error handling

This matters because an LLM can produce syntactically invalid JSON, missing fields, unexpected values, or the wrong data types.

A schema turns the LLM from an untrusted output generator into a component with a defined contract.

17. Deterministic fallback

The AI layer is not the only source of insights.

If the LLM fails, the audit pipeline can fall back to:

buildDeterministicInsights()

This rule engine handles deterministic findings without requiring an AI response.

That gives the system an important property:

An AI outage should not make the entire SEO audit useless.

For example, if the crawler already knows that a page has no canonical, that fact does not disappear because an LLM request failed.

18. SEO checks that should remain deterministic

Some checks are simply better implemented as code.

For example:

title exists?
meta description exists?
canonical exists?
H1 exists?
HTTP status is 200?
internal link is broken?
JSON-LD exists?
hreflang is valid?

An LLM is unnecessary here.

The strongest architecture is therefore:

Deterministic crawler/checks
          +
RAG retrieval
          +
LLM reasoning

not:

Everything → LLM

This reduces cost and makes the technical checks reproducible.

19. Content quality signals

The crawler also calculates content-quality signals such as:

Flesch reading ease
keyword stuffing
complex-word ratio
word count

The keyword-stuffing detector currently flags keyword density above:

3%

These metrics should be treated as signals, not absolute truth.

For example, a technical article can legitimately repeat a domain-specific term many times.

A simple density threshold cannot understand that context.

The useful approach is to use these metrics as evidence alongside the actual content.

20. Cannibalization detection

One of the more interesting uses of embeddings is content cannibalization.

The system can compare page embeddings using pairwise cosine similarity.

The current threshold is:

0.85

Conceptually:

Page A ────────┐
               │ cosine similarity
Page B ────────┘
       |
       v
 similarity >= 0.85
       |
       v
 potential semantic overlap

The current comparison is O(n²).

That is fine for a relatively small set of pages.

It becomes expensive as the number of pages grows.

For example:

100 pages  -> 4,950 pairs
1,000 pages -> 499,500 pairs
10,000 pages -> 49,995,000 pairs

So pairwise comparison is useful as an initial implementation, but a large-scale crawler would eventually need a more efficient nearest-neighbor approach.

And again, similarity does not prove cannibalization.

It identifies candidates for further analysis.

21. Incremental refresh

A crawler that only works when a user manually starts an audit is not enough for monitoring.

AuditMe also has an incremental refresh process.

The refresh job checks the:

50 most recent scans

using HTTP validators:

If-None-Match
If-Modified-Since

If the page has not changed, there is no reason to download and re-embed it.

If it has changed:

re-scan
   ↓
re-extract
   ↓
re-chunk
   ↓
re-embed
   ↓
update audit data
   ↓
update score history

This is significantly cheaper than reprocessing every page every time.

22. Crawl job monitoring

Long-running jobs need a failure mechanism.

The crawler marks stale jobs as failed after:

10 minutes

Sites that have not been crawled for more than:

7 days

can be scheduled for another crawl.

This creates a basic monitoring loop:

crawl
 |
 v
fresh data
 |
 v
7 days
 |
 v
refresh

The exact interval can be changed depending on how frequently the target sites change.

23. Database model beyond embeddings

The RAG table is only one part of the application.

The other important tables are:

`scans`

Used for a 24-hour cache.

It includes:

url
data JSONB
etag
last_modified

The URL is the primary key.

`audit_results`

Stores permanent audits.

Each audit has a UUID and includes:

rag_debug

This is particularly useful when debugging retrieval quality.

If an AI recommendation looks wrong, you want to know what context the model actually received.

`score_history`

Stores score changes over time.

This makes trend analysis possible instead of showing only the current score.

`crawl_jobs`

Tracks crawler execution and status.

Old jobs are cleaned up using a seven-day TTL policy.

24. Why `rag_debug` matters

RAG systems are notoriously difficult to debug if retrieval is invisible.

Suppose the LLM makes a bad recommendation.

There are several possible causes:

crawler extracted the wrong content
        ↓
chunking lost context
        ↓
embedding was poor
        ↓
retrieval selected wrong chunks
        ↓
prompt was unclear
        ↓
LLM reasoned incorrectly

Without retrieval diagnostics, these problems all look like:

"The AI gave a bad answer."

That is not actionable.

Keeping retrieval information in rag_debug makes the pipeline observable.

You can inspect:

query
retrieved chunks
similarity scores
source URL
chunk indexes

and determine where the failure occurred.

25. Environment configuration

The main environment variables are:

GOOGLE_GEMINI_API_KEY
SUPABASE_SERVICE_ROLE_KEY
PAGESPEED_API_KEY
CRUX_API_KEY
CRON_SECRET

Their responsibilities are separated:

GOOGLE_GEMINI_API_KEY
    -> Gemini embeddings + LLM

SUPABASE_SERVICE_ROLE_KEY
    -> server-side database administration

PAGESPEED_API_KEY
    -> PageSpeed enrichment

CRUX_API_KEY
    -> Chrome UX / Core Web Vitals enrichment

CRON_SECRET
    -> protected scheduled jobs

The service-role key should never be exposed to browser-side code.

26. What I learned from building it

The main lesson is that RAG is only one part of the system.

A useful web intelligence pipeline needs several layers.

Layer 1: Fetching

Get the actual page safely.

HTTP
redirects
robots
throttling
SSRF protection

Layer 2: Parsing

Turn HTML into useful information.

metadata
content
links
schema
headings

Layer 3: Structured analysis

Run deterministic checks.

status
canonical
titles
links
schema
hreflang

Layer 4: Retrieval

Find semantically relevant content.

chunk
embed
search
rank

Layer 5: Reasoning

Let the LLM interpret the evidence.

retrieved content
+
technical facts
+
audit instructions

Layer 6: Validation

Never trust generated output blindly.

LLM
 ↓
Zod
 ↓
structured audit

That separation makes the system much easier to reason about.

27. What I would improve next

There are several obvious directions for future iterations.

Better retrieval

The current page-local brute-force retrieval is simple and fast, but a larger corpus could use database-side HNSW retrieval or a hybrid search strategy.

Better chunking

The current 1400-character sliding window works, but heading-aware and semantic chunking could preserve context more effectively.

Better reranking

Top-K vector similarity is not necessarily the final ranking.

A reranker could improve retrieval precision.

Better site-wide RAG

The current audit query is primarily page-focused.

A broader retrieval layer could answer questions across the entire site.

Faster cannibalization detection

O(n²) pairwise comparison will eventually become a bottleneck.

Nearest-neighbor search can reduce the number of comparisons.

More browser rendering

Some JavaScript-heavy sites require full rendering to accurately inspect their content.

That should remain an optional expensive path rather than the default for every URL.

28. The bigger picture

The crawler is not just an ingestion script.

Once the site is represented as:

pages
+
links
+
metadata
+
content
+
embeddings
+
crawl history

it becomes possible to build much more than a basic SEO checker.

For example:

semantic content clusters
        ↓
internal linking recommendations

similar pages
        ↓
cannibalization candidates

content changes
        ↓
historical SEO monitoring

retrieved page content
        ↓
grounded AI recommendations

This is the direction I am taking with AuditMe.

The objective is not to replace deterministic SEO tooling with an LLM.

It is to combine deterministic analysis with semantic retrieval and AI reasoning.

29. Final architecture

The complete system can be summarized as:

                         WEBSITE
                            |
                            v
                    +---------------+
                    | Safe Crawler  |
                    +-------+-------+
                            |
             +--------------+--------------+
             |              |              |
             v              v              v
          Metadata        Links         Content
             |              |              |
             +--------------+--------------+
                            |
                            v
                     Chunking 1400/150
                            |
                            v
                  Gemini embedding-001
                            |
                            v
                    768-dim vectors
                            |
                            v
                +-----------------------+
                | Supabase PostgreSQL   |
                | pgvector + HNSW       |
                +-----------+-----------+
                            |
                            v
                 Page-local Top-K
                 cosine retrieval
                            |
                            v
                RETRIEVED PAGE CONTENT
                            |
                            v
                       Grounded LLM
                            |
                            v
                     Zod validation
                            |
             +--------------+--------------+
             |                             |
             v                             v
       Structured Audit          Deterministic Fallback
             |
             v
                    AuditMe Results

The key idea is simple:

The crawler collects evidence. PostgreSQL stores facts. pgvector stores semantic representations. Retrieval selects context. The LLM reasons over that context.

That division of responsibilities is what makes the system practical.

Conclusion

Building RAG for websites is fundamentally different from building RAG over static documents.

The difficult part is not generating an embedding.

It is building a reliable representation of the website before the embedding is ever created.

You need to handle:

safe URL fetching;
redirects;
robots.txt;
crawl limits;
SPA rendering;
HTML extraction;
content normalization;
chunking;
embeddings;
vector storage;
retrieval;
structured SEO checks;
LLM validation;
incremental refresh;
historical data.

Once those pieces are separated, the architecture becomes much easier to extend.

And that is the real value of combining a crawler with RAG:

You are not simply giving an AI access to a webpage. You are giving it a structured, retrievable representation of the website it is supposed to analyze.

You can see the resulting SEO auditing product at AuditMe and try the Website SEO Checker.

The code examples in this article are simplified versions of the implementation. Production crawlers should additionally account for rate limiting, resource limits, security hardening, rendering costs, database lifecycle management, and workload isolation.

How to Actually Raise Your Google PageSpeed Score in 2026

Eduard — Wed, 05 Aug 2026 01:45:27 +0000

A few weeks ago I finished a three-hour deep dive into font loading for a client. Subsetting, preloading, font-display: swap — the works. PageSpeed came back at 92.
The client asked the only question that mattered: “Did we fix the real problem?”

The hero image was still 4.2 MB. One conversion to AVIF and a proper <picture> tag later the score dropped to 58. Suddenly Lighthouse could see everything else that was broken.

That’s the thing about PageSpeed scores. They show you what the lab thinks is slow. Real users often experience something completely different.

What PageSpeed Insights Really Measures
Real Case Study: 42 → 91 in One Afternoon
The Five Things That Move the Needle Most
When the Score Lies
My Real Workflow (Not a Checklist)
The WebP (and AVIF) Trap + Code Example
Fonts That Don’t Block the Page
Why INP Matters More Than the Old FID
The Real 80/20
When to Stop
Quick Checks Before the Next Test
Bottom Line
Related Guides
Sources & Further Reading

What PageSpeed Insights Really Measures

Google PageSpeed Insights runs Lighthouse in a controlled lab. You get a 0–100 number. That number itself is not a ranking factor. Core Web Vitals are. And the two are related, but not identical.

Here’s the simple rule I give clients:

Below 50 → real problems that users feel
50–89 → fix the obvious stuff
90+ → open Chrome DevTools and look at actual load time before you celebrate

For a deeper look at the ranking metrics that actually matter, see our guide on how to fix Core Web Vitals issues.

Real Case Study: 42 → 91 in One Afternoon

Last month a mid-size e-commerce client came in with a mobile PageSpeed score of 42. Field data from CrUX showed LCP at 4.1 s and INP at 380 ms. Users were bouncing hard on product pages.
We did two things only:

Converted the 3.8 MB hero JPEG to AVIF (primary) + WebP (fallback) and served it with a proper <picture> element.
Removed or deferred 1.2 MB of third-party trackers and chat widgets that were blocking the main thread.

Results after one afternoon of work:

Metric	Before	After
Mobile PageSpeed	42	91
LCP	4.1 s	1.3 s
INP	380 ms	95 ms
Total page weight	6.4 MB	1.9 MB

No fancy critical CSS, no server migration, no CDN change. Just images and JavaScript hygiene. That’s why I keep saying the big wins are almost always the boring ones.

Want to see where your own site stands right now? Open the free SEO analyzer in another tab and run your URL while you keep reading.

The Five Things That Move the Needle Most

After looking at more than two hundred sites, the same five issues keep showing up.

Images. Almost always the biggest culprit. I have yet to audit a site under 70 where images weren’t the main drag. Full details in our Image SEO complete guide.

JavaScript execution time. This one hides. You can score 95 and still have three seconds of main-thread blocking. Check the Performance panel, not just the Lighthouse report. Slow JS is also the primary reason INP fails on mobile.

Server response time (TTFB). If the server takes more than 800 ms to answer, nothing else you do will feel fast.
Caching. Set it once and every return visitor benefits.
CDN. If your audience is spread across countries and your origin is in one place, a CDN is no longer optional.

When the Score Lies

Last quarter I checked a Shopify store that scored 98. Actual load time for a real user: 4.2 seconds. Forty-seven tracking scripts, a chat widget, a popup, and three recommendation engines. Lighthouse tested a clean version without most of that noise.
Lab data and field data are different animals. Real visitors have ad blockers, older phones, and flaky connections that no lab simulation fully captures.

My Real Workflow (Not a Checklist)

When someone asks me to improve their score, this is what I actually do.

Open DevTools first. Look at the network waterfall. Total size, number of requests, longest assets. Try it now: open Chrome DevTools on your own site, go to Network, disable cache, reload, and sort by size. If anything over 200 KB is an image or a third-party script, you already know your first target.
Find the biggest offenders. Usually images. Sometimes a 2 MB JavaScript bundle that does almost nothing useful.
Fix images. Convert to modern formats, add srcset, lazy-load everything below the fold. Fifteen minutes of work often gains 10–20 points.
Kill unused JavaScript. Open the Coverage tab. Sixty to eighty percent of the code is frequently never executed. Remove it.
Stop over-optimizing. Chasing the last three points on a 95 score is rarely worth the time. If you want a broader technical check after these steps, run a full SEO audit.

The WebP (and AVIF) Trap + Code Example

Everyone says “just convert to WebP.” I did that once and the score went down. Older Safari versions didn’t support it cleanly, and the fallback JPEG was three times larger.
Now I always use the full <picture> element with multiple sources and a solid fallback. More markup, but it works everywhere.

Here’s the exact pattern I ship:

<picture>
  <source
    srcset="hero-800.avif 800w, hero-1200.avif 1200w, hero-1600.avif 1600w"
    type="image/avif"
    sizes="(max-width: 768px) 100vw, 1200px">
  <source
    srcset="hero-800.webp 800w, hero-1200.webp 1200w, hero-1600.webp 1600w"
    type="image/webp"
    sizes="(max-width: 768px) 100vw, 1200px">
  <img
    src="hero-1200.jpg"
    width="1200"
    height="675"
    alt="Product hero showing the main benefit"
    fetchpriority="high"
    decoding="async">
</picture>

Key points:

AVIF first, WebP second, JPEG last.
Explicit width and height prevent CLS.
fetchpriority="high" on the LCP image.
Never put loading="lazy" on the hero.

For third-party scripts that are not needed on first paint, the same principle applies:

<!-- Bad: blocks parsing and execution -->
<script src="https://example.com/tracker.js"></script>
<!-- Better: does not block parsing -->
<script src="https://example.com/tracker.js" defer></script>
<!-- Best for non-critical widgets: load only after interaction or idle -->
<script>
  window.addEventListener('load', () => {
    const s = document.createElement('script');
    s.src = 'https://example.com/chat-widget.js';
    s.async = true;
    document.body.appendChild(s);
  });
</script>

Fonts That Don’t Block the Page

font-display: swap is not a magic bullet. It just means invisible text for a moment, then a flash. Fine for body copy. Ugly for headlines.
My current approach: system fonts for body text, and for brand fonts — preload only the exact weights you actually use. I still see sites loading twelve font files when they need three.

Why INP Matters More Than the Old FID

FID (First Input Delay) only measured the delay before the very first interaction. Google replaced it with INP (Interaction to Next Paint) in 2024, and by 2026 it is the only interactivity metric that counts for ranking.

INP looks at every interaction throughout the page life — clicks, taps, key presses — and reports the worst ones (98th percentile). That is why a site can have a perfect lab score and still feel laggy on a mid-range Android phone.
The main cause of poor INP is almost always JavaScript that monopolizes the main thread for longer than 50 ms. Long tasks from analytics, chat widgets, A/B testing libraries, or heavy event handlers are the usual suspects.

Practical ways to improve INP:

Break long tasks into chunks under 50 ms (setTimeout, requestIdleCallback, or the newer Scheduler API).
Defer or remove third-party scripts that run on every page load.
Use the Coverage tab and the Performance panel together — Coverage shows unused code, Performance shows the actual long tasks. If your INP is above 200 ms on mobile, fixing JavaScript will usually give a bigger ranking and conversion lift than shaving another 5 points off the PageSpeed score.

The Real 80/20

Three actions deliver most of the gain:

Compress and correctly size every image (Squoosh is still free and excellent).
Remove unused JavaScript with the Coverage tab.
Turn on caching and put a CDN in front of the site. On platforms like Vercel or Netlify this is already handled. Everything else — critical CSS, advanced preloading, Speculation Rules — is polishing. Do the big three first.

When to Stop

I once spent eight hours with a client trying to turn a 99 into a 100. The missing point was a 12 KB analytics script that was already async.
My rule now: if the score is above 90 and real load time is under two seconds, stop. Your time is more valuable than that last point.

Quick Checks Before the Next Test

Run the URL through a proper combined audit (PageSpeed + real traffic signals) at AuditMe.
Look at your largest image. Anything over 200 KB needs attention.
In DevTools → Network, disable cache and reload. If the page is over 2 MB, something is wrong.
Check the JavaScript bundle size. Over 500 KB usually means dead code.
For a focused score check, use the dedicated SEO Score Checker.
Want the full picture in under a minute? The free SEO analyzer pulls it all together. ## Bottom Line PageSpeed scores are useful signals, not the final truth. I have seen perfect 100s that felt sluggish and mid-60s that loaded faster than most sites. Focus on what users actually experience. Under two seconds, green Core Web Vitals (especially LCP under 2.5 s and INP under 200 ms) put you ahead of the majority of the web. That’s the only score that really matters.

Related Guides

Continue with these practical guides from the same series:

Complete SEO Audit Guide 2026: How to Find and Fix Every Issue — full 12-point checklist covering meta tags, schema, content, and more.
How to Fix Core Web Vitals Issues: LCP, INP, CLS Explained — deep dive into the three metrics that actually affect rankings.
Image SEO: Complete Guide to Optimizing Images for Search — formats, responsive images, alt text, and performance impact.

Sources & Further Reading

This article draws on practical audit experience and the following public resources:

Google PageSpeed Insights — official lab and field data tool.
Web Vitals documentation — Google’s definitions and thresholds for LCP, INP, and CLS.
Lighthouse scoring methodology — how the 0–100 score is calculated.
Optimize LCP and Optimize INP — official Chrome performance guides.
Squoosh — free image compression and format conversion tool referenced in the 80/20 section.
Internal site resources: AuditMe, SEO Score Checker, Free SEO Analyzer.

WordPress SEO Audit Checklist — 15 Things Worth Checking in 2026

Eduard — Sun, 02 Aug 2026 21:54:34 +0000

WordPress still powers about 43% of the web. That means your competition is not only other WordPress sites — it’s also headless setups, well-funded content teams, and stores that ship updates every week.
Most ranking drops I see on WordPress installs come from the same three places:

Plugin and theme conflicts that quietly break technical SEO
On-page issues that accumulate over months
Content that became thin or outdated while nobody was looking A focused 30–40 minute audit once a month (or at least once a quarter) catches most of this damage early. Here’s the exact checklist I use.

How to use this checklist
Technical SEO (5–8 minutes)
- 1. Permalink structure
- 2. XML Sitemap
- 3. robots.txt
- 4. Broken links (internal first)
- 5. HTTPS and mixed content
On-Page SEO (10–15 minutes)
- 6. Title tags
- 7. Meta descriptions
- 8. H1 tags
- 9. Heading hierarchy
- 10. Image alt text + performance
- 11. Internal links
- 12. Content length and quality
Schema, Social & Performance (5–10 minutes)
- 13. Structured data (Schema)
- 14. Open Graph and Twitter Cards
- 15. Core Web Vitals
Priority order after the audit
Useful free (or freemium) tools
Final notes
Similar useful articles
Sources & further reading

How to use this checklist

Do it in order. Technical issues first — they affect everything else.
Spend more time on your top 10–20 pages by traffic or revenue.
Write down every problem you find. Don’t try to fix everything in one sitting.
After the audit, prioritize: high impact + low effort first.
Re-check rankings and organic traffic 2–4 weeks later.

You don’t need expensive tools for the first pass. Browser, Search Console, PageSpeed Insights, and one free crawler are enough. If you want to speed things up, you can also run a quick automated check with a free WordPress SEO audit tool that covers many of these points in under a minute.

Technical SEO (5–8 minutes)

1. Permalink structure

Where: Settings → Permalinks
What to look for:
If it’s set to “Plain” (?p=123), change it immediately to Post name (/%postname%/).
Plain permalinks are bad for users and for crawlers. Almost every site that isn’t a massive news portal with very specific URL requirements should use Post name.
After changing:
Check 3–5 older posts. If any 404, set up redirects (Redirection plugin or Rank Math / Yoast redirect features work fine). Most modern SEO plugins handle the bulk of this automatically.
Common mistake: Changing permalinks on a live site without checking redirects. Always test.

2. XML Sitemap

Check: yoursite.com/sitemap.xml or yoursite.com/sitemap_index.xml
Yoast, Rank Math, All in One SEO and most other plugins generate one automatically. If you get a 404, the sitemap is either not generated or blocked.

What to do:

Confirm the sitemap loads and contains your important pages
Submit it in Google Search Console and Bing Webmaster Tools
Watch the Coverage / Pages report for sudden drops in indexed URLs — this often points back to sitemap or robots problems

Pro tip: If you have a large site, check that the sitemap is split into multiple files and that pagination works. A single giant sitemap can cause issues.

3. robots.txt

Check: yoursite.com/robots.txt
Dangerous pattern I still see regularly:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/

Blocking /wp-content/ is almost never correct. CSS, JS, fonts and images live there. Search engines need to fetch them to understand how the page renders.
Safer minimal version:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Anything more aggressive (blocking specific plugins, query parameters, etc.) should be intentional and documented. When in doubt, be less restrictive.
Also check that important pages or directories are not accidentally blocked.

4. Broken links (internal first)

Broken internal links waste crawl budget and create dead ends for users. External 404s are less critical but still look unprofessional.
Quick approach:

Use a free crawler (Screaming Frog free version, Sitebulb trial, or online tools)
Or use the link checker in your SEO plugin / audit tool
Focus first on internal links from high-traffic pages and from the homepage / main menus Fix strategy:
Update or remove the link
Or set a 301 redirect if the target page moved For large sites, schedule a deeper crawl monthly. Don’t try to fix every external 404 on the first pass.

5. HTTPS and mixed content

The entire site should load over HTTPS. Mixed content (HTTP images, scripts or stylesheets on HTTPS pages) still triggers browser warnings and can affect trust signals.
How to check:

Chrome DevTools → Security tab
Or look in the Console for mixed-content warnings
PageSpeed Insights and most audit tools also flag this Fix: Update the source URLs to HTTPS or force HTTPS at the server / CDN / plugin level (Really Simple SSL, Cloudflare, or host-level settings). --- ## On-Page SEO (10–15 minutes)

6. Title tags

Every important page needs a unique title. Aim for under ~60 characters so it doesn’t get truncated in search results.
What to check on your top pages:

Is the title unique?
Does it contain the primary keyword near the beginning (when it still reads naturally)?
Is it written for humans or just stuffed with keywords? Generic titles like “Home”, “Services” or “Blog” waste valuable space. Tool tip: You can bulk-export titles with most SEO plugins or use a free browser extension / Screaming Frog.

7. Meta descriptions

Same rules as titles: unique, roughly under 155–160 characters, written for humans first.
Meta descriptions do not directly influence rankings, but they heavily influence click-through rate. A page that ranks well but has a weak or missing description is leaving clicks (and therefore rankings) on the table.
Write them like a short ad: clear benefit + keyword + reason to click.

8. H1 tags

One H1 per page. It should describe the main topic and preferably contain the primary keyword.
Multiple H1s still appear frequently with page builders (Elementor, Divi, Bricks, etc.) and poorly coded themes.
How to check: Inspect the page source or use the browser inspector. Search for <h1. If you see more than one, fix the template or the specific page.

9. Heading hierarchy

Correct order: H1 → H2 → H3 → H4.
Don’t skip levels just for visual design. Screen readers and search engines both prefer a logical outline.
Page builders make it very easy to create a visual hierarchy that doesn’t match the actual HTML heading tags. Always verify the real markup.

10. Image alt text + performance

Every meaningful image needs descriptive alt text.
IMG_20260725.jpg or empty alt attributes tell Google (and screen readers) nothing useful.
“WordPress SEO audit checklist screenshot” is useful.
Decorative images should have empty alt (alt="") so assistive technology skips them.
While you’re there:

Compress images
Serve modern formats (WebP / AVIF) where possible
Add proper width/height attributes to reduce CLS Performance and accessibility often improve together.

11. Internal links

Aim for 3–5 relevant contextual internal links on most content pages. Link to related posts, category hubs, pillar pages or key service pages.
This helps distribute authority and makes the site structure clearer to crawlers.
Avoid:

Footer or sidebar link spam
Linking to the same page over and over
Using exact-match anchor text on every link Contextual links inside the content work best.

12. Content length and quality

Thin pages (under ~300 words of real content) rarely rank for competitive terms. Pillar pages, money pages and important guides usually need 1,000+ words of genuinely useful material.
This does not mean padding. It means:

Answering the main query thoroughly
Covering related questions people also ask
Updating outdated sections
Removing fluff that adds no value Practical check: Look at your top landing pages by traffic. If organic traffic has slowly declined while the SERP became more competitive, content quality and freshness are often the first places to look. Also check for near-duplicate pages (especially category/tag pages and pagination) that may be competing with each other.

Schema, Social & Performance (5–10 minutes)

13. Structured data (Schema)

Run several key pages through Google’s Rich Results Test.

Look for valid JSON-LD. The most common high-value types on WordPress sites are:

Article / BlogPosting
FAQ
HowTo
Product
Organization
BreadcrumbList
WebSite (with sitelinks search box)

Most SEO plugins can output basic schema. Custom post types, complex templates or page-builder pages often need manual JSON-LD or a more flexible solution.
Invalid or missing schema means you are leaving rich results (and higher CTR) on the table.

14. Open Graph and Twitter Cards

Share 2–3 important pages on LinkedIn, Facebook or X and look at the preview.
Check:

Correct title
Correct description
Correct image (and that it looks good, not cropped or low-res) Missing or wrong OG tags produce ugly social shares and lower click-through from social traffic. Yoast and Rank Math handle the basics well. Double-check featured image dimensions if previews look bad.

15. Core Web Vitals

Test the homepage and 2–3 important templates (single post, archive, product page, etc.) in PageSpeed Insights.
Targets (mobile):

LCP < 2.5 s
CLS < 0.1
INP < 200 ms Most common WordPress culprits:
Heavy page builders
Unoptimized hero images
Render-blocking plugins and scripts
Too many marketing / tracking scripts
Lack of proper caching and CDN High-impact fixes:
Good caching plugin (or host-level caching)
CDN
Image optimization + modern formats
Delay or conditionally load non-critical JavaScript
Reduce the number of active plugins Deeper issues sometimes require changing the theme or how the page builder is used.

Priority order after the audit

Don’t try to fix everything at once. Suggested order:

Critical technical — Permalinks, robots.txt, HTTPS, major broken internal links, sitemap not working
High-visibility on-page — Titles and meta descriptions of top pages, missing H1s, obvious thin content on money pages
Performance — Core Web Vitals on homepage and key templates
Schema & social — Structured data on important pages, Open Graph
Ongoing content & internal linking — Systematic improvements over the following weeks Track the pages you changed. Give Google 2–4 weeks and then review rankings + organic traffic.

Useful free (or freemium) tools

Task	Tool	Notes
Crawl & broken links	Screaming Frog (free)	500 URLs limit
Core Web Vitals	PageSpeed Insights	Official
Rich results	Google Rich Results Test	Official
Search performance	Google Search Console	Essential
Bulk title/meta check	SEO plugin + export or SF	Rank Math / Yoast work well
robots.txt & sitemap	Browser + Search Console	Simple
Quick site audit	AuditMe	Free automated checks + WordPress-specific fixes

You can do a solid first-pass audit with almost no paid tools. For a faster overview of titles, meta tags and overall score, the free SEO analyzer is also handy.

Final notes

Most WordPress sites I look at have at least 4–7 clear, fixable issues sitting in the technical and on-page sections. The difference between “we should do an SEO audit someday” and actually finishing one is usually just having a concrete checklist and 40 focused minutes.
Do the technical part first. Then the top pages. Write everything down. Fix the high-impact items. Repeat in a few weeks or next month.
That’s it. Go run it on your site.
If you want, reply with the biggest issue you found — I’m curious what still surprises people in 2026.
You can also run a full automated check anytime at AuditMe.

Sources & further reading

This checklist is based on practical experience plus official documentation and established SEO references:

Official / primary sources

Google Search Central – Core Web Vitals
Google Rich Results Test
Google Search Console documentation
PageSpeed Insights Books
The Art of SEO (Eric Enge, Stephan Spencer, Jessie Stricchiola) — still one of the most solid foundational texts.
SEO for Growth by John Jantsch & Phil Singleton — practical and business-oriented.
The Beginner’s Guide to SEO (Moz) — free and regularly updated online version is excellent for fundamentals. Selected articles & guides
Search Engine Land – Core Web Vitals guides and updates
Semrush – On-page SEO checklist (regularly refreshed)
Ahrefs & Moz blog posts on technical SEO, internal linking and content audits

The core principles (crawlability, clear structure, useful content, good user experience metrics) have been stable for years. The tools and exact thresholds (especially Core Web Vitals and INP) continue to evolve, which is why re-checking every few months still makes sense.

Content Optimization for Search Engines: Beyond Keywords

Eduard — Sat, 01 Aug 2026 11:02:31 +0000

Content Optimization for Search Engines: Beyond Keywords

After auditing a SaaS blog last month I saw the same pattern I’ve been seeing for years: 200 well-written posts, almost none of them ranking. The writer came from journalism, so the prose was clean and readable. The problem was structural. Zero keyword targeting, no topical clusters, almost no internal linking. Pretty words with zero organic traffic.
This is the gap I keep running into. People either write for robots (keyword-stuffed nonsense nobody finishes) or they write purely for humans and act like search engines don’t exist. The real work sits in the middle — and it’s not as complicated as most SEO guides make it sound.

Keywords Aren’t Dead. They’re Just Different.
Where the Keywords Actually Belong
Topical Authority Is the Long Game That Still Works
E-E-A-T Isn’t a Checklist
Stop Writing Like a Textbook
AI Content Is the Elephant in the Room
Content Freshness: Update Only When It Matters
Final Thought

Keywords Aren’t Dead. They’re Just Different.

A few years ago you could force “best project management software” into a page eight times and rank. Those days are gone. Google understands context far better now. That doesn’t mean keywords stopped mattering. It means the way you use them changed.
When I start optimizing a piece I still do keyword research, but volume is only one signal. Intent is the bigger one. Someone searching “project management software” wants comparisons and pricing. Someone searching “how to manage a remote project team” wants a practical walkthrough. Same broad topic, completely different content.

I mostly use Ahrefs. SEMrush works fine too. The rule I learned the expensive way: match commercial-intent keywords to product pages and informational-intent keywords to the blog. Trying to rank a blog post for a high-intent transactional term is usually a multi-month waste of time.
Question-based keywords are still underrated. Featured snippets love clear, direct answers. I’ll take a question like “how many internal links should a page have,” answer it in the first two sentences of a section, then expand. That short direct answer often wins the snippet.

Where the Keywords Actually Belong

You don’t need to be surgical, but there is a practical hierarchy I follow:

Primary keyword near the front of the title tag, in the H1, and inside the first 100 words
Secondary keywords in H2s and H3s
Meta description should contain the primary keyword naturally — not forced, just present

Early in my career I cost a client three months of traction with a simple meta description mistake. We were targeting “enterprise SEO platform.” I wrote a clean, persuasive description that somehow never included the exact phrase. Google didn’t bold anything in the SERP. CTR sat at 1.2%. I added the keyword, kept the same position and the same content, and CTR moved to 3.8% in two weeks.

One thing I’ll push back on: obsessing over “LSI keywords.” Google does use related terms and entities, but most tools turn that idea into a checklist you sprinkle in. Don’t. Write thoroughly about the topic and the related concepts appear naturally. If you’re covering coffee makers you’ll mention grind size, water temperature, filter types and brew methods without forcing a list.

Topical Authority Is the Long Game That Still Works

Most people quit too early here. Topical authority is Google deciding you actually know a subject, not just ranking one lucky page. It takes consistent coverage of related subtopics over months.
I learned this with a project-management client. We had a strong pillar ranking around position 6 for “project management.” Traffic stalled. Looking at Asana and Monday.com showed the gap immediately — they had dozens of pages covering every angle: agile vs waterfall, remote team management, timeline estimation, tool comparisons. We were competing with one page against an entire ecosystem.

So we built a proper cluster: one 4,000-word pillar (“Complete Guide to Project Management”) plus four focused pieces on the subtopics. Descriptive anchor text both ways. Four months later the pillar moved from 6 to 2. The cluster pages together brought in roughly 22K monthly organic visits. Nothing revolutionary in any single post — just breadth and clear connections.

The practical version: pick your core topics, build 5–8 solid pieces around each one, and link them tightly. That’s still the most reliable way I’ve found to move the needle on competitive terms. For a deeper breakdown of this approach you can check the topical authority guide.

E-E-A-T Isn’t a Checklist

Google’s E-E-A-T framework (Experience, Expertise, Authoritativeness, Trustworthiness) gets treated like a box-ticking exercise: add an author bio, cite three sources, done. That misses the point.
I’ve audited plenty of sites with perfect author bios and “meet the experts” pages that still rank poorly. The content itself was shallow rehashes of whatever already sat on page one. E-E-A-T isn’t something you bolt on. It’s something the content demonstrates if it’s actually useful.
What moved the needle for me was adding first-hand experience: specific client numbers, what worked, what failed, the exact mistakes I made. Traffic improved — not because Google has a magic “experience detector,” but because those details make the piece more useful, and usefulness is what the ranking systems are getting better at measuring.
The Experience signal is still the most underused. If you’re reviewing a tool, show real screenshots of you using it. If you’re giving advice, say when it worked and when it didn’t. Generic advice doesn’t signal experience. Specific, slightly imperfect, real-world advice does.
For trustworthiness, cite sources. Not because Google necessarily gives a direct ranking boost (though it might), but because unsourced claims read like marketing copy. When I link to a study or reference concrete data, readers trust the piece more. Trust improves engagement metrics, and those still matter.

Stop Writing Like a Textbook

I used to write long dense paragraphs because I thought that felt “comprehensive.” It just made people leave. Time on page dropped, scroll depth was terrible, bounce rate sat around 78%.
Now I keep paragraphs short — usually two to four sentences. I drop a subheading every 200–300 words. I use bullets when a list is clearer than narrative. I deliberately vary sentence length so the rhythm doesn’t become hypnotic and boring.

I aim for roughly grade 8–10 readability. Not because Google cares about Flesch-Kincaid scores (it doesn’t), but because that range correlates with people actually finishing the article. If your content reads like a legal brief, most readers won’t get past the intro. For practical tips on writing that ranks without sounding robotic, see the SEO copywriting guide.

AI Content Is the Elephant in the Room

Almost every SEO is using AI to draft now. I use it too. Publishing raw AI output is still a bad idea.
Google’s helpful content systems specifically target low-effort mass-produced content. Even without that, generic AI writing underperforms. I tested it directly. I published an 800-word post that was pure AI draft — well structured, factually correct, zero edits. It landed at position 34 and stayed there. I rewrote the same topic with personal examples, specific numbers from my work, and actual opinions. Same structure, different soul. It moved to position 7 in about two months.
The workable approach is simple: let AI build the skeleton, then add the parts AI still can’t reliably produce — your experience, your specific data, your judgments, the stories that only come from doing the work. Verify every claim. Rewrite anything that sounds generic. Make sure at least one section could only have been written by someone who’s actually done this.

Content Freshness: Update Only When It Matters

Google cares about freshness for time-sensitive queries and mostly ignores it for evergreen ones. I still see people rewriting “What Is SEO?” every quarter when nothing substantial has changed. Don’t.
Update when the information is actually outdated, when you have meaningful new data, when competitors have clearly surpassed you, or when a real development lands in the topic. Changing the publish date after a meaningful update does help re-crawling. I had a post stuck at position 8 for six months. I added a solid new section with 2026 data, updated the date, and it moved to position 3 within a month. Freshness signals work — but only when the content itself is genuinely fresher.

Final Thought

Content optimization in 2026 is less about tricks and more about doing the unglamorous work consistently: matching intent, covering topics thoroughly, showing real experience, writing so people actually finish reading, and treating AI as a starting point rather than the finished product.

None of this is new. Most of it just requires patience and the willingness to go deeper than the average page ranking for the same terms. That still works.
If you want a quick second pair of eyes on a page you’re optimizing, drop the URL into the free SEO analyzer and look at the keyword placement, readability, and structure scores. The numbers won’t rank the page for you, but they usually show the gaps faster than staring at the draft yourself.

Image SEO: Complete Guide to Optimizing Images for Search

Eduard — Thu, 30 Jul 2026 07:37:16 +0000

A while back I audited an e-commerce site that looked solid on paper. Fast hosting, clean code, decent content. Mobile PageSpeed score: 34. I kept digging until I checked the page weight. Homepage was pulling 14MB of images on first paint. Fourteen megabytes. For a homepage.
Every product photo had been uploaded as a 3000px-wide PNG straight from the photographer’s camera. No compression, no resizing, no modern formats. Mobile users on 4G were downloading desktop-sized files. We converted everything to WebP, added proper responsive sizes, and got the total image payload down to about 800KB. Score jumped to 87. That single change — just the images — moved their Core Web Vitals from “poor” to “good” across the whole site.
That’s when it clicked for me. Image SEO isn’t some nice-to-have checklist item. On a lot of sites it’s the biggest performance bottleneck you’ll find.

Alt Text (and the mistakes I keep seeing)

Alt text does two jobs that people constantly mix up: it helps screen-reader users understand the image, and it gives search engines context so the image can show up in Google Images. Good alt text does both at once.
What I see in almost every audit:

“image of product”
or pure keyword stuffing: “buy blue widget online best price” Neither helps anyone. Something specific works better: “Woman hiking on mountain trail with red backpack at sunset” That’s useful for a screen reader and it tells Google exactly what it’s looking at. Keep it under roughly 125 characters. Screen readers often cut off after that, so longer text just gets ignored. Decorative images (background patterns, spacers, pure design elements) should get an empty alt: alt="". That way screen readers skip them instead of announcing “image”. Early on I made a classic mistake myself. I used the same alt text for every color variant of a product — “Blue running shoe” for the blue, red, and green versions. Google couldn’t tell them apart in image search. Each variant needs its own unique description. ## File names — the five-minute fix most people skip I still open sites and see IMG_4729.JPG and photo-1.png. File names are a ranking signal for images. Renaming them costs nothing and takes almost no time. What I do:
hyphens between words
include the main keyword
make it descriptive red-leather-wallet.jpg is already better than IMG_4729.JPG. seo-audit-checklist-2026.png tells Google (and future-you) exactly what the file is. ## Formats — my actual take, not the marketing version WebP is the right default for most photos. You usually get 25–35% smaller files than JPEG at similar quality, and support is over 96%. For product shots and complex images it’s the clear winner. But I don’t buy the “WebP for everything” advice. On screenshot-heavy tutorial pages I’ve seen WebP create ugly compression artifacts around sharp text edges. In those cases PNG looked better even though the file was larger. Sometimes the “best” format isn’t the best for your images. AVIF is technically stronger — up to 50% smaller than WebP — but support still sits around 80%. I use it as a progressive enhancement with a WebP fallback. Most people won’t notice, the ones on modern browsers get a faster load. SVG for logos and icons is non-negotiable. If your logo is still a PNG, fix it. Vectors scale cleanly and the files are tiny. Quick rule of thumb I use:
photos → WebP (or AVIF + WebP fallback)
screenshots / text-heavy images → PNG
logos & icons → SVG ## Responsive images — stop sending desktop files to phones This is the second biggest performance killer after raw file size. Without srcset, every mobile visitor downloads the full desktop image. Basic pattern:

<img
  src="image-800.jpg"
  srcset="image-400.jpg 400w,
          image-800.jpg 800w,
          image-1200.jpg 1200w"
  sizes="(max-width: 600px) 400px,
         (max-width: 1000px) 800px,
         1200px"
  alt="Descriptive alt text">

Phone gets the 400px version, tablet 800px, desktop 1200px. On that e-commerce site, just adding responsive images cut average image payload by about 60% for mobile users. That’s the difference between a 3-second load and a 1-second load on typical 4G.

Lazy loading — easy to get wrong

Native lazy loading has been around since 2020. Just add loading="lazy":

<img src="image.jpg" alt="Description" loading="lazy">

The mistake I still see constantly: people lazy-load the hero image. Above-the-fold content should load immediately. Lazy-loading the main visual tanks LCP because the browser waits to fetch the most important image on the page.
Also check your CDN or image optimization service. Some inject their own JavaScript lazy-loading that fights with the native attribute. Open Chrome DevTools and watch the network order — make sure the hero loads first and everything else waits.

Image sitemaps — Google can’t index what it never finds

Most images get discovered through the HTML. That fails for images loaded with JavaScript, CSS background images, or stuff buried in carousels and galleries.
Putting images in your XML sitemap gives Google a direct path:

<url>
  <loc>https://example.com/page</loc>
  <image:image>
    <image:loc>https://example.com/image.jpg</image:loc>
    <image:caption>Description of the image</image:caption>
  </image:image>
</url>

I’ve seen image search traffic double on sites after we added proper image entries to the sitemap. The images weren’t new — Google just couldn’t find them before.

Mistakes that show up in almost every audit

Oversized files — 3000px wide when the layout only needs 800px. Resize before you upload.
Wrong format — PNG product photos that should be WebP. The size difference hurts.
Missing or useless alt text — still the most common issue. In my last 20 audits, 14 had serious gaps.
Generic file names — banner-final-v3.jpg tells Google nothing.
No lazy loading (or lazy-loading everything, including the hero).
Leaving EXIF data — location and camera info. Some people keep it for context, I usually strip it on client sites for privacy. ## Tools I actually open
Squoosh — still my go-to for quick one-off compression. The quality slider is excellent.
TinyPNG — solid for batch PNG/WebP work.
Lighthouse (Chrome DevTools) — run it after every change so you can see the real impact.
Auditme — useful for spotting missing alt text, wrong formats, and images that lack responsive sizing across a whole site. ## Quick check If you want a fast look at your own images: Run a free image SEO analysis It flags alt text coverage, format issues, and whether lazy loading is set up correctly.

Eduard Tymchenko
Building tools that help people find and fix the SEO problems that actually move the needle.