How LLM Search Changes the SEO

Sandeep Goel — Fri, 16 Oct 2026 09:08:36 +0000

Most SEO workflows were built around one question: where does this page rank for this keyword? LLM search adds a second question. When someone asks a chatbot or an AI answer box about your topic, can the system find your page, read it, and pull a clean piece of it into an answer?
I have been working through that second question with clients since AI answers started showing up in their reports. This post is the workflow I use now, written for developers and SEOs who share the same site. If you want the short comparison first, I wrote about what actually changes between traditional SEO and GEO on my own site.

Here I will Stay on the Technical Side.

How LLM Search Works, in Plain Terms

Most AI search products follow a similar pattern. A person asks a question. The system rewrites it into one or more search queries, retrieves pages or passages, ranks them, and a language model writes an answer from that material. Some products show citations and some don't. The details differ by product, and nobody outside those companies has the full picture, so I treat this as a working model and not a spec.

Two things follow from it. First, your page has to be reachable by whatever does the retrieval. Second, the passage that gets pulled has to make sense on its own, because the model often sees a slice of your page and not the whole thing.

Everything else in this post hangs on those two points. I keep the steps in one place, which I call my AI search visibility framework, so every project goes through the same checks in the same order.

The Six Steps are:

Crawl access: can the right bots fetch the page and see the content?
Entity clarity: is it obvious who or what the page is about?
Structured data: does the markup match what a reader sees?
Passage design: does each section work when it is read alone?
Retrieval testing: what do real prompts return?
Log monitoring: which bots visit, and what do they get back?

Step 1: Check Crawlability for AI Bots

This is where I start, because nothing else matters if the bot gets a 403.
Different companies run different crawlers for different jobs. Here is the short version, and you should check each vendor's current documentation because names and behavior change.

Googlebot crawls for Google Search. Google says AI Overviews and AI Mode use the same index, and that no special requirements apply to appear in AI features beyond normal SEO fundamentals. Blocking Googlebot removes you from Search entirely.
Google-Extended is a robots.txt token, not a separate crawler. It controls whether your content is used for Gemini model training and grounding. It does not affect Search ranking.
OpenAI uses GPTBot for training data, OAI-SearchBot for ChatGPT search results, and ChatGPT-User when a person asks ChatGPT to open a page.
Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User in a similar split.
Perplexity uses PerplexityBot and Perplexity-User.

The training bots and the search bots are separate decisions. Plenty of publishers allow the search bots and block the training bots. That is a business call, and a robots.txt like this expresses it:
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

In two recent Semrush audits I ran at Obelisk Infotech, the agency I founded, neither site blocked any AI bot. The problems were ordinary ones: broken images, pages with very little text, and missing meta descriptions. Still, I have seen sites where a security plugin or a CDN bot rule quietly challenges unknown crawlers, so I test instead of assuming.

A quick first pass with curl shows what a bot might receive:
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://example.com/your-page/
curl -s -A "OAI-SearchBot" https://example.com/your-page/ | grep -i "a phrase from your main content"

The first command checks the status code. The second checks that your main content is in the raw HTML. Many crawlers do not run JavaScript, so a page that builds its text in the browser can look empty to them. A spoofed user agent is only a rough test, because firewalls often judge bots by IP range too. The real answer is in your server logs, which I come back to in step 6. You can do every check in this post yourself with curl, your logs and Search Console.

Step 2: Make Entities Obvious
An entity is a thing a system can identify: a person, a company, a product, a place. When an LLM answers a question about "the best tool for X" or "who is this consultant", it is working with entities and the claims attached to them. If your site describes the same business three different ways, you make that job harder. For a quick example, I keep a short profile page on WordPress that uses the same wording as my site.

What I do in Practice:
Use one canonical name for the person, brand and product, and use it everywhere.
Write an About page that says plainly who you are, what you do, where you work and who you serve. No slogans.

Keep profile descriptions consistent across LinkedIn, GitHub, directories and your own site.

Link to those profiles from the site, and mark them with sameAs in structured data.

None of this is magic. Repeated, consistent facts are easier for a system to connect than scattered ones, and I can't prove how much weight any single source gets. I treat it as cheap hygiene.

Step 3: Add Structured Data that matches the Page

Structured data helps machines read a page without guessing. Google's introduction to structured data explains how markup helps it understand content, and it prefers JSON-LD. For AI features specifically, Google says no special schema is needed. The rule I follow is simple: mark up what is visibly on the page, and nothing more.
Here is a minimal Organization example:

{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Example Studio",
"url": "https://example.com/",
"logo": "https://example.com/logo.png",
"description": "A web design studio for small clinics in Pune.",
"sameAs": [
"https://www.linkedin.com/company/example-studio",
"https://github.com/example-studio"
]
}

Validate it with the Rich Results Test, then compare it with the page. If the markup says "web design studio" and the page says "marketing agency", fix one of them. Two mistakes show up again and again. One is markup copied from a template that still describes another business, which is easy to do after a redesign. The other is markup for details that appear nowhere on the page, such as ratings nobody can see. Both turn a helpful signal into a mismatch. Local businesses have a few extra properties, like address and opening hours, which I covered in schema markup for local businesses.

Step 4: Write Passages that work on their own

This is the part that changes day-to-day writing the most. Retrieval systems tend to work with chunks of a page. If a chunk starts with "This also fixes the earlier problem," the system has no idea what "this" is.

My rules of thumb, which are habits and not laws:
One question per section. Make the heading close to the question a person would type.

Put the answer in the first one or two sentences, then add detail.
Name the subject again at the start of a section instead of leaning on "it" or "this".

Keep sections short, often 100 to 250 words, with plain tables for comparisons.

Show the author and a last-updated date, and cite your sources.

A Weak Opening: "It also makes pages faster, so that helps." A better one: "Enabling gzip or Brotli compression shrinks text files such as HTML, CSS and JavaScript, which makes pages load faster." The second one can be lifted out and still be true.

I run a simple scoring pass on drafts using my C.A.R.E. framework, which stands for Clarity, Authority, Relevance and Experience. If a section fails on clarity, I rewrite it before anything else.

Step 5: Test Retrieval with real prompts

Rank tracking tells you about a keyword. It does not tell you what an AI product says. So I keep a fixed list of 20 to 30 prompts per site: the questions customers ask, comparison questions, and a few that use the brand name directly.
Once a month I run them in logged-out sessions and record whether the brand is mentioned, whether the page is cited, and whether the answer is correct. Answers change from one run to the next, so I run each prompt more than once and look at patterns, not single results. A shared spreadsheet is enough.

Step 6: Watch your Logs

Logs are the closest thing to ground truth. This command counts AI bot requests by status code and path from a standard access log:
grep -E "OAI-SearchBot|GPTBot|ChatGPT-User|PerplexityBot|ClaudeBot|Claude-SearchBot" access.log \
| awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -20

Look for three things. Which pages do the bots request? Do they get 200s, or a pile of 403s and 404s? And do they come back? Pair this with referral data in your analytics. Visits from domains such as chatgpt.com or perplexity.ai are small today for most sites, but they are real and they are countable.

One optional extra is an llms.txt file in the site root. Chrome's Lighthouse documentation describes llms.txt as a developing standard, a Markdown file that gives AI systems a short overview of the site and its key links, and notes that a missing file is not required yet. I add one because it takes twenty minutes, not because I have evidence that it moves anything.

Notice that nothing in the left column goes away. A page that does not rank, load or index will not be retrieved either. LLM search adds work on top of the basics, and it rewards the sites that already did the basics well.

Where I Would Start

If you have one afternoon, check your robots.txt and CDN rules for AI search bots, confirm your main content is in the raw HTML, and read your last 30 days of bot logs. Those three checks find most of the real problems I see, and none of them needs a tool you don't already have.

I'm Sandeep Goel, an SEO consultant who works on AI search visibility. If you try this workflow on a site, tell me what your logs showed. I'm curious which bots surprised you.

DEV Community: Sandeep Goel

How LLM Search Changes the SEO