For years, we were told to build websites for search engines.
Add the right keywords.
Write a good title.
Generate a sitemap.
Add structured data.
Build backlinks.
Improve Core Web Vitals.
And, of course, make sure Google can crawl everything.
That worked.
But the way people discover information is changing.
Today, someone might not search for your website at all.
They might ask ChatGPT.
Or Claude.
Or Gemini.
Or another AI system.
And that's where things get interesting.
Because a website can be perfectly optimized for Google and still be terrible for AI to understand.
The New Problem Isn't Just SEO
Imagine you have a website with 10,000 pages.
Google can crawl it.
Your Lighthouse score is great.
Your sitemap is valid.
Your metadata looks perfect.
You have structured data.
From an SEO perspective, everything looks healthy.
Now ask an AI:
"What does this company actually do?"
The answer might still be incomplete.
Or worse, the AI might misunderstand the business entirely.
Why?
Because being crawlable isn't the same as being understandable.
That's the problem I became interested in.
What Does "AI-Friendly" Actually Mean?
I started thinking about a website from the perspective of an AI system rather than a search engine.
An AI needs to answer questions about your website.
That means it needs to understand things like:
- What is this website?
- Who is it for?
- What does this company offer?
- Which pages are important?
- How are different pieces of content related?
- Which information is authoritative?
- Can important facts be extracted reliably?
- Is the content structured consistently?
- Are there conflicting or ambiguous signals?
This isn't exactly SEO.
It's closer to machine-readable communication.
And that requires a different mindset.
So I Built an Experiment
Instead of asking:
"How well optimized is this website for Google?"
I wanted to ask:
"How understandable is this website to AI?"
That led me to build an AI Visibility analysis system.
The idea is simple:
Website
↓
Crawler
↓
Page Analysis
↓
Content Analysis
↓
Schema Analysis
↓
Internal Link Analysis
↓
AI Optimization Analysis
↓
AI Visibility Score
↓
Actionable Recommendations
The important part is that this isn't based on a few manually selected pages.
The system starts with the actual website.
It crawls the real pages.
Then analyzes what is actually there.
Crawling Is Only the Beginning
A crawler can tell you that a page exists.
That's not enough.
For every page, I wanted to understand several dimensions.
Content
Is the content clear?
Does the page actually explain what it is about?
Are important entities and concepts explicit?
Structure
Are headings meaningful?
Is the document hierarchy logical?
Can an AI distinguish the main topic from navigation, boilerplate and unrelated content?
Structured Data
Does the page expose useful schema?
Is the schema actually consistent with the visible content?
Are important entities represented?
Internal Links
Can an AI understand the relationship between pages?
Or is the website essentially a collection of disconnected documents?
Metadata
Titles and descriptions still matter.
But they're only one signal.
The bigger question is:
Does the metadata agree with the actual content?
The Interesting Part: Scoring
Once you have all this information, you need a way to turn it into something useful.
So instead of returning a giant technical report, I wanted to produce a score.
Something like:
AI Visibility Score
-------------------
Overall Score 85/100
Content Clarity 91
Schema Health 88
Internal Linking 79
Machine Readability 87
AI Optimization 82
The exact score isn't the important part.
The important part is that the score should be explainable.
A number without evidence isn't very useful.
Every Score Should Have Evidence
One of the things I care about most in this system is traceability.
If the system says:
Internal Linking: 79
I don't want the user to just trust an algorithm.
I want to show:
- which pages were analyzed
- which links were found
- which relationships are weak
- what rules were triggered
- and why the score changed
The same applies to content quality and structured data.
Instead of:
"Your schema is bad."
The system should be able to say:
"These 4 pages declare Organization schema, but the Organization entity isn't consistently represented across the site."
That's much more actionable.
From Analysis to llms.txt
Then I reached another interesting question.
If we're trying to make websites easier for AI systems to understand, shouldn't websites explicitly communicate their structure to AI?
That's where llms.txt becomes interesting.
The concept is straightforward:
Instead of making an AI discover everything from thousands of pages, provide a curated machine-readable entry point.
Something like:
# Example Company
> Example Company provides developer tools for...
## Products
- Product A
https://example.com/product-a
- Product B
https://example.com/product-b
## Documentation
- Getting Started
https://example.com/docs/getting-started
- API Reference
https://example.com/docs/api
The goal isn't to replace the website.
It's to give machines a map.
But There's a Bigger Shift Happening
This isn't really about llms.txt.
It's not even really about AI crawlers.
It's about a fundamental change in how websites are consumed.
For a long time, the primary question was:
"Can a search engine find my page?"
Increasingly, another question matters:
"Can an AI understand my website well enough to represent it correctly?"
Those are different problems.
SEO Isn't Dead
I don't think SEO is going away.
Quite the opposite.
A technically healthy website still needs:
- crawlability
- indexability
- good performance
- useful content
- semantic HTML
- structured data
- internal linking
- strong information architecture
But these things are becoming part of something larger.
Think of it like this:
Website
│
┌───────────┴───────────┐
↓ ↓
Search AI Systems
│ │
SEO AI Visibility
│ │
└───────────┬───────────┘
↓
Machine Understanding
The goal isn't to choose between Google and AI.
The goal is to build websites that machines can understand.
The New Optimization Layer
I think we're moving toward a stack that looks something like this:
Traditional Web
↓
Technical SEO
↓
Semantic Structure
↓
Structured Data
↓
Machine Readability
↓
AI Visibility
And the interesting thing is that most websites are still somewhere around the middle.
They were designed for humans.
Then optimized for search engines.
But they weren't necessarily designed to be interpreted by language models.
What I Learned Building This
The biggest lesson wasn't about AI.
It was about abstraction.
A website contains an enormous amount of information.
Humans can navigate it intuitively.
Machines need signals.
Those signals can come from:
- URLs
- headings
- semantic HTML
- metadata
- schema
- links
- content hierarchy
- consistent entities
- explicit relationships
When these signals agree with each other, the website becomes much easier to understand.
When they contradict each other, even a powerful AI can struggle.
The Future Website Might Be Designed for Three Audiences
We used to think about two:
1. Humans
2. Search Engines
I think we're adding a third:
3. AI Agents
And AI agents are different from search engines.
A search engine wants to discover and rank documents.
An AI agent wants to understand information and use it to answer questions or perform actions.
That's a much bigger responsibility for the web.
So, Should You Start Optimizing for AI?
Yes.
But don't start by stuffing your website with AI-generated text.
Don't add meaningless schema everywhere.
Don't create thousands of pages just because an LLM can generate them.
Instead, start with something much simpler:
1. Make your website understandable.
If you can't explain what a page is in one sentence, an AI probably won't have an easy time either.
2. Make important entities explicit.
Companies, products, people, organizations and concepts should have consistent representations.
3. Build a strong information architecture.
Your website shouldn't feel like a pile of URLs.
4. Use structured data correctly.
Schema should describe reality, not just exist for the sake of SEO.
5. Make relationships obvious.
Internal links aren't only navigation.
They're signals about how information is connected.
6. Measure it.
This is the part we're still figuring out.
SEO has decades of tools and metrics.
AI visibility is still a relatively new space.
That's what makes it interesting.
We're Entering the "Answer Engine" Era
The web was built around documents.
Search engines organized those documents.
Social networks organized people and attention.
Now AI systems are starting to organize knowledge.
And when the interface becomes an AI answer instead of a list of ten blue links, the optimization problem changes.
The question is no longer only:
"How do I rank?"
It becomes:
"How do I become the source an AI understands, trusts and chooses to use?"
That's a very different question.
And I think we're just getting started.
Final Thought
Maybe the next generation of web development won't be about building websites that are optimized for humans or machines.
It'll be about building websites that are understandable to both.
Humans need good UX.
Search engines need crawlable, structured content.
AI systems need context, relationships and clear signals.
The best websites will increasingly provide all three.
Don't stop building for Google.
Just don't build only for Google anymore.
Top comments (0)