DEV Community: JSON-LEE

LLMs.txt vs Robots.txt - What's the Difference and Do You Need Both?

JSON-LEE — Mon, 29 Jun 2026 09:38:29 +0000

The short answer

robots.txt controls which crawlers can access parts of your site. It is for
crawl access management.

LLMs.txt provides an AI-readable summary of important site content. It is
for content orientation and discovery.

They serve different purposes. You need robots.txt for crawler control. LLMs.txt
is optional, but useful when you want to summarize key pages for AI-assisted
retrieval systems.

robots.txt: the gatekeeper

A robots.txt file sits at your site root and tells crawlers which paths they can
and cannot access:

User-agent: GPTBot
Allow: /

User-agent: *
Disallow: /admin/

Google describes robots.txt as a way to manage crawler access, not as a complete
indexing control. A URL blocked in robots.txt can still be discovered through
links. To prevent indexing, use the appropriate noindex mechanism on pages that
crawlers can access.

Our LLMs.txt Checker audits robots.txt for major AI
crawlers including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.

LLMs.txt: the guidebook

An LLMs.txt file also sits at your site root, but it is written in Markdown and
designed to summarize important pages:

# Your Site Name
> Short description of what your site offers

## Key Pages
- [Home](/): Main landing page
- [Documentation](https://aeocheck.xyz/docs): API and integration guides

## Tools
- [AEO Checker](https://aeocheck.xyz/tools/aeo-checker): Technical AEO audit

It is a human-readable, AI-parsable site summary. Think of it as a structured
table of contents with context.

Do you need both?

robots.txt: Yes, if you want clear crawler access rules.

LLMs.txt: Maybe. It is not a confirmed ranking factor and does not guarantee
AI citations. It is still low effort and useful for sites that want to expose a
clean summary of tools, docs, guides, or key resources.

Our recommendation: create one because it improves clarity, not because it is a
magic visibility switch. Use our free LLMs.txt Generator
to create yours in minutes.

The LLMs-full.txt companion

LLMs-full.txt is an extended version that can include more detail than the
shorter LLMs.txt file. It is most useful for documentation sites, knowledge
bases, or content collections where a compact index is not enough.

If your site has only a few public pages, a well-structured LLMs.txt may be
sufficient.

How they work together

Site root/
|-- robots.txt     - Who can crawl and where
|-- llms.txt       - What matters on the site
|-- llms-full.txt  - Expanded AI-readable overview
`-- sitemap.xml    - Crawlable URL inventory

Each file serves a different audience: crawlers use robots.txt and sitemap.xml,
while AI-assisted systems can use LLMs.txt and LLMs-full.txt for orientation.

Checking your setup

Use our LLMs.txt Checker to verify:

All four files are accessible.
AI crawlers are not accidentally blocked.
Your LLMs.txt is properly formatted.
Links are valid and accessible.

It takes a few seconds and gives you a complete picture of your AI search file
readiness.

Sources and further reading

Originally published at aeocheck.xyz — free AI search readiness tools.

LLMs.txt Guide — What It Is, Why It Matters, and How to Make One That Actually Works

JSON-LEE — Mon, 29 Jun 2026 09:37:52 +0000

I've looked at a lot of /llms.txt files over the past few months. Most of them are broken. Not in a "the server returns 500" way, but in a "this is a copy-pasted robots.txt with the wrong MIME type" way. A few are genuinely useful. This guide is about making yours one of the useful ones.

What LLMs.txt actually is

An LLMs.txt file is a plain Markdown file served at /llms.txt on your domain. It's meant to give AI language models a structured summary of your site's key pages — what they're about, why they matter, and how they connect.

It was proposed in 2024 by Jeremy Howard as a lightweight alternative to scraping and parsing full HTML pages. The idea is simple: if an LLM wants to understand your site, give it a clean, structured Markdown file instead of making it dig through your HTML.

Think of it as a README.md for your website. Not a sitemap. Not a robots.txt replacement. A human-readable (and machine-readable) summary of what matters on your site.

The format

Dead simple. Three parts:

# Site Name

> A one-sentence description of what this site provides.

## Section Name

- [Page Title](https://example.com/page): What this page is about
- [Another Page](https://example.com/another): Brief description

What I see people getting wrong

After checking hundreds of sites with our LLMs.txt Checker, here are the most common mistakes:

1. Wrong content type

Your server needs to serve /llms.txt as text/plain or text/markdown with charset=utf-8. A common failure is returning text/html because the server treats the file like a 404 and serves the homepage. In that case, an AI crawler receives a full HTML page instead of your clean Markdown file.

2. Listing every page on the site

I see people dumping their entire sitemap into LLMs.txt. Don't do this. The point is curation. Pick 10-30 pages that actually matter. An LLM doesn't need to know about your cookie policy page or your 47th blog post from 2023.

3. Missing or useless descriptions

A link with no description is just a URL. A link with "Click here to learn more" is noise. Each link should have a one-line description that gives the LLM enough context to understand what's on that page without visiting it.

4. Broken links

You'd be surprised how many LLMs.txt files link to pages that 404. Test your links. Our checker catches these automatically.

5. Only having LLMs.txt, no LLMs-full.txt

LLMs.txt is the summary — 10-30 links with one-line descriptions. LLMs-full.txt (at /llms-full.txt) is the expanded version with full Markdown content. If you have documentation or guides, having both gives AI models the option to read the detailed version when they need deeper context.

What a good LLMs.txt looks like

Here's a real example:

# Acme Docs

> Technical documentation for the Acme platform — APIs, SDKs, and integration guides.

## Getting Started

- [Quickstart](https://docs.acme.com/quickstart): 5-minute setup guide
- [Installation](https://docs.acme.com/install): Install the CLI and SDK
- [Core Concepts](https://docs.acme.com/concepts): How Acme models resources

## API Reference

- [REST API](https://docs.acme.com/api): Full REST API reference
- [Webhooks](https://docs.acme.com/webhooks): Event notifications

## Guides

- [Authentication](https://docs.acme.com/auth): OAuth 2.0 and API key setup
- [Error Handling](https://docs.acme.com/errors): Error codes and recovery
- [Migration Guide](https://docs.acme.com/migrate): Upgrading from v1 to v2

Notice: curated list, clear descriptions, logical sections.

Sections that work well

Core Pages or Getting Started — for the homepage and entry points
Documentation or Guides — for docs, tutorials, how-to content
API or Reference — for technical reference material
Blog or Articles — for key blog posts only (not all of them)
Tools or Products — for interactive tools and product pages

Don't use vague section names like "Other" or "Misc." If a page doesn't fit into a clear section, ask yourself whether it belongs in the file at all.

Template you can use

# Your Site Name

> A one-sentence description of what your site provides.

## Core Pages

- [Home](https://example.com/): What your site does
- [About](https://example.com/about): Who you are

## Key Content

- [Guide Title](https://example.com/guide): What this guide covers
- [Article Title](https://example.com/article): What this article explains

Use absolute URLs. Test every link. Keep it under 50 links.

How many sites actually have LLMs.txt?

On June 25, 2026, I checked 13 well-known developer and SaaS sites by requesting their /llms.txt. Here's what I found:

Site	Status	Notes
tanstack.com	200 OK	Excellent — product docs index with full routing guide
stripe.com	200 OK	Clean summary with link to LLMs-full.txt
cloudflare.com	301 Redirect	Exists but redirects to www subdomain first
openai.com	403 Forbidden	Has a file but intentionally blocks direct access
cursor.com	No file	Connection refused
vercel.com	No file	Connection refused
tailscale.com	No file	Connection refused
linear.app	No file	Connection refused
anthropic.com	No file	Connection refused
perplexity.ai	No file	Connection refused
supabase.com	No file	Connection refused
github.com	No file	Connection refused

That's 2 out of 13 with a proper working LLMs.txt. Even among the companies building AI tooling, adoption is near zero. Stripe and TanStack are the exceptions — and their implementations are solid reference examples.

If you ship an LLMs.txt today, you're ahead of most of the internet.

Testing your file

Use our LLMs.txt Checker — it validates format, checks content type, tests every link, and verifies your LLMs-full.txt if you have one. If you're building from scratch, our LLMs.txt Generator can create a baseline file from your sitemap.

Does LLMs.txt guarantee AI citations?

No. Having an LLMs.txt file does not guarantee that ChatGPT, Claude, Perplexity, or Google AI Overviews will cite your site. It removes a barrier — making your site easy to understand — but the AI still needs to decide whether your content is relevant and trustworthy.

That said, I've seen sites with well-maintained LLMs.txt files get cited more consistently than those without. It's not a ranking signal. It's just good communication.

Originally published at aeocheck.xyz — free AI search readiness tools.

How to Get Cited by ChatGPT, Perplexity, and AI Search Engines in 2026

JSON-LEE — Mon, 29 Jun 2026 09:37:15 +0000

The citation problem

You rank well in Google for your target keyword. But when someone asks an AI
search engine the same question, your page is not cited. The answer links to a
competitor, a documentation page, a forum thread, or an industry publication.

This happens because AI search systems select sources, not just ranked pages.
The work is partly technical and partly reputational: your content must be
crawlable, extractable, trustworthy, and externally corroborated.

1. Allow the relevant crawlers

If a crawler cannot access your public content, it cannot evaluate that content
for retrieval or citation. Check your robots.txt:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

Any broad block should be intentional. Our
LLMs.txt Checker audits common AI crawler rules in
one scan.

OpenAI documents several user agents, including GPTBot, OAI-SearchBot, and
ChatGPT-User. Treat crawler names and purposes as platform-specific, and review
official docs before changing sitewide rules.

2. Make sure search indexes can find you

AI search systems often rely on web search indexes, direct crawling, or both.
If important pages are missing from conventional search discovery paths, they
are harder for answer engines to find.

Submit your sitemap to Bing Webmaster Tools.
Keep sitemap URLs canonical and indexable.
Avoid hiding critical content behind client-only JavaScript.
Use internal links from visible, crawlable pages.

This is not a guarantee of ChatGPT citations. It is a basic discoverability
step that also supports traditional SEO.

3. Structure content for extraction

AI systems need to identify the answer, the context, and the source quickly.

Do:

Put the direct answer near the beginning of each section.
Use question-format headings when the query is question-driven.
Keep paragraphs focused.
Use numbered lists and comparison tables when they make the answer clearer.
Link to primary sources for technical claims.

Do not:

Bury the answer under several paragraphs of setup.
Use clever headings that hide the topic.
Put key facts only in images.
Publish unsupported statistics without a source.

4. Use structured data honestly

Schema markup helps machines interpret the page type, publisher, author, dates,
and content relationships. Useful types include:

Organization for the brand or publisher.
WebSite for the site entity.
Article or TechArticle for editorial pages.
FAQPage for real FAQ sections.
HowTo for procedural content.
SoftwareApplication for tools.

Our AEO Checker detects which schema types your page has
and recommends additions.

5. Build external evidence

The strongest citation signal is not your own claim that you are credible. It is
other credible sources mentioning, reviewing, citing, or linking to your work.

What helps:

Industry publication mentions.
Documentation or tool roundups that include your product.
Customer reviews on relevant third-party platforms.
Public case studies with real examples.
Community discussions where people mention the tool naturally.
Original data or research that others can cite.

You cannot control all of this directly. You can make your work easier to cite:
publish clear pages, name your methodology, include references, and make contact
and correction paths visible.

6. Keep freshness visible

Different AI and search products refresh at different rates. You usually cannot
force a recrawl, but you can make update signals obvious:

Show visible "last updated" dates on guides.
Update sitemap lastmod where appropriate.
Keep RSS feeds current for editorial content.
Avoid silently changing technical recommendations without updating metadata.

7. Track what you can

There is no universal "Search Console for ChatGPT." Track proxies:

GA4 referrals from AI products where available.
Server logs for known crawler user agents.
Bing and Google indexing coverage for key pages.
Manual prompt-bank checks for target queries.
Mentions and links from third-party sites.

Bottom line

Getting cited by AI search engines is not about gaming an algorithm. It is about
being a crawlable, well-structured, externally supported source for a specific
question. Technical signals get you into consideration. Content quality and
off-site authority determine whether you deserve the citation.

Run a free AEO audit on your site to see where you stand.
Also check the AEO checker alternatives comparison
if you are evaluating manual audits or traditional SEO platforms for AI search work.

Sources and further reading

Originally published at aeocheck.xyz — free AI search readiness tools.

AI Search and SEO Are Not the Same Thing — Here's the Difference That Actually Matters

JSON-LEE — Mon, 29 Jun 2026 09:34:32 +0000

I used to think AI search readiness was just SEO with a new name. It's not. The more time I spend on this, the clearer the distinction becomes.

The core difference

Traditional SEO optimizes for ranking in a list of links. You want to be the #1 blue link on Google for "best project management software." The user clicks through to your page, you get the traffic, you monetize.

AI search optimizes for being the source of an answer. When someone asks Perplexity or ChatGPT "what's the best project management software?", the AI reads multiple sources, synthesizes an answer, and cites the ones it used. The user may never click through.

The fundamental units are different:

SEO operates on pages and rankings
AI search operates on facts, claims, and citations

You can be #1 on Google for a keyword and never appear in a single AI-generated answer. And you can be cited in AI answers without ranking in the top 10 for anything.

What still matters

Some things carry over from SEO:

Technical quality — Fast pages, HTTPS, crawlable content. AI crawlers care about this just like Googlebot.
Clear content structure — Headings, lists, tables. Well-structured content is easier for AI models to parse.
Internal linking — AI crawlers follow links like any other crawler. Good information architecture matters.
Backlinks from authoritative sources — Being cited by Wikipedia, academic papers, and major publications signals trust to AI models just like it does to search engines.

What matters for AI search that barely matters for SEO

A few things that are critical for AI search but don't move the needle much for traditional rankings:

LLMs.txt / LLMs-full.txt — These files don't affect your Google ranking at all. But they give AI models a clean, structured map of your site. I've seen sites with great LLMs.txt files get cited more consistently than sites with better backlink profiles but no AI-readable summary.

Structured data for disambiguation — In SEO, schema markup helps with rich snippets. In AI search, it helps the model understand exactly what entity your page is about. When your page says it's about "Mercury" with Organization schema, the AI knows it's the car brand, not the planet or the element.

Cite-worthy claims — SEO content often writes around claims to avoid being wrong. "Many people say..." or "It's generally believed that..." — this is noise to an AI. The models want specific, attributable statements they can use. "According to our analysis of 500 websites, only 12% have a valid LLMs.txt file" is a cite-worthy claim. "Many websites might benefit from LLMs.txt" is not.

Answer positioning — Put your key answer in the first 100 words. AI models have limited context windows for extraction. If your answer is buried in paragraph 12, the model may never get to it before synthesizing its response.

What matters for SEO that barely matters for AI search

And the flip side:

Keyword density — AI models understand semantic meaning. They don't count how many times you wrote "best CRM software." Write naturally. Keyword stuffing doesn't help and may hurt if it degrades readability.

PageRank-style link equity — Internal PageRank distribution through link structures matters much less for AI search. The AI cares about whether your page answers a question well, not whether it receives enough link juice.

Meta descriptions as click-through optimization — In SEO, meta descriptions exist to get people to click. In AI search, the description is a content summary the model uses to understand page purpose. Write it as a summary, not as ad copy.

Freshness signals for non-news content — Google cares about fresh content for certain queries. AI models care more about accuracy than recency. A well-researched page from 2025 with strong structured data may be cited over a hastily written page from this week.

What I do differently now

After a year of paying attention to AI search, here's what changed in my workflow:

I write answers first, context second — I put my main point in the first paragraph, then back it up. Not the other way around.
I make every claim specific and attributable — "We analyzed X and found Y" instead of "Many experts believe..."
I maintain LLMs.txt — It takes 10 minutes to update when I publish something important. Low effort, unclear upside, but consistently correlated with better AI citation rates in my own data.
I use structured data aggressively — Organization schema, Article schema, FAQ schema, BreadcrumbList. Every page gets the schema types that apply.
I stopped obsessing over keyword rankings — I still check them, but I spend more time monitoring which of my pages get cited in AI search results. Different metrics, different priorities.

The honest take

AI search is not replacing SEO. But it's creating a parallel discovery channel that works by different rules. The sites that win on both surfaces will be the ones that understand both games and optimize accordingly.

Run our AEO Checker to see where your site stands on the AI search readiness signals. Then check your Google Search Console for your traditional SEO metrics. The two reports will tell you different things. That's the point.

If you are comparing AEO tools and SEO platforms side by side, we also have a
detailed comparison: AEO checker vs SEO tools
with Ahrefs, Semrush, and Sitechecker.

Originally published at aeocheck.xyz — free AI search readiness tools.

AI Overviews Optimization Guide - What Actually Works in 2026

JSON-LEE — Mon, 29 Jun 2026 09:34:31 +0000

The AI Overviews reality

AI Overviews and chatbot search have changed how informational queries are
answered. For many searches, users now see a synthesized answer with source
links before they see the traditional list of blue links.

That does not mean every page needs a separate "AI SEO" playbook. The durable
work is still clear crawl access, helpful content, structured data, strong
entity signals, and transparent publisher information.

Content structure is your biggest lever

AI systems extract answers from well-structured content. The format matters more
than word count.

Answer-first architecture: Every section should lead with a direct one- or
two-sentence answer, followed by supporting detail. This helps both human
readers and retrieval systems quickly identify the claim being made.

Question-format headings: H2s and H3s that mirror real search questions are
easier to match against answer-seeking queries. "What does X cost?" is clearer
than "X Pricing Overview."

FAQ sections with appropriate markup: If the page genuinely contains
question-and-answer content, FAQPage structured data can make that structure
explicit. Do not add FAQ markup to promotional copy that is not actually an FAQ.

Schema: the technical disambiguation layer

Structured data tells machines what your content represents, not just what it
says. The most useful schema types depend on the page:

Organization identifies the publisher or brand entity.
WebSite identifies the site and its canonical home.
Article or TechArticle identifies editorial content, author, publisher, and dates.
FAQPage identifies real question-and-answer sections.
SoftwareApplication identifies web tools and product pages.
BreadcrumbList clarifies page hierarchy.

Schema alone is not enough. The visible page must match the structured data, and
the content still needs to answer the query better than competing sources.

Entity optimization over keyword repetition

AI search systems reason about entities: organizations, people, tools, topics,
and relationships. They need to know exactly what a page is about and who is
behind it.

Practical steps:

Use your brand name consistently.
Link to official profiles and authoritative references when available.
Include sameAs links in Organization schema only when the profiles are real.
Reference adjacent concepts and related topics in the visible content.
Keep About, Contact, Privacy, Methodology, and References pages easy to find.

Our AEO Checker analyzes entity clarity signals including
brand inference, og:site_name, and Organization schema presence.

Trust signals to prioritize

Search and AI products do not publish a complete citation formula. But public
quality guidance consistently rewards transparency, accountability, and helpful
content.

Prioritize:

Signal	Why it matters
Author or publisher attribution	Shows who is responsible for the page
Published and updated dates	Helps with time-sensitive topics
External references	Lets readers verify technical claims
About and contact pages	Makes the publisher accountable
Privacy and terms pages	Establishes a baseline of operational trust
Methodology page	Explains how tools and scores are produced

Pages that score well in our AEO audit have strong technical readiness across
crawlability, AI-readable files, structured data, content structure, entity
clarity, and trust signals. A high score is not a guarantee of citations or
rankings.

What does not work

LLMs.txt as a magic switch: LLMs.txt can make important pages easier to
discover and summarize, but it is not a confirmed ranking factor and does not
guarantee AI citations.

Keyword stuffing: Repeating a phrase does not make a page more useful.
Clear structure and direct answers matter more.

AI-generated content alone: Thin generated content without examples,
evidence, or original judgment is unlikely to become a trusted source.

Measure what you can

Traditional rank tracking is not enough for AI search. Track proxies instead:

Google Search Console impressions and query patterns.
GA4 referral traffic from AI surfaces where available.
Bot access logs for crawler activity.
Manual prompt-bank checks for your highest-value topics.
Changes in branded search demand and third-party mentions.

Run a free AI Overview readiness check on your pages to
find and fix technical gaps.

Sources and further reading

Originally published at aeocheck.xyz — free AI search readiness tools.

AI Crawlers Are Scanning Your Site Right Now - How to Check and Control Access

JSON-LEE — Mon, 29 Jun 2026 09:33:36 +0000

AI crawlers now appear in many server logs alongside traditional search bots.
Some are used for search retrieval, some for training, and some for broader web
indexing. If you care about AI search visibility, you need to know which ones
can access your public pages.

The most common accidental blocker is simple: a robots.txt rule or CDN bot
setting that prevents AI crawlers from reaching the content you want discovered.

The major AI crawler tokens to check

Here are crawler tokens you may see in logs or robots.txt rules:

Crawler token	Company	Notes
GPTBot	OpenAI	Documented OpenAI crawler token
OAI-SearchBot	OpenAI	Documented OpenAI search-related crawler token
ChatGPT-User	OpenAI	Documented OpenAI user-triggered agent token
ClaudeBot	Anthropic	Documented Anthropic crawler token
Claude-SearchBot	Anthropic	Documented Anthropic search-related crawler token
Google-Extended	Google	Google control token for Gemini Apps and Vertex AI use
CCBot	Common Crawl	Web corpus crawler used by many downstream systems
PerplexityBot	Perplexity	Commonly referenced Perplexity crawler token

Crawler names and purposes change. Always confirm against official platform
documentation before making sitewide access decisions.

First, check what is actually happening

Before you change anything, find out who is already crawling. If you have server
logs:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Google-Extended|CCBot|PerplexityBot" access.log

If you use Cloudflare, check bot and security events and filter by user agent.

Three quick diagnostic steps:

Open https://yourdomain.com/robots.txt and look for broad Disallow: / rules.
Confirm the sitemap is listed in robots.txt or discoverable at /sitemap.xml.
Use our AEO Checker to validate robots.txt and flag restrictive AI crawler rules.

The most common mistake

The blunt rule that makes sites invisible to many crawlers:

User-agent: *
Disallow: /

This blocks every well-behaved crawler that follows the wildcard rule. If you
see it on a public marketing site, blog, or documentation site, it is probably
too restrictive.

A more common pattern is:

User-agent: *
Disallow: /admin
Disallow: /api
Disallow: /private

This can be reasonable. The key is to make sure public content is allowed and
sensitive areas are blocked intentionally.

The allow vs block decision

Allow public content when you want search and AI discovery.

Selectively block sensitive paths such as admin, account, checkout, API, and
private areas.

Block completely only when you intentionally do not want a crawler to access
any public content.

For most content sites, SaaS marketing sites, and documentation sites, the
practical approach is to allow public pages and block private or operational
paths.

Configuring robots.txt

Here is a simple template:

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /admin
Disallow: /api
Disallow: /private

Sitemap: https://example.com/sitemap.xml

Place it at /robots.txt. Make sure it returns a 200 status and a plain text
response.

What blocking actually does

Robots.txt is a crawler instruction, not an authentication system. Major
well-behaved crawlers generally respect it. Bad actors may not.

If a path contains sensitive information, protect it with authentication and
authorization. Do not rely on robots.txt as a security boundary.

Watch out for CDN bot protection

Even if robots.txt is correct, CDN bot protection can still block or challenge
AI crawlers at the network level. If you use Cloudflare or another CDN, review
bot events and WAF rules after changing crawler access.

The 5-point AI search readiness checklist

Robots.txt is accessible and returns plain text.
Sitemap is discoverable and contains canonical public URLs.
AI crawler rules are intentional rather than accidental.
LLMs.txt exists at /llms.txt if you want an AI-readable site summary.
Structured data is present on important pages.

Run our AEO Checker to audit these signals in one scan.

The bottom line

Most accidental AI crawler blocks come from broad robots.txt rules or CDN bot
settings. Both are fixable. The right setup is not "allow everything forever";
it is to make public discovery intentional and private areas truly private.

Sources and further reading

Originally published at aeocheck.xyz — free AI search readiness tools.

AEO vs SEO - What's Different and Why It Matters for AI Search Visibility

JSON-LEE — Mon, 29 Jun 2026 09:33:35 +0000

Two different discovery surfaces

SEO optimizes for search engines that return ranked links. AEO optimizes for
answer engines that synthesize responses and cite sources.

The two overlap, but they are not identical. SEO asks whether a page can rank
and attract clicks. AEO asks whether a page can be understood, extracted,
trusted, and cited as an answer.

SEO vs AEO: the comparison that matters

	SEO	AEO
Goal	Rank in search results	Get selected as a source for an answer
Output	A list of links	A synthesized response with citations
Key signals	Crawlability, relevance, links, performance	Crawlability, schema, content structure, entity clarity, trust signals
Success metric	Impressions, rankings, clicks, conversions	Citations, AI referrals, answer inclusion, brand mentions
Measuring tools	Google Search Console, Bing Webmaster Tools, analytics	Analytics, server logs, prompt testing, citation monitoring

Why the distinction matters

Traditional SEO is still essential. Search engines remain a major discovery
channel, and many AI products use conventional search indexes or web crawling as
part of retrieval.

But answer engines compress the user journey. If an answer is synthesized before
the user clicks, the source must be clear, trustworthy, and easy to extract.
That puts more weight on structure, attribution, freshness, and external
evidence.

What AEO checks that SEO tools often miss

Standard SEO tools commonly check:

Rankings and backlinks.
Keyword targeting and metadata.
Page speed and Core Web Vitals.
Indexability and technical crawl issues.

AEO tools additionally check:

AI crawler access - Are key AI crawler user agents blocked?
LLMs.txt presence - Is there a structured site summary for AI-assisted systems?
Schema completeness - Which schema types exist, and are there parse errors?
Content structure - Are answers, headings, lists, and FAQ sections easy to extract?
Entity clarity - Can machines identify the brand, site, and topic?
Trust signals - Are author, dates, contact pages, policies, methodology, and references visible?

Our AEO Checker audits these dimensions and gives you a
technical readiness score with prioritized fixes. See how it
compares to traditional SEO tools like
Ahrefs, Semrush, and Sitechecker.

They overlap more than vendors admit

AEO and SEO are not enemies. Good SEO fundamentals help AEO: crawlable pages,
clear titles, useful content, internal links, sitemaps, and fast rendering all
matter.

Good AEO practices also help SEO: structured data, transparent authorship,
clear page hierarchy, and helpful references improve the page for users and
search systems.

The difference is emphasis. SEO focuses on rankings and traffic. AEO focuses on
being a reliable, extractable source for a direct answer.

What to prioritize

Start here:

Check crawler access - Make sure robots.txt does not accidentally block important crawlers.
Add appropriate schema - Use Organization, WebSite, Article, FAQPage, SoftwareApplication, or BreadcrumbList where they match the page.
Restructure content - Lead sections with direct answers and use clear headings.
Build entity signals - Use consistent names, canonical URLs, and real sameAs links.
Add trust pages - Make About, Contact, Methodology, References, Privacy, and Terms easy to discover.

Use our free AEO Checker to find where your site stands
and which fixes to prioritize.

Sources and further reading

Originally published at aeocheck.xyz — free AI search readiness tools.

Technical AI search readiness: what to check before chasing AI citations

JSON-LEE — Fri, 26 Jun 2026 08:58:05 +0000

Classic SEO audits usually answer questions like: can Google crawl this page, does the title tag exist, is the sitemap discoverable, and are canonical tags configured correctly?

Those checks still matter. But AI-assisted search adds a slightly different question:

Can an answer engine crawl, understand, extract, and trust this page?

I have been building a free technical checker for that question. The goal is not to predict rankings or promise citations from ChatGPT, Perplexity, Gemini, Claude, Bing, or Google AI Overviews. The goal is much narrower: surface concrete readiness issues a site owner can actually fix.

The checks include:

Whether robots.txt blocks common AI and search crawlers.
Whether LLMs.txt or LLMs-full.txt exists and is readable.
Whether sitemap discovery works.
Whether important pages expose structured data.
Whether headings and page sections are answer-ready.
Whether the publisher/entity is clear.
Whether trust signals such as about, contact, author, dates, and references are visible.

The most useful part so far has been separating "AI visibility" from "AI readiness." Visibility is the downstream result. Readiness is the technical and editorial surface that makes the site easier to crawl, parse, and cite.

That distinction matters because site owners cannot directly force an AI system to cite them. But they can remove crawler blocks, expose clearer files, improve content structure, add useful schema, and make sources and ownership easier to verify.

I published the checker here:
https://ai-search-readiness.s01071233604.workers.dev/tools/aeo-checker

Methodology:
https://ai-search-readiness.s01071233604.workers.dev/methodology

References:
https://ai-search-readiness.s01071233604.workers.dev/references

Feedback welcome, especially on false positives, missing crawler rules, and which checks are genuinely useful in production SEO workflows.

I changed my JSON tool’s privacy copy from “trust me” to “verify it in DevTools”

JSON-LEE — Tue, 16 Jun 2026 11:13:18 +0000

I made another round of changes to SafeJSON today.

Not a redesign.
Not a new theme.
Not a big UI change.

Mostly copy, positioning, and how the product explains its privacy model.

That sounds small, but I think it matters more than I expected.

SafeJSON is a browser-based JSON toolkit. It has tools like a JSON formatter, validator, beautifier, viewer, parser, CSV ↔ JSON converter, JSON diff, JWT decoder, JSONPath query, and JSON Schema validator.

The tools themselves are not hard to explain.

The harder part is explaining why this JSON tool should exist when there are already so many JSON tools.

For me, the answer keeps coming back to trust.

“Privacy-first” was too vague

At first, I described SafeJSON as a privacy-first JSON toolkit.

That is not wrong, but it is also not strong enough.

“Privacy-first” is a claim.

And developer tools should not depend too much on claims.

If a tool asks developers to paste API responses, logs, JWTs, webhooks, AI outputs, or backend data into a browser window, the privacy wording needs to be more concrete than that.

Sometimes the pasted JSON is harmless.

Sometimes it contains internal IDs, tokens, credentials, customer data, headers, production logs, or things the developer did not notice were sensitive.

So the real question is not:

“Does this website say it cares about privacy?”

The better question is:

“Can I verify what happens to the content I paste?”

That is the part I wanted SafeJSON to make clearer.

The actual claim should be testable

The wording I moved toward is:

no pasted-content upload

That is much more specific than “privacy-first.”

It does not try to sound bigger than it is.

It does not ask the user to believe a broad marketing promise.

It points to something a developer can check.

Open DevTools → Network.
Paste JSON.
Run the tool.
Check the requests.

The thing to verify is simple:

No request should contain the pasted content.

That is the trust model I want SafeJSON to communicate.

Not “trust this site.”

More like:

“Here is the thing you can test yourself.”

Why this matters for JSON tools

A JSON formatter is boring until the pasted data is not boring.

If you paste sample JSON from a tutorial, it does not matter much.

If you paste a real API response from a production system, it starts to matter.

If you paste a JWT, webhook payload, internal log, or AI output that includes private context, it matters even more.

I do not think every developer will check DevTools every time.

Most probably will not.

But I still think the product should make the verification path obvious.

Because that changes the relationship between the tool and the user.

Instead of saying:

“Please trust that we handle your data safely.”

SafeJSON should say:

“You can check whether your pasted content leaves the browser.”

That feels like a better standard for this category.

What I changed in the product

The recent changes were mostly about making this idea easier to understand.

I tightened the wording around browser-based processing.

I avoided vague security language.

I made the privacy claim more specific.

I tried to make the main difference clearer:

SafeJSON is not trying to win because it has every possible JSON feature.

It is trying to win because it gives developers a simple way to work with JSON without blind trust.

The product is still small.

Current state:

51 users in the last 28 days.
0 paying customers.
Edge extension live.
Chrome extension still pending review.

So this is not a success story.

It is more of a positioning note from building a tiny developer tool.

But this update made one thing clearer for me:

For developer tools, privacy copy should be verifiable.

Not just reassuring.

If you work with JSON a lot, I’d be curious whether “verify it in DevTools” feels like a real wedge or just something I personally care about.

How to Check If an Online JSON Formatter Uploads Your Data

JSON-LEE — Mon, 15 Jun 2026 12:52:47 +0000

Most developers have done this at least once.

You get a messy API response.

You need to inspect a JWT.

You have a webhook payload, a log object, or a config file that is hard to read.

So you open a JSON formatter, paste the content, and move on.

That habit is convenient. But it also deserves a second look.

Not every JSON tool behaves the same way. Some tools process your input entirely in the browser. Some send content to a server. Some store snippets for sharing. Some extensions have permissions that are broader than you expect.

The problem is not that every online formatter is unsafe.

The problem is that you often do not know what happens after you paste.

What you should avoid pasting blindly

Before using any random online tool, be careful with:

production JWTs
API responses containing user data
logs from real systems
config files
webhook payloads
database URLs
cloud keys
internal endpoints
tenant IDs
error traces from production systems

A JSON payload does not need to contain an obvious password to be sensitive.

Sometimes the risky part is context: user IDs, internal URLs, tokens, customer data, or system structure.

A quick DevTools check

You can do a basic check with your browser’s DevTools.

Open the JSON tool.
Open DevTools.
Go to the Network tab.
Clear existing requests.
Paste a harmless test JSON first.
Run format, validate, diff, decode, or whatever action the tool provides.
Watch the Network tab.
Look for POST, PUT, fetch, XHR, or beacon requests after your input.
Inspect request payloads if they exist.
Check whether your pasted JSON appears in any request.

Do this with harmless test data first.

If the tool uploads the test JSON, do not paste production content into it.

What to look for

A few signs deserve attention:

POST requests after you paste or click format
request bodies containing your JSON
share-link features that save snippets
server-side validation APIs
analytics events that include pasted content
extension background requests that are not clearly explained

Analytics by itself is not the same thing as pasted-content upload.

A tool can have normal page analytics and still avoid sending your JSON payload.

The important question is narrower:

Is my pasted content included in a network request?

That is what you are trying to verify.

Browser extensions need the same scrutiny

Extensions can be useful, but they deserve the same level of inspection.

A JSON formatter extension may need content scripts or host access to detect JSON pages and format them. That does not automatically mean it is malicious.

But the extension should explain:

why each permission exists
whether it loads remote code
whether it injects ads or affiliate scripts
whether it uploads JSON content
where the source code can be audited

If an extension handles developer data, its permission model should not be vague.

How SafeJSON approaches this

I built SafeJSON around one idea:

Do not ask developers to trust a privacy claim. Give them a way to verify it.

SafeJSON’s core tools process pasted JSON locally in the browser and do not upload pasted content for core operations.

That includes common workflows like formatting, validating, viewing, parsing, JSON Diff, JWT decoding, JSONPath queries, and schema validation.

The important part is not just the claim.

The important part is that you can check it yourself.

Open DevTools → Network, paste JSON, run the tool, and inspect whether pasted content is uploaded.

I wrote a full step-by-step guide here:

https://www.safejson.dev/security/check-json-formatter-upload

There is also a dedicated verification page:

https://www.safejson.dev/privacy/verify-local-processing

And for the browser extension permissions:

https://www.safejson.dev/extension/permissions

Final thought

The goal is not to stop using web tools.

The goal is to stop pasting sensitive developer data into tools whose behavior you have never checked.

A quick Network tab check takes less than a minute.

For production JSON, JWTs, logs, configs, and API payloads, that minute is worth it.

I rewrote SafeJSON’s privacy copy because “privacy-first” was too vague

JSON-LEE — Mon, 15 Jun 2026 08:35:03 +0000

I changed a lot of the copy on SafeJSON today.

Not the design.
Not the theme.
Not the layout.

Mostly the wording around privacy.

At first, I thought the message was simple:

SafeJSON is a privacy-first JSON toolkit.

But the more I looked at that sentence, the less useful it felt.

“Privacy-first” sounds good, but it still asks the user to trust the website.

And for a developer tool, that is not enough.

The problem with vague privacy claims

A lot of online JSON tools ask developers to paste API responses, logs, JWTs, webhooks, AI outputs, or backend data into a text box.

Sometimes the data is harmless.

Sometimes it is not.

It may contain tokens, internal IDs, customer data, credentials, headers, production logs, or things the developer did not even notice were sensitive.

The uncomfortable part is that many tools do not make it obvious what happens after you paste.

Is the content processed locally?

Is it sent to a server?

Is it stored?

Is it logged somewhere?

Most users will not know unless they check.

And most websites do not encourage them to check.

That is the part I wanted to make clearer with SafeJSON.

The wording had to be more precise

I do not want SafeJSON’s privacy message to depend on big claims.

I also do not want to use wording that sounds stronger than what is technically true.

So I moved away from vague phrases and tightened the message around this:

No pasted-content upload.

That is the actual claim.

Not magic security language.
Not a broad trust statement.
Not “just believe me.”

The tools process pasted JSON in the browser, and the important thing is that the pasted content is not uploaded in requests.

The better promise: verify it yourself

The main idea behind SafeJSON is not just privacy.

It is verifiable privacy.

A developer can open DevTools → Network, paste JSON into a SafeJSON tool, run the formatter, validator, diff, JWT decoder, or other tools, and check the requests.

The thing to verify is simple:

Does any request contain the pasted content?

That is a much better test than reading another privacy headline.

It is observable.
It is repeatable.
It does not require trusting my marketing copy.

Why this matters for JSON tools

JSON tools are boring until the pasted data is not boring.

A formatter is just a formatter when you paste sample data.

It becomes different when you paste a real API response, a production log, a webhook payload, a JWT, or an AI output that includes internal context.

That is why I think browser-based processing matters for this category.

Not because every JSON file is sensitive.

But because developers should not have to think too hard before formatting data they are already working with.

What SafeJSON is now trying to say

SafeJSON is a browser-based JSON toolkit.

It includes tools like JSON Formatter, Validator, Beautifier, Viewer, Parser, CSV ↔ JSON, JSON Diff, JWT Decoder, JSONPath Query, and JSON Schema Validator.

But the positioning is not “we have many JSON tools.”

There are already many JSON tools.

The positioning is:

You should be able to use a JSON tool without blind trust.

Open DevTools → Network and verify that no request contains pasted content.

That is the part I want to make clearer across the product.

Still very early

SafeJSON is still tiny.

Current state:

51 users in the last 28 days.
0 paying customers.
One Edge extension live.
Chrome extension still pending review.

So this is not a “big launch” post.

It is more of a product positioning note.

I am trying to figure out whether “verifiable privacy” is a real wedge for a JSON developer toolkit, or just something I personally care about.

But after rewriting the site today, I feel more confident about one thing:

For developer tools, privacy copy should be testable.

Not just reassuring.

How to Verify Your JSON Formatter Is Safe: A 30-Second Test

JSON-LEE — Tue, 09 Jun 2026 10:46:44 +0000

ou use online JSON tools every day. But how do you know they are not sending your data to a server?

Here is a 30-second test that works on any online tool:

The Network Tab Test
Open your JSON formatter of choice
Open DevTools (F12) → Network tab
Paste any JSON data into the tool
Look for new network requests
If you see XHR or fetch requests when you paste or format JSON, your data has left your browser and is now on someone else's server.

This is not hypothetical. In November 2025, security researchers at watchTowr discovered that jsonformatter.org and codebeautify.org had been storing user-submitted data without authentication. Over 80,000 code snippets — including AWS keys, GitHub tokens, and database passwords — were publicly accessible. Attackers were actively scraping the data within 48 hours.

Red Flags in Online Developer Tools
Beyond the Network Tab test, here are signs a tool is processing your data server-side:

"Save" or "Share" features. If a tool offers to save your work or generate a shareable link, your data is stored on a server.
"Recent" or "History" pages. jsonformatter.org's "Recent Links" page was the exact feature that caused the credential leak.
Loading spinners during formatting. If formatting is not instant, the tool is likely making a round trip to a server.
No explicit privacy claim. If the tool does not state "all processing is client-side" or "your data never leaves your browser," assume it is server-side.
Client-Side Tools That Pass the Test
The safest online tools process everything in your browser. Here is how to identify them:

Open DevTools → Network tab
Paste data
Zero new requests = your data stayed local
This is the design principle behind SafeJSON. All formatting, tree view rendering, JWT decoding, and JSON diff comparison happens in JavaScript running in your browser. There is no backend processing user data.

The Bottom Line
You would not paste your AWS credentials into a random person's terminal. But every time you use a server-side online tool, that is essentially what you are doing.

The fix is simple: take 30 seconds to check. Open DevTools. Look at the Network tab. If you see requests going out, find a tool that does not.

I built SafeJSON after learning about the jsonformatter.org breach. It is open source (MIT) at github.com/s01071233604-tech/safejson.