Originally published at chudi.dev
To optimize for Perplexity, ChatGPT, and Claude, make your site crawlable and format content for extraction. That means allowing AI crawlers in robots.txt, adding llms.txt, and using structured headings plus schema on priority posts. This guide walks through the exact steps.
Quick Wins (Do These Today)
- Update
robots.txtto allow AI crawlers - Create
/llms.txtat site root - Add BlogPosting schema to 10 top articles
- Structure one article with 15+ headers instead of 3
That's a 30-minute investment that unlocks visibility in Perplexity, ChatGPT, and Claude.
Why Do AI Engines Need Different Optimization Than Google?
Google ranks pages based on backlinks and authority signals. AI engines — Perplexity, ChatGPT, Claude — retrieve content by semantic relevance and structure. They need explicit crawler permission, clean schema markup, and answer-first formatting. Without these signals, your content is invisible to AI search regardless of how well it ranks on Google.
Step 1: Update robots.txt for AI Crawlers
Your robots.txt is the bouncers list for search crawlers. Most sites have something like:
User-agent: *
Disallow: /admin
Disallow: /api
Sitemap: https://yoursite.com/sitemap.xml
This is Google-focused. AI engines need explicit permission. Update it:
# Allow all standard crawlers
User-agent: *
Disallow: /admin
Disallow: /api
Disallow: /private
# Explicitly allow AI crawlers
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot-Extended
Allow: /
User-agent: CCBot
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Why each one matters:
| Bot | Owner | Used By |
|---|---|---|
| GPTBot | OpenAI | ChatGPT, GPT-5 |
| ClaudeBot | Anthropic | Claude.ai, API users |
| PerplexityBot | Perplexity | Perplexity search |
| Googlebot-Extended | Google's AI Overview, SGE | |
| CCBot | CommonCrawl | Hugging Face, open-source models |
Test it: Use this to verify:
curl -I https://yoursite.com/robots.txt
Step 2: Create /llms.txt (New File)
While robots.txt controls access, /llms.txt controls attribution. Create a new file at the root. For a comprehensive deep-dive on this file, see my guide on llms.txt and robots.txt for AI crawlers:
https://yoursite.com/llms.txt
# LLM Content Policy
All articles on this site are available for training, search indexing, and answer generation by LLMs.
## How to attribute our content:
For articles: [Article Title] by [Author Name] (sitename.com)
For data: Link to the specific section
For code: Preserve license headers
## How we'd like to be credited:
If citing multiple articles, link to: https://yoursite.com
## Content discovery:
- Sitemap: https://yoursite.com/sitemap.xml
- RSS: https://yoursite.com/rss.xml
- Blog archive: https://yoursite.com/blog
## Content we don't want indexed:
- Drafts (marked `draft: true`)
- Private tools or dashboards
- Archived content older than 5 years
## Preferred citation format:
[Article Title] — Author Name on yoursite.com
---
Last updated: January 2025
Why it matters: AI engines scan for llms.txt to understand your content policy. Without it, some engines might skip you (too risky). With it, you're explicitly inviting them in.
What Schema Markup Should You Add for AI Search?
Start with BlogPosting schema on every article — it tells AI engines the content type, author, date, and topic. Add FAQPage schema wherever you have Q&A sections, and HowTo schema on step-by-step guides. These three schema types cover most blog content and have the highest citation impact in AI search.
Step 3: Add Schema.org Structured Data
AI engines parse JSON-LD to understand content structure. This structured approach to content is what enables the extraction and synthesis that powers answer engines. Add this to your blog post template:
For Articles (BlogPosting)
Add this in your page's <head>:
For How-To Content (HowToSchema)
If you're teaching a process:
For FAQ Content (FAQPage)
Verify schema: Use Google's Rich Results Tester to validate.
Step 4: Structure Content for Extraction
AI engines need to find the answer within your content. This means:
Use Headers to Break Up Content
Bad structure:
# Article Title
<p>Long paragraph explaining the concept...</p>
<p>More context...</p>
<p>Finally, the key insight...</p>
Good structure:
# Article Title
## What is AEO?
<p>AEO (Answer Engine Optimization) is the practice of structuring content so AI engines can extract, cite, and surface it in generated answers. It focuses on crawl access, structured data, and answer-first formatting.</p>
## Why does AEO matter for visibility?
<p>AI engines like Perplexity and ChatGPT now answer queries directly without sending users to search results. If your content is not optimized for extraction, you are invisible to this growing traffic source.</p>
## How to implement AEO
<p>Steps...</p>
## Common mistakes
<p>What to avoid...</p>
## FAQ
- Q1: Answer
- Q2: Answer
Every 2-3 paragraphs, add a header. This makes it easier for AI to:
- Find the specific section answering a user's question
- Extract just that section (not the whole article)
- Cite the correct part of your content
Use Lists for Dense Information
Instead of:
"To optimize your site, you need to update your robots.txt file, create an llms.txt file, add schema to your articles, and structure your content with headers."
Write:
"To optimize your site:
- Update robots.txt for AI crawlers
- Create llms.txt at site root
- Add schema to articles
- Structure with semantic headers"
AI engines can extract lists more reliably than paragraph prose.
Put the Answer at the Top
Don't make readers scroll for the punchline. If your headline is "Why AEO Matters," answer it in the first paragraph:
"AEO matters because 30% of searches now go through AI engines instead of Google. If your content isn't optimized for Perplexity, ChatGPT, and Claude, you're invisible to an entire audience."
Then expand with context, examples, and proof.
Use Definition Boxes
For key concepts, use a highlighted box:
> **Definition:** AEO (Answer Engine Optimization) is optimizing your content to be found, extracted, and cited by AI search engines.
This signals to AI engines: "This is important context."
Include Tables and Structured Data
Tabular data is easier for AI to extract:
| Factor | SEO | AEO |
|---|---|---|
| Ranking | Backlinks | Content structure |
| Speed | Important | Less important |
Don't just describe in paragraphs. Use tables.
How Do You Know If AI Engines Are Citing Your Content?
Search your target keywords directly in Perplexity, ChatGPT search, and Claude. If your site doesn't appear in sources within 4-6 weeks of publishing, you likely have a crawl access problem, a freshness issue, or your content isn't structured as direct answers. Manual citation checks take five minutes weekly.
Step 5: Monitor Visibility in AI Engines
Search Your Topics in Perplexity
Go to perplexity.ai and search your main keyword. Do you see your content cited?
If yes ✅ → Your content is discoverable
If no ❌ → You need to audit (usually a robots.txt or freshness issue)
Check ChatGPT Search
OpenAI's ChatGPT now searches the web. Search your site name + keyword. Does your content appear?
Use Perplexity Citation Tracking
When your content is cited, you'll see traffic from perplexity.com in your analytics. Track this growth.
2026 Update: Platform-Specific Citation Strategies
Since this article was first published, research has revealed that each AI platform has distinct source preferences. A one-size-fits-all approach to AI search optimization is no longer sufficient.
Perplexity: The Reddit Connection
Perplexity cites approximately 6.6 sources per answer, more than any other platform. Its source pool leans heavily on Reddit, with 46.7% of top-cited sources coming from Reddit discussions. This means genuine participation in subreddits relevant to your expertise (r/SEO, r/webdev, r/artificial) directly increases your chances of being cited.
The mechanism: Perplexity indexes Reddit threads and cites them as sources. When your expertise appears in those threads (with or without links to your site), Perplexity associates your knowledge with your brand.
ChatGPT: Recency and Authority
ChatGPT cites only about 2.6 sources per answer, making it the most selective platform. It shows strong preference for Wikipedia (7.8% of all citations) and recently updated content. 95% of ChatGPT citations come from content published or updated within the last 10 months.
To improve ChatGPT citations: update your highest-value articles quarterly with new data, add dateModified schema, and ensure your content contains specific statistics AI can attribute.
Google AI Overviews: Still Tied to Rankings
Google AI Overviews show 76% overlap with traditional top 10 results, making it the most SEO-correlated AI feature. If you rank well on Google, you're likely to appear in AI Overviews. Focus traditional SEO efforts here.
Measuring Your AI Visibility
You can now audit your site's AI infrastructure readiness with automated tools. citability.dev runs 10 checks against your site covering robots.txt, sitemap, structured data, answer-first content, and freshness signals. It measures three tiers: AI visibility (can AI find you?), AI recommendability (does AI suggest you?), and AI citability (does AI link to you?).
To measure your citation rate directly, query ChatGPT, Perplexity, and Claude with 20 questions your site should answer. Track two metrics per query: whether the AI mentions your brand (visibility) and whether it links to a URL on your domain (citation). In our benchmark of 7 sites, the gap between these two numbers ranged from 25 to 95 points. Ahrefs was 100% visible but only 5% cited. A site with DA under 10 achieved 15% citation rate by focusing on original data and answer-first structure. The full methodology is documented in I Audited 7 Websites for AI Citability.
Complete Optimization Checklist
- [ ] robots.txt allows GPTBot, ClaudeBot, PerplexityBot
- [ ]
/llms.txtexists at site root - [ ] Top 10 articles have BlogPosting schema
- [ ] How-to content has HowToSchema
- [ ] FAQ content has FAQPageSchema
- [ ] Content has 10+ headers (not 3)
- [ ] Key answers in first paragraph
- [ ] Important data in tables, not paragraphs
- [ ] Meta descriptions under 160 chars (Google habit)
- [ ] Images have descriptive alt text
- [ ] No critical content in images only
- [ ] Content updated in last 12 months (fresh signal)
- [ ] Run infrastructure scan at citability.dev to identify gaps
- [ ] Measure AI citation rate with 20 manual queries across 3 platforms
- [ ] Add original data (benchmarks, tables, unique statistics) to top articles
Why Isn't My Content Showing Up in AI Search Results?
The four most common causes: AI bots blocked in robots.txt, conflicting canonical URLs from cross-posting, duplicate content diluting citation chances, and schema errors preventing structured data from registering. Run through each diagnostic in order — most optimization failures trace back to one of these four fixable issues.
When Your Optimization Isn't Working
If you've done the five steps and still don't see citations in Perplexity or ChatGPT after 4-6 weeks, run this diagnostic.
Check if AI bots are reaching your pages. Look in your server logs or analytics for user-agent strings containing GPTBot, ClaudeBot, or PerplexityBot. If you see no traffic from these bots, there's a crawl access issue—either robots.txt is still blocking them, or your hosting provider's firewall doesn't recognize these newer bot names.
Verify your canonical is consistent. AI systems avoid citing content with conflicting canonical signals. If you're cross-posting to Dev.to, Medium, or LinkedIn, make sure each cross-post has your original URL in the canonical field—not the platform's default. One conflicting canonical can suppress citations across all AI engines.
Check for duplicate content. AI engines, like Google, downweight duplicate content. If you have two posts targeting the same query, consolidate them. Two 800-word posts on the same topic compete with each other and dilute citation chances compared to one 1,600-word authoritative post.
Test your schema. Run your top 3 posts through Google's Rich Results Test. If schema errors appear, fix them—AI engines use the same structured data signals to understand content type.
Most optimization failures trace back to one of these four issues. The good news: all four are diagnosable and fixable in an afternoon.
The Advantage
Here's the thing: most creators still treat AEO as optional.
You just did these 5 steps. Your competitors haven't. This gives you a 6-12 month window where your content will be cited more often in AI answers, driving visibility and traffic.
This is the opposite of SEO, where the first-mover advantage is gone. AEO is still day one. If you want to go deeper into answer engine optimization as a comprehensive strategy, check out my detailed AEO guide.
Next: Run a free AI visibility scan at citability.dev to check your infrastructure readiness, or dive deeper with the full AEO guide.
Sources
- Google Search Central: robots.txt overview (Google Search Central)
- Schema.org BlogPosting (Schema.org)
- Sitemaps XML format (Sitemaps.org)
- Evergreen Media: Answer Engine Optimization (Evergreen Media)
- AI Visibility Readiness Framework (citability.dev)
Top comments (0)