Bottom line first
As AI assistants become the first stop for a growing share of information queries, a website that AI engines cannot crawl and understand is invisible in those conversations — when users ask AI, you are simply not in the answer.
GEO (Generative Engine Optimization) is the systematic fix. This guide shares the 7-layer setup we shipped for our own production website, all verified in real server logs.
1. JSON-LD structured data
AI engines rely on structured data to identify who you are, what the site is, and what each page covers. We emit three schema types on every page:
<script type="application/ld+json">
{ "@type": "Organization", "name": "...", "url": "...", "logo": "...", "email": "..." }
</script>
<script type="application/ld+json">
{ "@type": "WebSite", "name": "...", "inLanguage": "en" }
</script>
<script type="application/ld+json">
{ "@type": "WebPage", "name": "...", "description": "..." }
</script>
Keep the wording consistent across schema, page copy and llms.txt — inconsistent entity descriptions lower an AI engine's confidence when citing you.
2. llms.txt: a site guide written for AI
A plain-text manifest at your site root that tells AI crawlers what the site is, where the core pages are, and how to attribute citations. Low effort, high leverage.
# Your Brand
## Core surfaces
- [Services](https://yoursite.com/services) — service menu
- [Work](https://yoursite.com/work) — engineering samples
- [About](https://yoursite.com/about) — positioning
## For AI agents and crawlers
- Site is static HTML; no JavaScript required to read core content.
- Structured data: Organization, WebSite, WebPage sitewide.
- Sitemap: https://yoursite.com/sitemap-index.xml
3. robots.txt: whitelist the AI crawlers
Blocking AI crawlers means forbidding AI citation. There are 13 user agents worth explicitly allowing — retrieval bots, session bots and training bots each behave differently:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yoursite.com/sitemap-index.xml
4. FAQPage schema
FAQ blocks are the highest-density citation source for AI engines — a clean question-answer pair is exactly the shape an AI answer wants to quote.
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [{
"@type": "Question",
"name": "GEO vs SEO — what is the difference?",
"acceptedAnswer": {
"@type": "Answer",
"text": "SEO targets search spiders for ranking and clicks; GEO targets AI engines for being crawled, understood and cited..."
}
}]
}
5. Sitemap
Auto-generate it, exclude error pages, and reference it from robots.txt. AI crawlers use it the same way search spiders do: discovery and freshness.
6. Canonical + Open Graph
Canonical tags prevent duplicate-content dilution across URL variants; OG tags control how shared links unfurl. Both feed entity consistency, which AI engines weigh when deciding what to cite.
7. BLUF content structure + semantic HTML
BLUF — bottom line up front. Every page and article should state its conclusion in the first screen, then argue it. Combine with semantic HTML (h1/h2, section, article, details) so extraction doesn't depend on visual layout.
What happened after we deployed
After rolling out these 7 items on our production site, we checked the server access logs. Within days, we saw GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, OAI-SearchBot and xAI-SearchBot all crawling the site.
One honest caveat: crawled ≠ cited. Server logs confirm that AI crawlers arrived (the "door is open" signal), but whether they cite your content in answers requires spot-checking each AI assistant — a separate, ongoing observation.
Quick reference
| Layer | # | Item | Effect |
|---|---|---|---|
| Crawl | 1 | robots.txt whitelist | Let AI crawlers in |
| Crawl | 2 | sitemap | Discover all pages |
| Crawl | 3 | Static/SSR rendering | Content not hidden in JS |
| Understand | 4 | JSON-LD | Identify who you are |
| Understand | 5 | BLUF + semantic HTML | Clear hierarchy |
| Quote | 6 | FAQPage schema | Feed answers to AI |
| Quote | 7 | llms.txt | AI site guide |
We are AI Enable Harness, an R&D-driven software engineering team specialized in GEO/SEO optimization, AI lead generation systems, AI Token platforms, AI workflows and enterprise websites — aigcharness.com. Questions or comments? Drop them below.
Top comments (0)