Search is splitting in two. Classic Google still matters, but a growing share of discovery now happens inside ChatGPT, Perplexity, Claude, and Google's AI Overviews. Those engines don't rank ten blue links — they synthesize an answer and cite a handful of sources. Getting cited is the new ranking. The practice of engineering for it is called Generative Engine Optimization (GEO), and this post documents the exact setup running on my site.
1. Let the AI crawlers in
Every AI engine has its own crawler, and many sites block them by accident with a blanket rule. Your robots.txt should explicitly allow the bots you want: GPTBot and OAI-SearchBot (OpenAI), PerplexityBot, ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Gemini training), Applebot-Extended (Apple Intelligence), and CCBot (Common Crawl, which feeds many models).
User-Agent: GPTBot
User-Agent: OAI-SearchBot
Allow: /
User-Agent: PerplexityBot
Allow: /
User-Agent: ClaudeBot
Allow: /
Sitemap: https://haiderfarooq.dev/sitemap.xml
2. Ship an llms.txt
llms.txt is an emerging convention: a plain-markdown file at your site root that gives language models a compressed, structured summary of who you are and what your pages contain. Think of it as a sitemap for meaning rather than URLs. Keep it under a few hundred lines, lead with a one-sentence identity statement, and list your key pages with one-line summaries each. A fuller llms-full.txt can carry the long version.
3. Build an entity graph with JSON-LD
AI engines resolve entities, not keywords. Schema.org JSON-LD is how you declare them unambiguously. The minimum viable graph for a personal site: a Person node with @id, sameAs links to GitHub/LinkedIn/X, knowsAbout, and hasOccupation; a WebSite node; and CreativeWork or Article nodes for project pages. Crucially, use stable @id URIs and reference them across pages so crawlers can stitch the graph together.
{
"@type": "Person",
"@id": "https://haiderfarooq.dev/#person",
"name": "Haider Farooq",
"jobTitle": "AI Engineer & Data Scientist",
"sameAs": [
"https://github.com/haiderfarooq3",
"https://www.linkedin.com/in/haiderfarooqdev/"
]
}
4. Write outcome-first copy
LLMs extract and repeat concrete numbers. "$12.1M in client revenue", "141 districts", "5,500+ patients" — figures like these survive summarization; adjectives don't. Put at least one hard number in the first 150 words of every page. Write bios in third person ("Haider Farooq is an AI engineer…") because that's the frame AI answers use when citing you.
5. Freshness and internal linking still matter
- Update pages on a real cadence — stale
lastmoddates in your sitemap depress recrawl priority. - Interlink related pages with descriptive anchor text; AI crawlers follow links to build context just like Googlebot does.
- Use IndexNow to ping Bing (which feeds ChatGPT search and Copilot) the moment content changes.
- Keep canonical URLs consistent — one host, one protocol, no trailing-slash ambiguity.
The checklist
-
robots.txtallows GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, CCBot -
llms.txt+llms-full.txtat site root - JSON-LD Person/WebSite/Article graph with stable
@idURIs - One concrete metric in the first 150 words of every page
- Sitemap with honest
lastmoddates, submitted to Google Search Console and Bing - IndexNow key installed and pinged on every deploy
None of this replaces good content — it makes good content legible to machines.
Originally published at haiderfarooq.dev.
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.