Answer Engine Optimization for Developers: What I Check First

Nitish Kumar Yadav — Fri, 14 Aug 2026 19:06:04 +0000

Most AEO advice starts with content. When I review a site, I start lower in the stack.

What does the crawler receive? Is the useful information in the HTML? Can a machine tell which company and product the page is about? Only after those checks do I look at how the answer is written.

I work on FixAEO, so I spend a lot of time thinking about why one brand appears in an AI answer while another does not. There is no single switch that controls this. In practice, the problems are usually spread across access, page structure, entity data, content, and measurement.

This is the order I use when checking them.

The short version: Make the right pages easy to fetch, put the important information in reliable HTML, describe your entities accurately, answer specific questions, and track mentions separately from citations. Add llms.txt if it helps agents navigate your site, but do not treat it as a shortcut.

Last reviewed: August 15, 2026.

First, define what AEO means here

Answer Engine Optimization (AEO) is the work of making a brand or website easier for answer systems to find, understand, and cite.

It overlaps heavily with technical SEO. The difference is the final output. A search engine usually returns a ranked list of pages. An answer engine may read several pages, write its own response, and show only a few sources.

For a developer, the rough path looks like this:

request -> crawler or user-triggered fetch -> retrieval index
        -> candidate selection -> generated answer -> mention or citation

You cannot force the last two stages. You can make the inputs less ambiguous.

My review order is:

crawler access
rendered HTML
entity and product data
answer-friendly content
repeatable measurement

1. Check the response before changing robots.txt

People often open robots.txt, see an Allow, and assume the crawler can reach the page. That is only one layer.

A CDN, Web Application Firewall, rate limiter, or bot-management rule can still return a challenge page or a 403. I check the actual response and the server logs before changing the file.

I also keep search crawlers and training crawlers separate. OpenAI documents OAI-SearchBot for ChatGPT search and lists GPTBot separately for potential model training. Perplexity documents PerplexityBot for its search index and Perplexity-User for user-requested fetches. Anthropic also publishes separate crawler guidance.

That lets a site owner make a more precise choice than blocking every AI-related user agent.

For example:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

This is a policy example, not a recommended default. The right rules depend on how you want your content used. User-agent names and provider policies can also change, so check the current documentation before deploying anything:

One practical test is to request the page with the intended user agent and inspect the status, headers, and body. This does not prove that the provider will index the page, but it will expose obvious blocking and challenge-page problems.

2. Look at the HTML a crawler actually gets

The next question is simple: if JavaScript does not run, is the useful information still there?

Server-side rendering is not mandatory for every site. It does remove a common source of failure, especially when product details, pricing, documentation, or comparison data load through several client-side requests.

I look for these problems:

the product description appears only after hydration
pricing is inserted after a click
documentation requires a logged-in session
the visible headings are styled <div> elements
the canonical URL points to another page
a noindex directive is present by mistake
the server returns different content to a non-browser client

Fetch the page without running JavaScript and read the response. At minimum, I want to see the title, main explanation, headings, canonical URL, update date, and the links that establish context.

If those pieces are missing, I would fix that before rewriting paragraphs for AI search.

3. Add JSON-LD that matches the visible page

Structured data is useful because it gives machines explicit entity information. It is not a citation guarantee.

For a SaaS homepage, I normally begin with Organization and SoftwareApplication. A small graph is enough if it is accurate:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://example.com/#organization",
      "name": "Example",
      "url": "https://example.com/",
      "logo": "https://example.com/logo.png",
      "sameAs": [
        "https://www.linkedin.com/company/example",
        "https://github.com/example"
      ]
    },
    {
      "@type": "SoftwareApplication",
      "name": "Example App",
      "url": "https://example.com/",
      "applicationCategory": "BusinessApplication",
      "operatingSystem": "Web",
      "publisher": {
        "@id": "https://example.com/#organization"
      },
      "offers": {
        "@type": "Offer",
        "price": "0",
        "priceCurrency": "USD"
      }
    }
  ]
}
</script>

The important part is consistency. The name, URL, logo, price, and social profiles should agree with what a person sees on the page. Do not add ratings, reviews, prices, or affiliations that are not visible and true.

I validate the result with the Schema.org validator and Google's Rich Results Test. Google also documents the supported properties for Organization and SoftwareApplication.

4. Treat llms.txt as a map, not a ranking file

I would not make llms.txt the first AEO task on a backlog.

The file is an emerging proposal for giving agents a short Markdown map of useful resources. It can be handy for a documentation-heavy product, but it is not an IETF standard and does not guarantee that commercial answer engines will use it.

A basic version might look like this:

# Example

> Example monitors distributed jobs and alerts teams when a run fails.

## Documentation

- [Quick start](https://example.com/docs/quick-start.md): Run the first job.
- [API reference](https://example.com/docs/api.md): Endpoints and authentication.

## Product

- [Pricing](https://example.com/pricing): Current plans and limits.
- [Security](https://example.com/security): Data handling and compliance.

Keep it short and point it at pages you maintain. It does not replace robots.txt, sitemap.xml, canonical tags, semantic HTML, or normal internal links. The proposal is documented at llmstxt.org.

5. Write sections that can stand on their own

AI retrieval systems often work with passages. That changes how I edit a page.

I ask whether a useful section still makes sense when it is lifted out of the surrounding article. If the answer depends on three earlier paragraphs, the passage is harder to reuse accurately.

The formats that work well are not exotic:

a direct definition below a question heading
a comparison table with named criteria
numbered steps with a clear outcome
examples that show an input and an output
a claim followed by its source
version numbers, prerequisites, and known failure cases

Specific wording helps. “A modern platform for intelligent growth” tells a reader almost nothing. “A tool that checks brand mentions in five answer engines and exports the cited URLs” describes an entity and a job.

This is also why I avoid filling a page with synonyms for the same idea. One clear term, used consistently, is easier for both people and machines to follow.

6. Measure mentions and citations separately

A single “AI visibility” score can hide what actually changed.

I keep at least four signals separate:

Signal	What it tells you
Mention rate	How often the answer names the brand
Citation rate	How often the answer links to the brand's pages
Sentiment	Whether the answer recommends, describes, or criticizes the brand
Share of voice	How often the brand appears compared with competitors

Use a fixed prompt set, run it on a schedule, and keep the raw answers. AI output varies, so one good response is not a trend.

For example, 20 prompts across 5 engines, checked weekly for 4 weeks, produce 400 observations. That is enough to see direction while still acknowledging that the output is non-deterministic.

Referral traffic belongs in the same review, but it answers a different question. A citation can influence brand consideration even when nobody clicks it.

The checklist I use

[ ] Important pages return 200 without authentication
[ ] Canonical URLs point to the intended originals
[ ] Search-oriented AI crawlers are not blocked by mistake
[ ] CDN and firewall rules agree with robots.txt
[ ] Primary information is present in reliable HTML
[ ] Each page has one descriptive H1 and a logical heading order
[ ] Organization data matches the visible company details
[ ] Product pages use accurate SoftwareApplication or Product data
[ ] JSON-LD validates without critical errors
[ ] Sitemap dates represent real updates
[ ] Optional llms.txt links to maintained pages
[ ] Important claims have named evidence
[ ] Mentions and citations are reported separately
[ ] Raw answers are saved for comparison

Three questions I hear often

Is AEO different from SEO?

Yes, but the two share the same foundation. SEO helps a page get crawled, indexed, and ranked. AEO adds work that helps answer systems extract a useful passage, connect it to the correct entity, and cite it. Skipping technical SEO usually creates AEO problems too.

Will llms.txt improve visibility on its own?

No. Use it as a navigation aid if it suits your site. Fix access, HTML, canonical tags, structured data, and internal links first.

Does schema guarantee a citation?

No. Accurate JSON-LD can remove ambiguity, but the answer engine still decides what to retrieve and cite. Good schema supports accessible pages and well-sourced content; it does not replace them.

Where I would start

If a site is missing from AI answers, I would not begin with a large content rewrite. I would fetch the important pages, check the response codes and HTML, compare the visible information with the JSON-LD, and verify the crawler policy.

Once that foundation is clean, improve the passages and start measuring a fixed prompt set. That order makes the investigation easier because each change has a clear reason.

If you want a starting point, FixAEO has a free AEO audit. I also wrote a longer overview of the topic here: What is AEO?.

How AI engines actually decide what to cite (ChatGPT, Perplexity, Gemini, AI Overviews)

Nitish Kumar Yadav — Sun, 21 Jun 2026 05:19:39 +0000

Everyone keeps asking "is SEO dead." Wrong question.

AI search doesn't show ten blue links. It generates one answer and names a few brands. If you're not in that answer, you don't exist for that query. So the real question is: how do these engines decide who to name?

I went down a rabbit hole on how four of them actually retrieve and cite sources. Here's what's true in 2026, with real numbers.

ChatGPT: being known beats ranking

ChatGPT answers in two modes. Default mode answers from trained-in memory, no live web. Search mode browses and attaches citations. The key fact: when it browses, it cites only about 15% of the pages it pulls (AirOps study of 548k pages). And it names brands roughly 3x more often than it links them.

So two things get you in:

Entity strength. If you're a consistent entity across Wikipedia, Wikidata, Reddit and press, ChatGPT names you from memory without browsing at all. Being a known entity beats ranking #1 anywhere.
Allow OAI-SearchBot in robots.txt. It's separate from GPTBot (training). Block it and you vanish from ChatGPT Search. A lot of sites do this by accident.

Perplexity: it's mostly Reddit

Perplexity does live retrieval and grounds every answer in sources. Its defining trait: it leans on community content hard. One 2025 study found Reddit was its most-cited source, ~47% of top citations. It also rewards answer-first pages, because its reranker scores for how cleanly it can extract a passage. A page can rank #1 on Google and never get cited here if the answer is buried.

Gemini: it's basically Google + the Knowledge Graph

Gemini is the only major assistant running on Google's own live index plus the Knowledge Graph. So classical SEO is the floor, not optional. The twist: ranking #1 isn't enough anymore. Only about 38% of Google's AI Overview citations come from the top 10 results, down from ~76% a year earlier. It pulls from deeper now, via sub-queries.

Google AI Overviews: authority over freshness

AI Overviews uses "query fan-out" - it splits your question into 8-12 sub-queries and pools the results. Most citations come from below position #1 (roughly 63% from below the top 10). And counterintuitively, it has the weakest freshness bias of the major engines. Established, authoritative pages keep getting cited even without recent updates, which is the opposite of ChatGPT and Perplexity.

What this means if you're building something

Lead every page with the answer in the first few lines. Most AI citations come from the top of the page.
Be a real entity (Wikipedia, Wikidata, Crunchbase, consistent name everywhere).
Let the AI crawlers in. Check robots.txt for OAI-SearchBot, PerplexityBot, Google-Extended.
Show up off your own site - Reddit and YouTube get cited constantly.
Track over time, not off one screenshot. These answers are non-deterministic; the same prompt gives different brands run to run.

I got tired of checking this by hand, so I built FixAEO - a free tool to see how AI engines describe and recommend your brand across 8 engines, plus a free llms.txt validator. Sharing in case it saves you the manual prompting.

What have you noticed about getting cited by AI? Curious if others are seeing the same patterns.

DEV Community: Nitish Kumar Yadav