Answer Engine Optimization for Developers: What I Check First

Most AEO advice starts with content. When I review a site, I start lower in the stack.

What does the crawler receive? Is the useful information in the HTML? Can a machine tell which company and product the page is about? Only after those checks do I look at how the answer is written.

I work on FixAEO, so I spend a lot of time thinking about why one brand appears in an AI answer while another does not. There is no single switch that controls this. In practice, the problems are usually spread across access, page structure, entity data, content, and measurement.

This is the order I use when checking them.

The short version: Make the right pages easy to fetch, put the important information in reliable HTML, describe your entities accurately, answer specific questions, and track mentions separately from citations. Add llms.txt if it helps agents navigate your site, but do not treat it as a shortcut.

Last reviewed: August 15, 2026.

First, define what AEO means here

Answer Engine Optimization (AEO) is the work of making a brand or website easier for answer systems to find, understand, and cite.

It overlaps heavily with technical SEO. The difference is the final output. A search engine usually returns a ranked list of pages. An answer engine may read several pages, write its own response, and show only a few sources.

For a developer, the rough path looks like this:

request -> crawler or user-triggered fetch -> retrieval index
        -> candidate selection -> generated answer -> mention or citation

You cannot force the last two stages. You can make the inputs less ambiguous.

My review order is:

1. crawler access
2. rendered HTML
3. entity and product data
4. answer-friendly content
5. repeatable measurement

1. Check the response before changing robots.txt

People often open robots.txt, see an Allow, and assume the crawler can reach the page. That is only one layer.

A CDN, Web Application Firewall, rate limiter, or bot-management rule can still return a challenge page or a 403. I check the actual response and the server logs before changing the file.

I also keep search crawlers and training crawlers separate. OpenAI documents OAI-SearchBot for ChatGPT search and lists GPTBot separately for potential model training. Perplexity documents PerplexityBot for its search index and Perplexity-User for user-requested fetches. Anthropic also publishes separate crawler guidance.

That lets a site owner make a more precise choice than blocking every AI-related user agent.

For example:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

This is a policy example, not a recommended default. The right rules depend on how you want your content used. User-agent names and provider policies can also change, so check the current documentation before deploying anything:

• OpenAI publisher and developer guidance
• Anthropic crawler guidance
• Perplexity crawler guidance

One practical test is to request the page with the intended user agent and inspect the status, headers, and body. This does not prove that the provider will index the page, but it will expose obvious blocking and challenge-page problems.

2. Look at the HTML a crawler actually gets

The next question is simple: if JavaScript does not run, is the useful information still there?

Server-side rendering is not mandatory for every site. It does remove a common source of failure, especially when product details, pricing, documentation, or comparison data load through several client-side requests.

I look for these problems:

• the product description appears only after hydration
• pricing is inserted after a click
• documentation requires a logged-in session
• the visible headings are styled <div> elements
• the canonical URL points to another page
• a noindex directive is present by mistake
• the server returns different content to a non-browser client

Fetch the page without running JavaScript and read the response. At minimum, I want to see the title, main explanation, headings, canonical URL, update date, and the links that establish context.

If those pieces are missing, I would fix that before rewriting paragraphs for AI search.

3. Add JSON-LD that matches the visible page

Structured data is useful because it gives machines explicit entity information. It is not a citation guarantee.

For a SaaS homepage, I normally begin with Organization and SoftwareApplication. A small graph is enough if it is accurate:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://example.com/#organization",
      "name": "Example",
      "url": "https://example.com/",
      "logo": "https://example.com/logo.png",
      "sameAs": [
        "https://www.linkedin.com/company/example",
        "https://github.com/example"
      ]
    },
    {
      "@type": "SoftwareApplication",
      "name": "Example App",
      "url": "https://example.com/",
      "applicationCategory": "BusinessApplication",
      "operatingSystem": "Web",
      "publisher": {
        "@id": "https://example.com/#organization"
      },
      "offers": {
        "@type": "Offer",
        "price": "0",
        "priceCurrency": "USD"
      }
    }
  ]
}
</script>

The important part is consistency. The name, URL, logo, price, and social profiles should agree with what a person sees on the page. Do not add ratings, reviews, prices, or affiliations that are not visible and true.

I validate the result with the Schema.org validator and Google's Rich Results Test. Google also documents the supported properties for Organization and SoftwareApplication.

4. Treat llms.txt as a map, not a ranking file

I would not make llms.txt the first AEO task on a backlog.

The file is an emerging proposal for giving agents a short Markdown map of useful resources. It can be handy for a documentation-heavy product, but it is not an IETF standard and does not guarantee that commercial answer engines will use it.

A basic version might look like this:

# Example

> Example monitors distributed jobs and alerts teams when a run fails.

## Documentation

- [Quick start](https://example.com/docs/quick-start.md): Run the first job.
- [API reference](https://example.com/docs/api.md): Endpoints and authentication.

## Product

- [Pricing](https://example.com/pricing): Current plans and limits.
- [Security](https://example.com/security): Data handling and compliance.

Keep it short and point it at pages you maintain. It does not replace robots.txt, sitemap.xml, canonical tags, semantic HTML, or normal internal links. The proposal is documented at llmstxt.org.

5. Write sections that can stand on their own

AI retrieval systems often work with passages. That changes how I edit a page.

I ask whether a useful section still makes sense when it is lifted out of the surrounding article. If the answer depends on three earlier paragraphs, the passage is harder to reuse accurately.

The formats that work well are not exotic:

• a direct definition below a question heading
• a comparison table with named criteria
• numbered steps with a clear outcome
• examples that show an input and an output
• a claim followed by its source
• version numbers, prerequisites, and known failure cases

Specific wording helps. “A modern platform for intelligent growth” tells a reader almost nothing. “A tool that checks brand mentions in five answer engines and exports the cited URLs” describes an entity and a job.

This is also why I avoid filling a page with synonyms for the same idea. One clear term, used consistently, is easier for both people and machines to follow.

6. Measure mentions and citations separately

A single “AI visibility” score can hide what actually changed.

I keep at least four signals separate:

Signal	What it tells you
Mention rate	How often the answer names the brand
Citation rate	How often the answer links to the brand's pages
Sentiment	Whether the answer recommends, describes, or criticizes the brand
Share of voice	How often the brand appears compared with competitors

Use a fixed prompt set, run it on a schedule, and keep the raw answers. AI output varies, so one good response is not a trend.

For example, 20 prompts across 5 engines, checked weekly for 4 weeks, produce 400 observations. That is enough to see direction while still acknowledging that the output is non-deterministic.

Referral traffic belongs in the same review, but it answers a different question. A citation can influence brand consideration even when nobody clicks it.

The checklist I use

• [ ] Important pages return 200 without authentication
• [ ] Canonical URLs point to the intended originals
• [ ] Search-oriented AI crawlers are not blocked by mistake
• [ ] CDN and firewall rules agree with robots.txt
• [ ] Primary information is present in reliable HTML
• [ ] Each page has one descriptive H1 and a logical heading order
• [ ] Organization data matches the visible company details
• [ ] Product pages use accurate SoftwareApplication or Product data
• [ ] JSON-LD validates without critical errors
• [ ] Sitemap dates represent real updates
• [ ] Optional llms.txt links to maintained pages
• [ ] Important claims have named evidence
• [ ] Mentions and citations are reported separately
• [ ] Raw answers are saved for comparison

Three questions I hear often

Is AEO different from SEO?

Yes, but the two share the same foundation. SEO helps a page get crawled, indexed, and ranked. AEO adds work that helps answer systems extract a useful passage, connect it to the correct entity, and cite it. Skipping technical SEO usually creates AEO problems too.

Will llms.txt improve visibility on its own?

No. Use it as a navigation aid if it suits your site. Fix access, HTML, canonical tags, structured data, and internal links first.

Does schema guarantee a citation?

No. Accurate JSON-LD can remove ambiguity, but the answer engine still decides what to retrieve and cite. Good schema supports accessible pages and well-sourced content; it does not replace them.

Where I would start

If a site is missing from AI answers, I would not begin with a large content rewrite. I would fetch the important pages, check the response codes and HTML, compare the visible information with the JSON-LD, and verify the crawler policy.

Once that foundation is clean, improve the passages and start measuring a fixed prompt set. That order makes the investigation easier because each change has a clear reason.

If you want a starting point, FixAEO has a free AEO audit. I also wrote a longer overview of the topic here: What is AEO?.