DEV Community

TCO Audit
TCO Audit

Posted on Originally published at tcoaudit.com

Auditing Your Website for AI Search Engines in 2026: TTFB, JSON-LD, robots.txt, and /llms.txt

When a potential buyer asks ChatGPT Search, Perplexity, or Claude to compare B2B software in your category, an automated crawler has roughly 800 milliseconds to fetch your page, strip the HTML, parse your pricing, and feed the context window before synthesis begins.

If your server stalls, or your security headers panic at an unfamiliar user-agent, the model simply moves on. Your competitor gets cited instead.

Most engineering and marketing teams unknowingly break AI search visibility across four technical layers. Here is what goes wrong and how to fix each one.

1. Origin TTFB > 500ms (The Live RAG Timeout)

Traditional Googlebot queues crawls asynchronously; if your origin takes 1.2 seconds to respond, Google still indexes the page eventually.

LLM answer engines do not work that way. During live search, the bot queries the web in real time while a human waits for a streaming answer. Retrieval pipelines operate on strict 600ms–1,000ms socket timeouts. If your origin sits on slow shared hosting or an un-cached PHP stack with high Time-to-First-Byte (TTFB), the crawler aborts the connection.

Fix: Put static pages and documentation behind an edge CDN (Cloudflare / Fastly) or upgrade origin hosting to NVMe-backed dedicated instances with sub-100ms TTFB.

2. Accidental robots.txt Blanket Blocks

Security plugins (Wordfence, Sucuri, default WAF rule sets) often group all new bot tokens under scrapers.

Check your live robots.txt right now. If your file includes generic Disallow: / directives or fails to explicitly grant permissions to:

  • GPTBot (OpenAI live retrieval)
  • ClaudeBot & Anthropic-AI (Anthropic citation fetches)
  • PerplexityBot (Perplexity real-time indexing)

...you have instructed answer engines to ignore your domain.

Fix: Explicitly define crawl access for legitimate AI user-agents in robots.txt, while keeping proprietary data directories disallow-flagged.

3. Missing JSON-LD SoftwareApplication Schemas

LLMs parse unstructured HTML through automated Markdown converters. When your pricing page relies on complex nested React divs, dynamic currency toggles, or CSS table hacks, the extractor often produces garbled text.

The model cannot determine whether your $49 plan includes 5 users or 50 users. Rather than risk a hallucination, it drops your product from the comparison table.

Fix: Embed clean, server-rendered JSON-LD structured data in your <head> using schema.org types (SoftwareApplication, Product, or FAQPage). Explicitly declare operatingSystem, applicationCategory, and offers.priceCurrency. Models read JSON-LD directly without guesswork.

4. No /llms.txt Root Manifest

The /llms.txt proposal is a lightweight standard (similar to sitemap.xml) providing a clean Markdown index of your core documentation, pricing models, and system capabilities at https://yourdomain.com/llms.txt.

Instead of forcing a crawler to scrape 50 bloated client-side JavaScript pages, /llms.txt delivers a concise, plain-text digest that fits directly into an LLM context window.

Fix: Generate a flat /llms.txt file at your root directory linking to raw Markdown documentation and structured pricing overviews.

Grade Your Live URL in Real Time

We built a free, zero-cookie edge inspector using Cloudflare streaming workers to measure all four factors in a single test:

Top comments (0)