DEV Community: Amit Kumar

Why AI Engines Ignore Your Content (Even When They Can Crawl It)

Amit Kumar — Sun, 26 Apr 2026 12:55:53 +0000

Fixing your robots.txt and disabling Cloudflare Bot Fight Mode is step one.

Most developers stop there and wonder why they still don't appear in ChatGPT, Gemini, Claude or Perplexity answers.

Crawlability is access. Citations are trust. They're two different problems.

I run LearnQ.ai and VEGA AI. After fixing the crawlability issues I wrote about in my previous article, I still wasn't getting cited consistently.

The bots could reach the content. They just weren't using it. Here's what was actually wrong.

The Real Reason AI Engines Skip Your Content

Your answers are buried inside paragraphs

Google trained us to write flowing prose with the answer somewhere in the middle. AI engines work differently.

They extract the most direct, self-contained answer to a query and surface it.

If your answer is buried three sentences into a paragraph, the engine either misses it or picks a competitor who answered more directly.

The fix is structural. Every H2 and H3 section should open with a direct answer in the first sentence. Supporting detail follows. Not the other way around.

Before:

Schema markup is a type of structured data that you add to your HTML.
It has been used by Google for years and is now becoming important for
AI search as well. Adding it can help AI engines understand your content.

After:

Schema markup helps AI engines understand what your content is about.
Add it to your HTML using JSON-LD format. This is the single most
impactful technical change you can make for AI search visibility.

The second version is extractable. The first one isn't.

Your FAQ section doesn't exist or isn't self-contained

FAQ sections are the highest-leverage content format for AI citations.
AI engines frequently pull FAQ answers verbatim because they are already
structured as question-answer pairs.

Two rules for FAQ content that actually gets cited:

Each answer must be self-contained. It should make complete sense
without the reader having seen any other part of the article. If your
FAQ answer says "as mentioned above," it will not get extracted cleanly.

Use the exact question phrasing your audience types. Not "What is GEO?" but "What is generative engine optimization and how is it different from SEO?"

The closer your question matches the actual query, the higher the extraction probability.

You have no FAQPage schema

Writing FAQ content is not enough. You need to tell AI engines it's a FAQ section using structured data. FAQPage schema in JSON-LD format does this.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is generative engine optimization?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Generative engine optimization (GEO) is the practice of 
        optimizing your content so it gets cited and recommended by AI 
        search engines like ChatGPT, Perplexity, and Google AI Overviews. 
        It differs from traditional SEO in that the goal is citation and 
        direct answer extraction, not ranking position."
      }
    }
  ]
}
</script>

Add this to every page that has FAQ content. On WordPress, WPCode makes this straightforward without touching theme files.

Your paragraphs are too long

AI engines have a strong preference for short, dense, information-rich paragraphs. Three sentences maximum per paragraph is the practical rule.

Long flowing text gets skipped in favour of content that is easier to
extract and attribute.

Audit your existing content. Any paragraph over three sentences is a
candidate for splitting.

You have no structured data identifying your entity

AI engines build a model of what your website is about based on structured data, not just content.

Without Organization schema on your homepage and Article or BlogPosting schema on your articles, you are an anonymous
source. Anonymous sources don't get cited.

Minimum schema set for AI search visibility:

Organization on homepage: name, URL, logo, description, sameAs (your social profiles)
Article or BlogPosting on every article: headline, author, datePublished, dateModified, publisher
FAQPage on every page with Q&A content
Person on author pages: name, jobTitle, url, sameAs

A Quick Content Audit Process

Run this on your five most important pages before doing anything else:

Open the page and read only the first sentence of each H2/H3 section. Does each one answer the section question directly? If not, rewrite the opening sentence.
Check whether a FAQ section exists. If not, add one with at least
five self-contained questions relevant to the page topic.
Validate your schema at schema.org/validator. Fix any errors shown.
Count sentences per paragraph. Split any paragraph over three sentences.
Disable JavaScript in your browser and reload. Confirm all content
is still visible in raw HTML.

This process takes about 90 minutes per page. The structural changes
compound quickly because AI engines re-crawl and update their citation pool regularly.

What Changed When I Applied This

After restructuring the key pages on LearnQ.ai — rewriting section openings to lead with direct answers, adding FAQ schema, and reducing paragraph length - citation frequency in Perplexity responses increased within two to three weeks.

The crawlability fix gets you in the door. The content structure fix gets you cited.

Full guide on AI-readable content structure:
proaisearch.com/robots-txt-ai-crawlers/

How to Set Up llms.txt and robots.txt for AI Crawlers on WordPress (2026 Guide)

Amit Kumar — Sun, 05 Apr 2026 14:32:20 +0000

If AI crawlers cannot reach your WordPress site, your content will never appear in ChatGPT or Perplexity answers. Here is the exact setup, file by file.

I manage WordPress sites for EdTech brands in India. A few months back I noticed something that made no sense: pages ranking well on Google were getting zero citations on Perplexity. Same content, same domain, completely invisible to AI search.

The problem was not the content. It was three technical things I had never thought to check.

Here is exactly what I fixed, with the actual code.

## Step 1: Fix your robots.txt for AI crawlers

Most WordPress robots.txt files either block AI crawlers explicitly or do not mention them at all. Neither is ideal.

Open yourdomain.com/robots.txt in your browser and check what is there. If you see any of these, AI crawlers are blocked:

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

To fix this, you need to edit your robots.txt. On WordPress, the easiest way is using WPCode:

Go to WPCode in your WordPress dashboard
Click Code Snippets, then Add Snippet
Choose Text snippet type
Set the insertion location to "robots.txt"
Paste this:

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: GoogleOther-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

Save and verify at yourdomain.com/robots.txt.

Alternatively use the Yoast SEO or Rank Math robots.txt editor under their Tools section if you have either plugin installed.

Step 2: Turn off Cloudflare Bot Fight Mode

This one is the most impactful fix and the least obvious. Cloudflare's Bot Fight Mode is enabled by default on all plans.

It blocks automated traffic, which includes legitimate AI crawlers like PerplexityBot and ClaudeBot.

They hit your site, get blocked at the Cloudflare layer, and your server never even sees the request.

I was running this for months without realising it was silently blocking AI citations for our VEGA AI brand.

To fix it:

Log into your Cloudflare dashboard
Select your domain
Go to Security, then Bots
Turn Bot Fight Mode OFF

If you are on a paid Cloudflare plan using Super Bot Fight Mode:

Go to Security, then Bots
Find "Definitely automated"
Change it from Block to Allow

Verify it worked by checking your server access logs a few days later. You should start seeing GPTBot and PerplexityBot in the logs.

Step 3: Create your llms.txt file

llms.txt is a plain text file that tells large language models what your site is about and which pages matter most. Think of it as a sitemap specifically for AI models.

Create a file called llms.txt and place it in your WordPress root directory. The easiest method:

Go to your hosting control panel, open File Manager
Navigate to public_html (or your WordPress root)
Create a new file called llms.txt
Paste your content and save

Here is a template you can adapt:

# Your Site Name

> One sentence describing what your site covers and who it is for.

## Key Pages

- [Page Title](https://yourdomain.com/page-slug/): What this page covers in one sentence.
- [Page Title](https://yourdomain.com/page-slug/): What this page covers in one sentence.
- [Page Title](https://yourdomain.com/page-slug/): What this page covers in one sentence.

## About

Your name, your role, and your expertise in two sentences.

## Contact

your@email.com

Keep it simple. You do not need to list every page, just the most important ones per topic cluster. Verify it is live at yourdomain.com/llms.txt.

Step 4: Add FAQPage schema markup

AI models prioritize structured question and answer content. FAQPage schema tells them exactly where to find it.

In WordPress, add this via WPCode on any page that has FAQ content:

Go to WPCode, Add Snippet
Choose JavaScript snippet type
Set insertion to "Site Wide Footer"
Paste and adapt this for each page (or use a separate snippet per page):

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "Your question here?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Your answer here. Keep it direct and complete."
      }
    },
    {
      "@type": "Question",
      "name": "Your second question here?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Your second answer here."
      }
    }
  ]
}
</script>

If you use Rank Math, you can add FAQ schema directly in the post editor under the Schema tab without touching code.

Select FAQPage as the schema type and fill in your questions there.

Step 5: Add Article schema with author details

AI models weight authorship when deciding what to cite. Anonymous content gets cited less. Add this to your key pages:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Your Page Title",
  "author": {
    "@type": "Person",
    "name": "Your Name",
    "jobTitle": "Your Job Title",
    "url": "https://yourdomain.com/about/"
  },
  "publisher": {
    "@type": "Organization",
    "name": "Your Site Name",
    "url": "https://yourdomain.com"
  }
}
</script>

Again, Rank Math handles this automatically if you set the rich snippet type to Article and fill in the author details in your profile settings.

Verify everything is working

Once all four steps are done:

Test robots.txt: visit yourdomain.com/robots.txt and confirm AI crawler directives are present
Test llms.txt: visit yourdomain.com/llms.txt and confirm it loads
Test schema: paste your URL into Google's Rich Results Test (search.google.com/test/rich-results) and confirm FAQPage and Article schema show clean
Check Cloudflare: confirm Bot Fight Mode is off in your dashboard

Results are not immediate. Give it four to six weeks and check whether your pages start appearing in Perplexity answers for your target topics.

Search your main keywords directly in Perplexity and see if you get cited.

I documented the full setup for each of these in more detail at proaisearch.com/llm-seo/ if you want deeper implementation notes.

For a broader guide on AI search optimization beyond just the technical setup, Pro AI Search covers GEO, AEO, and LLM SEO in one place.