DEV Community: Daniil Lutsiuk

Yoast and All in One SEO store a GPTBot switch their free code never reads

Daniil Lutsiuk — Mon, 21 Sep 2026 14:29:30 +0000

Of five free WordPress SEO plugins checked on September 21, 2026, one writes robots.txt rules that block AI crawlers. Two of the largest, Yoast SEO and All in One SEO, store a GPTBot setting in their free builds, and no code in those builds reads it. Rank Math and SEOPress contain no AI crawler names at all.

Disclosure: the author develops Prime SEO, the fifth plugin in this comparison. Every claim below comes from the free packages downloaded from the WordPress.org plugin directory, searched for crawler names and for the code that acts on each setting; file paths are given so that anyone can repeat the check.

Yoast SEO free stores three switches it never reads

Yoast SEO, active on 10 million sites according to the directory, defines three options in inc/options/class-wpseo-option-wpseo.php: deny_gptbot_crawling, deny_ccbot_crawling and deny_google_extended_crawling. All default to false. The settings screen even carries the label "Prevent OpenAI GPTBot from crawling" under Crawl optimization.

What the free build lacks is a reader. Its robots.txt code, src/integrations/front-end/robots-txt-integration.php, checks three neighbouring options, among them internal search and Google AdsBot, and stops there. In the settings script the GPTBot toggle receives isDummy whenever the site's isPremium preference is false, and the same component loads a link named block-unwanted-bots-upsell. A free user sees the switch as a preview of Premium.

All in One SEO Lite follows the same pattern

All in One SEO, installed on 2 million sites, declares an unwantedBots group in app/Common/Options/Options.php with openAiGptBot and commonCrawlCcBot next to a Google AdsBot entry. Across the Lite package those names appear in two option files and nowhere else. The only other code that touches the group resets or exports it.

The Lite download ships the app/Common and app/Lite folders. No file in either one turns the setting into a robots.txt line, which places the working part in the paid tiers.

Rank Math and SEOPress free have no AI crawler code

A search of Rank Math (4 million active installs) for 14 crawler names, from GPTBot to Amazonbot, returns nothing outside a bundled user agent library. SEOPress returns nothing at all. Rank Math free generates an llms.txt file, and SEOPress keeps that feature for its PRO version.

SEOPress PRO, a paid product, goes further: the 9.5 copy checked for this article offers a robots.txt template with rules for CCBot, GPTBot, Google-Extended and ClaudeBot. None of the four free builds logs visits from AI crawlers. Each package reads the visitor's user agent in at most one file, and no crawler name appears in any of them.

Free version	Checked	Writes AI crawler rules to robots.txt	llms.txt	Logs AI crawler visits
Yoast SEO	28.5	No (setting stored, not read)	Yes	No
Rank Math	1.0.278	No	Yes	No
All in One SEO	5.0.1.1	No (setting stored, not read)	Yes	No
SEOPress	10.2	No	No (PRO)	No
Prime SEO	1.2.0	Yes, 15 crawlers	Yes, plus llms-full.txt	Yes

Three lines of robots.txt need no plugin

Blocking a crawler does not require any of the five. OpenAI's crawler documentation says "GPTBot is used to make our generative AI foundation models more useful and safe" and accepts an ordinary robots.txt rule:

User-agent: GPTBot
Disallow: /

The same two lines work for ClaudeBot or for Google-Extended, the token that, in Google's words, "does not impact a site's inclusion in Google Search." On WordPress the catch is practical. The default robots.txt is virtual, many hosts offer no file access, and a list of 15 names kept by hand goes stale as vendors rename their agents.

Few site owners bother either way. Among the robots.txt files that Cloudflare could read on top domains, GPTBot was the most refused AI agent and was still, João Tomé of Cloudflare wrote in a July 2025 study, "disallowed by 312 domains" only. A rule also binds nobody: agents that fetch a page for a live user, ChatGPT-User and Perplexity-User among them, are documented by their vendors as outside robots.txt, a point covered in the guide to AI crawlers in WordPress logs.

Prime SEO 1.2.0 writes rules for 15 crawlers

Prime SEO puts the feature in its free version. The AI Bots Manager lists 15 crawlers with a switch each, plus three presets (allow all, block training crawlers only, block all). Matching robots.txt lines are appended to the virtual file, and a separate screen logs which AI agents visited and which page they opened.

Its weak side is plain. Prime SEO has 100+ active installs against Yoast's 10 million and a record that starts in January 2026. Two of the names on its list (anthropic-ai and Claude-Web) are ones Anthropic no longer documents. Those two rules block nothing today.

A site that needs one blocked crawler and nothing else is served as well by the three-line rule above (no plugin, no updates to follow). For the rest of the setup, see the llms.txt plugin for WordPress page.

The option keys are already in place

Yoast and All in One SEO would need little work to change this table, because the settings exist and only the reader is missing from their free code. If either one moves the switch out of the paid tier, free AI crawler control stops being a reason to choose a smaller plugin. Until then the comparison stands at one in five.

Originally published on the Prime SEO blog.

3,590 downloads, 100 active installs: what I changed in my WordPress plugin today

Daniil Lutsiuk — Mon, 21 Sep 2026 09:42:05 +0000

Prime SEO, the WordPress plugin I've been building since January, has 3,590 downloads on WordPress.org as of September 21, 2026.

The same API reports 100+ active installs. WordPress.org rounds that figure down, so the real number sits between 100 and 199, and at most one download in eighteen became a site that kept the plugin.

I don't know yet why people leave. Version 1.1.9 added a deactivation survey yesterday (it sends nothing unless the user presses Submit, and never the site address), and the only two answers in it so far are my own tests. What I can look at today is the first-run path, and the hole in it is easy to see.

Ten menu items and no next step

A new install opened a setup wizard with three questions: homepage title, modules, robots.txt. After that the user landed on a dashboard next to ten menu items. The wizard itself ended on a green tick and three links, and nothing on either screen said which of the ten pages mattered first.

All in One SEO 5.0.1.1 handles this with a setup checklist. In its free build the file is app/Common/SeoChecklist/SeoChecklist.php: every step has a priority and a time estimate, a callback decides whether the step is complete, and the names of completed steps are appended to a stored list. I borrowed that shape.

Two kinds of steps

Version 1.2.0 went out today with a checklist on the dashboard. Nothing in it is ticked by hand.

In the All in One SEO code a completed step is stored by name and reopens only in a few coded cases: indexing gets switched off, Search Console is disconnected, the user unticks it by hand. I wanted every step that depends on a setting to follow that setting, because a bar that says 9 of 9 while the sitemap is off tells the user nothing. My steps come in two kinds:

// live: follows a setting. Switch the sitemap off and the step opens again.
'completed' => $module( 'module_sitemap' ),

// sticky: a one-time event. Once done, stays done.
'completed' => (bool) get_option( 'prime_seo_wizard_complete' ),

Live steps cover settings such as the sitemap and the AI bot policy (the $module() helper treats a missing key as on, because modules are enabled by default). Any of them can reopen. Sticky ones cover events, such as a finished wizard or the day one post reached a score of 80. The wizard's last page now shows the same list, with the steps it has just completed on the left.

The step nobody could complete

That bug was mine alone. My first version of the IndexNow step looked for the API key inside the plugin's settings array. That key lives in a separate option, prime_seo_indexnow_key, and is generated on first use. The step stayed open forever, and I noticed only because my test site sat at 9 of 10 after I had done everything on the list.

Fixing it exposed a second problem. Since 1.1.6 the plugin deliberately sends nothing to IndexNow from local, development and hidden sites, which made the step impossible to finish on exactly those sites. It's now offered to public sites only, so a local install counts nine steps and a live one counts ten (eleven when another SEO plugin left data to import). Both counts are honest now.

If the survey answers come back as "too complicated", the checklist was the right first move. If they say "feature X is missing", I spent a day on the wrong problem, and the next entry will be about X.

I'm the developer of Prime SEO, a free SEO plugin for WordPress. Entry one of a build log; the numbers come from the public WordPress.org API.

AI crawlers in WordPress logs: GPTBot shows up, Google-Extended never does

Daniil Lutsiuk — Mon, 21 Sep 2026 08:10:01 +0000

Google-Extended and Applebot-Extended, two of the names that robots.txt guides list most often for AI, cannot appear in a WordPress access log. Neither one sends a request. Google's crawler documentation states that Google-Extended "doesn't have a separate HTTP request user agent string." The second name gets even less: "Applebot-Extended does not crawl webpages," Apple's support page for Applebot reads, which leaves both names as robots.txt tokens rather than visitors and means that a log of AI crawlers in WordPress has to be built from the agents that send requests.

Disclosure: the author develops Prime SEO, a WordPress plugin that includes such a log. Every user agent below was checked against vendor documentation on September 21, 2026, and the code runs without any plugin.

Three vendors document nine user agents

OpenAI lists four agents on its crawler page. GPTBot gathers training material (the page says it exists "to make our generative AI foundation models more useful and safe"), while OAI-SearchBot builds the index behind ChatGPT search. ChatGPT-User fetches a page when a person asks for it. OAI-AdsBot checks pages submitted as ads.

Anthropic documents three, split the same way: ClaudeBot collects training data, while Claude-User and Claude-SearchBot serve live questions and search. Perplexity runs two.

User agent substring	Operator	Purpose, per the vendor	robots.txt
`GPTBot`	OpenAI	Model training	Honored
`OAI-SearchBot`	OpenAI	ChatGPT search index	Honored
`ChatGPT-User`	OpenAI	Fetch started by a user	"may not apply"
`OAI-AdsBot`	OpenAI	Safety check of pages submitted as ads	Not stated
`ClaudeBot`	Anthropic	Model training	Honored
`Claude-User`	Anthropic	Fetch started by a user	Honored
`Claude-SearchBot`	Anthropic	Search quality	Honored
`PerplexityBot`	Perplexity	Perplexity search results	Honored
`Perplexity-User`	Perplexity	Fetch started by a user	"generally ignores"

Missing from Anthropic's page are anthropic-ai and Claude-Web, two names that still circulate in robots.txt templates. A rule aimed at them blocks nothing the vendor admits to sending.

User-triggered fetchers sit outside robots.txt

Crawlers that build an index obey robots.txt, by the vendors' own account. Anthropic's help page says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt." Fetchers started by a person are a different matter.

OpenAI describes ChatGPT-User as an agent that "is not used for crawling the web in an automatic fashion," and adds that "robots.txt rules may not apply" because a user starts the action. Perplexity's bot guide is blunter about Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

For a site owner, the split has a practical side. A Disallow line can keep a page out of a training set, yet the same page may still be fetched for a ChatGPT user five minutes later. Only a log shows that second kind of visit.

A short mu-plugin records every match

WordPress needs no plugin for a basic log. The file below goes into wp-content/mu-plugins/, hooks into template_redirect at priority 1, and compares the user agent of each request with the nine documented names.

<?php
/**
 * Plugin Name: AI Crawler Log
 * Description: Records visits from documented AI user agents.
 */

add_action( 'template_redirect', function () {
    if ( empty( $_SERVER['HTTP_USER_AGENT'] ) ) {
        return;
    }

    $agent = sanitize_text_field( wp_unslash( $_SERVER['HTTP_USER_AGENT'] ) );
    $names = array(
        'GPTBot', 'OAI-SearchBot', 'ChatGPT-User', 'OAI-AdsBot',
        'ClaudeBot', 'Claude-User', 'Claude-SearchBot',
        'PerplexityBot', 'Perplexity-User',
    );

    foreach ( $names as $name ) {
        if ( false === stripos( $agent, $name ) ) {
            continue;
        }

        $log   = get_option( 'ai_crawler_log', array() );
        $log[] = array(
            'bot'  => $name,
            'url'  => esc_url_raw( wp_unslash( $_SERVER['REQUEST_URI'] ?? '/' ) ),
            'time' => gmdate( 'Y-m-d H:i:s' ),
        );

        // Keep the last 500 visits; "false" keeps the option out of autoload.
        update_option( 'ai_crawler_log', array_slice( $log, -500 ), false );
        break;
    }
}, 1 );

Each match lands in a single option, ai_crawler_log, stored with autoload switched off so that ordinary page loads never carry it. The array_slice() call caps the record at 500 visits. On a local WordPress 7.1 test install, a request sent with the GPTBot agent string appeared in the option within the same second, while a Chrome agent left no trace. Reading the result takes one WP-CLI command: wp option get ai_crawler_log --format=json.

Owners with shell access can skip PHP and count matches in the server log instead:

grep -Eio "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User" access.log | sort | uniq -c | sort -rn

Page caches and spoofed agents distort the count

PHP-based logging has a blind spot. When a full-page cache (WP Rocket or a CDN edge, for instance) answers the request, WordPress never starts and the visit goes unrecorded. The server access log catches those hits; a PHP log on a cached site shows only cache misses.

User agent strings are also trivial to fake. Any script can call itself GPTBot, which is why all three vendors publish their IP ranges as JSON files (openai.com/gptbot.json is one of them). A visit from outside those ranges is an impostor.

The opposite error exists too. Cloudflare reported on August 4, 2025 that Perplexity used, besides its declared agents, "a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked." No list of names catches a crawler that refuses to give one.

Cloudflare counted a 305% rise in GPTBot requests

Traffic from these agents is no longer marginal. GPTBot "increased its share from 2.2% to 7.7%" of crawler requests in the year to May 2025, João Tomé of Cloudflare wrote in a study of crawler traffic published that July. Raw request volume from the same bot, the figure in the heading above, more than quadrupled.

Growth was uneven. In the same dataset ClaudeBot, Anthropic's training crawler, "fell from 11.7% to 5.4% of total traffic." One site's log can look nothing like the global chart, and a quiet month for one vendor says little about the others.

Blocking remains rare among large sites. Of the 3,816 robots.txt files that Cloudflare could read among top domains, 546 addressed AI bots at all. GPTBot led the refusals: the study found it "disallowed by 312 domains."

Prime SEO 1.1.9 matches 14 names, two of them in vain

Prime SEO, the plugin named in the disclosure, ships the same stripos() approach behind a dashboard screen called AI Crawler Stats, which also records requests for llms.txt and the XML sitemap. Version 1.1.9 matches 14 substrings. Two of them are Google-Extended and Applebot-Extended, which, per the documentation quoted above, can never match. Those two counters will stay at zero.

The plugin's list also predates Claude-User, Claude-SearchBot and Perplexity-User, so it undercounts user-triggered visits today; an update is planned for version 1.2.0. Background on the file these agents are meant to read is in the guide to what llms.txt is and on the llms.txt plugin for WordPress page.

Vendor lists change faster than robots.txt files

All three vendors now separate training crawlers from agents that act for a person. If that split continues, a robots.txt file written for GPTBot and ClaudeBot alone will govern a shrinking part of AI traffic, and the raw access log (not the rule file) becomes the place where a new agent name first shows up. Until a vendor page changes, nine names are enough.

Originally published on the Prime SEO blog.

What Is llms.txt? How to Set It Up on WordPress (2026 Guide)

Daniil Lutsiuk — Sun, 20 Sep 2026 18:19:03 +0000

In September 2024, Jeremy Howard — the guy behind fast.ai and a former president of Kaggle — published a short proposal. He called it llms.txt, and the idea was almost annoyingly simple: give AI models a plain-text map of your website so they don't have to guess what's important.

No one paid much attention at first. Then Cloudflare added one. Then Stripe. Then Anthropic, Vercel, Supabase, and Zapier. Today, over 780 websites have published an llms.txt file, and three separate community directories track who's adopted it.

The question isn't really whether llms.txt will matter. It's whether you'll set yours up before or after your competitors do.

What llms.txt actually is (in plain English)

Think of your website from an AI model's perspective for a second.

When ChatGPT or Perplexity needs to answer a question about your product, it sends a crawler to your site. That crawler sees HTML — headers, footers, navigation menus, cookie banners, newsletter popups, sidebar widgets. Buried somewhere in that noise is the content that actually matters.

Your XML sitemap doesn't help much here. It lists every URL on the site equally — the homepage, the privacy policy, that test page from 2019 you forgot to delete. There's no hierarchy, no context, no "start here."

llms.txt solves this by giving AI models exactly what they need: a structured Markdown file at yoursite.com/llms.txt that says "here's what this site is about, and here are the pages that matter most."

That's it. A curated table of contents for machines that read.

The format: simpler than you'd expect

An llms.txt file is just Markdown with a specific structure:

# Your Site Name
> A one-line summary of what your site does.
Some optional context about your project, company, or product.
## Documentation
- [Getting Started](https://yoursite.com/docs/start): Setup guide for new users
- [API Reference](https://yoursite.com/docs/api): Complete API documentation
## Blog
- [Latest Release](https://yoursite.com/blog/v2): What's new in version 2.0

The rules are minimal:

One H1 heading — your site or project name (the only required element)
A blockquote — brief summary with key facts
H2 sections — groups of links, each with an optional description
Markdown links — [Page Name](URL): optional description

That's the entire spec. Jeremy Howard intentionally kept it simple because "websites already have sitemaps, robots.txt, and meta tags. What they don't have is a curated overview written specifically for language models."

llms.txt vs llms-full.txt — yes, there are two

The standard actually defines two files:

llms.txt is the summary — your site's elevator pitch plus links to key pages. Think of it as the table of contents. Most llms.txt files are a few hundred lines at most.

llms-full.txt is the entire book. It contains the full Markdown content of your key pages, concatenated into one massive file. Cloudflare's llms-full.txt is 3.7 million tokens. Anthropic's is 481,000 tokens. Vercel's has been described as "a 400,000-word novel."

Why would you want that? Because an AI model with a large enough context window can ingest your entire documentation in one shot — no crawling, no parsing HTML, no missing pages.

For most WordPress sites, llms.txt alone is enough. But if you run a documentation-heavy site or want AI to have deep access to your content, llms-full.txt is worth considering.

Who's actually using it?

The short answer: mostly developer-facing companies. The long answer is more interesting.

Stripe organizes their llms.txt by product category, with an "Optional" section for niche tools like Stripe Climate. Their file tells AI agents exactly how to retrieve plain-text versions of documentation pages.

Cloudflare structures theirs by product line — Workers, Pages, R2, and so on. Each entry has a description that helps AI models decide which page to fetch based on the user's question.

Anthropic (the company behind Claude) maintains both files. Their llms.txt is 8,364 tokens; the full version is 481,349 tokens covering their entire API documentation.

Vercel went a step further and published a proposal for inline LLM instructions in HTML — extending the llms.txt idea into individual pages.

Other adopters include Supabase, Zapier, Modal, Coinbase, and hundreds of smaller companies tracked across three community directories.

The honest truth: does it actually work?

Here's where most articles about llms.txt get vague. We won't.

The data is mixed. Search Engine Land tracked 10 websites for 180 days — 90 days before adding llms.txt, 90 days after. The results:

2 out of 10 sites saw AI traffic increases (12.5% and 25%)
8 sites saw no measurable change
1 site actually declined by 19.7%

But here's the catch: the two sites that grew had also launched new content and PR campaigns around the same time. The llms.txt file alone didn't drive the increase — it was part of a broader effort.

Adoption is still early. Rankability scanned the top 1,000 most-visited websites globally. As of mid-2025, only 0.3% had an llms.txt file. That's 3 out of 1,000.

No AI provider has officially committed to reading llms.txt. OpenAI, Anthropic, Google — none have confirmed their crawlers consistently follow llms.txt instructions. Google's John Mueller compared the standard to the deprecated keywords meta tag.

So why bother?

Three reasons:

Low effort, no downside. Setting up llms.txt takes 5 minutes. There's zero risk. It won't hurt your rankings, slow your site, or cause conflicts with anything.
The direction is clear. Even if no one reads llms.txt today, AI search is growing at 34% CAGR. Gartner predicts 25% of search volume will shift away from traditional engines by late 2026. When AI providers do formalize how they read websites, llms.txt (or something very similar) will be the standard.
First-mover advantage is real. If only 0.3% of top sites have llms.txt, that means 99.7% of your competitors don't. Early adopters set the baseline that later standards build on.

How llms.txt differs from robots.txt and XML sitemaps

This trips people up, so here's a clear comparison:

	robots.txt	XML Sitemap	llms.txt
Purpose	Controls crawler access	Lists all URLs for indexing	Curated overview for AI understanding
Audience	Search engine crawlers	Search engine crawlers	Language models (LLMs)
Format	Custom syntax	XML	Markdown
Content	Allow/Disallow rules	URLs + metadata	Descriptions + key page links
Analogy	Security guard	Phone book	Tour guide

They serve completely different purposes. You need all three — robots.txt controls who gets in, the sitemap lists what exists, and llms.txt explains what matters.

Setting up llms.txt on WordPress

You have two options: manual or plugin. Manual works, but it's tedious to maintain. Here's why.

Option 1: Manual setup

Create a file called llms.txt in your WordPress root directory (same place as wp-config.php):

# Your Site Name
> Brief description of your website and what it offers.
## Pages
- [Home](https://yoursite.com/): Main landing page
- [About](https://yoursite.com/about/): Company information
- [Services](https://yoursite.com/services/): What we offer
## Blog
- [Latest Post](https://yoursite.com/blog/latest/): Description

Upload it via FTP or your hosting file manager.

The problem: Every time you publish a new post, update a page, or change your site structure, you need to manually edit this file. For a site with 50+ posts, maintaining llms.txt by hand gets old fast.

You also need to handle URL rewrites so WordPress serves the file correctly, which means editing .htaccess or adding a custom rewrite rule in functions.php.

Option 2: Automatic generation with a plugin

This is where things get practical. Several WordPress plugins now generate llms.txt automatically. The setup typically looks like:

Install the plugin
Go to settings and choose which post types to include (posts, pages, products, etc.)
Click "Generate" — the plugin creates llms.txt based on your actual content
It updates automatically when you publish or edit content

What Prime SEO does specifically:

Prime SEO includes llms.txt generation as part of its AI Settings module. Here's what it offers beyond basic generation:

Auto-generation of both llms.txt and llms-full.txt
Post type selection — choose which content types appear in the file
Section management — organize links by custom sections, not just post type
AI Bots Manager — control which AI crawlers can access your site (12 bots including GPTBot, ClaudeBot, PerplexityBot)
AI Crawler Stats — see which AI bots actually visit your site and how often
AI Visibility Score — a quick check of how AI-ready your site is

The llms.txt and AI bots features are free — no Pro license required.

Install Prime SEO on WordPress.org — takes about 2 minutes.

What to include in your llms.txt (and what to skip)

Not every page belongs in your llms.txt. The whole point is curation — helping AI focus on what matters.

Include:

Homepage
Core product/service pages
Documentation and how-to guides
Key blog posts (evergreen content, not news)
Pricing page
FAQ or knowledge base

Skip:

Privacy policy, terms of service (AI doesn't need these for answering questions)
Category and tag archives
Author pages
Thin pages with little content
Pages marked noindex
Test/staging pages

Good descriptions matter. Don't just list URLs — add context:

## Documentation
- [Getting Started](https://yoursite.com/docs/start): Step-by-step installation
  and configuration guide for new users
- [API Reference](https://yoursite.com/docs/api): Complete REST API documentation
  with code examples in Python, JavaScript, and PHP

That description tells an AI model exactly what it'll find at each URL, so it can pick the right source for the user's question.

A real-world example

Here's a shortened version of our own llms.txt at prime-seo-plugin.com/llms.txt (the full text of the same pages is in llms-full.txt):

# Prime SEO
> Free WordPress SEO plugin for Google and AI search: meta tags, XML sitemap,
> schema markup, redirects, llms.txt and control over 12 AI crawlers.
## Documentation
- [Getting Started](https://prime-seo-plugin.com/docs/getting-started/):
  Install Prime SEO, run the setup wizard, and start optimizing your site
- [LLMs.txt](https://prime-seo-plugin.com/docs/llms-txt/):
  Generate and manage LLMs.txt files for your WordPress site
- [AI Bots Manager](https://prime-seo-plugin.com/docs/ai-bots-manager/):
  Control how AI search engines crawl your WordPress site
- [Schema Markup](https://prime-seo-plugin.com/docs/schema-markup/):
  Add JSON-LD structured data to your WordPress site
## Product
- [Prime SEO PRO](https://prime-seo-plugin.com/pro.html):
  AI-written SEO titles and descriptions, bulk generation, video sitemap

Notice how each link has a description that tells the AI model what the page contains. An AI answering "How do I set up schema markup in Prime SEO?" knows exactly which link to pull from.

Checklist: your llms.txt in 15 minutes

Here's the quick version for people who skim (no judgment — we do it too):

Decide on content — list 10-20 most important pages on your site
Choose your method — manual file or plugin (we recommend plugin for WordPress)
Write descriptions — every link needs a 1-line explanation
Organize sections — group by type (Docs, Blog, Product, etc.)
Test the URL — visit yoursite.com/llms.txt and verify it loads
Set up llms-full.txt (optional) — if you have documentation-heavy content
Monitor AI traffic — check if AI bots are reading your new file
Update regularly — or use a plugin that updates automatically

What's next for the standard?

llms.txt is still a proposal, not an official web standard. It doesn't have W3C backing or formal support from major AI providers.

But the momentum is real. Three community directories track adoption. GitHub discussions are active. Major tech companies are implementing it on their documentation sites. And the underlying problem — AI models struggling to understand website structure — isn't going away.

Whether the final standard is called llms.txt or something else, the concept of "a curated, machine-readable summary of your website" is becoming essential. Setting it up now costs nothing and positions your site for whatever comes next.

The sites that prepared for mobile-first indexing early didn't regret it. The same pattern is playing out with AI search readiness.

FAQ

Is llms.txt required for WordPress? No. It's optional, and your site will function perfectly without it. But adding it improves your chances of being correctly cited when AI search engines reference your content.

Will llms.txt improve my Google rankings? No. Google does not use llms.txt for ranking. It specifically targets AI language models like ChatGPT, Claude, and Perplexity. Your traditional SEO setup (sitemap, meta tags, schema) remains separate.

Do I need both llms.txt and llms-full.txt? For most sites, llms.txt alone is sufficient. llms-full.txt is useful if you have extensive documentation and want AI models to access the full content without crawling each page individually.

How often should I update llms.txt? Every time you add or remove important content from your site. If you use a plugin like Prime SEO, this happens automatically when you publish or update posts.

Is llms.txt the same as robots.txt? No. robots.txt controls which crawlers can access your site. llms.txt tells AI models what your site is about and which pages are most important. They serve different purposes, and you need both.