DEV Community: Daniil Lutsiuk

How to add llms.txt to WordPress in 2026: spec v2, the Lighthouse audit and five plugins compared

Daniil Lutsiuk — Thu, 24 Sep 2026 05:56:20 +0000

Jeremy Howard published version 2 of the llms.txt proposal on August 10, 2026, two years after the original. One line is required. WordPress.org now serves such a file at its own root, Chrome's Lighthouse checks for it since version 13.3, and three of the four biggest WordPress SEO plugins generate it in their free builds.

For a site owner the decision has three parts. Whether to publish the file at all. Whether to write it by hand or let a plugin build it, and, if a plugin, which one, since the default outputs of the five compared below differ in length by a factor of 200.

The v2 spec keeps one required line and adds two link relations

The format did not change. According to the proposal at llmstxt.org, a valid file contains "An H1 with the name of the project or site. This is the only required section," followed by a blockquote summary, then free paragraphs, then H2 sections holding markdown link lists in the form - [name](url): notes. A file with the single line # Example Shop is valid.

What v2 adds is discoverability. "v1 permitted llms.txt files in subpaths without saying what that meant," the changes page states. "v2 defines it: a file covers the pages under its path, and the most specific file applies." A page can now announce its file with <link rel="describedby" href="/llms.txt"> in the head, or with an HTTP Link: header set at the CDN, and the markdown twin of a page may live at either page.html.md or page.md, where v1 allowed only the longer form.

The ## Optional section survived with less meaning. Tools no longer treat it as a mechanical signal; the convention stays for links an agent can skip. Howard also writes that "thousands of sites now publish an llms.txt file," and the AI labs are among them. On September 24, 2026, developers.openai.com/llms.txt and docs.claude.com/llms.txt both answered 200, and so did wordpress.org/llms.txt, a 75-line file that opens by stating WordPress "powers more than 43% of all websites on the internet."

Lighthouse 13.3 audits the file but gives no score for it

Chrome's Lighthouse gained an Agentic Browsing category in version 13.3, which DebugBear covered on May 10, 2026. Four audits sit in it, and one is named "llms.txt does not follow recommendations." That audit fails a file with no H1 header, a file that is too short, or a file without links. Everything else passes.

A site without the file is not penalised. When the server answers 404, the Chrome documentation (last updated May 5, 2026) says "the audit is marked as Not Applicable (N/A)," and the whole category carries no 0 to 100 score, "because the standards for the agentic web are still emerging." Lighthouse gathers data here. Ranking is not part of it.

The audit also has a bug open against it. Issue 17082 in the Lighthouse repository, opened June 18, 2026, describes a spec-compliant file that returns 200 and still produces "Fetch of llms.txt failed" in PageSpeed Insights, which was running an older Chrome build than the documentation calls for. A green result there is not yet a reliable signal either way.

Google Search says the file goes unread, and server logs agree

John Mueller, Search Advocate at Google, wrote on Bluesky on June 17, 2025: "FWIW no AI system currently uses llms.txt." His evidence is in every access log: "It's super-obvious if you look at your server logs. The consumer LLMs / chatbots (the ones that SEOs want traffic from) will fetch your pages - for training and grounding, but none of them fetch the llms.txt file."

Fifteen months later the two halves of Google disagree. Search Engine Journal put it in a headline, "Google's llms.txt guidance depends on which product you ask": Search says the file is unused, while Chrome ships an audit for it.

On narrower ground, the llmstxt.org text sides with Chrome, because the file is used "most heavily for software documentation, where coding agents follow them to find API references and tutorials." A WooCommerce store is not documentation. No vendor has published a statement that its shopping or search crawler reads the file.

Publishing one costs a few kilobytes and, with a plugin, no time. Trusting it costs more, because the crawlers measurably do fetch the HTML pages, and access to those pages is decided by robots.txt, a file that llms.txt never replaces.

What five WordPress plugins write into the file

A search for "llms.txt" in the WordPress.org plugin directory returned 521 plugins on September 24, 2026. Website LLMs.txt, the largest single-purpose one, lists 40,000 active installs. The findings below come from reading the free builds of the four major SEO suites and of Prime SEO, the plugin developed by the author, on the same date.

Yoast SEO 28.5 ships the feature off by default (enable_llms_txt in the wpseo option) and writes a physical file to the site root once a week through WP-Cron, with a first run five minutes after the toggle. Each post type gets five links. Cornerstone content comes first, then the most recently modified, and posts older than twelve months are dropped. Every string goes through html_entity_decode and a backslash escape of markdown characters, and the plugin refuses to overwrite a file whose MD5 hash it did not record itself, so a hand-written file stays.

Rank Math 1.0.279 keeps llms.txt as a module outside the default set, so it also starts off. Once enabled, it serves the file on every request through a rewrite rule, with nothing on disk, a default of 50 links per post type and the header X-Robots-Tag: noindex, nofollow. Line one is a credit sentence, removable with the filter rank_math/llms_txt/remove_credit, and the title is the Knowledge Graph name rather than the blog name.

All in One SEO 5.0.2 also starts off and also writes a physical file, but through Action Scheduler, daily, plus a single run ten seconds after any post or term is saved. Its default is 1,000 links per post type, pillar content first. Version 5.0.2 is the release that switched the title from HTML escaping to entity decoding, with a docblock reading "this is a plain-text file."

SEOPress 10.2 is the odd one. Its directory readme lists "llms.txt & Agent Readiness" among the free features, while the free build on WordPress.org holds a toggle default and a documentation link, and no code that writes or serves the file. The generator sits in the paid version, which the author has not examined.

Prime SEO 1.2.1, whose llms.txt generator for WordPress is documented on the plugin site, serves the file on activation without a toggle, dynamically, the way Rank Math does. Its list holds 20 pages, 20 posts, 20 WooCommerce products and 10 categories, newest first; noindex posts and the front page are left out. Each link is described by the SEO description or the excerpt, cut to 20 words.

If a physical llms.txt exists at the root, the plugin steps aside and the server delivers that file. An optional llms-full.txt, off by default, appends the body text of up to 50 posts at 2,000 words each; no other free build here produces a full-text file, although the AIOSEO code carries the filename and a flag that the free build ignores.

One defect belongs in this list too. In 1.2.1 the site name comes straight from get_bloginfo('name'), which WordPress stores already HTML-escaped, so a site called "Smith & Sons" appears as # Smith & Sons. The fix, wp_specialchars_decode, is committed for 1.2.2, and it is the same bug AIOSEO closed a day earlier. Nothing in any of the five builds emits the v2 rel="describedby" link or the Link: header, and none publishes .md twins of pages, so the discoverability half of version 2 has no implementation among them.

Two minutes by hand, or no minutes with a plugin

The manual route needs a text file at the web root, uploaded over SFTP or through the host's file manager: the site name after #, one blockquote sentence, a short H2 list of the pages an agent should read first. Yoast, Prime SEO and the Lighthouse audit all accept that file as it is. The trade is upkeep. A hand-written list is frozen on the day it is written, while the generated ones follow new posts weekly, daily, or on the next request.

A plugin is the better choice when the site publishes often and nobody will remember the file. The hand-written file wins when the site has a dozen pages that matter and a thousand that do not, because every generator above selects by date, cornerstone flag or sitemap priority, and none of them knows which page a support agent should read before quoting a return policy.

If an agent ever asks for the file while the catalogue changes every week, the generated version is the one that answers correctly; if the file has to say which twelve pages define the business, a person has to write it, and every plugin here steps aside for that file. Prime SEO is free on WordPress.org.

Google rewrites 63% of meta descriptions. The other 37% still decide the click

Daniil Lutsiuk — Wed, 23 Sep 2026 15:48:05 +0000

Google rewrites 62.78% of meta descriptions, according to a study of 20,000 keywords that Ahrefs published in 2020. The number gets quoted as a reason to stop writing them. The same study found a quarter of top-ranking pages with no meta description at all, which sounds like the second half of the same argument.

Both figures leave out the part that matters to a site owner. In the remaining 37% of cases Google shows the description as written, and Bing and the AI search engines quote a page before a human ever opens it. For a WordPress site with 400 posts the question is not whether to write descriptions. The question is who writes them.

What Google says it does with the tag

Google's own documentation on snippets is less dismissive than the study makes it sound. "Snippets are primarily created from the page content itself," the Search Central page on snippet control states, before adding that Google "sometimes uses the meta description HTML element if it might give users a more accurate description of the page than content taken directly from the page."

The rewrite is a fallback for descriptions that fail. One that repeats the title gets replaced by a sentence from the body, and so does one copied across a whole category, which the same page addresses directly: "identical or similar descriptions on every page of a site aren't helpful when individual pages appear in search results." On length the guidance is loose. There is no cap, and the snippet "is truncated in Google Search results as needed, typically to fit the device width."

Yoast, whose plugin runs on 10 million sites by the WordPress.org count, translates the truncation into a working number. Its guide on meta descriptions, updated by Edwin Toonen on June 23, 2025, recommends "up to 155 characters" and concedes the same limit of control: "Sometimes, it shows the meta description, and sometimes, it just grabs some sentences of your copy or generates something itself."

The empty tag is the common case

Michal Pecánek, who wrote the Ahrefs study, measured a weak correlation of 0.28 between having a description and ranking position. The figure is the counterargument to any claim that descriptions move rankings, and it holds. A description changes what the result says, not where it sits.

What happens when the tag is empty depends on the SEO plugin. Prime SEO, the plugin developed by the author, falls back to the post excerpt, and when the excerpt is empty too it prints the opening 25 words of the body. Most plugins do something similar, so the snippet starts mid-sentence, with a greeting to regular readers or a cookie notice. On a site where the excerpt field has never been used, that fallback is what searchers read for every post.

The free version of Prime SEO counts those posts on its dashboard, under Needs Attention, and its SEO score includes a check for a description between 120 and 160 characters. The count is usually larger than the owner expects. A blog that has published twice a week since 2019 holds about 700 posts, and at a few minutes per description written by hand, with the post open, the backlog is a working week.

Six fields in one click, a site in one run

Version 1.2.1 of Prime SEO Pro, released on September 23, 2026, ships the option the backlog needs. A Generate All button in the post editor writes six fields at once: the SEO title, the meta description, and the Open Graph and Twitter title and description, each cut to its own limit (155 characters for the meta description). The text comes from the post content and the focus keyword when one is set. The Bulk Generator runs the same generation over every post that is missing a title or a description, with a progress bar, a pause button and a stop button.

Since 1.2.1 the generations run through the plugin's own service. A Pro license includes 500 generations a month, where one generation is one Generate All click, one post in a bulk run, or one Content Score analysis, and the counter with its reset date sits in AI Settings. No OpenAI account is needed. Sites that want more, or a different model, paste their own OpenAI key on the same page; the limit is then gone, and the request travels from the site to OpenAI directly.

The generated text is a draft, not a verdict. Google will still rewrite the ones that read like every other page, and the 37% that survive are the ones that say what the page contains in words the searcher used. Pro subscriptions to the plugin have doubled since the start of September 2026, and the bulk run is the feature new subscribers open first.

If three posts in four on a site show an opening sentence as the snippet, the choice in 2026 is between a week of writing and an afternoon of reviewing. Prime SEO is free on WordPress.org; Generate All and the Bulk Generator are part of Pro, $59 a year with a 14-day trial and no card.

Yoast and All in One SEO store a GPTBot switch their free code never reads

Daniil Lutsiuk — Mon, 21 Sep 2026 14:29:30 +0000

Of five free WordPress SEO plugins checked on September 21, 2026, one writes robots.txt rules that block AI crawlers. Two of the largest, Yoast SEO and All in One SEO, store a GPTBot setting in their free builds, and no code in those builds reads it. Rank Math and SEOPress contain no AI crawler names at all.

Disclosure: the author develops Prime SEO, the fifth plugin in this comparison. Every claim below comes from the free packages downloaded from the WordPress.org plugin directory, searched for crawler names and for the code that acts on each setting; file paths are given so that anyone can repeat the check.

Yoast SEO free stores three switches it never reads

Yoast SEO, active on 10 million sites according to the directory, defines three options in inc/options/class-wpseo-option-wpseo.php: deny_gptbot_crawling, deny_ccbot_crawling and deny_google_extended_crawling. All default to false. The settings screen even carries the label "Prevent OpenAI GPTBot from crawling" under Crawl optimization.

What the free build lacks is a reader. Its robots.txt code, src/integrations/front-end/robots-txt-integration.php, checks three neighbouring options, among them internal search and Google AdsBot, and stops there. In the settings script the GPTBot toggle receives isDummy whenever the site's isPremium preference is false, and the same component loads a link named block-unwanted-bots-upsell. A free user sees the switch as a preview of Premium.

All in One SEO Lite follows the same pattern

All in One SEO, installed on 2 million sites, declares an unwantedBots group in app/Common/Options/Options.php with openAiGptBot and commonCrawlCcBot next to a Google AdsBot entry. Across the Lite package those names appear in two option files and nowhere else. The only other code that touches the group resets or exports it.

The Lite download ships the app/Common and app/Lite folders. No file in either one turns the setting into a robots.txt line, which places the working part in the paid tiers.

Rank Math and SEOPress free have no AI crawler code

A search of Rank Math (4 million active installs) for 14 crawler names, from GPTBot to Amazonbot, returns nothing outside a bundled user agent library. SEOPress returns nothing at all. Rank Math free generates an llms.txt file, and SEOPress keeps that feature for its PRO version.

SEOPress PRO, a paid product, goes further: the 9.5 copy checked for this article offers a robots.txt template with rules for CCBot, GPTBot, Google-Extended and ClaudeBot. None of the four free builds logs visits from AI crawlers. Each package reads the visitor's user agent in at most one file, and no crawler name appears in any of them.

Free version	Checked	Writes AI crawler rules to robots.txt	llms.txt	Logs AI crawler visits
Yoast SEO	28.5	No (setting stored, not read)	Yes	No
Rank Math	1.0.278	No	Yes	No
All in One SEO	5.0.1.1	No (setting stored, not read)	Yes	No
SEOPress	10.2	No	No (PRO)	No
Prime SEO	1.2.0	Yes, 15 crawlers	Yes, plus llms-full.txt	Yes

Three lines of robots.txt need no plugin

Blocking a crawler does not require any of the five. OpenAI's crawler documentation says "GPTBot is used to make our generative AI foundation models more useful and safe" and accepts an ordinary robots.txt rule:

User-agent: GPTBot
Disallow: /

The same two lines work for ClaudeBot or for Google-Extended, the token that, in Google's words, "does not impact a site's inclusion in Google Search." On WordPress the catch is practical. The default robots.txt is virtual, many hosts offer no file access, and a list of 15 names kept by hand goes stale as vendors rename their agents.

Few site owners bother either way. Among the robots.txt files that Cloudflare could read on top domains, GPTBot was the most refused AI agent and was still, João Tomé of Cloudflare wrote in a July 2025 study, "disallowed by 312 domains" only. A rule also binds nobody: agents that fetch a page for a live user, ChatGPT-User and Perplexity-User among them, are documented by their vendors as outside robots.txt, a point covered in the guide to AI crawlers in WordPress logs.

Prime SEO 1.2.0 writes rules for 15 crawlers

Prime SEO puts the feature in its free version. The AI Bots Manager lists 15 crawlers with a switch each, plus three presets (allow all, block training crawlers only, block all). Matching robots.txt lines are appended to the virtual file, and a separate screen logs which AI agents visited and which page they opened.

Its weak side is plain. Prime SEO has 100+ active installs against Yoast's 10 million and a record that starts in January 2026. Two of the names on its list (anthropic-ai and Claude-Web) are ones Anthropic no longer documents. Those two rules block nothing today.

A site that needs one blocked crawler and nothing else is served as well by the three-line rule above (no plugin, no updates to follow). For the rest of the setup, see the llms.txt plugin for WordPress page.

The option keys are already in place

Yoast and All in One SEO would need little work to change this table, because the settings exist and only the reader is missing from their free code. If either one moves the switch out of the paid tier, free AI crawler control stops being a reason to choose a smaller plugin. Until then the comparison stands at one in five.

Originally published on the Prime SEO blog.

3,590 downloads, 100 active installs: what I changed in my WordPress plugin today

Daniil Lutsiuk — Mon, 21 Sep 2026 09:42:05 +0000

Prime SEO, the WordPress plugin I've been building since January, has 3,590 downloads on WordPress.org as of September 21, 2026.

The same API reports 100+ active installs. WordPress.org rounds that figure down, so the real number sits between 100 and 199, and at most one download in eighteen became a site that kept the plugin.

I don't know yet why people leave. Version 1.1.9 added a deactivation survey yesterday (it sends nothing unless the user presses Submit, and never the site address), and the only two answers in it so far are my own tests. What I can look at today is the first-run path, and the hole in it is easy to see.

Ten menu items and no next step

A new install opened a setup wizard with three questions: homepage title, modules, robots.txt. After that the user landed on a dashboard next to ten menu items. The wizard itself ended on a green tick and three links, and nothing on either screen said which of the ten pages mattered first.

All in One SEO 5.0.1.1 handles this with a setup checklist. In its free build the file is app/Common/SeoChecklist/SeoChecklist.php: every step has a priority and a time estimate, a callback decides whether the step is complete, and the names of completed steps are appended to a stored list. I borrowed that shape.

Two kinds of steps

Version 1.2.0 went out today with a checklist on the dashboard. Nothing in it is ticked by hand.

In the All in One SEO code a completed step is stored by name and reopens only in a few coded cases: indexing gets switched off, Search Console is disconnected, the user unticks it by hand. I wanted every step that depends on a setting to follow that setting, because a bar that says 9 of 9 while the sitemap is off tells the user nothing. My steps come in two kinds:

// live: follows a setting. Switch the sitemap off and the step opens again.
'completed' => $module( 'module_sitemap' ),

// sticky: a one-time event. Once done, stays done.
'completed' => (bool) get_option( 'prime_seo_wizard_complete' ),

Live steps cover settings such as the sitemap and the AI bot policy (the $module() helper treats a missing key as on, because modules are enabled by default). Any of them can reopen. Sticky ones cover events, such as a finished wizard or the day one post reached a score of 80. The wizard's last page now shows the same list, with the steps it has just completed on the left.

The step nobody could complete

That bug was mine alone. My first version of the IndexNow step looked for the API key inside the plugin's settings array. That key lives in a separate option, prime_seo_indexnow_key, and is generated on first use. The step stayed open forever, and I noticed only because my test site sat at 9 of 10 after I had done everything on the list.

Fixing it exposed a second problem. Since 1.1.6 the plugin deliberately sends nothing to IndexNow from local, development and hidden sites, which made the step impossible to finish on exactly those sites. It's now offered to public sites only, so a local install counts nine steps and a live one counts ten (eleven when another SEO plugin left data to import). Both counts are honest now.

If the survey answers come back as "too complicated", the checklist was the right first move. If they say "feature X is missing", I spent a day on the wrong problem, and the next entry will be about X.

I'm the developer of Prime SEO, a free SEO plugin for WordPress. Entry one of a build log; the numbers come from the public WordPress.org API.

AI crawlers in WordPress logs: GPTBot shows up, Google-Extended never does

Daniil Lutsiuk — Mon, 21 Sep 2026 08:10:01 +0000

Google-Extended and Applebot-Extended, two of the names that robots.txt guides list most often for AI, cannot appear in a WordPress access log. Neither one sends a request. Google's crawler documentation states that Google-Extended "doesn't have a separate HTTP request user agent string." The second name gets even less: "Applebot-Extended does not crawl webpages," Apple's support page for Applebot reads, which leaves both names as robots.txt tokens rather than visitors and means that a log of AI crawlers in WordPress has to be built from the agents that send requests.

Disclosure: the author develops Prime SEO, a WordPress plugin that includes such a log. Every user agent below was checked against vendor documentation on September 21, 2026, and the code runs without any plugin.

Three vendors document nine user agents

OpenAI lists four agents on its crawler page. GPTBot gathers training material (the page says it exists "to make our generative AI foundation models more useful and safe"), while OAI-SearchBot builds the index behind ChatGPT search. ChatGPT-User fetches a page when a person asks for it. OAI-AdsBot checks pages submitted as ads.

Anthropic documents three, split the same way: ClaudeBot collects training data, while Claude-User and Claude-SearchBot serve live questions and search. Perplexity runs two.

User agent substring	Operator	Purpose, per the vendor	robots.txt
`GPTBot`	OpenAI	Model training	Honored
`OAI-SearchBot`	OpenAI	ChatGPT search index	Honored
`ChatGPT-User`	OpenAI	Fetch started by a user	"may not apply"
`OAI-AdsBot`	OpenAI	Safety check of pages submitted as ads	Not stated
`ClaudeBot`	Anthropic	Model training	Honored
`Claude-User`	Anthropic	Fetch started by a user	Honored
`Claude-SearchBot`	Anthropic	Search quality	Honored
`PerplexityBot`	Perplexity	Perplexity search results	Honored
`Perplexity-User`	Perplexity	Fetch started by a user	"generally ignores"

Missing from Anthropic's page are anthropic-ai and Claude-Web, two names that still circulate in robots.txt templates. A rule aimed at them blocks nothing the vendor admits to sending.

User-triggered fetchers sit outside robots.txt

Crawlers that build an index obey robots.txt, by the vendors' own account. Anthropic's help page says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt." Fetchers started by a person are a different matter.

OpenAI describes ChatGPT-User as an agent that "is not used for crawling the web in an automatic fashion," and adds that "robots.txt rules may not apply" because a user starts the action. Perplexity's bot guide is blunter about Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

For a site owner, the split has a practical side. A Disallow line can keep a page out of a training set, yet the same page may still be fetched for a ChatGPT user five minutes later. Only a log shows that second kind of visit.

A short mu-plugin records every match

WordPress needs no plugin for a basic log. The file below goes into wp-content/mu-plugins/, hooks into template_redirect at priority 1, and compares the user agent of each request with the nine documented names.

<?php
/**
 * Plugin Name: AI Crawler Log
 * Description: Records visits from documented AI user agents.
 */

add_action( 'template_redirect', function () {
    if ( empty( $_SERVER['HTTP_USER_AGENT'] ) ) {
        return;
    }

    $agent = sanitize_text_field( wp_unslash( $_SERVER['HTTP_USER_AGENT'] ) );
    $names = array(
        'GPTBot', 'OAI-SearchBot', 'ChatGPT-User', 'OAI-AdsBot',
        'ClaudeBot', 'Claude-User', 'Claude-SearchBot',
        'PerplexityBot', 'Perplexity-User',
    );

    foreach ( $names as $name ) {
        if ( false === stripos( $agent, $name ) ) {
            continue;
        }

        $log   = get_option( 'ai_crawler_log', array() );
        $log[] = array(
            'bot'  => $name,
            'url'  => esc_url_raw( wp_unslash( $_SERVER['REQUEST_URI'] ?? '/' ) ),
            'time' => gmdate( 'Y-m-d H:i:s' ),
        );

        // Keep the last 500 visits; "false" keeps the option out of autoload.
        update_option( 'ai_crawler_log', array_slice( $log, -500 ), false );
        break;
    }
}, 1 );

Each match lands in a single option, ai_crawler_log, stored with autoload switched off so that ordinary page loads never carry it. The array_slice() call caps the record at 500 visits. On a local WordPress 7.1 test install, a request sent with the GPTBot agent string appeared in the option within the same second, while a Chrome agent left no trace. Reading the result takes one WP-CLI command: wp option get ai_crawler_log --format=json.

Owners with shell access can skip PHP and count matches in the server log instead:

grep -Eio "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User" access.log | sort | uniq -c | sort -rn

Page caches and spoofed agents distort the count

PHP-based logging has a blind spot. When a full-page cache (WP Rocket or a CDN edge, for instance) answers the request, WordPress never starts and the visit goes unrecorded. The server access log catches those hits; a PHP log on a cached site shows only cache misses.

User agent strings are also trivial to fake. Any script can call itself GPTBot, which is why all three vendors publish their IP ranges as JSON files (openai.com/gptbot.json is one of them). A visit from outside those ranges is an impostor.

The opposite error exists too. Cloudflare reported on August 4, 2025 that Perplexity used, besides its declared agents, "a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked." No list of names catches a crawler that refuses to give one.

Cloudflare counted a 305% rise in GPTBot requests

Traffic from these agents is no longer marginal. GPTBot "increased its share from 2.2% to 7.7%" of crawler requests in the year to May 2025, João Tomé of Cloudflare wrote in a study of crawler traffic published that July. Raw request volume from the same bot, the figure in the heading above, more than quadrupled.

Growth was uneven. In the same dataset ClaudeBot, Anthropic's training crawler, "fell from 11.7% to 5.4% of total traffic." One site's log can look nothing like the global chart, and a quiet month for one vendor says little about the others.

Blocking remains rare among large sites. Of the 3,816 robots.txt files that Cloudflare could read among top domains, 546 addressed AI bots at all. GPTBot led the refusals: the study found it "disallowed by 312 domains."

Prime SEO 1.1.9 matches 14 names, two of them in vain

Prime SEO, the plugin named in the disclosure, ships the same stripos() approach behind a dashboard screen called AI Crawler Stats, which also records requests for llms.txt and the XML sitemap. Version 1.1.9 matches 14 substrings. Two of them are Google-Extended and Applebot-Extended, which, per the documentation quoted above, can never match. Those two counters will stay at zero.

The plugin's list also predates Claude-User, Claude-SearchBot and Perplexity-User, so it undercounts user-triggered visits today; an update is planned for version 1.2.0. Background on the file these agents are meant to read is in the guide to what llms.txt is and on the llms.txt plugin for WordPress page.

Vendor lists change faster than robots.txt files

All three vendors now separate training crawlers from agents that act for a person. If that split continues, a robots.txt file written for GPTBot and ClaudeBot alone will govern a shrinking part of AI traffic, and the raw access log (not the rule file) becomes the place where a new agent name first shows up. Until a vendor page changes, nine names are enough.

Originally published on the Prime SEO blog.

What Is llms.txt? How to Set It Up on WordPress (2026 Guide)

Daniil Lutsiuk — Sun, 20 Sep 2026 18:19:03 +0000

In September 2024, Jeremy Howard — the guy behind fast.ai and a former president of Kaggle — published a short proposal. He called it llms.txt, and the idea was almost annoyingly simple: give AI models a plain-text map of your website so they don't have to guess what's important.

No one paid much attention at first. Then Cloudflare added one. Then Stripe. Then Anthropic, Vercel, Supabase, and Zapier. Today, over 780 websites have published an llms.txt file, and three separate community directories track who's adopted it.

The question isn't really whether llms.txt will matter. It's whether you'll set yours up before or after your competitors do.

What llms.txt actually is (in plain English)

Think of your website from an AI model's perspective for a second.

When ChatGPT or Perplexity needs to answer a question about your product, it sends a crawler to your site. That crawler sees HTML — headers, footers, navigation menus, cookie banners, newsletter popups, sidebar widgets. Buried somewhere in that noise is the content that actually matters.

Your XML sitemap doesn't help much here. It lists every URL on the site equally — the homepage, the privacy policy, that test page from 2019 you forgot to delete. There's no hierarchy, no context, no "start here."

llms.txt solves this by giving AI models exactly what they need: a structured Markdown file at yoursite.com/llms.txt that says "here's what this site is about, and here are the pages that matter most."

That's it. A curated table of contents for machines that read.

The format: simpler than you'd expect

An llms.txt file is just Markdown with a specific structure:

# Your Site Name
> A one-line summary of what your site does.
Some optional context about your project, company, or product.
## Documentation
- [Getting Started](https://yoursite.com/docs/start): Setup guide for new users
- [API Reference](https://yoursite.com/docs/api): Complete API documentation
## Blog
- [Latest Release](https://yoursite.com/blog/v2): What's new in version 2.0

The rules are minimal:

One H1 heading — your site or project name (the only required element)
A blockquote — brief summary with key facts
H2 sections — groups of links, each with an optional description
Markdown links — [Page Name](URL): optional description

That's the entire spec. Jeremy Howard intentionally kept it simple because "websites already have sitemaps, robots.txt, and meta tags. What they don't have is a curated overview written specifically for language models."

llms.txt vs llms-full.txt — yes, there are two

The standard actually defines two files:

llms.txt is the summary — your site's elevator pitch plus links to key pages. Think of it as the table of contents. Most llms.txt files are a few hundred lines at most.

llms-full.txt is the entire book. It contains the full Markdown content of your key pages, concatenated into one massive file. Cloudflare's llms-full.txt is 3.7 million tokens. Anthropic's is 481,000 tokens. Vercel's has been described as "a 400,000-word novel."

Why would you want that? Because an AI model with a large enough context window can ingest your entire documentation in one shot — no crawling, no parsing HTML, no missing pages.

For most WordPress sites, llms.txt alone is enough. But if you run a documentation-heavy site or want AI to have deep access to your content, llms-full.txt is worth considering.

Who's actually using it?

The short answer: mostly developer-facing companies. The long answer is more interesting.

Stripe organizes their llms.txt by product category, with an "Optional" section for niche tools like Stripe Climate. Their file tells AI agents exactly how to retrieve plain-text versions of documentation pages.

Cloudflare structures theirs by product line — Workers, Pages, R2, and so on. Each entry has a description that helps AI models decide which page to fetch based on the user's question.

Anthropic (the company behind Claude) maintains both files. Their llms.txt is 8,364 tokens; the full version is 481,349 tokens covering their entire API documentation.

Vercel went a step further and published a proposal for inline LLM instructions in HTML — extending the llms.txt idea into individual pages.

Other adopters include Supabase, Zapier, Modal, Coinbase, and hundreds of smaller companies tracked across three community directories.

The honest truth: does it actually work?

Here's where most articles about llms.txt get vague. We won't.

The data is mixed. Search Engine Land tracked 10 websites for 180 days — 90 days before adding llms.txt, 90 days after. The results:

2 out of 10 sites saw AI traffic increases (12.5% and 25%)
8 sites saw no measurable change
1 site actually declined by 19.7%

But here's the catch: the two sites that grew had also launched new content and PR campaigns around the same time. The llms.txt file alone didn't drive the increase — it was part of a broader effort.

Adoption is still early. Rankability scanned the top 1,000 most-visited websites globally. As of mid-2025, only 0.3% had an llms.txt file. That's 3 out of 1,000.

No AI provider has officially committed to reading llms.txt. OpenAI, Anthropic, Google — none have confirmed their crawlers consistently follow llms.txt instructions. Google's John Mueller compared the standard to the deprecated keywords meta tag.

So why bother?

Three reasons:

Low effort, no downside. Setting up llms.txt takes 5 minutes. There's zero risk. It won't hurt your rankings, slow your site, or cause conflicts with anything.
The direction is clear. Even if no one reads llms.txt today, AI search is growing at 34% CAGR. Gartner predicts 25% of search volume will shift away from traditional engines by late 2026. When AI providers do formalize how they read websites, llms.txt (or something very similar) will be the standard.
First-mover advantage is real. If only 0.3% of top sites have llms.txt, that means 99.7% of your competitors don't. Early adopters set the baseline that later standards build on.

How llms.txt differs from robots.txt and XML sitemaps

This trips people up, so here's a clear comparison:

	robots.txt	XML Sitemap	llms.txt
Purpose	Controls crawler access	Lists all URLs for indexing	Curated overview for AI understanding
Audience	Search engine crawlers	Search engine crawlers	Language models (LLMs)
Format	Custom syntax	XML	Markdown
Content	Allow/Disallow rules	URLs + metadata	Descriptions + key page links
Analogy	Security guard	Phone book	Tour guide

They serve completely different purposes. You need all three — robots.txt controls who gets in, the sitemap lists what exists, and llms.txt explains what matters.

Setting up llms.txt on WordPress

You have two options: manual or plugin. Manual works, but it's tedious to maintain. Here's why.

Option 1: Manual setup

Create a file called llms.txt in your WordPress root directory (same place as wp-config.php):

# Your Site Name
> Brief description of your website and what it offers.
## Pages
- [Home](https://yoursite.com/): Main landing page
- [About](https://yoursite.com/about/): Company information
- [Services](https://yoursite.com/services/): What we offer
## Blog
- [Latest Post](https://yoursite.com/blog/latest/): Description

Upload it via FTP or your hosting file manager.

The problem: Every time you publish a new post, update a page, or change your site structure, you need to manually edit this file. For a site with 50+ posts, maintaining llms.txt by hand gets old fast.

You also need to handle URL rewrites so WordPress serves the file correctly, which means editing .htaccess or adding a custom rewrite rule in functions.php.

Option 2: Automatic generation with a plugin

This is where things get practical. Several WordPress plugins now generate llms.txt automatically. The setup typically looks like:

Install the plugin
Go to settings and choose which post types to include (posts, pages, products, etc.)
Click "Generate" — the plugin creates llms.txt based on your actual content
It updates automatically when you publish or edit content

What Prime SEO does specifically:

Prime SEO includes llms.txt generation as part of its AI Settings module. Here's what it offers beyond basic generation:

Auto-generation of both llms.txt and llms-full.txt
Post type selection — choose which content types appear in the file
Section management — organize links by custom sections, not just post type
AI Bots Manager — control which AI crawlers can access your site (12 bots including GPTBot, ClaudeBot, PerplexityBot)
AI Crawler Stats — see which AI bots actually visit your site and how often
AI Visibility Score — a quick check of how AI-ready your site is

The llms.txt and AI bots features are free — no Pro license required.

Install Prime SEO on WordPress.org — takes about 2 minutes.

What to include in your llms.txt (and what to skip)

Not every page belongs in your llms.txt. The whole point is curation — helping AI focus on what matters.

Include:

Homepage
Core product/service pages
Documentation and how-to guides
Key blog posts (evergreen content, not news)
Pricing page
FAQ or knowledge base

Skip:

Privacy policy, terms of service (AI doesn't need these for answering questions)
Category and tag archives
Author pages
Thin pages with little content
Pages marked noindex
Test/staging pages

Good descriptions matter. Don't just list URLs — add context:

## Documentation
- [Getting Started](https://yoursite.com/docs/start): Step-by-step installation
  and configuration guide for new users
- [API Reference](https://yoursite.com/docs/api): Complete REST API documentation
  with code examples in Python, JavaScript, and PHP

That description tells an AI model exactly what it'll find at each URL, so it can pick the right source for the user's question.

A real-world example

Here's a shortened version of our own llms.txt at prime-seo-plugin.com/llms.txt (the full text of the same pages is in llms-full.txt):

# Prime SEO
> Free WordPress SEO plugin for Google and AI search: meta tags, XML sitemap,
> schema markup, redirects, llms.txt and control over 12 AI crawlers.
## Documentation
- [Getting Started](https://prime-seo-plugin.com/docs/getting-started/):
  Install Prime SEO, run the setup wizard, and start optimizing your site
- [LLMs.txt](https://prime-seo-plugin.com/docs/llms-txt/):
  Generate and manage LLMs.txt files for your WordPress site
- [AI Bots Manager](https://prime-seo-plugin.com/docs/ai-bots-manager/):
  Control how AI search engines crawl your WordPress site
- [Schema Markup](https://prime-seo-plugin.com/docs/schema-markup/):
  Add JSON-LD structured data to your WordPress site
## Product
- [Prime SEO PRO](https://prime-seo-plugin.com/pro.html):
  AI-written SEO titles and descriptions, bulk generation, video sitemap

Notice how each link has a description that tells the AI model what the page contains. An AI answering "How do I set up schema markup in Prime SEO?" knows exactly which link to pull from.

Checklist: your llms.txt in 15 minutes

Here's the quick version for people who skim (no judgment — we do it too):

Decide on content — list 10-20 most important pages on your site
Choose your method — manual file or plugin (we recommend plugin for WordPress)
Write descriptions — every link needs a 1-line explanation
Organize sections — group by type (Docs, Blog, Product, etc.)
Test the URL — visit yoursite.com/llms.txt and verify it loads
Set up llms-full.txt (optional) — if you have documentation-heavy content
Monitor AI traffic — check if AI bots are reading your new file
Update regularly — or use a plugin that updates automatically

What's next for the standard?

llms.txt is still a proposal, not an official web standard. It doesn't have W3C backing or formal support from major AI providers.

But the momentum is real. Three community directories track adoption. GitHub discussions are active. Major tech companies are implementing it on their documentation sites. And the underlying problem — AI models struggling to understand website structure — isn't going away.

Whether the final standard is called llms.txt or something else, the concept of "a curated, machine-readable summary of your website" is becoming essential. Setting it up now costs nothing and positions your site for whatever comes next.

The sites that prepared for mobile-first indexing early didn't regret it. The same pattern is playing out with AI search readiness.

FAQ

Is llms.txt required for WordPress? No. It's optional, and your site will function perfectly without it. But adding it improves your chances of being correctly cited when AI search engines reference your content.

Will llms.txt improve my Google rankings? No. Google does not use llms.txt for ranking. It specifically targets AI language models like ChatGPT, Claude, and Perplexity. Your traditional SEO setup (sitemap, meta tags, schema) remains separate.

Do I need both llms.txt and llms-full.txt? For most sites, llms.txt alone is sufficient. llms-full.txt is useful if you have extensive documentation and want AI models to access the full content without crawling each page individually.

How often should I update llms.txt? Every time you add or remove important content from your site. If you use a plugin like Prime SEO, this happens automatically when you publish or update posts.

Is llms.txt the same as robots.txt? No. robots.txt controls which crawlers can access your site. llms.txt tells AI models what your site is about and which pages are most important. They serve different purposes, and you need both.