Of five free WordPress SEO plugins checked on September 21, 2026, one writes robots.txt rules that block AI crawlers. Two of the largest, Yoast SEO and All in One SEO, store a GPTBot setting in their free builds, and no code in those builds reads it. Rank Math and SEOPress contain no AI crawler names at all.
Disclosure: the author develops Prime SEO, the fifth plugin in this comparison. Every claim below comes from the free packages downloaded from the WordPress.org plugin directory, searched for crawler names and for the code that acts on each setting; file paths are given so that anyone can repeat the check.
Yoast SEO free stores three switches it never reads
Yoast SEO, active on 10 million sites according to the directory, defines three options in inc/options/class-wpseo-option-wpseo.php: deny_gptbot_crawling, deny_ccbot_crawling and deny_google_extended_crawling. All default to false. The settings screen even carries the label "Prevent OpenAI GPTBot from crawling" under Crawl optimization.
What the free build lacks is a reader. Its robots.txt code, src/integrations/front-end/robots-txt-integration.php, checks three neighbouring options, among them internal search and Google AdsBot, and stops there. In the settings script the GPTBot toggle receives isDummy whenever the site's isPremium preference is false, and the same component loads a link named block-unwanted-bots-upsell. A free user sees the switch as a preview of Premium.
All in One SEO Lite follows the same pattern
All in One SEO, installed on 2 million sites, declares an unwantedBots group in app/Common/Options/Options.php with openAiGptBot and commonCrawlCcBot next to a Google AdsBot entry. Across the Lite package those names appear in two option files and nowhere else. The only other code that touches the group resets or exports it.
The Lite download ships the app/Common and app/Lite folders. No file in either one turns the setting into a robots.txt line, which places the working part in the paid tiers.
Rank Math and SEOPress free have no AI crawler code
A search of Rank Math (4 million active installs) for 14 crawler names, from GPTBot to Amazonbot, returns nothing outside a bundled user agent library. SEOPress returns nothing at all. Rank Math free generates an llms.txt file, and SEOPress keeps that feature for its PRO version.
SEOPress PRO, a paid product, goes further: the 9.5 copy checked for this article offers a robots.txt template with rules for CCBot, GPTBot, Google-Extended and ClaudeBot. None of the four free builds logs visits from AI crawlers. Each package reads the visitor's user agent in at most one file, and no crawler name appears in any of them.
| Free version | Checked | Writes AI crawler rules to robots.txt | llms.txt | Logs AI crawler visits |
|---|---|---|---|---|
| Yoast SEO | 28.5 | No (setting stored, not read) | Yes | No |
| Rank Math | 1.0.278 | No | Yes | No |
| All in One SEO | 5.0.1.1 | No (setting stored, not read) | Yes | No |
| SEOPress | 10.2 | No | No (PRO) | No |
| Prime SEO | 1.2.0 | Yes, 15 crawlers | Yes, plus llms-full.txt | Yes |
Three lines of robots.txt need no plugin
Blocking a crawler does not require any of the five. OpenAI's crawler documentation says "GPTBot is used to make our generative AI foundation models more useful and safe" and accepts an ordinary robots.txt rule:
User-agent: GPTBot
Disallow: /
The same two lines work for ClaudeBot or for Google-Extended, the token that, in Google's words, "does not impact a site's inclusion in Google Search." On WordPress the catch is practical. The default robots.txt is virtual, many hosts offer no file access, and a list of 15 names kept by hand goes stale as vendors rename their agents.
Few site owners bother either way. Among the robots.txt files that Cloudflare could read on top domains, GPTBot was the most refused AI agent and was still, João Tomé of Cloudflare wrote in a July 2025 study, "disallowed by 312 domains" only. A rule also binds nobody: agents that fetch a page for a live user, ChatGPT-User and Perplexity-User among them, are documented by their vendors as outside robots.txt, a point covered in the guide to AI crawlers in WordPress logs.
Prime SEO 1.2.0 writes rules for 15 crawlers
Prime SEO puts the feature in its free version. The AI Bots Manager lists 15 crawlers with a switch each, plus three presets (allow all, block training crawlers only, block all). Matching robots.txt lines are appended to the virtual file, and a separate screen logs which AI agents visited and which page they opened.
Its weak side is plain. Prime SEO has 100+ active installs against Yoast's 10 million and a record that starts in January 2026. Two of the names on its list (anthropic-ai and Claude-Web) are ones Anthropic no longer documents. Those two rules block nothing today.
A site that needs one blocked crawler and nothing else is served as well by the three-line rule above (no plugin, no updates to follow). For the rest of the setup, see the llms.txt plugin for WordPress page.
The option keys are already in place
Yoast and All in One SEO would need little work to change this table, because the settings exist and only the reader is missing from their free code. If either one moves the switch out of the paid tier, free AI crawler control stops being a reason to choose a smaller plugin. Until then the comparison stands at one in five.
Originally published on the Prime SEO blog.
Top comments (0)