Generate llms.txt on WordPress and Verify AI Crawlers Actually Fetch It

You can publish an llms.txt in ten minutes. Proving that a crawler actually fetched it — right URL, right content type, no redirect, no WAF block — takes another twenty. This post does both, in order, with commands you can paste into a terminal.

Honest framing first: llms.txt is a community proposal, not a ratified standard, and no AI provider promises to read it. What you can verify objectively is five facts:

1. The file is served as text/plain; charset=utf-8.
2. It is reachable at the exact URL a crawler would request, with no redirect.
3. It is reachable without cookies, JavaScript or a logged-in session.
4. robots.txt and your edge (CDN/WAF) do not block AI user agents from it.
5. Requests for it show up in your access log.

Everything below targets those five facts.

1. Decide what goes in the file

The proposed shape is small: an H1 with the site name, a blockquote summary, then sections of markdown links with one-line descriptions.

# Example Studio

> A two-person studio building custom WordPress themes and WooCommerce checkouts.

## Pages

- [Services](https://example.com/services): Scope, timeline and what a project costs.
- [Contact](https://example.com/contact): How to start a project.

## Posts

- [Speeding up a slow WooCommerce cart](https://example.com/blog/slow-cart): Where the time actually goes.

Two rules worth following: one line per link (wrapping breaks parsers that read line by line), and no content that only exists behind a login.

2. Serve it from WordPress

If you would rather not maintain PHP for this, AumViso generates llms.txt on a WordPress site, alongside schema, sitemap, meta and internal links. It is free on wordpress.org.

If you want the file under your own control, drop this into wp-content/mu-plugins/llms-txt.php. MU-plugins load automatically, so there is no activation step — you flush rewrite rules once from the CLI.

<?php
/**
 * Plugin Name: llms.txt endpoint
 * Description: Serves /llms.txt generated from published pages and posts.
 */

add_action('init', function (): void {
    add_rewrite_rule('^llms\.txt$', 'index.php?aum_llms=1', 'top');
});

add_filter('query_vars', function (array $vars): array {
    $vars[] = 'aum_llms';
    return $vars;
});

// Stop WordPress from 301-ing /llms.txt to /llms.txt/.
add_filter('redirect_canonical', function ($redirect, $requested) {
    return ((int) get_query_var('aum_llms') === 1) ? false : $redirect;
}, 10, 2);

function aum_llms_line(string $title, string $url, string $desc = ''): string {
    $desc = trim((string) preg_replace('/\s+/u', ' ', wp_strip_all_tags($desc)));
    if ($desc !== '' && function_exists('mb_substr')) {
        $desc = mb_substr($desc, 0, 200);
    }
    return $desc === ''
        ? sprintf('- [%s](%s)', $title, $url)
        : sprintf('- [%s](%s): %s', $title, $url, $desc);
}

function aum_llms_build(): string {
    $out   = [];
    $out[] = '# ' . wp_strip_all_tags(get_bloginfo('name'));
    $out[] = '';
    $summary = trim(wp_strip_all_tags(get_bloginfo('description')));
    $out[] = '> ' . ($summary !== '' ? $summary : 'What this site covers and who it is for.');
    $out[] = '';

    $sections = [
        'Pages' => get_posts([
            'post_type'      => 'page',
            'post_status'    => 'publish',
            'posts_per_page' => 50,
            'orderby'        => 'menu_order title',
            'order'          => 'ASC',
        ]),
        'Posts' => get_posts([
            'post_type'      => 'post',
            'post_status'    => 'publish',
            'posts_per_page' => 100,
            'orderby'        => 'date',
            'order'          => 'DESC',
        ]),
    ];

    foreach ($sections as $heading => $items) {
        if (empty($items)) {
            continue;
        }
        $out[] = '## ' . $heading;
        $out[] = '';
        foreach ($items as $item) {
            $desc  = $item->post_excerpt !== '' ? $item->post_excerpt : $item->post_content;
            $out[] = aum_llms_line($item->post_title, (string) get_permalink($item), $desc);
        }
        $out[] = '';
    }

    return implode("\n", $out) . "\n";
}

add_action('template_redirect', function (): void {
    if ((int) get_query_var('aum_llms') !== 1) {
        return;
    }
    status_header(200);
    header('Content-Type: text/plain; charset=utf-8');
    echo aum_llms_build(); // phpcs:ignore WordPress.Security.EscapeOutput
    exit;
});

Check the syntax before you upload, because a parse error here takes the whole site down:

php -l wp-content/mu-plugins/llms-txt.php

If you would rather keep it managed as a normal plugin, add a register_activation_hook(__FILE__, 'aum_llms_flush'); that calls flush_rewrite_rules(), and activate it through the admin. Same file body, one extra function.

3. Flush rewrite rules and make the first request

wp rewrite flush --hard

curl -sS -o /tmp/llms.txt -D /tmp/llms.headers \
  -w 'status=%{http_code} redirects=%{num_redirects} type=%{content_type} bytes=%{size_download}\n' \
  https://example.com/llms.txt

Pass condition: status=200 redirects=0 type=text/plain; charset=utf-8 and a nonzero byte count. A 301 here is the single most common failure, and it comes from one of two places: WordPress canonical redirects (handled by the filter above) or a server-level trailing-slash rule. If it still redirects, look at your web server config before touching PHP again.

4. Verify the body is text, not a themed HTML page

head -n 6 /tmp/llms.txt
grep -c '<html' /tmp/llms.txt || true
file -i /tmp/llms.txt
xxd -l 16 /tmp/llms.txt

What you want: the first line starts with #, grep -c '<html' prints 0, file -i reports charset=utf-8 (or us-ascii), and the hex dump does not start with efbbbf — a UTF-8 BOM at the top of a PHP file will corrupt the response and is invisible in most editors.

If the body is HTML, something is catching the 404 path before your handler runs: a redirect plugin, a "smart 404", or a theme fallback. template_redirect firing before the template loads is what prevents this, so confirm the rewrite rule actually matched rather than a catch-all route returning 200 with a page.

5. Verify every link inside the file resolves

A dead link in llms.txt is worse than no file, because it advertises URLs that 404 to the one reader you wrote it for.

grep -oE '\((https?://[^)]+)\)' /tmp/llms.txt \
  | tr -d '()' \
  | sort -u \
  | while read -r u; do
      code=$(curl -s -o /dev/null -w '%{http_code}' -L --max-time 15 "$u")
      [ "$code" = "200" ] || echo "BROKEN $code $u"
    done

Run this after any permalink structure change and after bulk edits. Silence means all links returned 200.

6. Verify robots.txt and your edge do not block the fetch

llms.txt does not need a robots.txt permission line, but blocking AI user agents blocks the file too — they never get far enough to request it.

curl -sS https://example.com/robots.txt | grep -nE 'User-agent|Disallow|Allow' | head -n 40

Two cases matter. A User-agent: * block on / kills it for everyone. A named block on GPTBot, ClaudeBot, PerplexityBot or CCBot kills it for that one. Either remove the block or add an explicit Allow: /llms.txt line inside that agent's group — matching is prefix-based, so the more specific path wins.

Then test the edge with the user agent strings themselves, which is how you find out whether a WAF is returning 403 to bots:

for UA in GPTBot ClaudeBot PerplexityBot CCBot; do
  printf '%-14s -> ' "$UA"
  curl -sS -o /dev/null -w '%{http_code}\n' -A "$UA" https://example.com/llms.txt
done

All four should print 200. Anything else is a bot rule at the CDN or proxy layer, not a WordPress problem, and no amount of PHP will fix it. This test tells you what your edge does with that header — it is not proof the real crawler behaved identically, but it is the closest thing you can check on demand.

7. Verify requests actually land — read the log

This is the step people skip, and it is the only one that produces evidence rather than intent.

Nginx:

sudo grep -h ' /llms.txt' /var/log/nginx/access.log /var/log/nginx/access.log.1 \
  | grep -Ei 'gptbot|claudebot|perplexitybot|ccbot|oai-searchbot|google-extended' \
  | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head

Apache, same idea:

sudo grep -h 'GET /llms.txt' /var/log/apache2/access.log* \
  | grep -Ei 'gptbot|claudebot|perplexitybot|ccbot|oai-searchbot|google-extended' \
  | tail -n 20

If nothing appears yet, that is a real answer, not a bug: crawler schedules are not yours to control. Keep the endpoint live, keep it in robots.txt-visible territory, and check again later. If you see the request but with a 403 or 301, you have found the exact layer to fix — that is the whole value of this log query.

8. Make the check recurring

A file that is correct today can break on the next plugin update. Put it on a timer:

#!/usr/bin/env bash
set -euo pipefail
URL="https://example.com/llms.txt"
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT

read -r code ctype < <(curl -sS -o "$tmp" -w '%{http_code} %{content_type}\n' "$URL")

[ "$code" = "200" ] || { echo "FAIL status $code"; exit 1; }
case "$ctype" in text/plain*) ;; *) echo "FAIL content-type $ctype"; exit 1;; esac
head -n 1 "$tmp" | grep -q '^# ' || { echo "FAIL missing H1"; exit 1; }
grep -qi '<html' "$tmp" && { echo "FAIL html in body"; exit 1; }

lines=$(grep -c '^- \[' "$tmp" || true)
[ "$lines" -gt 0 ] || { echo "FAIL no link entries"; exit 1; }

echo "OK $lines entries"

Run it from cron every fifteen minutes and alert on a non-zero exit. Once it is silent, you can stop thinking about it.

Failure modes, mapped

Symptom	Likely cause	Fix
301 to /llms.txt/	Canonical redirect or trailing-slash rule	The redirect_canonical filter, then the server config
200 but body starts with <!DOCTYPE html>	Catch-all 404 handler returning a page	Return early on template_redirect; check redirect plugins
Content-Type: text/html	Server overrides the header	Send the header explicitly and inspect the raw response
Empty or stale file	Page cache or CDN holding an old 404	Purge the path, exclude it from HTML caching
Links inside 404	Permalinks changed after generation	Flush and regenerate
Crawler gets 403	Bot rule at the edge	Allow the path or the user agent

After the file is verified

The log query in step 7 answers a narrower question than you think: it tells you that something fetched /llms.txt, not who is crawling the rest of the site. If you want per-crawler visibility — which AI agents are hitting your pages, and the option to allow or block them individually — that is a separate signal from the file itself. AumCrawl shows which AI crawlers are hitting a WordPress site and lets you decide per crawler. Published llms.txt first; then measure who actually arrives.