DEV Community: AumCreate

Identifying AI crawlers in your access log: user agent, source, and crawl frequency

AumCreate — Fri, 23 Oct 2026 04:19:08 +0000

Your access log already answers "which AI crawlers hit my site, how hard, and are they who they claim to be". You don't need a new SDK or a tag manager. You need three passes over the same file: classify by user agent, verify by source, measure frequency.

Here are the commands and one Python script that produce all three reports.

What the log gives you

nginx combined format, one line per request:

203.0.113.7 - - [23/Oct/2026:13:45:01 +0000] "GET /blog/wordpress-hosting-cost HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (compatible; GPTBot)"

Fields: client IP, timestamp, request line, status, bytes sent, referer, user agent. That's enough. If your log is JSON or a custom format, only step 2 changes.

Step 1 — look at the raw user agents first

awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

The top twenty UAs on your site. AI crawlers almost always declare themselves: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, Bytespider, Amazonbot, meta-externalagent. Treat that as data, not code — keep it in a config file you edit, because operators rename crawlers and new ones show up.

One check before anything else: if you sit behind a CDN or reverse proxy, $remote_addr may be the proxy, not the crawler. Confirm you're logging the real client IP before you trust any per-IP number.

Step 2 — classify

Save this as ai_crawlers.py:

#!/usr/bin/env python3
"""Classify AI crawler traffic in an nginx combined access log."""
import re
import sys
from collections import Counter, defaultdict
from datetime import datetime

LINE = re.compile(
    r'^(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] '
    r'"(?P<method>[A-Z]+) (?P<path>\S+)(?: [^"]*)?" '
    r'(?P<status>\d{3}) (?P<size>\S+) "(?P<ref>[^"]*)" "(?P<ua>[^"]*)"'
)

BOTS = {
    "gptbot": "GPTBot",
    "oai-searchbot": "OAI-SearchBot",
    "chatgpt-user": "ChatGPT-User",
    "claudebot": "ClaudeBot",
    "perplexitybot": "PerplexityBot",
    "ccbot": "CCBot",
    "bytespider": "Bytespider",
    "amazonbot": "Amazonbot",
    "meta-externalagent": "meta-externalagent",
}

def classify(ua):
    low = (ua or "").lower()
    for token, name in BOTS.items():
        if token in low:
            return name
    if not low or low == "-":
        return "no-ua"
    if re.search(r"bot|crawl|spider|slurp", low):
        return "other-bot"
    return "browser"

def main(path):
    rows = []
    with open(path, encoding="utf-8", errors="replace") as fh:
        for line in fh:
            m = LINE.match(line)
            if not m:
                continue
            d = m.groupdict()
            d["bot"] = classify(d["ua"])
            d["day"] = datetime.strptime(
                d["ts"], "%d/%b/%Y:%H:%M:%S %z"
            ).date().isoformat()
            rows.append(d)

    per_day = defaultdict(Counter)
    ips = defaultdict(Counter)
    paths = defaultdict(Counter)
    status = defaultdict(Counter)

    for r in rows:
        per_day[r["day"]][r["bot"]] += 1
        ips[r["bot"]][r["ip"]] += 1
        paths[r["bot"]][r["path"]] += 1
        status[r["bot"]][r["status"]] += 1

    totals = {b: sum(d[b] for d in per_day.values()) for b in ips}
    bots = sorted(totals, key=lambda b: -totals[b])

    print(f"{'bot':<18} {'hits':>8} {'ips':>6}  top path")
    for b in bots:
        top = paths[b].most_common(1)[0][0] if paths[b] else "-"
        print(f"{b:<18} {totals[b]:>8} {len(ips[b]):>6}  {top}")

    print("\ndaily volume")
    days = sorted(per_day)
    print("day".ljust(12) + "".join(b[:12].rjust(14) for b in bots))
    for day in days:
        print(day.ljust(12) + "".join(str(per_day[day][b]).rjust(14) for b in bots))

    print("\nstatus mix")
    for b in bots:
        mix = ", ".join(f"{s}:{n}" for s, n in status[b].most_common(4))
        print(f"  {b:<18} {mix}")

    print("\ntop ips per bot")
    for b in bots:
        print(f"\n[{b}]")
        for ip, n in ips[b].most_common(5):
            print(f"  {ip:<40} {n}")

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "/var/log/nginx/access.log")

python ai_crawlers.py /var/log/nginx/access.log

You get per-crawler hit totals, unique IP counts, the top path, a daily matrix, a status mix, and the busiest IPs. The status mix is the first filter — a bot with mostly 404s is not reading your site, it's walking a list.

Step 3 — verify the source

A user agent is a string the client chooses. Anyone can send GPTBot. So a matching UA means claimed, not verified. Verification means the IP is inside the range the operator publishes:

import ipaddress
import json

# published-ranges.json: {"GPTBot": ["203.0.113.0/24", ...], "ClaudeBot": [...]}
RANGES = json.load(open("published-ranges.json"))
NETS = {bot: [ipaddress.ip_network(c) for c in cidrs]
        for bot, cidrs in RANGES.items()}

def is_verified(ip, claimed_bot):
    nets = NETS.get(claimed_bot)
    if not nets:
        return None  # no published list for this name: unknown, not verified
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in nets)

Most crawler operators publish their ranges as a machine-readable file and document the reverse-DNS suffix they use. Fetch the file on a schedule, store it, diff it — the diff is your early warning that a range moved.

Reverse DNS is the second opinion, and it should only run on hits that failed the range check, not on every line:

import socket

def fcrdns(ip):
    """Forward-confirmed reverse DNS. Returns hostname or None."""
    try:
        host = socket.gethostbyaddr(ip)[0]
        _, _, addrs = socket.gethostbyname_ex(host)
    except (socket.herror, socket.gaierror):
        return None
    return host if ip in addrs else None

Record three outcomes, not two: verified (IP inside the published range), claimed (UA says bot, IP outside every range), unknown (no UA at all). Only the first is evidence that a crawler is actually crawling you.

Step 4 — measure crawl frequency

Hourly volume for one crawler, without touching the script:

grep -i 'ClaudeBot' /var/log/nginx/access.log \
  | awk -F'[][]' '{print $2}' \
  | cut -d: -f1,2 \
  | sort | uniq -c

How to read the numbers:

Steady hits per day from a handful of IPs — a real crawler on a schedule. Note the rate; that's your baseline.
One IP, thousands of URLs, mostly 404 — probing, or a sitemap pointing at pages you deleted.
A burst inside a few minutes — a re-crawl of pages that changed. Check what you deployed that day.
Many IPs, few hits each — distributed. Verify against ranges before you assume one operator.
The same path with varying query strings — cache-busting. Your cache may be serving fresh HTML on every request, which invites more crawling.

Pair the frequency with the path list. A crawler that only touches /blog/* is indexing. One that touches /cart, /checkout, or /api/* is doing something you probably want to stop regardless of who it is.

Step 5 — turn the log into a decision

Four buckets, four actions:

Evidence	Action
Verified, and you want it (answer engines that send referrals)	Leave open, make sure the page renders server-side
Verified, and you don't want it (training-only)	robots.txt token, or a block at the proxy
Claimed but not verified	Don't block on UA alone — block on "outside published range"
Unknown, high volume, odd paths	Rate-limit at the edge, then watch

robots.txt is a request. A block at nginx is enforcement. If you need enforcement, do it with a map and keep logging the blocked requests, so six weeks later you can still see what you cut off.

Step 6 — keep it running

Keep enough history to compare week over week; a single day tells you nothing about a trend.
Run the script from cron, write JSON, diff against yesterday. A new UA with more than a hundred hits is the thing worth reading.
Split logs by crawler family so you stop grepping a multi-gigabyte file:

map $http_user_agent $ai_log {
    default          main;
    ~*GPTBot         ai_gptbot;
    ~*ClaudeBot      ai_claude;
    ~*PerplexityBot  ai_perplexity;
    ~*CCBot          ai_ccbot;
}

open_log_file_cache max=100 inactive=60s;
access_log /var/log/nginx/$ai_log.log combined;

The map goes in the http block; the variable in the path is why open_log_file_cache is there.

If you'd rather not maintain the lists

Keeping the token map, the published ranges, and the daily diff in sync is the boring half of this job. If you run WordPress, AumCrawl shows which AI crawlers are hitting the site and lets you allow or block per crawler. The access log stays the source of truth either way — this is the dashboard on top of it, not a replacement for the evidence.

Generate llms.txt on WordPress and Verify AI Crawlers Actually Fetch It

AumCreate — Fri, 23 Oct 2026 03:57:47 +0000

You can publish an llms.txt in ten minutes. Proving that a crawler actually fetched it — right URL, right content type, no redirect, no WAF block — takes another twenty. This post does both, in order, with commands you can paste into a terminal.

Honest framing first: llms.txt is a community proposal, not a ratified standard, and no AI provider promises to read it. What you can verify objectively is five facts:

The file is served as text/plain; charset=utf-8.
It is reachable at the exact URL a crawler would request, with no redirect.
It is reachable without cookies, JavaScript or a logged-in session.
robots.txt and your edge (CDN/WAF) do not block AI user agents from it.
Requests for it show up in your access log.

Everything below targets those five facts.

1. Decide what goes in the file

The proposed shape is small: an H1 with the site name, a blockquote summary, then sections of markdown links with one-line descriptions.

# Example Studio

> A two-person studio building custom WordPress themes and WooCommerce checkouts.

## Pages

- [Services](https://example.com/services): Scope, timeline and what a project costs.
- [Contact](https://example.com/contact): How to start a project.

## Posts

- [Speeding up a slow WooCommerce cart](https://example.com/blog/slow-cart): Where the time actually goes.

Two rules worth following: one line per link (wrapping breaks parsers that read line by line), and no content that only exists behind a login.

2. Serve it from WordPress

If you would rather not maintain PHP for this, AumViso generates llms.txt on a WordPress site, alongside schema, sitemap, meta and internal links. It is free on wordpress.org.

If you want the file under your own control, drop this into wp-content/mu-plugins/llms-txt.php. MU-plugins load automatically, so there is no activation step — you flush rewrite rules once from the CLI.

<?php
/**
 * Plugin Name: llms.txt endpoint
 * Description: Serves /llms.txt generated from published pages and posts.
 */

add_action('init', function (): void {
    add_rewrite_rule('^llms\.txt$', 'index.php?aum_llms=1', 'top');
});

add_filter('query_vars', function (array $vars): array {
    $vars[] = 'aum_llms';
    return $vars;
});

// Stop WordPress from 301-ing /llms.txt to /llms.txt/.
add_filter('redirect_canonical', function ($redirect, $requested) {
    return ((int) get_query_var('aum_llms') === 1) ? false : $redirect;
}, 10, 2);

function aum_llms_line(string $title, string $url, string $desc = ''): string {
    $desc = trim((string) preg_replace('/\s+/u', ' ', wp_strip_all_tags($desc)));
    if ($desc !== '' && function_exists('mb_substr')) {
        $desc = mb_substr($desc, 0, 200);
    }
    return $desc === ''
        ? sprintf('- [%s](%s)', $title, $url)
        : sprintf('- [%s](%s): %s', $title, $url, $desc);
}

function aum_llms_build(): string {
    $out   = [];
    $out[] = '# ' . wp_strip_all_tags(get_bloginfo('name'));
    $out[] = '';
    $summary = trim(wp_strip_all_tags(get_bloginfo('description')));
    $out[] = '> ' . ($summary !== '' ? $summary : 'What this site covers and who it is for.');
    $out[] = '';

    $sections = [
        'Pages' => get_posts([
            'post_type'      => 'page',
            'post_status'    => 'publish',
            'posts_per_page' => 50,
            'orderby'        => 'menu_order title',
            'order'          => 'ASC',
        ]),
        'Posts' => get_posts([
            'post_type'      => 'post',
            'post_status'    => 'publish',
            'posts_per_page' => 100,
            'orderby'        => 'date',
            'order'          => 'DESC',
        ]),
    ];

    foreach ($sections as $heading => $items) {
        if (empty($items)) {
            continue;
        }
        $out[] = '## ' . $heading;
        $out[] = '';
        foreach ($items as $item) {
            $desc  = $item->post_excerpt !== '' ? $item->post_excerpt : $item->post_content;
            $out[] = aum_llms_line($item->post_title, (string) get_permalink($item), $desc);
        }
        $out[] = '';
    }

    return implode("\n", $out) . "\n";
}

add_action('template_redirect', function (): void {
    if ((int) get_query_var('aum_llms') !== 1) {
        return;
    }
    status_header(200);
    header('Content-Type: text/plain; charset=utf-8');
    echo aum_llms_build(); // phpcs:ignore WordPress.Security.EscapeOutput
    exit;
});

Check the syntax before you upload, because a parse error here takes the whole site down:

php -l wp-content/mu-plugins/llms-txt.php

If you would rather keep it managed as a normal plugin, add a register_activation_hook(__FILE__, 'aum_llms_flush'); that calls flush_rewrite_rules(), and activate it through the admin. Same file body, one extra function.

3. Flush rewrite rules and make the first request

wp rewrite flush --hard

curl -sS -o /tmp/llms.txt -D /tmp/llms.headers \
  -w 'status=%{http_code} redirects=%{num_redirects} type=%{content_type} bytes=%{size_download}\n' \
  https://example.com/llms.txt

Pass condition: status=200 redirects=0 type=text/plain; charset=utf-8 and a nonzero byte count. A 301 here is the single most common failure, and it comes from one of two places: WordPress canonical redirects (handled by the filter above) or a server-level trailing-slash rule. If it still redirects, look at your web server config before touching PHP again.

4. Verify the body is text, not a themed HTML page

head -n 6 /tmp/llms.txt
grep -c '<html' /tmp/llms.txt || true
file -i /tmp/llms.txt
xxd -l 16 /tmp/llms.txt

What you want: the first line starts with #, grep -c '<html' prints 0, file -i reports charset=utf-8 (or us-ascii), and the hex dump does not start with efbbbf — a UTF-8 BOM at the top of a PHP file will corrupt the response and is invisible in most editors.

If the body is HTML, something is catching the 404 path before your handler runs: a redirect plugin, a "smart 404", or a theme fallback. template_redirect firing before the template loads is what prevents this, so confirm the rewrite rule actually matched rather than a catch-all route returning 200 with a page.

5. Verify every link inside the file resolves

A dead link in llms.txt is worse than no file, because it advertises URLs that 404 to the one reader you wrote it for.

grep -oE '\((https?://[^)]+)\)' /tmp/llms.txt \
  | tr -d '()' \
  | sort -u \
  | while read -r u; do
      code=$(curl -s -o /dev/null -w '%{http_code}' -L --max-time 15 "$u")
      [ "$code" = "200" ] || echo "BROKEN $code $u"
    done

Run this after any permalink structure change and after bulk edits. Silence means all links returned 200.

6. Verify robots.txt and your edge do not block the fetch

llms.txt does not need a robots.txt permission line, but blocking AI user agents blocks the file too — they never get far enough to request it.

curl -sS https://example.com/robots.txt | grep -nE 'User-agent|Disallow|Allow' | head -n 40

Two cases matter. A User-agent: * block on / kills it for everyone. A named block on GPTBot, ClaudeBot, PerplexityBot or CCBot kills it for that one. Either remove the block or add an explicit Allow: /llms.txt line inside that agent's group — matching is prefix-based, so the more specific path wins.

Then test the edge with the user agent strings themselves, which is how you find out whether a WAF is returning 403 to bots:

for UA in GPTBot ClaudeBot PerplexityBot CCBot; do
  printf '%-14s -> ' "$UA"
  curl -sS -o /dev/null -w '%{http_code}\n' -A "$UA" https://example.com/llms.txt
done

All four should print 200. Anything else is a bot rule at the CDN or proxy layer, not a WordPress problem, and no amount of PHP will fix it. This test tells you what your edge does with that header — it is not proof the real crawler behaved identically, but it is the closest thing you can check on demand.

7. Verify requests actually land — read the log

This is the step people skip, and it is the only one that produces evidence rather than intent.

Nginx:

sudo grep -h ' /llms.txt' /var/log/nginx/access.log /var/log/nginx/access.log.1 \
  | grep -Ei 'gptbot|claudebot|perplexitybot|ccbot|oai-searchbot|google-extended' \
  | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head

Apache, same idea:

sudo grep -h 'GET /llms.txt' /var/log/apache2/access.log* \
  | grep -Ei 'gptbot|claudebot|perplexitybot|ccbot|oai-searchbot|google-extended' \
  | tail -n 20

If nothing appears yet, that is a real answer, not a bug: crawler schedules are not yours to control. Keep the endpoint live, keep it in robots.txt-visible territory, and check again later. If you see the request but with a 403 or 301, you have found the exact layer to fix — that is the whole value of this log query.

8. Make the check recurring

A file that is correct today can break on the next plugin update. Put it on a timer:

#!/usr/bin/env bash
set -euo pipefail
URL="https://example.com/llms.txt"
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT

read -r code ctype < <(curl -sS -o "$tmp" -w '%{http_code} %{content_type}\n' "$URL")

[ "$code" = "200" ] || { echo "FAIL status $code"; exit 1; }
case "$ctype" in text/plain*) ;; *) echo "FAIL content-type $ctype"; exit 1;; esac
head -n 1 "$tmp" | grep -q '^# ' || { echo "FAIL missing H1"; exit 1; }
grep -qi '<html' "$tmp" && { echo "FAIL html in body"; exit 1; }

lines=$(grep -c '^- \[' "$tmp" || true)
[ "$lines" -gt 0 ] || { echo "FAIL no link entries"; exit 1; }

echo "OK $lines entries"

Run it from cron every fifteen minutes and alert on a non-zero exit. Once it is silent, you can stop thinking about it.

Failure modes, mapped

Symptom	Likely cause	Fix
`301` to `/llms.txt/`	Canonical redirect or trailing-slash rule	The `redirect_canonical` filter, then the server config
`200` but body starts with `<!DOCTYPE html>`	Catch-all 404 handler returning a page	Return early on `template_redirect`; check redirect plugins
`Content-Type: text/html`	Server overrides the header	Send the header explicitly and inspect the raw response
Empty or stale file	Page cache or CDN holding an old 404	Purge the path, exclude it from HTML caching
Links inside 404	Permalinks changed after generation	Flush and regenerate
Crawler gets `403`	Bot rule at the edge	Allow the path or the user agent

After the file is verified

The log query in step 7 answers a narrower question than you think: it tells you that something fetched /llms.txt, not who is crawling the rest of the site. If you want per-crawler visibility — which AI agents are hitting your pages, and the option to allow or block them individually — that is a separate signal from the file itself. AumCrawl shows which AI crawlers are hitting a WordPress site and lets you decide per crawler. Published llms.txt first; then measure who actually arrives.

Inquiry Form to CRM: A Routing Process That Doesn't Drop Leads

AumCreate — Fri, 23 Oct 2026 03:44:57 +0000

A B2B inquiry form that emails a shared inbox is not a lead pipeline. It's a race condition.

Two people reply to the same inquiry, or nobody does. The CRM gets a contact with no owner. The form handler times out while the CRM API is restarting and the submission is gone — the customer sees a success message and never hears back.

This is a process writeup, not a plugin recommendation. It's the sequence I'd use to wire any WordPress form to any CRM: persist first, route deterministically, sync through an outbox, and watch the clock.

The four failure modes worth designing against

Failure	What it looks like	Fix
Lost on the way out	CRM API times out, submission discarded	persist before sending
Nobody owns it	record created, `owner` empty	routing rules evaluated at capture time
Two people reply	same company submits twice, two deals	dedupe key, unique index
Silence	owned, but no first contact for days	SLA query + escalation

Every step below exists to kill one of those four.

Step 0 — Define the field contract

Before wiring anything, split the form fields into three groups. This split is what makes routing possible at all.

Routing fields — structured, validated, never free text:

email (required)
company (required)
country (required, 2-letter ISO)
product_line (select, one of your lines)
qty (number, optional but strongly encouraged)

Context fields — hidden inputs, captured automatically:

landing page URL, referrer
utm_source / utm_medium / utm_campaign
form id and locale
submit timestamp (server side)

Free text — message. Store it, show it to the owner, but never route on it. A regex over a paragraph written in three languages is not a routing rule.

If qty is empty, don't guess the tier. The autoresponder asks one question, and the reply updates the record.

Step 1 — Persist before you send

Your database is the source of truth. The CRM is a downstream copy. That ordering is the whole trick: if the CRM is down for twenty minutes, nothing is lost.

register_activation_hook(__FILE__, function () {
    global $wpdb;
    $table   = $wpdb->prefix . 'inquiries';
    $charset = $wpdb->get_charset_collate();

    $sql = "CREATE TABLE {$table} (
        id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
        created_at DATETIME NOT NULL,
        email VARCHAR(190) NOT NULL,
        company VARCHAR(190) NULL,
        country CHAR(2) NULL,
        product_line VARCHAR(64) NULL,
        qty INT UNSIGNED NULL,
        message TEXT NULL,
        payload LONGTEXT NOT NULL,
        dedupe_key CHAR(64) NOT NULL,
        owner VARCHAR(190) NULL,
        crm_state VARCHAR(16) NOT NULL DEFAULT 'pending',
        crm_id VARCHAR(64) NULL,
        attempts TINYINT UNSIGNED NOT NULL DEFAULT 0,
        next_attempt_at DATETIME NULL,
        first_touch_at DATETIME NULL,
        PRIMARY KEY (id),
        UNIQUE KEY dedupe_key (dedupe_key),
        KEY crm_state_next (crm_state, next_attempt_at)
    ) {$charset};";

    require_once ABSPATH . 'wp-admin/includes/upgrade.php';
    dbDelta($sql);
});

The raw payload column is not laziness — when someone claims they asked for 500 units and the record says 5, you want the original request body, untouched.

Step 2 — One dedupe key, hashed once

Dedupe on email + company inside a rolling window. A monthly bucket is usually right for B2B: the same buyer asking about a different product line next month is a new conversation, the same buyer double-submitting after a timeout is not.

function acx_dedupe_key(array $lead) {
    $bucket = (int) floor(time() / 2592000); // 30-day window

    return hash('sha256', implode('|', array(
        strtolower(trim($lead['email'])),
        strtolower(trim((string) $lead['company'])),
        $bucket,
    )));
}

With the unique index in place, INSERT IGNORE makes the race safe — two simultaneous submissions produce one row, and the second call returns the existing id instead of a duplicate deal.

function acx_capture_inquiry(array $raw) {
    global $wpdb;
    $lead   = acx_normalize($raw);
    $lead['dedupe_key'] = acx_dedupe_key($lead);
    $table  = $wpdb->prefix . 'inquiries';

    $wpdb->query($wpdb->prepare(
        "INSERT IGNORE INTO {$table}
         (created_at, email, company, country, product_line, qty, message, payload, dedupe_key, owner)
         VALUES (UTC_TIMESTAMP(), %s, %s, %s, %s, %d, %s, %s, %s, %s)",
        $lead['email'],
        $lead['company'],
        $lead['country'],
        $lead['product_line'],
        (int) $lead['qty'],
        $lead['message'],
        wp_json_encode($raw),
        $lead['dedupe_key'],
        acx_pick_owner($lead)
    ));

    return (int) $wpdb->get_var($wpdb->prepare(
        "SELECT id FROM {$table} WHERE dedupe_key = %s",
        $lead['dedupe_key']
    ));
}

function acx_normalize(array $raw) {
    $country = strtoupper(preg_replace('/[^A-Za-z]/', '', (string) ($raw['country'] ?? '')));

    return array(
        'email'        => sanitize_email($raw['email'] ?? ''),
        'company'      => sanitize_text_field($raw['company'] ?? ''),
        'country'      => substr($country, 0, 2),
        'product_line' => sanitize_key($raw['product_line'] ?? ''),
        'qty'          => absint($raw['qty'] ?? 0),
        'message'      => sanitize_textarea_field($raw['message'] ?? ''),
    );
}

Hook it from whatever your form fires after a successful submit — one line, one entry point, and every form on the site goes through the same contract:

add_action('acx_inquiry_submitted', 'acx_capture_inquiry', 10, 1);

Step 3 — Route deterministically, not by luck

Two rules, evaluated in this order:

Territory — country maps to a region, region maps to a pool of owners.
Tier — quantity above your threshold swaps the pool for the senior one.

function acx_region($country) {
    $apac = array('SG','MY','ID','TH','VN','PH','AU','NZ','JP','KR','IN','CN','HK','TW');
    $emea = array('GB','DE','FR','NL','ES','IT','PL','SE','AE','SA','ZA','TR');
    $amer = array('US','CA','MX','BR','CL');

    if (in_array($country, $apac, true)) { return 'apac'; }
    if (in_array($country, $emea, true)) { return 'emea'; }
    if (in_array($country, $amer, true)) { return 'amer'; }
    return 'row';
}

function acx_pick_owner(array $lead) {
    $region = acx_region($lead['country']);
    $tier   = ((int) $lead['qty'] >= 500) ? 'key_' : '';

    $pools = apply_filters('acx_owner_pool', array(
        'apac'     => array('sia@example.com', 'jun@example.com'),
        'key_apac' => array('sia@example.com'),
        'emea'     => array('omar@example.com'),
        'amer'     => array('dana@example.com'),
        'row'      => array('sia@example.com'),
    ), $lead);

    $key = $tier . $region;
    if (empty($pools[$key])) {
        $key = isset($pools[$region]) ? $region : 'row';
    }
    if (empty($pools[$key])) {
        return null;
    }

    $candidates = $pools[$key];
    $index = hexdec(substr($lead['dedupe_key'], 0, 8)) % count($candidates);

    return $candidates[$index];
}

Hashing the dedupe key instead of using a counter is the important detail. Assignment becomes stateless and idempotent: the retry, the duplicate submission and the backfill all compute the same owner, so you can never end up with two rows owned by two people. The tradeoff is that distribution is only approximate — fine for pools of two to five people, not fine for a 50-seat call centre.

Because the owner is decided at capture time, the CRM record arrives with an owner already set. No assignment rule to debug on the other side.

Step 4 — Sync through an outbox

Nothing calls the CRM during the request. A worker drains the queue:

add_action('acx_crm_flush', 'acx_flush_outbox');

function acx_flush_outbox() {
    global $wpdb;
    $table = $wpdb->prefix . 'inquiries';

    $rows = $wpdb->get_results(
        "SELECT * FROM {$table}
         WHERE crm_state = 'pending'
           AND (next_attempt_at IS NULL OR next_attempt_at <= UTC_TIMESTAMP())
         ORDER BY id ASC LIMIT 20"
    );

    foreach ($rows as $row) {
        $result = acx_push_to_crm($row);

        if (is_wp_error($result)) {
            $backoff = min(3600, 60 * pow(4, (int) $row->attempts));
            $wpdb->query($wpdb->prepare(
                "UPDATE {$table}
                 SET attempts = attempts + 1,
                     next_attempt_at = DATE_ADD(UTC_TIMESTAMP(), INTERVAL %d SECOND)
                 WHERE id = %d",
                $backoff, $row->id
            ));
            continue;
        }

        $wpdb->update($table, array(
            'crm_state' => 'synced',
            'crm_id'    => $result['id'],
        ), array('id' => $row->id));
    }
}

function acx_push_to_crm($row) {
    $response = wp_remote_post('https://crm.example.com/api/v1/leads', array(
        'timeout' => 10,
        'headers' => array(
            'Content-Type'    => 'application/json',
            'Idempotency-Key' => $row->dedupe_key,
            'Authorization'   => 'Bearer ' . ACME_CRM_TOKEN,
        ),
        'body' => wp_json_encode(array(
            'email'        => $row->email,
            'company'      => $row->company,
            'country'      => $row->country,
            'product_line' => $row->product_line,
            'qty'          => (int) $row->qty,
            'message'      => $row->message,
            'owner'        => $row->owner,
            'source'       => 'website_form',
        )),
    ));

    if (is_wp_error($response)) {
        return $response;
    }

    $code = wp_remote_retrieve_response_code($response);
    if ($code < 200 || $code >= 300) {
        return new WP_Error('acx_crm_push', 'CRM returned HTTP ' . $code);
    }

    $data = json_decode(wp_remote_retrieve_body($response), true);
    return array('id' => isset($data['id']) ? $data['id'] : null);
}

Retries are safe because the Idempotency-Key is the same on every attempt — even a "failed" call that actually reached the CRM won't create a second lead.

One honest caveat: WP-Cron fires on page views, so it is not a reliable one-minute scheduler. Point a real system cron at wp-cron.php every minute, or run the drain from a small worker outside WordPress.

Step 5 — Watch the clock, not the pipeline

A lead with an owner and no first contact is worse than an unassigned one, because it looks handled. Run this on the same schedule and escalate:

$stale = $wpdb->get_results(
    "SELECT id, owner, email, company, created_at
     FROM {$wpdb->prefix}inquiries
     WHERE crm_state = 'synced'
       AND first_touch_at IS NULL
       AND created_at < DATE_SUB(UTC_TIMESTAMP(), INTERVAL 4 HOUR)"
);

What "escalate" means is your call, but it must be an action, not a report: reassign to the pool, notify the owner's backup, and write a line into the record so the next audit sees it happened. Set first_touch_at from the CRM webhook when the owner sends the first reply — if you have no webhook, a manual "mark contacted" button in the admin list beats nothing.

Step 6 — Three queries you run weekly

-- stuck: never reached the CRM
SELECT id, email, attempts, next_attempt_at
FROM wp_inquiries
WHERE crm_state = 'pending' AND created_at < DATE_SUB(UTC_TIMESTAMP(), INTERVAL 1 HOUR);

-- unowned: routing table has a hole
SELECT country, product_line, COUNT(*) AS n
FROM wp_inquiries
WHERE owner IS NULL GROUP BY country, product_line;

-- dedupe rate: a spike means double submits or a bot
SELECT DATE(created_at) AS d, COUNT(*) AS rows_total,
       COUNT(DISTINCT dedupe_key) AS unique_leads
FROM wp_inquiries
GROUP BY DATE(created_at) ORDER BY d DESC LIMIT 14;

The first query is the one that matters. Any row older than an hour in pending means either the worker isn't running or the CRM is rejecting your payload — and both are silent without this check.

What you've actually built

A submission that survives a CRM outage, because the row is written first
One record per real inquiry, enforced by a unique index rather than a hope
An owner decided at capture time, computed idempotently from the lead itself
Retries with backoff, safe to run twice
An SLA query that turns silent leads into an action

None of that is CRM-specific, and none of it needs a new dependency. It's a table, a routing function and a worker.

If you'd rather have this built against your own field list and CRM rather than assembling it, that's the kind of WordPress work we take on: aumcreate.com/services.

Giving a WordPress Support Bot Your Real Product Data (Specs, Stock, Shipping)

AumCreate — Fri, 23 Oct 2026 03:42:04 +0000

Most "AI support" setups fail in the same three places: they invent a spec, they claim something is in stock when it isn't, and they promise a delivery window nobody can honour.

The cause isn't the model. It's that the answers live in WooCommerce — in product attributes, stock tables, shipping classes and post meta — and the bot was never given a way to read them at question time.

This tutorial builds the missing piece: a small, read-only context endpoint per product, a rules layer that answers stock and shipping deterministically, and a model that only gets to paraphrase what the endpoint returned. Everything below is plain WordPress + WooCommerce code you can drop into an mu-plugins file.

The design rule: ground, don't fine-tune

Specs change when someone edits a product. Stock changes when someone buys. Shipping classes change maybe twice a year.

A fine-tune bakes all three into weights and goes stale silently. A retrieval call per product is cheaper, debuggable, and you can curl it to see exactly what the bot saw.

Step 1 — Decide what the bot may answer

Write this down before writing code, because it defines the payload:

Question type	Source of truth	Answered by
Specs (material, dimensions, compatibility)	product attributes	model, quoting the payload
Stock	`_stock_status`, `_stock`, backorder setting	rules
Restock date	a custom meta field you maintain	rules
Delivery window	shipping class + your own ETA map	rules
Price, discounts, contracts	not in this payload	hand off to a human

Two of those rows say rules, not model. That's deliberate: a language model reformatting "in_stock" into a sentence is safe. A language model deciding whether something is in stock is not.

Step 2 — A read-only context endpoint

<?php
/**
 * Plugin Name: Catalog Context
 * Description: Read-only product context for a support assistant.
 */

add_action('rest_api_init', function () {
    register_rest_route('catalog-ctx/v1', '/product/(?P<id>\d+)', array(
        'methods'             => 'GET',
        'permission_callback' => '__return_true',
        'callback'            => 'acx_product_context',
    ));
});

function acx_product_context(WP_REST_Request $request) {
    $ctx = acx_build_context((int) $request['id']);
    if (is_wp_error($ctx)) {
        return $ctx;
    }
    return rest_ensure_response($ctx);
}

function acx_build_context($id) {
    $id     = (int) $id;
    $cached = get_transient('acx_ctx_' . $id);
    if (is_array($cached)) {
        return $cached;
    }

    $product = wc_get_product($id);
    if (!$product || 'publish' !== $product->get_status()) {
        return new WP_Error('acx_not_found', 'Unknown product', array('status' => 404));
    }

    $payload = array(
        'id'       => $id,
        'name'     => $product->get_name(),
        'url'      => get_permalink($id),
        'specs'    => acx_specs($product),
        'stock'    => acx_stock($product),
        'shipping' => acx_shipping($product),
        'updated'  => gmdate('c'),
    );

    set_transient('acx_ctx_' . $id, $payload, 10 * MINUTE_IN_SECONDS);

    return $payload;
}

Now the three builders. Specs come straight from visible attributes, so an editor changing a product updates the bot with no code change:

function acx_specs(WC_Product $product) {
    $out = array();

    foreach ($product->get_attributes() as $attribute) {
        if (!$attribute->get_visible()) {
            continue;
        }
        $label = wc_attribute_label($attribute->get_name());

        if ($attribute->is_taxonomy()) {
            $values = wc_get_product_terms($product->get_id(), $attribute->get_name(), array('fields' => 'names'));
        } else {
            $values = $attribute->get_options();
        }

        if (!empty($values)) {
            $out[$label] = implode(', ', array_map('wp_strip_all_tags', $values));
        }
    }

    return $out;
}

Stock is returned as a band, not a number. Publishing exact quantities invites people to do arithmetic on your inventory, and the bot never needs the digit — it needs to know whether to say "yes", "only a few left", or "no":

function acx_stock(WC_Product $product) {
    $qty = null;
    if ($product->get_manage_stock() && null !== $product->get_stock_quantity()) {
        $qty = (int) $product->get_stock_quantity();
    }

    if (!$product->is_in_stock()) {
        $band = 'out_of_stock';
    } elseif (null === $qty) {
        $band = 'in_stock';
    } elseif ($qty > 20) {
        $band = 'in_stock';
    } else {
        $band = 'low_stock';
    }

    return array(
        'band'        => $band,
        'backorders'  => $product->backorders_allowed() ? 'allowed' : 'not_allowed',
        'restock_eta' => get_post_meta($product->get_id(), '_acx_restock_eta', true) ?: null,
    );
}

Shipping: map the product's shipping class to a window you maintain yourself, and keep the map filterable so you can change it without touching the file:

function acx_shipping(WC_Product $product) {
    $class_id  = $product->get_shipping_class_id();
    $class     = $class_id ? get_term($class_id, 'product_shipping_class') : null;
    $class_key = ($class && !is_wp_error($class)) ? $class->slug : 'standard';

    $eta_map = apply_filters('acx_shipping_eta', array(
        'standard' => '3-5 business days after dispatch',
        'bulky'    => '7-12 business days after dispatch',
    ), $product);

    return array(
        'class'      => $class_key,
        'eta'        => isset($eta_map[$class_key]) ? $eta_map[$class_key] : null,
        'dispatches' => get_post_meta($product->get_id(), '_acx_dispatch_days', true) ?: null,
    );
}

If a product has no ETA entry, eta is null — and null is a signal to the bot to say "let me confirm that", not to guess. More on that below.

Step 3 — Invalidate the cache on the events that matter

A ten-minute cache is fine until someone drops the stock to zero. Hook the three writes that change the answer:

add_action('woocommerce_update_product', 'acx_flush_context');
add_action('woocommerce_product_set_stock', 'acx_flush_context');
add_action('save_post_product', 'acx_flush_context');

function acx_flush_context($subject) {
    $id = $subject instanceof WC_Product ? $subject->get_id() : (int) $subject;
    if ($id > 0) {
        delete_transient('acx_ctx_' . $id);
    }
}

Variations need the same treatment — hook woocommerce_variation_set_stock and flush the parent id too, or a variable product will keep serving the pre-purchase stock band for ten minutes.

Step 4 — Answer stock and shipping with rules, not with the model

Intent routing, before anything is sent to a language model:

function acx_route_message($message, $product_id) {
    $text = strtolower($message);
    $ctx  = acx_build_context($product_id);

    if (is_wp_error($ctx)) {
        return array('mode' => 'handoff', 'answer' => null, 'ctx' => null);
    }

    if (preg_match('/\b(in stock|available|availability|how many left)\b/', $text)) {
        return array('mode' => 'rule', 'answer' => acx_stock_sentence($ctx), 'ctx' => $ctx);
    }

    if (preg_match('/\b(ship|shipping|delivery|dispatch|arrive|lead time)\b/', $text)) {
        return array('mode' => 'rule', 'answer' => acx_shipping_sentence($ctx), 'ctx' => $ctx);
    }

    return array('mode' => 'model', 'answer' => null, 'ctx' => $ctx);
}

function acx_stock_sentence(array $ctx) {
    $stock = $ctx['stock'];

    if ('out_of_stock' === $stock['band']) {
        $msg = sprintf('%s is out of stock right now.', $ctx['name']);
        if ('allowed' === $stock['backorders']) {
            $msg .= ' Backorders are open, so you can still place an order.';
        }
        if (!empty($stock['restock_eta'])) {
            $msg .= ' Next restock: ' . $stock['restock_eta'] . '.';
        }
        return $msg;
    }

    if ('low_stock' === $stock['band']) {
        return sprintf('%s is in stock, but only a few units are left.', $ctx['name']);
    }

    return sprintf('%s is in stock.', $ctx['name']);
}

function acx_shipping_sentence(array $ctx) {
    $ship = $ctx['shipping'];

    if (empty($ship['eta'])) {
        return sprintf('I need to confirm the delivery window for %s — handing this to the team.', $ctx['name']);
    }

    return sprintf('%s ships on our %s service: %s.', $ctx['name'], $ship['class'], $ship['eta']);
}

Two of the three intents never touch the model at all, which means zero hallucination surface and near-zero latency.

Step 5 — A grounding contract for the remaining questions

For everything else (specs, compatibility, "will this fit"), send the model a tight system prompt and the JSON:

You answer questions about exactly one product.
Use only the JSON in <product>. Do not use outside knowledge.

Rules:
1. If a field is null or missing, say you will confirm it with the team. Never estimate.
2. Never state a delivery date that is not in shipping.eta.
3. Quote specs verbatim; do not round or convert units.
4. Maximum 3 sentences. End with the product URL when the answer is about a spec.
5. If the user asks about price exceptions, bulk contracts, or refunds, reply HANDOFF.

If you cannot answer from <product>, reply with exactly: HANDOFF

Then treat the literal string HANDOFF as a routing signal, not as text to show the user.

Step 6 — Test it with curl before you wire the chat layer

curl -s "https://example.com/wp-json/catalog-ctx/v1/product/123" \
  | python3 -m json.tool

Check four cases:

A product with no shipping class → shipping.class is standard, eta is set.
A product whose class has no ETA entry → eta is null, and the rule sentence says it needs confirmation.
A product set to backorder → stock.band is out_of_stock, backorders is allowed.
A variable product → each variation id returns its own context; flush on variation stock change.

If any of those four are wrong, the bot will be wrong in production, and no prompt tuning will fix it.

Step 7 — Make the handoff a first-class path

Set the escalation trigger in the routing layer, not in the prompt: refunds, bulk or contract pricing, complaints, and anything the model tagged HANDOFF.

This is the part most builds get wrong — they escalate by opening a ticket and telling the user to email in. The conversation should continue with a person in the same thread, with the product context already attached so the customer doesn't repeat themselves.

That relay — rules first, then the model, then a human taking over inside the same conversation — is how AumChat is built; it runs as a standalone service, with an installer plugin that puts it on a WordPress site. There's a public demo where you can see the shop-owner side without registering: chat.aumcreate.com/demo. If you want the setup described above running against your own catalog, start here: aumcreate.com/aumchat.

The checklist

[ ] Specs read from visible attributes, never hard-coded in a prompt
[ ] Stock returned as a band, with backorder state
[ ] Shipping ETA from your own filterable map, null when unknown
[ ] Cache invalidated on product save and stock change
[ ] Stock and shipping answered by rules, not by the model
[ ] null fields produce "let me confirm", never an estimate
[ ] Handoff continues in the same conversation

Get those seven right and the bot stops being a liability on the three questions customers ask most.

robots.txt and AI crawlers: what it actually controls, and what it only asks for

AumCreate — Fri, 23 Oct 2026 03:25:22 +0000

The useful mental model for robots.txt is: it is a fetch directive, not an access control system. It asks a crawler not to request a URL. Whether the request is honored depends entirely on the crawler, and it says nothing about what happens to content once it has already been fetched.

That single sentence explains almost every surprise people hit when they try to manage AI crawlers with robots.txt. Here is the breakdown, with tests you can run against your own site.

What robots.txt can do

1. Address crawlers individually. Grouping by User-agent uses longest-match precedence: a group naming a specific token beats the * group. So one file can allow search crawlers and disallow model-training crawlers at the same time.

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

2. Keep compliant crawlers out of paths you never wanted fetched. Admin areas, internal search results, faceted URLs, staging leftovers. This works because the compliant crawler reads the file before requesting anything else.

3. Publish an auditable statement of intent. The file is public, versionable, and timestamped by your git history. If a dispute about usage ever comes up, "our policy has said this since date X" is checkable evidence. Nothing else in your stack gives you that for free.

4. Feed vendor-side switches. Several AI vendors read robots.txt as the input to whether they may use a site's content. Changing the file changes their behavior without a support ticket or a dashboard login.

What robots.txt cannot do

1. It cannot stop a crawler that ignores it. There is no enforcement mechanism. A scraper that never requests /robots.txt is completely unaffected by anything in it.

2. It cannot un-publish anything already collected. Blocking today does not remove yesterday's fetch from a dataset. The directive is forward-looking only.

3. It cannot prevent indexing of a URL discovered elsewhere. For search engines, Disallow prevents the fetch, not the URL from being listed. If other pages link to it, it can still appear as a bare URL with no snippet.

4. It cannot rate-limit. Crawl-delay is honored by some crawlers and ignored by others, and it has no effect on crawlers that don't read the file at all. Rate limiting belongs at the edge.

5. It cannot block by IP or ASN. There is no syntax for it. If a request comes from somewhere you don't want, the decision has to happen in your server or CDN config.

6. It cannot tell "AI search citation" apart from "AI model training". Those are separate vendor-defined tokens, and each vendor documents its own. You have to decide per token, not per "AI" in general.

7. It does not cover your whole site automatically. robots.txt is scoped per origin — scheme, host, and port. https://example.com/robots.txt does not govern https://www.example.com/ or http://example.com/. Check every hostname you actually serve.

8. It is not a security boundary. Treat it as a sign on the door, not a lock.

Testing your own site

Test 1 — what does your server return for a given user-agent?

Before blaming robots.txt, confirm your CDN or WAF isn't already blocking the token.

for ua in "GPTBot/1.0" "ClaudeBot/1.0" "CCBot/2.0" "Mozilla/5.0 (compatible; Googlebot/2.1)"; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/robots.txt)
  echo "$code  $ua"
done

A 403 here means the block is happening above the application, and your robots.txt content is irrelevant for that token.

Test 2 — did the crawler actually read the file?

If a token appears in your access log but never appears fetching /robots.txt, that token is not following the directive. Grep for both:

# count hits per known AI token
grep -oE '(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|anthropic-ai|PerplexityBot|CCBot|Bytespider|Amazonbot|meta-externalagent|Applebot-Extended)' access.log \
  | sort | uniq -c | sort -rn

# did they fetch robots.txt first?
grep robots.txt access.log | grep -icE 'gptbot|claudebot|ccbot'

If the second number is zero and the first isn't, you have your answer about that crawler — regardless of what your file says.

Test 3 — is the same content reachable through a path you didn't think about?

This is the most common gap. You block a directory, and the same content is served at /feed/, /wp-json/wp/v2/posts, a print view, or an AMP variant. Blocking one path is not blocking the content.

for p in /feed/ /wp-json/wp/v2/posts /?print=1; do
  echo "$p -> $(curl -s -o /dev/null -w '%{http_code}' https://example.com$p)"
done

If any of those return 200 with your text in the body, your robots.txt rule is decorative.

Test 4 — confirm the file itself is being served correctly

A 5xx on /robots.txt is treated by major search crawlers as a full disallow while it persists, and a 404 is treated as no restrictions. Both outcomes are usually accidental; a redirect is worse, because the directive is defined per origin.

curl -sI https://example.com/robots.txt | head -n 5

You want a 200 with Content-Type: text/plain.

When robots.txt is not enough

If you have decided a crawler should not be served at all, the decision has to be enforced where you can actually refuse a request. A blunt user-agent check at the edge:

# http context
map $http_user_agent $block_training_crawler {
    default     0;
    ~*GPTBot    1;
    ~*ClaudeBot 1;
    ~*CCBot     1;
    ~*Bytespider 1;
}

server {
    listen 443 ssl;
    server_name example.com;

    if ($block_training_crawler) {
        return 403;
    }

    # ... rest of the server block
}

Two caveats worth writing down before you deploy it: user-agent strings are trivially spoofable, so pair this with verified-bot identification from your edge provider rather than treating it as authentication; and blocking at the edge means the crawler never sees your robots.txt explanation, which is fine for enforcement but removes the audit trail.

The decision you actually have to make

robots.txt forces a policy question that most sites haven't answered: do you want to be fetchable by AI systems, citable by them, or trainable by them? Those are three different permissions, and the tokens are separate for a reason. Blocking the training crawler while allowing the user-triggered fetcher is a coherent position. Blocking everything and then wondering why AI-assisted answers cite a competitor is also a position — just not the one most people meant to take.

If you want the raw data first — which AI crawlers are actually hitting your site, per token, before you write a single line of policy — AumCrawl logs AI crawler hits and lets you allow or block per crawler.

Checklist

[ ] robots.txt returns 200 on every hostname you serve
[ ] Your policy is written per token, not per *
[ ] You checked whether the tokens in your log ever fetched /robots.txt
[ ] You checked whether the same content is reachable via feed, JSON API, or print view
[ ] Anything you truly need refused is refused at the edge, not only in the file
[ ] You have a written answer to "fetchable, citable, or trainable"

More on crawler policy and per-crawler controls: aumcreate.com/crawlers

WordPress Feels Slow? Profile Plugins and Queries Before You Buy More Hosting

AumCreate — Thu, 22 Oct 2026 15:10:16 +0000

A slow WordPress page is not automatically a hosting problem. Before you upgrade, split the time into three buckets: PHP/plugin execution, database query time, and network time to first byte. If query time is low and PHP time is high, a bigger database does not help. If query time is high, a faster CPU can hide a missing index for a while. Profile before you upgrade.

Step 1: Measure a stable baseline

Use a staging copy with the same plugins and data. Measure the uncached page for profiling, then measure the cached page separately.

URL="https://staging.example.com/"

for i in 1 2 3 4 5; do
  curl -s -o /dev/null -w '%{time_starttransfer}' "$URL"; echo
done

Do not trust one run. Use the median of several runs. If the page is cached, the uncached number is the one that tells you about plugins and queries.

Step 2: Log page time, query count, and query time

Add this to wp-config.php while profiling:

define('SAVEQUERIES', true);

Create wp-content/mu-plugins/wp-profile.php:

<?php
add_action('shutdown', function () {
    if (!defined('SAVEQUERIES') || !SAVEQUERIES) {
        return;
    }

    global $wpdb;

    $total = 0.0;
    $slow = [];

    foreach ((array) $wpdb->queries as $q) {
        $seconds = (float) $q[1];
        $total += $seconds;

        if ($seconds >= 0.05) {
            $slow[] = [
                'sql' => $q[0],
                'seconds' => round($seconds, 4),
                'caller' => $q[2],
            ];
        }
    }

    error_log(sprintf(
        'WP_PROFILE page=%ss query=%ss queries=%d slow=%s',
        timer_stop(0),
        round($total, 4),
        count($wpdb->queries),
        wp_json_encode($slow)
    ));
});

Load the page, then read the PHP error log. Remove the constant and the mu-plugin after profiling. SAVEQUERIES adds overhead and should not stay on in production.

Step 3: Sweep plugins on staging

Do not do this on production. Deactivate all active plugins, then activate one at a time and measure the same URL.

cd /path/to/wordpress

measure() {
  for i in 1 2 3; do
    curl -s -o /dev/null -w '%{time_starttransfer}' "$1"; echo
  done | sort -n | sed -n '2p'
}

wp plugin list --status=active --field=name > /tmp/active-plugins.txt
wp plugin deactivate --all

while read -r plugin; do
  wp plugin activate "$plugin"
  echo "$plugin $(measure https://staging.example.com/)"
  wp plugin deactivate "$plugin"
done < /tmp/active-plugins.txt

wp plugin activate $(cat /tmp/active-plugins.txt)

Skip plugins that the site needs to boot. Also test with the default theme if you suspect the theme. A plugin that adds a small query can still add a large external HTTP call on every page load.

Step 4: Read the slow queries

Open the log line that starts with WP_PROFILE. Take the SQL from the slow array and run EXPLAIN in MySQL:

EXPLAIN SELECT post_id
FROM wp_postmeta
WHERE meta_key = 'example_key';

Look for:

type=ALL on a large table
a large rows estimate
Using temporary
Using filesort

Add an index only after you check the real query pattern and test write performance. An index that helps one report can hurt checkout or editorial saves.

Step 5: Check autoloaded options

Large autoloaded options are loaded on every request. Find the largest ones:

SELECT option_name, LENGTH(option_value) AS bytes
FROM wp_options
WHERE autoload = 'yes'
ORDER BY bytes DESC
LIMIT 20;

Do not delete rows blindly. Identify stale options from plugins you no longer use, then remove them through the plugin or a backup-backed cleanup.

Step 6: Check transients

Expired transients can pile up in wp_options. List the largest:

SELECT option_name, LENGTH(option_value) AS bytes
FROM wp_options
WHERE option_name LIKE '%transient%'
ORDER BY bytes DESC
LIMIT 20;

If you see many expired transients, clean them from the database or with a maintenance routine you control.

Step 7: Decide what to buy

Query time high, page time high: fix queries, indexes, or autoloaded data before buying a bigger database.
Query time low, page time high: profile plugins, theme hooks, external HTTP calls, and PHP workers.
Time to first byte high before WordPress runs: look at DNS, TLS, network, and host-level caching.
Only logged-in admin pages are slow: profile admin hooks, not the front end.

Hosting upgrades are useful when the profile shows a real resource limit. They are a poor substitute for a missing index or a plugin that calls a remote API on every request.

AumCreate is a one-person WordPress shop in Singapore. It does custom WordPress development and deployment setup. If you want help turning this profile into a fix, the services page is here: AumCreate services.

Schema.org for Small Business Sites: Organization, Article, and Product JSON-LD You Can Ship

AumCreate — Thu, 22 Oct 2026 15:06:59 +0000

Search engines and AI answer engines read structured data more reliably when each page has one clear job. For a small business site, the useful starting set is small: Organization for the business, Article for editorial pages, and Product for product pages. Add them in that order, validate, then add extras.

Rule: markup must describe what a visitor can see. Do not mark up a rating, price, author, or address that is not on the page.

1. Organization: the business entity

Use one Organization node sitewide, or at least on the homepage and about page. Keep the same @id so other nodes can reference it.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://example.com/#organization",
  "name": "Example Pte Ltd",
  "url": "https://example.com/",
  "logo": "https://example.com/logo.png",
  "email": "hello@example.com",
  "telephone": "+65-0000-0000",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "1 Example Road",
    "addressLocality": "Singapore",
    "postalCode": "000000",
    "addressCountry": "SG"
  },
  "sameAs": [
    "https://www.linkedin.com/company/example",
    "https://github.com/example"
  ]
}

Use sameAs only for profiles you control. Leave out aggregateRating unless real reviews exist on the page.

2. Article: editorial pages

Use Article on blog posts, guides, and news posts. The visible headline, author, and dates must match the markup.

{
  "@context": "https://schema.org",
  "@type": "Article",
  "@id": "https://example.com/blog/example-post/#article",
  "headline": "A clear headline under 110 characters",
  "description": "One sentence that matches the visible summary.",
  "image": ["https://example.com/img/example.jpg"],
  "datePublished": "2026-10-21T09:00:00+08:00",
  "dateModified": "2026-10-21T09:00:00+08:00",
  "author": {
    "@type": "Person",
    "name": "Author Name",
    "url": "https://example.com/about/author/"
  },
  "publisher": {
    "@id": "https://example.com/#organization"
  },
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://example.com/blog/example-post/"
  }
}

If you do not output a sitewide Organization node, replace the publisher reference with a full Organization object.

3. Product: product pages

Use Product on pages that sell one product. Product needs offers or review/aggregateRating. For a small shop, start with offers and skip ratings unless they are real.

{
  "@context": "https://schema.org",
  "@type": "Product",
  "@id": "https://example.com/products/example/#product",
  "name": "Example Product",
  "image": ["https://example.com/img/example-product.jpg"],
  "description": "What the product is and who it is for.",
  "sku": "EX-001",
  "brand": {
    "@type": "Brand",
    "name": "Example"
  },
  "offers": {
    "@type": "Offer",
    "url": "https://example.com/products/example/",
    "priceCurrency": "SGD",
    "price": "49.00",
    "availability": "https://schema.org/InStock",
    "priceValidUntil": "2026-12-31"
  }
}

4. Add Organization with a small WordPress snippet

If you are not using a schema plugin, this prints an Organization node on the front page. Replace the sameAs URL with your own profile.

add_action('wp_head', function () {
    if (!is_front_page()) {
        return;
    }

    $data = [
        '@context' => 'https://schema.org',
        '@type' => 'Organization',
        '@id' => home_url('/#organization'),
        'name' => get_bloginfo('name'),
        'url' => home_url('/'),
        'logo' => get_site_icon_url(),
        'sameAs' => [
            'https://www.linkedin.com/company/example',
        ],
    ];

    echo '<script type="application/ld+json">';
    echo wp_json_encode($data, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
    echo '</script>';
});

5. Validate before you ship

Check one Organization page, one Article page, and one Product page. Fix errors before adding more types. Common mistakes:

@id values that change on every request
price as a number with a currency symbol
image URLs that 404
author set to the company when the visible byline is a person
Product markup on category pages that list many products

If you use WordPress and want these nodes emitted from one place, AumViso is a free plugin that handles schema structured data, sitemap, meta, and internal links: AumViso.

WordPress Backup and Restore Verification: What to Include, How Often, and How to Prove It Works

AumCreate — Thu, 22 Oct 2026 15:01:50 +0000

Most backup conversations stop at "we take a nightly backup". That sentence describes a cron job, not a recovery plan. Until you have restored an artifact and checked the result, you have a hypothesis.

Here is the process, in the order it should be written down.

1. A backup is three artifacts, not one

Artifact	Contents	Why it is separate
Database dump	Every table with the site prefix, plus any table a plugin created outside it	Small, changes constantly, needs frequent capture
Files archive	`wp-content`: uploads, themes, plugins, mu-plugins	Large, changes rarely, needs versioning not frequency
Config + manifest	`wp-config.php`, URLs, prefix, table count, active plugin list	Needed to know what "complete" means

What must be in the dump: all prefixed tables and anything else the site writes to. Excluding tables you do not recognise is how restores lose order and membership data.

What must not be in the archive: wp-content/cache, wp-content/upgrade, wp-content/backups, log directories. They inflate the artifact and slow the restore for zero recovery value.

wp-config.php holds salts, database credentials, and sometimes hardcoded WP_HOME / WP_SITEURL. Store it separately and encrypt it.

The manifest is the part people skip. Record: UTC timestamp, siteurl, home, core version as reported by the install, table prefix, table count, active plugins. Without it you cannot answer "is this restore complete?" — only "does it load?".

2. Files and database must come from the same window

A dump taken at 02:00 and a tar taken at 02:40 can disagree: an attachment row exists for a file that is not in the archive, or the reverse. On a quiet blog nobody notices. On a store it is a missing product image during a campaign.

For anything taking orders, wrap the job:

wp --path="$SITE" maintenance-mode activate
trap 'wp --path="$SITE" maintenance-mode deactivate' EXIT

Or snapshot at the storage layer, which is atomic and cheaper. Pick one approach per site; do not mix them.

3. Frequency is an RPO decision

Decide how much data loss is acceptable, then derive the schedule.

Site type	Database	Files	Retention example
Brochure / blog	daily	weekly	14 daily, 8 weekly, 6 monthly
Store	every 4–6 hours	daily	48 hourly, 14 daily, 8 weekly
Membership / LMS	every 4–6 hours	daily	same as store

Two event-driven runs matter more than the schedule: immediately before any core, plugin or theme update, and before any DNS, SSL or hosting migration. Those are the moments you need an artifact from ten minutes ago, not from last night. A third: before a bulk import or a search-replace.

4. Write, seal, and ship

#!/usr/bin/env bash
set -euo pipefail

SITE="/var/www/example.com"
DEST="/srv/backups/example.com"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
WORK="$DEST/$STAMP"
mkdir -p "$WORK"

# 1. database
wp --path="$SITE" db export "$WORK/db.sql" --single-transaction --default-character-set=utf8mb4

# 2. files (cache and old backups excluded)
tar -C "$SITE" -czf "$WORK/wp-content.tar.gz" \
  --exclude='wp-content/cache' \
  --exclude='wp-content/uploads/cache' \
  --exclude='wp-content/upgrade' \
  --exclude='wp-content/backups' \
  wp-content

# 3. config
cp "$SITE/wp-config.php" "$WORK/wp-config.php"

# 4. manifest
PREFIX="$(wp --path="$SITE" config get table_prefix)"
TABLES="$(wp --path="$SITE" db query \
  "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = DATABASE();" \
  --skip-column-names)"
{
  echo "created_utc=$STAMP"
  echo "site_url=$(wp --path="$SITE" option get siteurl)"
  echo "home=$(wp --path="$SITE" option get home)"
  echo "core=$(wp --path="$SITE" core version)"
  echo "db_prefix=$PREFIX"
  echo "tables=$TABLES"
} > "$WORK/manifest.txt"

# 5. integrity checks at write time
gzip -t "$WORK/wp-content.tar.gz"
DUMPED="$(grep -c '^CREATE TABLE' "$WORK/db.sql")"
EXPECTED="$(grep -oE '^tables=[0-9]+' "$WORK/manifest.txt" | cut -d= -f2)"
if [ "$DUMPED" != "$EXPECTED" ]; then
  echo "FATAL: table count $DUMPED != expected $EXPECTED" >&2
  exit 1
fi

# 6. seal
( cd "$WORK" && sha256sum db.sql wp-content.tar.gz wp-config.php manifest.txt > SHA256SUMS )

# 7. ship offsite (adjust to your object storage CLI)
# aws s3 cp --recursive "$WORK" "s3://example-backups/$STAMP/"

The check that catches the most real failures is not gzip -t — it is the table count. A dump that hit a timeout mid-postmeta is still a valid gzip file and will still import; it is just short. Compare the dump size against the median of the last seven runs and alert on a large negative deviation.

5. Verify the restore, not the backup

Once a month, and after any major change, restore the newest artifact into staging and run a checklist. Time the drill — that number is your real RTO.

Prerequisite: a staging install with its own wp-config.php pointing at a scratch database. Do not copy the production config into staging; it holds production credentials and possibly hardcoded URLs.

#!/usr/bin/env bash
set -euo pipefail

BACKUP="/srv/backups/example.com/20261020T020000Z"
STAGING="/srv/staging/example.com"
SRC_URL="https://example.com"
DST_URL="https://staging.example.com"

# 0. verify the seal before touching anything
( cd "$BACKUP" && sha256sum -c SHA256SUMS )

# 1. files
rm -rf "$STAGING/wp-content"
tar -C "$STAGING" -xzf "$BACKUP/wp-content.tar.gz"

# 2. database
wp --path="$STAGING" db drop --yes 2>/dev/null || true
wp --path="$STAGING" db create
wp --path="$STAGING" db import "$BACKUP/db.sql"

# 3. rewrite URLs (handles serialized options; sed on the dump does not)
wp --path="$STAGING" search-replace "$SRC_URL" "$DST_URL" --all-tables --precise

# 4. caches and rewrite rules
wp --path="$STAGING" cache flush || true
wp --path="$STAGING" rewrite flush --hard || true

Then assert the content actually arrived. Row counts are the cheapest strong signal:

PREFIX=wp_
SRC="/var/www/example.com"
DST="/srv/staging/example.com"

for T in posts postmeta term_relationships terms term_taxonomy; do
  A="$(wp --path="$SRC" db query "SELECT COUNT(*) FROM ${PREFIX}${T}" --skip-column-names)"
  B="$(wp --path="$DST" db query "SELECT COUNT(*) FROM ${PREFIX}${T}" --skip-column-names)"
  [ "$A" = "$B" ] && S=OK || S=MISMATCH
  printf '%-18s prod=%-8s restored=%-8s %s\n' "$T" "$A" "$B" "$S"
 done

Skip options and users in the first pass: staging plugins can legitimately write rows on load, and a mismatch there sends you chasing a non-problem. Compare posts, postmeta, term_relationships and your order tables first.

Uploads:

diff <(cd "$SRC/wp-content/uploads" && find . -type f | sort) \
     <(cd "$DST/wp-content/uploads" && find . -type f | sort) | head -20

Empty output means the file lists match.

HTTP smoke test:

for p in / /sample-page/ /wp-login.php /robots.txt; do
  printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "https://staging.example.com$p")" "$p"
done

6. The manual checklist

Automation will not tell you whether checkout works.

Check	Pass condition
Admin login	dashboard loads, no database error
Permalinks	a post URL returns 200, not 404 (`wp rewrite flush --hard` if not)
Media	an existing image renders and the file is on disk
Forms	submit a test entry, confirm it is stored
Checkout	test-mode order completes, order note and confirmation mail arrive
Cron	`wp cron event list` shows the expected events
Outbound mail	a test message arrives
File ownership	web server can write to `wp-content/uploads`

That last row is the one that bites six weeks later, when a customer tries to upload a file and nobody can reproduce it.

7. Failure modes that survive a naive check

Truncated dump. Valid gzip, valid import, missing rows. Caught by the table count and a size-band alert.
--exclude-tables inherited from an old runbook, silently dropping a plugin's tables.
Uploads outside wp-content, because a host moved them. The archive is complete and contains no images.
Serialized data edited with sed. Use wp search-replace --precise, never text substitution on the dump.
Cron lost after a partial table restore, so scheduled posts and cleanup silently stop.
The backup stored inside wp-content, so tonight's job archives last night's job.
Wrong ownership after restore, so writes fail without an error the browser sees.

8. Log every drill

One line per drill: date, artifact timestamp, restore duration, row-count result, smoke test result, who ran it. When something breaks at 3am, that log is the difference between a runbook and a guess.

If you want this runbook written into the handover of a build rather than bolted on afterwards, backup and restore procedure is part of our custom WordPress work at AumCreate — custom development starts at $1,500, and the setup option covers deployment on a five-working-day turnaround.

A repeatable process for sitemaps and per-page meta that AI crawlers can actually use

AumCreate — Thu, 22 Oct 2026 14:45:27 +0000

Most "AI SEO" advice is about prompts. The unglamorous truth is that an AI crawler has to do five boring things to your site before any of that matters: discover the URL, fetch it, parse the markup, chunk the content, and cite it. Each of those steps has a signal you control. This post is the process I run to make all five work.

It's ordered deliberately. Doing step 4 before step 1 is how you end up with a beautiful schema graph on a page that no crawler can reach.

Step 0 — Inventory before you touch anything

List every URL template on the site (post, page, product, category, tag archive, author archive, search, paginated). For each one write down: is it indexable, is it canonical, does it need to be discovered at all?

Sitemaps get bloated because nobody ever does this. A sitemap listing 4,000 tag archives is not a discovery aid, it's noise that dilutes the crawl budget you wanted to spend on the 200 pages that matter.

Step 1 — Fix status and canonical first

A sitemap entry is a promise: this URL is 200, canonical, and indexable. Break that promise and crawlers learn to trust the file less.

Rules:

Only 200 URLs go in the sitemap.
Only self-canonical URLs go in the sitemap. Never list a URL that canonicals elsewhere.
Never list a noindex URL. It's a contradiction — the page says "don't index me," the sitemap says "here's what to index."
Redirects don't go in. The destination does.

On WordPress, core already excludes most of this via wp-sitemap.xml, but check the tag and author archives — those are the usual leaks.

Step 2 — Build the sitemap properly

A minimal, valid file:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guides/robots-txt/</loc>
    <lastmod>2026-10-14</lastmod>
  </url>
  <url>
    <loc>https://example.com/guides/sitemaps/</loc>
    <lastmod>2026-09-30</lastmod>
  </url>
</urlset>

Mechanics that matter:

Hard limits are 50,000 URLs and 50 MB uncompressed per file. Past either, split into a sitemap index.
lastmod must reflect a meaningful content change. Bumping it on every deploy trains crawlers to ignore the field entirely.
priority and changefreq are widely ignored. Skip them; they add noise without signal.
Keep the file UTF-8, well-formed, and free of BOM characters.

A sitemap index, for when you split by type:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-posts.xml</loc>
    <lastmod>2026-10-14</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2026-10-12</lastmod>
  </sitemap>
</sitemapindex>

Step 3 — Advertise it in robots.txt and verify

Sitemap: https://example.com/wp-sitemap.xml

That's the whole step — but verify it, because a broken sitemap URL in robots.txt is worse than none:

# 1. Is the sitemap well-formed XML?
curl -s https://example.com/wp-sitemap.xml | xmllint --noout - && echo "XML OK"

# 2. How many URLs, and do they look right?
curl -s https://example.com/wp-sitemap.xml \
  | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' \
  | wc -l

# 3. Spot-check the first 50 for noindex (should return nothing)
curl -s https://example.com/wp-sitemap.xml \
  | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' | head -50 \
  | while read -r u; do
      curl -sL "$u" | grep -qi 'name="robots" content="noindex"' && echo "NOINDEX: $u"
    done

That third check catches the most common sitemap defect in the wild: a plugin change turned a template to noindex and nobody noticed the sitemap still advertises it.

Step 4 — The per-page meta layer

Four tags per page, in this order of importance:

<title> — unique, describes the actual page, no template boilerplate.
<meta name="description"> — a real summary, ~150 characters. It gets used as snippet text and as chunk context.
<link rel="canonical"> — self-referencing on every indexable page.
<meta name="robots"> — only where it differs from the default. noindex,follow on search results and 404s.

WordPress outputs the canonical for you but not a description. A minimal, runnable addition:

add_action( 'wp_head', function () {
    if ( is_singular() ) {
        $id   = get_queried_object_id();
        $desc = wp_strip_all_tags( get_the_excerpt( $id ) );
        $desc = trim( preg_replace( '/\s+/', ' ', $desc ) );
        $desc = mb_substr( $desc, 0, 155 );

        if ( $desc ) {
            printf( "<meta name=\"description\" content=\"%s\">\n", esc_attr( $desc ) );
        }
    }

    if ( is_search() || is_404() ) {
        echo "<meta name=\"robots\" content=\"noindex,follow\">\n";
    }
}, 1 );

Swap get_the_excerpt() for whatever field holds your real summary if you have one — an excerpt built from the first 155 characters of body copy is a fallback, not a description. And if another plugin already emits a description, delete one of the two rather than shipping both.

Step 5 — Structured data, one shape per template

Schema is how you tell a parser what kind of thing the page is, instead of making it guess from prose. Match the type to the template — Article for posts, Product for products, BreadcrumbList site-wide, Organization once on the homepage.

add_action( 'wp_head', function () {
    if ( ! is_singular( 'post' ) ) {
        return;
    }

    $id = get_queried_object_id();

    $schema = [
        '@context'         => 'https://schema.org',
        '@type'            => 'Article',
        'headline'         => wp_strip_all_tags( get_the_title( $id ) ),
        'description'      => wp_strip_all_tags( get_the_excerpt( $id ) ),
        'datePublished'    => get_the_date( DATE_W3C, $id ),
        'dateModified'     => get_the_modified_date( DATE_W3C, $id ),
        'mainEntityOfPage' => [
            '@type' => 'WebPage',
            '@id'   => get_permalink( $id ),
        ],
    ];

    echo '<script type="application/ld+json">'
        . wp_json_encode( $schema, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE )
        . "</script>\n";
}, 20 );

Two rules: emit JSON-LD once per type (duplicate graphs from two plugins is a mess to debug), and only mark up what's visibly on the page. Marking up an FAQ that isn't rendered is the fastest route to losing rich results.

Step 6 — Internal links, because crawlers don't browse

Discovery isn't only the sitemap. It's also the link graph. A page with no inbound internal links is an orphan, and orphan pages are effectively undiscoverable no matter how clean the XML is.

Every indexable page reachable within three clicks of the homepage.
Descriptive anchor text — robots.txt patterns for AI crawlers, not click here.
Link from high-traffic posts down to the pages you actually want cited.
Check pagination: series and archives that only expose page 1 hide everything after it.

Step 7 — `llms.txt`, cheaply

llms.txt is a proposed convention, not an adopted standard — no guarantee any crawler reads it. Cost is near zero, so publish one, but don't build a strategy on it.

# Example Studio

> One-person WordPress studio. Templates, custom builds, deployment.

## Guides
- [robots.txt patterns](https://example.com/guides/robots-txt/): allowing search crawlers while blocking training crawlers
- [Sitemap setup](https://example.com/guides/sitemaps/): sitemap structure and per-page meta

## Optional
- [Changelog](https://example.com/changelog/)

Keep it to canonical URLs with one-line descriptions. A stale llms.txt full of 404s is worse than not having one.

Step 8 — The verification loop and its cadence

Per page, before publishing:

curl -sL https://example.com/guides/robots-txt/ \
  | grep -Eio '<link rel="canonical"[^>]*>|<meta name="robots"[^>]*>|<meta name="description"[^>]*>|<title>[^<]*</title>'

What you're looking for: one canonical, self-referencing; no stray noindex; a description that isn't just the title repeated.

Then on a schedule:

Weekly — new URLs appear in the sitemap, lastmod is honest.
Monthly — full sitemap sweep for 404s and noindex (the script in step 3, without the head -50).
On any template change — re-check the schema output and the canonical. Template edits are where noindex accidents happen.

Where the manual part stops

The process above is four files and a bash loop — which is fine for one site and miserable for five. This is exactly the boundary I hit, and why the sitemap, llms.txt, schema, meta and internal-link work on my own sites runs through AumViso instead of hand-maintained snippets.

But the tooling is the small part. The part that actually moves the needle is step 1: deciding which URLs deserve to be discovered, and then not contradicting yourself in the markup.

robots.txt patterns that let search crawlers in and keep AI training crawlers out (without wrecking indexing)

AumCreate — Thu, 22 Oct 2026 08:30:02 +0000

Every few months this request lands in my inbox: "can we block the AI crawlers but stay in Google?" The answer is yes, and it takes about 20 lines of robots.txt — but only if you stop treating robots.txt as one switch.

This is the walkthrough I run on WordPress sites.

1. Separate the three decisions

People conflate these constantly. They are different mechanisms with different failure modes.

Decision	Controlled by	If you get it wrong
May you fetch the page?	`robots.txt`	Page never gets read
May you index the page?	`meta robots`, `X-Robots-Tag`, canonical	Page appears in results you didn't want
May you train on / ground answers in the content?	Crawler-specific tokens	Content ends up in a corpus you didn't intend

robots.txt is a voluntary crawl request. It is not access control. If you need enforcement, that belongs at the server or CDN layer. Keep that in mind before you ship a Disallow and assume the problem is solved.

2. Sort the crawlers into three buckets

Before writing a single line, decide which bucket each token goes in. Write the decision down — future-you will not remember why PerplexityBot is allowed.

Bucket A — search index crawlers. Allow.
Googlebot, Bingbot, DuckDuckBot, Applebot, YandexBot.

Bucket B — training corpora. Usually block.
GPTBot, CCBot (Common Crawl), ClaudeBot, Google-Extended, Applebot-Extended, Bytespider.

Note that Google-Extended and Applebot-Extended are not crawlers. They're tokens that control whether already-crawled content is used for model training/grounding. Blocking them has no effect on how your pages rank or whether they're indexed — that's exactly why they're the safe ones to disallow.

Bucket C — AI answer surfaces. This is a business call.
OAI-SearchBot, PerplexityBot, Claude-SearchBot, plus user-initiated fetchers like ChatGPT-User and Claude-User.

These fetch pages so an assistant can cite them. Blocking them does not remove you from classic search results. It removes you from being quoted in answers. For a docs site that's a distribution channel; for a paywalled publisher it's a leak. Pick deliberately — the mistake is picking by accident because you copy-pasted a blocklist.

3. The file

# ---- Search: allowed ----
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Applebot
Allow: /

User-agent: YandexBot
Allow: /

# ---- AI training: blocked ----
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# ---- AI answer engines: allowed (flip to Disallow if you disagree) ----
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# ---- Fallback for everything else ----
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /*?replytocom=
Disallow: /*?add-to-cart=

Sitemap: https://example.com/wp-sitemap.xml

4. The four rules that actually bite

A crawler obeys exactly one group. It matches the most specific User-agent line, and then reads only that group. So the moment you add a Googlebot group, the Disallow: /wp-admin/ under User-agent: * stops applying to Googlebot. This is the single most common way people break their own site while "just adding AI blocking."

If you use specific groups, you have two options: duplicate the shared rules into every group, or leave the shared rules out entirely and keep the specific groups narrow. Don't mix and hope.

Disallow: with nothing after it means allow everything. It is not "block all". Disallow: / is block all.

robots.txt must return 200 at the root. A 404 is read by most crawlers as "no restrictions, go ahead". A 5xx tells crawlers something is broken and they may back off crawling. And never disallow robots.txt itself.

Blocking a URL does not remove it from the index. If a page is linked from somewhere crawlable, Disallow stops the content being read, not the URL being listed. To remove a page you need noindex — and noindex only works if the page is crawlable. Blocking and de-indexing are opposite operations; running both is how you get "why is this still in Google."

Wildcards (*, $) work on the major search engines. Support elsewhere varies, so treat wildcard lines as best-effort cleanup, not as a security boundary.

5. Verify it like a crawler, not like a human

Read the file back through a parser. Five seconds of Python saves a week of confusion:

import urllib.robotparser
from urllib.parse import urljoin

BASE = "https://example.com"

rp = urllib.robotparser.RobotFileParser()
rp.set_url(urljoin(BASE, "/robots.txt"))
rp.read()

checks = [
    ("Googlebot", "/blog/post-1/"),
    ("Googlebot", "/wp-admin/"),
    ("GPTBot", "/blog/post-1/"),
    ("CCBot", "/"),
    ("OAI-SearchBot", "/blog/post-1/"),
    ("SomeRandomBot", "/blog/post-1/"),
]

for ua, path in checks:
    allowed = rp.can_fetch(ua, urljoin(BASE, path))
    print(f"{ua:15} {path:22} {'ALLOW' if allowed else 'BLOCK'}")

Expected shape of the output with the file above: Googlebot /blog/post-1/ ALLOW, Googlebot /wp-admin/ ALLOW (because the specific group swallowed the * rules — that's the bug this test exists to catch), GPTBot BLOCK, OAI-SearchBot ALLOW.

One caveat: Python's robotparser doesn't implement * and $ wildcards. Use it for group-matching sanity and check wildcard lines by hand.

Then confirm the basics over HTTP:

curl -sI https://example.com/robots.txt | head -5
# expect: HTTP/2 200, content-type: text/plain

curl -s https://example.com/robots.txt | head -20

6. The indexing safety checklist

Run this every time you touch the file:

[ ] /robots.txt returns 200 with text/plain
[ ] A Googlebot group exists and contains no Disallow on page templates
[ ] No blanket Disallow: / sitting in the * group
[ ] CSS, JS and image paths are not blocked — a search crawler that can't render the page sees a blank template
[ ] The Sitemap: line points at a URL that returns 200
[ ] noindex pages are crawlable, so the directive can actually be read
[ ] Live URL test in Search Console on one post, one page, one product

That last item is the one people skip. Do it on a real URL from each template, not the homepage.

7. `robots.txt` is a request — the log is the truth

Once the file is live, the interesting question changes from "did I write it right" to "is anyone honouring it." You can't answer that from the file; you answer it from request data, per user agent, over time.

On WordPress I use AumCrawl for that: it surfaces which AI crawlers are actually hitting the site and lets you allow or block each one individually at the server level — which is the part robots.txt genuinely cannot enforce, since a crawler that ignores the file never reads it in the first place.

So the loop is: declare intent in robots.txt, enforce it at the server, verify the result in the logs. All three, or you're just writing polite suggestions.