An Audit for a Site That Wants to Be Read by Machines

#ai #seo #tutorial #llm

Twenty checks, in four groups, each with a command to run and a condition that decides pass or fail. Every one is either specified in a public document or observable on your own infrastructure — nothing here rests on a claim about the inside of a ranking system.

How to run this

Pick three URLs before you start: your home page, your best content page, and one generated page if you have a template. Run every check against all three. Set SITE and PAGE once and the commands below paste directly.

SITE=https://example.com
PAGE=$SITE/learn/some-page

The groups are ordered by dependency: a failure in access makes everything after it moot, and a failure in delivery makes structure irrelevant. Work top to bottom and stop fixing when you reach a green group.

Access: can it be fetched

1. robots.txt returns 200 as plain text. curl -sSI $SITE/robots.txt. Pass: status 200 and a content type of text/plain. A framework returning an HTML 404 page here parses to no rules at all, and a 5xx tells a compliant crawler to stop entirely.
2. Each named agent group is complete on its own. Read the file and cover every group but one with your hand. Pass: each group expresses your full intent alone, because a crawler with its own group never reads the * group — the rule explained in robots.txt for AI crawlers.
3. The sitemap exists and is referenced. curl -sS $SITE/robots.txt | grep -i sitemap then fetch it. Pass: the line exists and the URL it names returns 200 XML.
4. No sitemap URL is disallowed by robots.txt. Extract the URLs and check each path against your rules. Pass: zero contradictions. This is the mismatch a search console reports back to you.

5. Every sitemap URL returns 200.

curl -sS $SITE/sitemap.xml \
  | grep -oE '<loc>[^<]+' | sed 's/<loc>//' \
  | while read -r u; do
      printf '%s %s\n' "$(curl -o /dev/null -sS -w '%{http_code}' "$u")" "$u"
    done | grep -v '^200' | head -20

Pass: no output. Redirects count as failures here — a sitemap should name final URLs.

6. Known crawlers are not being challenged or rate-limited into failure. Group your access log by agent and status. Pass: no verified crawler is receiving a material share of 429 or 5xx responses. The parser in the crawler audit prints exactly this.

Delivery: is the content in the response

7. The main text is in the unrendered HTML.
```
curl -sS -A "OAI-SearchBot" "$PAGE" | grep -c "a phrase from paragraph six"
```
Pass: at least 1. A zero here makes most of this cluster irrelevant until it is fixed.
8. The raw-to-rendered word ratio is above 90%. Run the diff script in JavaScript rendering and what crawlers see. Pass: the kept percentage is 90 or above on all three URLs.
9. A bot user agent gets the same page a browser does. Fetch with a browser agent and a bot agent, and compare byte counts. Pass: within a few per cent. A large gap means a WAF, a consent wall or a geographic redirect is intercepting the exact clients you care about.
10. Missing pages return 404 or 410. curl -o /dev/null -sS -w '%{http_code}' $SITE/definitely-not-a-page. Pass: 404 or 410, not 200. A soft 404 is indistinguishable from content to everything downstream.
11. Redirects are permanent and single-hop. curl -sSIL $PAGE | grep -E '^HTTP|^[Ll]ocation'. Pass: at most one redirect, and a 301 rather than a 302 where the move is permanent.
12. Non-HTML assets carry the directives you intend. curl -sSI $SITE/some.pdf | grep -i x-robots-tag. Pass: the header is present where you want a directive and absent where you do not. A meta robots tag cannot reach a PDF.

Structure: can it be split usefully

13. Exactly one h1, and it states the question. curl -sS $PAGE | grep -c '<h1'. Pass: 1, and reading it alone tells you what the page answers.
14. Heading levels do not skip. Extract the headings in order and check the sequence. Pass: no h2 followed directly by an h4. A structure-aware splitter reconstructs the trail from these.
15. No section exceeds roughly 400 words. Count words between consecutive h2 elements. Pass: sections mostly under 400, which is about one chunk — the arithmetic is in retrieval-friendly site architecture.
16. A mid-page passage stands alone. Copy four hundred words from the middle into a blank file and read it. Pass: a stranger can tell what it is about and what is being claimed.
17. The canonical is self-referential and exact. curl -sS $PAGE | grep -i 'rel="canonical"'. Pass: it matches the served URL character for character — scheme, host, trailing slash and case included.
18. Structured data is present in the raw response and valid. curl -sS $PAGE | grep -c 'application/ld+json', then paste the page into the Schema Markup Validator. Pass: at least 1 in the raw HTML and no errors from the validator. Injected client-side counts as absent.
19. Every value in the structured data appears in the visible text. Extract the JSON values and grep the rendered text for each. Pass: all of them found. This is the honesty rule from structured data that machines read, enforced mechanically.

Evidence: can you tell what happened

20. Your logs can answer the question “who fetched what”. Check three things: that the real client address is recorded rather than your CDN’s, that the user-agent string is retained, and that retention is at least thirty days. Pass: all three. Without this check, none of the others can be re-run against reality, and you have no record if a question about crawling ever becomes concrete.

Two optional additions that are observations rather than pass conditions, and are worth recording alongside the audit even though they cannot fail:

A baseline mention rate. Run the sampling protocol in measuring whether AI assistants mention you once, with its interval, so a future comparison has something to compare to. Do not treat the number as a target.
A layout depth series. Run the harness in the ten blue links, measured on your top queries and keep the screenshots. One measurement is a data point; four quarters is a series.

Scoring, and what to fix first

The checks are not equally weighted, because they are not independent.

Failure	Description
Check 7 or 8 fails	Stop and fix it. If your text is not in the response, every other item is theoretical. This is the single most expensive failure in the list and the most likely to be present without anyone knowing.
Check 1 or 3 fails	Fix next, and in the same change: a robots.txt and a sitemap that disagree are worse than either alone.
Check 9 fails	Your own infrastructure is blocking the clients you are trying to reach. Usually a WAF rule or a consent wall, and usually a five-minute fix once located.
Check 20 fails	Fix before the others, not after. Until the logs are right you cannot verify that any fix worked.
Checks 13 to 16 fail	Content work rather than engineering work, done page by page. Real, slower, and it is where the remaining value is once the mechanics are green.
Checks 18 and 19 fail	Lowest priority of the failures here. Worth doing, worth doing honestly, and not worth delaying anything above it for.

Re-run the whole thing quarterly and after any framework upgrade or infrastructure change. Every failure in this list is the kind that reappears silently: a refactor moves a fetch to the client, a CDN rule starts challenging an agent, a redesign changes a heading level. The audit is cheap enough to repeat, which is the only property that makes a checklist useful over more than one afternoon.