How to Audit AI Crawler Access with robots.txt and Server Logs

AI visibility work often starts with prompts, citations, and answer tracking. But there is a more basic question to settle first:

Can the systems you care about reliably fetch the pages you expect them to use?

A site can publish excellent research and still create avoidable discovery problems through a restrictive robots.txt rule, a CDN challenge, inconsistent redirects, or pages that return different status codes to different user agents.

This guide shows a practical, repeatable audit for AI-crawler access using two evidence sources:

1. the site's declared policy in robots.txt, and
2. observed requests in server or CDN logs.

The goal is not to guarantee inclusion in any AI answer. It is to make access decisions visible, testable, and easier to maintain.

1. Start with an explicit crawler policy

A short robots.txt file is easier to review than a long collection of overlapping rules.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

If your organization intentionally applies different rules to specific crawlers, document the business reason next to the configuration in version control. Robots.txt itself does not support inline operational context well, so keep a companion policy file.

# ai-crawler-policy.yml
owner: growth-platform
review_interval_days: 90
default_policy: allow
exceptions:
  - user_agent: ExampleBot
    path: /private-research/
    action: disallow
    reason: unpublished source material
    approved_by: security
    review_on: 2026-10-01

The important part is not the exact schema. It is having an owner, a review date, and a reason for every exception.

2. Parse robots.txt as data

Manual inspection is useful, but automated checks catch accidental changes during deployment.

The following TypeScript example reads a robots.txt file and extracts simple allow/disallow rules. Production parsers should account for the full standard and the matching behavior of the crawlers you monitor, but even a small check can surface unexpected policy changes.

type Rule = {
  userAgent: string;
  directive: "allow" | "disallow";
  path: string;
};

export function parseRobots(input: string): Rule[] {
  const rules: Rule[] = [];
  let currentAgents: string[] = [];

  for (const rawLine of input.split(/\r?\n/)) {
    const line = rawLine.replace(/#.*/, "").trim();
    if (!line) continue;

    const separator = line.indexOf(":");
    if (separator === -1) continue;

    const key = line.slice(0, separator).trim().toLowerCase();
    const value = line.slice(separator + 1).trim();

    if (key === "user-agent") {
      currentAgents = [...currentAgents, value.toLowerCase()];
      continue;
    }

    if (key === "allow" || key === "disallow") {
      for (const userAgent of currentAgents) {
        rules.push({
          userAgent,
          directive: key,
          path: value,
        });
      }
    }
  }

  return rules;
}

A useful CI assertion is that important public paths remain allowed for the default agent.

import { readFile } from "node:fs/promises";
import { parseRobots } from "./parse-robots";

const robots = await readFile("public/robots.txt", "utf8");
const rules = parseRobots(robots);

const blockedPublicPaths = rules.filter(
  (rule) =>
    rule.userAgent === "*" &&
    rule.directive === "disallow" &&
    ["/blog", "/research", "/docs"].some((path) =>
      rule.path.startsWith(path),
    ),
);

if (blockedPublicPaths.length > 0) {
  throw new Error(
    `Public content blocked in robots.txt: ${JSON.stringify(blockedPublicPaths)}`,
  );
}

This does not replace a standards-compliant parser. It creates a focused regression test for the paths your team has declared important.

3. Test the real HTTP response

Robots.txt is only one layer. A crawler can be allowed by policy and still receive a 403, 429, redirect loop, or JavaScript-only shell.

Run a small set of HTTP checks from outside your production network:

curl -I https://example.com/robots.txt
curl -I https://example.com/sitemap.xml
curl -I https://example.com/research/
curl -L -o /dev/null -s -w "%{http_code} %{url_effective}\n" \
  https://example.com/research/

For each important URL, record:

• initial status code
• final status code after redirects
• final canonical URL
• response content type
• cache or challenge headers
• response time
• whether the page contains meaningful server-rendered text

Do not rely on a single homepage test. Sample templates separately: documentation, articles, comparison pages, pricing pages, and structured research assets can behave differently at the edge.

4. Query logs for observed crawler traffic

Server and CDN logs show what actually happened. Start with a bounded time window and retain only the fields needed for diagnosis.

Example SQL for a generic request-log table:

SELECT
  DATE_TRUNC('day', request_time) AS day,
  user_agent,
  status_code,
  COUNT(*) AS requests,
  COUNT(DISTINCT path) AS unique_paths
FROM edge_requests
WHERE request_time >= CURRENT_TIMESTAMP - INTERVAL '30 days'
  AND (
    LOWER(user_agent) LIKE '%bot%'
    OR LOWER(user_agent) LIKE '%crawler%'
    OR LOWER(user_agent) LIKE '%spider%'
  )
GROUP BY 1, 2, 3
ORDER BY day DESC, requests DESC;

User-agent text is a discovery signal, not proof of identity. If identity matters, validate requests using the publisher's documented verification method, such as IP or reverse-DNS procedures where available. Do not create a universal list from memory and assume it stays current.

Then inspect errors by path:

SELECT
  path,
  status_code,
  COUNT(*) AS requests,
  MAX(request_time) AS last_seen
FROM edge_requests
WHERE request_time >= CURRENT_TIMESTAMP - INTERVAL '30 days'
  AND status_code >= 400
  AND LOWER(user_agent) LIKE '%bot%'
GROUP BY 1, 2
ORDER BY requests DESC
LIMIT 100;

This turns a vague concern into a queue of concrete fixes.

5. Separate access, discovery, and citation

These are different layers and should have different metrics.

Layer	Question	Example evidence
Access	Can the page be fetched?	200 response, no challenge
Discovery	Was the page requested?	verified log event
Interpretation	Was usable content returned?	rendered text and metadata
Citation	Did an answer reference the page?	observed answer citation

A successful fetch does not prove citation. A citation does not prove that every page is accessible. Keeping the layers separate prevents teams from drawing causal conclusions that the data cannot support.

6. Build a small weekly audit

A useful weekly job can be simple:

1. fetch robots.txt and store a hash
2. compare it with the previous version
3. test a representative URL set
4. aggregate verified crawler requests
5. flag new 4xx/5xx patterns
6. attach owners and deadlines
7. review access changes alongside AI-visibility changes

Example audit record:

{
  "runAt": "2026-08-03T07:00:00Z",
  "robotsHash": "sha256:replace-with-real-hash",
  "testedUrls": 24,
  "successfulUrls": 23,
  "blockedUrls": 1,
  "newErrorPatterns": [
    {
      "pathPattern": "/research/*",
      "status": 403,
      "owner": "edge-platform"
    }
  ]
}

Use real values from your own audit. The placeholder hash above is intentionally labeled so it cannot be mistaken for production evidence.

7. Review changes with an evidence checklist

Before closing an access issue, verify:

• the robots.txt rule is intentional
• the live response matches the declared policy
• the canonical URL is stable
• the page returns meaningful content without a browser-only dependency
• bot-management rules are not creating unintended challenges
• log retention is long enough to compare periods
• verified and unverified user agents are reported separately
• access metrics are not presented as citation metrics

Where this fits in an AI-visibility program

Crawler access is infrastructure hygiene. The next layer is measuring how a brand appears across AI answers, how fresh those observations are, and which cited sources recur.

Corank is built around AI-visibility measurement. Its workflow can complement an access audit by helping teams observe the answer layer after the underlying pages are available and monitorable.

The practical sequence is:

1. make access policy explicit,
2. verify real HTTP behavior,
3. observe crawler requests,
4. measure answer visibility and citations,
5. review changes with dated evidence.

That sequence keeps technical fixes, content work, and visibility reporting connected without treating any one signal as proof of the others.

Final takeaway

An AI-crawler audit should be boring in the best way: versioned rules, repeatable tests, bounded log queries, clear owners, and explicit uncertainty.

When those basics are in place, teams can spend less time guessing whether a page was reachable and more time improving the content and evidence that people—and answer engines—may choose to use.``