How to Check AI Crawler Access (robots.txt / llms.txt) Without a Paid API

#apify #webscraping #api #ai

The problem

Whether ChatGPT, Claude, Perplexity, Google or a dozen other AI crawlers can reach your site is decided by three small public text files: robots.txt, llms.txt, and llms-full.txt. Checking them sounds trivial until you actually do it. A single robots.txt can name each bot in its own user-agent group, with specific groups taking precedence over wildcard ones and Allow rules winning equal-specificity ties. A site can block training bots while allowing search bots, or the reverse — and a WAF or CDN rule can change what the file serves without anyone touching page code.

Now multiply that by a client portfolio of dozens of sites, or add the question "did last week's deployment change anything?" Reading the files by hand is slow, misreading the syntax is easy, and a paid audit SaaS is a heavy answer for what is ultimately three HTTP GETs per site.

What the actor does

The AI Crawler Access Checker audits up to 100 sites in one run. Per origin it requests only the three conventional policy paths — robots.txt, llms.txt, llms-full.txt — and interprets the observed rules against a tracked set of 16 named AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Amazonbot, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User), segmented by training, search, and user-fetch purpose.

For each unique website the delivered row includes:

an AI Access Score across all 16 tracked bots and a separate AI Search Score over the search/user-fetch subset, so training policy and discovery policy can be reviewed independently;
per-bot verdicts — allowed, partial, blocked, or unknown — plus the restricted path patterns and whether a specific named group or the wildcard group matched;
robots.txt policy state (available, not_published, unavailable), llms.txt/llms-full.txt presence and HTTP status, and declared sitemap URLs;
direct evidence URLs with observation timestamps, a confidence block, explicit data gaps, a recommended human action, and a review priority.

Unknown states are honest: if the policy endpoint times out, returns 403/429/5xx, or serves HTML instead of text, the verdict is unknown, both scores are null, the row is free, and the recommended action is to retry after source recovery — a temporary outage is never scored as permission.

Repeat audits are built in: paste a previous run's exported dataset rows into the optional previousResults input and each new row carries a comparison block with changed fields and before/after values, included in the same per-site price. Safety controls include private-address rejection, DNS pinning, bounded and revalidated redirects, and per-endpoint byte caps. The actor never crawls page content or executes JavaScript.

Example: input and output

{
  "websites": [
    "apify.com",
    "openai.com",
    "anthropic.com"
  ],
  "maxConcurrency": 10
}

An abbreviated example row from the README's dataset output section:

{
  "website": "https://example.com",
  "found": true,
  "robotsTxt": { "exists": true, "status": 200, "state": "available" },
  "llmsTxt": { "exists": false, "status": 404 },
  "sitemaps": ["https://example.com/sitemap.xml"],
  "aiAccessScore": 88,
  "aiSearchScore": 100,
  "blockedBots": ["GPTBot", "CCBot"],
  "bots": [
    { "bot": "GPTBot", "vendor": "OpenAI", "purpose": "training", "verdict": "blocked", "restrictedPaths": ["/"], "matchedRule": "specific" },
    { "bot": "OAI-SearchBot", "vendor": "OpenAI", "purpose": "search", "verdict": "allowed", "restrictedPaths": [], "matchedRule": "specific" }
  ],
  "summary": "Blocks 2/16 AI crawlers: GPTBot, CCBot.",
  "recommendedAction": "CONSIDER_PUBLISHING_LLMS_TXT",
  "actionPriority": "low",
  "safeToAutomate": false,
  "billing": { "billable": true, "eventName": "result-found" }
}

The README notes this row is illustrative, not a live observation of example.com; actual values depend on the files served when your run executes.

Pricing and the free limit

The actor is pay-per-event: $0.005 per run start plus $0.02 per successfully delivered unique website audit. Duplicates are normalized before any source work and never billed twice; core source failures come back as free outcome rows.

Apify's free plan gives $5 of usage credits per month. At this tariff, $5 covers about 249 site audits in a single run (0.005 + 0.02 × 249 = $4.985) — or two full 100-site audits at roughly $2.005 each.

Try it

Paste a domain list, press Start, and read the dataset overview as a review queue: AI Crawler Access Checker

For AI agents and MCP

The actor takes JSON in and returns structured JSON rows plus a run-level OUTPUT record with completeness and replay-safety counters, so agents can call it directly through the Apify API and triage results without human parsing. The README ships explicit guardrails for that use: cite evidence[].sourceUrl when describing a policy, never replace unknown or null with an allow/block guess, preserve dataGaps in the answer, and require human approval before changing a live site policy. An MCP server setup for the actor is documented in the README's API section.

DEV Community