robots.txt makes two decisions, and most sites only make one

#seo #ai #webdev #marketing

A crawler arriving at your site is doing one of two jobs, and they have nothing in common:

Answering a question right now — retrieving your page so an assistant can summarise or cite it.
Filling a training corpus — taking the text away to shape a future model.

Most robots.txt files on the web treat both the same way: either everything is allowed by omission, or a
single Disallow: / turns the site invisible. That is the wrong granularity in both directions. Blocking
citations removes you from the answers people ask today; blocking training changes nothing about how you
appear in those answers. They are separate decisions, and they are written as separate groups in the same
file.

The two groups

Citation crawlers are the ones that fetch a page because a person asked a question: OAI-SearchBot,
ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, DuckAssistBot. If these are blocked,
the assistant can still mention your brand — it just cannot read the page it would cite, so the answer
quotes somebody else's page about you instead of yours.

Training crawlers take content for model development: GPTBot, ClaudeBot, Google-Extended,
Applebot-Extended, CCBot, meta-externalagent, Bytespider, Amazonbot, cohere-ai. Blocking these
is a legitimate business decision — it is simply a different decision, and it is the one most sites never
make explicitly because the default is silence.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin
Disallow: /staging

Sitemap: https://example.com/sitemap.xml

Three details that actually decide whether the file works:

The user-agent token has to be exact. GPTBot is not gptbot, and a typo produces a file that reads as a policy while behaving as silence.
The Sitemap line is an absolute url — a crawler does not guess your scheme, and a relative path is ignored.
A per-group allow does not override a path disallow in another group. The most specific matching group wins, which is why private paths belong in User-agent: * and not only in your own list.

Why this is a marketing decision, not an IT chore

The answer engines of the last two years are the channel with the lowest cost per qualified read that
exists: a page that is quotable and citable gets shown to people who typed the exact problem it solves.
A single line in a text file decides whether that channel exists for you. That is marketing work with an
engineering shape, and nobody in a typical org owns it — legal is worried about training, IT is worried
about load, and marketing never sees the file.

There is a second half to it too: permission is not the same as usability. A page can be fully crawlable and
still unciteable — no stated answer, no schema, no dates, nothing an agent can quote in one sentence. Both
halves have to be true before the line matters.

The tools I use for this, free and public

AI crawler robots.txt generator — writes both groups from your host and private paths, in your browser, nothing sent anywhere.
AEO readiness checklist — 23 weighted checks on whether a page is quotable at all.
llms.txt generator — the same facts in one file an agent can read without crawling.
FAQ schema generator — the markup that makes a Q&A quotable.

The same method ships as a CLI: npx marketing-mindset aeo runs the readiness checklist locally.

Where I do this work

I run it on my own domain first and the artifacts are public: robots.txt that names every crawler
explicitly, a sitemap, an llms.txt, schema on every commercial page, and a dated proof page listing every
artifact with a clickable link. If you want the AI-readable layer built on your site instead of explained:

Sprint $900 — one focused build, five working days: an offer page with schema, five test hypotheses with kill rules, one channel live, tracking that makes the result readable.
Engine $1,900/month — a running queue of pages, tools and distribution with a monthly read-out.
Full Build $2,900 — site architecture, the full page set, the AI-readable layer, tooling and handover.

Scope, prices and the proof page are all public — no invented case studies, no benchmarks I did not
compute. The full picture: marketing engineer for hire.

The one-line version

Write down which crawlers may quote you and which may train on you, in two groups, with exact tokens and an
absolute sitemap url — then make sure the pages they are allowed to read are worth quoting.

Method, free tools and published scope: axelfreeman.com
· the method as an npm CLI

DEV Community