Nakodo's robots.txt is 30 lines of text generated from 34 lines of TypeScript, and more than half of it is the same five rules written twice. That duplication is the only correct way to write the file, and it took reading the specification to believe it.
Here is the whole thing, live:
User-Agent: *
Allow: /
Disallow: /campaigns
Disallow: /settings
Disallow: /onboarding
Disallow: /billing
Disallow: /api/
Disallow: /auth/
User-Agent: GPTBot
User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
User-Agent: ClaudeBot
User-Agent: Claude-SearchBot
User-Agent: Claude-User
User-Agent: PerplexityBot
User-Agent: Perplexity-User
User-Agent: Google-Extended
User-Agent: Applebot-Extended
User-Agent: CCBot
Allow: /
Disallow: /campaigns
Disallow: /settings
Disallow: /onboarding
Disallow: /billing
Disallow: /api/
Disallow: /auth/
Sitemap: https://nakodo.app/sitemap.xml
Groups are chosen, not merged
The rule in RFC 9309 that makes this file look redundant: a crawler finds the group whose User-Agent line matches it most specifically, obeys that group, and ignores every other group in the file, including the wildcard one.
So this, which is what I wrote first, is a mistake:
User-Agent: *
Disallow: /campaigns
User-Agent: GPTBot
Allow: /
That does not mean "everyone keeps out of /campaigns, and GPTBot is also allowed the rest of the site". It means GPTBot has an empty disallow list and the run of the whole site, including the pages the wildcard group was protecting. Naming a crawler and then writing less than the wildcard group gives it more access, not less.
Which is why the rules are repeated verbatim. In the source there is exactly one copy:
import { PRIVATE_PREFIXES } from "@/lib/public-paths";
// The app behind sign-in has nothing for a crawler; everything else is open.
// Links with tracking parameters stay crawlable on purpose: every page sets a
// canonical URL, and a crawler has to fetch the page to see it.
const DISALLOW = [...PRIVATE_PREFIXES, "/auth/"];
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: "*", allow: "/", disallow: DISALLOW },
{ userAgent: AI_CRAWLERS, allow: "/", disallow: DISALLOW },
],
sitemap: absoluteUrl("/sitemap.xml"),
};
}
Next's MetadataRoute.Robots takes userAgent as a string or an array of strings. An array becomes one group with eleven User-Agent lines, which is what RFC 9309 calls a group with several product tokens: they share one set of rules. That is the difference between eleven lines of file and eleven groups of eight lines, and it also means the two policies cannot drift, because they are the same constant.
Why name them when the policy is identical
The wildcard group already allows everything except the app. Every one of these eleven crawlers would be allowed the same by *. So the group adds nothing to the file's behaviour today, and I keep it anyway:
// AI search and assistant crawlers, named so the policy toward them is a
// decision on record rather than an accident of the wildcard. A crawler obeys
// only the most specific group that names it, so each gets the full policy.
const AI_CRAWLERS = [
"GPTBot",
"OAI-SearchBot",
"ChatGPT-User",
"ClaudeBot",
"Claude-SearchBot",
"Claude-User",
"PerplexityBot",
"Perplexity-User",
"Google-Extended",
"Applebot-Extended",
"CCBot",
];
A wildcard that allows AI crawlers is indistinguishable from never having thought about AI crawlers. A named group is a position: somebody wrote these names down, and changing the position is a one-line diff on an array rather than an argument about what the wildcard was supposed to imply. Nakodo's public pages exist to be read and quoted; a prospective customer asking an assistant about creator outreach and getting our pages quoted back is the outcome we want, so the honest value of the array is "yes, on purpose".
The eleven are not all the same kind of thing, which is the part worth knowing before you copy the list:
- Training and corpus crawlers: GPTBot, ClaudeBot, CCBot. These fetch pages to build datasets.
- Search index crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot. These build the index an assistant searches when it answers.
- User-triggered fetchers: ChatGPT-User, Claude-User, Perplexity-User. These fetch a page because a person in a chat asked about it, one URL at a time.
-
Two that are not crawlers at all: Google-Extended and Applebot-Extended. No fetcher sends those strings as a user agent. They are control tokens: Google and Apple read them out of robots.txt to decide whether content their ordinary crawlers already have may be used for AI features. A group that names them is the only way to say yes or no, and
Allow: /there is a yes.
That last group is the one I would have got wrong. Disallowing Google-Extended does not stop Googlebot fetching anything, and allowing it does not add a single request to the server. It is a permission, sitting in a file whose other lines are all traffic rules.
The disallow list is the auth boundary, read from the other side
The five paths are not typed in robots.ts. They come from the module that decides what needs a session:
// Where robots.txt sends crawlers away from: the signed-in app. The auth pages
// are left crawlable so crawlers can read their noindex tag.
export const PRIVATE_PREFIXES = ["/campaigns", "/settings", "/onboarding", "/billing", "/api/"];
Two details in there took a while to settle.
/auth/ is appended in robots.ts but the sign-in pages themselves are not blocked. Blocking a page in robots.txt and marking it noindex are mutually exclusive instructions: a crawler that is not allowed to fetch the page can never read the tag that asks it not to index the page, so a blocked-and-noindexed URL can still show up in results as a bare link. The sign-in pages therefore stay crawlable and carry the tag instead:
// Sign-in pages stay out of search results. robots.txt leaves them crawlable so
// crawlers can read this.
export const metadata = { robots: { index: false, follow: true } };
You can check that one from a terminal: curl -s https://nakodo.app/login | grep robots prints name="robots" content="noindex, follow".
And none of the prefixes has a trailing slash. Matching in robots.txt is a literal prefix, not a path segment match, so Disallow: /campaigns also covers a hypothetical /campaigns-public. For this site that is the behaviour I want, and the one URL space where the difference matters, the API, is written with the slash because /api/ needs to block what is under it and there is no page at /api itself.
The test that keeps the two boundaries honest
The real failure mode here is not a crawler getting into the app. It is a public page being treated as private, which is silent: the page works perfectly for anyone signed in, and crawlers get a redirect to the sign-in form. So the same module's test walks the sitemap:
test("every page in the sitemap is open to signed-out visitors", () => {
for (const path of [...STATIC_PAGES.map((p) => p.path), ...ENTRIES.map((e) => e.path)]) {
assert.ok(isPublicPath(path), `${path} would redirect crawlers to /login`);
}
for (const path of ["/opengraph-image", ...CARD_PATHS.map((p) => `/og${p}`)]) {
assert.ok(isPublicPath(path), `${path}: social cards must be public for link previews`);
}
});
The second half of that test is there because a link preview is an unauthenticated fetch of an image route: a social card behind the session check turns every shared link into a picture of a login form, and nothing in the app would ever show you that.
What the crawlers get once they are in
Allowing an assistant to crawl is only half of it. The pages are also published as markdown for exactly these clients, at llms.txt and llms-full.txt, generated from the same arrays that render the HTML, so a quote from an assistant and a quote from a reader are the same sentence.
There is one place where naming yourself a bot changes the response, and it is not the robots file. Prices are rendered per request in the visitor's currency, and anything that looks like a crawler gets the base currency instead of a datacentre's:
// Crawlers get pounds: they're placed wherever their servers are, and search
// results should show the base price, not a datacentre's currency.
export async function displayCurrency(): Promise<Currency> {
if (BOT.test((await headers()).get("user-agent") ?? "")) return BASE_CURRENCY;
return currencyForCountry(await requestCountry());
}
Which is checkable in one line:
$ curl -s -A GPTBot https://nakodo.app/pricing | grep -oE '£[0-9]+' | sort -u
£0
£39
£119
Those are the real plan prices on the pricing page. Fetch the same URL from a browser in the euro area and the same numbers arrive as euros.
The whole of it
The file is boring on purpose and it still needed three decisions: repeat the rules because groups are chosen rather than merged, name the AI clients because a wildcard records no decision, and keep the blocked list in the module that already knows what is private so robots.txt and the sign-in check can never disagree. The one line of the file nobody argues about, the Sitemap: line, is the one that gets fetched most.
Top comments (1)
We need to produce a short comment, casual, one sentence maybe two, about the video. Must not be formal. Must start with a reaction or question specific to this video. Could ask about why certain names aren't crawlers, or about the repeated whole policy. Must avoid "Great video". Must not use prohibited phrases. Must not include URLs. Must be short. Must not include double hyphen. Must not use em-dash. Must be ASCII. Write something like: "so why did they list two non‑craw