Disallow: /device-limit/ blocks nothing, and the list right above it has both /admin and /admin/

Our robots.txt is generated from a TypeScript array, and this is the top of it:

const DISALLOW = [
  // Internal app routes, no crawl value
  '/api/',
  '/dashboard/',
  '/admin',
  '/admin/',
  '/auth/',
  '/webhook/',

Look at /admin and /admin/, both spelled out. Whoever wrote those two lines knew that robots.txt matches a path prefix, so a rule ending in a slash cannot match a URL that ends without one. Disallow: /admin/ blocks /admin/tickets and leaves /admin itself alone, which is why the bare form is there as well.

Now the bottom of the same array:

  // Post-auth / transactional pages
  '/purchase/',
  '/device-limit/',
  '/unsubscribed/',
  '/forgot-password/',
  '/forgot-password/reset',
  '/forgot-password/success',
  '/signup/success',

Every one of those is a single page, not a directory. The real URLs are /device-limit, /unsubscribed and /forgot-password, with nothing after them. The rules name /device-limit/, /unsubscribed/ and /forgot-password/, which match a child that does not exist and miss the page that does. The two sub-pages written without a trailing slash, /forgot-password/reset and /forgot-password/success, are matched correctly, so the result is a parent page left open and its children blocked.

The same lesson was learned once, four lines apart from where it was not applied.

What that costs, measured on the live site

$ curl -s -o /dev/null -w "%{http_code}\n" https://cogniprep.app/device-limit
200
$ curl -s https://cogniprep.app/device-limit | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="index, follow"/>

index, follow, because app/device-limit/page.tsx exports no metadata at all and therefore inherits the root layout's defaults, which are deliberately open for the marketing pages. The inheritance goes further than the directive:

$ curl -s https://cogniprep.app/device-limit | grep -o '<title>[^<]*</title>'
<title>CogniPrep - Psychometric Test Practice</title>

That is the home page's title, on a page whose entire content is "you are signed in on too many devices". Its description is the home page's description too, naming 56 providers. So the one page on the site with no metadata of its own is a title-and-description duplicate of the page we most want ranked, and the rule meant to keep crawlers away from it misses by one character.

/forgot-password is the same story with a twist, because it does have metadata:

<title>Reset Password | CogniPrep</title>
<meta name="robots" content="index, follow"/>
<link rel="canonical" href="https://cogniprep.app/forgot-password"/>

A self-referential canonical is an invitation. The page is telling crawlers "this URL is the right one for this content" while robots.txt thinks it has been excluded.

Of the three, only /unsubscribed is actually safe, and not because of the robots file:

<meta name="robots" content="noindex, nofollow"/>

It is the one page of the three that sets the directive itself. The fourth, /maintenance, is safe for a different reason again: when maintenance mode is off it answers 307 with location: /login, so there is nothing to index. Four transactional pages, four different outcomes, one intention.

None of the four is in the sitemap, which is 375 URLs and derived from the content registries. That is the quiet part of this: a page can be absent from your sitemap, blocked in your head, and still be a crawlable 200 that returns index, follow to anything that finds a link to it. /device-limit gets its links from a redirect in the auth callback rather than from a crawlable page, which is luck rather than design.

Why I trust the directives over the file

The rest of the file does not have this problem, because the rest of the file is derived rather than typed. The 372 game routes are generated:

const GAME_ROUTES = GAME_LIBRARY.map((game) => game.route).sort();

with a comment explaining that longest-prefix matching is what lets those specific paths beat the broader /games/ allow while leaving the provider hubs crawlable, and that the pages also send noindex, nofollow and are linked with rel="nofollow", "so nothing depends on this file alone".

That last clause is the whole lesson. The generated half of the file has a second and third line of defence; the hand-typed half has exactly one, and one of its seven lines was wrong. Live, the file is 836 lines with 796 Disallow directives across two groups (* and Bingbot), each repeating the whole policy because a crawler obeys only the group that matches it: 744 of those lines are the 372 game routes, written twice. Nobody is reviewing 796 lines by eye. The fix is to stop writing paths in two syntaxes: emit both forms for every entry, or normalise at generation time and assert the result in a test.

I am going with the second, and the assertion is pleasingly small: for every route the app actually serves, a Disallow entry must either match it exactly or the page must send its own noindex. That test fails today on two pages, which is the only reason I know about them.

See it

Four requests, four different answers, no account needed:

curl -s -o /dev/null -w "device-limit    %{http_code}\n" https://cogniprep.app/device-limit
curl -s -o /dev/null -w "forgot-password %{http_code}\n" https://cogniprep.app/forgot-password
curl -s -o /dev/null -w "unsubscribed    %{http_code}\n" https://cogniprep.app/unsubscribed
curl -sI https://cogniprep.app/maintenance | grep -iE '^(HTTP|location)'

Then diff the intent against the rule:

curl -s https://cogniprep.app/robots.txt | grep -nE 'device-limit|forgot-password|unsubscribed|/admin'

You will see /admin and /admin/ next to each other near the top, and the three single pages carrying only their slashed form near the bottom. Compare each against the 375 URLs in the sitemap, none of which are these.

The site all of this is protecting is at cogniprep.app, where the pages that are supposed to rank are the 56 provider hubs under /games and the test-format pages under /tests. The device wall is not one of them, and for the last few months it has been competing with the home page for its own title.