Our robots.txt allows /games/ and then disallows 165 paths underneath it

CogniPrep's robots.txt is 11.5 KB. Most of that is 165 Disallow lines that all sit underneath a path the same file has already allowed:

Allow: /games/
...
Disallow: /games/acr-abstract
Disallow: /games/acr-literacy
Disallow: /games/acr-mechanical
...
Disallow: /games/wlc-personality

That reads like a mistake. It is the whole point of the file.

Two different kinds of page share one prefix

/games/shl is a provider hub. It is a public marketing page: it describes the test suite, it is in the sitemap, and it is one of the pages we actually want ranking.

/games/shl-verbal is a game. It is auth gated. An anonymous visitor gets redirected to /login, which means a crawler gets an empty shell with no outgoing links and nothing to index.

Both live under /games/. A crawler has no way to tell them apart from the URL, so before we split them, Site Audit reported the same two issues over and over: "noindex page" and "page has no outgoing links", once per game. Worse than the noise, those fetches were crawl budget spent on pages that can never rank, taken from the hubs that can.

Longest prefix wins, which is what makes the contradiction legal

The robots.txt rule most people half remember is "Allow beats Disallow". That is only the tiebreak. The actual rule in Google's specification is that the most specific rule wins, where specific means the longest matching path, and Allow only wins when two rules are exactly the same length.

/games/shl-verbal is matched by Allow: /games/ (7 characters) and by Disallow: /games/shl-verbal (18 characters). The Disallow is longer, so the game is blocked. /games/shl is matched only by the Allow, so the hub stays crawlable. One prefix, two outcomes, decided by string length.

This is why the specific paths have to be spelled out in full. A shorter pattern that tried to be clever, something like Disallow: /games/*-, would be both fragile and shorter than several of the paths it was trying to beat.

Nobody types 165 paths

The list is generated from the same array the app routes from:

import { GAME_LIBRARY } from '@/lib/games/constants';

const GAME_ROUTES = GAME_LIBRARY.map((game) => game.route).sort();

const DISALLOW = [
  '/api/',
  '/dashboard/',
  '/admin',
  // ...
  ...GAME_ROUTES,
  // ...
];

This is the part I would keep if I rewrote the file tomorrow. A hand maintained blocklist is a list nothing derives from, and those drift silently: the failure mode is not an error, it is a game that shipped last week quietly burning crawl budget for a month before anyone opens Search Console. Deriving it means a new game is disallowed on the day it ships, with no config change and nothing to remember.

robots.ts in the Next.js App Router is an ordinary TypeScript module that exports a function, so importing the game library into it costs nothing at runtime. It is evaluated at build time and serialised to text.

Bingbot gets its own identical copy

return {
  rules: [
    { userAgent: '*', allow: ALLOW, disallow: DISALLOW },
    { userAgent: 'Bingbot', allow: ALLOW, disallow: DISALLOW },
  ],
  sitemap: `${baseUrl}/sitemap.xml`,
  host: baseUrl,
};

That duplication is not redundancy. A crawler picks the single most specific group whose user agent matches and obeys only that group. It does not merge groups. So the moment you write any User-agent: Bingbot section at all, Bingbot stops reading the * section entirely, and anything you left out of the specific group is no longer a rule that applies to it.

Which means the dangerous version of this file is not the one with two copies. It is the one where somebody adds a three line Bingbot group for one special case and silently unblocks 165 game routes for Bing.

It also doubles the file. 382 Disallow lines, 34 Allow lines. Google's limit is 500 KiB, so 11.5 KB is not close to a problem.

The parameter we deliberately did not block

The bottom of the file blocks the usual tracking parameters:

'/*?utm_*',
'/*?source=*',
'/*?campaign=*',
'/*?fbclid=*',
'/*?gclid=*',
'/*?msclkid=*',
'/*?mc_*',

?ref= is missing from that list, and there is a comment in the source explaining why, because it looks exactly like an oversight:

?ref= is deliberately NOT blocked. Launch directories link to /?ref=<name>, and blocking it stops crawlers from seeing the homepage's rel="canonical", so the backlink cannot consolidate.

This is the thing about robots.txt that costs people real links. Blocking a URL does not tell a crawler the URL is a duplicate. It tells the crawler not to fetch it. A URL that is never fetched is a URL whose rel="canonical" is never read, and canonical is the only mechanism by which the link equity from /?ref=tinylaunch gets credited to /.

So blocking ?ref= to avoid duplicate content achieves the opposite of what it looks like it achieves: the duplicate stays in the index as an unfetchable URL, and the backlink you earned points at a page Google has decided not to look at. Letting it through and answering with a canonical tag is what actually merges it.

The tracking parameters above are safe to block for exactly the inverse reason: nobody links to you with a gclid, so there is no equity to consolidate and nothing lost by never fetching them.

Defence in depth, because robots.txt is a request

None of this is enforcement. robots.txt is advisory, and a page that must not be indexed cannot rely on it. The game routes also send robots: noindex, nofollow in their metadata, and the public pages that link to them use rel="nofollow" on the link itself. If the file vanished tomorrow, nothing would be indexed that should not be.

There is a subtlety there worth stating, because the two mechanisms interact badly if you get them backwards: a noindex meta tag only works if the crawler is allowed to fetch the page and read it. Disallowing a URL in robots.txt and putting noindex on it means the noindex is never seen. In our case the belt and braces are in the right order because the routes redirect anonymous traffic to /login anyway, so there is no content to index either way. If your pages return real HTML, pick one: noindex and allow the fetch, or Disallow and accept that the URL can still appear as a bare link.

See it

Open cogniprep.app/robots.txt in a browser. Scroll to the /games/ block and count: the first group runs from acr-abstract to wlc-personality, then the whole thing repeats under User-agent: Bingbot.

Then compare the two page types the file is separating:

• cogniprep.app/games/shl is a hub. It returns full HTML to anyone, and it is in the sitemap.
• /games/shl-verbal is a game route from the same prefix. Open it signed out and you land on /login. That empty shell is what the 165 Disallow lines exist to keep crawlers away from.

Finally, load cogniprep.app/?ref=tinylaunch and view source. The <link rel="canonical" href="https://cogniprep.app"/> in the head is the tag that only works because the file does not block the parameter that got you there.

One last note if you try to check any of this with curl: the site sits behind a bot checkpoint, so you will get an interstitial rather than the file. Use a browser.