robots.txt cannot protect a URL somebody forwards, and that is the whole reason for the meta tag

CogniPrep has an affiliate signup page that is never linked from anywhere on the site. You reach it from a single-use invite link we email a creator. It is disallowed in robots.txt, and it also sends noindex, nofollow in its page metadata.

Belt and braces, except the two halves protect against different things, and the one that most people would call redundant is the one actually doing the work.

The disallow is the half that can be bypassed

User-Agent: *
Disallow: /affiliate/

A crawler reads robots.txt for a host, then obeys it while crawling that host. That model has a gap: a crawler handed a URL directly, from a shared link, a referrer header, a pasted message, a browser extension that reports visited URLs, does not necessarily fetch and apply robots.txt rules for a page it was pointed at. And even when a disallowed URL is fully respected, the disallow only says "do not fetch". It does not say "do not index". A URL with enough external signal can appear in results as a bare link with no snippet, because the crawler was told not to look at what is inside it.

An invite link gets forwarded. That is what invite links are for. Whichever inbox, group chat or note-taking tool it passes through is a place the URL can escape from, and none of those paths involve reading our robots.txt first.

The meta tag is the half that travels with the page

export const metadata = {
  title: 'Affiliate Signup',
  robots: { index: false, follow: false },
};

This is in the response. It does not matter how the crawler arrived, whether it read robots.txt, or what linked to it. It is attached to the thing being evaluated rather than to the host.

Note the dependency, which is the part people get backwards: for a crawler to see noindex, it has to fetch the page. If a URL is both disallowed and noindexed, and a crawler obeys the disallow, it never learns about the noindex. The two do not stack neatly. They cover different arrival paths, and you want both precisely because you cannot predict which path a leaked URL takes.

For a page like this, the disallow saves crawl budget on a URL with no search value, and the meta tag is what actually keeps it out of the index.

Neither of them is security

Here is the rule this page exists to demonstrate: crawler directives are requests, not access control. An indexing directive is a note to well-behaved software. A page that must not be used by strangers has to be enforced by the code that serves it.

So the page is force-dynamic and validates the invite server side before rendering anything. Its server component shows either the form, or a specific message: this link is not valid, this link has already been used, this link has expired.

And then the form posts to an API route that re-validates the invite authoritatively. Nothing the page did is load bearing for security. The page's validation exists to give a human a useful message instead of a form that fails on submit.

The redemption itself is one transaction that locks the invite row:

return db.transaction(async (tx) => {
  const [invite] = await tx
    .select()
    .from(affiliateInvitesTable)
    .where(eq(affiliateInvitesTable.token, token))
    .for('update');
  // ...validate, mark used, create the code
});

FOR UPDATE is what makes "single use" true rather than aspirational. A token that leaks, or is double submitted by an impatient click, still yields at most one code.

One more thing sits in the invite rather than in the form: the commission rate and the buyer discount are fixed when the invite is generated. The person redeeming it supplies only the code they want. An affiliate can never set their own rates, because the rates are not an input to the endpoint they can reach.

The related decision: one parameter we deliberately do not block

The same robots file blocks the usual tracking parameters (utm_*, fbclid, gclid and friends) because they generate duplicate thin URLs. It does not block ?ref=.

Launch directories link to us as /?ref=<name>. Blocking that pattern stops crawlers from fetching the page, which means they never see the homepage's canonical tag pointing at /, which means the backlink cannot consolidate into the canonical URL. The duplicate content concern is real, and the canonical tag is the right tool for it. Blocking the crawl solves the smaller problem by preventing the fix for it.

That is the same lesson as the page above, in the other direction: a disallow prevents a fetch, and quite a lot of the machinery you want, canonical tags and noindex included, only works if the fetch happens.

See it

Both halves are public and take about twenty seconds to check.

1. Open cogniprep.app/robots.txt and search for /affiliate/. You will find it in the wildcard group and again in the Bingbot group, which gets its own explicit copy of the same policy because it crawls more aggressively with explicit rules than with an inherited wildcard.
2. Open cogniprep.app/affiliate/join and view source, or run this in the console:

document.querySelector('meta[name="robots"]').content
// "noindex, nofollow"

Without an invite parameter the page does not render a form at all. It renders "This page needs a valid invite link. Please ask CogniPrep for one." That is the server-side check above, visible from the outside.

While you are in that file, note that it is 423 lines long. Almost all of it is generated: every individual auth-gated game route is disallowed, derived from the game library rather than typed out, so a newly added route is disallowed the day it ships rather than the day somebody remembers.

The takeaway

Ask what each protection is attached to. robots.txt is attached to a host and consulted before a fetch. A meta directive is attached to a response and read during one. Access control is attached to the code that produces the response, and is the only one of the three that a determined visitor cannot simply ignore.