A dead URL on our site answers 404 in one namespace and a 307 to the login form in the next

PubTrivia is one Next.js app holding two very different things: a public marketing site of 72 sitemap URLs, like the guide to hosting a pub quiz and the features index, and a signed-in dashboard where a venue sets up tables and runs the night. One proxy decides, on every request, which of those two you are allowed to see.

Here is what that decision does to URLs that do not exist. Run it yourself, no account needed:

$ curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://pub-trivia.app/guides/not-a-real-guide
404

$ curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://pub-trivia.app/blog
307 https://pub-trivia.app/login?next=%2Fblog

Neither page exists. One of them says so. The other one asks you to sign in first.

The list that decides

The gate is a prefix allowlist, and it lives in one file, lib/routes.ts, whose docstring explains why it exists at all:

/**
 * The one place that says which routes are public, which are indexable, and
 * which crawlers should stay out.
 *
 * These three questions used to be answered independently in three files, the
 * auth allowlist in utils/supabase/middleware.ts, the sitemap, and robots.txt,
 * and they disagreed: /about shipped in the sitemap while the auth gate bounced
 * every crawler that followed it to /login.
 */

PUBLIC_ROUTES has 21 entries. Sixteen are pages or content clusters, and five are infrastructure: /robots.txt, /sitemap.xml, the IndexNow key file, the maintenance page and /too-many-requests.

The matcher is three lines:

function matchesAny(routes: readonly string[], pathname: string): boolean {
    return routes.some(
        (route) => pathname === route || pathname.startsWith(route + '/')
    )
}

That + '/' is doing more work than it looks. It is why /playlist is not covered by the /play entry, and why /loginx is not covered by /login:

$ curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://pub-trivia.app/playlist
307 https://pub-trivia.app/login?next=%2Fplaylist

It is also why '/' being in the list does not make the entire site public. '/blog' === '/' is false, and '/blog'.startsWith('//') is false, so the root entry matches exactly one path: the root.

The flip side is that every entry is a prefix whether or not it has children. /about is a single leaf page with nothing beneath it, and the allowlist cannot express that:

$ curl -s -o /dev/null -w '%{http_code}\n' https://pub-trivia.app/about/nope
404

So /about/nope gets a correct 404 from a namespace that does not exist, and /blog gets a redirect to a login form. The difference is not about which URLs are real. It is about which first segment somebody added to a list.

What the redirect costs

For a human, a 307 to a login form is a shrug. For a crawler it is a worse answer than the 404, for three reasons that compound.

A 404 is a removal signal. Google drops a URL that keeps answering 404. A 307 is not a removal signal, it is a statement that the content moved to the target, so the dead URL keeps getting recrawled and the target absorbs the signal.

The target is /login, and /login is not noindex:

$ curl -s https://pub-trivia.app/login | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="index, follow"/>

It has a self-referential canonical, so it is not pointing that signal anywhere else either. And robots.txt does not keep crawlers out of it:

User-Agent: *
Allow: /
Disallow: /dashboard
Disallow: /play
Disallow: /api
Disallow: /auth
Disallow: /subscribe
Disallow: /too-many-requests
Disallow: /maintenance

Eight disallows, and /login and /signup are deliberately not among them, because both are real pages a person might reasonably arrive at from a search. The effect is that /login is an indexable page that is not in the sitemap, and every mistyped or stale link anywhere outside those 21 prefixes redirects into it.

The 404 page is also telling two stories

The page a reader actually gets from the namespaces that do work is app/not-found.tsx, which is 21 lines: a heading, a sentence, and a link home. It exports no metadata. In the App Router that means it inherits whatever the root layout declares, which is the site-wide default, which is the homepage's:

$ curl -s https://pub-trivia.app/guides/nope | grep -o '<title>[^<]*</title>'
<title>PubTrivia | Run Live Pub Quiz Nights</title>

So a 404 claims to be the homepage in the browser tab, and in the og:title of any link preview of it. And because Next adds its own robots tag to a notFound() response while the root layout contributes the site default, the same document carries both of these:

$ curl -s https://pub-trivia.app/guides/nope | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="noindex"/>
<meta name="robots" content="index, follow"/>

The restrictive directive wins when a crawler sees both, so this one is cosmetic rather than harmful. It is still two tags disagreeing in one head, which is the kind of thing that costs an hour when you are debugging something else.

The root layout comment next to it shows the lesson was already learned once, for a different tag:

// No `alternates` here on purpose: a canonical set at this level is inherited
// by every page that does not set its own, and each of them then declares
// itself a duplicate of the homepage.

Canonical was pulled out of the root layout for exactly the reason robots and title should have been thought about for the 404 page.

What we are changing

Two edits, in this order.

The 404 page gets its own metadata: its own title, and robots: { index: false, follow: false } set deliberately rather than inherited from two directions at once.

The gate gets a fall-through for unknown paths. The allowlist answers "is this public", and it should not also be answering "does this exist". A request that matches no entry, carries no session cookie, and is a GET for text/html has no destination to remember, so instead of a redirect with a next parameter it should be allowed through to the 404. Authenticated-only pages keep the redirect, because for those the gate really is the reason you cannot see the page.

If you want to watch the gate work before we change it, the public site is entirely crawlable and the two curls at the top of this post are the whole experiment. Add any segment you like after /guides/ for a 404, and ask for anything we never shipped at the root for the redirect.