DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Six admin pages, three refusals, and a noindex tag that no crawler can ever reach

PubTrivia got an admin area: six page files under app/admin, for the one person who operates the thing. A list of venues, a list of quizzes, a mailing composer, and two detail pages behind dynamic segments.

It is the least glamorous code in the repository and it made me check three things I thought I already knew. One of them turned out to be wrong, and the wrong one is at the end.

Three refusals, written three times on purpose

A page behind one gate is a page behind one bug. There are three here, and none of them trusts the others.

The proxy. The app's route matcher checks a session on anything that is not in the public allowlist. /admin is not in it, so a signed-out visitor never reaches the layout:

$ curl -s -o /dev/null -w '%{http_code} -> %{redirect_url}\n' https://pub-trivia.app/admin
307 -> https://pub-trivia.app/login?next=%2Fadmin
Enter fullscreen mode Exit fullscreen mode

Identical with a Googlebot user agent, and identical for a path that does not exist:

$ curl -s -o /dev/null -w '%{http_code} -> %{redirect_url}\n' https://pub-trivia.app/nonsense-path-xyz
307 -> https://pub-trivia.app/login?next=%2Fnonsense-path-xyz
Enter fullscreen mode Exit fullscreen mode

That sameness is the useful property. The 307 on /admin is not evidence that /admin exists, because every unknown path in the application answers exactly the same way. Try it on your own invented path above.

The layout. A signed-in customer who types the URL does reach the layout, and gets a 404:

export default async function AdminLayout({ children }: { children: React.ReactNode }) {
  const admin = await getAdmin()

  // A 404 rather than a 403: a signed-in customer who guesses the URL learns
  // nothing. Signed-out visitors never get here; the proxy sends them to /login.
  if (!admin) notFound()

  return <AdminShell email={admin.email}>{children}</AdminShell>
}
Enter fullscreen mode Exit fullscreen mode

403 is the honest status and 404 is the better one. "You are not allowed here" confirms there is a here.

Every action, separately. This is the one people skip:

/**
 * For server actions. Every export of a "use server" file is a public POST
 * endpoint, so each admin action checks for itself rather than trusting that
 * the page it was rendered on was gated.
 */
export async function requireAdmin(): Promise<Admin> {
  const admin = await getAdmin()
  if (!admin) throw new Error('Not found')
  return admin
}
Enter fullscreen mode Exit fullscreen mode

A server action is not protected by the page that rendered it. It is a POST route with a generated id, callable by anyone who has seen that id, and the id lives in the HTML of a page that an admin loaded. So the mailing composer's send action, the CSV export, every quiz action: each one opens with requireAdmin().

Who counts as an admin is a code change

/** Lowercase. Compared against the signed-in user's email after normalising it. */
export const ADMIN_EMAIL_ALLOWLIST: readonly string[] = Object.freeze([/* two addresses */])
Enter fullscreen mode Exit fullscreen mode

An email allowlist in a frozen array, two entries, rather than a role column or an environment variable. Three reasons, and the third is the one I like:

  • It cannot be escalated from inside the product. There is no row anybody can flip to become an admin. Only this file decides, and changing it is a reviewed commit and a deploy.
  • It has no bootstrap step. No migration, no one-off script to mark the first admin before the dashboard works at all.
  • A missing environment variable can never lock me out of my own admin area, because there is no environment variable.

The lookup is cache() wrapped, so the layout, the page and the header share one call to the auth provider rather than making three:

export const getAdmin = cache(async (): Promise<Admin | null> => { /* ... */ })
Enter fullscreen mode Exit fullscreen mode

And the segment is pinned to per-request rendering, because an admin dashboard that gets statically generated at build time is a different kind of bug:

export const dynamic = 'force-dynamic'
export const revalidate = 0
Enter fullscreen mode Exit fullscreen mode

The list that never heard about any of this

Our repository has one file that answers which routes are public, which are indexable and which crawlers should stay out of. It exists because those three questions used to be answered in three places and the three disagreed: /about shipped in the sitemap while the auth gate bounced every crawler that followed it to /login.

/admin is correctly absent from the public allowlist, correctly absent from the sitemap, and correctly absent from the list of routes that stay up during maintenance. Here is the fourth list, live:

$ curl -s https://pub-trivia.app/robots.txt
User-Agent: *
Allow: /
Disallow: /dashboard
Disallow: /play
Disallow: /api
Disallow: /auth
Disallow: /subscribe
Disallow: /unsubscribe
Disallow: /too-many-requests
Disallow: /maintenance

Sitemap: https://pub-trivia.app/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

Eight prefixes. /admin is not one of them, and it is the newest private segment in the app.

The dashboard layout, written months earlier, states the pattern in its own comment: robots.txt stops crawling rather than indexing, so the private segments carry robots: { index: false } on their layouts as well. The list is the belt and the meta tag is the braces.

The admin layout has the braces:

export const metadata: Metadata = {
  title: 'Admin',
  robots: { index: false, follow: false, nocache: true },
}
Enter fullscreen mode Exit fullscreen mode

And now the part I had not thought about. That meta tag is unreachable. A crawler cannot read a noindex on a page it is never served: asking for /admin gets a 307 before any of our rendering happens, so the only robots directive a crawler will ever see for that URL is the one in robots.txt, which does not mention it.

Then it follows the redirect, and the destination is indexable:

$ curl -s https://pub-trivia.app/login | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="index, follow"/>
Enter fullscreen mode Exit fullscreen mode

That is deliberate for /login itself, which is crawlable but deliberately not in the sitemap, because a sitemap says which pages you want ranked and a sign-in form is not one of them. But it means the redirect chain from a private URL lands on a page that invites indexing, under a 307 that says "keep asking for the original".

None of this is a leak. Nothing in the product links to /admin, the gate is real, and a crawler that follows the redirect learns only that we have a sign-in page. What it costs is smaller and duller than a leak: a crawler that finds the URL somewhere, a referrer log, a browser history sync, a post like this one, will keep asking for it, and will keep being redirected, forever, because a 307 promises that the URL is worth retrying. One line in the disallow list ends that conversation, and it belongs there. The reason I am writing it down rather than quietly adding it is that the bug is interesting and the fix is not: four lists are three too many to keep in your head, which is exactly why they were centralised, and centralising them did not make me read all four when I added a segment.

The census, if you want to check

$ curl -s https://pub-trivia.app/sitemap.xml | grep -c '<url>'
111
$ curl -s https://pub-trivia.app/sitemap.xml | grep -c admin
0
Enter fullscreen mode Exit fullscreen mode

111 URLs, 108 of them content pages from the registry and 3 legal documents, and no private segment among them. The repository has 141 page.tsx files, so 30 of them are pages the sitemap does not mention: 11 under the dashboard, 6 under admin, 4 in the password reset flow, 2 under the player route, and then sign-in, sign-up, the checkout return, the unsubscribe confirm page, the rate limit page, the maintenance page and the auth callback. Some of those are public and deliberately unindexed, which is a different thing from private, and telling the two apart is the whole job of that one file.

The public ones are all reachable from pub-trivia.app, and if you want the full list without crawling for it, the sitemap is the honest version of this post's numbers.

Top comments (0)