Letting crawlers read noindex on Next.js auth pages

Sign-in pages are the classic thing to hide from search. The reflex is to put them in robots.txt and move on. On RadarMakers we did exactly that, and then changed our minds about one path. This post shows the two small edits, the reasoning behind them, and, just as important, what the change does not prove.

The rule that is easy to get backwards

robots.txt and noindex answer different questions. robots.txt says whether a crawler may fetch a URL. noindex says whether a page that was fetched may appear in search results. The second one can only be read from a response, so a crawler that is not allowed to fetch the URL never sees it.

Google's documentation on blocking indexing with noindex says this directly: for a noindex rule to be effective, the page must not be blocked by robots.txt. If it is blocked, the crawler never sees the rule, and the URL can still show up in search, for example when other pages link to it.

That makes a Disallow line a poor tool for removing a URL that search engines already know about. It stops the visit that would have delivered the instruction to leave.

What we had

Our frontend has a sign-in flow, and the OAuth callback lives at /auth/callback. It is a client component that reads the query string and either sends the person on or shows an error. A URL such as /auth/callback?error=... is the kind of address that appears when a sign-in does not complete.

Before the change, src/app/robots.ts included /auth/ in its disallow list:

// frontend/src/app/robots.ts, excerpt, before commit 46f72d7
      disallow: ["/admin", "/dashboard", "/auth/", "/api/", "/reset-password", "/forgot-password"],

The commit message and code comment name /auth/callback?error= style URLs as the reason for the change. This post does not include crawl data about them, and it makes no claim about whether or when any of them left an index.

The change

One commit, 46f72d7, made two edits that only make sense together.

First, "/auth/" came out of the disallow list. Nothing else in the list moved:

// frontend/src/app/robots.ts, excerpt, current
      disallow: ["/admin", "/dashboard", "/api/", "/reset-password", "/forgot-password"],

Second, next.config.ts got a headers() function that adds X-Robots-Tag: noindex to responses handled by Next.js for paths matching /auth/:path*, which includes /auth and /auth/callback:

// frontend/next.config.ts, excerpt
  async headers() {
    return [{ source: "/auth/:path*", headers: [{ key: "X-Robots-Tag", value: "noindex" }] }];
  },

The diagram shows the intended behavior for a crawler that follows the rules. In the first panel the crawl is blocked, so the response is never fetched. In the second the crawl is allowed, the response carries the header, and the directive can be read. Removing the block and adding the header happen in the same commit, so the first panel is not claiming that a noindex header already existed behind the block.

Why a header and not a meta tag

A robots meta tag in the page <head> would also work for an HTML page. We chose a response header for these reasons, and they are design choices rather than requirements:

• One entry in next.config.ts covers everything under the path. In the Next.js route syntax :path* matches zero or more path segments, so /auth/callback is covered, and so is any page we add under /auth/ later, without touching that page.
• The rule lives next to the other routing configuration, not scattered across page files.
• The callback is a client component, and we preferred not to build a separate server layout only to carry a robots tag.

Google documents X-Robots-Tag as the way to send the same directive in an HTTP response, and it applies to responses that have no HTML head at all.

What we left alone

The other disallow entries are still in robots.txt: /admin, /dashboard, /api/, /reset-password and /forgot-password. This commit leaves them unchanged. The comment above that list mentions authentication and crawl budget, but it does not establish that every listed route requires a signed-in session, and this post does not evaluate those rules.

For /auth/ the commit did one specific thing: because the goal was for the directive to be readable, the crawl block went away for that path.

The cost we accepted

Allowing the crawl means a crawler can fetch /auth/ URLs again. The comment in the file gave crawl budget, meaning the resources a search engine spends crawling a site, as a reason to block paths like these. We decided that, for this one path, letting the directive be read is worth more than the goal of reducing crawler requests. That is a judgment call for a small site, not a rule. If your auth area is large, or your crawl budget is a real constraint, weigh it differently.

Checking what the server sends

It is worth looking at the actual responses instead of trusting the config file. When I checked on 2026-10-28, robots.txt on the live site no longer listed /auth/:

$ curl -s https://radarmakers.com/robots.txt
User-Agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/
Disallow: /reset-password
Disallow: /forgot-password

Sitemap: https://radarmakers.com/sitemap.xml

And a callback URL with an error parameter answered with the header:

$ curl -sI "https://radarmakers.com/auth/callback?error=access_denied" | grep -i x-robots-tag
x-robots-tag: noindex

That is all this check shows: the server is configured the way the commit intended. It says nothing about what any search engine has done with those URLs.

What this does not tell you

• It is not a measured result. This post describes a configuration and the reasoning for it. It does not report indexing changes, timing or traffic.
• Search engines decide for themselves when to fetch a page again and what to do with it. A crawler that never returns will never see the header.
• noindex is a request to compliant crawlers. It is not access control. Keep the real protection on the server, which is what the authentication is for.
• I describe the behavior Google documents. Other search engines publish their own rules.

Takeaways

• A Disallow stops the crawl, and a crawl is the only way a crawler can read noindex. Do not use both on the same URL when you want the page out of an index.
• Use Disallow to restrict crawling. If the goal is exclusion from search through noindex, let crawlers that support it fetch the directive, whether the URL is new or already known.
• An X-Robots-Tag header in next.config.ts covers a whole path with one entry, including pages you add later.
• Check the real response with curl -I after you deploy.
• Say what you changed and why. Leave out results you have not measured.