Why Your Next.js Page Returns 200 OK but Still Isn't Indexed

Farhan Khan — Mon, 02 Nov 2026 21:12:19 +0000

A page can load perfectly in your browser, return 200 OK, and still be absent from Google Search.

That situation is frustrating because the usual first check—opening the URL and confirming that it works—does not test the things a crawler needs to discover, render and understand the page.

For a Next.js App Router site, I would debug this in layers rather than immediately changing metadata or requesting indexing. The goal is to establish what the server actually returns, what the browser renders, what Google is allowed to fetch, and which URL the page claims as its canonical version.

This guide walks through that process with commands and code you can adapt to a real project.

First, understand what `200 OK` tells you

An HTTP 200 response means that the server successfully handled the request. It does not mean that:

the page contains its intended content in the initial HTML;
search engines are allowed to index it;
the page has a valid canonical URL;
Google has rendered the JavaScript successfully;
the page is included in the sitemap;
Google has selected the URL for indexing.

Google describes JavaScript Search processing as three stages: crawling, rendering and indexing. A page can pass one stage and still have a problem at another. Google generally queues crawlable 200 pages for rendering, but rendering is a separate step and may not happen immediately. Google's JavaScript SEO guide explains the process in more detail.

The practical consequence: treat 200 OK as the beginning of the investigation, not the conclusion.

1. Check the actual HTTP response

Start with the exact URL that is missing from search. Do not test only the homepage or a nearby route.

On macOS, Linux, WSL or Git Bash, run:

URL="https://www.example.com/services/it-support"

curl -sS -L \
  -D response-headers.txt \
  -o response-body.html \
  -w '\nFinal status: %{http_code}\nFinal URL: %{url_effective}\n' \
  "$URL"

This follows redirects, saves the response body and records the response headers. The header file can contain more than one response block when redirects occur.

Inspect the relevant headers:

grep -Ei '^(HTTP/|location:|content-type:|x-robots-tag:)' response-headers.txt

On Windows PowerShell, call curl.exe explicitly to avoid the older curl alias:

$Url = "https://www.example.com/services/it-support"
curl.exe -sS -L -D response-headers.txt -o response-body.html `
  -w "`nFinal status: %{http_code}`nFinal URL: %{url_effective}`n" $Url
Select-String -Path .\response-headers.txt -Pattern '^(HTTP/|location:|content-type:|x-robots-tag:)'

Look for these outcomes:

Result	What to investigate
`200`	Continue. It does not prove indexability.
`301` or `308`	Confirm that the redirect reaches the intended final URL and that internal links use that destination.
`302` or `307`	Check whether the redirect is intentional and stable.
`404`	The route may be missing, incorrectly generated or using the wrong slug.
`5xx`	Investigate deployment errors, server exceptions and upstream data dependencies.
`200` with an error or empty-state page	Possible soft 404 or a route that is returning success for missing content.

A useful test is to compare the response for a real page with the response for a deliberately invalid URL. If both return the same status and effectively the same page, your not-found handling may be wrong.

In the App Router, use Next.js not-found handling for missing resources instead of rendering a normal-looking error message with a successful response. See Next.js error handling and not-found routes.

2. Inspect the HTML the server sends before JavaScript runs

This is one of the highest-value checks for a JavaScript-heavy website.

Open response-body.html and search for the page's actual main heading and copy. You can also run:

grep -oi '<title[^>]*>[^<]*</title>' response-body.html
grep -oi '<h1[^>]*>[^<]*</h1>' response-body.html
grep -oi "<meta[^>]*name=[\"\']robots[\"\'][^>]*>" response-body.html
grep -oi "<link[^>]*rel=[\"\']canonical[\"\'][^>]*>" response-body.html

These quick searches are useful, but HTML can be minified, multiline or ordered differently. For a more reliable first-pass inspection, here is a small Python script using only the standard library.

Save this as inspect_html.py:

from html.parser import HTMLParser
from pathlib import Path
import sys


class PageInspector(HTMLParser):
    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.title_parts = []
        self.h1_parts = []
        self.links = []
        self.robots = []
        self.canonicals = []
        self._in_title = False
        self._in_h1 = False

    def handle_starttag(self, tag, attrs):
        attrs = dict(attrs)

        if tag == "title":
            self._in_title = True
        elif tag == "h1":
            self._in_h1 = True
        elif tag == "meta":
            name = (attrs.get("name") or "").lower()
            http_equiv = (attrs.get("http-equiv") or "").lower()
            if name == "robots" or name == "googlebot":
                self.robots.append(attrs.get("content", ""))
            if http_equiv == "refresh":
                self.robots.append("META REFRESH: " + attrs.get("content", ""))
        elif tag == "link":
            rel = (attrs.get("rel") or "").lower().split()
            if "canonical" in rel:
                self.canonicals.append(attrs.get("href", ""))
        elif tag == "a" and attrs.get("href"):
            self.links.append(attrs["href"])

    def handle_endtag(self, tag):
        if tag == "title":
            self._in_title = False
        elif tag == "h1":
            self._in_h1 = False

    def handle_data(self, data):
        if self._in_title:
            self.title_parts.append(data)
        if self._in_h1:
            self.h1_parts.append(data)


def main():
    if len(sys.argv) != 2:
        raise SystemExit("Usage: python inspect_html.py response-body.html")

    html = Path(sys.argv[1]).read_text(encoding="utf-8", errors="replace")
    parser = PageInspector()
    parser.feed(html)

    print("Title:", " ".join(" ".join(parser.title_parts).split()) or "NOT FOUND")
    print("H1:", " ".join(" ".join(parser.h1_parts).split()) or "NOT FOUND")
    print("Robots directives:", parser.robots or "None found in HTML")
    print("Canonical URLs:", parser.canonicals or "None found in HTML")
    print("Anchor links found:", len(parser.links))
    for href in parser.links[:25]:
        print(" -", href)


if __name__ == "__main__":
    main()

Run it:

python inspect_html.py response-body.html

This script is deliberately a diagnostic aid, not a complete HTML validator or a Googlebot emulator. It reports what is present in the downloaded response. It does not execute JavaScript, resolve relative URLs or decide whether a page should be indexed.

What if the HTML contains only an app shell?

Suppose the server response contains a root element and JavaScript bundles, but none of the page's main text. The page may rely on client-side fetching to render its content.

Google can render JavaScript, so an empty initial response does not automatically prove that the page cannot be indexed. It does mean you have added another dependency: the renderer must fetch the scripts, execute them, obtain the data and produce the content. It also creates problems for crawlers that do not execute JavaScript.

For important public pages, prefer server-rendered or statically generated content where it makes architectural sense. In Next.js, Server Components are the default in the App Router; use Client Components for the interactive parts that actually need browser state or event handlers.

A useful separation is:

Server-render the page's purpose and primary content. The title, main heading, service description and important links should not depend on a button being clicked.
Hydrate interactive elements. Menus, filters, calculators and other client-side controls can remain interactive components.
Fetch critical data before returning the page. Do not show a permanent loading skeleton because a client-side request failed or was delayed.

The question is not whether a framework uses JavaScript. It is whether the essential page is available and understandable when the crawler processes the rendered document.

3. Check `robots.txt`, meta robots and `X-Robots-Tag` separately

These controls are related, but they do different jobs.

robots.txt controls whether a crawler may fetch a URL.
A noindex meta tag or X-Robots-Tag tells a crawler not to index a fetched resource.
A canonical link suggests which URL should represent duplicate or substantially similar content.

Check the robots file:

curl -sS https://www.example.com/robots.txt

Then inspect the response headers for X-Robots-Tag and the HTML for a robots meta tag. A page can be blocked in one place and allowed in another; check the actual URL and any relevant assets, not just the homepage.

A simple Next.js App Router app/robots.ts might look like this:

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: "/",
      disallow: ["/admin/"],
    },
    sitemap: "https://www.example.com/sitemap.xml",
  };
}

Only disallow paths that should genuinely be excluded from crawling. Do not block JavaScript or CSS resources that are needed to render important public pages.

One common mistake is using noindex as if it were a crawl blocker. It is not. A crawler must be able to fetch a page to see its noindex instruction. Conversely, blocking a URL in robots.txt can prevent a crawler from seeing the page-level directive.

Also check for an accidental deployment-wide noindex. It can be introduced through a shared layout, a staging configuration or a response header at the hosting layer. If the rule is inherited from a root layout, individual pages may look correct in their own files while still receiving the wrong directive.

4. Make the canonical URL agree with the URL you want indexed

Canonical problems are particularly easy to miss when a site has multiple hostnames, trailing-slash rules or preview deployments.

For example, these may all resolve to a page that looks identical:

https://example.com/services/it-support
https://www.example.com/services/it-support
https://www.example.com/services/it-support/
https://preview.example.com/services/it-support

Choose the preferred production URL and make redirects, internal links, sitemap entries and canonical tags agree with it.

In a Next.js App Router page or server layout, metadata can be declared like this:

import type { Metadata } from "next";

export const metadata: Metadata = {
  metadataBase: new URL("https://www.example.com"),
  title: "Managed IT Support for Small Businesses in Leeds",
  description:
    "IT support, device management and practical help for small businesses in Leeds.",
  alternates: {
    canonical: "/services/it-support",
  },
};

The metadataBase setting lets Next.js resolve URL-based metadata such as a relative canonical path. Use the real production origin, not a preview or localhost address.

There is an important App Router detail: Next.js supports the metadata object and generateMetadata only in Server Components. If a page begins with "use client", move the metadata export to a server component or parent layout rather than trying to export it from the client component. See the Next.js Metadata API documentation.

After deployment, inspect the actual HTML. Do not assume the source file you edited is the only place emitting canonical tags. Check for:

more than one canonical tag;
a canonical that points to the homepage;
a canonical that points to a preview hostname;
a canonical that points to a different page with different intent;
a canonical that changes after client-side rendering.

Google recommends declaring the canonical consistently in the original HTML. If JavaScript also changes it, the rendered canonical should not contradict the original one. See Google's canonical guidance for JavaScript sites.

5. Make internal links discoverable as links

A page can exist and still be difficult to discover if nothing links to it.

For normal navigation, use an actual anchor with a meaningful href:

import Link from "next/link";

export function ServiceNavigation() {
  return (
    <nav aria-label="Services">
      <ul>
        <li>
          <Link href="/services/it-support">IT support</Link>
        </li>
        <li>
          <Link href="/services/cyber-security">Cyber security</Link>
        </li>
      </ul>
    </nav>
  );
}

Avoid making a div or button act as the only way to navigate to another page. A click handler can work for a human using a browser, but it is not a substitute for a crawlable link with an href.

Google's documentation is explicit that it extracts URLs from the href attribute of links. JavaScript-injected links can be discovered when they are valid anchor elements, but ordinary links in the HTML are easier to inspect and more robust across tools. Crawlable link guidance.

When reviewing a route, ask:

Is there at least one relevant internal link to it?
Does that link point directly to the final canonical URL?
Is the anchor text descriptive enough to explain the destination?
Is the link present in the rendered page, not hidden behind an interaction that a crawler may never trigger?

A sitemap is useful, but it should not be the only discovery path for important pages.

6. Check that the sitemap lists real, canonical, public URLs

For an App Router project, Next.js supports a sitemap.ts convention that generates /sitemap.xml. The official documentation is here: Next.js sitemap files.

Download the production sitemap and check it rather than assuming that a successful build means its entries are correct:

curl -sS https://www.example.com/sitemap.xml -o sitemap.xml

For a sitemap index, inspect the child sitemap URLs as well. For each important entry, verify that:

it returns the expected successful response;
it does not redirect to a different URL unnecessarily;
it is not blocked from crawling;
it is not marked noindex;
its canonical points to itself (or to the intended preferred URL);
it contains useful, current content rather than a placeholder or deleted record.

Do not add every URL your application can technically generate. Exclude private dashboards, search-result combinations, empty category pages, test routes and duplicate parameter variations. A sitemap is a list of URLs you want considered for indexing, not a way to override noindex, canonicalisation or quality decisions.

Also avoid setting every lastmod value to the current build time. Use a real content modification date when you have one. A deployment is not necessarily a content update.

7. Test the rendered page in Search Console

Once the response, directives, canonical and links are correct, use Google's URL Inspection tool in Search Console.

Inspect the exact production URL and compare:

the indexed result and Google's last crawl information;
the user-declared canonical and Google's selected canonical;
the live test result;
the HTML and resources shown in the rendered page;
any page availability or indexing messages.

The live test is useful because it helps separate a current fetch/rendering problem from an older indexed snapshot. It does not guarantee that Google will index the page. After correcting a meaningful issue, you can request indexing, but repeatedly requesting the same unchanged URL is not a fix for the underlying problem.

If the rendered HTML is missing the main content, investigate the data request, the component boundary and any blocked scripts or API dependencies. If the content is present but Google selects another canonical, investigate duplication and URL consistency instead. If the URL is not known to Google, improve internal discovery and check sitemap coverage.

A compact debugging order

When a Next.js page is missing from search, I would check it in this order:

Order	Check	Evidence to collect
1	Exact URL and response	Final status, redirects and response headers
2	Initial HTML	Title, H1 and meaningful page content in the downloaded body
3	Indexing directives	`robots.txt`, meta robots and `X-Robots-Tag`
4	Canonical	One correct canonical matching the preferred production URL
5	Links	Crawlable internal `<a href>` links to the page
6	Sitemap	The final canonical URL appears in the appropriate sitemap
7	Google rendering	Search Console URL Inspection and rendered HTML

This order prevents a common waste of time: rewriting content or changing keywords when the actual problem is a redirect, a noindex header or a route that returns an empty application shell.

Final thought

A successful browser load is a user-facing test. Indexability is a separate technical outcome that depends on access, rendering, directives, canonicalisation, discovery and Google's own selection systems.

For a Next.js site, the fastest route to an answer is to inspect the deployed URL from the outside first. Start with the HTTP response and HTML that actually exist in production. Then compare that evidence with the rendered page and Search Console. It is much more reliable than guessing from the component tree or repeatedly changing metadata.

AI assistance disclosure: This article was prepared with AI assistance. The publisher should test the commands against a real project, verify the framework-specific details for the installed Next.js version, and edit the article to reflect their own technical understanding before publication.

DEV Community: Farhan Khan

Why Your Next.js Page Returns 200 OK but Still Isn't Indexed

First, understand what `200 OK` tells you

1. Check the actual HTTP response

2. Inspect the HTML the server sends before JavaScript runs

What if the HTML contains only an app shell?

3. Check `robots.txt`, meta robots and `X-Robots-Tag` separately

4. Make the canonical URL agree with the URL you want indexed

5. Make internal links discoverable as links

6. Check that the sitemap lists real, canonical, public URLs

7. Test the rendered page in Search Console

A compact debugging order

Final thought

Further reading

Technical references

DEV Community: Farhan Khan

Why Your Next.js Page Returns 200 OK but Still Isn't Indexed

First, understand what 200 OK tells you

1. Check the actual HTTP response

2. Inspect the HTML the server sends before JavaScript runs

What if the HTML contains only an app shell?

3. Check robots.txt, meta robots and X-Robots-Tag separately

4. Make the canonical URL agree with the URL you want indexed

5. Make internal links discoverable as links

6. Check that the sitemap lists real, canonical, public URLs

7. Test the rendered page in Search Console

A compact debugging order

Final thought

Further reading

Technical references

First, understand what `200 OK` tells you

3. Check `robots.txt`, meta robots and `X-Robots-Tag` separately