Why your pages are crawled but not indexed

"Crawled, currently not indexed" in Search Console gets treated as one problem with one fix,
usually "publish better content." The status covers several distinct failure modes, and content
quality is only one of them. The other three are almost always visible in the server response
itself, if anyone actually diffs what Googlebot receives against what a browser receives.

What "crawled but not indexed" actually means

Google fetched the URL, read the response, and made a deliberate decision not to add it to the
index. That is different from "discovered, not crawled", which is a budget or robots issue, and
different from a canonical replacing the URL with another one Google chose to index instead. The
Search Console UI collapses these into similar-looking rows. The API is more precise, and it is the
only practical way to check hundreds of URLs without clicking through the UI one at a time.

Querying status through the API

The URL Inspection API exposes the same data the UI shows for a single URL, built for automation.
Each call needs the property's verified siteUrl and the full inspectionUrl, sent to
urlInspection.index.inspect, which maps to POST /v1/urlInspection/index:inspect on the Search
Console API.

const { google } = require('googleapis');

async function inspectUrl(auth, siteUrl, inspectionUrl) {
  const searchconsole = google.searchconsole({ version: 'v1', auth });
  const res = await searchconsole.urlInspection.index.inspect({
    requestBody: { siteUrl, inspectionUrl },
  });
  const result = res.data.inspectionResult.indexStatusResult;
  return {
    url: inspectionUrl,
    verdict: result.verdict,               // PASS, PARTIAL, FAIL, NEUTRAL
    coverageState: result.coverageState,    // e.g. "Crawled - currently not indexed"
    lastCrawlTime: result.lastCrawlTime,
    googleCanonical: result.googleCanonical,
    userCanonical: result.userCanonical,
  };
}

The quota on this endpoint is small, so a full-site sweep needs to run as a slow batch rather than
a loop firing as fast as the client allows. Space the calls, and cache the results, since a page's
status does not move minute to minute.

The two canonical fields are the first thing worth comparing. When googleCanonical and
userCanonical disagree, Google decided the site's own canonical tag was wrong and picked a
different URL to index in its place. That alone explains a large share of "crawled, not indexed"
rows, and it needs no further diagnosis beyond fixing the tag.

When the tag is correct but the server still lies

If the canonical fields agree and the page still is not indexed, the next check is whether
Googlebot and an ordinary visitor are looking at the same document. Some setups serve different
HTML by user agent, deliberately or by accident, through a caching layer or a bot-protection rule
that treats crawlers differently from everyone else.

#!/usr/bin/env bash
# Compare what Googlebot fetches against what a browser fetches
URL="$1"

curl -s -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  -o googlebot.html -w "googlebot: %{http_code}\n" "$URL"

curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
  -o browser.html -w "browser:   %{http_code}\n" "$URL"

diff <(grep -o '<title>.*</title>' googlebot.html) \
     <(grep -o '<title>.*</title>' browser.html)

A status code mismatch is easy to catch this way. A content mismatch hiding inside two identical
200 responses is the one that survives for months, because nothing in the logs looks like an
error.

The failure that produces no error at all

JavaScript-rendered content is the quieter version of the same problem. Googlebot fetches the
initial HTML, queues the page for rendering, executes the scripts, and only then sees the final
DOM. If a script throws, if a third-party dependency times out, or if the content depends on a
client-side fetch that the render sandbox blocks, the rendered page ends up thinner than the one a
visitor sees, with an HTTP 200 at every step along the way. There is no error to alert on. The
only way to catch it is to render the page the way the crawler does and diff the result against the
server-side output, the same comparison as above, run against the rendered DOM rather than the raw
response.

Redirect chains as a slow leak

A URL that resolves through two or three redirects before reaching its final destination usually
still gets indexed, but it competes for crawl budget against everything else on the domain, and
each additional hop is another place for a canonical or a status code to quietly diverge from what
was intended. Checking chain length is a short loop built on curl -I --location-trusted with
manual redirect handling, and it is worth running across the whole URL list on the same cadence as
the nooralto.com technical audits that catch
this alongside the canonical and rendering checks above.

Reading the four checks together

None of these checks needs the content rewritten first. Canonical agreement, user-agent parity,
rendered-DOM parity and redirect chain length are structural facts about the response, and they
answer whether Google is capable of indexing the page before the conversation moves to whether the
page deserves to be. Running the content-quality argument before ruling these out wastes a sprint
on the wrong hypothesis.

Written by the team at Nooralto, a web and SEO studio working out of Agadir and Paris.