DEV Community

Roman SEO
Roman SEO

Posted on Fully Autonomous

Debugging "Discovered" vs "Crawled, currently not indexed": sitemap, internal links and canonical

If you build sites for clients, sooner or later someone forwards you a Search Console screenshot with "Crawled, currently not indexed" and asks what's broken. Often nothing is broken in the usual sense. The page returns 200, renders fine and passes every check you'd run locally. Google just isn't including it.

Here's how I approach these two statuses from the technical side.

Know which stage failed

  • Discovered, currently not indexed: the URL is in Google's queue (from a sitemap or a link) but hasn't been fetched. This is a crawl scheduling problem.
  • Crawled, currently not indexed: the URL was fetched and processed, and Google chose not to index it. This is a quality or duplication decision.

Different stage, different fix. Start by opening the URL in "Inspect URL" and noting four things: sitemap presence, referring pages Google knows about, last crawl date, and the canonical Google selected versus the one you declared.

Sitemap: make it a list of pages you actually want indexed

A common pattern on Shopify, WordPress and custom builds is a sitemap that includes everything: tag archives, filtered collections, paginated URLs, parameter variants. That dilutes the signal and burns crawl time on near-duplicates.

Things to check:

  • only canonical, indexable, 200 status URLs in the sitemap;
  • no URLs that are noindex or redirect;
  • filters, sort orders and internal search kept out;
  • lastmod only changes when content really changes.

Internal links: the most underrated fix for "Discovered"

A URL that exists only in the sitemap and isn't reachable from navigation looks unimportant. If an important page has zero or one internal link pointing at it, add links from the menu, the parent category, the homepage or related pages. Plain <a href> links in the rendered HTML, not something injected only on click.

Also check Crawl stats in Search Console settings. Slow response times or 5xx errors make Google crawl more cautiously, which shows up as a growing Discovered list.

Canonical: where "Crawled" often comes from

For "Crawled", look at duplication first:

  • product variants (color, size) on separate URLs with near-identical content: point rel="canonical" to the main version;
  • city or service pages generated from one template with swapped names: merge or make them genuinely different;
  • two pages targeting the same topic: keep the stronger one and 301 the other.

If "Inspect URL" shows Google picked a different canonical than yours, that's your signal that Google considers the pages duplicates.

Things that don't help

Repeatedly hitting "Request indexing" doesn't speed anything up, and there's a daily quota. The Indexing API is officially intended for job posting and livestream pages, so using it for regular pages isn't a fix.

Verify

After shipping fixes, use "Validate fix" on the status group in the Pages report, then watch the Performance report for impressions on those URLs over the next few weeks.

I put together a fuller walkthrough (which pages are fine to leave in these lists, a 10 minute diagnosis order and a FAQ) on my site: Crawled or Discovered, currently not indexed: how to fix it.

Top comments (0)