How We Tackled 1,600 “Crawled – Currently Not Indexed” URLs on Shopify

Jeffrey Del Rosario — Tue, 15 Sep 2026 14:58:50 +0000

If you manage a large Shopify store, Google Search Console can occasionally show you a number that looks much worse than it actually is.

For us, that number was roughly 1,600 URLs under:

Crawled – currently not indexed

At first glance, the obvious reaction is:

We need to get all 1,600 pages indexed.

That turned out to be the wrong goal.

The more useful question was:

Which of these URLs actually deserve to be indexed?

That distinction changed how we approached the entire problem.

This article explains the workflow we used to audit the affected URLs, identify patterns, prioritize important pages, and avoid wasting time trying to force low-value URLs into Google's index.

What “Crawled – currently not indexed” actually tells you

The important word here is crawled.

Google already discovered the URL and successfully requested it.

So this is different from a discovery or accessibility problem.

The page was crawled, but Google decided not to include it in the index at that point.

That does not automatically mean something is technically broken.

A URL can be perfectly accessible and still not be worth indexing.

For an e-commerce store, this distinction matters because one product catalog can generate many different types of URLs:

product pages
collection pages
blog posts
parameter URLs
filtering variations
pagination
outdated products
near-duplicate pages
utility pages

Treating all of them as equally important creates a lot of unnecessary work.

Our situation

The issue appeared while working on MySeeds, a Shopify store with more than 1,000 products.

Google Search Console showed roughly 1,600 URLs in the “Crawled – currently not indexed” bucket.

Instead of inspecting them randomly one by one, we exported the URLs and started looking for patterns.

That was the first useful breakthrough.

Step 1: Group URLs before investigating them

An export containing 1,600 URLs looks intimidating.

But 1,600 individual problems rarely means there are 1,600 different causes.

Most large indexing problems are really a handful of patterns repeated hundreds of times.

We started by grouping URLs based on their structure.

For example:

/products/
/collections/
/blogs/
/pages/
?sort_by=
?filter=
?page=

You can automate much of this classification with a simple script.

from urllib.parse import urlparse
import pandas as pd

df = pd.read_csv("gsc-urls.csv")

def classify_url(url):
    parsed = urlparse(url)
    path = parsed.path
    query = parsed.query

    if "/products/" in path:
        return "Product"

    if "/collections/" in path:
        if query:
            return "Collection with parameters"
        return "Collection"

    if "/blogs/" in path:
        return "Blog"

    if "/pages/" in path:
        return "Page"

    if query:
        return "Other parameter URL"

    return "Other"

df["url_type"] = df["URL"].apply(classify_url)

print(df["url_type"].value_counts())

The exact script is not important.

The principle is.

Turn a large URL problem into several smaller pattern problems.

Once we did that, the report became much easier to reason about.

Step 2: Decide whether each URL type should be indexed at all

Before trying to “fix” a URL, we asked:

If Google indexed this page tomorrow, would we actually want it appearing in search results?

That question immediately eliminates a lot of noise.

We used a decision model similar to this:

URL type	Usually index-worthy?	Typical action
Important product	Yes	Investigate
Main collection	Yes	Investigate
Useful article	Yes	Investigate
Core informational page	Yes	Investigate
Sort URL	Usually no	Leave excluded
Filter variation	Usually no	Check canonical/indexing setup
Duplicate URL	No	Consolidate
Empty/thin collection	Maybe not	Improve, merge or remove
Discontinued product	Depends	Redirect, retain or remove deliberately

This is where the original number started becoming much less scary.

The objective was never:

Indexed URLs = maximum possible

The objective was:

Important URLs = discoverable, crawlable, canonical and useful enough to index

Those are very different goals.

Step 3: Compare indexing signals

For URLs that genuinely mattered, we looked for conflicting signals.

A page might technically exist while sending Google mixed messages about whether it should be treated as the primary URL.

We checked four things together:

Internal links
      ↓
Preferred URL
      ↓
Canonical
      ↓
XML sitemap

Ideally, all of them point toward the same URL.

For example, imagine that a product can effectively be encountered as:

/products/example-product

while different navigation paths or parameters expose variants of that URL.

If your sitemap references one version, internal links frequently point toward another, and canonical signals say something else, you are making Google's job harder than necessary.

Canonicalization is not simply about inserting a canonical tag.

It is about making your signals consistent.

Step 4: Look at internal linking

One of the easiest things to underestimate during an indexing audit is internal linking.

A page may be technically accessible while being practically invisible inside the site architecture.

We asked questions such as:

How many internal links point to this page?

Is it linked from a major collection?

Can a visitor reach it naturally from the store navigation?

Is it several layers deep?

Does another page receive most of the relevant internal links instead?

For important pages, we wanted Google to see clear evidence that the page was part of the primary site structure.

This was particularly important for products, collections and informational content we actually wanted to rank.

A sitemap can help Google discover a page.

But a sitemap does not replace a good internal site architecture.

Step 5: Check whether the page actually adds value

Technical SEO was only part of the investigation.

Some URLs were technically fine.

That did not necessarily make them useful.

Consider two collection pages containing almost the same products, similar headings and very little unique context.

Both can be:

200 OK
crawlable
indexable
self-canonical
included in the sitemap

And Google may still decide that only one is worth indexing.

This is why we also reviewed page purpose.

For important pages we looked at things like:

Search intent
Unique content
Product selection
Page purpose
Internal links
Overlap with another page

In some cases, improving a page was appropriate.

In others, the better solution was consolidation.

And sometimes the correct decision was simply to stop worrying about that URL.

Step 6: Keep the sitemap clean

A sitemap is another useful prioritization signal.

For a large store, we want it to represent pages we genuinely consider useful search destinations.

That means we do not want to think about the sitemap as:

Here is every URL the platform is capable of producing.

Instead:

Here are the canonical pages we believe search engines should discover and consider.

For important URLs, we checked whether the preferred version was represented consistently.

This was especially useful after identifying duplicate or outdated URL patterns.

Step 7: Inspect samples, not every URL manually

Manually inspecting 1,600 URLs would have been inefficient.

Once we had clusters, we selected representative URLs from each group.

For example:

5 important products
5 secondary products
5 collections
5 blog articles
5 parameter URLs
5 older URLs

We then compared their behavior.

If ten URLs sharing the same template show the same indexing pattern, investigating the template or URL type is generally more useful than repeating the exact same manual inspection hundreds of times.

Of course, high-value pages still deserve individual attention.

But large-scale SEO problems should usually be investigated pattern first, URL second.

What we changed

After the audit, the work became much more targeted.

Instead of attempting to push every excluded URL into Google, we focused on four areas:

Index-worthy pages: important products, collections and content that should genuinely be searchable.

Consolidation: URLs that duplicated another page or competed with a stronger destination.

Internal linking: pages that deserved more visibility inside the site architecture.

Content quality: pages whose purpose or differentiation was not strong enough.

That also meant intentionally leaving some URLs unindexed.

And that is perfectly fine.

Google Search Console itself distinguishes between URLs that are not indexed because of a problem and URLs that simply do not need to be indexed.

What we deliberately did not do

We did not repeatedly submit hundreds of URLs through URL Inspection hoping Google would eventually accept them.

We did not assume that every non-indexed URL represented lost traffic.

And we did not create extra text just to make every page longer.

Those approaches address the number displayed in Search Console rather than the underlying website architecture.

Our goal was to improve the set of pages Google had good reasons to index.

The most important lesson

The biggest lesson from the project was surprisingly simple:

Indexing is not a numbers game.

Having 10,000 indexed URLs is not necessarily better than having 3,000.

What matters is whether the right 3,000 pages are indexed.

When a large e-commerce site shows hundreds or thousands of excluded URLs, start with classification.

Ask:

What type of URL is this?

Then:

Should this type of URL be indexed?

Only after answering those questions should you investigate why an individual page is missing from the index.

That approach turns a Search Console report containing thousands of URLs into a manageable technical SEO project.

A reusable workflow

Our current process can be summarized as:

Export affected URLs
        ↓
Group by URL pattern
        ↓
Decide which groups deserve indexing
        ↓
Inspect representative samples
        ↓
Check canonical + sitemap + internal links
        ↓
Evaluate content and search intent
        ↓
Fix patterns instead of individual URLs
        ↓
Monitor Search Console

For Shopify stores in particular, this is much more scalable than treating every URL as an independent SEO problem.

And perhaps more importantly, it prevents you from spending weeks trying to index pages that Google never needed in the first place.

DEV Community: Jeffrey Del Rosario