Discovered, not indexed: the Shopify crawl bug clients hit

TL;DR

• "Discovered - currently not indexed" means Google found the URL but hasn't fetched it yet; it says nothing about content quality (that's the neighboring "Crawled - currently not indexed" status).
• On Shopify the usual cause is URL inflation: recommendation tracking parameters, collection-scoped product URLs, and app-added query strings that the default robots.txt doesn't block.
• The fix lives in robots.txt.liquid and internal link markup, not in rewriting product copy or spamming "Request indexing."

A client's Shopify store shows a growing pile of URLs stuck under "Discovered - currently not indexed" in Search Console, and nothing in the site throws an error. No noindex tag, no 404, no robots block anyone remembers writing. This status shows up on stores you build and maintain, not just ones you audit after the fact, because it's produced by defaults nobody chose: Shopify's Product Recommendations API appends tracking parameters to product links, collection pages create a second URL for every product they list, and apps bolt on filters and widgets with their own query strings. None of it is flagged as an SEO risk anywhere in the app install flow. This article walks through what the status actually means, why Shopify stores generate far more crawlable URLs than products, and the exact checks (view-source, robots.txt, sitemap counts) you can run against a client's store in about five minutes.

What does "Discovered - currently not indexed" actually mean?

Google's Page indexing report documentation defines it precisely: "The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl."

Compare that with the status directly beneath it in the same report: "The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling."

One word apart, and the fixes are opposite. A crawled page has been fetched and read, so its title, markup and content have actually been judged. A discovered page has been seen by nothing at Google. Editing product copy on a page in that state changes nothing, because the change isn't visible to the entity that put it in the queue. The first question to ask on a client's store isn't "what's wrong with these pages," it's "what is the crawler doing instead."

Why does Google flag this as a crawl-budget signal?

Google opens its large-site crawl budget guide by narrowing who needs it: "If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." Two of the three qualifying conditions it lists are about raw size (a million pages, or ten thousand changing daily). The third isn't:

• "Sites with a large portion of their total URLs classified by Search Console as Discovered - currently not indexed"

Google is naming the status itself as a qualifying condition, independent of catalog size. A four-hundred-product store isn't a large site by any normal reading, but if its Page indexing report is dominated by this status, it's in scope. The real question isn't how many products the client sells, it's how many addresses the store hands out for them.

Why does a Shopify store have more URLs than products?

A Shopify store doesn't present one URL per product. It presents the canonical product URL plus every alternative address the platform, the theme, and installed apps are willing to link. Three mechanisms account for most of it, and none is visible in the admin.

Where do the recommendation tracking parameters come from?

Shopify's Product Recommendations API reference documents this directly: "the url property for each product in the products response contains URL parameters that lets you build a conversion funnel." The example URL in Shopify's own docs:

/products/gorgeous-wooden-computer?pr_choice=default&pr_prod_strat=description&pr_rec_pid=13&pr_ref_pid=17&pr_seq=alternating

Those parameters exist so Shopify's reports can attribute a sale to a recommendation strategy. The side effect: the same product page gets a distinct address for every combination of referring product and strategy. A Shopify Community thread from March 2024 describes roughly 80,000 parameter URLs discovered on one store. A case study by developer Eduard Fastovski describes a 200-product store that accumulated over a million indexed URLs (plus several million more not indexed) from ?pr_prod and ?filter parameters; after blocking both patterns: "It worked! Down to just 1k pages now, from 5 million!"

What lets this persist: Shopify's default robots.txt blocks the recommendations endpoint, not the URLs it hands out. Checking the live robots.txt of two unrelated Shopify stores shows the standard Disallow: /recommendations/products rule, and no rule matching pr_prod_strat, pr_rec_id, pr_rec_pid, pr_ref_pid, or pr_seq. The Ajax endpoint is closed; every product URL carrying those params, wherever it's linked from (a theme section, an app, a third-party sitemap), stays fully crawlable.

Why do collection-scoped product URLs duplicate every product?

Shopify creates a second address whenever a product card links through its collection: /collections/summer-sale/products/linen-shirt alongside /products/linen-shirt. Shopify's own Liquid reference for the within filter warns: "Because a standard product page and a product page in the context of a collection have the same content on separate URLs, you should consider the SEO implications of using the within filter."

A product sitting in six collections can be reached at seven addresses. The default robots.txt blocks sorted and filtered collection URLs (Disallow: /collections/*sort_by*, Disallow: /collections/*+*), but nothing blocks /collections/*/products/*. Those URLs carry a canonical tag pointing back at the clean product URL, which is correct, but a canonical is only readable after a fetch. Consolidating them still costs a crawl each.

What URLs are your client's apps adding without telling anyone?

The default file blocks named patterns, not parameters in general: oseid, preview_theme_id, preview_script_id, a specific duplicated ls= shape, sorted/filtered collections, and /search. Any parameter an app or theme introduces outside those patterns is crawlable by default: a filter app with its own query format, a "recently viewed" widget, a wishlist link, a campaign parameter that leaks into an internal link instead of staying outbound-only. No app declares its URL footprint at install time, so this is invisible until it shows up in Search Console weeks later.

How do you check a client's store in five minutes?

1. Read the shapes, not the count. Open Page indexing, select Discovered - currently not indexed, and bucket the sample URLs: clean /products/…, collection-scoped /collections/…/products/…, and anything with a ?. The proportions tell you which mechanism above is dominant.
2. Compare what Google knows against what the sitemap declares. Add the indexed and not-indexed totals, then open yourstore.com/sitemap.xml. Shopify auto-generates it with separate child sitemaps for products, collections, blogs, and pages. If Google knows an order of magnitude more URLs than the sitemap declares, the surplus arrived through internal links.
3. View source on a product page and search for pr_prod_strat. If the theme renders a recommendations block into the page, the parameters sit in the href attributes. Search for pr_rec_pid too.
4. Read yourstore.com/robots.txt properly. Check whether anything blocks the parameter shapes from step 3, and whether a past edit went further than intended. Shopify's own help page warns that "Incorrect use of the feature can result in loss of all traffic."
5. Inspect one clean product URL, not fifty. Run URL Inspection and note the last crawl date. Google's documentation states this "is not a live test" and that there's "a daily limit of inspection requests for each property." One URL is a diagnosis; a full catalog isn't auditable this way.

How do you fix each cause?

Parameter URLs. Two real options, with a trade-off. Strip the parameters at the theme level (loses the conversion-funnel attribution those params exist for), or block the shapes in robots.txt.liquid (keeps tracking working for real visitors, stops Google from fetching them). Google's guidance is explicit about method: "Block crawling of URLs using robots.txt," and separately, "Don't use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time." A noindex on a parameter URL is the intuitive fix and it makes the crawl problem worse. Edit robots.txt.liquid carefully and re-check it after theme updates, since it's theme code and a theme change can silently overwrite your edit.

Collection-scoped duplicates. Where a product card doesn't need collection context, link to the plain product URL instead of the collection-scoped one. Google's crawl guidance: "Eliminate duplicate content to focus crawling on unique content rather than unique URLs."

Products that are genuinely gone. Return a 404 or 410, per Google's guidance, rather than redirecting a deleted product's URL to something convenient, which just keeps a competing URL alive in the queue.

Server response. Google's own definition of the status mentions overloading the site as a possible cause. Efficient page loads, HTTP caching, and 304 responses all matter here; heavy third-party scripts and app-injected assets are usually the biggest levers you actually control.

What not to do: don't work down the list requesting indexing on every URL. Google states "Submitting a request does not guarantee that the page will appear in the Google Index," and the daily limits make a catalog-sized pass impossible anyway. It's a diagnostic tool, not a remedy, and it leaves the underlying crawl surface untouched.

Why does this bug keep coming back after you fix it?

Every mechanism above comes from ordinary store maintenance, not a one-time mistake. A merchandiser adds three collections and every product in them gains three addresses. A theme update ships a new recommendations section. An app adds a filter with a query format nobody reviewed. A seasonal range gets deleted and redirected instead of allowed to 404. None of it announces itself, and the crawl surface only grows unless someone deliberately shrinks it. Worth building a recurring check into your maintenance routine (robots.txt diff after theme updates, periodic sitemap-vs-indexed count) rather than treating this as a one-time cleanup.

Has anyone found other URL shapes leaking through a Shopify store's default robots.txt, beyond the recommendation params and collection-scoped duplicates covered here?

The full version of this article — with screenshots and ongoing updates — lives on the StoreCanary blog.