Shopify robots.txt.liquid is blocking your clients' products

#shopify #webdev #seo #tutorial

TL;DR

robots.txt on Shopify is generated from robots.txt.liquid inside the theme. Any app or developer with theme access can add a Disallow rule that blocks Googlebot from product pages, with zero warning in the admin.
The only signals are in Google Search Console (Page indexing report, "Blocked by robots.txt" status) and Merchant Center diagnostics.
Adding noindex to a blocked page will not fix it. Googlebot must be unblocked first so it can read the tag.

A single line in robots.txt.liquid can silently drop every product in a client's Shopify catalogue from Google's crawl queue. The store looks healthy: pages are active, the theme is live, Shopify reports no errors. Google just stops fetching the URLs.

This happens because Shopify generates /robots.txt from a Liquid template inside the theme, not from a static file. Any app, developer, or agency with theme code access can modify it. The most common pattern: an SEO app or a previous developer adds a Disallow rule that overreaches, catching /products/ paths alongside the intended targets like /admin or /cart.

This article covers how to detect the block in Search Console, read the correct user-agent group in robots.txt.liquid, remove the offending rule, and verify that both Googlebot and Googlebot-image can reach product pages and images. It also explains why noindex on a blocked page will not work.

Is a crawl block the same as a noindex?

No, and the difference is the source of most failed fixes.

A noindex tag says: fetch this page, then keep it out of the index. A robots.txt Disallow says: do not fetch this page at all. The second sounds stricter. It is actually weaker, because a page Google never fetches is a page Google knows nothing about, including whether you wanted it indexed.

Google Search Console names two separate statuses here. "Blocked by robots.txt" means the URL was blocked; Google notes this does not guarantee the page stays out of the index. "Indexed, though blocked by robots.txt" means it appeared in results anyway, built from signals on pages that link to it.

For a product page, the second status is the worst outcome: the URL surfaces in results, but Google has no title, description, price, or structured data to build a listing from.

Why does this appear in client stores that nobody touched?

Because robots.txt on Shopify is not a file you upload once. It is generated from robots.txt.liquid, a template in the templates/ folder of the published theme. Shopify's own documentation for editing robots.txt.liquid calls this an unsupported customization, warns that Shopify Support cannot help with edits, and states that incorrect use of the feature can result in loss of all traffic.

Anything with theme code access can write to it: an SEO app installed and later forgotten, an agency hardening the file during a migration, a developer restoring a theme backup. A Shopify Community thread from March 2025 shows the typical pattern: 34 blocked product pages traced to Disallow rules added by three installed apps.

Shopify's defaults are sensible: /admin, /cart, /checkout, /search, /policies/, and filtered collection URLs are blocked out of the box and should stay that way. The damage comes from what gets added beneath them.

Why will adding noindex to a blocked page not fix it?

This catches experienced developers. If a page shows as "Indexed, though blocked by robots.txt", the instinct is to add a noindex meta tag. Google's documentation says directly that this will not work: for a noindex rule to be honored, the page must not be blocked in robots.txt, because a blocked crawler never reads the tag.

The correct order: unblock first, so Googlebot can read the instruction. This is the inverse of the phantom noindex pattern. There, Googlebot is allowed in but told to forget the page. Here, Googlebot is not allowed in at all, so it receives no instruction.

What does this cost in Merchant Center?

Google Merchant Center re-crawls the landing page behind every feed item. Its documentation lists a specific failure for pages it cannot reach: product landing pages that robots.txt forbids Googlebot to download. Those products remain disapproved and stop showing in Shopping ads and free listings until the crawler can reach the page.

Two user agents matter: Googlebot for landing pages and Googlebot-image for images. A rule that only blocks an image CDN path breaks image crawling while landing pages appear unaffected.

One parsing rule worth knowing before editing: Google's robots.txt specification resolves conflicts by longest matching path. A named User-agent: Googlebot block replaces the wildcard block entirely for that crawler, so a permissive * group gives no protection if a restrictive Googlebot group exists below it.

How do you check a client store in five minutes?

Read the live file. Open https://clientstore.com/robots.txt in a browser. It is public with no login needed. Read every Disallow line and check whether any could match a product URL.
Open Search Console, Page indexing. Look for "Blocked by robots.txt" and "Indexed, though blocked by robots.txt". Each status lists the affected URLs, which is faster than reasoning about wildcard patterns by hand.
Run URL Inspection on a real product URL. It reports directly whether crawling was allowed.
Find the template. Online Store, Themes, Edit code on the published theme, then the templates/ folder. If robots.txt.liquid is listed, the store runs a customized file worth reading in full.
Check Merchant Center diagnostics for product-page crawl issues, which name the URLs Google could not download.

How do you fix it?

Remove or narrow the overreaching Disallow rule in robots.txt.liquid on the published theme. Leave Shopify's defaults for /cart, /checkout, and /admin intact; they are there for good reasons.

Check which user-agent group actually applies to Googlebot. A named User-agent: Googlebot block replaces the wildcard block for that crawler; a clean * group gives no protection if a restrictive Googlebot group exists below it.

Do not check immediately after saving. Google generally caches robots.txt for up to 24 hours, and Merchant Center states a change may take 24 to 72 hours to reflect. Re-run URL Inspection after that window.

If the pages genuinely need to stay out of the index: unblock in robots.txt.liquid first, then add noindex, in that order.

Why does this bug come back?

robots.txt.liquid belongs to a theme, not to the store. Whichever theme is published determines what Google is served. Shopify documents that uploading a theme through the admin's Themes section will not import robots.txt.liquid, while ThemeKit or command-line deployments preserve it. Swapping the published theme can quietly replace the rules Google reads with no visible change in the admin.

Add an app with theme code access or a developer restoring a backup, and the same Disallow line can reappear months later with no symptom in the admin, only on Google's side.

Have you traced a client's lost Merchant Center traffic back to a robots.txt.liquid change? What process do you use to catch it before the client does?

The full version of this article, with screenshots and ongoing updates, lives on the StoreCanary blog.