Filters and sort options on a category page can multiply a few hundred products into millions of crawlable URLs. Most of those URLs are near-duplicates, and letting a crawler wander through all of them wastes requests and dilutes signals. The standard advice sounds simple ("block them", "canonicalise them"), but the three tools do different jobs, and using the wrong one gives a result you did not intend.
What each tool actually does
-
robots.txtdisallow stops compliant crawlers from fetching a URL. It does not remove the URL from the index if other pages link to it, and Google cannot see anoindexor a canonical on a page it is not allowed to fetch. -
rel="canonical"is a hint that a page is a duplicate of another URL. Google may follow it or may not, and the duplicate still has to be crawled for the hint to be read. -
noindex(meta robots orX-Robots-Tag) keeps a crawlable page out of the index. The page still gets crawled, so it does not save crawl requests.
A decision table
| Facet type | Example | Index? | Treatment |
|---|---|---|---|
| Has real search demand and a unique page purpose | /shoes/running/ |
Yes | Clean URL, unique title and copy, self-referencing canonical, in the sitemap |
| Combination with demand but fragile | /shoes/running/red/ |
Maybe | Only if the page has distinct content and stock; otherwise canonical to the parent |
| Sort order | ?sort=price_asc |
No | Canonical to the unsorted URL; keep out of internal links |
| Pagination | ?page=2 |
Yes, as its own page | Self-referencing canonical; do not canonicalise page 2 to page 1 |
| Stackable filters (3+ at once) | ?color=red&size=9&brand=x |
No | Disallow in robots.txt if crawling them is the problem; make the links non-crawlable |
| Internal search results | /search?q= |
No |
noindex and disallow, depending on how they are linked |
| Session or tracking parameters |
?sessionid=, ?utm_
|
No | Do not generate them in internal links; canonical to the clean URL |
Order of operations
- Decide which facets deserve to exist as pages. This is a product and SEO decision, not a technical one. A facet earns a URL only if people search for it and you can make the page useful.
-
Stop generating crawlable links to everything else. Filters implemented as buttons or form posts, not plain
<a href>links with parameters, are the cleanest fix because the URLs are never discovered. - Add canonicals on the variants that will still exist as URLs.
-
Use
robots.txtlast, and only for patterns that are already not worth crawling. Do not disallow a URL you also want de-indexed throughnoindex, because the crawler will never see the tag. - Measure. Compare crawled URLs in your logs and the pages in the Search Console indexing report before and after.
A quick way to see how big the problem is: count the distinct parameter combinations in a crawl or log export. If that number is many times larger than your real catalogue, the site is producing crawl space you did not design.
Rebuilding filters and templates so the crawlable set matches the pages you actually want indexed is work our team handles under web platform development.
References
- Google Search Central: Managing crawling of faceted navigation URLs
- Google Search Central: Consolidate duplicate URLs
- Google Search Central: Introduction to robots.txt
Top comments (0)