DEV Community

seotto.com
seotto.com

Posted on Edited on

Your site search creates infinite URLs: an SEO checklist for internal search result pages

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com

Internal site search is usually built as a product feature, not an SEO surface. A search box, a /search?q= route, a results template, done. Then six months later someone opens Search Console and finds thousands of URLs like /search?q=cheap+watches+free+shipping under "Crawled, currently not indexed", or worse, indexed.

Search result pages are an unusual kind of page: there's an unlimited number of them, anyone can create new ones by typing or linking, and most of them are thin. That combination makes them one of the most common sources of crawl waste and index bloat on content and e-commerce sites. The fixes are mostly a few lines in your templates and routing. Here's the checklist.

Indexing

1. Decide whether any search pages should be indexed at all.
For most sites the answer is no. Search result pages duplicate your category, tag and product pages with worse structure. Google's own guidelines have long recommended keeping auto-generated search results out of the index. Make the default explicit: <meta name="robots" content="noindex"> on the search results template.

2. If some should be indexed, turn them into real pages.
Sometimes a handful of queries are genuinely valuable, for example "red running shoes" on a shop with no such category. Don't index the /search?q= version. Create a proper landing page with its own URL, a written intro, a stable product set and internal links to it. That's a category page, not a search page, and it should be built like one.

3. Use noindex, not only robots.txt, for pages already in the index.
If search URLs are already indexed, blocking them in robots.txt stops Google from recrawling them, which means it never sees the noindex. Add noindex first, let the URLs drop out, then consider a robots.txt rule to save crawl budget.

Crawl traps

4. Don't link to search URLs from crawlable pages.
"Popular searches" widgets, tag clouds that link to /search?q=tag, and "did you mean" links all hand crawlers an endless supply of query URLs. Link to real category or tag pages instead, or render those widgets without crawlable <a href> links if they're purely for users.

5. Make search forms submit with GET, but don't let pagination and sorting multiply.
GET is right for search because users can bookmark and share results. But q × page × sort × filter creates combinatorial URLs. Keep parameters in a fixed order, drop defaults (page=1, sort=relevance) from the URL, and cap pagination depth on search results.

6. Normalise queries.
?q=Shoes, ?q=shoes, ?q=shoes+ and ?q=%20shoes are four URLs with identical results. Lowercase and trim the query, collapse whitespace, and redirect to the normalised URL. This alone can cut the number of distinct search URLs dramatically.

Response codes and content

7. Return a proper status for empty results, but don't return 404.
A search with no results is a valid response, so 200 is technically correct. Combine it with noindex so empty result pages don't become soft-404 noise. What you shouldn't do is render a full page of "related products" for a nonsense query and make it look like a real, indexable page.

8. Guard against spam queries.
Spammers sometimes link to your search URL with a query containing their brand name, phone number or a URL, hoping your page echoes it in the title or heading and gets indexed. Escape the query (you should for security anyway), keep it out of the <title> on noindexed pages or at least truncate it, never turn it into a link, and consider rejecting queries that contain URLs or are longer than a sensible limit.

9. Keep canonicals simple.
On noindexed search pages, a self-referencing canonical on the normalised URL is fine; don't canonicalise every search to the homepage, which sends confusing signals. Don't combine noindex with a canonical pointing to a different, indexable page and expect both to work as you intended; pick one signal.

Faceted and search-result URLs are a classic way to waste crawl budget after a small parameter change. seotto.com checks for indexing and robots regressions on those patterns when they start getting out of hand.

Measurement

10. Use search logs as research, not as pages.
Your internal search logs are one of the best keyword research sources you have: they're the exact words visitors use, including the things they couldn't find. Review the top queries and the top zero-result queries monthly. The fix for a frequent zero-result query is usually new content or a synonym, not a new indexable search URL.

Two places search URLs sneak in that the template won't catch

  • Sitemaps. Some generators include every route the app knows about, and a cached search page or a "popular searches" route ends up listed as if it were content. Check your sitemap for your search path; a URL you've marked noindex should never be in it.
  • Client-side search. If your search runs in JavaScript and updates the URL with history.pushState, the URLs it creates still get shared, linked and crawled. The server must return the same noindex and normalisation for those URLs when they're requested directly, not only when the page is rendered in the browser.

A quick test for your own site

You can check most of this in a few minutes:

# 1. Is the results template noindexed?
curl -s "https://example.com/search?q=test" | grep -io '<meta name="robots"[^>]*>'

# 2. Are queries normalised?
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "https://example.com/search?q=%20TEST%20"

# 3. Is the query echoed unescaped?
curl -s "https://example.com/search?q=%3Cb%3Ehello%3C/b%3E" | grep -c '<b>hello</b>'
Enter fullscreen mode Exit fullscreen mode

Then search Google for site:example.com inurl:search (or whatever your search path is) to see how many result pages are already indexed. A rough number is enough to know whether you have a problem.

The short version

Noindex the results template by default, stop linking to search URLs, normalise queries, keep user input out of titles and links, and turn the few valuable queries into real pages. Spotting new indexable URL patterns like these before they bloat the index is part of what we built seotto.com to do.

Related reading

Does your site index any internal search pages on purpose? I'm curious whether anyone has made it work well, and what it took.

Top comments (3)

Collapse
 
astap profile image
Юрий •

Great checklist — the "infinite URL" issue from internal search is easy to overlook. I'd add: make sure search pages have a clear noindex strategy by default, and only allow indexing for curated, high-value queries (if any). Also worth logging popular search terms to spot content gaps.

Collapse
 
seotto profile image
seotto.com •

Noindex by default with a short allowlist of curated queries is pretty much where I'd land too. Logging the terms is underrated, since searches that return zero results are basically a list of pages people expected you to have. Would you turn those curated queries into proper landing pages, or keep them as indexed search URLs?

Collapse
 
astap profile image
Юрий •

I'd turn consistently valuable, intent-matched queries into proper landing pages (unique copy, clear structure, controllable metadata and internal links). I'd keep internal search noindex by default with a tight allowlist, and only promote queries that show real demand in logs (especially repeated searches and zero-result terms) into standalone pages.