DEV Community

Toolore
Toolore

Posted on

My domain came with 957,000 ghost URLs. One robots.txt line fixed it.

#seo #webdev #googlesearchconsole #nextjs

Six weeks after launching a static Next.js site, Google had indexed exactly 1 page out of 66.

Search Console said Google knew about 957,000 URLs on the domain.

I had built sixty-six.

The report that actually explains things

The sitemap report said Success, 66 discovered pages. That felt fine, and it is the report most people stop at. It only means Google fetched the XML file — it says nothing about indexing.

The Pages report says what is wrong:

Reason Pages
Crawled – currently not indexed 621,728
Soft 404 335,238
Not found (404) 358
Discovered – currently not indexed 65
Indexed 1

That last row is the site I built. The rest is somebody else's.

Where a million URLs come from

The domain had a previous life as an e-commerce store. Google's index does not forget quickly, and it was still crawling URLs like:

https://toolore.com/?products/7899130/
https://toolore.com/?ctg/search/similarImageSearchResultView/?ctgItemCd=77118274
Enter fullscreen mode Exit fullscreen mode

Look at where the ? is. These are not paths — they are query strings on the homepage. My static site, quite correctly, served the homepage with a 200 for every one of them.

So Google saw ~335,000 pages that looked like the homepage wearing different hats (soft 404s), plus ~622,000 more it had crawled and declined to index. Crawl budget is finite. It was being spent on a store that no longer exists, and my actual pages sat in "Discovered – currently not indexed", which means we know this URL exists, we have not got round to fetching it.

A separate 358 URLs were the old store's real product pages, like /products/hunter-specialties-dod-buck-bark-deer-call/. Those returned a genuine 404, which is exactly right, and I left them alone.

The fix

No page on my site reads a query parameter. Not one. So:

User-Agent: *
Allow: /
Allow: /icon.svg
Disallow: /*?

Sitemap: https://toolore.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

In Next.js that is app/robots.ts:

import type { MetadataRoute } from "next";

export const dynamic = "force-static";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      // The favicon is served with a cache-busting query;
      // the longer rule wins.
      allow: ["/", "/icon.svg"],
      disallow: "/*?",
    },
    sitemap: "https://toolore.com/sitemap.xml",
  };
}
Enter fullscreen mode Exit fullscreen mode

Two details worth knowing:

  1. Check what else carries a query string. My favicon is served as /icon.svg?icon.2dadt9otxkdjj.svg. Blocking it would have cost the favicon in search results. Google resolves conflicting rules by rule length, so /icon.svg (9 characters) beats /*? (3).
  2. Drop the Host: directive if a generator adds one. Search Console flags it — only Yandex reads it.

Three things I nearly did instead, all wrong

Redirecting the ghost URLs to the homepage. This feels helpful and creates a soft 404 for every one of them. Google treats a redirect to an irrelevant page as a soft 404, which is the problem I already had.

Using the Removals tool. It hides URLs from results for six months. It does not remove them from the index, and doing it 957,000 times is not a plan.

Hitting "Validate Fix" and waiting for a green tick. I ran it anyway. It will fail — once the URLs are blocked by robots.txt, Google cannot re-crawl them to confirm anything. That is the intended outcome, not an error. The number to watch is Indexed, not the validation badge.

While I was in there: lastmod

My sitemap stamped lastmod with the build time. Every deploy told search engines that all 94 pages had just changed. Do that often enough and the field stops being believed — for the pages that really did change, too.

So each page now carries the date its content actually changed:

const toolUpdated = (tool: Tool) => tool.updated ?? CONTENT_BASELINE;

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    ...allTools.map((tool) => ({
      url: absoluteUrl(toolPath(tool)),
      lastModified: toolUpdated(tool),
      changeFrequency: "monthly" as const,
      priority: 0.7,
    })),
    // …
  ];
}
Enter fullscreen mode Exit fullscreen mode

That also made IndexNow straightforward: after a deploy, read the live sitemap, take the URLs whose lastmod is today, and submit only those. Bing and Yandex pick them up in minutes instead of days.

Where it stands

Honest answer: too early to claim a win. Search Console data runs several days behind, and the report I am looking at today still predates the fix.

What has moved so far: "Discovered – currently not indexed" has fallen from 65 to 55, which means Google has started fetching pages it previously only knew about. Indexed is still 1. I will post an update when there is a real number rather than a hopeful one.

What I would tell myself before buying the domain

  1. Check the history. The Wayback Machine takes thirty seconds and would have shown me an outdoor-gear store sitting on this domain a year earlier.
  2. Open the Pages report first. Sitemap "Success" means the file parsed. Nothing more.
  3. If your site answers 200 to anything, that is a URL Google can index. Query strings on a static site are the easy way to accidentally publish a million duplicates of your homepage.
  4. A 404 is a fine answer. Do not redirect dead URLs somewhere polite. The site this happened to is Toolore — 78 calculators, converters and file tools that run entirely in the browser. The guides are new too.

If you have inherited a domain and your pages are not getting indexed, open Search Console → Pages → the "Why pages aren't indexed" table, and look at the example URLs. If they are not URLs you wrote, you have the same problem I did — and possibly the same one-line fix.

Top comments (0)