Programmatic SEO pages and thin content: noindex until there are three items

Generating pages from your own data is the cheap way to give a site hundreds of URLs. It is also the cheap way to publish hundreds of pages that say almost nothing. We chose to treat listings with fewer than three live products as too small for index eligibility. This post shows the rule we use for our category and tag pages: when the lookups succeed, a page with fewer than three live products emits noindex metadata while it stays available to visitors. The examples come from RadarMakers, where categories like Developer Tools are built from the catalog.

Three is our number. It is an application rule, not a search engine requirement and not a measured quality score. The rest of this post is about how to make such a rule behave well, including what it does when the data is missing.

Two kinds of generated pages

Category pages live at /categories/<slug>. Selected categories have editorial copy in src/lib/category-pages.ts, followed by their live product listings.

Tag pages live at /tag/<slug>. Their text is assembled from facts about the products carrying the tag, using helpers such as listFacts, tagIntro and tagFaq in src/lib/tag-pages.ts. The helpers describe what the catalog returns at that moment and avoid quality claims.

Both kinds apply the same threshold.

One gate, one definition of "live"

The category threshold is a constant and a one-line function in the same file:

// frontend/src/lib/category-pages.ts, excerpt
export const MIN_INDEXED_PRODUCTS = 3;

// frontend/src/lib/category-pages.ts, excerpt
export function categoryIndexable(liveCount: number): boolean {
  return liveCount >= MIN_INDEXED_PRODUCTS;
}

Tag pages use a separate constant with the same value, MIN_PRODUCTS = 3, from src/lib/best-ranking.ts. The same file defines what counts as a product for this purpose:

// frontend/src/lib/best-ranking.ts, excerpt
export function liveProducts(products: Product[]): Product[] {
  return [...new Map(products.filter((p) => p.status === "live").map((p) => [p.id, p])).values()];
}

For category and tag metadata, liveProducts counts only products with the status live and removes duplicate IDs, so a product that appears twice in the fetched data counts once. A product that is not live never raises the number.

Here is what that rule looked like against the live catalog on October 31, 2026. I counted distinct live product IDs for each of the 16 categories through the public products endpoint, then read the robots meta tag from each served category page.

Source: radarmakers.com public products endpoint and served category HTML, checked October 31, 2026 at 04:55 UTC. Chart: RadarMakers.

Seven categories had three or more live products and served no robots override. Nine had fewer than three and served noindex, follow. Three of those nine had no live products at all. The two categories sitting exactly on the line, Apps and Other, both had three live products and both were index eligible. That is the boundary doing what the constant says.

What the page emits

The category layout builds its metadata and adds a robots field only when the page is below the threshold:

// frontend/src/app/categories/[slug]/layout.tsx, excerpt
    ...(empty ? { robots: { index: false, follow: true } } : {}),

At or above the threshold the layout emits no robots field, so the default applies. The tag page is explicit in both directions:

// frontend/src/app/tag/[slug]/page.tsx, excerpt
    robots: { index: products.length >= MIN_PRODUCTS, follow: true },

follow: true is deliberate. A thin page is still a normal page for people, and crawlers that support the directive are still allowed to follow its links to the product pages. The metadata asks search engines that support it not to index the page. It does not stop them from crawling it, and a crawler can only read the instruction if it is allowed to fetch the page, which is why these pages are not blocked in robots.txt.

When I checked the live site on 2026-10-28, the served HTML matched this. The Developer Tools category page had no robots meta tag, the Other, Web Games and Health category pages had <meta name="robots" content="noindex, follow"/>, and a tag page such as /tag/ai had index, follow. Those are the pages as served on that day. Counts change as products are added, and this check says nothing about what any search engine did with them.

The sitemap follows the same rule

Our sitemap policy excludes listings below the same threshold used by their metadata. A URL that we mark noindex and also offer in the sitemap would send two opposite signals, and the comment in sitemap.ts says the map must not offer such pages. The sitemap performs its own requests, so its inclusion decision is not derived from the metadata response.

For categories, the sitemap code keeps only entries that pass categoryIndexable. It also needs two more things before a URL goes in: the ranked list for the category must load, and the list must give a valid date to use as lastmod. That date comes from a small helper:

// frontend/src/lib/category-pages.ts, excerpt
    const t = Date.parse(p.content_updated_at || p.launched_at || p.created_at || "");

It takes the latest valid date among the products, preferring a content update date, then the launch date, then the creation date. A category whose list cannot be loaded drops out of the sitemap rather than appearing with a guessed date.

Tags need an eligible tag, at least three live matching products in the data the sitemap fetched itself, and a valid product date. Passing the count gate is necessary but not sufficient. On 2026-10-28 the live sitemap listed these five category URLs, and none of the three noindex category pages I fetched, Other, Web Games and Health, was among them:

$ curl -s https://radarmakers.com/sitemap.xml | grep -o "<loc>https://radarmakers.com/categories/[^<]*</loc>"
<loc>https://radarmakers.com/categories/ai-tools</loc>
<loc>https://radarmakers.com/categories/developer-tools</loc>
<loc>https://radarmakers.com/categories/productivity</loc>
<loc>https://radarmakers.com/categories/design</loc>
<loc>https://radarmakers.com/categories/fintech</loc>

Three days later, the same check

Catalogs move, so a dated check goes stale quickly. On October 31, 2026 I repeated the check across all 16 category pages and the sitemap. The sitemap now listed seven category URLs instead of five. The two new ones were Apps and Other. Other had been one of the noindex pages on October 28. By October 31 it had reached three live products, it served no robots override, and the sitemap listed it. Nothing in the code changed for that to happen. The catalog crossed the threshold and both signals followed.

Source: radarmakers.com/sitemap.xml and served category HTML, checked October 31, 2026 at 04:55 UTC. Chart: RadarMakers.

The useful result is the empty cells. No category page served noindex while also appearing in the sitemap, and no index eligible category was missing from it. That agreement is what the shared threshold is supposed to produce. It is a snapshot of what the server sent at one time, and it still says nothing about what any search engine did with those pages.

When the product lookup fails

This is the part worth spelling out, because a rule that depends on a count has to say what happens when the count is unavailable. The category layout makes two lookups at the same time, the category itself and the live products of that category, each with a short timeout:

// frontend/src/app/categories/[slug]/layout.tsx, excerpt
  const [category, empty] = await Promise.all([
    withinDeadline(getCategories())
      .then((c) => (c ?? []).find((x) => x.slug === slug) ?? null)
      .catch(() => null),
    withinDeadline(getProducts({ status: "live", category: slug, limit: MIN_INDEXED_PRODUCTS }))
      .then((p) => !categoryIndexable(liveProducts(p ?? []).length))
      .catch(() => false),
  ]);

Three outcomes follow from this code:

• If the category is not found, or its lookup fails, the layout emits index: false, follow: true. We deliberately emit noindex when category data is unavailable.
• If the category is found and the product lookup succeeds, the count decides. A lookup that resolves with null is treated as an empty list, so it counts as zero products and gets noindex.
• If the category is found and the product lookup throws or times out, the catch handler returns false for empty. The layout then emits no robots field, and the page stays index eligible.

The third case is a policy, and the comment in the file states it: when the API fails, we do not guess. For a response whose category lookup succeeds but whose product lookup rejects or times out, the layout omits the robots override. This post does not establish how long any indexing effect of that would last. We chose to leave the page open in that case, instead of marking it noindex on the strength of a count we never got. Another policy would emit noindex whenever the count cannot be fetched, and a site with many thin pages and an unreliable API might prefer it. The point is to decide on purpose and write it down, not to inherit whatever an empty catch happens to do.

This failure path covers category metadata only. It does not describe successful rendering of the page, and I am not describing tag page failure behavior here.

Renaming without losing the old URLs

Category slugs were renamed in migration 050: ai became ai-tools, devtools became developer-tools, and mobile became apps. Products point to categories by ID, so the rename did not touch them. The old addresses are kept by permanent redirects built from one table in the code, RENAMED_CATEGORY_SLUGS, for both /categories/<old> and /best/<old>. On the live site, /categories/devtools answers with a 301 to /categories/developer-tools.

What this does not tell you

• "Index eligible" means the metadata allows indexing. It does not mean any page is indexed.
• Three products is not a threshold that search engines publish. It is where we drew the line for these pages.
• No traffic, ranking or indexing results are reported here. The checks above show what the server sends, not what happened afterwards.
• The count gate is only part of the sitemap rule. The data checks decide inclusion too.

Test boundary behavior, not only the happy case

The threshold is meaningful only when the record definition is consistent. A list with two distinct live IDs should fail. A list with three should pass. Repeating one live ID must not make a two product category pass. Adding a pending or archived listing must not raise the live count. Put these cases in tests for both categories and tags because separate constants can drift even when they currently share a value.

Do not count a client side display if server metadata uses another query. The visitor could see three cards while metadata sees two records after duplicate removal. The reverse could happen if metadata counts something not rendered. Keep the filtering rule visible and compare the page, metadata, and sitemap against the same source where practical.

A successful empty lookup is different from a failed request. When a request times out, the code cannot honestly conclude that the category has zero products. Return distinct states for success with products, success with no products, and failure. The renderer can then show a useful empty state for the first case and a temporary error or retry path for the second. Metadata and sitemap policy should also be explicit for failure.

Avoid catching every exception and replacing it with an empty array. That hides operational failure and may change robots metadata by accident. If you cache the last successful result, document its age limit and invalidation behavior. A retry can help with a transient problem, but excessive retries can increase load during an outage. Do not claim a recovery guarantee unless the service provides one.

Keep metadata and sitemap checks together

Robots metadata and sitemap generation are separate code paths. For the same category fixture, compare the live count, robots directive, canonical, response status, and sitemap decision. A deliberately noindex page should not also be included in the sitemap under this policy. An index eligible page can still be excluded if the data request fails or there is no valid last modified date.

A sitemap is a discovery hint, not a promise of indexing. Google says it helps search engines learn which URLs a site considers important, but it does not guarantee that a URL appears in results. A noindex route may still be discoverable through links. Align signals, but do not describe a sitemap inspection as an indexing result.

Test below, at, and above the boundary, plus duplicate IDs, no valid date, and request failure. Run the cases for categories and tags. Add a test that inspects generated metadata and a separate one for sitemap output. Parse generated output rather than relying only on a screenshot.

Keep redirects in the same review. When a category slug changes, assert the old path resolves to the intended new path. Check redirect status, final URL, canonical, and internal links. A redirect map preserves old paths only when its destination is correct. Test the destination instead of merely checking that a redirect happened.

Separate page quality from index eligibility

A mechanical count cannot decide whether the page answers a real query. Three unrelated products can satisfy a code condition and still make a weak comparison. A carefully edited page about a smaller category may answer a narrow need. Treat the threshold as a guardrail and review the content separately.

Read each generated page as a visitor. Does the introduction describe the category without claiming unsupported features? Can someone compare products using facts that are actually present? Is there a useful next step? A heading followed by a handful of cards may be too little. Consider adding a genuine editorial explanation or keeping the route out of search until it helps the intended audience.

Do not create interchangeable paragraphs by inserting a category name into a template. That makes URLs look different while saying the same thing. Use catalog facts only when they are accurate and useful. A missing price field does not prove that a product is free or paid. Avoid turning a blank feature field into a negative statement.

As listings change, review routes whose state crosses the boundary. Record which categories changed and check metadata and sitemap output. A page becoming index eligible is a code state, not proof of traffic or ranking. Label the date of a live inspection and keep operational changes separate from visitor analytics.

• [ ] Test two distinct live records, the threshold, and duplicate IDs.
• [ ] Handle successful empty results separately from failed requests.
• [ ] Inspect metadata and sitemap output with the same fixtures.
• [ ] Review whether the page answers a distinct visitor need.
• [ ] Test old slugs through to the final destination.

Preserve meaning when category routes are renamed

A renamed slug affects more than a route lookup. Search metadata, sitemap entries, breadcrumbs, internal links, and cached social cards can each retain the earlier path. Keep one redirect map and test the old and new address together. A permanent redirect should lead to the final canonical path rather than another redirect chain. Update internal navigation to point directly to the current address even when the redirect remains for old links.

Check that the destination still passes the index eligibility rule. A redirect to a category that is itself noindex may preserve access while not producing the intended search landing page. That is a policy choice worth recording. Do not keep a redirect merely because a prior slug existed if the destination no longer represents the same category.

When a category is merged, preserve the old route only if the new route is a genuine replacement. Otherwise, an honest not found response can be better than redirecting every retired category to a broad page. The destination should match the previous visitor intent. Review the heading and introduction after the redirect, not only the status code.

A deployment checklist can exercise the same URLs across HTML, sitemap, and redirects. Fetch the old route, follow redirects, inspect canonical metadata on the destination, then search the sitemap for the final path. This gives the team a repeatable record of how the policy behaved at release time. None of these checks alone proves that an external search engine has updated its index.

Keep decisions small and reversible

When evidence is incomplete, avoid turning a failed data request into a permanent content judgment. A temporary API failure should not make the application claim a category has no products. Store the failed state, preserve the last good result if the product policy allows it, and make the result age visible to the team. The public page should not quietly present stale data as current.

Review the threshold itself on a schedule, but change it only when a visitor need and the implementation evidence justify a change. A threshold set too low can expose pages with little distinct value. A threshold set too high can keep useful specialist pages out of search eligibility. The right value is a local tradeoff, not a universal SEO number.

Write down who can change the rule and which tests must pass. Include the sitemap, metadata, failure behavior, and page copy. That makes a policy adjustment reviewable and prevents a constant change from leaving contradictory comments or an outdated article behind.

Takeaways

• Keep the category and tag thresholds aligned. In this code they are two constants with the same value.
• Document what your metadata does when a lookup fails.
• Keep old URLs alive with permanent redirects when you rename slugs.
• Inspect the served HTML and the sitemap with curl after you deploy.

Sources

• RadarMakers implementation snapshot checked October 28, 2026: frontend/src/lib/category-pages.ts, frontend/src/lib/best-ranking.ts, frontend/src/app/categories/[slug]/layout.tsx, frontend/src/app/tag/[slug]/page.tsx, and frontend/src/app/sitemap.ts.
• Google Search Central, Block Search Indexing with noindex: https://developers.google.com/search/docs/crawling-indexing/block-indexing
• RadarMakers live category and sitemap checks described in this article, recorded October 28, 2026 and repeated October 31, 2026.
• RadarMakers public products endpoint, live product counts per category, checked October 31, 2026.