The noindex trap: why my category pages kept falling out of Google

I run KarachiAds, a classifieds marketplace for Karachi, Pakistan. It's a Laravel app, built solo, currently sitting at around 130 categories and 40+ location pages.

A few months in, I hit a problem that I haven't seen written up anywhere — and the fix turned out to be one line of logic and one database table.

The setup

Classifieds sites have a structural problem: you generate a lot of pages, and most of them are empty at the start.

I have a page for /dogs. I have one for /smart-watches. I have one for /bed-sets. Each is a real category a user might browse. But on day one, /dogs had one listing on it. /bed-sets had two.

Google calls these thin pages, and it does not like them. Too many of them and it stops trusting the whole site. So I did the obvious thing:

@php
    $tooThin = $ads->total() < 3;
    $noindex = $hasFilter || $tooThin;
@endphp

<meta name="robots" content="{{ $noindex ? 'noindex,follow' : 'index,follow' }}">

Fewer than 3 listings? Don't index it. More than 3? Let it in. Clean, simple, done.

It was also quietly broken, and it took me weeks to notice.

The bug

Listings expire.

A category page climbs to 5 listings, gets indexed, starts pulling in a trickle of traffic. Two listings expire. The count drops to 3. Another one goes. Now it's at 2.

And on the next page load, that $ads->total() < 3 check runs again, and the page that Google had indexed and started ranking silently serves noindex.

Google comes back, sees the tag, and drops it.

Then a new listing comes in, the count goes back to 3, and the page says index again. Then it drops. Then it comes back.

I had built a page that flickered in and out of the index depending on inventory that day. There was no memory anywhere in the system — every request recalculated from scratch, with no idea what the page had been yesterday.

This is worse than it sounds, because indexing is not a switch you flip. Getting a page indexed the first time takes weeks on a new domain. Losing it takes one crawl. I was throwing away the slowest, most expensive thing I had built and then paying for it again.

What I wanted instead

A one-way door.

Once a page has legitimately earned its way into the index, it should stay there. Not because I'm trying to trick anyone — the page is real, the category is real, and it had genuine content on it. It just happens to be empty right now.

The rule I wanted:

A page can go from noindex to index. It can never go back.

The implementation

One table:

Schema::create('seo_unlocks', function (Blueprint $table) {
    $table->id();
    $table->string('entity_key')->unique();
    $table->timestamp('unlocked_at');
});

entity_key is a string that identifies any indexable thing on the site:

category:47
location:12
landing:3:19

One table covers every page type. No new column on categories, no new column on locations, nothing to migrate on existing models.

Building the key is just a matter of working out which kind of page we're on:

$unlockKey = null;

if (!empty($current) && is_object($current) && !empty($current->id)) {
    $unlockKey = 'category:' . (int) $current->id;
} elseif (!empty($landingPage) && !empty($landingArea)) {
    $unlockKey = 'landing:' . (int) $landingPage->id . ':' . (int) $landingArea->id;
} elseif (!empty($areaObj) && is_object($areaObj) && !empty($areaObj->id)) {
    $unlockKey = 'location:' . (int) $areaObj->id;
}

And then the partial that decides the robots tag:

@php
    $hasFilter = request()->hasAny(['min','max','condition','sort','q','page']);

    $total = (isset($ads) && is_object($ads) && method_exists($ads, 'total'))
        ? (int) $ads->total()
        : null;

    $unlocked = false;

    if ($unlockKey) {
        try {
            $unlocked = DB::table('seo_unlocks')
                ->where('entity_key', $unlockKey)
                ->exists();

            // Not unlocked yet, but qualifies right now? Unlock it permanently.
            if (!$unlocked && !$hasFilter && $total !== null && $total >= 3) {
                DB::table('seo_unlocks')->insertOrIgnore([
                    'entity_key'  => $unlockKey,
                    'unlocked_at' => now(),
                ]);
                $unlocked = true;
            }
        } catch (\Throwable $e) {
            // Fall back to the old behaviour rather than break the page
            $unlocked = false;
        }
    }

    $tooThin = ($total !== null) ? ($total < 3) : false;
    $noindex = $hasFilter || ($tooThin && !$unlocked);

    $canonical = url(request()->path()); // strips query string
@endphp

<link rel="canonical" href="{{ $canonical }}">
<meta name="robots" content="{{ $noindex ? 'noindex,follow' : 'index,follow' }}">

The interesting line is this one:

$noindex = $hasFilter || ($tooThin && !$unlocked);

Thin and never unlocked → noindex. Thin but previously unlocked → index. The unlock is written on the first page load that qualifies, so it happens automatically — no cron, no queue, no manual step.

Three things I got wrong first

1. I assumed I'd already built this. I was convinced past-me had handled it. I went looking for the flag, the column, the config value. There was nothing. It had never existed. Half an hour of grep before I accepted that I had to write it.

2. I forgot the pages that were already ranking. The unlock only fires when a page is rendered. Pages sitting at 2 listings today would never trigger it, even though several of them had been indexed for weeks.

So I wrote a one-off backfill script that inserted an unlock row for every category and location that already had 3 or more listings. 174 rows on the first run — 131 categories and 43 locations. Everything that had legitimately earned its place got grandfathered in.

3. I nearly let filtered pages write unlock rows. This is the subtle one.

?page=2, ?sort=price, ?min=50000 — these should always be noindex, forever, no exceptions. They're infinite combinations of the same content. I had that part right: $hasFilter sits outside the unlock check entirely, as the first condition in the OR.

But the filter guard also has to be inside the unlock write:

if (!$unlocked && !$hasFilter && $total !== null && $total >= 3) {

Without that !$hasFilter, a filtered page showing 3 results would happily write an unlock row keyed to the canonical page — permanently unlocking a page that might have one listing on it. The unlock has to be earned by the canonical URL, on its own merits.

The try/catch is not decoration

This partial runs on every category, location and landing page on the site. If the table is missing, or the DB connection hiccups, or I fat-finger a migration, I don't want a white screen on my highest-traffic pages.

The catch block sets $unlocked = false, which is exactly the old behaviour. Worst case, the feature stops working and nobody notices. That felt like the right trade for an SEO optimisation.

Same instinct behind the method_exists($ads, 'total') check — this partial gets included from several different controllers, and I'd rather compute nothing than fatal on a page where $ads isn't a paginator.

The thing nobody tells you about noindex

While I was in here, I learned something that reframed the whole exercise:

noindex does not save you crawl budget.

Google has to fetch the page to see the tag. The request happens, the HTML renders, the queries run, the bandwidth is spent — and then Google reads the meta tag and decides not to index it. You paid the full cost and got nothing.

If you actually want to stop Google spending its limited budget on a URL pattern, robots.txt is the tool — it stops the fetch from happening at all. noindex is for pages you want crawled but not listed. They're different jobs, and I had been using one to do the other.

Was it worth it?

Test cases after deploying:

Page	Before	After
Category at exactly 3 listings	flickering	index,follow, permanently
Category at 1 listing, never unlocked	noindex,follow	noindex,follow — correct
?page=2	noindex,follow	noindex,follow — correct
Unlock row deleted, page reloaded	—	unlock rewrote itself

That last one is my favourite test. Delete the row, load the page, the row comes back. The system heals itself.

It's too early for me to show you a traffic graph — this went live recently and indexing moves in weeks, not days. What I can say is that the failure mode is gone. Pages that earn their place now keep it, and I'm no longer paying twice for the same index slot.

If you're running anything with generated category or location pages — a marketplace, a directory, a job board — go check whether your thin-page rule has a memory. Mine didn't, and I'd assumed for months that it did.

---

Building KarachiAds in public. Happy to answer questions about the Laravel side in the comments.