DEV Community: Atabek Murtazaev

One React component cost me 98 unindexed pages

Atabek Murtazaev — Mon, 12 Oct 2026 12:06:23 +0000

Google had indexed 21 pages of my site. Another 98 sat in Search Console under Discovered - currently not indexed, and the number was climbing.

I spent a week on the wrong explanations. Thin content, maybe. A new domain with no authority. Too many similar pages. All plausible, all wrong.

The cause was four characters in a React component.

The Symptom

-	Reason	Pages
-	Discovered — currently not indexed	98
-	Page with redirect	7
-	Alternate page with canonical tag	6
-	Excluded by noindex tag	2
-	Duplicate without user-selected canonical	1
-	Crawled — currently not indexed	1

Everything except that first row is benign. The redirects and canonicals are working as intended. The 98 are the problem, and the label is precise in a way that’s easy to skim past.

Discovered means Google knows the URL exists. It found it in my sitemap.
Currently not indexed means it chose not to spend crawl budget fetching it.

Google has not read those pages, so this isn’t a verdict on the content. It decided they weren’t worth fetching, and the strongest signal I can control there is internal links.

The Cause

The piece of code that

{categories.map(({ label, tools }) => {
  const isOpen = open === label;
  return (
    <div key={label}>
      <button onClick={() => setOpen(isOpen ? null : label)}>
        {label}
      </button>

      {isOpen && (
        <nav>
          {tools.map((tool) => (
            <Link key={tool.slug} href={`/${tool.slug}`}>{tool.name}</Link>
          ))}
        </nav>
      )}
    </div>
  );
})}

{isOpen && <nav>}

If the category is closed, the nav is not hidden. It does not exist. React never creates the element, so the links are not in the DOM, so they are not in the HTML a crawler receives.

One category is open by default. Five links. The other 25 tool pages had zero links from the homepage - the only page on the site with any external authority at all.

I confirmed it the way I should have started, by reading what is actually served rather than what the browser shows:

curl -s https://papyrio.net/ | grep -c 'href="/[a-z-]*-pdf"'

How I solved this problem

<nav hidden={!isOpen}>
  {tools.map((tool) => (
    <Link key={tool.slug} href={`/${tool.slug}`}>{tool.name}</Link>
  ))}
</nav>

hidden is an HTML attribute. The element renders, the links are in the markup, the browser does not paint it. Crawlers read markup.

Add aria-expanded on the button while you are there, so the state is exposed to assistive technology rather than implied by what is missing.

The result

-		Before	After
-	Tool links on the homepage	5	30
-	Tool pages with no internal links	3	0
-	Internal links site-wide	42	722

Why this is easy to miss

I found these 3 hidden areas

Everything worked. Not a bug. No error, no warning, no failing test. The
accordion behaved exactly as designed in every browser I opened.

My tooling agreed. Lighthouse was green. The pages had titles, meta
descriptions, canonical tags, hreflang, structured data. Every SEO checklist
item I could tick was ticked. None of them check whether the page is reachable.

The browser lied to me. Inspecting the homepage in DevTools, I saw thirty
links, because interacting with a page is how you end up looking at it. The
crawler never clicks anything.

My next analyze

curl your pages and count the links. Not DevTools. Not "view source" after you have clicked around. A cold fetch, the way a crawler sees it.
Grep for && before JSX that contains a Link. Conditional rendering and conditional visibility are different things, and the first one deletes content from the document.
Check tabs, modals, carousels and "load more" buttons. Same pattern, same consequence. Anything that mounts on interaction is invisible to a crawler.
Read the Search Console reason strings literally. "Discovered" and "Crawled" are different diagnoses. The first is a linking problem, the second is a content problem. I treated a linking problem as a content problem for a week.

I still annoy on this part

The actual fix was {isOpen && <nav>} to <nav hidden={!isOpen}>

By the way I forgot to mention the product. papyrio

Thanks for reading !!!

My PDF compressor was making files bigger

Atabek Murtazaev — Sun, 04 Oct 2026 11:09:41 +0000

I run a free PDF toolkit. Last month I checked something I had never actually measured: whether the compress tool compressed anything.

It did not. A 2,879,606-byte PDF came back at 2,938,188 bytes. Two percent larger. The meta description on that page promised image-heavy PDFs shrink 30-70%, and it had been promising that to every visitor for months. Examples written in Go

The cause

The handler sent the file to Gotenberg asking for PDF/A-2b:

writer.WriteField("pdfa", "PDF/A-2b")

PDF/A is an archival format. It embeds every font and ICC colour profile and leaves images at full resolution, so the file still renders the same way in forty years. I read "PDF/A" and pattern-matched: A for Archive, archives are small.

The fix

Ghostscript, which actually resamples images:

cmd := exec.Command(
         "gs",
         "-dBATCH", "-dNOPAUSE", "-dQUIET",
         "-sDEVICE=pdfwrite",
         "-dCompatibilityLevel=1.7",
         "-dPDFSETTINGS="+preset,        
         "-dDetectDuplicateImages=true",
         "-dCompressFonts=true", "-dSubsetFonts=true",
         "-dDownsampleColorImages=true",
         "-dColorImageResolution="+dpi,
         "-sOutputFile="+outPath, inPath,)

Same file, measured against production: 2,879,606 bytes in, 417,480 out. Minus 85.5%, and 3.2 seconds instead of 16.2. Text stays selectable because fonts are subset rather than rasterized.

if len(compressed) >= len(fileBytes) {
    compressed = fileBytes  // already optimized; hand back the original
}

Without it you ship the PDF/A bug again in a different costume. Files that are already optimized get bigger when you rebuild them, and a compression tool that sometimes inflates your file is worse than no tool.

Then I tried to sell it

The web app makes no money. Anonymous people convert one PDF and leave, which is the correct behaviour for a free converter and a terrible business. But the same endpoints behind an API serve a different customer - one who pays, because their invoicing system needs to merge PDFs every night.

So: list it on RapidAPI. Before writing any code, measure whether the hardware survives paying traffic.

Capacity first

The whole thing runs on one t3.micro. That is a burstable instance:
you get a baseline CPU allocation and a credit balance you spend when you exceed it. Run out of credits and you either throttle hard or start paying surplus charges.

I ramped concurrent conversions and watched CPUCreditBalance

144 conversions per hour sustained without touching the credit balance
5 concurrent requests comfortable, degrading past that
credit balance stayed full throughout

That is small. It is also more than enough - 144 per hour is 3,456 per day, and the paid tiers I planned top out at 25,000 per month. The go or no-go answered itself, and it cost an afternoon instead of a migration.

The bug that would have broken every paying customer

RapidAPI proxies all traffic through its own IP addresses.

My rate limiter was keyed on client IP. Ten requests per minute, per IP, which is sensible for a public web app. Behind the gateway, every paying subscriber in the world shares the same handful of proxy IPs. Customer A's requests would have exhausted Customer B's limit. The first two subscribers would have throttled each other into 429s and churned, and I would have had no idea why.

The fix is to key on the subscriber instead, and to verify the request actually came through the gateway before trusting the header that names them:

func (m *RapidAPIAuth) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    secret := r.Header.Get("X-RapidAPI-Proxy-Secret")
    if secret == "" {
        m.next.ServeHTTP(w, r)  // not a gateway request, carry on
        return
    }
    if subtle.ConstantTimeCompare([]byte(secret), []byte(m.expected)) != 1 {
        writeError(w, http.StatusForbidden, "forbidden", "...")
        return
    }
    user := r.Header.Get("X-RapidAPI-User")
    // ... attribute the request to `user`, and rate limit on that
}

A limit you are not allowed to set

I wanted paid plans at 30 requests per hour, matching what the box can actually do.

RapidAPI does not let you. Paid tiers are locked at 1,000 requests per hour and the provider has no field to lower it. So the platform advertises a number my hardware cannot serve.

The resolution is not technical, it is honesty. The server enforces 120 per hour

Better a customer reads that before subscribing than discovers it at request 121.

Endpoints do not exist until you declare them

Last one, and it cost me real time.

The origin server has twelve endpoints. I defined four in RapidAPI's dashboard and assumed the gateway was a dumb proxy that would forward anything.

$ curl -X POST "https://...p.rapidapi.com/ocr-pdf" -H "X-RapidAPI-Key: $KEY" ...
{"message":"Endpoint '/ocr-pdf' does not exist"}

What made this hard to catch: an undeclared path returns 401 without a key, exactly like a declared one. Authentication runs before path resolution. So the obvious probe tells you nothing, and you need a valid key to discover that eight of your twelve endpoints are unreachable.

I had already written documentation for all twelve. Had I shipped it, every subscriber would have hit a 404 on two thirds of the product.

You can find it on RapidAPI where 12 enpoinds live.

Free tier is 50 requests a month, no card. It is on RapidAPI as Papyrio PDF Tools, and the reference is at here.