the silent wrong-ranking bug

A customer told me my scraper's #1 product was wrong. They were right — and the bug was nastier than a crash.

I run a small catalogue of scrapers for Korean e-commerce and content platforms. Last week a customer using my Olive Young (a huge Korean beauty retailer) best-seller scraper sent me a short, polite message:

"The ranking looks off. Your #1 is a probiotics supplement, but the real #1 on the site is a MEDIHEAL mask sheet."

She was completely right. And the reason why is the kind of bug I now think about a lot.

It wasn't down. It wasn't empty. It was plausibly wrong.

My nightly health-checks were all green. Every run succeeded, returned 100 items, and every field was populated — real product names, real prices, real images. If you glanced at the output it looked perfect.

The problem: the order was wrong, and the products were the wrong set. My scraper's "#1 best-seller" was a diet-probiotics capsule. The site's actual #1 was a face mask.

Here's what happened. Olive Young serves ranking data from two different places:

• product-ranking-service.oliveyoung.com/v1/pages/ranking/sales/products — an internal "sales ranking" microservice.
• global.oliveyoung.com/display/product/best-seller/order-best — the endpoint the site's own "Best" page actually renders.

My scraper called the first one. The site shows the second one. Both return real, valid-looking products in a real order — they're just different boards. So the scraper produced data that passed every automated check and was still wrong to any human who'd looked at the actual page.

The scary part: it's a whole class of bug

That one report made me stop and audit my whole catalogue against the live sites — not "does it return data" but "does the top-N match what a real user sees right now."

I found the same shape more than once. The common trigger was innocent-looking:

// looks harmless
searchParams: { 'category-id': '' }   // "" = give me the overall ranking, right?

Except the site had quietly changed so that an empty/absent parameter no longer returns the ranking — it returns an unranked default set (sorted by internal product id, every rating 0, half of them sold out). No error. Just a plausible-looking list in the wrong order. A couple of my scrapers were shipping that as "the best-seller ranking."

Other variants I found the same day:

• a 6-day-stale cached list served under a no-filter request while the real ranking lived under one specific filter id;
• a marketplace where a boolean used flag was inverted (the API's 2 meant new, not 1), so every "new/used" label was backwards;
• a webtoon ranking labelled "by rating" that was actually the site's default popularity order.

None of these crashed. None returned zero rows. All of them would sail through a "is it up?" monitor forever.

The lesson I wish I'd internalised earlier

"It runs and returns data" is not "it's correct." For anything ranked, the only real test is:

Fetch what a real user sees on the actual page, and compare your top-N to it — order and identity, not just field presence.

Concretely, what I changed:

1. Verify against the human-facing view. For each ranking scraper I now diff the top ~5 against the live page the user would look at, not just against "an endpoint that returns products."
2. Prefer the endpoint the site itself renders. If you find a cleaner-looking internal microservice, double-check it's the same board the page shows. A different host is a red flag.
3. Add sanity guards, not just presence checks. Fail (loudly) the run if the list is near-monotonic by id, or every rating is 0, or the sold-out share is absurd — those are the fingerprints of a "silent default" response.

The fix for Olive Young itself was small once I understood it — point at /display/product/best-seller/order-best with the params the Best page sends, map the real field names, and now the scraper's #1 is MEDIHEAL, matching the site. (Bonus: the correct endpoint even carries review counts the old one didn't.)

The thing I keep coming back to

The most valuable message I got all month was one sentence from a user pointing out that my #1 was wrong. It didn't just fix one scraper — it exposed a blind spot in how I was validating all of them.

If you scrape ranked data: go look at one of your outputs next to the real page right now. Not the schema. The order.

I maintain a set of Korean-platform scrapers (kdatafactory). If you've hit this "plausibly wrong" class of bug, I'd love to hear how you catch it.