"Built on real paid demand data" — until I checked, and found out how wrong that can still be

If you're an indie dev, you've probably heard some version of this advice: "don't guess what to build — look at real demand data." I build a product called Idea Miner that's built entirely on that premise: it mines real, budget-backed paid tasks from global outsourcing platforms and surfaces the demand patterns that keep recurring.

A few days ago someone ran an outside teardown of the site and pointed out something embarrassing: a task literally titled "On-Site Physical Business Verification" was tagged under "Web Development." Not close. Not a stretch. Just wrong.

My first instinct was to explain it away — one weird edge case, not representative. So I wrote a script to actually check. Out of 3,401 published demand features, 1,446 of them (42.5%) carried the exact same wrong label. Not a handful of outliers. Almost half the catalog.

That's the story I actually want to tell here — not because "we fixed a bug" is interesting on its own, but because of what it means for any indie dev who leans on "real data" (mine or anyone else's) to decide what to build.

"The data source is real" and "the data is accurate" are not the same claim

Every one of those 1,446 mislabeled tasks was still a genuinely real, genuinely paid outsourcing task — the underlying source was never fake. What broke was a layer between the raw source and what a user actually sees: an old fallback classifier, from back when the real categorization model was unavailable, that defaulted anything unrecognized to "Web Development." That code got replaced a while ago. Fixing the code didn't retroactively fix the rows it had already written — those just sat there, silently wrong, for who knows how long.

If you're evaluating whether to build something based on a demand report, a trend dashboard, or frankly any tool that claims "real data" — that claim answers exactly one question: is the underlying source fabricated? It says nothing about whether something broke in the pipeline between that source and the number you're looking at. Those are two separate questions, and I'd been treating them as one.

The scarier part: this kind of bug doesn't announce itself

The historical logic itself isn't interesting — every codebase has some. What actually bothered me is that this ran in production, on a page people look at daily, for what was clearly a long stretch, without ever being caught by normal usage.

I think I know why: it's not a crash. The page still rendered. The link still worked. There was still data on the screen. Nothing about the experience signaled "something here is wrong" — it just quietly showed the wrong category next to a real, otherwise-fine listing. A broken pipeline that fails loudly gets fixed fast. One that fails quietly can run for a long time before anyone notices, and "anyone" includes the person who built it.

What I'd actually suggest, if you're picking a direction using someone else's "data-backed" report

Not "don't trust data" — that's not useful advice. More like: ask two questions separately, because they have different answers.

1. Is the source real? (Not fabricated, traceable back to an actual event.)
2. How many processing steps sit between that source and the number I'm looking at, and would a silent failure in any of them be visible to me?

Question 1 is usually the one that gets marketed. Question 2 is the one that actually determines whether you can trust the conclusion enough to spend a few months of solo, spare-time development on it.

The fix, briefly

Three rounds of deterministic, rule-based reclassification (zero LLM cost, batched not per-item) fixed 1,383 of the 1,446. What the rules genuinely couldn't place got moved to an existing honest "Other" category instead of guessing. A separate, unrelated historical bug (embedding-vector dimension mismatch from an old provider migration, affecting 1,283 features) got fixed the same day, for free, using a self-hosted local embedding model.

End state: the mislabeled-category count is down to 63 (all confirmed genuinely correct, left untouched on purpose), the embedding mismatch is at 0, and the site's own audit tooling now flags 92 remaining items (2.7%) — mostly minor text ambiguity, not real contamination.

I'm not going to claim "the data is perfect now." I claimed that once already, apparently incorrectly. What I can say is: the two processing layers behind this incident just got checked by someone actively looking for problems, instead of resting on "the source is real" as if that settled it.

Happy to go into the actual technical details in the comments if anyone wants them — the rule-based classifier, why a stale reference point was quietly breaking cluster-similarity checks too, the exact before/after numbers per fix round. And if you're an indie dev looking for a direction backed by actual paid demand (with the same "verify, don't just trust the label" mindset applied), that's what Idea Miner is for.