My custom filter showed "zero false positives"! Then I realized my validation logic was flawed, letting all the garbage through.

Hey there, it's your "Oji" (38-year-old AI/quant dev on the side). During the week, I'm a regular company employee; on weekends, I tinker with AI agents and automated trading bots.

Recently, I was building a filter to automatically extract "AI-related companies" from a market data list, based on their business descriptions. The mechanism is simple: it scores companies based on hits of predefined positive keywords (e.g., "machine learning," "natural language processing") and negative keywords (e.g., "gaming," "entertainment").

To improve the filter's accuracy, I set up a test to tune the threshold (how many positive keyword hits qualify a company). I prepared lists of positive examples (actual AI companies) and negative examples (non-AI companies) to see how well the filter classified them—standard stuff.

Then I ran the test, and the results were mind-blowing.

"Zero false positives, no matter the threshold I tried."

For a moment, I thought, "Is my filter a genius?" Zero false positives is usually impossible. But seconds later, I cooled down. Results that are too perfect usually mean something is wrong.

Sure enough, when I manually reviewed the list of companies that passed the filter, I found obvious non-AI companies like gaming studios mixed in. My filter was letting garbage through, yet the test reported "no issues." The despair I felt realizing this late on a weekday night was quite something.

Why "Zero False Positives"?

The root cause wasn't a bug in the code itself, but a design flaw in the validation logic.

Basic accuracy validation involves looking at two metrics:

• False Positive (FP): A non-AI company incorrectly classified as an AI company.
• False Negative (FN): An AI company incorrectly classified as a non-AI company.

The issue I encountered was with FP. To count FPs, the definition of negative examples (companies that are "not AI companies") is crucial.

And here was my definition of a negative example:

"A company with 0 positive keyword hits AND 2 or more negative keyword hits."

This was the source of all evil. This definition, seemingly sound at first glance, contained a self-contradiction.

My filter's rule for classifying a company as "passing (AI-related)" was: "2 or more positive keyword hits."

You probably see it now.

• Filter's passing condition: positive_hits >= 2
• Test's negative example condition: positive_hits == 0

These two conditions can never be true simultaneously. A company classified as "passing" by the filter already has positive_hits >= 2, so it can never fit the negative example definition of positive_hits == 0.

In essence, when trying to count "negative examples that were incorrectly classified as passing (i.e., false positives)," the structure itself made it impossible for any company to be both a "negative example" and "passing." Of course, false positives would always be zero.

Translating the concept to code:

// Filter rule: Pass if positive keyword count >= 2
function is_theme_company(positive_hits) {
  return positive_hits >= 2;
}

// Validation's negative example definition (buggy)
function is_negative_example(positive_hits, negative_hits) {
  // Only consider companies with zero positive hits as negative examples
  return positive_hits === 0 && negative_hits >= 2;
}

// Validation process
let false_positives = 0;
for (const company of all_companies) {
  const is_selected = is_theme_company(company.pos_hits);

  // If it's a negative example but selected, count as FP
  if (is_selected && is_negative_example(company.pos_hits, company.neg_hits)) {
    false_positives++;
  }
}
// With this logic, is_selected=true (pos_hits>=2) and is_negative_example=true (pos_hits==0)
// can never both be true, so false_positives will always be zero.

The test was too tightly coupled to the logic it was supposed to validate, preemptively deciding the outcome. It was a brutal rookie mistake.

How I Fixed It

Once I understood the cause, the fix was simple.

I changed the definition of negative examples to be independent of keyword hit counts. Specifically, I manually selected dozens of companies that were unequivocally not AI companies (e.g., food manufacturers, apparel brands, construction companies) and created a fixed "negative example list."

Running the test again with this list, false positives, as expected, came pouring out. Finally, the numbers showed that the filter was indeed picking up many irrelevant companies. This was the real starting point for tuning.

My Takeaway

The lessons from this failure are significant.

"The test passed" only means something if the test itself is correct.

Finding code bugs with tests is fundamental, but you must always question the possibility that the test logic itself is flawed. Especially when validating rules you've created yourself, there's a risk that the validation logic gets dragged by the tested logic, unconsciously leading to "conclusion-first" tests.

When you get "too perfect results," first question your own assumptions. This is a crucial reminder I'll engrave into my mind.

This story also applies to backtesting automated trading bots. When a backtest shows unusually good performance, it's rarely because you've discovered a brilliant strategy; it's usually because you're looking at future data (leakage) or not accounting for fees and slippage.

I'm logging embarrassing failures like this as part of building in public. Hopefully, it helps someone else in their solo dev journey.

---

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

If a provider-agnostic RAG Q&A API is useful to you, mine is MIT-licensed on GitHub: rag-faq-api. It runs and passes its full test suite **with no API key* (offline stub LLM + hashing embedder), swaps to Claude / Gemini / OpenAI via one env var, and ships a retrieval-quality harness (Hit@k / MRR / Recall@k) with a chunking sweep.*