My Bot's KPI: Trades 2 > Opportunities 1?! When Your Numerator and Denominator Don't Observe the Same Time.

Hey everyone, it's your friendly neighborhood "old man" developer here. I'm a 38-year-old part-time engineer, spending my evenings and weekends tinkering with AI bots.

Today, I want to share a particular moment from my bot development log that had me face-palming – one of those "how could I make such a dumb mistake?" moments. In short, a KPI I built to measure my bot's performance started spitting out logically impossible values. Specifically, my "fulfillment rate" exceeded 100%.

Imagine a scenario like, "there was only one trading opportunity, but somehow two trades were executed." It made no sense. And no, my bot hadn't gone rogue and started placing infinite orders. The cause was a simple, elementary error in my KPI calculation logic.

What Happened: My Tests Turned Red with an AssertionError

I've set up a decent suite of unit and integration tests for my bots. Last weekend, after adding some new logic, I ran the tests as usual, and they went red with an unfamiliar error:

AssertionError: fulfillment_rate > 1.0

fulfillment_rate is a custom KPI I use to see how many of the trading opportunities my bot identifies actually result in a successful trade. The formula is simple: actual_trades / total_opportunities.

For this to exceed 1.0 (or 100%) is impossible. Digging into the logs, I found bizarre entries like trades: 2 against total_opportunities: 1. For a moment, I thought there was a bug in my aggregation logic, double-counting trades. But a direct check of the DB confirmed two actual trades. And at that moment, only one opportunity was visible. What was going on...?

The Cause: The End of the Month Hadn't Arrived Yet

Stepping through the code with a debugger, the cause became immediately clear. The "observation time" for calculating the numerator and denominator was out of sync.

The problematic code, conceptually, looked something like this:

# Problematic logic (conceptual)

# Denominator: Fetch all trading opportunities as of the end of the month
opportunities = get_opportunities_as_of(end_of_month_date)

# Numerator: Fetch all trades that occurred within a specific period
trades = get_trades_in_period(start_date, end_date)

# Calculate fulfillment rate
fulfillment_rate = len(trades) / len(opportunities)

At first glance, this code might seem fine. But running it before the end of the month arrives leads to a nasty problem.

Let's say I ran the test on August 26th.

1. Numerator Calculation: get_trades_in_period would retrieve all trades from August 1st to August 26th. If a new trade was entered and executed on August 26th, the numerator len(trades) would be incremented.
2. Denominator Calculation: get_opportunities_as_of would try to fetch trading opportunities that should be visible as of end_of_month_date, i.e., August 31st. But it's only August 26th. Since future data isn't available yet, the trading opportunities that occurred on August 26th hadn't been included in the denominator calculation.

The result was a discrepancy: the numerator increased in real-time, while the denominator wouldn't update until the end of the month. If trades were concentrated near month-end, the fulfillment rate would easily exceed 100%. It's obvious in hindsight, but I didn't catch it until the test failed.

The Fix: Unifying the Observation Point to the "Entry Date"

Once I pinpointed the cause as a "mismatch in observation time," the fix was straightforward: align the numerator and denominator to be calculated from data observed at the same point in time.

Specifically, I changed the logic to re-calculate trading opportunities based on the actual "entry date" when the bot made a trade.

# Corrected logic (conceptual)

# Base calculation on the day the trade occurred
trade_entry_date = get_trade_entry_date(...)

# Denominator: Get opportunities as of "that specific day" the trade occurred
opportunities = get_opportunities_as_of(trade_entry_date)

# Numerator: Similarly, get trades that occurred "on that specific day"
trades = get_trades_on(trade_entry_date)

# Now, the observation points are aligned
fulfillment_rate = len(trades) / len(opportunities)

Now, without waiting for month-end, I can correctly calculate, for each trade, "what percentage of opportunities existing at that specific moment were executed." By leveraging the bot's own business day determination logic, I was able to align the observation points.

The Lesson: "Numerator and Denominator on the Same Playing Field" - A Fundamental Principle

This mistake of "misaligned aggregation criteria for numerator and denominator" isn't actually my first time. Since I started personal development, this is probably the fourth time.

• User registration counts and active user counts had different aggregation periods, causing active rates to exceed 100%.
• API total request counts and error counts had different time zones, leading to skewed error rates.
• In backtesting profit/loss calculations, the timing of fee accrual was off, overstating profits.

All of them were the same type of mistake.

"When calculating KPIs, always obtain the numerator and denominator from the same observation point, the same definition, and the same playing field."

This is a fundamental principle, yet it's easy to overlook when you're deep in the code. It's especially easy to fall into this trap when pulling numbers from different tables or data sources.

This latest failure was detected by an AssertionError in my tests, which was great. If I hadn't had tests and remained oblivious, I might have mistakenly thought, "Wow, this bot is performing incredibly well!" and made bad decisions based on that false impression.

This past weekend was another stark reminder that humble testing and strict adherence to basic principles are paramount. I hope sharing this failure log can be helpful to someone out there.

---

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

If a provider-agnostic RAG Q&A API is useful to you, mine is MIT-licensed on GitHub: rag-faq-api. It runs and passes its full test suite **with no API key* (offline stub LLM + hashing embedder), swaps to Claude / Gemini / OpenAI via one env var, and ships a retrieval-quality harness (Hit@k / MRR / Recall@k) with a chunking sweep.*