Built a small tool recently to flag which complaints are spiking in an app's recent reviews versus its own baseline — no LLM, just keyword frequency compared over time. The first version of the "recent vs baseline" split used the app's own version number to draw the line. It broke in two completely opposite ways, and both were only visible once I tested against real apps instead of a couple of hand-picked examples.
Too much data: the high-volume app
I tested against an app with a huge, constant stream of reviews. Its entire visible review window (App Store's public feed caps out around 500 recent reviews) turned out to be a single version. Zero prior-version reviews to compare against. Every keyword's baseline rate came out to zero, which meant every keyword hit my "brand new complaint" sentinel value — the signal was just noise dressed up as urgency.
Too little data: the fast-shipping app
Then I tested against an app that ships small point releases constantly. Its single latest version had exactly two negative reviews so far. Any word both of them happened to share mechanically hit a 100% rate for that version — again, an inflated, meaningless ratio, this time from the opposite direction: not too little history, too little current data to trust.
The fix: stop trusting the version number as a boundary
Neither failure was really about the math — spike_ratio = current_rate / baseline_rate is fine. The bug was upstream: using "version string" as the boundary between "recent" and "everything else" assumes every app updates its reviews at a similar granularity, and that's false at both ends of the spectrum. A version number is metadata about the app's release process, not about how much time has actually passed.
Switching the split to a time window (reviews from the last N days vs. everything older, anchored on the most recent review's own timestamp rather than wall-clock "now") fixed both cases at once, plus a guard requiring a minimum sample size on both sides before trusting any ratio at all — same principle as requiring enough trailing history before computing a moving average, just easy to forget when the "boundary" variable in front of you is a version string instead of an obviously time-based field.
The generalizable lesson: when you split data into "before" and "after" for a comparison, check what's actually determining the split. If it's some field the source system controls independently of the passage of time (a version number, a batch ID, a status flag), it can silently break your baseline assumption in ways a single test case will never reveal — only real distributions at both extremes will.
Part of a small portfolio of data actors I maintain — signal over data dump, always tested against the real source before shipping. 0xGollum, feeding the data mines.
Top comments (0)