In June, PeerPlay's 30-day active users peaked at 709. By late August the number was 126, a drop of about 82%. I build and run PeerPlay on my own, ...
For further actions, you may consider blocking this person and/or reporting abuse
churn is the comfortable story, a bug is the one that fixes things. the emotional pull point is the real insight here, blaming the users costs you nothing. one more check for the list: verify the metric definition itself hasn't changed. a renamed or dropped event quietly stops sending, and the graph looks exactly like churn.
Exactly. "Churn" was the comfortable answer, so I almost stopped looking. In my case the events were fine and the app list itself was wrong: Discovery was hiding every app older than 14 days. I only caught it by accident in a Firebase analytics export. Checking whether the metric definition changed is now the first thing I'd do before calling anything churn.
glad the checklist found a home. the accidental export catch is an origin story every postmortem secretly wants.
The "exists in the database but not on screen" line is the one I'd turn into an actual check. A small weekly query that counts rows per status (paused, expired, hidden) next to last week's numbers would probably have shown paused campaigns piling up before the active users graph moved at all. Was the 14 day filter written before auto restart existed, or did both ship together?
Agreed, that's the check I'd build first. A weekly count per status next to last week's would have shown paused campaigns piling up long before active users dropped. The 14-day filter and auto restart weren't built together, and nobody checked how they behaved side by side, which is exactly how the gap slipped through. Lesson learned: every time a campaign can change state, someone should ask whether it's still visible after that.
Thanks for writing up the wrong turn as well as the fix. Assuming churn first is so natural, and the numbers fit it well enough that nobody questions it.
What strikes me is that both bugs were invisible from the dashboard. The metric moved, but nothing told you which part of the product had stopped doing its job. One thing that helps on products like this is a small set of health numbers that track supply as well as users. Here that might be the count of active campaigns visible in Discovery each day. If it drops while campaigns are still running, it points straight at the product rather than at the people.
It is a cheap addition, and it turns weeks of quiet drift into an alert. Are you planning to add something like that, or did fixing the 14 day filter feel like enough for now?
Exactly that. The dashboard told me how many people showed up, but nothing about what they found when they got there. Fixing the 14-day filter solved this bug, not the blind spot, so a daily count of campaigns visible in Discovery next to the count of campaigns that should be live is next on my list. If those two numbers drift apart, it's the product, not the users. Thanks, that's a really useful way to put it.
Discovery quietly hiding every app older than 14 days is such a sneaky failure, because nothing errors and the graph looks exactly like users losing interest. What I take from this is to always ask "could the product have stopped showing people the thing" before "did people stop wanting the thing". Glad you caught it from the export instead of shipping a churn fix for a problem that didn't exist.
Thanks, that's a better way to put it than I managed in the post. "Could the product have stopped showing people the thing" is going on my checklist before I ever call something churn again. The scary part was how reasonable the wrong answer looked. Any churn fix would have done nothing, because the users never left, the apps just disappeared from their screen.
the second bug is the one that stands out: a paused campaign that nothing on screen shows. a cheap guard is a check that compares what the database says should be live against what Discovery actually lists, so a mismatch fires before the graph drops. it also gives a dip in active users a first place to look that isn't churn.
Yes, the second bug was the sneakier one, because nothing looked broken. A check that compares what the database says is live with what Discovery actually lists would have caught both bugs before the graph moved. Thanks, that's a much better first place to look than "users left".
this is the scariest kind of drop because every dashboard just says "normal churn". i've started keeping one boring metric next to active users, like how many items someone actually sees per session, so a supply problem can't pass itself off as a demand problem
That framing is great. It was exactly that: people still opened the app, there just wasn't enough to see. Items seen per session would have shown the drop on day one. That's going on my list next to active users.