In June, PeerPlay's 30-day active users peaked at 709. By late August the number was 126, a drop of about 82%. I build and run PeerPlay on my own, so there was nobody else to look at that graph with me.
I did what most developers would do. I assumed people had tried the app, decided it wasn't for them, and left. I started writing a churn analysis. I was wrong, and the way I was wrong is worth sharing, because it applies to almost any small app.
The story I told myself
Churn is a comfortable explanation. It's normal, every app has it, and a test-swapping app has an obvious reason for it: developers join, get their testers, pass their 14 days and move on. The numbers fit that story well enough that I didn't question it.
So I was looking at the users. Who left, when, after doing what. That's a reasonable question, but it assumes the product is working the way you think it is.
What was actually happening
The real cause was a bug in the Discovery tab, the screen where developers find apps to test. It hid every app older than 14 days. That sounds harmless for a product built around 14-day rounds, except PeerPlay campaigns restart automatically. Apps that were still active, still looking for testers, simply disappeared from Discovery once they passed day 14.
So the people who opened PeerPlay found less and less to do, and the apps that needed testers got fewer of them. From the outside, that looks exactly like users losing interest.
The second bug that kept it quiet
A second bug at the same time made it worse. A developer's own My Apps list never showed that a campaign was paused. So nothing on screen hinted that anything was wrong, not for the testers browsing Discovery and not for the developers waiting for testers.
That's the part that stays with me. Two bugs, each small, and together they made a broken flow look like a calm, normal app with fewer users.
How I found it, and what changed
I didn't find it through monitoring or a bug report. I found it by accident while going through a Firebase analytics export, and I fixed it the same day.
After the fix, 30-day active users recovered to roughly 420 by late September. Not back to the June peak, but a long way from 126, and a clear sign that a lot of the drop had never been churn at all.
Why this is easy to miss when you build alone
When you're a solo developer, you're also the only person who uses the app with a full picture of how it's supposed to work. You know which apps should be in Discovery, so you rarely look at it the way a new user does. The empty space a user sees doesn't look empty to you.
There's also an emotional pull toward the churn story. A bug is your fault and has to be fixed today. Churn feels like the market's fault and can be analysed calmly. It's tempting to pick the explanation that asks less of you.
None of this needs a team to fix. It needs a habit: whenever a number moves sharply, assume the app first and the users second, and go and look.
What I'd check before calling it churn
First, look for anything time-based. Filters like "newer than X days", expiry dates and automatic restarts are where old assumptions hide. A rule that made sense when you wrote it can quietly break when another part of the app changes.
Second, check every state your app can be in and whether the user can see it. Paused, expired, waiting, hidden: if a state exists in your database but not on screen, nobody will ever report it, because nobody knows it's there.
Third, look at your raw data, not just the dashboard graph. An export lets you ask questions the dashboard wasn't built for, and that's where the answer was for me.
The same habit helps during a Google Play closed test. When testers seem to go quiet, check that the app is actually working for them before you decide they've lost interest. That's part of why PeerPlay checks that a tester really opened an app for a real session: it tells you whether people are using the app, or just have it installed.
Top comments (13)
churn is the comfortable story, a bug is the one that fixes things. the emotional pull point is the real insight here, blaming the users costs you nothing. one more check for the list: verify the metric definition itself hasn't changed. a renamed or dropped event quietly stops sending, and the graph looks exactly like churn.
Exactly. "Churn" was the comfortable answer, so I almost stopped looking. In my case the events were fine and the app list itself was wrong: Discovery was hiding every app older than 14 days. I only caught it by accident in a Firebase analytics export. Checking whether the metric definition changed is now the first thing I'd do before calling anything churn.
glad the checklist found a home. the accidental export catch is an origin story every postmortem secretly wants.
The "exists in the database but not on screen" line is the one I'd turn into an actual check. A small weekly query that counts rows per status (paused, expired, hidden) next to last week's numbers would probably have shown paused campaigns piling up before the active users graph moved at all. Was the 14 day filter written before auto restart existed, or did both ship together?
Agreed, that's the check I'd build first. A weekly count per status next to last week's would have shown paused campaigns piling up long before active users dropped. The 14-day filter and auto restart weren't built together, and nobody checked how they behaved side by side, which is exactly how the gap slipped through. Lesson learned: every time a campaign can change state, someone should ask whether it's still visible after that.
Thanks for writing up the wrong turn as well as the fix. Assuming churn first is so natural, and the numbers fit it well enough that nobody questions it.
What strikes me is that both bugs were invisible from the dashboard. The metric moved, but nothing told you which part of the product had stopped doing its job. One thing that helps on products like this is a small set of health numbers that track supply as well as users. Here that might be the count of active campaigns visible in Discovery each day. If it drops while campaigns are still running, it points straight at the product rather than at the people.
It is a cheap addition, and it turns weeks of quiet drift into an alert. Are you planning to add something like that, or did fixing the 14 day filter feel like enough for now?
Exactly that. The dashboard told me how many people showed up, but nothing about what they found when they got there. Fixing the 14-day filter solved this bug, not the blind spot, so a daily count of campaigns visible in Discovery next to the count of campaigns that should be live is next on my list. If those two numbers drift apart, it's the product, not the users. Thanks, that's a really useful way to put it.
Discovery quietly hiding every app older than 14 days is such a sneaky failure, because nothing errors and the graph looks exactly like users losing interest. What I take from this is to always ask "could the product have stopped showing people the thing" before "did people stop wanting the thing". Glad you caught it from the export instead of shipping a churn fix for a problem that didn't exist.
Thanks, that's a better way to put it than I managed in the post. "Could the product have stopped showing people the thing" is going on my checklist before I ever call something churn again. The scary part was how reasonable the wrong answer looked. Any churn fix would have done nothing, because the users never left, the apps just disappeared from their screen.
the second bug is the one that stands out: a paused campaign that nothing on screen shows. a cheap guard is a check that compares what the database says should be live against what Discovery actually lists, so a mismatch fires before the graph drops. it also gives a dip in active users a first place to look that isn't churn.
Yes, the second bug was the sneakier one, because nothing looked broken. A check that compares what the database says is live with what Discovery actually lists would have caught both bugs before the graph moved. Thanks, that's a much better first place to look than "users left".
this is the scariest kind of drop because every dashboard just says "normal churn". i've started keeping one boring metric next to active users, like how many items someone actually sees per session, so a supply problem can't pass itself off as a demand problem
That framing is great. It was exactly that: people still opened the app, there just wasn't enough to see. Items seen per session would have shown the drop on day one. That's going on my list next to active users.