Praveen

Posted on Oct 26

My Closed Testers Hit a Crash on Day 3. Here's How I Triaged It Without Losing the 14 Days

#android #googleplay #kotlin #debugging

On day 3 of my Google Play closed test, one tester sent a short message: "it closes when I open settings." No stack trace, no device name, nothing else. I had a test running on the closed track, a 14-day window to protect, and a bug I couldn't reproduce on my own phone.

This is the routine I now follow when a crash shows up in the middle of a closed test. It's mostly about getting a readable stack trace out of Play Console and shipping a fix without breaking the test itself.

1. Make sure Play Console can actually read your stack traces

My first crash report in Android vitals looked like this:

java.lang.NullPointerException
    at a.b.c.d(Unknown Source:12)
    at a.b.e.onCreate(Unknown Source:4)

That's R8 doing its job. Without the mapping file, Play Console shows obfuscated names. Two things fixed it for me:

Upload the mapping file for every release. In Play Console, open App bundle explorer, pick the version, go to the Downloads tab and upload mapping.txt from app/build/outputs/mapping/release/. Do it for the exact versionCode testers are running.
Ship native symbols if you use NDK code. In the module's build.gradle.kts:

android {
    buildTypes {
        release {
            isMinifyEnabled = true
            ndk {
                debugSymbolLevel = "FULL"
            }
        }
    }
}

After that, new crashes from the same version showed real class and method names. Older reports don't always re-symbolicate, so I don't panic if the first few stay unreadable.

2. Find the crash for the right version

With a small tester group, vitals data is thin, so I don't wait for charts. I go to Monitor and improve > Android vitals > Crashes and ANRs, filter by the app version the closed track is serving, and open the cluster. Each cluster shows the stack trace, Android version and device model. That alone told me my crash only hit Android 10 devices, which explained why my Android 14 phone never saw it.

I also check Ratings and reviews > Testing feedback. Testers sometimes leave a private note there instead of messaging me, and those notes are the honest, app-specific feedback I'll want later anyway.

3. Reproduce on the same Android version

My bug was a settings screen reading a value that older Android versions return as null:

// crashed on API 29: value can be null there
val mode = prefs.getString("theme_mode", null)!!

// fixed
val mode = prefs.getString("theme_mode", null) ?: "system"

I borrowed an old Android 10 phone from a friend and reproduced it in under a minute. Fix it, add a quick unit test around it, move on.

4. Ship the fix without touching the testers

This is the part where I almost made a mess. Things I keep the same:

Same closed track. Upload the fixed AAB as a new release on the closed track the testers already joined. Don't create a new track for the fix.
Same tester list or Google Group. Don't remove and re-add anyone. Google's rule for new personal accounts is at least 12 testers opted in continuously for 14 days, and an opt-out followed by an opt-in means those days have to start again for that tester.
Higher versionCode. Bump it or the upload is rejected.

Then I check Publishing overview. If managed publishing is on, the update sits there until I press publish, and testers keep the crashing build in the meantime.

5. Tell testers there's an update

Play updates closed-test apps like any other app, but auto-update can take a while. I send one short line: "version 1.0.4 fixes the settings crash, update from the Play Store when you can." That's it.

What I'd do differently

I'd upload the mapping file before the test starts, not after the first crash. And I'd line up enough testers from day one so one person going quiet doesn't put the 14-day window at risk.

That second part is why I looked at managed tester pools at all. 12testers is a budget option at $8 per app: you add their Google Group or share your opt-in link, 14 physical Android devices join your closed track, and they keep an eye on opt-ins over a 16-day window. You still need your own feedback and honest answers for production access, and Google makes the final call.

If you've hit a crash only on one Android version during a closed test, I'd like to hear how you tracked it down.

DEV Community