We put a feedback button in every build we ship. Tap it, type a line, and the report arrives with the device model and the app version. Each report becomes a ticket, and each ticket starts a fix attempt.
Then a trivia app we had just uploaded to Play got seven reports in a single day. The texts were ospuuq, pzxoim, caokah. Six lowercase letters each. All seven came from the same OnePlus 8 Pro, all on app version 1.0.0.
It was not a person. It was the Play pre-launch report (Firebase Test Lab's Robo crawler), which taps every input it can find, types something, and presses the button next to it. It does this again on every upload.
So for a day we had counted a crawler as our most active tester, and some of those tickets had already started fix attempts for a bug that did not exist.
What we changed
- Test Lab devices report the system setting
firebase.test.labas"true". A webview cannot read that, so we added a tiny Capacitor plugin that exposes it. - The feedback widget now sends
where: "robot"from those devices, and the ticket collector skips that value. - We did not filter by "looks like random letters". That would also throw away the short, odd, real messages from actual players.
The day was lost to believing a number on the dashboard, not to the fix.
If you ship a free-text field in a mobile build, open your pre-launch report once and read what your first "users" typed into it.
We make small idle tycoon games, in case you want to see what the robot was testing: https://qqq1212341.github.io/pcbang-alba-sim/?utm_source=dev-to-gamedev-tag&utm_medium=community
Top comments (2)
Since the pre-launch report repeats on every upload, I’d keep each robot report in a test-only queue and attach the app build number or test-run ID where you can. That preserves a useful signal if the crawler finds a real crash or blocked field, while keeping robot entries out of player-feedback counts. Your
where: "robot"marker gives you the split; build-level IDs would make repeated runs easier to compare.Good call not filtering on "looks random". The short odd messages from real players are often the most useful ones.
I'd keep the robot reports in their own bucket instead of dropping them, though. Most of what it types is noise, but it also finds real things: a field that accepts an empty string, a button that sends twice on a double tap, a crash on a screen nobody on the team opens. Worth a quick look after each upload.
The same firebase.test.lab check is worth wiring into anything that counts users, not just feedback. On a new app, the pre-launch devices can be a big share of your day-one numbers. We log questionnaire started and finished to measure completion, and your post made me want to check whether the robot shows up in that log too.