Most of my Android "purchase errors" were Google's test phones

I build a few mobile apps on the side. One of them is a German citizenship test prep app, on iPhone and Android, with a paywall and RevenueCat for purchases.

Last week Android looked broken. The funnel said people opened the paywall, tried to buy, and got billing errors. iPhone was fine. I was about to go digging in the billing code.

Then I looked at who was producing the errors. Most of them were not people.

Google's test phones

When you upload an Android build to Google Play, Google can run a pre-launch report: it installs your app on a set of test devices, taps around, and sends you a report. Those devices are real sessions as far as your SDK is concerned. They open the app, they reach the paywall, and they try things.

In my data they were easy to see once I looked:

• The device model starts with sdk_gphone (the Android emulator), or it is a model that no real user of my app has.
• Mostly Android 11, the same emulator model every time, always an English locale.
• A US IP address, even though the app is for people in Germany.
• A burst of sessions around the same minute, and the device never comes back.

I'm only describing what I saw in my own data. I can't promise it is how every pre-launch device looks, so check yours before you filter on it. The model name is the part I would trust most.

Billing on these devices fails. RevenueCat returns "device or user is not allowed to make the purchase". Every time a test phone opens the paywall and taps buy, you get a purchase error.

On one busy day, test phones made all of the purchase errors in that app. A real user had nothing to do with them.

Apple has the same thing. Review runs your build too, usually as a US iPhone on a version that is not live yet. It belongs in the same filter.

Why leave them out of funnels

A funnel only works if each step is a person deciding something. Test phones break that in two ways.

They add people who were never going to buy. That pushes your conversion down. And they create errors that no real person saw. That makes a healthy payment flow look broken.

Mine did both. Android's paywall looked like it had a billing problem, so I wanted to spend a day on it. The problem was in my analytics, not the app.

What the numbers looked like after

I excluded the test devices and counted again. Of the Android people who saw the paywall, about 1 in 40 tapped buy. On iPhone it was about 1 in 9.

That is a real gap, and it is a very different problem from "Android billing is broken". The first one is a story about the paywall, the price in local currency, or who uses Android for this kind of app. The second would have sent me into the wrong code.

I don't know yet why Android is lower. I have ideas, and I'll test them one at a time.

The real bug the filter uncovered

With the noise gone, one thing stood out. A real user had tapped buy 12 times, and I had nothing logged for any of them. Not a success, not an error.

So I read the purchase code. There was a path where the purchase call returned, but the entitlement was not active yet, and the function simply ended. No message, no event, no alert. From the user's side, they tapped buy and nothing happened, so they tapped again.

That was the silent "no entitlement" path. It was my bug.

The fix: log every outcome

Every purchase attempt should end in exactly one named outcome, and the user should always see something. These are the four I use: unlocked, no_entitlement, cancelled, error.

Here is the shape of it, in TypeScript with a RevenueCat-style API:

type Outcome = "unlocked" | "no_entitlement" | "cancelled" | "error";

let buying = false;

export async function buy(pkg: Package) {
  if (buying) return; // guard double taps
  buying = true;
  track("purchase_tapped", { product: pkg.identifier });

  // if nothing has finished after 20s, say so
  const pending = setTimeout(() => track("purchase_pending"), 20_000);

  const finish = (outcome: Outcome, extra: object = {}) => {
    clearTimeout(pending);
    buying = false;
    track("purchase_outcome", { outcome, ...extra });
  };

  try {
    const { customerInfo } = await Purchases.purchasePackage(pkg);
    if (customerInfo.entitlements.active["pro"]) {
      finish("unlocked");
      return;
    }
    finish("no_entitlement");
    Alert.alert("Almost there", "The purchase went through but Pro is not active yet. Tap Restore, or try again in a minute.");
  } catch (e: any) {
    if (e.userCancelled) {
      finish("cancelled");
      return;
    }
    finish("error", { code: String(e.code ?? ""), message: String(e.message ?? "") });
    Alert.alert("Purchase failed", "Nothing was charged. Please try again.");
  }
}

The details that matter:

• One outcome per attempt. If you can't tell which of the four happened, a funnel step is hiding.
• No silent path. The no_entitlement branch used to be a bare return. Now it logs and shows an alert.
• Guard double taps. If the first tap is still running, the second one does nothing. Before this, 12 taps were 12 calls.
• A pending event after 20 seconds. Slow billing is not an error and not a success. Without this event you can't see it at all.

Keep the error code and message in the event. They are how you tell "device not allowed" from a real failure.

How to do this in your own data

1. Add the device model to your events if your SDK doesn't. In my case the iOS SDK did not send the model at all, so I couldn't filter iPhone test devices by model and had to go by app version.
2. Look at purchase errors grouped by device model. If a model like sdk_gphone... is on top, you've found them.
3. Check the pre-launch report in Play Console for the exact devices it used, and compare with what you see.
4. Filter the same way in every chart: funnels, retention, error counts. Doing it in one chart and not another gives you two numbers that disagree.
5. Re-run your funnel and read the ratios, not the totals.

Disclosure

I build Apparent, which leaves Google's and Apple's test phones out of every number by default. If you want to try it, it's at apparent.fyi. The same filter works if you do it by hand, and the code above doesn't need it.

The part I'd keep even if you never use a tool: log every outcome, and never let a purchase end without telling the user something.