Ján Turský

Posted on Oct 30 AI-assisted

I asked two phones where my office is. They did not agree.

#kotlin #android #ios #testing

I asked two phones where my office is located. They did not agree. One of them also thinks it's an account problem.

Before you ask, no, I don't have an office. I just wanted to check if a tiny prompt survives a model update, and the office question happend to be case number twelve. That's it. That's the whole origin story.

The "experiment" (I use that word loosely)

What did I do? I wrote a support ticket classifier. Five labels: Billing, Bug, Account, Feature, Other. Twelve tickets, each one run three times, temperature 0.2, exact match against the expected label. That's the whole experiment. (Source: Trust me bro, but also the JSON files, they are public.)

val suite = evalSuite("ticket-classifier") {
    prompt(version = "v1") { input ->
        "Classify this support ticket as one of Billing, Bug, Account, Feature or Other. " +
            "Reply with the category only.\n\nTicket: $input"
    }
    case("My card was charged twice", id = "double-charge") { expect("Billing") }
    case("Where is your office located?", id = "office") { expect("Other") }
    graders { exactMatch(ignoreCase = true) }
    repeat(3)
    passWhen { score >= 0.8 }
}

The suite lives in commonMain, and the runner is a small library I built called Proctor. It runs the same suite against the model that is already inside the phone and writes one JSON file per run. Gemini Nano on Android (through ML Kit), Apple Foundation Models on iOS.

I tested on a Pixel 11 Pro with all four Gemini Nano 4 variants you can pick in AICore: stable and preview, full and fast. For iOS I used the simulator, which borrows the Mac's Foundation Models. Yes, simulator. I'll come back to that, don't worry.

Android is boring (and that's good)

Twelve out of twelve tickets pass on nearly every run, about 140 ms per call. That includes the office question, which gets Other three times out of three. The only wobble in eight classifier runs was preview/fast on the first prompt, which missed one case.

Boring is exactly what you want from a classifier.

iOS has opinions

On iOS the office ticket fails. Three attempts, three answers:

Expected "Other", got "Account"

It fails with both prompts. The second prompt even adds a short definition per category (Billing is charges, refunds, invoices. Account is login, password, profile) and the iOS answer didn't move at all. Mean score was 92% on the first prompt and 97% on the second, one failed case each.

Why does Apple's model file an office location under Account? I have no idea. My guess is that "your" and "located" sounds like it's about the company details, but that's a guess, not a finding. What I can tell you is that it's very stable. Same answer every time, about 335 ms.

Here is the thing that I find interesting. Both phones are 11/12 or better. If you test it by hand with five tickets you would never notice. You only see it because the exact same twelve tickets ran on both.

"Valid JSON" is not the same as "correct"

The second suite pulls three fields out of a ticket as JSON: order id, product, urgent. One ticket says "I cannot log in to the web dashboard, blocking my whole team right now." and the expected product is web dashboard.

Every Gemini Nano variant and the iOS model answer with "product": null. Nobody finds the dashboard. That's a real model miss, fair enough. Three of four cases pass on Android, two of four on iOS.

But look at what Android actually returns:

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

Gemini Nano wraps the JSON in a Markdown code fence, the iOS model returns bare JSON. The jsonSchema grader passed on both (it strips the fences before decoding) and the field check detailsMatch is the one that failed with "product null instead of web dashboard".

Two graders, two different opinions about the same output. If I only checked "is it valid JSON?" I would have a perfect score and a bug in production.

So why bother?

The system model is not part of your app. The OS updates it, not you. Different phones run different versions, and the model that ships next autumn is not the one you tested. Google's own ML Kit docs recommend running an evaluation dataset per Gemini Nano version, and then give you no on-device tool for it. Apple has an Evaluations framework, but for Apple platforms only.

So the check I wanted was small: run the same cases on every model I can reach, keep the JSON, diff it later. Nothing fancy.

What I have NOT verified

Because someone in the comments will ask, so lets do it first:

iOS ran on a simulator with the Mac's Foundation Models. No real iPhone with Apple Intelligence was involved. And yes, I do own an iPhone 17e, so you would think this is easy. It's not. The iPhones sold in my country (Europe) come with a model number that says the phone was made in China, and for that variant the Apple Intelligence feature is locked. So the one iPhone I have can't run the model I wanted to test. Sadly. If you have an iPhone with Apple Intelligence working and want to run the same suite, I would really like to see your numbers.
Twelve tickets and four extraction cases is a smoke test, not a benchmark. Three repeats at low temperature shows stability, not coverage.
One device, one Android build. Another phone may ship a different Nano.
The sample report labels runs by model version, so stable and preview of the same model looks the same in one table. I know. It's on my list.

Try it

Proctor 1.0.0 is on Maven Central (io.github.jantursky:proctor, plus proctor-mlkit for Android and proctor-ios for iOS), MIT licensed. The sample report from the runs above is open, so you can look at every case without installing anything.

Repo: https://github.com/jantursky/proctor
Sample report: https://proctor-demo.vercel.app
Docs: https://proctor-docs-phi.vercel.app

Did a model update ever change the output of your Prompt API feature? Tell me what moved, I'm curious.

DEV Community