DEV Community

Cover image for My Android AI app has 400 unit tests. None of them can tell me if the model's answer is any good
Grokkris
Grokkris

Posted on

My Android AI app has 400 unit tests. None of them can tell me if the model's answer is any good

#android #kotlin #testing #showdev

I'm a student building an open-source Android AI app on my own, and at some point I hit 400 unit tests with zero failures. That sounds great until you ask what those tests actually prove.

Quick context: Kryzz AI is an MIT-licensed Android app (Kotlin + Jetpack Compose) for chat, voice and agent tasks. It doesn't run models on the phone. You bring your own OpenRouter key and the models are hosted by whichever provider you pick. Chats, memory, files and keys stay on the device, and there's no analytics or telemetry.

What the tests are good at

An AI app has a lot of plain, boring code around the model, and that's exactly where tests pay off. Looking at the test folder, most files fall into a few groups:

  • Parsing what comes back. Streaming responses (SseParserTest), API responses, tool calls written as XML, and the agent's plan (AgentPlanParserTest). Models return slightly broken output all the time, so the parser has to survive it.
  • Rules the agent must follow. Which links it's allowed to open (FetchAllowlistTest), how many steps a turn can take (AgentLoopPolicyTest, AgentTurnPolicyTest), and what happens when the model sends back nothing at all (EmptyResponsePolicyTest).
  • Cleaning up before the user sees it. Stripping leftover reasoning text (ReasoningScrubTest), shrinking old context so a long chat still fits (ContextCompactorTest).
  • Files the agent builds. Spreadsheet content, document layout, office files and code project zips.
  • Voice and UI details. When to decide you've stopped talking (AdaptiveEndOfSpeechDetectorTest), side panel swipes, navigation.

These catch real bugs. A change to the plan parser that breaks on one weird model output shows up in seconds instead of in someone's chat.

What they can't tell me

  • Whether the answer is good. Every test runs without a real model. They check that the plumbing works, not that the reply is useful, correct or well written.
  • Whether it works on your phone. Unit tests run on my computer. The new signed release build has so far only been checked on an emulator, which is the honest gap I most want to close.
  • Whether a new user gets through setup. The biggest drop-off I've seen is the first screen asking for an API key. No unit test would have found that; one person on a forum did.

So "400 tests, 0 failures" means the app is less likely to fall over. It doesn't mean it's good. For that I still need people actually using it.

If you build apps on top of LLMs: how do you check answer quality without paying for a model call on every test run? Recorded responses, a small eval set, or just trying it yourself?

Try it

Honest caveats: you need your own OpenRouter key, it's not on the Play Store, and the release build is emulator-tested only so far. Bug reports, especially from real devices, are very welcome.

Top comments (0)