Two Android ML tests that could never fail, and a TFLite check that can

Two tests in my Android app could never fail. One only counts the labels; the other compares the output with an
answer file that holds no answers. And both of them look at this picture:

The app runs a Keras model exported to LiteRT, the new name of TensorFlow Lite. This post is about one rule: a check
proves something only if you have seen it fail. Here is how mine could not, and the check I would build instead.

Two tests that could not fail

The classification test asks only this of the result
(ClassificationE2ETest.kt, line 66):

assertTrue("Expected top-5 results",
    success.output.size in 1..5)

Any five labels pass, including the five least likely ones the app shows
(part 1
explains why it shows those).

The detection test looks better. It reads a golden file, the saved answer a test compares against, and matches each
expected box by its label and its overlap. Here is the golden file
(detection_golden.json,
its note cut to fit):

{
  "_note": "Golden generation skipped: ...",
  "model": "ssd_mobilenet_v1_detect.tflite",
  "image": "images/test_image.jpg"
}

The note in full reads "Golden generation skipped: No module named 'ai_edge_litert'". No boxes. The match loop runs
once per expected box, so with none it checks nothing, and the size check becomes "at least zero"
(DetectionE2ETest.kt, lines 130 to 131).
The script that should write the answers wrote this placeholder and carried on.

The check that can fail

1. Get the notebook's answer

Start from a real photo with a known answer. KerasHub's own example uses a California quail, and "quail" is class 85
in ImageNet, the label set these classifiers use
(imagenet_utils.py).
Its example downloads the photo with keras.utils.get_file, and on 22 October 2026 Wikimedia answered
403 -- Forbidden. That photo is CC BY-SA 3.0, too, not public domain. So I downloaded a public-domain one from the
U.S. Fish and Wildlife Service,
cropped it square, and loaded it from disk. With my exported preset in the example's place:

image = keras.utils.load_img("california_quail.jpg")
classifier = keras_hub.models.ImageClassifier.from_preset(
    "mobilenet_v3_small_100_imagenet",
    activation="softmax",
)
preds = classifier.predict(np.array([image]))
top_labels = keras_hub.utils.decode_imagenet_predictions(preds)

predict() runs the preset's own preprocessing, so this is the answer the model was built to give. On a computer's
CPU it gave quail 10.30, partridge 3.94, prairie chicken 3.77, black grouse 3.57 and ruffed grouse 3.43. One surprise:
those are raw scores. Despite activation="softmax", the 1,000 scores summed to 162.5, not 1.

2. Check the .tflite file in Python first

Run the .tflite file on the array the Keras model sees after preprocessing (KerasHub's hint is
model(model.preprocessor(x))), with the interpreter code from the
keras.io guide. Then compare the two outputs the way KerasHub's own export
tests do
(test_case.py):

abs_diff = np.abs(keras_val - litert_val)
mean_abs_diff = np.mean(abs_diff)
max_abs_diff = np.max(abs_diff)

On the quail photo they agreed: max_abs_diff 1.1e-05 and mean_abs_diff 1.9e-06 over the 1,000 scores, with the
same five labels in the same order and quail at 10.31 in both. If the file disagrees with the notebook here, the phone will too, and the bug is
in the export, not the app.

3. Hold the phone to the saved answer

Save the five labels and scores as the golden file, or for detection the boxes as label, score, x1, y1, x2,
y2: the keys my detection test already reads. Push the same photo to the phone and keep the test's matching rule:
the same label, and a box overlap of 0.5 or more, measured as IoU (the area two boxes share divided by the area they
cover together). Two guards make it a test that can fail:

• An empty or missing golden file fails. It never counts as a pass.
• The same pixels go in. The preset crops the photo to a square before it resizes it, and my app stretches it. Use a square photo, or crop it first.

Here it is failing

I copied my app's preprocessing into Python (resize to 224 by 224, then the preset's numbers) and fed the same file
the same square photo. Quail still came first, at 10.11 instead of 10.31, but prairie chicken and partridge swapped
places. A check that compares the ordered five labels fails right there, with the right numbers in the sidecar: what
is left is the resize step. That is the rule working, on a computer's CPU; I did not run the phone for this post.

What I have not solved

How close is close enough? My app's resize alone moved the top score by 0.2 and swapped two labels, before any
quantization (weights stored in fewer bits to make the file smaller). KerasHub's thresholds are set by hand, per model
and per output, and I do not have a rule for picking one. If you have one, I would like to read it.

The two tests are still in the repo as they were at commit b8d3e2f (3 July 2026). The scores and differences above come from a run
on 22 October 2026 with keras 3.15.1, keras-hub 0.32.0, litert-torch 0.10.0 and ai-edge-litert 2.3.0.

Next: five ways the export itself breaks, each with its public record →

---

This article was written with the help of AI, from my own repository and the public docs it links.