Why an Image Classifier That Tests Well Fails in Production

#ai #debugging #llm #machinelearning

“Validation accuracy 0.97, production accuracy about 0.6.” Or the sharper version: “the model is 99% accurate and operators say almost every alert is wrong.” The architecture is almost never the cause. Four things are, and they leave different fingerprints.

The symptom, stated precisely

Before diagnosing, pin down which number moved. “Accuracy dropped” conflates three different situations: the model is genuinely worse on production inputs; the model is the same but the class balance changed so accuracy is no longer the right measure; or the model was never as good as the test set said. Those have different fixes and the second one involves no model change at all.

So: get a few hundred production images labelled by whoever adjudicates the alerts, and compute precision and recall per class on them. Then compute the same on your test set. If recall held and precision collapsed, look at the base rate first. If both fell, look at preprocessing and shift. If test accuracy looks implausibly high for the problem, look at the split.

The split was leaking

This is the most common single cause and the most embarrassing, because it means the reported number was never real. The tell is a test score suspiciously close to the training score, and often very high from the first few epochs.

Near-duplicates across splits. Capture is rarely one image per item. Burst mode, a video frame grab, three angles of the same part, the same product photographed on two days — a random row-wise split puts some in train and some in test, and the model recognises the item rather than the class. Detect it: compute a perceptual hash or an embedding for every image and count cross-split pairs above a similarity threshold. Any nonzero count is the size of the problem. See deduplicating with embeddings and dataset decontamination.
Augmenting before splitting. If five crops were generated per source image and the split came afterwards, the test set is crops of training images. Split first, augment the training side only.
Fitting statistics on everything. Normalisation constants, a PCA basis, or a class-rebalancing resample computed over the full dataset before splitting leaks distribution information into the test set. Smaller effect than duplicates, same category of error.
Splitting by row instead of by group. The correct unit is whatever will be shared between train and production: item id, capture session, day, camera, production lot, patient, store. Split by that group. If the score falls sharply when you re-split this way, the original split was the whole story.

Serving preprocesses differently

The second most common cause and the cheapest to rule out, because nothing raises an error — the pipeline runs perfectly and feeds the model different pixels than training did.

Resize semantics. Downscaling a 4000-pixel image to 224 pixels without antialiasing aliases severely, and libraries differ on whether they antialias by default. Training with one library’s bilinear and serving with another’s produces visibly different inputs at the same nominal setting.
Channel order. OpenCV loads BGR, most other things load RGB. A swap costs a great deal of accuracy and produces no error.
Normalisation range. Applying ImageNet means and standard deviations, which assume inputs scaled to 0–1, to 0–255 tensors saturates everything. The reverse under-uses the input range.
EXIF orientation. Not applied automatically by most libraries. Phone photographs arrive rotated ninety degrees and the model has never seen a sideways anything.
Crop policy. Training on random resized crops and serving a full-frame squash changes the apparent object scale, which for a scale-sensitive task is a large shift.
Compression. Training on archival originals and serving quality-60 JPEGs from a camera is a distribution shift you introduced yourself, and it is easy to fix by degrading the training data to match.

The diagnostic is one experiment and it should be the first thing you run: take your test set and evaluate it through the serving preprocessing path rather than the training one. If accuracy collapses there, you have reproduced the production gap on data you already have labels for, and the cause is entirely upstream of the model.

The base rate changed and the threshold did not

Accuracy measured on a balanced test set does not survive an unbalanced production stream, and the size of the effect is arithmetic rather than a matter of degree.

Take a genuinely good classifier: 99% true positive rate, 1% false positive rate. On a balanced test set that is 99% accuracy. Now deploy it where the defect rate is one in a thousand.

100,000 items, defect rate 0.1%

      100 defects      * 0.99 TPR = 99 true positives
   99,900 good items   * 0.01 FPR = 999 false positives

precision = 99 / (99 + 999) = 99 / 1098 = 0.090

nine percent of alerts are real. The model did not change;
the prior did.

Nothing about the network is wrong here, and no retraining fixes it. The corrections are: report precision and recall at the production prior rather than accuracy at all; choose the decision threshold on a validation set reweighted to that prior, or directly against the precision the operation requires; and if a fixed 0.5 threshold was inherited from a tutorial, replace it. See which classification metric answers which question and dataset balancing.

There is also an exact correction for the scores themselves when only the class prior has changed and the class-conditional distributions have not: add the log ratio of the production prior to the training prior to each class logit before the softmax. This is the standard prior-shift adjustment, and where the production prior is unknown it can be estimated from unlabelled production predictions by expectation-maximisation. It does not help if the inputs themselves shifted, which is the next section — and the scores need to be calibrated for it to mean anything; see confidence calibration.

Genuine covariate shift

Once leakage, preprocessing and the prior are ruled out, the remaining gap is real: production images are drawn from a different distribution than the ones you trained on. In vision this is usually physical and usually obvious once named.

The camera. Collection used a phone; production uses a fixed small-sensor camera with a different lens, different noise characteristics and aggressive in-camera sharpening.
The lighting. LED strips flicker at mains frequency and produce banding at some exposure times. Colour temperature differs from daylight collection. Specular highlights land in different places because the fixtures are fixed.
The background and pose. Items photographed on a bench during collection arrive on a moving steel conveyor with reflections and neighbouring items in frame.
Time. Suppliers change, packaging is redesigned, a lens fouls over three months, seasons change the ambient light. This one arrives gradually and is invisible without monitoring.

You can quantify it without labels. Embed a sample of production images and a sample of your test images with the same backbone, then train a small classifier to tell the two sets apart. If it succeeds well above chance, the sets are distinguishable and your test set is not representative — and the features it uses tell you what shifted. Monitoring the embedding distribution over time catches the gradual version; see embedding drift.

The fix for real shift is data, not architecture. Collect from production, label the cases the model gets wrong or is least certain about, retrain. Active learning on low-confidence production frames buys considerably more accuracy per label than random collection. Augmentation that simulates the specific production nuisances — motion blur, JPEG artefacts, colour temperature, exposure — is cheap and helps, and where real examples of a rare class are unobtainable, synthetic training images are a partial substitute with their own gap to manage.

The order to check these in

Re-evaluate your existing test set through the serving preprocessing path. Minutes of work, no new labels, and it resolves a large share of cases outright.
Hash or embed every image and count near-duplicate pairs that cross the train/test boundary. A nonzero count means the headline number was inflated and you do not yet know by how much.
Re-split by the correct group — session, day, device, lot — and retrain. Compare against the original split. The drop is the leakage.
Recompute precision and recall at the production prior and re-select the operating threshold against the precision the workflow needs.
Train a test-versus-production discriminator on embeddings to quantify the remaining shift, and inspect what it keys on.
Only now change the model or collect more data, targeting what steps three to five actually revealed.

One thing to put in place before any of it: log production inputs alongside their predictions and, where available, the eventual outcome. A gap you cannot reproduce offline is a gap you will be arguing about rather than fixing, and the log is also the only honest source of a representative test set for the next model. The related failure on continuously-scored feeds — where the same base-rate arithmetic governs the alarm rate — is in anomaly detection on a quality-control feed.