A machine learning model can achieve 95% accuracy during development and still perform terribly after deployment. This is one of the frustrating realities of machine learning.
You train the model, evaluate it on your test set, see impressive results, and think the hard part is over. Then real users start interacting with it. The predictions become less accurate. Why? Because a model is not trained on “the real world.” It is trained on the data you gave it. And those two things are not always the same. In essence, a machine learning model is an abstraction of reality.
What follows are some reasons why well-performing models may still fail in production.
1. Your training data does not represent production
Suppose you build a model to predict whether a transaction is fraudulent. Your training data might look like this:
| Amount | Country | Device | Fraud |
|---|---|---|---|
| 500 | Nigeria | Mobile | 0 |
| 1,200 | Nigeria | Web | 0 |
| 9,000 | Nigeria | Mobile | 1 |
| ... | ... | ... | ... |
If most of your historical transactions came from mobile users, the model may learn patterns that work particularly well for mobile transactions. But after deployment, suppose the company starts receiving many more transactions from web users. The distribution of the input data has changed. Your model has not suddenly become worse at mathematics. The world it is seeing has changed. This is commonly described as data drift or, more broadly, distribution shift [1].
2. Your test set may be too similar to your training data
A model can perform extremely well when the test data resembles the training data too closely. Consider a dataset containing customer records from 2024 and 2025. If you randomly split the entire dataset into training and test sets, records from both years may appear in both sets.
That might be appropriate for some problems. But suppose customer behaviour changes over time. A model trained on randomly mixed historical data may look excellent during evaluation while struggling when asked to predict behaviour in 2026.
For time-dependent problems, how you split the data matters. Sometimes a chronological split is more realistic:
Training: January 2024 – December 2025
Testing: January 2026 – March 2026
The goal is not simply to obtain a high test score. The goal is to create an evaluation that resembles how the model will actually be used.
3. Your preprocessing may not be the same
Imagine that you standardize numerical variables before training. During development, you calculate the mean and standard deviation from the training data. But when the model is deployed, someone implements the preprocessing differently. Perhaps a column is scaled incorrectly. Perhaps missing values are handled differently. Perhaps a categorical variable is encoded using a different mapping. The model itself has not changed. The data entering the model has. This is why preprocessing should normally be treated as part of the machine learning pipeline rather than as a collection of disconnected steps.
With Scikit-Learn, for example, you can combine preprocessing and modelling:
from sklearn.pipeline import Pipeline
pipeline = Pipeline([
("preprocessor", preprocessor),
("model", model)
])
The pipeline helps ensure that the same transformations are applied consistently.
4. The target can change
Sometimes the problem is not the input data. The meaning of the target variable changes. Imagine a model predicting whether a customer will default on a loan. If the company changes its definition of “default,” historical labels and future labels may no longer represent exactly the same thing. The model is still solving the problem it was trained to solve. But the problem itself has changed. This is sometimes referred to as concept drift [2].
5. Production has problems your notebook does not
A model in a notebook only needs to produce predictions. A production system has other requirements. It may need to:
- respond quickly;
- handle thousands of requests;
- deal with missing or unexpected values;
- work with changing databases;
- maintain consistent preprocessing;
- log predictions;
- monitor performance.
A model with excellent accuracy but unacceptable response time may still be a failed production system. Machine learning in production is therefore not just about the model. It is about the system surrounding the model.
Summary of the shifts and failures
| Reason | What changed? | Drift / Shift classification |
|---|---|---|
| 1. Training data does not represent production | The distribution of production inputs differs from training data | Data drift / Distribution shift |
| 2. Test set is too similar to training data | Evaluation data does not adequately represent the future/production distribution | Temporal distribution shift (potentially data drift in production) |
| 3. Preprocessing is not the same | The transformation/representation of features changes between training and production | Preprocessing mismatch (can create an input distribution shift) |
| 4. The target changes | The relationship between features and target, or the meaning/distribution of the target, changes | Concept drift |
| 5. Production has problems your notebook does not | System conditions change: latency, missing values, unexpected inputs, infrastructure, etc. | Not necessarily drift/shift — production/system failure |
The bigger lesson
A good test score answers a relatively narrow question:
“How well does this model perform on this evaluation data?”
Production asks a much harder question:
“How well does this entire system continue to perform when the real world starts changing?”
That is why a model can have excellent accuracy, precision, recall, or R2 during development and still fail after deployment. The solution is not necessarily a more sophisticated algorithm. Sometimes the real problem is data drift, leakage, an unrealistic evaluation strategy, inconsistent preprocessing, changing targets, or poor system design.
The model is only one component. A machine learning model succeeds in production when the assumptions made during development continue to hold when the model meets reality.
Top comments (0)