Design Patterns for Resilient Serving - Continued Model Evaluation

This design pattern helps detect and take action when a deployed model is no longer fit-for-purpose.

Reasons for model degradation

1. Concept drift
2. Data drift

Concept drift	Data drift
Relationship between the model inputs and target have changed	Any change that has occurred to the data being fed to the model for predictions as compared to the data that was used for training.

Identifying model deterioration

1. Continuous monitoring of the model's predictive performance over time
2. Assess this performance with the same evaluation metrics used during development

Continuous model evaluation provides a framework to evaluate a deployed mode's performance exclusively on new data. This means we can detect a model's staleness as early as possible. This information helps to do either of the following:

1. Retrain the model
2. Replace the existing model with a new version entirely.

This is done by capturing the following:

1. ground truth
2. prediction inputs and outputs for comparing with ground truth values
3. model versions
4. timestamp of prediction requests

Triggers for retraining

Whether to retrain or not based on an evaluation report depends on what amount of deterioration of performance is acceptable in relation to the cost of retraining.

Setting a higher threshold for model performance ensures a higher-quality model in production but will require more frequent retraining jobs which is costly.

Having a lower threshold on the other hand, would mean more cost effective but there is a chance of having a stale model in production.

Scheduled retraining

Continuous evaluation	Scheduled retraining
May happen everyday	May occur only every week or every month