Machine Learning models can perform well when they are trained, but that does not means the input data will remain the same over time.
As real world data changes, a model can receive data that looks very different from the data it was trained on. This is known as data drift.
For this Project, I built ChurnWatch, a small MLOps project using AWS to demonstrate how data drift can be monitored after model training.
The Goal
Train a model -> compare new data with training data -> detect drift -> monitor the result.
The Problem
A company may train a model using historical customer information to predict whether a customer is likely to leave the service.
Overtime, customer may spend more or less than before
customers may spend more or less than before
the average number of support calls may change
overall customer data may change
Even if model itself has not changed, the data going into it may no longer resemble the data used during training and this affects model reliability. ChurnWatch focuses on detecting this change.
- S3 is used to train the churn prediction model
- Sagemaker is used to train the churn prediction model
- A python based drift detection script compares the baseline training dara with stimulated production data using Population Stability Index (PSI)
- The resulting drift value is published to CloudWatch so that it can be monitored.
The dataset is divided into a baseline/training dataset and simulated production dataset
The production dataset is intentionally generated with changed feature distributions so that the monitoring system has an actual drift scenario to detect.
Training the model
The first step is training a simple churn prediction model using Amazon Sagemaker.
The training data is stored in Amazon S3, and Sagemaker trains the machine learning model using the historical customer data.
Detecting Data Drift
For drift detection, ChurnWatch uses Population Stability Index (PSI)
PSI compares the distribution of a feature in the baseline dataset
and new dataset.
A simplified interpretation is:
PSI < 0.10 -little or no significant change
0.20 - significant drift
For the experiment, production data was intentionally generated with different distributions.
The drift detection script calculates PSI for the selected features and identifies the maximum drift value.
for example :
Maximum PSI:0.xx
Threshold : 0.2
Result: Drift detected
The exact value depends on the generated production dataset.
Monitoring with CloudWatch
Once the drift calculation is complete, the resulting metric is published to Amazon CloudWatch
this provides a simple monitoring layer for the ML workflow, instead of manually checking the drift every time, the drift value can be viewed as a CloudWatch metric
What I learned
This Project helped me understand aan important difference between building an ML model and operating an ML system.
Training a model is only one part of the lifecycle, once a model is used with real-world data, the data itself needs to be monitored.
With ChurnWtach , I implemented a small workflow covering:
Dataset management with Amazon S3
Model training with Amazon Sagemaker
Data drift detection using PSI
Metric monitoring with CloudWatch
A model can remain unchanged while data around it changes
Monitoring that change is an important part of maintaining machine learning systems in production.
Top comments (0)