ChurnWatch - MLOps Data Drift Monitoring

rasi321 — Tue, 13 Oct 2026 09:52:05 +0000

Machine Learning models can perform well when they are trained, but that does not means the input data will remain the same over time.

As real world data changes, a model can receive data that looks very different from the data it was trained on. This is known as data drift.

For this Project, I built ChurnWatch, a small MLOps project using AWS to demonstrate how data drift can be monitored after model training.

The Goal
Train a model -> compare new data with training data -> detect drift -> monitor the result.

The Problem
A company may train a model using historical customer information to predict whether a customer is likely to leave the service.
Overtime, customer may spend more or less than before
customers may spend more or less than before
the average number of support calls may change
overall customer data may change

Even if model itself has not changed, the data going into it may no longer resemble the data used during training and this affects model reliability. ChurnWatch focuses on detecting this change.

S3 is used to train the churn prediction model
Sagemaker is used to train the churn prediction model
A python based drift detection script compares the baseline training dara with stimulated production data using Population Stability Index (PSI)
The resulting drift value is published to CloudWatch so that it can be monitored.

The dataset is divided into a baseline/training dataset and simulated production dataset

The production dataset is intentionally generated with changed feature distributions so that the monitoring system has an actual drift scenario to detect.

Training the model
The first step is training a simple churn prediction model using Amazon Sagemaker.
The training data is stored in Amazon S3, and Sagemaker trains the machine learning model using the historical customer data.

Detecting Data Drift
For drift detection, ChurnWatch uses Population Stability Index (PSI)
PSI compares the distribution of a feature in the baseline dataset
and new dataset.

A simplified interpretation is:

PSI < 0.10 -little or no significant change
0.20 - significant drift

For the experiment, production data was intentionally generated with different distributions.
The drift detection script calculates PSI for the selected features and identifies the maximum drift value.

for example :
Maximum PSI:0.xx
Threshold : 0.2
Result: Drift detected

The exact value depends on the generated production dataset.

Monitoring with CloudWatch
Once the drift calculation is complete, the resulting metric is published to Amazon CloudWatch
this provides a simple monitoring layer for the ML workflow, instead of manually checking the drift every time, the drift value can be viewed as a CloudWatch metric

What I learned
This Project helped me understand aan important difference between building an ML model and operating an ML system.
Training a model is only one part of the lifecycle, once a model is used with real-world data, the data itself needs to be monitored.
With ChurnWtach , I implemented a small workflow covering:
Dataset management with Amazon S3
Model training with Amazon Sagemaker
Data drift detection using PSI
Metric monitoring with CloudWatch

A model can remain unchanged while data around it changes
Monitoring that change is an important part of maintaining machine learning systems in production.

Getting Started with MLOps: ML models and Amazon Sagemaker

rasi321 — Mon, 12 Oct 2026 14:16:09 +0000

A machine learning model is more than just training an algorithm on data.

Once a model is trained, how do we deploy it, make predictions, monitor its performance, and update it when needed?

This is where MLOps comes in:

Machine Learning: Building and training models using data.
MLOps: Managing the ML lifecycle, from expermientation and training to deployment and monitoring
Amazon Sagemaker: Amazon Sagemaker is an AWS machine learning service that provides tools and infrastructure to build, train, deploy, and manage ML models

A simple way to understand the workflow:

Data -> Model training -> Model Evaluation -> Deployment -> Monitoring

I am starting by exploring the fundamental of MLOps and how Amazon Sagemaker helps take ML models from experimentation to production.

DEV Community: rasi321

ChurnWatch - MLOps Data Drift Monitoring

Getting Started with MLOps: ML models and Amazon Sagemaker