Detecting Anomalies with .NET and ML.NET: A Practical Guide

Introduction

Ever wondered how to find a needle in a haystack or detect when your data decides to throw a party? Anomaly detection can help you spot those unexpected patterns that might indicate system failures, financial fraud, or even just a misbehaving sensor. In this article, we’ll explore how to use ML.NET to integrate anomaly detection into your .NET applications. Let's dive in and make sure your data behaves!

What is Anomaly Detection?

Anomaly detection is the practice of identifying data that doesn't behave like the rest of its peers. Imagine finding a cat at a dog show – that's an anomaly. Whether you're monitoring systems, securing financial transactions, or ensuring quality control, spotting these outliers is crucial.

ML.NET and Anomaly Detection

ML.NET is a machine learning library developed by Microsoft that allows you to integrate AI models directly into .NET applications. With ML.NET, you can create, train, and use machine learning models without leaving the .NET environment. Think of it as having a trusty assistant to help you keep your data in check.

Setting Up the Environment

First, let’s set up the necessary environment to create our anomaly detection project.

Step 1: Install the .NET SDK
Make sure you have the .NET SDK installed. If you don’t have it yet, download it here. Trust us, it’s easier than assembling IKEA furniture.

Step 2: Create a New Project
Open Visual Studio or use the .NET CLI to create a new console project:

dotnet new console -n MLNetDemo
cd MLNetDemo

Step 3: Add the Necessary Packages
Add the ML.NET and ML.NET TimeSeries packages to your project:

dotnet add package Microsoft.ML
dotnet add package Microsoft.ML.TimeSeries

Implementing Anomaly Detection

Let's create a practical example of anomaly detection in sensor data. Don’t worry, we won’t leave you hanging!

Step 4: Create the Data Classes
Define the SensorData and AnomalyPrediction classes to represent the input data and the predictions.

public class SensorData
{
    [LoadColumn(0)]
    public DateTime Time { get; set; }

    [LoadColumn(1)]
    public float Value { get; set; }
}

public class AnomalyPrediction
{
    [VectorType(3)]
    public double[] Prediction { get; set; }
}

Explanation:

SensorData:

This class represents the input data containing the time (Time) and sensor value (Value). Think of it as the guest list for your data party.

AnomalyPrediction:

This class represents the anomaly prediction returned by the model. The Prediction property is a vector of three elements containing information about the anomaly detection. It’s like your bouncer at the party, deciding who’s out of place.

Step 5: Implement the Main Code
Create the Program.cs file with the following code:

using Microsoft.ML;

class Program
{
    static void Main(string[] args)
    {
        var context = new MLContext();

        // Simulated data for demonstration purposes
        var data = new List<SensorData>
        {
            new SensorData { Time = DateTime.Parse("2024-03-08 00:00"), Value = 1.0f },
            new SensorData { Time = DateTime.Parse("2024-03-08 01:00"), Value = 1.1f },
            new SensorData { Time = DateTime.Parse("2024-03-08 02:00"), Value = 1.0f },
            new SensorData { Time = DateTime.Parse("2024-03-08 03:00"), Value = 1.2f },
            new SensorData { Time = DateTime.Parse("2024-03-08 04:00"), Value = 1.1f },
            new SensorData { Time = DateTime.Parse("2024-03-08 05:00"), Value = 5.0f }, // Anomaly
            new SensorData { Time = DateTime.Parse("2024-03-08 06:00"), Value = 1.1f },
            new SensorData { Time = DateTime.Parse("2024-03-08 07:00"), Value = 1.0f },
            new SensorData { Time = DateTime.Parse("2024-03-08 08:00"), Value = 1.2f },
            new SensorData { Time = DateTime.Parse("2024-03-08 09:00"), Value = 1.0f }
        };

        var dataView = context.Data.LoadFromEnumerable(data);

        var pipeline = context.Transforms.DetectSpikeBySsa(
            outputColumnName: nameof(AnomalyPrediction.Prediction),
            inputColumnName: nameof(SensorData.Value),
            confidence: 95.0,
            pvalueHistoryLength: data.Count / 2,
            trainingWindowSize: data.Count,
            seasonalityWindowSize: data.Count / 4);

        var model = pipeline.Fit(dataView);
        var transformedData = model.Transform(dataView);

        var predictions = context.Data.CreateEnumerable<AnomalyPrediction>(transformedData, reuseRowObject: false).ToList();

        Console.WriteLine("Time\t\tValue\tAnomaly");
        for (int i = 0; i < predictions.Count; i++)
        {
            var sensorData = data[i];
            var prediction = predictions[i];
            var isAnomaly = prediction.Prediction[0] == 1;
            Console.WriteLine($"{sensorData.Time}\t{sensorData.Value}\t{isAnomaly}");
        }
    }
}

Explanation:

Creating the ML Context:

var context = new MLContext();

Creates a new ML.NET context, which is necessary for all ML.NET operations. It’s like our command center for ML tasks.

Creating Simulated Data:

var data = new List<SensorData>
{
    new SensorData { Time = DateTime.Parse("2023-01-01 00:00"), Value = 1.0f },
    new SensorData { Time = DateTime.Parse("2023-01-01 01:00"), Value = 1.1f },
    // ...
};

Simulates some sensor data that includes an anomalous value. It’s like our data party where one guest decided to go wild!

Loading the Data:

var dataView = context.Data.LoadFromEnumerable(data);

Converts the list of data into an IDataView, which is the data format used by ML.NET. Think of it as rolling out the red carpet for our data.

Configuring the Transformation Pipeline:

var pipeline = context.Transforms.DetectSpikeBySsa(
    outputColumnName: nameof(AnomalyPrediction.Prediction),
    inputColumnName: nameof(SensorData.Value),
    confidence: 95.0,
    pvalueHistoryLength: data.Count / 2,
    trainingWindowSize: data.Count,
    seasonalityWindowSize: data.Count / 4);

• Configures the transformation pipeline to detect spike anomalies using the DetectSpikeBySsa method.
• confidence: Confidence level for anomaly detection.
• pvalueHistoryLength: Length of the p-value history for detection.
• trainingWindowSize: Training window size.
• seasonalityWindowSize: Seasonality window size.

Training the Model:

var model = pipeline.Fit(dataView);

Trains the model using the provided data. It’s like teaching our bouncer what an unruly guest looks like.

Transforming the Data:

var transformedData = model.Transform(dataView);

Applies the trained model to the data to make predictions.

Extracting Predictions:

var predictions = context.Data.CreateEnumerable<AnomalyPrediction>(transformedData, reuseRowObject: false).ToList();

Converts the predictions into a list for analysis. Time to see who gets kicked out of the party!

Displaying the Results:

Console.WriteLine("Time\t\tValue\tAnomaly");
for (int i = 0; i < predictions.Count; i++)
{
    var sensorData = data[i];
    var prediction = predictions[i];
    var isAnomaly = prediction.Prediction[0] == 1;
    Console.WriteLine($"{sensorData.Time}\t{sensorData.Value}\t{isAnomaly}");
}

Iterates over the predictions and displays whether there is an anomaly for each data point.

Conclusion

With ML.NET, integrating artificial intelligence into your .NET applications becomes simple and accessible. Anomaly detection is a powerful tool for identifying unusual patterns in your data, helping to prevent failures, fraud, and more. We hope this guide has shown you how to start using ML.NET for anomaly detection in time series data. Give it a try ;)

To get the project running, just download it from my repository by clicking here