DEV Community: Abby

Powering Anomaly Detection for Industry 4.0

Abby — Mon, 24 Jul 2023 13:00:00 +0000

What is Industry 4.0?

You’ve probably heard the buzz: Industry 4.0 is revolutionizing the way companies manufacture, develop and distribute their products. But, what exactly is Industry 4.0?

To understand the Fourth Industrial Revolution, it helps to remember the first three. The First Industrial Revolution began at the end of the 18th century and focused on mechanizing industrial processes. The Second Industrial Revolution introduced electrification, and the Third Industrial Revolution championed automatization.

The Ford assembly line in 1913; source: Wikimedia commons/public domain

Each of these periods focused on reducing human intervention in industry, but what’s left once factories are mechanized, electrified, and automated? The Fourth Industrial Revolution is all about harnessing the power of data and leveraging it to simulate cognition in industry.

More than anything, Industry 4.0 is a paradigm shift in the way we organize and manage industrial processes to make the most of cyber-physical systems. These smart manufacturing processes include artificial intelligence, machine learning, cloud- and edge-computing, Industrial IoT, distributed computing, augmented reality, and much, much more.

Industrial IoT; image from Connected

What is Anomaly Detection?

One of the most popular ways that artificial intelligence is being incorporated into industrial manufacturing is through automated defect detection or anomaly detection. Anomaly detection is the process of identifying anomalous items in a stream of input data and is a critical component of quality assurance in any production line.

Traditional manual defect detection methods are not only expensive and time-consuming but can also often be ineffective, as not all anomalies are visible to the human eye. However, with the advent of machine learning, computer vision can be leveraged to facilitate human operator work– or even completely automate this process!

Challenges of Anomaly Detection

Anomaly detection faces several unique challenges. It’s often difficult to obtain a large amount of anomalous data, making traditional supervised learning techniques impractical. These class imbalances also mean that popular evaluation metrics like accuracy aren’t relevant.

Furthermore, the difference between a normal sample and an anomalous one can be microscopic. It isn't always feasible to predefine all types of anomalies at the outset of an experiment. Fortunately, to address each of these challenges and more, a team of artificial intelligence researchers at Intel have developed a cutting-edge, easy-to-implement, open-source package called Anomalib.

What is Anomalib?

Anomalib is an open-source deep learning library developed by Intel that makes it easy to benchmark different anomaly detection algorithms on both public and custom datasets, all by simply modifying a config file. As the largest public collection of anomaly detection algorithms and datasets, it has a strong focus on image-based anomaly detection. It’s a comprehensive, end-to-end solution that includes cutting-edge algorithms, relevant evaluation methods, prediction visualizations, hyperparameter optimization, and inference deployment code with Intel’s OpenVINO Toolkit.

Anomaly detection graphics from a Comet + Anomalib experiment by Sid Mehta; used with permission.

So, how does it work? Anomalib uses unsupervised ML techniques to learn an implicit representation of normality with AutoEncoders, GANs, or a combination of both. During inference, new samples are compared against the embeddings of normal samples to determine whether or not they are anomalous. In this way, Anomalib allows you to save your sparse anomalous data for testing purposes only.

Anomalib currently supports ten cutting-edge anomaly detection models, including FastFlow, PaDiM, PatchCore, and CFlow models, but is also continuously updated with the latest state-of-the-art algorithms. You can also train models with custom data or access public datasets like the MVTec or BeanTech datasets through the API. What’s best, Anomalib makes end-to-end anomaly detection possible straight out-of-the-box, and without additional GPUs or super long training times.

Anomalib can do more than just support POC projects, however. In the real world, machine learning is a highly iterative process, and the details of all these iterations can get pretty confusing, pretty fast. To optimize your model and get it production-ready, you’ll need to log, manage, and version these details in an experiment tracking tool. Anomalib recently announced a full integration with Comet, and in this article, we'll explore how to use the two together to power production-grade anomaly detection for Industry 4.0!

Anomaly detection of hazelnuts from the MVTec dataset; image from Anomalib.

Anomalib + Comet for Anomaly Detection

Comet is a powerful tool that allows you to manage and version your training data, track and compare training runs, and monitor your models in production — all in one platform. And now Comet is fully integrated with Anomalib for experiment management, benchmarking, and hyperparameter optimization!

The Comet + Anomalib integration offers the following features:

Autologging and custom logging of experiment- and project-level metrics and features, including system metrics, hyperparameters, graph definition, evaluation metrics, and more.
Organize your project-level dashboard with Comet’s custom panels for an overview that tailors to your team’s specific needs.
Image Panels allow you to compare your images across different experiments and throughout different steps. Search for individual images and showcase selected images across individual experiment runs.
Log benchmarked results to Comet as a means to track model drift.
Isolate the best hyperparameters with HPO powered by the Comet Optimizer.

Auto-logged experiment-level charts in Comet; image by author.

Logging

Experiment-level

In single-experiment view, Comet logs appropriate evaluation metrics in both tabular and chart form. By definition, anomaly detection is a problem of class imbalances, and this makes for poor performance with traditional ML metrics like accuracy, which are designed around an assumption of balanced class distribution.

Experiment-level graphics of a broken bottle from the MVTec dataset in Comet; image by author.

Instead, Anomalib + Comet calculates the F1 score and AUROC at both image and pixel levels. The F1 score combines precision and recall into a single metric by taking their harmonic mean, while still accounting for the precision-recall tradeoff. The AUROC curve describes how well a model can distinguish between classes by plotting a probability curve at various thresholds, or degrees of separability, and is another very important metric for evaluating classification problems with class imbalances.

Formula for F1 score, where TP are true positives, FP are false positives, and FN are false negatives

Pixel-level AUROC and F1 Scores, as auto-logged in Comet; image by author.

But that’s not all! Scrolling through the left-hand sidebar of the single-experiment view, you’ll find that quite a few metrics and features are automatically logged for you! Basic contextual information like source code, system metrics, installed packages, and output are all logged. Experiment-specific hyperparameters, metrics, and graph definitions are also auto-logged, and with a simple edit to the Anomalib config file, you can also log images and other graphics to Comet. Keeping track of all these metrics is essential for producing production-ready models and monitoring them for concept and data drift.

Auto-logged experiment-level hyperparameter tracking in Comet; GIF by author.

Project-level

In the panel-level view, you’ll see charts automatically populated with performance metrics for a bird’s eye view of your project across experiment runs. You can also add Comet’s new Image Panels to this view to visualize specific prediction images across different experiments, as shown below:

Project-level line charts in Comet; image by author.

Project-level image panel in Comet; image by author.

Lastly, it can be really important to see how things are shaping up between two specific experiment runs. Comet allows you to diff selected runs for a more cross-sectional view of your project. This also allows you to compare specific metrics and parameters:

Diffing two selected experiment runs in Comet; GIF by author.

Benchmarking

Anomalib also includes a benchmarking script for relating results across different combinations of models, their parameters, and dataset categories. Log the model performance and throughputs to Comet as a means to track model drift, or export them to a CSV file. You can check out the full documentation here, and once your configuration is decided, perform your benchmarking with one simple command:

python tools/benchmarking/benchmark.py \
--config <relative-or-absolute-path>/<paramfile>.yaml

Hyperparameter Optimization

Anomalib also supports hyperparameter optimization with the Comet Optimizer, making it easier to isolate the right combination of hyperparameters. See here for Anomalib’s HPO docs, or see here for details on other possible configurations with Comet's Optimizer.

At the top of your Comet optimizer report, you’ll find each of your hyperparameters ranked by the evaluation method of your choice, in order of the largest magnitude of Spearman correlation coefficient, to the smallest. In the snapshot below, the learning rate had the largest correlation coefficient with the model’s F1 score.

Comet Optimizer report; image by author.

You’ll also find all of your experiment runs ranked by your evaluation metric, with the additional option to toggle the ranking based on any of the model’s parameters. Lastly, Comet plots your evaluation metrics across all experiment runs in line plots, and feel free to add any of Comet’s publicly available custom panels!

Getting Started

To start tracking your Anomalib projects with Comet, just follow the four quick steps below. Feel free to follow along with this Colab tutorial, and be sure to check out this completed Anomalib project here, courtesy of Comet’s own ML Growth Engineer, Sid Mehta.

0. Setup and Installation

Clone the Anomalib repo into your environment and install the necessary dependencies:

git clone https://github.com/openvinotoolkit/anomalib.git
cd anomalib
pip install . --q

1. Configure Comet Credentials

If you don’t already have a Comet account, you can sign up for free here. Make sure to grab your API key from your account settings so you can configure your Comet credentials in any of several ways. For the sake of simplicity, we'll set them directly through environment variables here:

export COMET_API_KEY = <Your-Comet-API-Key>
export COMET_PROJECT_NAME = <Your-Comet-Project-Name> # this will default to the name of your dataset

2. Modify the Anomalib config File

Next, we’ll need to modify our Anomalib config file to enable logging. The easiest way to do this is to open the existing configuration file at anomalib/anomalib/models/<model-of-your-choice>/config.yaml and adjust the following parameters:

visualization:
    show_images: true
    save_images: true
    log_images: true
    mode: full # options: ["full", "simple"]
logging:
    logger: comet
    log_graph: true

Alternatively, you can copy your particular model's default config template into a new YAML file and adjust the parameters as needed. In this Colab notebook, we’ve also demonstrated how to use pyyaml to write a config file in an interactive environment. Note that each model supported by Anomalib has a different config file structure.

3. Training

By default, !python tools/train.py runs the PaDiM model on the bottle category from the MVTec AD (CC BY-NC SA 4.0) dataset.

To use a different algorithm, just switch out the model name in the config file path to another supported algorithm. To use a custom dataset, just update the relevant Anomalib config file accordingly with the path to your dataset.

python tools/train.py \
--config anomalib/models/<specific-model-name>/<config-file>.yaml

Now just head over to the Comet UI to check out your results!

Conclusion

In the real world, machine learning is a highly iterative process and you’ll likely have many more training runs to visualize and organize. Comet makes it easy to track these runs, share them with members of your team, and collaborate within your organization. By pairing Anomalib with Comet, you can take advantage of all the cutting-edge algorithms of Anomalib, and create production-worthy, maintainable models for your next Industry 4.0 project.

Thanks for making it all the way to the end of this article, and we hope you found this tutorial useful! Feel free to drop any questions or feedback in the comments below, and stay tuned for more content!

Credit Card Fraud Detection With Autoencoders

Abby — Sat, 15 Jul 2023 15:37:45 +0000

In this article, we'll leverage the power of autoencoders to address a key issue for banks and their customers: credit card fraud. Feel free to follow along with the full code tutorial in this Colab and get the Kaggle dataset here.

If you can't wait, check out the final public project here!

Credit Card Fraud

Credit card fraud represents an important, yet complex challenge for banks. In 2021 alone, Nilson estimated that credit card fraud had surpassed $28.5 billion dollars worldwide. And while effective fraud detection efforts, like the use of machine learning and deep learning algorithms, have helped bring about a steady decline in fraud since its peak in 2016, fraud still represents about 6.8% of international credit card transactions.

But fraudulent financial data comes with its own set of unique challenges. Fraudulent transactions represent anomalous data that make up a very small percentage of total transactions. This results in significant class imbalances that make many traditional evaluation metrics, like accuracy, irrelevant.

If you aren't already, you can follow along with the full code in this Colab.

The Data

Our data consists of two days' worth of real bank transactions made by European cardholders in 2013. In this dataset, fraud makes up just 0.172% of all transactions, meaning a model that predicted "no fraud" for every single observation would still achieve an accuracy of over 98%! In the pie chart below, fraudulent transactions are represented by the tiny orange sliver.

Additionally, in order to protect all personal identifiable information (PII) in this dataset, all features (besides Time and Amount) have undergone a PCA transformation. This means that we cannot use any domain knowledge for the purposes of feature engineering or selection.

Data Visualization and Pre-processing

Let's start out by visualizing our data. It would be impossible to plot 30 dimensions, so first we'll apply Scikit-Learn's implementation of t-SNE to the data. t-SNE is a dataset decomposition technique. Here we only plot the first two components with maximum information.

We can observe in this graph that there is little difference between fraudulent and non-fraudulent transactions. Most machine learning models would struggle to classify this data as-is.

First, we'll perform some very basic transformations to the data before feeding it to our model. The original Time feature represents the number of seconds elapsed between each transaction and the first transaction in the data. We'll convert this relative time measure to hour-of-the-day.

We also scale the Time and Amount features, as all other features (V1, V2, … V28) were previously scaled during the PCA transformation. Finally, because we are training an autoencoder to "learn" the embeddings of a normal transaction, we'll subset just normal samples as our training dataset, and use a 50:50 split of normal and fraudulent samples for our validation set. We'll also set aside 250 samples for a test dataset of completely unseen data.

# Convert relative 'Time' measure to hour of day
data["Time"] = data["Time"].apply(lambda x : x / 3600 % 24)

# Scale 'Time' and 'Amount'
data['Amount'] = StandardScaler().fit_transform(data['Amount'].values.reshape(-1, 1))
data['Time'] = StandardScaler().fit_transform(data['Time'].values.reshape(-1, 1))

# Separate fraudulent transactions from normal transactions
norm_data, fraud_data = data[data['Class']==0], data[data['Class']==1]

# Training data will be 2000 normal samples
# Validation data will be 1000 samples, approximately half fraud and half normal samples, shuffled and with indices reset
# Leave out 200 normal samples and 50 fraud samples for a final test dataset of 250 unseen samples; we'll choose a random subsample of these later
train_data = norm_data.iloc[:2000, :]
val_data = pd.concat([norm_data.iloc[2000:2558,:], fraud_data.iloc[:442, :]], axis =0).sample(frac=1).reset_index(drop=True)
test_samples = pd.concat([norm_data.iloc[2558:2608,:], fraud_data.iloc[442:,:]], axis=0).sample(frac=1).reset_index(drop=True)

Attacking Data Imbalance With Autoencoders

Due to the stark class imbalances in this dataset, for this challenge we will use an Autoencoder. Autoencoders learn an implicit representation of normality from the "normal" samples, allowing us to reserve our sparse fraudulent data samples for testing. During inference, new samples are compared against the embeddings of normal samples to determine whether or not they are fraudulent.

Model Training and Inference

We define a simple Autoencoder model with two fully connected linear layers and the Tanh activation function. We'll be using mean squared error as our loss function and an Adam optimizer. We'll log all of this information to Comet, an experiment tracking tool, so we can compare how different experiment runs performed later on.

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(30,14),
            nn.Tanh(),
            nn.Linear(14,7),
            nn.Tanh(),
            )
        self.decoder = nn.Sequential(
            nn.Linear(7, 14),
            nn.Tanh(),
            nn.Linear(14,30),
            nn.Tanh()
            )
    def forward(self,x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

We can quickly visualize this model architecture with the following diagram. Note that we start with 30 input nodes, then encode this information down to seven nodes, and then decode back to 30 output nodes.

Next, we define our training and validation loops. Note that we create our val_predictions table by defining a pandas DataFrame within our validation loop and appending the relevant columns. We'll use this DataFrame to examine sample-level metrics in Comet after inference.

for epoch in range(hyper_params['epochs']):
    losses = []
    for (data, _) in train_loader:
        # ===================forward=====================
        output = model(data)
        loss = criterion(output, data)     
        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        losses.append(loss.detach().cpu().numpy())
    # ===================log========================
    experiment.log_metric('epoch_loss', np.mean(losses), epoch = epoch+1)

    losses = []
    labels = []
    preds = []
    embeds = []

    for data, label in val_loader:
        # ===================forward=====================
        reconstructed = model(data)
        loss = torch.sigmoid(criterion(reconstructed, data)).item()
        losses.append(round((loss),6))
        labels.append(label.item())
        preds.append(reconstructed[0][-1].item())
        embeds.append(reconstructed[0][:-1].detach().cpu().numpy())
    # ===================log========================  
    experiment.log_metric('val_loss', np.mean(losses), epoch = epoch +1)
    if epoch == (hyper_params["epochs"] - 1):
      print("logging table")
      df = pd.DataFrame()
      df['Reconstruction_Loss'] = losses
      df['Labels_gt'] = labels
      df['Labels_preds']= [int(x >= hyper_params['threshold']) for x in losses]
      experiment.log_table('val_predictions.csv', df)
      print('complete')

Precision-recall Curve vs Threshold Values

Now that we've run training and validation loops, let's examine the precision-recall tradeoff of our model at various threshold levels and plot the reconstruction error against our chosen threshold. To see how we defined our custom plotting functions, see the Colab here.

From the plot above, we can see the optimal threshold value for fraudulent transactions is around 0.75. Now let's plot the precision-recall tradeoff of the model with the threshold set to this value.

Visualizing Reconstruction Error

Finally, let's visualize the reconstruction error with the threshold level applied to see how well it separates fraudulent transactions from non-fraudulent transactions.

Although some outliers are still misclassified, the threshold separates the vast majority of the reconstructed labels. Note that we used a 50:50 split of fraudulent and non-fraudulent data for the validation split.

To get a better understanding of how our autoencoder has learned to distinguish between the two classes, let's visualize plots of the raw data (before encoding) and the transformed data (after reconstruction). Here again, we'll apply t-SNE to each dataset and plot the first two components with maximum information.

Clearly the data on the right is much easier to separate (and therefore classify), than the data plotted on the left. It would be very difficult for a machine learning algorithm to classify the data on the left at all, so our autoencoder has proven very helpful.

Classification Report and Evaluation

Let's also take a look at the classification report of our validation data. Note that although the original dataset was highly imbalanced, we made a validation set that was roughly equally split between normal and fraudulent transactions. This gives our model a better opportunity to predict both classes, and also makes the evaluation metrics much more relevant.

With a precision, recall, and F1-score of 0.91, our model has done well! But can we do any better? Let's take a look at the data we logged to Comet to identify where our model might be improved. We can also check out how to monitor testing data statistics as our model gets a chance to look at new, unseen data.

Debugging Our Model With Data Panels

Once we've run training and inference, we can head over the Comet UI to check out our Data Panels. Follow along with my public project here and feel free to experiment with customization options like join types, column order, and more. We'll also take a peek at how to filter and sort the Data Panel to identify misclassified samples for retraining.

Data Panels are interactive so we can filter samples based on the ground truth labels and compare them to the model's predictions. Since not detecting fraudulent cases when they are actually occurring could potentially cost financial institutions a lot of money, we specifically want to look at instances where there was fraud, but the model predicted there wasn't (false negatives).

We can also gather all instances where the model predicted there was fraud but there wasn't (false positives). By re-training the autoencoder on these problematic samples, we can improve the model's representation of "normality" for future runs.

We can also track how prediction columns are changing for the same samples across experiments. By concatenating the tables via columns, we can see side-by-side how our predictions compare to previous runs.

Finally, we can also compare distribution shifts in the input data. If we were using different versions of data across experiments or running experiments over time, it would be a good idea to log some basic summary statistics (like those calculated with pd.df.describe) of the input data. This will allow us to quantify the statistical differences between data and whether this is having a substantial impact on model performance.

# We take 10 random samples from the unseen test set we defined earlier
test_data = test_samples.sample(10).reset_index(drop=True)

# Log summary statistics of test data (only 5 columns, round to 4 decimal places)
experiment.log_table('input_statistics.csv', test_data.describe().iloc[:,[0, 1, 2, 29, 30]].round(4))

Conclusion

Thanks for making it all the way to the end and we hope you found this tutorial useful! To recap everything we've accomplished, we:

Loaded a notoriously imbalanced dataset
Constructed a simple Autoencoder model
Plotted reconstruction labels and defined a classification threshold
Calculated validation metrics
Logged sample and project-level data to Comet, an experiment tracking tool
Examined specific instances where the model struggled and identified areas for improvement

For questions, comments, or feedback, feel free to drop a note in the comments below. Happy coding!

DEV Community: Abby

Powering Anomaly Detection for Industry 4.0

What is Industry 4.0?

What is Anomaly Detection?

Challenges of Anomaly Detection

What is Anomalib?

Anomalib + Comet for Anomaly Detection

Logging

Experiment-level

Project-level

Benchmarking

Hyperparameter Optimization

Getting Started

0. Setup and Installation

1. Configure Comet Credentials

2. Modify the Anomalib config File

3. Training

Conclusion

Credit Card Fraud Detection With Autoencoders

Credit Card Fraud

The Data

Data Visualization and Pre-processing

Attacking Data Imbalance With Autoencoders

Model Training and Inference

Precision-recall Curve vs Threshold Values

Visualizing Reconstruction Error

Classification Report and Evaluation

Debugging Our Model With Data Panels

Conclusion

References and Resources