DEV Community

umapyoi
umapyoi

Posted on

Performance of anomaly detection cloud platform with MVTec dataset

I verified the performance of the anomaly detection model automatic creation platform “ADFI” written in the previous article!

See the previous article for ADFI.

Screen of ADFI:

Image description

Experimental settings

Dataset

For the verification experiment, I used the MVTec dataset, which is a very famous image dataset for anomaly detection.

It contains datasets of 15 categories that are frequently used in verification experiments of anomaly detection related papers.

Dataset image example:

Image description

The number of data

Since the number of normal images and abnormal images differs depending on the data set, the below is used for each data set.

Training images were randomly extracted from the dataset. Test images are randomly extracted from images not used in Training images.

Normal training data: 50

Anomaly training data: 10

Normal test data: 20

Anomaly test data: 20

Verification items

  • Learning time
    Time taken to train the model

  • Test time
    Time from running the test to getting the results of all test data (40 images)

  • AUC (Area Under the Curve)
    An evaluation index often used for anomaly detection.
    ROC (Receiver Operating Characteristic) The area corresponding to the lower part of the curve.
    The closer the AUC is to 1, the higher the performance of the model. (If predicted completely randomly, the AUC will be 0.5.)
    I downloaded the score result CSV and calculated the AUC.

Experimental results

The table below shows the results of creating a deep distance learning (DML) model with ADFI for all datasets.

Image description

The average learning time is about 11 minutes.

The average test time is about 4 seconds.

The average AUC is 0.947.
Very good results with AUC above 0.9 for datasets other than Cable and Screw.

Performance comparison with methods of deep metric learning

I compared the AUC of the model of ADFI and the AUC of the model created by each method of deep metric learning without using ADFI.

AUCs of each method in the MVTec dataset:

Image description

It is not possible to simply give a superiority or inferiority because the experimental settings are different.

But the AUC values of ADFI are the highest in many datasets.

See also the previous article for ADFI.

https://dev.to/umapyoi/image-anomaly-detection-how-to-create-models-quickly-and-easily-i9l

Top comments (0)