DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

Cover image for Metrics Monitoring App with Anomaly Detection using MongoDB
madalinfasie
madalinfasie

Posted on • Updated on

Metrics Monitoring App with Anomaly Detection using MongoDB

Overview of My Submission

Schedule metric collectors to gather measurements from different external services and optionally run anomaly detection on the new values.

The project uses MongoDB as a storage system, storing data as time series entries.

To make the experience complete, a visualization tool is needed. Fortunately, Mongo gives us Mongo Charts which can be used directly from Mongo Atlas.

Submission Category:

Prime Time

Link to Code

The code can be found here:

Metric Monitoring App Using Mongo Time Series

Description

Schedule metrics collectors for different services and run anomaly detection on new values.

Getting started

Prerequisites

Make sure you have Docker and Docker Compose installed on your machine.

Create a MongoDB Atlas cluster with MongoDB 5.0 or above installed.

For security reasons, the .env was not included in the repository. The available configuration variables are provided inside .env.sample. Make sure to set all the configuration variables before running the app.

Running the app

Running this app is as easy as running:

$ docker-compose build
$ docker-compose up

Overview

Starting with the docker-compose.yml file, we can see that this application is split into a main webapp service and the scheduling service (celery in this case).

The metrics collection is done mainly by the celery workers. The metrics_web service providing only a way of pushing metrics from external service (when scheduling is…

Click above to read more.

Additional Info

To test the project, I created three metrics, two of them are randomly generated (as their name suggests) and another one is the number of Dev.to posts in the last 15 minutes (I chose 15 minutes just so I don't have to wait so long for the data).

In the first screenshot, we can see the number of posts in the last 15 minutes and anther graph, plotting the values returned by random.randrange(1, 100)

Dev.to Articles published last 15 minutes and a random metric

In the second screenshot, we can see the second randomly generated metric (on the right) and a plot of the anomaly prediction (on the left). It is considered an anomaly if the score is -1.

As you can see, due to the metric's randomness and few data points, the anomaly prediction is not giving us any real insight.

A second randomly generated metric and the anomaly detection plot

Built with

Flask: Flask is a lightweight WSGI web application framework

MongoDB: An open-source NoSQL database developed for high performance, high availability, and easy scalability. It also provides Time Series collections.

Celery: A task queue with focus on real-time processing, while also supporting task scheduling.

Docker: An open platform for developing, shipping, and running applications

scikit-learn: Simple and efficient tools for predictive data analysis

Top comments (0)

12 Gorgeous UI Components for Your Design Inspiration

12 Gorgeous UI components for your design inspiration: cards, text, buttons, checkboxes, icons, loaders and menus.