Scraping Custom Django Metrics with Prometheus

In the previous article, we explored setting up a Prometheus instance to scrape generic data(metrics) from our very basic Django application.

Now, we're taking it a step higher: We're going to send custom metrics to Prometheus so we can visualize the data.

To follow along with this article, you need:

• To have Prometheus installed

• To understand the basics of Prometheus. For that, refer to the previous entries in this series

• A basic Django application set up for you to work with. You can refer to Django's official documentation for this.

Using the Python Logging module

Python already has a module for logging various information in an application. We will build on this module to generate our custom logs, which will be exposed to Prometheus for scraping.

In your settings.py add the following lines:

import os # Add this at the top of the file
.
.
.

# Add this at the bottom of the file
LOGGING = {
    'version': 1,  # Specifies the logging configuration schema version
    'disable_existing_loggers': False,  # Keep default Django loggers active
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {message}',
            'style': '{',  # Use `{}` to format log messages
        },
    },
    'handlers': {
        'file': {  # Write logs to a file
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': os.path.join(BASE_DIR, 'app.log'),  # Log file path
            'formatter': 'verbose',  # Use the verbose format
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'INFO',  # Log everything INFO and above
            'propagate': True,  # Pass log messages to parent loggers
        },
    },
}

This defines a log handler that will write information into an app.log file.

To read more about log levels, refer to this documentation

Create Custom Prometheus Log Handler

In a new customlogger.py file, add the following code:

import logging
from prometheus_client import Counter

# Define Prometheus counters for different log levels
log_counters = {
    'INFO': Counter('django_log_info_total', 'Total number of INFO logs'),
    'WARNING': Counter('django_log_warning_total', 'Total number of WARNING logs'),
    'ERROR': Counter('django_log_error_total', 'Total number of ERROR logs'),
}

class PrometheusLogHandler(logging.Handler):  # Custom log handler
    def emit(self, record):
        log_level = record.levelname  # Get the log level (INFO, WARNING, ERROR)
        if log_level in log_counters:  # Check if we have a counter for this level
            log_counters[log_level].inc()  # Increment the counter

Expose Metrics to Prometheus

We will create a Django endpoint(Django view) for Prometheus to access the data from.

app/views.py:

from prometheus_client import generate_latest
from django.http import HttpResponse

def metrics_view(request):
    """Expose Prometheus metrics, including log counters."""
    metrics = generate_latest()  # Generate all Prometheus metrics
    return HttpResponse(metrics, content_type='text/plain')

yourdjangoproject/urls.py:

from django.contrib import admin
from django.urls import path, include
from demo.views import metrics_view

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('django_prometheus.urls')),
    path('metrics/', metrics_view, name='metrics'),
]

prometheus.yml:

scrape_configs:
  - job_name: 'django_app'  # A name for this job
    static_configs:
      - targets: ['localhost:8000']  # Django app's address

This will configure Prometheus to look for data at the address localhost:8000. Alternatively, you can use 127.0.0.1:8000

Finally, we update our settings.py to include the Custom handler we created:

from demo.customlogger import PrometheusLogHandler
.
.
LOGGING = {
    'version': 1,  # Specifies the logging configuration schema version
    'disable_existing_loggers': False,  # Keep default Django loggers active
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {message}',
            'style': '{',  # Use `{}` to format log messages
        },
    },
    'handlers': {
        'file': {  # Write logs to a file
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': os.path.join(BASE_DIR, 'app.log'),  # Log file path
            'formatter': 'verbose',  # Use the verbose format
        },
        'prometheus': {  # Send logs to Prometheus (custom handler)
            'level': 'INFO',
            'class': 'my_app.log_handlers.PrometheusLogHandler',  # Our custom handler
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file', 'prometheus'],  # Use both file and Prometheus handlers
            'level': 'INFO',  # Log everything INFO and above
            'propagate': True,  # Pass log messages to parent loggers
        },
    },
}

Putting it all together

Now we test what we have:

python manage.py runserver

Inside the Prometheus directory:

./prometheus --config.file=prometheus.yml 

On your browser, open 127.0.0.1:8000/metrics to confirm that the metrics are being generated.

Navigate to the Prometheus UI at 127.0.0.1:9090. Query the log metrics by searching for django_log_info_total and click on the Graph view.

What we've done so far

1. Enabled logging in our Django application using the logging module

2. Created a Custom logging handler to increment the Prometheus counter whenever a log is recorded

3. Exposed the Django logs to Prometheus and visualized it in Prometheus UI

For the full implementation, you can check out my repository