DEV Community: Senko Rasic

Simple Python and Django logging

Senko Rasic — Sat, 25 Mar 2023 16:55:25 +0000

Introduction

In this article, discuss logging in Python, specifically in the context of Django projects. Logging is essential for understanding the behavior of your application, diagnose problems, and monitor performance. More verbose logging makes it easier to pinpoint problems, but on the other hand, logging everything can lead to so manny log information that it is harder to find actual problems. The right amount of logging is different for every app and developer, but it is almost always more than "none".

In Python, the logging module is a standard way to implement logging functionality. The common pattern for using the logging module is by importing the getLogger function and creating a logger object with the module's name:

from logging import getLogger

log = getLogger(__name__)

log.info("Something interesting happened")

The logging module provides a flexible and powerful framework for capturing log messages in your applications. You can log messages with different levels of severity, such as DEBUG, INFO, WARNING, ERROR, and CRITICAL, depending on the nature of the message. Using the getLogger function, you can create a logger object for each Python module, allowing you to then specify logging level and different handlers for each module.

You can also use the logger functions directly on the logging module. This uses the root logger instead of creating a new one for each module:

import logging
logging.info("Something interesting happened")

This is useful for small scripts, but creating per-module logger is recommended for non-trivial Python applications because it makes it easier to show or handle logs differently based on where they originated.

Basic configuration for Python logging

Basic configuration that just outputs all logs above certain severity level can be setup using basicConfig:

from logging import getLogger, basicConfig, INFO

basicConfig(level=INFO)
log = getLogger(__name__)

# won't show up because DEBUG is less severe than INFO
log.debug("Not terribly interesting")  
# this will be shown in the log
log.info("Something interesting happened")

basicConfig has more options such as customizing the log format or configuring where the logs will be output (if not to console).

Logging exception details

The logger methods support including exception information in the log message, making it easy to provide stack traces helpful for diagnosing issues in your application. When logging a message, you can set the exc_info parameter to True to automatically include the exception information in the log message, like in this example:

import logging
log = getLogger(__name__)
try:
    result = 1 / 0
except ZeroDivisionError:
    logging.error("An error occurred while dividing by zero", exc_info=True)

Simple logging in Django

Django provides a default logging configuration out of the box. It outputs INFO or more severe logs to the console only if DEBUG is set to True . Otherwise, it only sends email to admins on server errors.

In many modern developments, it's preferrable to log everything interesting to console and leave it up to the platform (such as Docker, Systemd or Kubernetes) to take care of gathering and storing the logs. The default configuration can easily be customized by modifying the LOGGING dictionary in the Django settings file.

First, let's create a LOG_LEVEL variable that pulls the desired log level from an environment variable:

import os
import logging

LOG_LEVEL = os.environ.get("LOG_LEVEL", logging.INFO)

Next, update the LOGGING dictionary to output log messages to the console (only showing messages with severity equal to or higher than our configured LOG_LEVEL):

LOGGING = {
    "version": 1,
    # This will leave the default Django logging behavior in place
    "disable_existing_loggers": False,
    # Custom handler config that gets log messages and outputs them to console
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "level": LOG_LEVEL,
        },
    },
    "loggers": {
        # Send everything to console
        "": {
            "handlers": ["console"],
            "level": LOG_LEVEL,
        },
    },
}

Disabling loggers

Sometimes it's useful to disable some loggers. An example in Django is silencing the log error message when the site visitor uses an incorrect host name (ie. not among the ones whitelisted in ALLOWED_HOSTS Django setting). While in debugging this might be useful, it is often an annoyance once you deploy to production. Since the public web is full with automated crawlers checking for vulnerabilities, you'll get a ton of these as soon as you set up a public-facing web app.

Here's a modified logging configuration that works the same as before but ignores these messages:

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "console": {"class": "logging.StreamHandler"},
        # A null handler ignores the mssage
        "null": {"level": "DEBUG", "class": "logging.NullHandler"},
    },
    "loggers": {
        "": {
            "handlers": ["console"],
            "level": ENV_STR("LOG_LEVEL", "INFO"),
        },
        "django.security.DisallowedHost": {
            # Redirect these messages to null handler
            "handlers": ["null"],
            # Don't let them reach the root-level handler
            "propagate": False,
        },
    },
}

Serving static files with Django

Senko Rasic — Mon, 06 Dec 2021 11:12:03 +0000

As you start your Django journey with a new project, Django's runserver helpfully handles your static files for you. But the documentation emphatically tells you not to do that in production: "This method is grossly inefficient and probably insecure, so it is unsuitable for production".

So, what's the best way to do it?

Static and media files

Here's a quick refresher for what I mean by static and media files in the context of Django, as they are handled differently in some cases.

By static files, I mean files that are part of your project (committed to your code repository) and are served as-is to your visitors. These may be images, CSS files, JavaScript files, PDFs, audio or video files - anything that's served as is, but is built-in to your project.

By media files, I mean files that are uploaded by your users and afterward served as is. A typical example are photos uploaded by your users.

Local and cloud storage

Both static files and media files can be stored either locally, on the same server that hosts the Django project, or somewhere else, for example on a cloud storage service.

Back in the day when most projects were deployed on a dedicated or virtual server, most files were stored locally. Today, if using services or tools that treat the deployed server as an immutable (read-only) image, this is no longer possible. Examples of this approach are Heroku and docker. In this case, storing media files locally is impossible, requiring the use of some kind of cloud storage.

Serving via a web server

If your Django instance sits behind a web server such as Nginx or Apache and your files are stored locally, the webserver can serve them directly.

Assuming your static files are located (collected) in /var/app/static (so that's your STATIC_ROOT in Django settings) and served under /static/ prefix (that's your STATIC_URL in Django), here's an example Nginx config to serve them:

location /static/ {
    root /var/app;
}

If you'd also want to serve media files, assuming MEDIA_URL is /media/ and MEDIA_ROOT is /var/app/media, you'd add another block:

location /media/ {
    root /var/app;
}

As an aside: since Nginx appends the URI path to the document root, it's useful to have the STATIC_ROOT end in STATIC_URL and MEDIA_ROOT end in MEDIA_URL - it makes configuration simpler.

For Apache, the configuration would look something like this:

DocumentRoot /var/app;

Alias /static /var/app/static;
Alias /media /var/app/media;

<Directory /var/app>
  Deny from all
</Directory>
<Directory /var/app/static>
  Allow from all
</Directory>
<Directory /var/app/media>
  Allow from all
</Directory>

Note that both examples are minimal and ignore things like compression, setting expiration headers, and so on.

Serving from cloud storage

If you're using cloud storage such as Amazon S3, you probably use the excellent django-storages. It makes serving static files directly from the cloud service easy. In your Django settings file, you just set your STATICFILES_STORAGE (for static files) and DEFAULT_FILE_STORAGE (for media files) to your preferred backend.

Example for Amazon S3:

STATICFILES_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

When you call manage.py collectstatic as part of your deployment procedure, the storages engine will upload all static files to S3. Media files will be uploaded to S3 when you try to save them server-side.

Again, note that this is a minimalistic example - see django-storages documentation for detailed installation instructions and all the options.

Serving static files via whitenoise

If you're serving your media files from a cloud service, you still have the option to serve static files locally. You may want to do that to simplify deployment or rollback procedure or for some other reason.

If you're behind a web server you can use it to serve static files as explained previously. But if you're using docker or deployed to a platform like Heroku, you may not be able to do that. Instead, you can serve them directly from Django, using whitenoise.

Enabling whitenoise is as simple as adding it to the list of middlewares in Django settings:

MIDDLEWARE = [
  'whitenoise.middleware.WhiteNoiseMiddleware',
  # ...
]

Note the placement of middleware matters - be sure to read the docs.

Compression and caching

Both django-storages and whitenoise support compression and manifest storage for your content.

Compressing your static files where it makes sense can dramatically decrease their size and thus load time of your pages. CSS and JavaScript files are good candidates for compression, while images, audio, and video files are already compressed.

If you enable manifest storage, the collectstatic step will rename the files to a unique name. For example, script.js might be renamed to script.f9f204dfa.js, where f9f204dfa is a manifest, or checksum, of the file contents. The manifest changes each time the file contents change, meaning each change to a file will generate a new, unique, name.

This, in turn, means we can tell the browser (and a CDN if you use one) to cache files forever. When you deploy a new version of your project that changes the file, a new file name will be used and the browser (or a CDN) will know to fetch the new file.

An important caveat here is that you must always access the static files using the {% static %} template tag. The template tag will rewrite the name to use the actual (current) manifest. If you hardcode path like script.js in your template, it will not be found.

Use with CDN

All of the above approaches can be used irrespective of whether you're using a content delivery network (CDN) or not. If you do use CDN, using manifest storage will help you even more.