Writing Custom Log Handlers In Python

Have you ever wondered how logging services like logz .io or loggly .com ship your logs, onto their platform? For example logz .io has an opensource logging handler which they use to transport logs from your application to their platform where further analysis are carried out on these logs to enable you derive substantial insights on your application's performance. In this short article we are briefly going to cover the basics of writting user-defined log handlers using the python standard logging package.

You might be wondering why i chose to write about this rather "odd" topic, but the importance of a good logging system cannot be overstated, whether your application is a web application, mobile or even an operating system, logging enables you to derive useful informations on how your application is performing. A good logging system helps in catching errors as they happen, instead of relying on users to report them, the majority of whom simply wouldn’t or wouldn’t know how to do it.

The logging.StreamHandler

The python standard logging package provides a Handler class which basically defines how a particular log message is handled, we can configure a simple handler that prints log messages to the terminal like this:

LOGGING = {
    "version": 1, 
    "disable_existing_loggers": False,
    "formatters": {
        "stdformatter": {"format": "DateTime=%(asctime)s loglevel=%(levelname)-6s  %(funcName)s() L%(lineno)-4d %(message)s call_trace=%(pathname)s L%(lineno)-4d"},
    },
    "handlers": {
        "stdhandler": {
            "class": "logging.StreamHandler",
            "formatter": "stdformatter",
            'stream': 'ext://sys.stdout'
        },
    },
    "loggers" : {
        "root": {
            "handlers": ["stdhandler"], 
            "level": "DEBUG",
            "propagate": True
            }
        }
}

We can then initialize this configuration, in a log.py assuming the log.py file is the application we want to log

import logging
from logging.config import dictConfig


dictConfig(LOGGING)
logger = logging.getLogger()
logger.debug("logger started")

def logToConsole():
    logger.info("Application started, succesfully")
    logger.info("Logging to the console")
    logger.info("Finished logging to console!")

# call the function
logToConsole()

When we run the log.py file python3 log.py; we can see a similar output to this shown on the terminal

As you can see we've been able to configure a handler that logs directly to the console(stdout), keep in mind you don't need to create any congifuration to log to the terminal, since the StreamHandler is the default handler for the logging package. The logging package also comes with an array of already made Handlers you can use asides the StreamHandler, some of which are listed below:

• FileHandler: which sends logging output to a disk file
• NullHandler: It is essentially a ‘no-op’ handler for use by library developers. Which i've never had any reason to use :)
• BaseRotatingHandler: which is basically another type of FileHandler for special use cases
• SocketHandler: which sends logging output to a network socket.
• SysLogHandler: which sends logging messages to a remote or local Unix syslog.
• SMTPHandler: which sends log messages to an email address, using SMTP.
• HTTPHandler: which supports sending logging messages to a web server, using either GET or POST semantics.

There are several handlers provided by the logging package that can be found here, some of which you might never have a use case for since they are built for very specific use cases.

Custom Handlers

Supposing we are building a cloud based logging as a service software like logz.io, where we'd like to store, retrieve and analyze our logs for better insights; python provides a low-level "api" to enable us transports logs from python applications easily, using the logging.Handler class. In the following example we are going to create a simple Custom Handler that writes to a ".txt" file. Obviously log messages are not written to txt files but this is to illustrate how to write Custom Log Handlers in a simple way.

class SpecialHandler(logging.Handler):

    def __init__(self )-> None:
        self.sender = LogSender()
        logging.Handler.__init__(self=self)

    def emit(self, record) -> None:
        self.sender.writeLog(record)

In the code above we declare a class named SpecialHandler, which inherits from the logging.Handler base class, this basically allows us to extend the base Handler so as to make it fit our use. We then declare the __init__ method and initiates the class which handles each log record as it is retrieved i.e LogSender, the emit(self, record) method is quite special as it responsible for sending each log record to the specified reciever.

 class LogSender:
    def __init__(self) -> None:
        pass

    def writeLog(self, msg: logging.LogRecord) -> None:
        with open("application.txt",'a',encoding = 'utf-8') as f:
            f.write(f"{msg} \n")

The LogSender class acts as the retriever of log records as they are sent from the emit method of the Handler's class; the LogSender class declares a writeLog method, which accepts any argument of type msg: logging.LogRecord. On reception of a record it opens a txt file application.txt in append mode and writes the log message to the file, adding a new line at the end of it.
This is basically all it takes to create a customHandler, we can configure our LOGGING Dictionary to recognize this new handler with a few lines of code, like so:

LOGGING = {
    "version": 1, 
    "disable_existing_loggers": False,
    "formatters": {
        "stdformatter": {"format": "DateTime=%(asctime)s loglevel=%(levelname)-6s  %(funcName)s() L%(lineno)-4d %(message)s call_trace=%(pathname)s L%(lineno)-4d"},
    },
    "handlers": {
        "stdhandler": {
            "class": "logging.StreamHandler",
            "formatter": "stdformatter",
            'stream': 'ext://sys.stdout'
        },
        "specialhandler":{
            "class" : "writer.SpecialHandler",
            "formatter": "stdformatter",
        }
    },
    "loggers" : {
        "root": {
            "handlers": ["stdhandler", "specialhandler"], 
            "level": "DEBUG",
            "propagate": True
            }
        }
}

Keep in mind, the writer.SpecialHandler is our custom handler class written in a writer.py file. To initialize this changes we can import the writer.py file into our log.py file and load the logging configuration into dictConfig again, but this time with a custom handler that writes to our application.txt file.

import logging
from logging.config import dictConfig
import writer #writer.py contains our SpecialHandler and LogSender class respectively

LOGGING = {
    "version": 1, 
    "disable_existing_loggers": False,
    "formatters": {
        "stdformatter": {"format": "DateTime=%(asctime)s loglevel=%(levelname)-6s  %(funcName)s() L%(lineno)-4d %(message)s call_trace=%(pathname)s L%(lineno)-4d"},
    },
    "handlers": {
        "stdhandler": {
            "class": "logging.StreamHandler",
            "formatter": "stdformatter",
            'stream': 'ext://sys.stdout'
        },
        "specialhandler":{
            "class" : "writer.SpecialHandler",
            "formatter": "stdformatter",
        }
    },
    "loggers" : {
        "root": {
            "handlers": ["stdhandler", "specialhandler"], 
            "level": "DEBUG",
            "propagate": True
            }
        }
}

dictConfig(LOGGING)
logger = logging.getLogger()
logger.debug("logger started")

def logToConsole():
    logger.info("Application started, succesfully")
    logger.info("Logging to the console")
    logger.info("Finished logging to console!")

# call the function
logToConsole()

When we run python3 log.py we can see an application.txt file is created within the current directory, containing all the log records.

Conclusion

Logging applications can save us hours in debugging hidden bugs within our application, as well as enabling to derive useful insights about the performance, operatability and needs of applications without any hassles. In this short article, we've seen how to write basic custom log handlers for our applications, extending to fit our specific needs.

Let me know what you think about this article and possibly any amendments that can be made to improve it's user experience, thanks for reading and have a great time!