Key Points for Logging with Python's logging Module

Hajime Fujita — Fri, 11 Jul 2025 06:35:48 +0000

I Should Have Adopted This Sooner

After building a piece of software of a certain size, I thought, "I should probably enhance the logging functionality for operational purposes," and began to look into Python's logging module.

Once I had a grasp of it, I was stunned to realize, "If only I'd implemented this sooner, debugging would have been so much easier...!"

In this article, I'd like to summarize what I've learned about logging and share some of the key strategies I've devised.

When Is It Time to Graduate from print?

In any programming language, printing the value of a variable is a fundamental debugging technique, and Python is no exception. If you're just running small test programs, print is likely sufficient.

However, if any of the following apply to you, it might be time to consider graduating from print and using logging instead.

You can't expect a detailed explanation from the field when a bug occurs in production.
You want to output different logs based on complex conditions, such as severity level.
You want to output the same content to both the terminal and a file without extra hassle.
You want to efficiently debug a program that takes a long time to run.

By using logging, you can easily output information-rich logs, which streamlines operational monitoring and bug identification. The larger the scale of your software, the greater the benefits.

So, what kind of logs can you produce? For example, you can output a log like this:

2025-07-11 10:38:03,641 test_logger.py (logger:23) - info : This is a test.

This log is generated with just a single line of code.

logger.info("This is a test.")

Of course, the log format is freely configurable, and you can have the output sent to the terminal just as easily as to a file.

Configuration with YAML and Basic Usage

For configuring logging, I recommend using the highly readable YAML format. For instance, here is a configuration to output logs to ./logs/info.log, with rotation occurring at 10 MB.

version: 1
disable_existing_loggers: False

formatters:
  file:
    format: "%(asctime)s %(filename)s (%(name)s:%(lineno)s) - %(thread)d %(funcName)s [%(levelname)s]: %(message)s"

handlers:
  file_handler:
    class: logging.handlers.RotatingFileHandler
    level: DEBUG
    formatter: file
    filename: ./logs/info.log
    maxBytes: 10485760
    backupCount: 20
    encoding: utf8

root:
  level: INFO
  handlers: [file_handler]

Note that you'll need the PyYAML package to read the YAML file. Be sure to install it beforehand.

Then, you load this YAML file as follows:

import sys
import yaml
from logging import config, getLogger

logger = getLogger(__name__)

try:
    with open("log_conf.yaml", "rt") as f:
        config.dictConfig(yaml.safe_load(f.read()))
except FileNotFoundError:
    logger.critical("log_conf.yaml was not found.")
    sys.exit(1)

After that, you just use this logger as needed.

logger.info("Logger initialized.")
logger.debug("This is in debug mode.")
logger.warning("This is a warning.")
logger.error("An error has occurred.")
logger.critical("A critical, unrecoverable error has occurred.")

Of course, outputting variables using f-strings is also possible.

test_variable = 1
logger.info(f"{test_variable}")

Using RotatingFileHandler and RichHandler Together

You can use multiple handlers at the same time. Below is a configuration example that uses the RotatingFileHandler along with RichHandler for console output.

This setup allows you to log to both a file and the console simultaneously.

version: 1
disable_existing_loggers: False

formatters:
  console:
    format: "%(filename)s (%(name)s:%(lineno)s) - %(thread)d %(funcName)s [%(levelname)s]: %(message)s"

  file:
    format: "%(asctime)s %(filename)s (%(name)s:%(lineno)s) - %(thread)d %(funcName)s [%(levelname)s]: %(message)s"

  file_plain:
    '()': 'misc.StripRichFormatter'
    format: "%(asctime)s %(filename)s (%(name)s:%(lineno)s) - %(thread)d %(funcName)s [%(levelname)s]: %(message)s"

handlers:
  console:
    class: rich.logging.RichHandler
    level: INFO
    markup: True

  file_handler:
    class: logging.handlers.RotatingFileHandler
    level: DEBUG
    formatter: file_plain
    filename: ./logs/stocktrading.log
    maxBytes: 10485760
    backupCount: 20
    encoding: utf8

loggers:
  my_module:
    level: ERROR
    handlers: [console]
    propagate: no

root:
  level: INFO
  handlers: [console, file_handler]

Note that you will need the rich package to use RichHandler. Make sure to install it beforehand.

With RichHandler, you can enrich your console output. For example, you can add color to your messages using markup:

logger.info("[green]Starting the program.[/]")
logger.error("[red bold]An error has occurred.[/]")

This makes the logs very easy to read and is highly recommended ☺️

[Advanced] Decoupling Messages from Code with a Message Manager

As your program grows and the variety of messages increases, hard-coding each message directly in your program becomes difficult to maintain.

To address this, we can decouple the log messages from the code and manage them centrally. For instance, you can create a YAML file for message definitions like this:

# --- Error Messages ---
errors:
  # File not found
  file_not_found: "The log configuration file was not found at '{path}'. Exiting the program."

  # Environment variable not found
  env_not_found: "The definition for the log configuration file is not set in the environment variable {env_file}. Exiting the program."
  api_not_found: "[bold red]The API password is not correctly set in the environment variables.[/]"
  ip_address_not_found: "[bold red]The IP address is not correctly set in the environment variables.[/]"
  port_not_found: "[bold red]The port number is not correctly set in the environment variables.[/]"

# --- Informational Messages ---
info:
  # Program start/end
  program_start: "[green]=== Starting program ({today}) ===[/]"
  program_end: "[green]=== Exiting program ===[/]"

Next, implement a MessageManager class like the one below to retrieve messages from the log_message.yaml file.

import yaml

class MessageManager:
    def __init__(self, file_path="log_messages.yaml"):
        try:
            with open(file_path, "rt", encoding="utf-8") as f:
                self._messages = yaml.safe_load(f)
        except FileNotFoundError:
            self._messages = {}
            print(f"Warning: Message definition file '{file_path}' not found.")

    def get(self, key: str, **kwargs) -> str:
        try:
            template = self._messages
            for k in key.split("."):
                template = template[k]

            if not isinstance(template, str):
                return f"Message key '{key}' is not a valid string."

            return template.format(**kwargs)

        except (KeyError, TypeError):
            return f"Message key '{key}' was not found."

Now, you can instantiate the MessageManager and use it as follows:

from message_manager import MessageManager
msg = MessageManager()
...
path_name = "/path/to/directory"
except FileNotFoundError:
    logger.critical(msg.get("errors.file_not_found", path=path_name))
    sys.exit(1)

errors.file_not_found specifies the file_not_found message within the errors section of your log_message.yaml.

path=path_name is the argument used to substitute the value into the {path} placeholder in the file_not_found message.

You can set as many arguments as you need. For example, with a message defined like this:

sell_executed: "[yellow]{disp_name}[/] ([cyan]{symbol})[/]: Sell order for [cyan]{qty}[/] shares at [cyan]{price}[/] JPY has been executed."

You can call it as follows:

logger.info(
    msg.get(
        "sell_executed",
        disp_name=disp_name,
        symbol=symbol,
        price=f"{int(price):,}" if float(price).is_integer() else price,
        qty=f"{int(qty):,}" if float(qty).is_integer() else qty,
    )
)

Execution Result