Complete Python Logging Guide: From Fundamentals to Production

This comprehensive tutorial covers everything you need to know about logging in Python: best practices for cloud/containers, core concepts, the logger hierarchy, and building a production-ready colored logging setup.

---

1. Why Send Logs to stdout/stderr Instead of Files?

When building modern applications (especially inside Docker, Kubernetes, or serverless platforms), the standard best practice is to treat logs as an unbuffered event stream.

stdout vs stderr

• stdout (Standard Output): Used for standard application flow logs (DEBUG, INFO).
• stderr (Standard Error): Used for warnings, runtime errors, and fatal exceptions (WARNING, ERROR, CRITICAL).

Benefits Over File Logging

• Container Compatibility: Containers (like Docker) are temporary. Files stored inside a container disappear when it restarts. Container runtimes automatically capture stdout and stderr.
• Centralized Aggregation: Log collectors (Datadog, AWS CloudWatch, Grafana Loki, Fluentd) scrape standard output streams automatically.
• Disk Safety: Writing directly to disk risk filling up server storage and crashing your application.
• Separation of Concerns: Your app's job is to output events. The hosting platform's job is to store, rotate, and archive them.

---

2. Python Logging Basics

Python includes a built-in logging module. You should use it instead of print() statements.

Why Avoid print()?

• print() cannot easily filter messages by importance.
• print() lacks timestamps, file names, line numbers, or context.
• print() sends everything to stdout without distinguishing errors.

Log Levels

Python provides five standard log levels, ordered by severity:

Level	Numerical Value	When to Use
DEBUG	10	Low-level details during development.
INFO	20	Normal operational events (e.g., service started, request processed).
WARNING	30	Something unexpected happened, but the program can continue.
ERROR	40	A serious error occurred; a feature failed to execute.
CRITICAL	50	A fatal error; the entire program may crash.

---

3. The 4 Pillars of Python Logging

Advanced logging in Python relies on four core classes:

1. Logger: The primary interface your application code calls (e.g., logger.info()).
2. Handler: Directs log messages to their final destination (StreamHandler for console, FileHandler for disk files).
3. Formatter: Defines the layout and text structure of the final log message.
4. Filter: Provides fine-grained control to include or exclude specific log records.

---

4. Understanding the Logger Hierarchy

Loggers are organized in a parent-child tree using dot notation, similar to Python module import paths.

1. Hierarchy Structure

• "" (Empty string): The Root Logger at the top of the tree.
• "app": Child of the Root Logger.
• "app.database": Child of "app".
• "app.database.mysql": Child of "app.database".

Best practice in any file is to instantiate loggers using __name__:

logger = logging.getLogger(__name__)

2. Propagation Flow

When a child logger records a message:

1. It checks if the message meets its own level setting.
2. If allowed, it executes its attached Handlers.
3. It passes the message upward to its parent logger's handlers (this behavior is called propagation).

Root Logger ("")        ---> StreamHandler (Console Output)
       ▲
       │ (Propagates UP)
 Logger ("app")         ---> Custom FileHandler
       ▲
       │ (Propagates UP)
Logger ("app.database") ---> Calls logger.info("Query executed")

To stop a child logger from sending records to parent handlers:

logger.propagate = False

---

5. Production-Ready Configuration with colorlog

To make console output easily readable during development and log inspection, you can color-code your terminal logs using the third-party colorlog library:

pip install colorlog

The Configuration Function

import logging
import colorlog

def configure_logging(logger: logging.Logger, verbose: bool = False) -> None:
    """Configures a logger with ANSI color output and dynamic verbosity."""

    # 1. Create a stream handler targeting standard output/error
    handler = colorlog.StreamHandler()

    # 2. Format output with level-specific colors
    handler.setFormatter(colorlog.ColoredFormatter(
        "%(log_color)s%(asctime)s [%(name)s] %(levelname)s: %(message)s",
        datefmt="%H:%M:%S",
        log_colors={
            'DEBUG':    'cyan',
            'INFO':     'green',
            'WARNING':  'yellow',
            'ERROR':    'red',
            'CRITICAL': 'red,bg_white',
        }
    ))

    # 3. Clear existing handlers to prevent duplicate lines
    logger.handlers.clear()

    # 4. Attach configured handler
    logger.addHandler(handler)

    # 5. Set log level based on verbosity flag
    logger.setLevel(logging.DEBUG if verbose else logging.INFO)

---

6. Complete End-to-End Example

Here is a complete, executable script demonstrating logger initialization, child inheritance, and color formatting.

import logging
import colorlog

# --- Logging Setup Helper ---
def configure_logging(logger: logging.Logger, verbose: bool = False) -> None:
    handler = colorlog.StreamHandler()
    handler.setFormatter(colorlog.ColoredFormatter(
        "%(log_color)s%(asctime)s [%(name)s] %(levelname)s: %(message)s",
        datefmt="%H:%M:%S",
        log_colors={
            'DEBUG':    'cyan',
            'INFO':     'green',
            'WARNING':  'yellow',
            'ERROR':    'red',
            'CRITICAL': 'red,bg_white',
        }
    ))
    logger.handlers.clear()
    logger.addHandler(handler)
    logger.setLevel(logging.DEBUG if verbose else logging.INFO)


# --- Application Setup ---
# 1. Initialize parent logger
app_logger = logging.getLogger("my_app")
configure_logging(app_logger, verbose=True)  # Enable debug mode

# 2. Initialize child logger
db_logger = logging.getLogger("my_app.database")


# --- Execution ---
if __name__ == "__main__":
    app_logger.info("Initializing application services...")

    # Child logger inherits parent's configuration via propagation
    db_logger.debug("Connecting to PostgreSQL at 127.0.0.1:5432")
    db_logger.info("Database connection established.")

    # Simulate warning and error events
    app_logger.warning("Cache missed for key: 'user_1024'")

    try:
        result = 1 / 0
    except ZeroDivisionError:
        db_logger.error("Failed to calculate user metric", exc_info=True)

    app_logger.critical("Memory threshold exceeded! Shutting down process.")