📌 This article is Part 3 of a multi-part series: "Logging in Django: From Basics to Production".
Check out the previous parts: Part 1: Why Logging? | Part 2: Python Logging Fundamentals
Architectural Building Blocks of Logging in Python
To transform logging from a simple statement into a robust and flexible pipeline, Python builds its logging framework around 4 core modular components:
- Logger: The entry point for dispatching events and capturing logs.
- Handler: Determines the destination and routes log events (console, files, sockets, databases).
- Formatter: Architects the message layout, converting raw event metadata into human-readable text or structured formats (such as JSON).
- Filter: Provides granular, programmatic control for refining events or injecting extra context.
Let's examine these components inside a complete end-to-end pipeline:
import logging
# 1. Define logger and assign an identity
logger = logging.getLogger("my_app")
logger.setLevel(logging.DEBUG)
# 2. Create a formatter to structure the output
formatter = logging.Formatter(
"%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
# 3. Define handler and specify the destination (Console)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(formatter)
# 4. Attach handler to the logger
logger.addHandler(console_handler)
# 5. Emit logs at various severity levels
logger.debug("Debug log - ignored by handler")
logger.info("Info message - displayed")
logger.error("Error message - displayed")
Deep Dive: How the Pipeline Works
1. Logger Identity (getLogger)
By calling logging.getLogger("my_app") , you create or retrieve a specific logger instance. The logger name functions as its identifier across the system, immediately revealing which component or service initiated the event during root-cause analysis.
2. Threshold Control: Logger Level vs. Handler Level
Notice these two lines:
logger.setLevel(logging.DEBUG)
console_handler.setLevel(logging.INFO)
Levels are evaluated at two separate checkpoints:
- First, the logger evaluates whether the message meets its minimum threshold (
DEBUG). If it does, the event is packaged and passed along to attached handlers. - Next, the handler independently verifies whether the event meets its own threshold (
INFO).
Because our logger threshold is DEBUG but the handler requires INFO , the DEBUG message is dropped by the handler and never appears in console output.
Note: The inverse is also true: if the logger is set to a higher threshold (e.g., ERROR) while the handler is set to DEBUG, the logger rejects incoming lower-level events immediately. The handler never receives them.
3. LogRecord and Formatter Internals
Whenever you invoke a logging method, Python dynamically instantiates an internal LogRecord object. This object packages raw execution metadata, such as:
- Exact millisecond timestamp (
created) - Source file line number (
lineno) - Executing thread identifier (
threadName) - The actual raw log message
The Formatter takes this LogRecord object, translates the requested attributes, and shapes the final string representation based on your format definition.
4. Final Destinations via Handlers
The standard StreamHandler writes events directly to standard console streams (like sys.stderr). You can attach multiple handlers to a single logger simultaneously. For example, a single event can be logged to stdout via StreamHandler while errors are persistently written to disk using RotatingFileHandler.
The Power of Filters
Filters do far more than simple severity checks. They give you programmatic access to the event lifecycle for two essential production needs:
- Data Masking (PII Protection): Sanitizing sensitive parameters—such as passwords, tokens, API keys, or personal identifiable information—before logs leave memory.
- Context Injection: Injecting contextual fields (e.g.,
request_id,tenant_id) directly into theLogRecordso every downstream handler and formatter can include them automatically.
Understanding Propagation and Hierarchical Trees
Loggers follow a dot-delimited hierarchy similar to packages or file system directories:
- Parent:
app - Child:
app.api - Grandchild:
app.api.auth
By default, propagate = True. When an event hits a child logger, it is processed by the child's handlers and then passed up the tree to the parent's handlers.
The Duplicate Log Problem
import logging
# Parent logger writing to console
parent_logger = logging.getLogger("app")
parent_logger.setLevel(logging.INFO)
parent_handler = logging.StreamHandler()
parent_handler.setFormatter(logging.Formatter("PARENT -> %(message)s"))
parent_logger.addHandler(parent_handler)
# Child logger with its own console handler
child_logger = logging.getLogger("app.api")
child_logger.setLevel(logging.INFO)
child_handler = logging.StreamHandler()
child_handler.setFormatter(logging.Formatter("CHILD -> %(message)s"))
child_logger.addHandler(child_handler)
# Emit event via child
child_logger.info("Service initialized.")
Console Output:
CHILD -> Service initialized.
PARENT -> Service initialized.
The message was printed twice because propagation passed the event up to parent_logger.
Fixing Duplicate Logs
To restrict event handling strictly to the child logger without bubbling up to ancestors, disable propagation:
child_logger.propagate = False
With propagate = False, the output is clean:
CHILD -> Service initialized.
This prevents wasted I/O, duplicate log entries, and bloated storage costs in production.
What to expect in this series:
- Part 1: Why Logging? (The philosophy, MTTR, and real-world scenarios)
- Part 2: Python Logging Fundamentals: Root Logger, Levels & Module Namespaces
- Part 3: Deep Dive into Architecture: Loggers, Handlers, Filters & Propagation (Current)
- Part 4: Production Django Logging: Dictionaries, JSON Formatting, and Cloud Setups
💬 Discussion:
Have you ever run into unexpected duplicate logs in production caused by propagation? How do you currently handle sensitive data masking in your logs? Let's discuss below!
Top comments (0)