DEV Community

Thesius Code
Thesius Code

Posted on • Originally published at datanest-stores.pages.dev

Python Logging & Config: Python Logging Guide

#webdev #python #programming #tutorial

Python Logging Guide

Why Structured Logging?

Plain-text logs are hard to search, filter, and alert on at scale. Structured
logging (JSON) makes every log line a queryable record — essential for
production observability.

Feature Plain Text Structured (JSON)
Human-readable Yes With tooling
Machine-parseable Regex needed Native
Searchable grep Field queries
Alerting Pattern match Exact field match
Context (request_id) Manual Automatic

Configuration Strategy

Use environment-specific YAML configs:

configs/
├── logging_dev.yaml   # Colorized console, DEBUG level
├── logging_prod.yaml  # JSON to stdout + file, INFO level
└── logging_test.yaml  # WARNING only, minimal output
Enter fullscreen mode Exit fullscreen mode

Load with one call:

from src.setup import configure_logging
configure_logging("prod")
Enter fullscreen mode Exit fullscreen mode

Request Context Propagation

Every log line should identify which request it belongs to. This toolkit
uses contextvars (works with asyncio, threads, and sync code):

from src.context import set_context, generate_request_id

# In middleware (automatic):
set_context(request_id=generate_request_id())

# In your code:
set_context(user_id="usr-42", tenant="acme")

# Every log line now includes these fields automatically
Enter fullscreen mode Exit fullscreen mode

Middleware Integration

FastAPI / Starlette (ASGI) 

from src.middleware import ASGILoggingMiddleware
app.add_middleware(ASGILoggingMiddleware)
Enter fullscreen mode Exit fullscreen mode

Flask / Django (WSGI) 

from src.middleware import WSGILoggingMiddleware
app.wsgi_app = WSGILoggingMiddleware(app.wsgi_app)
Enter fullscreen mode Exit fullscreen mode

Both middleware:

  1. Generate or extract request_id from headers
  2. Set context variables
  3. Log request start/finish with duration
  4. Clean up context after the request

Filtering Noise

Suppress third-party loggers 

from src.filters import SuppressLoggerFilter
handler.addFilter(SuppressLoggerFilter(["urllib3", "botocore", "asyncio"]))
Enter fullscreen mode Exit fullscreen mode

Rate-limit repeated messages 

from src.filters import RateLimitFilter
handler.addFilter(RateLimitFilter(period_seconds=60))
Enter fullscreen mode Exit fullscreen mode

Performance Tips

  1. Use QueueHandler in production — create_async_handler() offloads
    I/O to a background thread so logging never blocks your request.

  2. Set appropriate levels — Don't log DEBUG in production. Every log
    line has a cost (CPU, I/O, storage).

  3. Use lazy formatting — logger.info("User %s", user_id) is faster
    than logger.info(f"User {user_id}") because formatting is skipped
    if the level is disabled.

  4. Rotate logs — Use RotatingFileHandler or TimedRotatingFileHandler
    to prevent disk exhaustion.

Common Patterns

Add context to exceptions 

try:
    process_order(order_id)
except Exception:
    logger.exception("Failed to process order", extra={"order_id": order_id})
Enter fullscreen mode Exit fullscreen mode

Structured extra fields 

logger.info(
    "Payment processed",
    extra={"amount": 49.99, "currency": "USD", "order_id": "ORD-123"},
)
Enter fullscreen mode Exit fullscreen mode

Conditional logging 

if logger.isEnabledFor(logging.DEBUG):
    # Expensive serialization only when DEBUG is active
    logger.debug("Full payload: %s", json.dumps(payload, indent=2))
Enter fullscreen mode Exit fullscreen mode

Anti-Patterns

  1. print() instead of logging — No levels, no formatting, no routing.
  2. Logging sensitive data — Never log passwords, tokens, or PII.
  3. Catching and logging without re-raising — Swallows the error.
  4. String concatenation in log calls — Use %s placeholders instead.
  5. One logger for everything — Use logging.getLogger(__name__) per module.

This is 1 of 14 resources in the Python Developer Pro toolkit. Get the complete [Python Logging & Config] with all files, templates, and documentation for $19.

Get the Full Kit →

Or grab the entire Python Developer Pro bundle (14 products) for $159 — save 30%.

Get the Complete Bundle →

Related Articles

Top comments (0)