Python Logging: Best Practices for Production Apps

Python Logging: Best Practices for Production Apps

You’ve deployed your Python app, and three hours later, a user reports a critical bug. You check the logs, but they’re a chaotic mess of print() statements, missing timestamps, and zero context. You can’t find the error. You can’t trace the request. You’re stuck.

This nightmare is avoidable. The difference between a sleepless night and a smooth debug session isn’t magic—it’s structured, intentional logging. In production, logs are your single source of truth. They tell you what your app is doing, where it’s failing, and why. But Python’s logging module is often misused, treated as a simple debugging tool rather than a production-grade observability system.

Let’s fix that. Here’s exactly how to log like a pro in production.

---

Use Named, Module-Level Loggers

Never use logging.info() or print() directly in your modules. Instead, create a named logger using __name__ in each file:

import logging

logger = logging.getLogger(__name__)

This creates a logger tied to the module’s path (e.g., myapp.services.user_service). Why? It lets you control logging behavior per module without breaking the global configuration.

Configure all handlers and formatters once at startup, ideally in a dedicated logging_config.py module or your app’s entry point. Avoid calling basicConfig() in imported modules—it can cause conflicts and unexpected behavior [6].

---

Centralize Configuration with dictConfig

For scripts, basicConfig() works. For production? No chance. You need handlers, formatters, and log levels that can route logs to consoles, files with rotation, and external collectors like Datadog or AWS CloudWatch.

Use logging.config.dictConfig() to define your setup in a dictionary. This makes config portable, testable, and easy to update without touching code.

Here’s a minimal but production-ready example:

import logging
import logging.config
import json

LOG_CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "json": {
            "()": "pythonjsonlogger.jsonlogger.JsonFormatter"
        },
        "standard": {
            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s"
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "standard",
            "level": "INFO"
        },
        "file": {
            "class": "logging.handlers.RotatingFileHandler",
            "formatter": "json",
            "filename": "logs/app.log",
            "maxBytes": 10_000_000,
            "backupCount": 5,
            "level": "DEBUG"
        }
    },
    "root": {
        "level": "DEBUG",
        "handlers": ["console", "file"]
    }
}

logging.config.dictConfig(LOG_CONFIG)
logger = logging.getLogger(__name__)

Install pythonjsonlogger for JSON formatting:

pip install pythonjsonlogger

This setup gives you:

• Timestamps, levels, logger names, and messages in every line
• JSON logs for file output (easy parsing in Elasticsearch, Loki, etc.)
• Rotating files to prevent disk overflow
• Separate handlers for console (human-readable) and file (structured)

---

Log at the Right Level

Your log levels are an ops contract. Misusing them creates noise or hides critical issues.

Level	When to Use
DEBUG	Targeted troubleshooting; verbose diagnostics
INFO	Normal milestones (e.g., “User logged in”, “Request started”)
WARNING	Abnormal but recoverable states (e.g., deprecated API used)
ERROR	Failed operations (e.g., DB query failed, API timeout)
CRITICAL	Service-threatening failures (e.g., app can’t start)

Never log DEBUG in production unless you’re actively debugging. Sample or drop them to save storage [2].

---

Add Context to Every Log Line

A log message without context is a guess. Always include:

• Request IDs
• User IDs
• Transaction IDs
• Resource names

Use extra, LoggerAdapter, or context variables to inject this data.

logger.info("Request processed", extra={"request_id": "abc123", "user_id": "u456"})

Or with LoggerAdapter:

from logging import LoggerAdapter

adapter = LoggerAdapter(logger, {"request_id": "abc123"})
adapter.info("Request processed")

For microservices, correlate logs with distributed tracing (span IDs, trace IDs) [2].

---

Never Log Secrets or PII

Logging passwords, tokens, or user data is a security risk and a compliance violation. Mask sensitive data early—before it hits the log stream.

def mask_token(token: str) -> str:
    return token[:4] + "..." + token[-4:] if len(token) > 8 else "..."

logger.info(f"Token used: {mask_token(user_token)}")

Add masking logic in your middleware or log adapters. Don’t “fix it later” after data is stored [1].

---

Log Exceptions with Stack Traces

When an exception occurs, use logger.exception() inside except blocks. It automatically includes the traceback.

try:
    db.query("SELECT * FROM users")
except Exception as e:
    logger.exception("Database query failed")

Alternatively, pass exc_info=True to logger.error().

---

Avoid Performance Pitfalls

Logging in hot paths or tight loops can kill performance. Use:

• Lazy formatting: logger.debug("x=%s", x) instead of f"x={x}"
• Queue-based logging: Use QueueHandler + QueueListener to offload I/O to a background thread [2][10]
• Sampling: Drop DEBUG logs in high-throughput systems [2]

---

Test Your Logging

Don’t assume logging works. Test it with:

• caplog in pytest
• Injecting custom handlers to verify output

import pytest

def test_logging(caplog):
    logger = logging.getLogger("test")
    logger.info("Test message")
    assert "Test message" in caplog.text

---

Start Simple, Iterate Fast

You don’t need a full observability stack on day one. Start with:

1. Named loggers
2. Centralized dictConfig
3. JSON logs + rotation
4. Context injection

Then add structured logging libraries (structlog, python-json-logger), integrate with Prometheus/OpenTelemetry, and stream logs via Kafka as your needs grow [2].

---

Your logs are your app’s diary. Treat them with care. Set up structured logging today, and you’ll debug faster, sleep better, and ship with confidence.

What’s your biggest logging pain point? Drop it in the comments—I’ll help you fix it. And if you found this useful, share it with your team. Let’s make production logging boringly reliable.

---

If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming!

Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.