🧠 From Chaos to Clarity: How I Designed a Structured Logging System for My Application

Logging is deceptively tricky. When I started building my app, I assumed logging would be simple — just add logger.info() or logger.error() where needed. But as the app grew, so did the problems:

• Inconsistent logs from different layers
• Inability to trace what failed and why
• Unstructured logs that Fluent Bit couldn’t reliably parse
• Missing context (like response time, request ID) in error logs

I had questions. A lot of them. Here's how I answered them — and built a logging system that’s clean, structured, and future-ready.

---

🤯 1. My First Realization: Logging Is Closely Tied to Exceptions

Doubt: “Logging needs all details at the end (status code, time taken, etc.), but most of those are only known at the response stage. Should I log in request and response both?”

Answer:

✅ Logging once — at the response/finalization stage — makes more sense. You get the complete picture: time taken, status code, route, etc.

❌ Logging both request and response separately introduces duplication and inconsistency.

---

🔍 2. Then I Asked: How Do I Get Debug Details from Inside the Library?

Doubt: “If I log inside the library, I miss outer context like time taken. But if I don't log there, I lose debug info.”

Insight:

I realized exceptions are the bridge — I can raise exceptions from deep inside the code, carrying debug metadata with them. Then, at the top-level (e.g., middleware), I catch and log the whole thing.

---

🏗️ 3. My First Design Attempt: Logging + Exception Carrying Details

class AppException(Exception):
    def __init__(self, message, level="error", details=None):
        super().__init__(message)
        self.level = level
        self.details = details or {}

💡 Now I could raise exceptions with structured metadata. I made sure every exception had a level and details dict — which the logger could later use.

---

📤 4. I Wanted JSON Logs for Fluent Bit

Doubt: “How do I send structured logs to Fluent Bit, and will all logs have the same JSON format?”

Answer:

I used Python's logging.config.dictConfig() with a JSON formatter (e.g., json-log-formatter or structlog). Now every log has fields like level, message, request_id, time_taken, etc.

---

🧪 5. Then I Faced: Should Exceptions Decide Log Level?

Doubt: “Wait, does the exception decide if this is info, debug, or error? Or does the logger?”

Answer:

✅ I let exceptions carry the log level as an enum (DEBUG, INFO, ERROR, etc.).

Then my logger dynamically calls logger.debug(), logger.info(), etc. based on that.

---

🧵 6. Then the Real Design Emerged: Middleware-Level Logging

I moved all logging to the final response handler:

• If there’s no exception, log INFO with full context (status, duration, etc.)
• If there's a custom exception, log using its level and details
• If it's unhandled, log as ERROR with traceback

Example final logger:

def log_exception(exc, request_context):
    if isinstance(exc, AppException):
        getattr(logger, exc.level)(exc.args[0], extra={**exc.details, **request_context})
    else:
        logger.error("Unhandled exception", exc_info=exc, extra=request_context)

---

🛑 7. I Also Asked: What About DEBUG Logs?

Doubt: “Do debug logs bubble up to middleware too?”

Answer:

✅ If debug details are important, I attach them as details in exceptions.

🚫 I don’t emit low-level debug logs directly — instead, I raise them with exceptions if they matter for observability.

---

🧠 8. I Realized: Logs and Exceptions Are Loosely Coupled, But Work Together

Insight:

Exceptions carry structured data, but they don't log themselves. The logger does that, at a single point, using info from the exception.

I even thought about creating a single field like log_level and one details dict — now the logger only looks at two fields, making it future-proof and decoupled.

---

💥 9. Final Problem: FastAPI’s HTTPException Doesn’t Work for This

Doubt: “I'm using FastAPI. Should I subclass HTTPException or replace it?”

Answer:

✅ I wrapped HTTPException into a custom LoggedException, adding level and details

✅ Middleware logs and re-raises it as usual

🚫 I never log inside business logic — I only raise with metadata

---

📦 Final Architecture

lib code
│
├── raises AppException("user not found", level="info", details={...})
│
controller
│
└── catches exception
    └── logs once with log level and details (JSON)
    └── returns response

---

🛠 Tools I Considered or Used

Tool	Use
logging + dictConfig	Core logger setup
json-log-formatter	JSON formatter for structured logs
structlog	More advanced context-based structured logging
contextvars	Track request-specific info like trace_id
Fluent Bit	Ship logs to log storage
Elasticsearch, Loki, Sentry	Log & error analysis

---

🧵 Key Takeaways

• Raise structured exceptions with log metadata
• Log once, centrally, with all context (request ID, time taken, etc.)
• Use JSON format for compatibility with log collectors
• Let exceptions carry log level, but log only at the final point
• Avoid scattered logs — treat logs as events, not prints

---

📝 Written on 2025-06-01