DEV Community

Cover image for Observability: Understanding Your System Through Logs, Metrics, Traces & Monitoring
Tanu Priya
Tanu Priya

Posted on

Observability: Understanding Your System Through Logs, Metrics, Traces & Monitoring

#microservices #architecture #monitoring

Modern distributed systems are complex.

A single user request might travel through multiple services, databases, caches, queues, and external APIs before a response is returned.

User
  │
  ▼
API Gateway
  │
  ▼
User Service ───→ Redis
  │
  ▼
Order Service
  │
  ▼
Payment Service
  │
  ▼
Database
Enter fullscreen mode Exit fullscreen mode

Now imagine a user reports:

"The application is very slow."

Where is the problem?

Is it:

  • The API server?
  • The database?
  • Redis?
  • A slow microservice?
  • An external API?
  • A network issue?

Without visibility into the system, finding the answer can feel like searching in the dark.

This is where Observability becomes essential.

What Is Observability?

Observability is the ability to understand what is happening inside a system by analyzing the signals it produces.

In simple terms:

Observability helps you answer: "What is happening inside my system, and why?"

A system is observable when engineers can investigate unexpected behavior without needing to manually reproduce every problem.

For example:

Problem Detected
      │
      ▼
Check Metrics
      │
      ▼
Find Affected Service
      │
      ▼
Check Logs
      │
      ▼
Follow Request Trace
      │
      ▼
Find Root Cause
Enter fullscreen mode Exit fullscreen mode

The foundation of observability is often built around three major signals:

  1. Logs
  2. Metrics
  3. Traces

These are commonly called the three pillars of observability.

1. Logs — What Happened?

Logs are detailed records of events that occur inside a system.

For example:

2026-08-23 10:30:12 INFO User logged in
2026-08-23 10:30:15 INFO Payment request started
2026-08-23 10:30:16 ERROR Payment service timeout
Enter fullscreen mode Exit fullscreen mode

Logs help answer questions like:

  • What happened?
  • When did it happen?
  • Which user or request was affected?
  • What error occurred?
  • Which service generated the error?

A useful log usually contains context.

For example:

{
  "timestamp": "2026-08-23T10:30:16Z",
  "level": "ERROR",
  "service": "payment-service",
  "request_id": "abc123",
  "message": "Payment provider timeout"
}
Enter fullscreen mode Exit fullscreen mode

Instead of just seeing:

Error occurred
Enter fullscreen mode Exit fullscreen mode

You now know:

  • When it happened
  • Which service failed
  • Which request was affected
  • What the failure was

This makes debugging significantly easier.

2. Metrics — How Is the System Performing?

Metrics are numerical measurements collected over time.

Examples include:

  • CPU usage
  • Memory usage
  • Request rate
  • Error rate
  • Response time
  • Database connections
  • Queue size

For example:

Requests per second: 12,000
Average latency: 120ms
Error rate: 0.4%
CPU usage: 68%
Enter fullscreen mode Exit fullscreen mode

Metrics are useful because they help identify trends and anomalies.

Imagine this:

10:00 → 0.2% errors
10:05 → 0.3% errors
10:10 → 0.4% errors
10:15 → 8.7% errors 🚨
Enter fullscreen mode Exit fullscreen mode

Something clearly changed.

Metrics help teams answer:

  • Is traffic increasing?
  • Is latency getting worse?
  • Is the error rate rising?
  • Is a server overloaded?

Unlike logs, metrics are especially useful for viewing the overall health of a system.

3. Traces — Where Did the Request Go?

Logs tell you what happened.

Metrics tell you how the system is performing.

But in a distributed system, you also need to know:

Where did this specific request travel?

This is where distributed tracing comes in.

Imagine a request:

User Request
     │
     ▼
API Gateway
     │ 20ms
     ▼
User Service
     │ 50ms
     ▼
Order Service
     │ 120ms
     ▼
Payment Service
     │ 2,500ms ⚠️
     ▼
Database
Enter fullscreen mode Exit fullscreen mode

The total request is slow.

A trace quickly reveals that the payment service is responsible for most of the latency.

A distributed trace typically uses a Trace ID that follows the request across services.

Trace ID: xyz-789

API Gateway
     │
     ▼
User Service
     │
     ▼
Order Service
     │
     ▼
Payment Service
Enter fullscreen mode Exit fullscreen mode

Each operation within the trace is called a span.

Trace
 ├── API Gateway Span
 ├── User Service Span
 ├── Order Service Span
 └── Payment Service Span
Enter fullscreen mode Exit fullscreen mode

This makes it possible to follow one request across an entire distributed architecture.

The Three Pillars Together

Logs, metrics, and traces become far more powerful when used together.

Imagine an alert:

🚨 Payment Error Rate Increased
Enter fullscreen mode Exit fullscreen mode

Step 1: Metrics

Metrics show:

Error Rate: 0.2% → 12%
Enter fullscreen mode Exit fullscreen mode

You know there is a major issue.

Step 2: Traces

Traces show:

Most failed requests → Payment Provider
Enter fullscreen mode Exit fullscreen mode

You now know where the failure is happening.

Step 3: Logs

Logs reveal:

Connection timeout after 5 seconds
Enter fullscreen mode Exit fullscreen mode

Now you understand the root cause.

The debugging flow becomes:

Metrics
   ↓
Detect the problem
   ↓
Traces
   ↓
Locate the problem
   ↓
Logs
   ↓
Understand the problem
Enter fullscreen mode Exit fullscreen mode

This combination dramatically reduces debugging time.

What Is Monitoring?

Observability and monitoring are related, but they are not exactly the same.

Monitoring focuses on tracking known signals and alerting you when something crosses a threshold.

For example:

CPU > 90% → Alert
Error Rate > 5% → Alert
Latency > 2 seconds → Alert
Enter fullscreen mode Exit fullscreen mode

Monitoring answers:

"Is something wrong?"

Observability goes further and helps answer:

"Why is it wrong?"

A simple comparison:

Monitoring Observability
Detects known problems Helps investigate unknown problems
Uses dashboards and alerts Uses multiple system signals
Answers "Is it failing?" Answers "Why is it failing?"
Focuses on symptoms Helps find root causes

You need both.

The Importance of Good Alerts

Not every metric should wake up an engineer at 3 AM.

Poor alerting can create:

100 Alerts
   ↓
Most Are Not Important
   ↓
Alert Fatigue
   ↓
Important Alert Gets Ignored
Enter fullscreen mode Exit fullscreen mode

Good alerts should be:

  • Actionable
  • Meaningful
  • Relevant
  • Prioritized

Instead of alerting:

CPU usage is 80%
Enter fullscreen mode Exit fullscreen mode

Consider alerting based on user impact:

Checkout error rate increased to 15%
Enter fullscreen mode Exit fullscreen mode

The second alert provides a clearer reason to investigate.

The Four Golden Signals

A useful framework for monitoring production systems focuses on four important signals:

1. Latency

How long does a request take?

Average: 120ms
P95: 450ms
P99: 2.1s
Enter fullscreen mode Exit fullscreen mode

Percentiles are important because averages can hide slow requests.

2. Traffic

How much demand is the system handling?

Requests per second: 15,000
Enter fullscreen mode Exit fullscreen mode

A sudden spike may indicate:

  • Viral traffic
  • A marketing campaign
  • Bots
  • Abuse
  • A potential attack

3. Errors

How many requests are failing?

Error Rate: 0.3%
Enter fullscreen mode Exit fullscreen mode

A sudden increase needs investigation.

4. Saturation

How close is the system to its capacity?

Examples:

CPU: 92%
Memory: 88%
Database Connections: 95%
Queue Capacity: 90%
Enter fullscreen mode Exit fullscreen mode

High saturation means the system may soon become unavailable.

Correlation IDs: Connecting Everything

One of the most useful practices in observability is using a correlation ID.

When a request enters the system:

Request ID = req-12345
Enter fullscreen mode Exit fullscreen mode

That ID travels with the request.

API Gateway → req-12345
User Service → req-12345
Order Service → req-12345
Payment Service → req-12345
Enter fullscreen mode Exit fullscreen mode

Now you can search logs across multiple services and connect all events related to the same request.

This is especially valuable in microservice architectures.

Dashboards: Seeing the Health of Your System

A dashboard gives teams a quick view of important signals.

For example:

┌──────────────────────────────────┐
│ System Health                    │
├──────────────────────────────────┤
│ Requests/sec        12,400       │
│ Error Rate          0.4%         │
│ P95 Latency         320ms        │
│ CPU Usage           65%          │
│ Database Health     Healthy ✅    │
└──────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

A good dashboard should answer:

"Is my system healthy right now?"

Different dashboards may exist for:

  • Infrastructure
  • APIs
  • Databases
  • Business metrics
  • Individual services

Observability in Microservices

Observability becomes even more important when an application uses many services.

Consider:

                API Gateway
                     │
        ┌────────────┼────────────┐
        ▼            ▼            ▼
   User Service   Order Service  Search Service
        │            │
        ▼            ▼
   Database      Payment Service
Enter fullscreen mode Exit fullscreen mode

A single request may involve multiple services.

Without distributed tracing and structured logs, debugging becomes extremely difficult.

With observability:

Request
   │
   ▼
Trace ID
   │
   ├── API Gateway
   ├── User Service
   ├── Order Service
   ├── Payment Service
   └── Database
Enter fullscreen mode Exit fullscreen mode

You can understand the complete request journey.

What Makes a System Observable?

A strong observability strategy usually includes:

Structured Logging

Instead of:

Something failed
Enter fullscreen mode Exit fullscreen mode

Use:

{
  "level": "ERROR",
  "service": "order-service",
  "request_id": "abc123",
  "error": "database_timeout"
}
Enter fullscreen mode Exit fullscreen mode

Meaningful Metrics

Track metrics that matter:

  • Request latency
  • Error rate
  • Throughput
  • Resource utilization
  • Queue depth

Avoid collecting thousands of metrics that nobody uses.

Distributed Tracing

Propagate trace context across:

  • Services
  • Queues
  • Background jobs
  • External calls

This makes end-to-end debugging possible.

Centralized Visibility

Instead of logging into multiple servers:

Server 1 Logs
Server 2 Logs
Server 3 Logs
Server 4 Logs
Enter fullscreen mode Exit fullscreen mode

Centralize your signals:

Logs ────┐
Metrics ─┼──→ Observability Platform
Traces ──┘
Enter fullscreen mode Exit fullscreen mode

Now teams have one place to investigate incidents.

Common Observability Mistakes

1. Logging Too Much

More logs don't automatically mean better observability.

Excessive logs can create:

  • Higher storage costs
  • Noise
  • Slower investigations

Log useful information, not everything.

2. Logging Too Little

This is equally dangerous.

If an incident occurs and the logs only say:

Error
Enter fullscreen mode Exit fullscreen mode

Finding the root cause becomes difficult.

Include meaningful context.

3. Monitoring Only Infrastructure

Your servers may look healthy while users are experiencing errors.

For example:

CPU: 30% ✅
Memory: 45% ✅

Checkout Success Rate: 40% ❌
Enter fullscreen mode Exit fullscreen mode

Always monitor user-facing and business-critical metrics too.

4. Too Many Alerts

If everything creates an alert, nothing feels important.

Prioritize alerts based on real impact.

5. Missing Trace Context

If every service creates a completely separate trace, following a request becomes impossible.

Always propagate the trace and correlation context.

A Simple Observability Architecture

A typical architecture may look like this:

                    Application
                         │
        ┌────────────────┼────────────────┐
        ▼                ▼                ▼
      Logs            Metrics           Traces
        │                │                │
        └────────────────┼────────────────┘
                         ▼
              Observability Platform
                         │
        ┌────────────────┼────────────────┐
        ▼                ▼                ▼
    Dashboards         Alerts        Investigation
Enter fullscreen mode Exit fullscreen mode

This gives teams visibility into both:

  • Current system health
  • The root cause of problems

Final Thoughts

You cannot reliably operate what you cannot understand.

As systems become larger and more distributed, observability becomes a core part of system design—not something added only after production issues begin.

Logs tell you what happened.

Metrics tell you how the system is performing.

Traces tell you where a request traveled.

Monitoring tells you when something needs attention.

Together, they help transform production incidents from confusing mysteries into problems that can be detected, investigated, and resolved faster.

Monitoring tells you that something is wrong. Observability helps you understand why.

Key Takeaways

  • Observability helps you understand what is happening inside a system.
  • The three core signals are logs, metrics, and traces.
  • Logs provide detailed event information.
  • Metrics reveal trends and overall system health.
  • Traces follow requests across distributed services.
  • Monitoring detects known problems and triggers alerts.
  • Good alerts should focus on meaningful user impact.
  • Use structured logs and correlation IDs for easier debugging.
  • Track latency, traffic, errors, and saturation.
  • Build observability into your system from the beginning.

*If reliability keeps your system running, observability helps you understand what happens when it doesn't. *

Top comments (0)