Distributed Tracing with OpenTelemetry: Observability for Modern Microservices

Introduction: The Observability Crisis in Microservices

A request enters your system. It touches five microservices. One service is slow. Which one? Why? How long does the bottleneck add to the end-to-end latency?

In monolithic applications, you could trace a single execution flow. In distributed systems with hundreds of services communicating asynchronously, that simple trace becomes impossible without proper instrumentation.

Observability is the answer—the ability to understand system behavior based on its external outputs. And OpenTelemetry is the modern, vendor-neutral standard for implementing it across your entire stack.

This article explores distributed tracing, OpenTelemetry's architecture, practical implementation patterns, and how to transform chaotic microservices logs into actionable insights.

The Problem: Why Traditional Logging Fails in Distributed Systems

Lack of Request Context

Traditional logging captures events on a single machine:

logger.info("Processing order: " + orderId);
logger.info("Fetching user from database");
logger.info("Order processed successfully");

In a monolith, these logs are sequential, readable, and correlated. In microservices:

• Order service logs: "Processing order: 12345"
• User service receives same request → logs separately
• Payment service logs independently
• These logs are scattered across different servers, files, and logging systems

Problem: You have no way to connect these logs into a single request flow.

The Gap Between Logs, Metrics, and Traces

Traditional monitoring focuses on three separate pillars:

1. Logs - Event records (text messages)
2. Metrics - Aggregated counters (response time, CPU usage)
3. Traces - Request flows (distributed context)

But they don't talk to each other:

• Your metrics show elevated latency
• Your logs are millions of lines
• You can't connect the metric spike to the exact request that caused it

Scale and Cost

APM (Application Performance Monitoring) solutions like Datadog and New Relic are powerful but expensive—often charging per event or trace. At scale (billions of requests/day), these costs become prohibitive.

OpenTelemetry: A Unified Standard for Observability

What is OpenTelemetry?

OpenTelemetry (OTel) is a vendor-neutral, open-source standard for collecting, processing, and exporting telemetry data (traces, metrics, logs). Key characteristics:

• Standardized API: Single instrumentation layer across all languages
• Vendor-neutral: Export to Jaeger, Prometheus, Datadog, AWS X-Ray, etc.
• Language support: Java, Python, Go, JavaScript, .NET, and more
• Zero-breaking changes: Built by the observability community (CNCF)
• Production-ready: Used at scale by Uber, Netflix, Google, Amazon

Architecture: Signals, Instrumentation, and Exporters

Your Application
    ↓
    ├─ Traces (distributed request paths)
    ├─ Metrics (counters, gauges, histograms)
    └─ Logs (structured events)
    ↓
OpenTelemetry SDK
    ├─ Instrumentation (automatic + manual)
    ├─ Processors (sampling, batching, filtering)
    └─ Exporters (OTLP, Jaeger, Prometheus, etc.)
    ↓
Backend (Jaeger, Tempo, Prometheus, etc.)

Practical Implementation: Setting Up OpenTelemetry in Java

Step 1: Add Dependencies

For Spring Boot 3.x with Maven:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-bom</artifactId>
            <version>1.35.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <!-- OpenTelemetry Spring Boot Starter -->
    <dependency>
        <groupId>io.opentelemetry.instrumentation</groupId>
        <artifactId>opentelemetry-spring-boot-starter</artifactId>
    </dependency>

    <!-- OTLP Exporter (exports to Jaeger, Tempo, etc.) -->
    <dependency>
        <groupId>io.opentelemetry.exporter</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
    </dependency>
</dependencies>

Step 2: Configure Application Properties

# application.yml
spring:
  application:
    name: order-service

management:
  otlp:
    tracing:
      endpoint: http://localhost:4317  # Jaeger OTLP receiver
  tracing:
    sampling:
      probability: 1.0  # 100% sampling (adjust for production)

Step 3: Automatic Instrumentation (Zero Code Changes)

Spring Boot automatically instruments:

• ✅ HTTP requests (incoming and outgoing)
• ✅ Database queries (JDBC, JPA)
• ✅ Message queues (Kafka, RabbitMQ)
• ✅ Caching (Redis, Caffeine)
• ✅ HTTP clients (RestTemplate, WebClient)

@RestController
@RequestMapping("/api/orders")
public class OrderController {

    @Autowired
    private OrderService orderService;

    @PostMapping
    public ResponseEntity<Order> createOrder(@RequestBody OrderRequest request) {
        // Automatic tracing:
        // - Incoming HTTP request is traced
        // - Span created with trace_id, span_id
        // - All downstream calls inherit this trace_id
        Order order = orderService.processOrder(request);
        return ResponseEntity.ok(order);
    }
}

@Service
public class OrderService {

    @Autowired
    private UserServiceClient userClient;

    @Autowired
    private PaymentServiceClient paymentClient;

    public Order processOrder(OrderRequest request) {
        // Automatic tracing for HTTP calls:
        User user = userClient.getUser(request.getUserId());

        // Automatic tracing for database calls:
        Order order = orderRepository.save(buildOrder(user, request));

        // Automatic tracing for outgoing HTTP:
        PaymentResponse payment = paymentClient.processPayment(order);

        return order;
    }
}

Result: Every request automatically traced with correlation IDs, timing, and dependencies—zero custom code.

Step 4: Custom Spans (When You Need More Granularity)

For business logic not covered by automatic instrumentation:

@Service
public class OrderService {

    @Autowired
    private Tracer tracer;

    public Order processOrder(OrderRequest request) {
        // Automatic spans for HTTP calls
        User user = userClient.getUser(request.getUserId());

        // Create custom span for business logic
        Span validationSpan = tracer.spanBuilder("validate-order")
            .startSpan();

        try (Scope scope = validationSpan.makeCurrent()) {
            validationSpan.addEvent("Validating inventory");
            validateInventory(request.getItems());
            validationSpan.addEvent("Inventory validated");

            // Add attributes for later filtering
            validationSpan.setAttribute("order.item_count", request.getItems().size());
            validationSpan.setAttribute("order.total", calculateTotal(request));
        } finally {
            validationSpan.end();
        }

        return orderRepository.save(buildOrder(user, request));
    }
}

Step 5: Deploy Jaeger for Local Development

Using Docker Compose:

version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP receiver

Start with: docker-compose up

Access Jaeger UI: http://localhost:16686

Real-World Patterns: Debugging Production Issues

Pattern 1: Finding Slow Requests

Scenario: P99 latency is 5 seconds, but P50 is 200ms. Something occasionally breaks.

Without distributed tracing: Check application logs across 20 services, look for timestamps, hope they're synchronized.

With OpenTelemetry:

1. Open Jaeger UI → Search for traces with latency > 5s
2. Click on a slow trace
3. See exact timeline: User Service (100ms) → Order Service (2000ms) → Payment Service (2800ms)
4. Payment Service has the bottleneck
5. Click Payment Service span → See all database queries
6. Identify slow query, fix database index

Time to resolution: 5 minutes (vs. 1-2 hours debugging)

Pattern 2: Debugging Distributed Failures

Scenario: Payment processing randomly fails, but no error logged.

Without tracing: Applications don't crash, no stack trace, mystery failure.

With OpenTelemetry:

1. Filter for traces with "error" status
2. Payment Service spans show HTTP 503 from external gateway
3. Inspect span attributes: http.status_code = 503, http.url = api.payment.com
4. External service was down, retry logic should kick in
5. Fix: Enable exponential backoff in HTTP client configuration

Pattern 3: Understanding Service Dependencies

Scenario: Order Service is slow. Is it our code, or downstream dependencies?

Without tracing: Deploy with more threads, hope it helps.

With tracing:

1. Open Jaeger → Service dependency graph
2. See Order Service → User Service, Inventory Service, Payment Service
3. Click trace → Breakdown:
   • Order Service code: 50ms
   • User Service: 300ms (slow database query)
   • Inventory Service: 800ms (timeout on external API)
   • Payment Service: 1500ms (gateway latency)
4. Optimize User Service query, add caching for Inventory API
5. Total latency: 2650ms → 450ms (5.8x improvement)

Performance Tuning and Sampling

Sampling Strategies

At high volume (1M requests/day), tracing everything is expensive:

management:
  tracing:
    sampling:
      probability: 0.1  # Trace 10% of requests (100k/day)

But sampling has a problem: You might miss rare errors or slow requests.

Solution: Intelligent sampling:

@Configuration
public class TracingConfig {

    @Bean
    public Sampler customSampler() {
        // Trace 100% of errors, 10% of successful requests
        return new Sampler() {
            @Override
            public SamplingResult shouldSample(
                Context parentContext,
                String traceId,
                String name,
                SpanKind spanKind,
                Attributes attributes,
                List<LinkData> parentLinks) {

                // Check for error signals
                if (name.contains("error") || attributes.get("http.status_code") >= 400) {
                    return SamplingResult.recordAndSample();
                }

                // Default: 10% sampling
                return Math.random() < 0.1 
                    ? SamplingResult.recordAndSample() 
                    : SamplingResult.drop();
            }
        };
    }
}

Exporting to Production Backends

Option 1: Jaeger (Self-Hosted)

management:
  otlp:
    tracing:
      endpoint: http://jaeger-collector.monitoring:4317

Pros: Open source, full control, low cost

Cons: Operational overhead

Option 2: Grafana Tempo (Cloud/Self-Hosted)

management:
  otlp:
    tracing:
      endpoint: https://tempo-distributor.grafana-cloud.com/v1/traces
      headers:
        Authorization: Bearer $GRAFANA_API_TOKEN

Pros: Serverless option, integrates with Grafana dashboards

Cons: Requires Grafana account

Option 3: Datadog

management:
  otlp:
    tracing:
      endpoint: https://opentelemetry-collector-http.us5.datadoghq.com/v1/traces
      headers:
        DD-API-KEY: $DATADOG_API_KEY

Pros: Managed service, excellent UI

Cons: Expensive at scale

Best Practices and Gotchas

✅ DO THIS

1. Correlate logs with traces: Include trace_id in all logs

@Bean
public LoggingEventListener loggingEventListener() {
    return event -> {
        String traceId = Tracer.current().getSpanContext().getTraceId();
        MDC.put("trace_id", traceId);  // Log Mapped Diagnostic Context
    };
}

1. Add business context to spans:

span.setAttribute("user.id", user.getId());
span.setAttribute("order.amount", order.getTotal());

1. Use semantic attribute conventions: Follow OpenTelemetry spec for standardized attributes

span.setAttribute("http.method", "POST");
span.setAttribute("http.url", "/api/orders");
span.setAttribute("http.status_code", 201);

❌ AVOID

1. Tracing everything at high volume: Use intelligent sampling
2. Storing PII in traces: Sanitize sensitive data
3. Ignoring trace context propagation: Ensure trace_id flows through all services

The Future: OpenTelemetry Roadmap

• Profiling integration: Link traces directly to CPU profiles
• Logs-first observability: Full logs support in OpenTelemetry
• Metrics to Traces correlation: Click metric spike → see causing trace
• GraphQL instrumentation: Better support for GraphQL APIs
• Serverless optimization: Cost-effective tracing for Lambda/Cloud Functions

Conclusion: From Chaos to Clarity

Microservices create complexity. Observability tames it. OpenTelemetry is the modern standard that brings sanity to distributed debugging:

✅ See request flows across your entire system

✅ Identify bottlenecks in milliseconds, not hours

✅ Correlate logs, metrics, and traces in one view

✅ Avoid vendor lock-in with open standards

✅ Scale cost-effectively with intelligent sampling

If you're running microservices without distributed tracing, you're flying blind. OpenTelemetry removes that blindfold.

Start small: Deploy Jaeger locally, add the Spring Boot starter, and watch your observability problems disappear.

Resources

• OpenTelemetry Official Documentation
• OpenTelemetry Java Instrumentation
• Jaeger Distributed Tracing
• OpenTelemetry Semantic Conventions
• Spring Boot OpenTelemetry Integration
• Grafana Tempo - Traces Backend

---

Tags: #Java #Observability #Microservices #Distributed-Tracing #OpenTelemetry #DevOps #Monitoring