Implementing Distributed Tracing in Go with OpenTelemetry

When a request fails in a distributed system, you end up grepping through five different log files across five services trying to figure out what happened. Logs tell you what, but not where — which service added 800ms of latency, or where an error actually originated across the call chain. Distributed tracing solves this by recording the full lifecycle of a request across service boundaries, and OpenTelemetry is the vendor-neutral standard for doing it without locking into a specific backend.

What OpenTelemetry Is (and What It Isn't)

OpenTelemetry (OTel) is not a backend — it's an instrumentation framework. It standardizes how you collect traces, metrics, and logs, and ships with:

• An API — the interface your code calls
• An SDK — the implementation (samplers, exporters, propagators)
• A Collector — optional intermediary that receives, processes, and routes telemetry
• Exporters — Jaeger, Zipkin, Datadog, Grafana Tempo, and others

You instrument your code once against the OTel API. You can route data to any backend by swapping the exporter. The instrumentation stays the same.

Initializing the Tracer Provider

Install the necessary packages:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk/trace
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Then initialize the tracer at startup:

package main

import (
    "context"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "google.golang.org/grpc"
)

func initTracer(ctx context.Context) (func(context.Context) error, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("localhost:4317"),
        otlptracegrpc.WithInsecure(),
        otlptracegrpc.WithDialOption(grpc.WithBlock()),
    )
    if err != nil {
        return nil, err
    }

    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName("order-service"),
            semconv.ServiceVersion("1.2.0"),
        ),
    )
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(
            sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)),
        ),
    )
    otel.SetTracerProvider(tp)
    return tp.Shutdown, nil
}

func main() {
    ctx := context.Background()
    shutdown, err := initTracer(ctx)
    if err != nil {
        log.Fatal(err)
    }
    defer shutdown(ctx)

    // start your HTTP server here
}

A few details worth noting:

• WithBatcher is what you want in production — it buffers spans and sends them in batches rather than flushing on every span end
• WithInsecure() is fine when the Collector runs on the same host; use TLS everywhere else
• The resource is metadata attached to every span emitted by this service — service name and version at minimum

Instrumenting Business Logic

The OTel API follows a simple pattern: create a tracer once per package, call Start to open a span, and End it when the operation is done.

package handler

import (
    "context"
    "fmt"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
)

var tracer = otel.Tracer("github.com/yourorg/order-service/handler")

func ProcessOrder(ctx context.Context, orderID string) error {
    ctx, span := tracer.Start(ctx, "ProcessOrder")
    defer span.End()

    span.SetAttributes(
        attribute.String("order.id", orderID),
        attribute.String("order.region", "eu-west-1"),
    )

    if err := validateOrder(ctx, orderID); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "order validation failed")
        return fmt.Errorf("validate: %w", err)
    }

    span.AddEvent("order validated")

    if err := chargePayment(ctx, orderID); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "payment failed")
        return fmt.Errorf("charge: %w", err)
    }

    span.SetStatus(codes.Ok, "")
    return nil
}

func validateOrder(ctx context.Context, orderID string) error {
    _, span := tracer.Start(ctx, "validateOrder")
    defer span.End()
    // validation logic
    return nil
}

func chargePayment(ctx context.Context, orderID string) error {
    _, span := tracer.Start(ctx, "chargePayment")
    defer span.End()
    // payment logic
    return nil
}

The critical discipline: pass ctx down through every function call. OpenTelemetry stores the active span in the context. Drop the context and child spans lose their parent — you end up with disconnected orphan spans instead of a coherent tree.

HTTP Middleware and Outbound Calls

For HTTP servers, the contrib package handles span creation automatically per request:

import (
    "net/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/orders", ordersHandler)

    handler := otelhttp.NewHandler(mux, "http-server",
        otelhttp.WithMessageEvents(otelhttp.ReadEvents, otelhttp.WriteEvents),
    )
    http.ListenAndServe(":8080", handler)
}

This creates a span per request, attaches standard HTTP attributes (method, route, status code), and — crucially — extracts incoming trace context from headers. If an upstream service already started a trace, your service joins that trace rather than starting a new one. That's what makes cross-service tracing work.

For outbound calls, wrap the HTTP client transport:

client := &http.Client{
    Transport: otelhttp.NewTransport(http.DefaultTransport),
}
resp, err := client.Get("https://inventory-service/items/42")

The transport injects the current trace context into the outgoing traceparent header automatically.

Sampling Strategy

Recording every span in a high-traffic service is expensive. The options in order of practical usefulness:

AlwaysSample — every request is traced. Fine for development or low-traffic services. Not for production at scale.

TraceIDRatioBased(0.05) — traces 5% of requests. Inexpensive, but you can miss rare events.

ParentBased(TraceIDRatioBased(0.1)) — respects the sampling decision of the upstream service. If the caller sampled this trace, you continue it. If not, you don't start one. This is the correct default for production: it keeps traces coherent across service boundaries without every service independently deciding whether to sample.

One real-world wrinkle: you want to always trace errored requests regardless of the base sample rate. The OpenTelemetry Collector supports this with its tailsampling processor, which can override the head-based decision after a trace completes. This is more complex to operate but gives you both cost efficiency and full fidelity on failures.

One operational note: span attributes can easily capture sensitive data — JWT fragments, query parameters, internal IDs. Before deploying to production, check what your instrumentation is recording. A security hardening checklist for your environment is a practical starting point for what to review.

The Takeaway

Distributed tracing in Go with OpenTelemetry breaks down to four steps: initialize a TracerProvider with a batching exporter, propagate ctx everywhere, use contrib middleware for protocol-level spans (HTTP, gRPC, database drivers), and configure ParentBased sampling for production.

The diagnostic value comes from the attributes and events you attach to spans — not just the default fields. span.SetAttributes(attribute.String("order.id", id)) is what turns "payment service returned an error" into "payment failed on order #4521 in eu-west-1 at 14:03:22." That's the difference between tracing and useful tracing.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.