Implementing Distributed Tracing in Go with OpenTelemetry

Distributed tracing is one of those things that's easy to skip when a project is small and genuinely painful to retrofit when it's not. A request hits your API gateway, fans out across three services, spawns a database query and two external calls, then returns — and when something is slow or broken, you're left grepping through logs hoping timestamps line up. OpenTelemetry fixes this, and integrating it into a Go service takes less time than you think.

What OpenTelemetry actually is (and isn't)

OpenTelemetry (OTel) is a vendor-neutral observability framework — it handles the instrumentation layer so you're not locked to Datadog, Jaeger, Zipkin, or any other backend. You instrument once, then ship traces anywhere that speaks OTLP (OpenTelemetry Protocol). The Go SDK is stable and actively maintained.

OTel is not a tracing backend. You still need something to store and visualize traces. For local development, Jaeger is the fastest option to spin up. In production, most teams ship to Grafana Tempo, Honeycomb, or a commercial APM that supports OTLP ingest. The exporter endpoint is a one-line config change — so the backend decision is reversible.

Setting up the tracer provider

Install the required packages first:

go get go.opentelemetry.io/otel \
    go.opentelemetry.io/otel/sdk/trace \
    go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
    go.opentelemetry.io/otel/propagation \
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp \
    google.golang.org/grpc

Then initialize the tracer provider at startup. The key decisions here are which exporter to use, which sampler, and how to propagate context across service boundaries.

package telemetry

import (
    "context"
    "fmt"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func InitTracer(ctx context.Context, serviceName, collectorAddr string) (*sdktrace.TracerProvider, error) {
    conn, err := grpc.NewClient(
        collectorAddr,
        grpc.WithTransportCredentials(insecure.NewCredentials()),
    )
    if err != nil {
        return nil, fmt.Errorf("failed to connect to collector: %w", err)
    }

    exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
    if err != nil {
        return nil, fmt.Errorf("failed to create exporter: %w", err)
    }

    res := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName(serviceName),
        semconv.ServiceVersion("1.0.0"),
    )

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))),
    )

    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    return tp, nil
}

ParentBased(TraceIDRatioBased(0.1)) means sample 10% of root spans, but always follow the sampling decision of an upstream parent. If an upstream service decides to trace a request, you trace it too. This is almost always what you want in a multi-service setup — a gateway controls the sampling decision and all downstream services defer to it.

Call tp.Shutdown(ctx) in your graceful shutdown path — it flushes buffered spans before the process exits.

Instrumenting HTTP handlers

The fastest way to instrument HTTP handlers is through the otelhttp middleware. For standard net/http:

package middleware

import (
    "net/http"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func TraceMiddleware(next http.Handler) http.Handler {
    return otelhttp.NewHandler(next, "http.request",
        otelhttp.WithMessageEvents(otelhttp.ReadEvents, otelhttp.WriteEvents),
    )
}

For Gin, use go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin. For Fiber, otelfiber does the same. All of them:

• Extract trace context from incoming traceparent headers (W3C Trace Context standard)
• Create a server span for each request
• Record HTTP method, route, and status code as span attributes
• Inject span context into the request's context.Context for downstream propagation

Propagating context to downstream calls

The most common mistake is creating spans correctly but not propagating the context when making outbound HTTP calls. This severs the trace — you get disconnected spans instead of a single trace tree. Distributed tracing only works when every hop in a call chain forwards the traceparent header.

import (
    "context"
    "net/http"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
)

func callDownstream(ctx context.Context, url string) (*http.Response, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return nil, err
    }

    // Inject trace context into outbound request headers
    otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

    client := &http.Client{
        Transport: otelhttp.NewTransport(http.DefaultTransport),
    }
    return client.Do(req)
}

Using otelhttp.NewTransport handles injection and creates a client span for the outbound call. The downstream service receives traceparent and links its spans to the same trace. For database calls, go.opentelemetry.io/contrib/instrumentation/database/sql/otelsql wraps database/sql and traces every query with no additional code changes.

Adding custom spans to business logic

Auto-instrumentation captures HTTP and DB calls, but interesting latency usually lives in your business logic. Add custom spans to the code paths that matter:

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
)

func processOrder(ctx context.Context, orderID string) error {
    tracer := otel.Tracer("order-service")
    ctx, span := tracer.Start(ctx, "processOrder")
    defer span.End()

    span.SetAttributes(
        attribute.String("order.id", orderID),
        attribute.String("order.channel", "api"),
    )

    if err := validateInventory(ctx, orderID); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }

    return chargePayment(ctx, orderID)
}

Keep span names stable and low-cardinality. processOrder is good; processOrder-550e8400-e29b-41d4-a716 is not — high-cardinality names break trace aggregation in every backend.

Sampling strategy: the decision that matters most

At 10 req/s, sampling 100% is fine. At 10k req/s, you're looking at real storage and egress costs. The sampler you configure on the tracer provider is the main lever.

Two practical patterns:

Head-based sampling (decided at trace start): use TraceIDRatioBased for uniform sampling, or implement the Sampler interface for rule-based decisions — always trace requests containing auth failures, never trace health checks.

Tail-based sampling (decided after the trace completes): run the OpenTelemetry Collector with tailsamplingprocessor. More infrastructure to maintain, but you can keep all traces containing errors and drop the healthy majority. This is the right choice for security-sensitive services where you need full fidelity on anomalies.

For security-focused services — access control layers, authentication endpoints, audit pipelines — trace retention requirements often align with compliance requirements. The security hardening checklists at AYI NEDJIMI Consultants cover observability retention policies alongside the usual hardening items.

The takeaway

Getting OTel running in a Go service takes under two hours, including a local Jaeger instance for validation. The hard part is getting the team to propagate context correctly through every outbound call — that discipline is what makes distributed tracing useful rather than a collection of disconnected spans. Start with otelhttp middleware and the OTLP gRPC exporter pointed at Jaeger locally. Add custom spans where business logic lives. Once the structure is right, swapping the collector endpoint for production is a one-line change.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.