Go OTEL Traces without the Globals

So I've been doing some work adding OTEL Traces to a Go service and have some notes. The service in question receives requests with trace headers, but also makes calls to other services and so should be injecting headers into subsequent requests too.

Firstly, the OTEL SDK is vast and takes a while to get your head around. Also, don't think you can just add one plugin for your HTTP router and be done- no such luck unfortunately. The SDK examples make use of global variables in various places which I really don't like. It actually feels like the Prometheus library from a few years ago. Anyway, this guide aims to avoid all the global variables and do things properly.

Setup

First things first- you're going to need a bunch of things that you instantiate and then pass around your app. I found that the 3 things I needed to pass around were the trace.Tracer, trace.TracerProvider and the propagation.TextMapPropagator.

The actual setup of each of those looks something like this:

...
otel.SetTracerProvider(noop.NewTracerProvider())
exporter, err := otlptracehttp.New(ctx)
r := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName("myService"),
        semconv.ServiceVersionKey.String("myVersion"),
    )
traceProvider := sdktrace.NewTracerProvider(
            sdktrace.WithBatcher(exporter),
            sdktrace.WithResource(r),
        )
tracer = traceProvider.Tracer(serviceName, trace.WithInstrumentationVersion(version))
textMapPropagator := propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})
...

What I then ended up doing is creating a struct with these things in it, plus some other observability items that all get passed around the app together which makes thing a lot cleaner

...
type O11y struct {
    Logger            *zerolog.Logger
    Metrics           *PromMetrics
    Tracer            trace.Tracer
    TraceProvider     trace.TracerProvider
    TextMapPropagator propagation.TextMapPropagator
}
...

This setup will use some environment variables to actually get the OTEL endpoint details. More detail on these can be found in the Environment Variable Specification. I like this pattern because an endpoint change can happen without an actual code change.

Now we're ready to start using these somewhere.

HTTP Server

Let's assume that you're using some kind of router package- I'd highly recommend that you then use the OTEL plugin for that package. You can find these plugins in the OTEL registry. In this case we're using the Julien Schmidt HTTPRouter and the plugin is from Splunk (nothing to do with Splunk logging mind you)- SplunkHTTPRouter.

The import path for this is a bit ridiculous, but I like to alias it to know that we're working with the trace version:

import (
...  
tracerouter "github.com/signalfx/splunk-otel-go/instrumentation/github.com/julienschmidt/httprouter/splunkhttprouter"
...
)

Now you actually need to make the router and add it to the server struct:

type Server struct {
    router *tracerouter.Router
    O11y
}

func New(o o11y.O11y) (*Server, error){
  ...
  s := &Server{
    router: tracerouter.New(
        otelhttp.WithTracerProvider(o.TraceProvider),
        otelhttp.WithPropagators(o.TextMapPropagator),
    ),
  }
  ...
  return s, nil
}

One small note here- you may want to exclude some paths from having any tracing on them. This is done as follows:

func OtelReqFilter(req *http.Request) bool {
    return req.URL.Path != "/healthcheck" &&
        req.URL.Path != "/metrics"
}

Then pass this in as another option in the tracerouter.New(), i.e.

...
router: tracerouter.New(
            otelhttp.WithFilter(tracer.OtelReqFilter),
...

Now we’re finally getting somewhere and can start adding some spans.

func (s *Server) myRouteHandler(w http.ResponseWriter, r *http.Request) {    
    span := trace.SpanFromContext(r.Context())
        defer span.End()
    ....
}

One more thing- error handling is a two step operation:

span.RecordError(err)
span.SetStatus(codes.Error, "some error")

Counter-intuitively the first operation does not actually set the span status.

HTTP Client

Now the client is a lot simpler, however because we are not using the global values in the OTEL SDK, we still need to pass in some things in order for trace headers to be injected on outgoing requests.

For each client that you create it has to look something like:

client := &http.Client{Transport: otelhttp.NewTransport(
http.DefaultTransport,
        otelhttp.WithTracerProvider(o.TraceProvider),
       otelhttp.WithPropagators(o.TextMapPropagator),
)}

Conclusion

• Don't use global variables
• Pass around the things your need- there are more than you think for OTEL.
• Use environment variables to set OTEL parameters like sampling percentage. This means you don't need code changes to update them.
• Filter out requests for /metrics and /healthchecks.