Setting Up Telemetry in Golang

Introduction

Observability is a key aspect of modern applications, enabling developers to monitor and troubleshoot their systems effectively. OpenTelemetry (OTel) provides a standardized framework for tracing and metrics collection. This blog post covers the setup of telemetry in a Golang application using OpenTelemetry, focusing on traces and metrics.

By integrating OpenTelemetry into your application, you can:

• Collect detailed traces of API calls and internal function executions.
• Monitor system performance through structured metrics.
• Gain insight into potential bottlenecks and optimize resource utilization.

This guide will walk you through setting up telemetry for a Golang service, including trace and metric collection, middleware integration, and best practices.

Initializing Telemetry Objects

1. serviceResource

The serviceResource object defines metadata about the service, such as its name. It uses OpenTelemetry's semantic conventions to describe the running application:

var serviceResource = resource.NewWithAttributes(
    semconv.SchemaURL,
    semconv.ServiceNameKey.String("pth-auth-service"),
)

This helps identify telemetry data originating from this service when viewed in monitoring tools like Grafana or Jaeger.

2. initTracer

This function initializes the OpenTelemetry Tracer Provider, which enables distributed tracing. Distributed tracing is useful for tracking requests across different microservices and identifying latency issues.

func initTracer(ctx context.Context, otlpEndpoint string) error {
    exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint(otlpEndpoint), otlptracehttp.WithInsecure())
    if err != nil {
        return fmt.Errorf("failed to create OTLP trace exporter: %w", err)
    }

    traceProvider = trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(serviceResource),
    )

    otel.SetTracerProvider(traceProvider)
    return nil
}

This function sets up an HTTP-based OpenTelemetry exporter, which sends trace data to an OpenTelemetry collector or a monitoring backend. The WithBatcher(exporter) option ensures that traces are batched before being exported to improve performance.

3. initMetricProvider

This function sets up the Metric Provider, allowing the application to collect and export metrics for monitoring performance:

func initMetricProvider(ctx context.Context, otelMetricsEndpoint string) error {
    exporter, err := otlpmetrichttp.New(ctx, otlpmetrichttp.WithEndpoint(otelMetricsEndpoint), otlpmetrichttp.WithInsecure())
    if err != nil {
        return fmt.Errorf("failed to create OTLP metric exporter: %w", err)
    }

    MeterProvider = sdkmetric.NewMeterProvider(
        sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter)),
        sdkmetric.WithResource(serviceResource),
    )
    otel.SetMeterProvider(MeterProvider)

    return nil
}

Metrics are crucial for tracking application behavior over time. Examples include HTTP request durations, memory usage, and active database connections.

4. registerMetrics

This function registers various HTTP server metrics to track API performance:

func registerMetrics(meter metric.Meter) {
    metrics := []struct {
        name        string
        description string
        register    func(metric.Meter) (interface{}, error)
    }{
        {HTTPServerDuration, "Measures the duration of inbound HTTP requests", func(m metric.Meter) (interface{}, error) { return m.Float64Histogram(HTTPServerDuration) }},
        {HTTPServerActiveRequests, "Number of concurrent HTTP requests in flight", func(m metric.Meter) (interface{}, error) { return m.Int64UpDownCounter(HTTPServerActiveRequests) }},
        {HTTPServerRequestSize, "Measures the size of HTTP request messages", func(m metric.Meter) (interface{}, error) { return m.Float64Histogram(HTTPServerRequestSize) }},
        {HTTPServerResponseSize, "Measures the size of HTTP response messages", func(m metric.Meter) (interface{}, error) { return m.Float64Histogram(HTTPServerResponseSize) }},
    }

    for _, metricDef := range metrics {
        if _, err := metricDef.register(meter); err != nil {
            fmt.Printf("[Telemetry] Failed to create metric %s: %v\n", metricDef.name, err)
        }
    }
}

These metrics help monitor key aspects of API performance, such as latency and data transfer sizes.

5. InitTelemetry

This function ties everything together by initializing both tracing and metrics. It reads necessary configurations from environment variables and ensures that telemetry is properly set up before the application starts handling requests.

func InitTelemetry(ctx context.Context) error {
    if strings.ToLower(os.Getenv("OTEL_SDK_DISABLED")) == "true" {
        return nil
    }

    otelEndpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    if otelEndpoint == "" {
        return fmt.Errorf("failed to create OTLP trace exporter: endpoint not set")
    }

    if err := initTracer(ctx, otelEndpoint); err != nil {
        return err
    }

    appLogger.Debug().Msg("[Telemetry] Tracer initialized")

    otelMetricsEndpoint := os.Getenv("OTEL_EXPORTER_OTLP_METRICS_ENDPOINT")
    if otelMetricsEndpoint == "" {
        otelMetricsEndpoint = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    }

    if otelMetricsEndpoint == "" {
        return fmt.Errorf("failed to create OTLP metric exporter: endpoint not set")
    }

    if err := initMetricProvider(ctx, otelMetricsEndpoint); err != nil {
        return err
    }

    appLogger.Debug().Msg("[Telemetry] Metric Provider initialized")

    meter := MeterProvider.Meter("pth-auth-service")
    registerMetrics(meter)

    return nil
}

Integrating Telemetry with the Router

To enable telemetry in the router, we use OpenTelemetry middleware, which ensures all incoming requests are instrumented for tracing and metrics:

func SetupRouter() *gin.Engine {
    router := gin.Default()

    router.Use(otelgin.Middleware("pth-auth-service"))
    if telemetry.MeterProvider != nil {
        router.Use(middlewares.RequestMetricsMiddleware(telemetry.MeterProvider.Meter("pth-auth-service")))
    }

    router.Use(middlewares.LogRequestResponseMiddleware)

    router.GET("/401", controllers.UnauthorizedResponse)
    router.GET("/403", controllers.ForbiddenResponse)

    router.Use(middlewares.TenantValidation())
    router.Use(middlewares.TokenVerification())

    router.POST("/auth", controllers.Auth)
    router.GET("/health", controllers.Health)
    router.GET("/version", controllers.Version)

    return router
}

By adding OpenTelemetry middleware, we ensure that all HTTP requests are traced, allowing developers to debug issues more effectively.

References

• OpenTelemetry for Go
• OpenTelemetry GitHub Repository
• Gin Middleware for OpenTelemetry

This setup ensures observability for our Golang service, helping developers gain insights into system behavior and performance.