DEV Community

Stephanie Bergamo
Stephanie Bergamo

Posted on • Originally published at stephanie.bergamo.fr

Go - Sending traces, logs and metrics to the Victoria* stack

#go #opentelemetry #observability #victoriametrics

The VictoriaMetrics folks now have a component for each OTel signal: VictoriaMetrics for metrics, VictoriaLogs for logs, and VictoriaTraces for traces. Nice. No collector needed, the Go SDK can talk to all three directly over OTLP/HTTP.

The catch: each one listens on its own port, with its own path, and the paths are almost consistent. Almost.

First, the dependency dance

The OTel Go SDK is split across a lot of modules, and each signal needs its own exporter. I ran go get about fifteen times before everything compiled. Here's the deduplicated version, per component; grab only what you need.

Common to everything:

go get \
  go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/sdk
Enter fullscreen mode Exit fullscreen mode

Traces:

go get \
  go.opentelemetry.io/otel/trace \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
Enter fullscreen mode Exit fullscreen mode

Metrics:

go get \
  go.opentelemetry.io/otel/sdk/metric \
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp
Enter fullscreen mode Exit fullscreen mode

Logs:

go get \
  go.opentelemetry.io/otel/sdk/log \
  go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp
Enter fullscreen mode Exit fullscreen mode

Logs, the logrus way:

go get \
  github.com/sirupsen/logrus \
  go.opentelemetry.io/contrib/bridges/otellogrus
Enter fullscreen mode Exit fullscreen mode

Two things worth knowing:

  • sdk/metric/metricdata comes with sdk/metric: no separate go get. Same for semconv, which ships with otel itself; you just import the version you want, e.g. go.opentelemetry.io/otel/semconv/v1.41.0.
  • Watch the spelling on otlptracehttp. I typed otltracehttp first and spent a minute being confused. The pattern is otlp + signal + protocol, consistently, across all three.

The three endpoints

Each signal goes to its own component, on its own port, with its own path:

  • Metrics → VictoriaMetrics, port 8428 /opentelemetry/v1/metrics
  • Logs → VictoriaLogs, port 9428 /insert/opentelemetry/v1/logs
  • Traces → VictoriaTraces, port 10428 /insert/opentelemetry/v1/traces

Spot the odd one out: metrics has no /insert prefix. Logs and traces do. The docs are clear about it, but it's the kind of thing you'll copy-paste wrong at least once (and see the next section for why you won't find out immediately).

Note that this is a local discovery setup, with everything on one box behind raw ports. In real life you'd put a reverse proxy in front and end up with metrics.example.org, logs.example.org, traces.example.org, and save the whole which port for what struggle.

The hidden trap: the SDK swallows your 404s

If you get the path wrong, nothing happens. No error, no panic, no angry log line. The exporter POSTs, gets a 404, and quietly moves on. Your app runs perfectly and your dashboards stay empty.

This is by design (telemetry that crashes your app is worse than no telemetry) but it makes debugging the setup phase miserable. So before anything else, wire up the error handler:

otel.SetErrorHandler(otel.ErrorHandlerFunc(func(err error) {
    log.Printf("otel error: %v", err)
}))
Enter fullscreen mode Exit fullscreen mode

Now you get told. Keep it while setting things up, and honestly, just keep it.

WithEndpoint or WithEndpointURL?

Both exist and they don't take the same thing, which is a fun five minutes.

WithEndpoint takes host:port without a scheme, and needs friends:

otlptracehttp.WithEndpoint("thanos.local:10428"),
otlptracehttp.WithURLPath("/insert/opentelemetry/v1/traces"),
otlptracehttp.WithInsecure(),   // because no scheme, you say it here
Enter fullscreen mode Exit fullscreen mode

Pass http://thanos.local:10428 to WithEndpoint and it breaks — the scheme
isn't part of the value, it's what WithInsecure() is for.

WithEndpointURL takes the whole thing:

otlptracehttp.WithEndpointURL("http://thanos.local:10428/insert/opentelemetry/v1/traces"),
Enter fullscreen mode Exit fullscreen mode

One option instead of three, and http:// already says "insecure", so no
WithInsecure() needed. Use this one.

Both are available on all three exporters (otlptracehttp, otlploghttp,
otlpmetrichttp), with the same behaviour.

Wiring it up 

const otelHost = "thanos.local"
Enter fullscreen mode Exit fullscreen mode

Metrics:

metricExporter, err := otlpmetrichttp.New(ctx,
    otlpmetrichttp.WithEndpointURL(
        fmt.Sprintf("http://%s:8428/opentelemetry/v1/metrics", otelHost)),
    otlpmetrichttp.WithTemporalitySelector(
        func(kind sdkmetric.InstrumentKind) metricdata.Temporality {
            return metricdata.DeltaTemporality
        },
    ),
)
Enter fullscreen mode Exit fullscreen mode

That WithTemporalitySelector matters. By default the SDK sends cumulative temporality; delta is what you want here, otherwise your counters get double-counted on restart.

Logs:

exporter, err := otlploghttp.New(ctx,
    otlploghttp.WithEndpointURL(
        fmt.Sprintf("http://%s:9428/insert/opentelemetry/v1/logs", otelHost)),
)
Enter fullscreen mode Exit fullscreen mode

Traces:

exp, err := otlptracehttp.New(ctx,
    otlptracehttp.WithEndpointURL(
        fmt.Sprintf("http://%s:10428/insert/opentelemetry/v1/traces", otelHost)),
)
Enter fullscreen mode Exit fullscreen mode

Share the resource

Build the resource once and hand it to all three providers:

func newResource(service, version string) (*resource.Resource, error) {
    return resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName(service),
            semconv.ServiceVersion(version),
        ),
    )
}
Enter fullscreen mode Exit fullscreen mode

Pro tip: if you build with ldflags, wire it to the same variable you already inject.

If you run several instances of the same service, add semconv.ServiceInstanceID(...) while you're here. Costs one line now, saves a refactor later.

Bonus: logrus straight into VictoriaLogs

If you're on logrus, the contrib bridge means zero changes to your existing log calls:

hook := otellogrus.NewHook(service, otellogrus.WithLoggerProvider(loggerProvider))
logrus.AddHook(hook)
Enter fullscreen mode Exit fullscreen mode

Every logrus.WithField(...).Info(...) now lands in VictoriaLogs as a structured record. Nothing else to do.

Don't forget to shut down

All three providers buffer. Batch processors for logs and traces, a periodic reader for metrics. If your process exits without flushing, you lose whatever was in flight (which, for short-lived processes, can be everything).

Collect the shutdown funcs and call them all:

type Observers struct {
    shutdowns []func(context.Context) error
}

func (o *Observers) Shutdown(ctx context.Context) error {
    var errs error
    for _, fn := range o.shutdowns {
        errs = errors.Join(errs, fn(ctx))
    }
    return errs
}
Enter fullscreen mode Exit fullscreen mode

Call site:

defer func() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    obs.Shutdown(ctx)
}()
Enter fullscreen mode Exit fullscreen mode

errors.Join rather than an early return, so one failing provider doesn't stop the other two from flushing.

Two things to watch: give Shutdown a fresh context with its own timeout, because, you know, if you pass the one that just got cancelled, the final flush dies before it leaves. And os.Exit() doesn't run deferred functions, so if your code exits that way, your telemetry never ships.

The Victoria* docs are at docs.victoriametrics.com.
Each component has its own OpenTelemetry setup page, which is where the endpoint paths above come from.

Top comments (0)