DEV Community

Teruo Kunihiro
Teruo Kunihiro

Posted on

Linking Go pprof Samples to OpenTelemetry Traces with OTLP Profiles

#opentelemetry #go #observability #profiling

Traces tell us which request was slow. CPU profiles tell us which functions consumed CPU time.

Can we start from a slow span and open only the CPU profile samples captured while that span was running?

This article explores that question by converting Go's built-in pprof data into OTLP Profiles, then linking individual Profile Samples to OpenTelemetry Trace and Span IDs. The final result is a Grafana workflow that starts from a GET /cpu-heavy span and opens a Pyroscope flame graph containing main.countPrimes.

The complete lab is available in the go-pprof-otlp-profile-link repository.

Go application -- OTLP Traces --> Collector --> Tempo
       |
       +-- Go pprof pull -------> Collector --> OTLP Profiles --> Pyroscope

Tempo + Pyroscope --> Grafana
Enter fullscreen mode Exit fullscreen mode

This is a fixed-version experiment, not a production deployment guide. OpenTelemetry Profiles is still evolving, so API, Collector, and backend compatibility must be revalidated when versions change.

The main versions used in the lab are:

Component Version
OpenTelemetry Collector Core / Contrib 0.158.0
Pyroscope 2.2.0
Tempo 3.0.2
Grafana 13.1.2
Go 1.25

What is an OpenTelemetry Profile?

A Profile aggregates observations of a running program over a period of time. A CPU profiler periodically samples the active call stack and records where CPU time was spent.

A four-second CPU Profile can be pictured like this:

Profile (four-second CPU capture)
├─ Sample A
│  ├─ stack: main.countPrimes → main.burnCPU
│  └─ value: CPU time
├─ Sample B
│  ├─ stack: runtime.gc
│  └─ value: CPU time
└─ Sample C
   ├─ stack: main.countPrimes → main.burnCPU
   └─ value: CPU time
Enter fullscreen mode Exit fullscreen mode

A Sample is an entry in the Profile. Multiple observations with the same stack and labels may be aggregated into one Sample entry. A Sample can contain:

  • A call stack.
  • One or more measured values, such as CPU time.
  • Attributes derived from profiler labels.
  • An optional Link to a Trace and Span.

OTLP Profiles represents this data using shared dictionary tables. Samples refer to strings, functions, stacks, attributes, and Links by index instead of repeating the same data.

Profiles
├─ Dictionary
│  ├─ StringTable
│  ├─ FunctionTable
│  ├─ StackTable
│  └─ LinkTable
└─ ResourceProfiles
   └─ ScopeProfiles
      └─ Profile
         └─ Samples
Enter fullscreen mode Exit fullscreen mode

What the OpenTelemetry Go SDK provides

At the time of this experiment, the official OpenTelemetry Go API and SDK do not implement the Profiles signal. They create Traces and Spans and provide Trace IDs and Span IDs, but they do not capture CPU profiles, produce OTLP Profiles, or create Profile Links.

OpenTelemetry Go SDK
├─ Create Traces and Spans                 yes
├─ Provide Trace IDs and Span IDs          yes
├─ Capture CPU profiles                    no
├─ Export OTLP Profiles                    no
└─ Link Profile Samples to Spans           no
Enter fullscreen mode Exit fullscreen mode

Go's runtime/pprof package fills a different part of the pipeline:

runtime/pprof
├─ Sample CPU stacks                       yes
├─ Add labels to profile samples           yes
├─ Convert pprof into OTLP Profiles        no
└─ Create OTLP Profiles Links              no
Enter fullscreen mode Exit fullscreen mode

The experiment combines the two: the OpenTelemetry SDK supplies the current Trace and Span IDs, while runtime/pprof attaches those IDs to CPU samples as labels.

Add the current Span IDs to pprof labels

Each application endpoint is wrapped by otelhttp.NewHandler. The wrapper creates a Server Span for every incoming request and places that Span in the request context before calling the inner handler.

The inner handler retrieves the existing SpanContext and adds its IDs to a pprof label set:

func tracedAndProfiledHandler(
    serviceName, profileMode, spanName string,
    handler http.HandlerFunc,
) http.Handler {
    profiled := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        labelValues := []string{
            "service.name", serviceName,
            "profile_mode", profileMode,
        }

        if profileMode == "pprof-link" {
            spanContext := trace.SpanContextFromContext(r.Context())
            if spanContext.IsValid() {
                labelValues = append(labelValues,
                    "otel.profile.trace_id", spanContext.TraceID().String(),
                    "otel.profile.span_id", spanContext.SpanID().String(),
                    "otel.profile.span_name", spanName,
                )

                trace.SpanFromContext(r.Context()).SetAttributes(
                    attribute.String(
                        "pyroscope.profile.id",
                        spanContext.SpanID().String(),
                    ),
                )
            }
        }

        labels := pprof.Labels(labelValues...)
        pprof.Do(r.Context(), labels, func(ctx context.Context) {
            handler.ServeHTTP(w, r.WithContext(ctx))
        })
    })

    return otelhttp.NewHandler(profiled, spanName)
}
Enter fullscreen mode Exit fullscreen mode

trace.SpanContextFromContext does not create a Span. It reads the SpanContext that otelhttp.NewHandler already placed in the request context.

profileMode is a switch used only by this lab; pprof-link selects the Case D correlation path.

pprof.Labels creates a key/value label set. pprof.Do applies the augmented labels while the callback runs and restores the previous labels afterward. Goroutines created inside the callback inherit the augmented label set.

This code does not start one CPU profiler per Span. The Collector starts a process-wide CPU capture by pulling the pprof endpoint:

receivers:
  pprof/cpu:
    remote:
      endpoint: http://app:8080/debug/pprof/profile?seconds=4
      collection_interval: 1s
Enter fullscreen mode Exit fullscreen mode

If the CPU profiler samples the labeled goroutine during that four-second window, the resulting pprof sample contains the Trace ID and Span ID labels.

One process-wide capture can include Samples from several requests, as well as runtime work that has no request Span at all.

What is an OTLP Profiles Link?

An OTLP Profiles Link associates a Profile Sample with an existing Trace and Span.

ProfilesDictionary.LinkTable[1]
  trace_id: <16 bytes>
  span_id:  <8 bytes>

Profile.Sample
  stack_index: 1
  link_index:  1
Enter fullscreen mode Exit fullscreen mode

link_index: 1 means that the Sample refers to LinkTable[1].

The Link contains only a Trace ID and Span ID. It does not contain the Trace itself. The Trace travels through a separate Traces pipeline and is stored in a tracing backend such as Tempo.

Why is the Link attached to each Sample instead of the whole Profile?

A single four-second Profile can overlap several requests:

Profile (four seconds)
├─ Sample A → Span 111
├─ Sample B → Span 222
├─ Sample C → no Link
└─ Sample D → Span 111
Enter fullscreen mode Exit fullscreen mode

Assigning one Span ID to the entire Profile would lose this distinction. Correlation must therefore happen at Sample level.

Convert pprof labels into Profile Links

In Collector Contrib 0.158.0, the pprof receiver converts pprof sample labels into OTLP Profile Sample attributes. It does not promote the correlation attributes into Links.

pprof sample labels
  otel.profile.trace_id = aaa...
  otel.profile.span_id  = 111...
        ↓ pprof receiver
OTLP Profile Sample attributes
  otel.profile.trace_id = aaa...
  otel.profile.span_id  = 111...
Enter fullscreen mode Exit fullscreen mode

To bridge that gap, the lab implements a custom profilelink processor in Go. This is not a processor included in the stock Collector distribution. OpenTelemetry Collector Builder packages it with the official receivers and exporters:

processors:
  - gomod: example.com/otel-profile-lab/profilelinkprocessor v0.1.0
    path: ./profilelinkprocessor
Enter fullscreen mode Exit fullscreen mode

The processor walks the Profiles hierarchy and visits each Sample once:

ResourceProfiles
└─ ScopeProfiles
   └─ Profile
      └─ Sample
Enter fullscreen mode Exit fullscreen mode

After validating the IDs, it deduplicates Links by the Trace ID and Span ID pair, appends a Link when necessary, and sets the Sample's link_index:

key := linkKey{traceID: traceID, spanID: spanID}
linkIndex, ok := linkIndices[key]
if !ok {
    link := links.AppendEmpty()
    link.SetTraceID(traceID)
    link.SetSpanID(spanID)

    linkIndex = int32(links.Len() - 1)
    linkIndices[key] = linkIndex
}

sample.SetLinkIndex(linkIndex)
Enter fullscreen mode Exit fullscreen mode

Index zero is the empty sentinel, so real Link entries begin at index one. Samples from the same Span reuse the same Link entry.

LinkTable
├─ [0] empty sentinel
├─ [1] Trace=aaa, Span=111
└─ [2] Trace=aaa, Span=222

Samples
├─ Sample A → LinkTable[1]
├─ Sample B → LinkTable[2]
└─ Sample D → LinkTable[1]
Enter fullscreen mode Exit fullscreen mode

The Profiles pipeline places the custom processor between the pprof receiver and the Pyroscope exporter:

service:
  pipelines:
    profiles:
      receivers: [pprof/cpu]
      processors: [profilelink]
      exporters: [otlp_grpc/pyroscope, debug/profiles]
Enter fullscreen mode Exit fullscreen mode

The processor mutates the incoming pprofile.Profiles data in place. p.next.ConsumeProfiles(ctx, profiles) then forwards the modified Profile to the next component in the pipeline.

Verify the correlation end to end

Clone the repository and run:

git clone https://github.com/trknhr/go-pprof-otlp-profile-link.git
cd go-pprof-otlp-profile-link
make case-d
make verify-d
Enter fullscreen mode Exit fullscreen mode

verify-d checks the same Trace ID and Span ID across:

  1. The application's HTTP response.
  2. The Trace stored in Tempo.
  3. The Profile.Sample.link created by the Collector.
  4. Pyroscope's Trace ID selector.
  5. Pyroscope's Span ID selector.
  6. main.countPrimes in the returned flame graph.

Grafana can then open the GET /cpu-heavy Trace in Tempo and navigate from the selected Span to Related profiles.

Grafana showing a Tempo trace beside a Pyroscope flame graph filtered by the selected Span ID

The Pyroscope query uses both the service name and the selected Span ID:

labelSelector: {service_name="otel-profile-demo"}
spanSelector:  [selected Span ID]
Enter fullscreen mode Exit fullscreen mode

There are two related but separate mechanisms here:

Grafana navigation
  Span attribute: pyroscope.profile.id
        ↓
  Build a spanSelector query

Data correlation
  Profile.Sample.link
        ↓
  Pyroscope returns Samples linked to that Span
Enter fullscreen mode Exit fullscreen mode

The pyroscope.profile.id Span attribute helps Grafana build the query. The OTLP Profile Link carries the correlation in the Profile data itself.

Caveats

CPU profiling is sampled

CPU profiling does not record every operation. A short Span can complete between samples, and a request that does not overlap the four-second pull window will not appear in that Profile.

CPU profiles also measure CPU execution, not wall-clock latency. A slow request that mostly waits for a database, network response, lock, or timer may contribute little CPU profile data.

The verification script retries the target request when it misses the current sampling window.

Span IDs create high cardinality

pprof can aggregate observations that have the same stack and labels. Adding a unique Span ID changes the aggregation key:

Without a Span ID label
  countPrimes × 3 observations → one aggregated Sample

With Span ID labels
  countPrimes + Span A → Sample A
  countPrimes + Span B → Sample B
  countPrimes + Span C → Sample C
Enter fullscreen mode Exit fullscreen mode

This approach does not start one Profile per Span, but it can increase the number of Samples, transfer size, storage, and query cost. A production design should decide which Spans to include and how long to retain the resulting data.

The Profiles ecosystem is still evolving

The OpenTelemetry Profiles specification is Alpha, the OTLP Profiles protobuf packages are in Development, and the official OpenTelemetry Go SDK does not yet expose a Profiles API or SDK. The Collector, Pyroscope, and Grafana versions should be upgraded and tested together.

Measure the custom processor

The processor visits each Sample once, so the nested loops follow the Profiles hierarchy rather than producing combinatorial work. Even so, production use should measure processing time, allocations, Profile size, and log volume under realistic load.

The pprof endpoint also exposes internal application information and should not be publicly accessible without appropriate network controls and authentication.

Summary

Converting pprof data into OTLP Profiles and correlating Profile Samples with a specific Span are separate operations.

This lab divides the work as follows:

OpenTelemetry Go SDK
  → create Server Spans and Trace/Span IDs

Go runtime/pprof
  → capture CPU profiles
  → retain IDs as sample labels

Collector pprof receiver
  → convert pprof into OTLP Profiles
  → convert labels into Sample attributes

Custom profilelink processor
  → turn Sample attributes into OTLP Profiles Links

Tempo, Pyroscope, and Grafana
  → navigate from a Span to its linked CPU Samples
Enter fullscreen mode Exit fullscreen mode

The result is a direct path from a slow Trace Span to the code that consumed CPU while that Span was running.

References

Top comments (0)