04 – eBPF Uprobes: Tracing gRPC Headers by Unpacking Go Function Internals

Introduction to Uprobes

In the previous post, we explored kprobes and demonstrated a simple eBPF example that instruments kernel-space HTTP servers to extract HTTP headers—without modifying the source code. In today's blog, we shift our focus to uprobes, which enable similar instrumentation for user-space applications, addressing use cases that kprobes cannot cover.

Much like how kprobes allow us to attach eBPF programs to kernel functions, uprobes let us hook into functions in user-space binaries. This provides powerful observability capabilities for modern applications written in languages like Go. In particular, we'll explore how to instrument Go binaries by attaching uprobes to standard library functions (e.g., net/http, google.golang.org/grpc) and user-defined logic.

To hook into a Go function using uprobes, we must understand how Go lays out its function arguments in memory. This is especially important for Go 1.17+, which introduced a register-based calling convention that changed how arguments are passed between functions. We'll learn how to inspect process memory and extract function arguments by understanding Go's calling conventions, memory addressing, and pointer dereferencing techniques.
Go binaries are increasingly common in cloud-native environments. Instrumenting them at runtime using eBPF uprobes enables deep observability without modifying or recompiling source code—making it ideal for production debugging and monitoring.

In this post, we will:

• Understand how Go passes function arguments (e.g., int, string, struct) in Go 1.17 and beyond
• Use tools like nm and objdump to locate function symbols in Go binaries.
• Attach uprobes to Go functions and extract runtime arguments.
• Use delve to debug Go programs and inspect memory layout for function hooking.
• Apply these techniques to parse gRPC headers from Go-based gRPC services using eBPF uprobes.

By the end of this post, you'll have a solid understanding of how to leverage uprobes for deep introspection of Go applications, opening up new possibilities for observability and debugging in production environments.

Architecture Overview

Prerequisites and Setup

Before diving into the technical details, let's establish our testing environment and the tools we'll need throughout this tutorial.

System Requirements

Linux x86_64 (kernel 4.1+ for uprobe support)
Root privileges or appropriate capabilities for eBPF operations

Development Tools

Rust: Our userspace framework for eBPF program development and attachment.

Go 1.17+: Target language for our instrumentation examples
delve: Go debugger for inspecting memory layouts and validating our approach

binutils: For nm, objdump, and other binary analysis tools.

Installation and Setup

Install delve if you haven't already:

go install github.com/go-delve/delve/cmd/dlv@latest

Enable ptrace for delve to attach to running processes:

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Sample Go Application

We'll work with two Go applications throughout this tutorial:

sample_go: A simple Go program designed to help us analyze Go's memory layout and calling conventions for different argument types (int, string, struct).

sample_grpc: A Go gRPC server that we'll instrument to extract gRPC headers using our Rust+eBPF tracer.

Rust + eBPF

trace-grpc-headers. Go through the readme to get it up and running

Go Calling Convention: The Register-Based Revolution (Go 1.17+)

Note: This blog focuses specifically on the amd64 architecture.

Starting from Go 1.17, the Go compiler adopted a register-based calling convention, moving away from the traditional stack-only approach used in earlier versions. This change significantly impacts how we extract function arguments in eBPF uprobes.

The Register Allocation Strategy

Go 1.17+ uses a predefined set of 9 registers for passing function arguments on x86_64:

RAX, RBX, RCX, RDI, RSI, R8, R9, R10, R11

Our Instrumentation Strategy

To successfully extract function arguments in our eBPF uprobe, we need to follow a systematic approach:

1. Locate the target function: Get the offset address of the Go function we want to instrument
2. Analyze argument layout: Determine whether arguments are passed in registers, on the stack, or a combination of both
3. Map arguments to locations: Identify which specific registers or stack offsets contain our target data

Once we have this information, we can accurately probe the respective Go function when it's invoked and extract the arguments we're interested in.

Essential Reading

This approach is based on Go's official ABI specification. I highly recommend reading the Go ABI Internal Documentation to understand the detailed rules governing how Go passes arguments to functions.
With this foundation in place, let's move on to the practical steps of locating functions and determining memory offsets.

Locating Functions in Go Binaries

Before we can attach uprobes, we need to identify the specific functions we want to instrument within the Go binary. Go provides several approaches for function discovery.

Symbol Tables and Debugging Information

Go binaries contain symbol tables that map function names to their memory addresses. However, the availability and format of these symbols depend on how the binary was compiled.

In our case,lets compile it with

# Preserves all symbols and disables optimizations
go build -gcflags="all=-N -l" -o sample

Finding Target Functions

Use these tools to explore available functions in your Go binary:

# List all functions with addresses:
nm -n myapp | grep " T "

# Search for specific function:
nm -n myapp | grep "main.hello"

you can also use objdump or go tool to extract the info

Determining Memory Offsets

Once we've located our target function in the symbol table, we need to calculate the correct offset address for uprobe attachment. This offset represents the function's location within the process's virtual address space.

Calculating Function Offsets

The offset address is part of the virtual address space memory layout. We attach our uprobe to this calculated offset address.

Step 1: Get the function's virtual address

nm -n server | grep MyHandler

Output:

0000000000498f60 T main.MyHandler

Step 2: Find the base load address

readelf -l server | grep "LOAD"

Output:

LOAD 0x0000000000000000 0x0000000000400000 0x0000000000400000

Step 3: Calculate the offset

offset = function_address - base_load_address
offset = 0x498f60 - 0x400000 = 0x98f60

Programmatic Offset Calculation

In our Rust userspace program, this offset calculation is handled by helper functions:

// Calculate offset for the target function
let offset = calculate_function_offset(&binary_path, "main.MyHandler")?;

// Attach uprobe to the calculated offset
go_sk
    .progs
    .handle_hello_int
    .attach_uprobe(false, pid, &binary_path, offset as usize)?;

Deep Dive: Go Argument Passing Patterns

Let's examine how Go passes different data types as function arguments, following the official ABI specification.

Integers

Simple types occupy one register each:

func hello_int(x int) int {
    fmt.Printf("hello_int %d\n", x)
    return x
}

eBPF extraction:

 int x = (int)ctx->ax;

Important: If the function has a receiver, the receiver takes RAX and arguments shift to subsequent registers.

Strings: Pointer + Length Structure

Go strings are internally represented as:

type string struct {
    ptr *byte //pointer to data
    len int // Length
}

For a function with multiple arguments:

func hello_int_string(x int, y string) int {
    fmt.Printf("hello_int_string %d %s\n", x, y)
    return x
}

eBPF extraction:

    int x = (int)ctx->ax;            // First argument (int) -> RAX
    void *str_ptr = (void *)ctx->bx; // String data pointer -> RBX
    long str_len_raw = ctx->cx;      // String length (as long) -> RCX

Structs: Value vs Pointer Passing

Struct by value: Individual fields are assigned to registers sequentially

"If T is a struct type, recursively register-assign each field of V."

Struct by pointer: Only the pointer occupies one register

"If T is a pointer type, assign V to register I and increment I."

The choice dramatically affects extraction strategy. See our sample code for both scenarios.

Slices: Three-Field Structure

Go slices contain three components:

type slice struct {
    ptr *elementType  // Data pointer
    len int          // Element count  
    cap int          // Capacity
}

Each field occupies consecutive registers when passed by value.

Floating Point Numbers: The XMM Challenge

Floats use separate XMM registers (X0-X14), not general-purpose registers. This creates limitations:

• Extractable: Floats in structs passed by pointer (dereference from memory)
• Not directly extractable: Individual float arguments or floats in value-passed structs

eBPF currently cannot access XMM registers directly. See this StackOverflow discussionfor details.

Workaround: Use pointer-based struct passing for float extraction.

Practical Application: gRPC Header Extraction

Now comes the exciting part—applying everything we've learned to solve a real-world problem: extracting gRPC headers from a running Go service without touching the source code.

The Challenge: gRPC vs HTTP/1.1

Unlike our previous blog where we parsed HTTP/1.1 headers (which are plain text), gRPC presents a tougher challenge. gRPC headers are compressed using HPACK, making direct parsing from network packets nearly impossible.
But here's where our uprobe knowledge becomes powerful—we can intercept the headers after they've been decompressed by the Go runtime.

Finding the Right Function to Hook

The key insight came from studying how observability tools like Pixie solve this problem. However, their examples target older Go versions and don't work with Go 1.17+'s register-based calling convention.
After digging through the gRPC-Go codebase, I identified the perfect target:

func (t *http2Server) operateHeaders(
    ctx context.Context, 
    frame *http2.MetaHeadersFrame, 
    handle func(*ServerStream)
) error

This function in the grpc-go package processes decompressed header fields just before they're sent to the client—exactly what we need!

Detective Work with Delve

But where does the frame argument live—register or stack? Time for some debugging:

dlv attach 366200
(dlv) break google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders
(dlv) continue

# When breakpoint hits...
(dlv) args
frame = ("*golang.org/x/net/http2.MetaHeadersFrame")(0xc00020e0c0)

(dlv) regs
Rdi = 0x000000c00020e0c0  # Bingo! Frame pointer is in RDI

Perfect match! The frame pointer lives in the RDI register, exactly as our calling convention knowledge predicted.

The Structs We Need to Parse

Armed with the register location, we now need to parse two key structures:

1. MetaHeadersFrame : Contains the frame metadata
2. Fields : Individual header name-value pairs

The eBPF Implementation

Using our memory layout knowledge from the previous sections, we can now traverse these structures in eBPF:

void *frame_ptr = (void *)ctx->di;

Try it yourself

Full Working code: https://github.com/maheshrayas/blogs/tree/main/ebpf/04-tracing-grpc-headers/code

The repo includes:

• Sample gRPC server with various header types
• Complete eBPF program with detailed comments
• Rust userspace code for uprobe attachment
• Step-by-step instructions to reproduce the results : readme.md

What You'll See
Currently, the basic implementation outputs header data to the kernel trace pipe:

Currently it prints all the headers key, value in /sys/kernel/tracing/trace_pipe, we can also pass these headers to userspace using Maps(RingBuffer or PerfEventArray) just like we did in our pervious example.

Debugging, Troubleshooting and Learnings

Working with Go uprobes can be challenging, especially when dealing with memory layout parsing and argument location detection. Here are the key issues you'll likely encounter and approaches to solve them.

Memory Layout Parsing: The Silent Killer

The most frustrating bugs come from incorrect memory address parsing. A single byte offset error can lead to:

• Garbage data extraction
• Kernel panics in extreme cases
• Silent failures with no obvious symptoms

Best practices:

• Always validate extracted pointers before dereferencing
• Use boundary checks in your eBPF programs
• Test with simple data types first, then move to complex structs. Thats how I got started doing a simple types and I have attached the sample for you to familar.

Register vs Stack: The Ongoing Mystery

Determining whether arguments are passed in registers or on the stack remains tricky. While I used delve for this analysis and my understanding on ABI specs, I'm not entirely confident this is the most reliable approach.

Potential alternative approaches (still researching):

• DWARF debug information: Parse function signatures and calling conventions from debug symbols
• Static analysis using go tool compile -S
• Assembly inspection with objdump -d

Call for input: If you have experience with static analysis techniques for determining Go argument passing, I'd love to hear your insights! Feel free to reach out or contribute to the repository.

Conclusion

Throughout this post, we've journeyed from understanding Go's register-based calling convention to successfully extracting gRPC headers from running applications using eBPF uprobes. We've learned how to locate functions in Go binaries, calculate memory offsets, parse different argument types, and apply these techniques to solve real observability challenges.
The techniques we've covered form the foundation for powerful runtime introspection capabilities. By combining uprobe instrumentation with efficient userspace communication through ring buffers, we can build sophisticated observability tools that operate without any application code changes.

What's Coming Next

I'm currently building a comprehensive observability tool that combines both gRPC and HTTP/1.1 header parsing capabilities, designed specifically for Kubernetes environments. This tool will provide complete HTTP path visibility and distributed tracing across your entire service mesh—all through eBPF instrumentation.
But that's just the beginning. My next blog post will be even more exciting, where I'll share a complete open-source project ready for production use in Kubernetes environments. We'll explore how to deploy eBPF-based observability at scale, handle multi-node clusters, and use the tool to increase the kubernetes security.

Stay tuned for the next installment where theory meets production reality. The best is yet to come!