[TIL] Debugging Go in Production with eBPF

Refer article “Debugging Go in prod using eBPF” by Zain Asgar : https://blog.pixielabs.ai/blog/ebpf-function-tracing/post/

Preface:

There are actually many ways to debug on Golang. Whether it's the most commonly used logging or through delve or even GDB, they are all common ways to debug Go. But recently, I saw this article and thought that the eBPF (extended Berkeley Packet Filter) mentioned in it was quite interesting, so I'll do a simple summary here.

Debugging in Go

Generally speaking, there are the following ways to debug in a Go application:

Add log:

As everyone is familiar with, use fmt.Println() or log.Println().

Using debugger (delve or GDB)

Whether it's through the built-in delve in vscode, or through GDB for debugging. Debugging through a debugger actually consumes a lot of system resources and often interrupts the overall application's operation.

Tracing

This refers to using external tracing tools, whether it's USDT or strace, they are both a way.

What is eBPF

eBPF (extended Berkeley Packet Filter) according to the official website (https://ebpf.io/), is a technology that allows applications to run on a sandbox, and can view relevant data through Syscall hook without modifying any kernel source code.

Application areas include:

Security: Through eBPF, it can be used as an intermediate layer between system calls (Syscall) and hardware devices.

Tracking and Profiling: Since Linux systems all provide eBPF interfaces, you can directly track and profile your application through this. (with very low latency).

This is also the main technique mentioned in this article.

Why use this?

Since all the tools for Debugging Go App are mentioned, it must be compared with other methods. You can see that eBPF has the following characteristics:

• Performance Impact is very low (disruption is also very low), similar to Tracing tool. But like GDE and Delve, it can trace Application Code.
• However, it cannot be used for distributed system measurement, because one eBPF can only be used for debugging and tracing the application you need to hook.

More introduction: