DEV Community

Cover image for Instrumenting Rust TLS with eBPF
Coroot
Coroot

Posted on • Originally published at coroot.com

Instrumenting Rust TLS with eBPF

eBPF collects telemetry directly from applications and infrastructure. One of the things it does is capture L7 traffic from TLS connections without any code changes, by hooking into TLS libraries and syscalls.

Works great for OpenSSL. Works for Go.

Then rustls enters the picture and everything stops being obvious. With OpenSSL, everything is nicely wrapped:

SSL_write(ssl, plaintext)
└─ write(fd, encrypted)

SSL_read(ssl, plaintext)
└─ read(fd, encrypted)
Enter fullscreen mode Exit fullscreen mode

From eBPF’s point of view this is perfect:

  • hook SSL_write, stash plaintext
  • write() fires immediately → same thread → you know the FD
  • same idea for reads Everything happens inside one call. Correlation is trivial.

Rustls does things differently

Rustls doesn’t own the socket and never calls read or write itself. It works on buffers, and the application (or runtime) is responsible for actually moving bytes over the network.

The API reflects that separation pretty clearly:

// application writes plaintext into rustls
writer.write(plaintext);

// rustls produces encrypted bytes and writes them via io::Write
conn.write_tls(&mut socket);

// application reads encrypted bytes and feeds them into rustls
conn.read_tls(&mut socket);

// rustls decrypts and updates internal state
conn.process_new_packets();

// application reads decrypted data
reader.read(plaintext_buf);
Enter fullscreen mode Exit fullscreen mode

So instead of one call doing everything, you get a pipeline:

  • plaintext is buffered first
  • encryption happens later
  • syscalls happen outside of rustls
  • decryption happens before the app reads

The key difference for eBPF:

  • writes: syscall happens after plaintext
  • reads: syscall happens before plaintext

So the OpenSSL-style correlation only works in one direction.

Writes work as usual

On the write side, nothing fundamentally new is needed. You hook Writer::write, stash the plaintext, and correlate it with the following sendto. The ordering is preserved, so the same approach as OpenSSL still applies here.

Reads are inverted

The read path is where things really break.

recvfrom(fd, encrypted_buf, ...);   // happens first

conn.read_tls(&mut socket);
conn.process_new_packets();

reader.read(plaintext_buf);         // plaintext appears here
Enter fullscreen mode Exit fullscreen mode

By the time we see plaintext, the syscall is already gone.

So the logic has to be reversed. Instead of:

  • “see plaintext → wait for syscall”

we do:

  • “see syscall → remember it → use it later”

Concretely:

  • on recvfrom → stash FD per thread
  • on reader.read → pick up that FD and attach it to plaintext

It’s basically reverse correlation. Not pretty, but it matches how rustls works.

When “ret=1” doesn’t mean 1 byte

This one took longer than expected. We reused the OpenSSL-style exit probe:

ret = PT_REGS_RC(ctx)

The probe fired, but results were weird:

ret=1
ret=0
Enter fullscreen mode Exit fullscreen mode

Which made no sense for a read. Turns out Rust returns Result like this:

  • rax → success or error flag
  • rdx → actual number of bytes

So we were reading rax and treating it as a size. Meaning:

  • ret=1 → actually an error
  • ret=0 → success, but size is somewhere else Fix was straightforward once understood:
if (PT_REGS_RC(ctx) == 0) { // success
    size = ctx->dx; // actual byte count
}

Enter fullscreen mode Exit fullscreen mode

Classic case of “everything works, but the numbers are garbage”.

Finding rustls in binaries

Rust symbols are heavily mangled:

_ZN55_$LT$rustls..conn..Writer$u20$as$u20$std..io..Write$GT$5write17h0ee1e61402b1a37cE

It looks messy, but it encodes the full path: rustls::stream::Writer implementing std::io::Write::write.

The tricky part is that mangling isn’t stable:

  • different compiler versions use different schemes (legacy vs v0)
  • optimizations and stripping can change what’s left in the binary

So matching exact names is fragile.

Instead, we:

  • check ELF .comment for rustc to detect that the binary was built with Rust
  • then scan symbols for patterns like “rustls”, “Writer”+”write”, “Reader”+”read”

Not perfect, but reliable enough in practice.

Results

Coroot is an open source observability tool that uses eBPF to simplify setup. Because we instrument rustls at the library level, not the frameworks, this works across most Rust clients that use rustls under the hood.

That includes HTTP stacks like hyper when paired with rustls (hyper-rustls, and frameworks like axum or warp when configured with rustls), database clients like sqlx when using its rustls TLS feature, and any async Rust service using tokio-rustls.

No code changes, no SDKs, no wrappers.

For Rust apps using OpenSSL via native-tls or openssl, the existing OpenSSL instrumentation already works. rustls was the missing piece.

Below is an example of a service talking to MySQL over TLS. Coroot shows the actual queries even though everything on the wire is encrypted.

If you’d like to give our open source tool a try and simplify your own observability, you can check it out at here on Github. You can also view this guide and other open source observability articles on our blog.

Top comments (0)