The Unix Way — Episode 17
A production server is slow. The senior engineer wants to know which syscall is blocking, on which thread, for how long, without restarting the service. The honest answer in 2026 is: that question has had a clean answer on Solaris and FreeBSD since the mid-2000s, and a clean answer on Linux since 2018. The fifteen-year gap is the story.
FreeBSD: DTrace (In Base Since 2009)
Bryan Cantrill, Mike Shapiro and Adam Leventhal designed DTrace at Sun Microsystems beginning in 2001 and shipped a working implementation in November 2003. Their USENIX 2004 paper, Dynamic Instrumentation of Production Systems, is still the canonical reference. Solaris 10 made DTrace generally available in January 2005.
The principle was rather radical for its time. Existing tracing tools (truss, strace, ptrace) interposed on the traced process; they either stopped it at each syscall or used signals to interrupt it, and either way the act of measurement disturbed what was being measured. DTrace took the opposite approach: instrument the kernel itself with passive probes, fire the probes only when armed, and never block the path of the traced code. The cost of an unarmed probe was zero; the cost of an armed probe was a small constant.
To make this safe in production, the team did something rather elegant: they made the DTrace script language deliberately Turing-incomplete. No unbounded loops, no recursion, no dynamic memory allocation in the probe action. The compiler could prove that every probe terminated within a bounded time and consumed a bounded amount of memory before loading it into the kernel. A misused script could not crash production; it would be rejected at compile time or terminate cleanly at the bound.
DTrace was released under the Common Development and Distribution License (CDDL) as part of OpenSolaris in 2005. John Birrell of the FreeBSD project ported DTrace to FreeBSD, where it landed in 7.1-RELEASE on 6 January 2009. The port covered the kernel provider, the userland USDT (User-Statically-Defined-Tracing) probes, and the full D script language. On a modern FreeBSD host, the number of installed probes runs into the tens of thousands; one enumerates them with dtrace -l.
A complete one-liner that counts open syscalls by process:
dtrace -n 'syscall::open*:entry { @[execname] = count(); }'
Press Control-D and DTrace prints a histogram. The traced processes did not pause; the production load did not skip; the open call observed was the actual open call, not a replay.
Linux: The Licence Wall
The Linux kernel could not adopt DTrace upstream. The structural cause is worth naming clearly, because the popular framing reverses it.
The CDDL is a file-level weak copyleft licence: it requires that CDDL files remain CDDL, but it does not demand that other files in the same project be relicensed. CDDL accepts coexistence with files under any other licence. The GPL is strong copyleft: any work that links GPL code must, as a whole, be distributed under GPL. The GPL does not accept coexistence with any licence that is not itself GPL.
The two are not symmetric. The CDDL would have accepted Linux as a neighbour without conditions. The GPL cannot accept any neighbour that does not relicense itself to GPL. The block was not in the CDDL's behaviour; it was in the GPL's design. One could equally describe the GPL as a licence that absorbs every other licence it touches.
This was not a hypothetical. Sun (and later Oracle) made clear that DTrace would not be relicensed to GPL. The Linux community concluded that the integration of DTrace into the mainline kernel was not legally possible without the GPL's claim on the combined work being honoured, which Sun would not do. A loadable kernel module port from Oracle has existed for some years, but it requires a CDDL kernel module to be loaded into a GPL kernel, a configuration that most distributions decline to ship out of caution about the GPL's reach.
The Linux answer was therefore not a port. The Linux answer was a rebuild, starting from a substrate that had existed since 1992.
Linux: eBPF, Built From BPF
Steven McCanne and Van Jacobson at Lawrence Berkeley Laboratory introduced BPF, the Berkeley Packet Filter, in 1992 (afaik; the design was first described at USENIX in 1993). BPF was originally a small in-kernel virtual machine used by tcpdump to filter packets without copying every packet up to userland. The kernel ran a small bytecode program against each packet header; only matching packets were forwarded. The design was simple, fast and provably safe: the bytecode was bounded and could not loop unboundedly.
For two decades BPF remained a packet-filter substrate. Then, in 2013-2014, Alexei Starovoitov (then at PLUMgrid, now Meta) and Daniel Borkmann (then at Cisco, now Isovalent) rewrote it. The result was eBPF: a more general virtual machine with a register-based instruction set, broader data types, kernel-side maps for state, and verifier guarantees borrowed in spirit from DTrace. The eBPF rework was accepted by David Miller into the networking tree in March 2014 and shipped in Linux 3.18 on 7 December 2014.
The verifier is the safety-critical piece. Before loading, the kernel walks every possible path of the eBPF program and proves that the program terminates, that all memory accesses are bounded, that all pointers are valid for their context, that no loop runs unboundedly. The proof is, on its terms, more elaborate than DTrace's Turing-incomplete language because eBPF chose a more powerful instruction set. The result is the same in production: a misused script cannot crash the kernel.
User-facing tracing tools came in the years that followed.
The BPF Compiler Collection (BCC), led by Brenden Blanco at IO Visor with substantial contributions from Brendan Gregg and many others, arrived in 2015. BCC let one write tracing scripts in Python with embedded C, which the kernel-side runtime would compile to eBPF bytecode and load. The ergonomics were better than raw eBPF assembly; they were still not DTrace.
bpftrace was the closer match. Alastair Robertson (then a researcher) wrote the initial implementation; Brendan Gregg announced it in October 2018 as "DTrace 2.0 for Linux". The bpftrace script language deliberately echoes the D language: probe specification, action block, aggregations, histograms, stack traces. A bpftrace one-liner counting open syscalls by process:
bpftrace -e 'tracepoint:syscalls:sys_enter_open* { @[comm] = count(); }'
The same answer as the DTrace example above. The five letters at the front are different; the shape is identical.
The Bridge: Brendan Gregg
The biographical detail is worth naming. Brendan Gregg had been one of the most prolific DTrace authors at Sun and Joyent. He co-wrote DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD with Jim Mauro in 2011. He moved to Netflix in 2014, where the production fleet runs Linux. He spent the following decade rebuilding, on Linux, the observability he had taken for granted on Solaris. His 2019 book BPF Performance Tools: Linux System and Application Observability is the spiritual sequel to the DTrace book; the same author, the same shape of question, two ecosystems, fifteen years apart.
His public reflection has been measured. In 2018, on announcing bpftrace, he wrote that the goal was not to compete with DTrace but to deliver, on Linux, the production-tracing experience that DTrace had defined. The Linux engineering community has not generally disputed the framing.
The Point
The Unix way is to expose the system to the operator and trust them to ask honest questions. DTrace was the canonical answer in 2003. The shape was: instrument every probe, run on demand, prove safety at load, return aggregated results.
Linux shipped the equivalent shape in 2018, fifteen years later. The engineering on Linux was not harder than the engineering on Solaris in 2003; the verifier work was more elaborate because eBPF chose a more powerful instruction set, but the safety guarantee is comparable. The journey was longer for a reason that had nothing to do with engineering: the original work was released under a licence the Linux kernel could not accept, the rebuild had to start from a different substrate, and the user-facing tools had to be written essentially from scratch.
Two practical takeaways for a working engineer in 2026:
First, on FreeBSD the tracing answer has been settled for the better part of two decades. dtrace -l enumerates the probes; the D language is small enough to learn in a weekend; the production safety is, on the architectural record, the strongest of any tracing system in active use.
Second, on Linux the answer has finally settled too. bpftrace is the closest match to D in syntax and intent; BCC remains useful for richer scripts; perf, ftrace and the underlying tracepoints continue to coexist for narrower use cases. The shape of "ask the kernel anything in production" is now a settled Linux capability, and Brendan Gregg has, between two books, written most of the documentation one will need.
The longer point is not which tracer is better. The longer point is that an architectural idea, however clearly demonstrated, can be delayed by fifteen years on a neighbouring system if the receiving licence refuses coexistence with the giving one. The shape was always the same. The journey was a great deal longer.
Read the full article on vivianvoss.net →
By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.
Top comments (0)