How Cilium Cut Restart Times by Reusing Compiled BPF Objects
If you’ve ever worked with eBPF programs in production, you know that compiling them can be a time-consuming bottleneck, especially during restarts. In Cilium, a popular networking and security project, the agent used to recompile all BPF programs from scratch on every restart. This led to delays stretching into minutes and even caused missed regeneration deadlines in continuous integration (CI) environments.
In this article, we’ll explore the root cause of this inefficiency, the technical details of the caching solution introduced, and the broader engineering lesson about build artifact reuse.
The Problem: Unnecessary Recompilation on Every Restart
Cilium’s agent manages a set of BPF programs that it compiles using clang. Before the change, each time the agent restarted, it triggered a full recompilation of all BPF programs. This was painfully slow — in some cases, endpoint template compilations took up to 553 seconds.
Why did this happen? Because the agent had no way to detect if the BPF source code or its build environment had changed since the last compilation. Without such detection, it always assumed it needed to recompile everything.
This behavior caused two main issues:
- Long startup delays: Developers and CI pipelines had to wait for lengthy recompilations even if nothing changed.
- Missed deadlines: In CI, where quick feedback loops matter, these delays led to missed regeneration deadlines and slowed down testing.
Root Cause: Lack of Input Hashing and Cache
The core reason for the redundant recompilation was the absence of a mechanism to track build inputs and reuse previously compiled artifacts.
The agent’s compilation process depends on multiple inputs:
- BPF source files
- Clang compiler configuration and environment
- Go build ID (which can affect generated headers)
- Generated headers and other build artifacts
Before this change, none of these inputs were hashed or checked for differences. The compilation was always triggered unconditionally, causing expensive clang invocations even if the inputs were identical.
The Solution: Input Hashing and Caching Compiled Objects
The fix introduced a robust caching mechanism keyed by a hash that uniquely identifies the compilation inputs. Here’s how it works in detail:
Hashing all relevant inputs: The system computes a single hash that combines the contents of all BPF source files, the clang configuration and environment variables, the Go build ID, and any generated headers. This produces a unique fingerprint representing the exact compilation context.
Cache lookup on agent restart: When the agent starts, it calculates this hash and checks if compiled BPF objects for this hash already exist on disk.
-
Reuse or recompile:
- If a matching cached object is found, the agent loads it directly, skipping the expensive compilation.
- If not, it proceeds with a normal compilation and then caches the resulting objects under the new hash.
This caching logic is primarily implemented in pkg/datapath/loader/cache.go. To ensure correctness, new tests were added in cache_test.go and hash_test.go verifying that the hashing accurately reflects input changes and that cached objects are reused appropriately.
This approach ensures that recompilation only happens when something truly changes, drastically reducing restart times.
Takeaway: Hashing Inputs to Cache Build Artifacts Saves Time and Resources
Build systems and runtime components often depend on multiple inputs that rarely change between runs. Without tracking these inputs, systems waste time recompiling or rebuilding unnecessarily.
By hashing all relevant inputs and caching build artifacts keyed by this hash, you can:
- Avoid redundant expensive work
- Speed up restarts and deployments
- Reduce CPU and memory usage during builds
- Improve developer and CI pipeline productivity
In Cilium’s case, this change turned restart times from minutes down to seconds, enabling faster iteration and more reliable CI.
If you’re building systems that compile code or generate artifacts dynamically, consider implementing a similar input hashing and caching mechanism.
This improvement came from the pull request titled "loader: Reuse compiled BPF objects across agent restarts" in the cilium/cilium repository.
Read the full breakdown on nossletter.tech.
Originally published on Nossletter.
Top comments (0)