Remote Execution, Explained for People Who Just Run `make`

Most engineers think of the build as something that happens on their laptop. You type a command, the fans spin up, and a few minutes later you have a binary. That model is so ingrained that we rarely question it — until the codebase gets big enough that it quietly collapses.

At a certain scale, the laptop build is too big for one machine, too slow to repeat, and too wasteful to run from scratch on every change. Ten engineers rebuild the same unchanged files ten times before lunch. A clean build takes forty minutes. CI is the bottleneck for the whole team. Remote execution is the answer large codebases reach for, and once you understand it, a lot of otherwise-baffling build-system design suddenly makes sense. Here is the mental model, minus the vendor jargon.

The core idea: the build is a graph of pure functions

Stop thinking of a build as a sequence of commands and start thinking of it as a graph. Each node is an action: a single command plus the exact set of inputs it reads and outputs it produces. Compiling one file is an action. Linking is an action. Generating code from a schema is an action. The edges are dependencies — the linker action consumes the object files the compile actions produced.

Here is the key insight that everything else rests on. If you treat each action as a pure function — same inputs always produce the same outputs — then two things become possible that are not possible with a pile of shell scripts. You can run independent actions anywhere, on any machine, in parallel. And you can cache the result of any action, because identical inputs mean you already know the output.

Remote execution is what you get when you take those two possibilities seriously.

What an action actually is

Concretely, an action is described by its command, the content of every input it depends on, and the environment it runs in. Crucially, inputs are identified by the hash of their content, not their path or timestamp:

Action {
  command:       ["clang", "-c", "parser.c", "-o", "parser.o"]
  input_digests: { "parser.c": sha256(abc1...),
                   "parser.h": sha256(def2...) }
  platform:      { os: linux, arch: x86_64, ... }
}

action_digest = sha256(serialize(Action))

That action_digest is the core lookup key. It fingerprints everything the build system has declared as relevant to the output. Correctness therefore depends on complete input and environment declarations: an undeclared dependency is invisible to the key. The build system computes the digest and asks a shared service one question: have you seen this exact declared action before?

• Cache hit: the service returns the output blobs immediately. No compilation happens at all — not locally, not remotely. You just fetched the answer.
• Cache miss: the service schedules the action on a remote worker, the worker runs the command in a clean sandbox, uploads the resulting parser.o to a content-addressed store, and records the action-to-output mapping so the next person gets a hit.

Multiply that across a team and a CI fleet and the effect is dramatic. The first engineer to build a given commit pays the cost; everyone else, and every CI run on that commit, gets cache hits. Work is done once and shared.

The two pieces of machinery

Underneath, remote execution rests on two services that are simpler than they sound.

The first is a content-addressed store (the CAS). It is a blob store keyed by the hash of the content. Put bytes in, get a hash back; present a hash, get the bytes. Because the key is the content's fingerprint, identical files can be deduplicated and a retained blob cannot become stale under the same digest. Storage policies may evict blobs, so clients still have to handle missing content. Source files, intermediate objects, and final binaries can all live here.

The second is the action cache: a map from action_digest to the set of output digests it produced. That is the lookup that turns "compile this" into "you already compiled this, here are the bytes."

Bazel and Buck2 both support the common Remote Execution API for these services, which allows compatible deployments to point either client at the same class of remote backend. You do not need to know the protocol to use it, but it helps to know it exists: the build tool on your laptop coordinates the graph while remote workers and shared caches handle much of the expensive execution and storage.

Why this reshapes how build systems are designed

Once you see the build as a graph of cacheable pure functions, the design decisions of modern build systems stop looking arbitrary.

Why does Bazel make you declare every input and output explicitly, instead of letting a command read whatever files it wants? Because the action digest has to capture every input. An undeclared input is an input the cache cannot see, which means the cache will hand you a stale result. Why the obsession with hermetic, sandboxed actions? Because a worker on the other side of the cluster has to be able to run your action with nothing but the declared inputs and get the same answer your laptop would. The strictness that feels like bureaucracy when you are writing a BUILD file is the exact thing that makes the action a pure function — and a pure function is the thing you can distribute and cache.

Where it gets hard

It is not free, and the failure modes are worth knowing before you adopt it.

The network becomes the new bottleneck. You have traded local CPU for cache lookups and blob transfers, so a slow link or a far-away CAS can make a "cached" build feel slower than a local one. Large inputs and outputs strain the CAS and the network; build graphs with giant artifacts need care. And the whole edifice rests on one assumption that deserves its own discussion: that actions are actually deterministic. The moment an action produces different output from identical inputs, the cache starts handing out wrong or useless answers, and the value proposition quietly inverts.

That last point is important enough that it is the subject of its own post. For now, the mental model is the takeaway: a build is a graph of actions, each action is a pure function fingerprinted by its inputs, and remote execution is just the infrastructure for running those functions anywhere and remembering their answers. Once that clicks, the rest of the modern build world — the explicit inputs, the sandboxing, the content hashing — reads as the obvious consequence it is.