DEV Community

Constant Itis
Constant Itis

Posted on

Why a Reusable Agent Environment Shouldn't Be a Filesystem Snapshot

At the end of Part 2, the agent had done real work inside a Bench. It installed a toolchain. It compiled something. It got a pipeline actually working. The task was complete and the Bench was disposable, so it vanished. So did the work.

I want to keep it.

The question is what "it" even is.

The obvious move is to save the Bench. Freeze its filesystem. Run podman commit on the container to turn it into an image, then reuse that image next time. It was the obvious move, so it is the one I reached for. One command, and it obviously works.

It is a trap.

There are three concrete reasons a snapshot is the wrong thing to keep.

First, it captures debris. A Bench at the end of a task is full of accidental state. Package managers leave caches behind. Build artifacts linger in temporary directories. You have log files from failed attempts. There is the half-finished experiment that did not make it to production. There is the failed apt install that came before the one that worked.

If you snapshot the filesystem, you freeze all of that noise together with the one thing you cared about. Six months later, you cannot tell why any given file in that image exists. You are shipping garbage because the snapshot does not distinguish signal from noise.

Second, it can freeze secrets. While the Bench ran, the workload may have read a granted file or fetched an API token over its allowed network. A filesystem snapshot bakes whatever was sitting on disk into a durable, reusable artifact.

Now your "reusable environment" is something you will copy around that quietly contains a credential. Disposable state became permanent state. This includes the parts that were supposed to stay disposable. You have created a security liability by accident because you saved the context, not just the configuration.

Third, it becomes an opaque parallel truth. You can inspect a saved image. You can even diff two of them. But a filesystem diff shows you what changed, not what was intentional, what was required, what was accidental, or what is safe to promote. It tells you a file appeared. It cannot tell you whether that file was ever supposed to be there.

And it becomes a new thing the operating system has to trust. It is a user-writable artifact standing in for a clean base. I sealed the host in Part 1 specifically to not have opaque mutable state I have to trust. A saved Bench smuggles that trust right back in through the front door.

I do not actually want the Bench's history.

I do not care about the order it installed things in. I do not care about the temp files it left behind. I want its intent.

What did this environment declare it needed? Which base did it start from? Which packages, by name? Which network destinations? Which directories it was granted?

That set is small. It is readable.

I want the intent, not the history.

That declared intent is what I call a Workshop.

A Workshop is a recipe, not a snapshot. When I decide a Bench is worth keeping, I promote it. Promotion does not freeze bytes. It records the recipe.

The recipe contains the seed it started from. It contains the packages it declared, using apt and pip, by name, with versions where I pin them. It contains the network profile it used. It contains the directories it was granted, recorded as the approved upper bound on what a launch may touch, not as standing ambient access the Workshop carries around.

The whole recipe is small enough that I read it. I read it as a plain diff. I confirm it before it is saved.

What gets stored is the meaning. This meaning is owned by the privileged supervisor, not a blob owned by whatever ran the agent. The agent is a client. It does not own the state. It requests resources. The supervisor manages the lifecycle.

To use a Workshop later, the operating system does not restore an image. It rebuilds a fresh environment from the recipe.

Fresh base. Install the declared packages again. Re-apply the declared grants, within the bounds the recipe approved. Clean every single time.

No debris, because debris was never recorded. No frozen secret, because the recipe never contained one.

The environment is regenerated from its meaning instead of resurrected from its corpse.

This approach solves the three problems with snapshots. It excludes noise because the recipe only lists what is required. It excludes secrets because the recipe only lists declarations, not runtime artifacts. It excludes opacity because the recipe is a plain text file that anyone can read and verify.

None of this is new.

Nix and Guix have been rebuilding environments from declarations for years. A Dockerfile is a recipe. Configuration management tools have argued "declare the end state, do not snapshot it" for a long time.

So the embarrassing question is fair.

Isn't a Workshop just a Dockerfile I generate from a session?

Honestly, kind of. A Workshop is a declarative environment spec, and the world has plenty of those. If I had invented a new recipe file format, that would be the least interesting thing about this.

The interesting part is not the format. It is the boundary the recipe crosses to exist. A Workshop is captured by promoting a real, messy, live agent session across a line. On one side is whatever the agent actually did, imperative and unreviewed. On the other is clean, human-reviewed, durable state the system is willing to keep. That promotion goes through a human confirmation step, and the result lands in a canonical record owned by the privileged supervisor. The agent cannot write or replace that record directly. It can propose. It cannot author the durable truth.

The agent might have run twenty commands to get a library installed correctly. It might have had to fix a dependency conflict along the way. The Workshop records the final, correct intent, not the twenty attempts.

The promotion boundary is the contribution, not the file format.

Reuse means re-derivation.

When I clone a repository that uses a Workshop, I do not get a pre-built image. I get the recipe. The operating system reads the recipe. It pulls the base seed. It installs the packages. It applies the grants.

It re-derives from the same recipe on my machine, on a CI server, or in the agent's next Bench. How identical the result is depends on how much I pinned. Name a package without a version and I get whatever the approved index serves that day, which is re-derivable but not bit-for-bit reproducible. That is a deliberate tradeoff, not Nix-level determinism.

This is the point of declarative environments. You keep a declaration and re-derive from it, instead of preserving a blob. How close two derivations land depends on how completely the declaration pins its inputs.

But there is a cost.

Rebuilding cleanly from the recipe is correct. Done naively, it is also absurdly wasteful.

If I throw the Bench away and rebuild the Workshop tomorrow, I download and compile the exact same toolchain from scratch all over again. I fetch the same source code. I run the same configure scripts. I link the same libraries. Correct, clean, and slow. A complex Workshop that took minutes to build takes those same minutes again.

I want to be careful about what I am NOT claiming here. The recipe is the source of truth, and it must always be able to re-derive the environment from scratch. That property is non-negotiable. But nothing in it says I have to pay the full derivation cost on every launch. The declaration staying authoritative and the launch being fast are not actually in conflict.

I want the cleanliness of rebuilding from a recipe. I do not want to pay the full cost of the rebuild every single time.

How do I keep the cleanliness of rebuilding from a recipe without paying the full cost of the rebuild every single time?

Top comments (0)