An agent renamed two methods, updated two files, and missed the third. The test suite went from green to two failures in one commit. Getting back to working code took one push and 1028 milliseconds, and it did not involve a sandbox at all.
Sponsored by Tensorlake. All experiments, numbers, and terminal output in this piece are my own, run live on their infrastructure.
The mistake agents actually make
Bad file edits are not the interesting failure. A single wrong line gets caught by the next test run and fixed in place.
The expensive failure is the coherent-looking refactor that is wrong across files. I gave an agent a normal task: rename the verbose DocumentStore methods and make build_index take a mapping instead of a list of pairs.
It did exactly that, in store.py and indexer.py. It did not touch api.py, which still passes a list.
v1: 2 passed
v2: 2 failed
AttributeError: 'list' object has no attribute 'items' src/indexer.py:6
Two of four files changed. Every changed file is internally consistent. The break lives in the one file the agent did not open, which is exactly the case code review is worst at catching and exactly what you get when an agent works from a partial view of the repository.
You cannot prompt your way out of this reliably. What you need is a cheap, exact way back.
Byte-exact, not approximately
I pushed the working tree, let the agent push its refactor, then pushed the original tree back.
v1 = client.push_worktree(repo, root="v1", branch="main", message="working service")
v2 = client.push_worktree(repo, root="v2", branch="main",
message="agent refactor", expect_oid=v1.commit)
rb = client.push_worktree(repo, root="v1", branch="main",
message="rollback", expect_oid=v2.commit)
v1 commit=b564a708 tree=d5aa2a70 1136 ms
v2 commit=9da6a695 tree=40144a43 1036 ms <- build broken
rollback commit=26262ee3 tree=d5aa2a70 1028 ms
Look at the tree hashes rather than the commits. d5aa2a70cae2e48f507367d6898e9ae8a81d4d4f before, and the identical hash after. That is not "the tests pass again", it is a cryptographic statement that the content is the same content.
The commit hash is new, and that matters. Nothing was rewritten, no history was force-pushed away. The broken state stays in the log where you can go and read what the agent did.
push objs=14 main: 00000000->b564a708
push objs=14 main: b564a708->9da6a695
push objs=14 main: 9da6a695->26262ee3
For an autonomous system that is the right shape. You want to undo the damage and keep the evidence, because the failed attempt is your training signal for what to stop the agent doing next time.
The distinction that gets blurred
Sandbox snapshots and versioned code get talked about as if they solve the same problem. They do not, and picking the wrong one is expensive in both directions.
I ran the same accident on the sandbox side: booted a machine, wrote the code, dropped an 8 MB model artifact next to it, checkpointed, then broke the code and deleted the artifact.
boot 1542 ms
checkpoint 1390 ms snapshot=dq4oexvl69cg
restore into new sandbox 2018 ms
restored: "v1 code", model.bin present, src/ present
The snapshot brought back the 8 MB artifact. The git rollback would not have, because it was never pushed. That is the whole difference in one file.
A snapshot restores the machine: everything on disk, tracked or not, and it hands you a new sandbox id. Versioned code restores the tree you chose to record, needs no machine to exist, and hands you a new commit on a history you can read.
No sandbox was harmed in this rollback
The part worth pausing on: the entire rollback sequence ran with no sandbox alive.
That is not a detail, it is the operational point. Agent runs are ephemeral. The sandbox that made the mess is usually gone by the time a human notices, torn down at the end of the run or timed out. If your only undo is a snapshot of that machine, your recovery story depends on having predicted the failure well enough to checkpoint before it happened.
Code history does not expire with the run. You can roll back an agent’s work an hour later, from a laptop, with nothing running, and materialise the good tree into a fresh sandbox whenever you next need one.
Rolling back the code was also the fastest of the three operations, at 1028 ms against 1390 ms to take a checkpoint and 2018 ms to restore one.
Wiring it into the agent loop
None of this helps if a human has to notice the breakage. And the stronger move is not to roll main back at all: give each agent a private workspace, and let nothing reach main until it has passed.
client.fork(f"ws-{agent}", base_repo=BASE)
tip = client.push_worktree(f"ws-{agent}", root=work, branch="main",
message=task, expect_oid=base_tip)
if tests_pass(work):
pre = client.merge(BASE, ours="main", theirs=tip.commit,
preflight=True, deep=True)
if pre.clean:
client.merge(BASE, ours="main", theirs=tip.commit,
materialize=True, message=f"promote {task}")
I ran two agents against the same base, one of them the broken refactor from earlier:
main = 1a27a30c
agent B rename refactor fork 183 ms commit 00b2d709 tests FAIL
not promoted; main still 1a27a30c
agent A add size() + test fork 161 ms commit 82fc24fe tests PASS
preflight clean 179 ms
promoted; main -> 82fc24fe
The broken refactor never reaches main. It still exists, in its own workspace, as a commit you can read and diff, which is what you want when you are trying to work out why an agent keeps failing a task. But main only ever moves to a tree something mechanical has approved.
Three properties make this safe rather than merely tidy.
The workspace is a repository fork, not a sandbox fork, which is the distinction this whole piece turns on. It copies refs rather than a filesystem, and cost 161 and 183 ms in these runs, cheap enough to do per task. The agent starts from the exact base commit and its mistakes are contained by construction rather than by discipline.
If the machine matters as much as the code, take a sandbox checkpoint at the same moment you push the workspace. Both halves of the state then sit behind the same gate, and you still promote only what passed.
expect_oid is a compare-and-swap on the branch you are pushing to. In this loop that is the workspace’s own main, which is the right scope for it: the workspace belongs to one agent, so the guard is there to stop two writers racing on a single branch.
The shared base is looked after one layer up, at promote time, and it holds. I let one agent promote while another was still working, then promoted the second. The merge simply stopped being a fast-forward and became a real three-way merge against the original base, and it came back clean. materialize re-resolves main when it runs rather than trusting the earlier preflight, so a promote that queues behind another one lands on top of it rather than over it. Two guarantees at two layers, each scoped to the job it is doing.
The promotion is gated on a deep preflight. preflight=True computes the merge and writes nothing, and deep=True makes it do the real line-level merge rather than a tree comparison, which is the difference between a correct answer and a conservative guess. That check cost 179 ms.
The gate does not have to be a test suite. A type check, a lint pass, a smoke request, a diff size threshold: anything returning a boolean works. The point is that the agent’s work is provisional until something mechanical says otherwise.
So when do you still need the byte-exact rollback from earlier? For what already landed. An agent promoted before you had this gate, a human pushed directly, a change passed its tests and broke something they did not cover. Isolation prevents; rollback repairs. You want both, and they are the same primitive pointed at different moments.
Use both, for different jobs
This is not an argument that snapshots are worse. They answer a question git cannot: what did this machine look like, including the things nobody committed.
Reach for the snapshot when the machine is the state. A long dependency install you do not want to repeat, a GPU environment that took real effort to get right, a heisenbug you want frozen exactly as it occurred.
Reach for versioned code when the code is the state, which for a coding agent is nearly always. It is cheaper, it outlives the sandbox, and it gives you a line-by-line diff of exactly what changed.
Snapshots are simply a different shape from routine code changes, and it is worth being deliberate about which you reach for. A filesystem image is stored whole, compared as a whole, and exists for the moments you chose to capture. That is exactly what you want for an environment, and more than you need for a source tree you are changing many times an hour.
What to watch for
Three things bit me or nearly did.
Pass expect_oid on every write. Each push above pins the commit it expects to replace, so a rollback lands on the state you actually looked at rather than one that moved while you were deciding. It is one argument, it costs nothing, and it turns a race into a clean error you can retry.
Rollback is only as good as what you pushed. Untracked build output, caches and model files are not in the tree and will not come back. Decide deliberately what belongs in the repository and what belongs in a snapshot.
The operations log is detailed, down to ref transitions, object counts and pack bytes, and it attributes each operation to the credential that made it. So give each agent role its own API key. That log then becomes per-agent provenance: who moved which ref, when, and by how much.
Final Thoughts
The useful mental model is that a coding agent needs a version control system, not a time machine for computers.
If you are building anything that lets a model write to a real codebase, the recovery path deserves as much design as the prompt. Record the tree before the agent runs, pin every write to the commit you expect, and keep the broken attempt in the history where you can learn from it. Then a bad refactor at 3am is a one second push instead of an incident.
Top comments (0)