Your Coding Agent's Workspace Is a Cache, Not a Disk: Make Artifact Durability an Invariant

A refactor step ran for forty minutes on a throwaway workspace and produced twelve files. Then the machine went away, the workspace went with it, and the only thing I had left was a diff pasted into a chat log — which, of course, no longer applied cleanly to the tree it came from. I spent the next hour reconstructing work that had already been done correctly, and none of my logs told me which writes had actually survived. Have you ever tried to answer "was that file committed or just written?" from a transcript? You cannot, because "written" and "committed" were never distinguished in the first place.

The week's DEV conversations kept circling whether AI writes our code and whether AI testing drives real organizational change. Both are interesting, but they skip a duller question that decides whether any of it counts: does the artifact outlive the machine that produced it? In this review I treat the agent workspace as an L1 cache with an unspecified eviction policy, and I make durability an invariant that a test can falsify.

What I am assuming, so you can reject the conclusion cheaply

Declared assumptions matter more than the diagram, because the diagram is only true inside them.

1. The remote workspace has no published lifetime guarantee and can be reclaimed mid-step, without a shutdown hook.
2. Reclamation is not a rare fault; on free compute it is the normal end of a workspace, so the protocol must be correct per step, not per session.
3. The agent step is the unit of replay: re-running a step is allowed, re-running half a step is not.
4. Artifacts are small enough to hash and upload (source files, patches, test reports), not multi-gigabyte datasets.
5. Some durable store exists outside the workspace — object storage, a database, a git remote — with atomic replacement or compare-and-swap.

If assumption 1 is false for you, because you rent a persistent volume with an SLA, most of this article is unnecessary. If assumption 1 is true and you have not written down an invariant, you are running on luck.

The invariant, stated so a test can actually fail it

Here is the property I want to hold for every step s and every artifact a that s declares:

Durability invariant. If a manifest for s is visible to a reader, then for every declared artifact a, the content referenced by that manifest is retrievable. Otherwise no manifest for s is visible, and s must be replayed.

In predicate form: committed(s) => forall a in declared(s): retrievable(hash(a)), and not committed(s) => replay(s). The second clause is the part people forget. A protocol that can leave a dangling manifest is worse than one that loses everything, because a dangling manifest lies to every downstream consumer, including your evaluation harness and your cost accounting.

The critical consequence: the content must be durable before the pointer to it is durable. Publish the manifest first and you have built a directory of broken promises.

Data flow: five edges, five ways to lose a file

agent step
   |
   (1) local write        workspace/fs      <- dies with the box
   |
   (2) hash + PUT blob    durable store     <- survives
   |
   (3) CAS manifest       durable store     <- survives, and it is a promise
   |
   (4) ack to orchestrator
   |
   (5) replay on unknown  (same step_id)    <- must be idempotent

Edges (1) and (2) are safe to lose: a blob with no manifest is garbage, and garbage is collectable. Edge (3) is the commit point of the whole system, and everything before it must be replayable. Edge (5) is where teams quietly cheat — they re-run with a fresh step_id, get a second manifest, and now two "authoritative" artifact sets exist for one logical step.

A fixture that kills the sandbox at every edge

Rather than argue about this, I wrote a small simulator. It is deliberately boring: a content-addressed blob directory, a JSON manifest with compare-and-swap on a revision number, and a workspace whose death I can schedule at an exact operation index. Percent-level reliability claims are meaningless here; I want every interleaving, not a sample.

Step 1 — content store with atomic replacement.

import hashlib, json, os

class ContentStore:
    def __init__(self, root):
        self.root = root
        os.makedirs(f"{root}/blobs", exist_ok=True)
        os.makedirs(f"{root}/manifests", exist_ok=True)

    def put(self, data: bytes) -> str:
        h = hashlib.sha256(data).hexdigest()
        path = f"{self.root}/blobs/{h}"
        tmp = path + ".tmp"
        with open(tmp, "wb") as f:
            f.write(data); f.flush(); os.fsync(f.fileno())
        os.replace(tmp, path)          # atomic within one filesystem
        return h

    def exists(self, h: str) -> bool:
        return os.path.exists(f"{self.root}/blobs/{h}")

    def manifest(self, step_id: str):
        p = f"{self.root}/manifests/{step_id}.json"
        return json.load(open(p)) if os.path.exists(p) else None

    def cas_manifest(self, step_id, expected_rev, body):
        cur = self.manifest(step_id)
        cur_rev = -1 if cur is None else cur["rev"]
        if cur_rev != expected_rev:
            raise RuntimeError(f"CAS conflict on {step_id}: want {expected_rev}, have {cur_rev}")
        body = dict(body, rev=cur_rev + 1)
        p = f"{self.root}/manifests/{step_id}.json"
        with open(p + ".tmp", "w") as f:
            json.dump(body, f); f.flush(); os.fsync(f.fileno())
        os.replace(p + ".tmp", p)
        return body

The single-writer expected_rev check is doing the real work here. In production this is a conditional write on object storage or a WHERE rev = ? update, and if your store cannot express it, stop reading and fix that first.

Step 2 — a workspace that can die on demand.

class SandboxDeath(Exception):
    pass

class Sandbox:
    def __init__(self, root, die_at=None):
        self.root, self.die_at, self.ops = root, die_at, 0

    def _tick(self, label):
        self.ops += 1
        if self.die_at is not None and self.ops >= self.die_at:
            raise SandboxDeath(label)
        return label

    def write(self, rel, data: bytes):
        self._tick(f"write:{rel}")
        path = os.path.join(self.root, rel)
        os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
        with open(path, "wb") as f:
            f.write(data)
        return rel

Step 3 — the protocol, in the order I claim is correct.

def run_step(store, sandbox, step_id, artifacts):
    declared = []
    for rel, data in artifacts:
        sandbox.write(rel, data)                       # (1) local, volatile
        declared.append(rel)

    hashes = {}
    for rel in declared:
        body = open(os.path.join(sandbox.root, rel), "rb").read()
        hashes[rel] = store.put(body)                  # (2) durable content

    return store.cas_manifest(step_id, -1, {           # (3) durable pointer
        "step_id": step_id, "state": "committed", "artifacts": hashes,
    })

Step 4 — the checker, which is where the invariant becomes executable.

def classify(store, step_id):
    m = store.manifest(step_id)
    if m is None:
        return "no-commit"                 # acceptable: replay required
    for rel, h in m["artifacts"].items():
        if not store.exists(h):
            return f"VIOLATION dangling {rel} -> {h[:12]}"
    return "committed"

Step 5 — kill the workspace at every operation index and classify.

def chaos(artifacts, operations):
    outcomes = {}
    for die_at in range(1, operations + 1):
        store = ContentStore(f"/tmp/mc/store-{die_at}")
        sandbox = Sandbox(f"/tmp/mc/sb-{die_at}", die_at=die_at)
        try:
            run_step(store, sandbox, "step-42", artifacts)
            result = "completed"
        except SandboxDeath as e:
            result = f"died:{e}"
        outcomes[die_at] = (result, classify(store, "step-42"))
    return outcomes

Run it with python3 -m chaos_manifest after dropping the three classes in one module. My local run over a five-artifact step produced a single completed outcome at the final operation index, a spread of died:* outcomes before that, and no-commit or committed for every one of them — never VIOLATION.

Then flip the order and watch the property fail

This is the part I actually care about, because a test that only ever passes proves nothing about the property you claim to hold. Swap edges (2) and (3) so the manifest is published before the blobs are uploaded:

def run_step_wrong(store, sandbox, step_id, artifacts):
    declared = [sandbox.write(rel, data) for rel, data in artifacts]
    hashes = {rel: hashlib.sha256(open(os.path.join(sandbox.root, rel), "rb").read()).hexdigest()
              for rel in declared}
    store.cas_manifest(step_id, -1, {"step_id": step_id, "state": "committed", "artifacts": hashes})
    for rel in declared:                               # too late
        store.put(open(os.path.join(sandbox.root, rel), "rb").read())

Now a death between the CAS and the first upload leaves a manifest that points at a blob nobody ever wrote. classify returns VIOLATION on that index, and it will return it on most indices after it. If your durable store has eventual visibility on the manifest side, you can reproduce the same bug even with the correct call order.

Failure domains and what each one costs you

Failure	Detected by	Symptom if unhandled	Mitigation
Workspace reclaimed before (2)	client observes disconnect	work lost, cost lost	replay step with same step_id
Death between (2) and (3)	nothing — silent	orphan blob, wasted bytes	blob GC keyed on manifest absence
Death between (3) and (4)	client sees no ack	duplicate replay attempt	CAS on expected_rev rejects it
Partial local file write	hash mismatch vs declared length	truncated artifact published	hash at read time, never at write time
Concurrent replay, two attempts	CAS conflict	two artifact sets	single writer per step_id
Store outage mid-step	upload error	half-committed step	no manifest until all blobs land

The row that costs the most in practice is the second one, because nothing reports it. You do not lose correctness, you lose money and storage quietly, and nobody notices until the billed byte count stops matching the number of committed steps.

Tradeoff table: when write-through is the wrong answer

Strategy	Loss window	Store ops per step	Complexity	Fits when
Per-write write-through	one write	N writes + 1 CAS	low	steps are long, workspace is fragile
Per-step batch publish	whole step	1 CAS	low	steps are short and cheap to replay
Periodic checkpoint	checkpoint interval	K + 1	medium	large workspaces, expensive steps
Push to git per step	one commit	1 push	medium	artifacts are text and reviewable
Persistent volume + SLA	none	0	low until it is not	you actually bought durability

I default to batch publish with a hard step timeout, because replay is usually cheaper than the extra round trips. I switch to per-write write-through when a single step's compute exceeds the cost of N uploads, which is common for long agent loops — and that is precisely the regime where a free, short-lived workspace is attractive in the first place.

Where the free tier actually fits in this design

MonkeyCode is an open-source coding-agent client, and the part relevant to this review is that it advertises free model access and a free server option for the workspace — the operator states a 10M-token free allowance, and I am deliberately not publishing a quota table, because quotas are the first thing to change and the second thing to be misquoted. Verify the current numbers on the project page before you plan a budget around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why does that belong in an architecture post rather than a getting-started post? Because a free, reclaimable server makes the failure mode above the default rather than the exception, and that is genuinely useful: my chaos fixture needs a real machine to die on, not just a die_at integer. The honest workflow I would recommend is to build the manifest protocol against the simulator first, then run the same step loop against a free server and compare committed-step counts to started-step counts. If those two numbers drift, you have found a durability bug that no amount of retry logic will paper over.

Who should not do this

If your workspace is a persistent volume with a real SLA, adding a manifest layer is ceremony that buys you nothing. If your steps finish in under a minute and produce one file, replay is so cheap that per-step batch publish is already the correct answer. And if every step ends in git push to a remote you control, you have a durable pointer already — measure before you build a second one.

Limits of what I tested

My fixture proves the protocol under a single-writer, single-store model, and it cheats by assuming os.replace is atomic on the same filesystem. It says nothing about object-storage eventual consistency, network partitions between the client and the store, or blobs above a few megabytes. It also cannot tell you whether the agent produced the right files, only whether the ones it declared are retrievable — evaluation correctness is a separate invariant, and I have written about that elsewhere.

So here is the counterexample question I would put to any team running agents on borrowed compute. A step dies after the blob upload and before the manifest CAS; a reader sees no commit and the orchestrator replays the step, and the replay produces byte-different but semantically equivalent output. Should the system reject the second attempt, replay it under the same step_id and let the CAS decide, or compensate by writing a superseding revision with both hashes recorded? Which event order breaks your invariant first — and what does your ledger say when it does?