DEV Community

dreamyman
dreamyman

Posted on Originally published at pierceliu.top

Containers die whenever; two hours of progress should not die with them

Three posts in: tasks are isolated from each other, and the wait went from 45 seconds to a fraction of one. But one thing has not changed since the first post — progress still lives in the container's own memory.

That used to mean "a crash loses it". Now it is worse: containers in a warm pool are replaced as a matter of routine. Upgrades replace them, machine failures replace them, resource pressure reclaims them. They are consumables by design.

Which reframes the question: why would you let a consumable hold two hours of your work?

Write it somewhere else: the site blackboard, not the worker's head

Picture a construction site. Progress kept in one worker's head ends when the shift changes; progress chalked on a blackboard on the wall can be read by whoever comes next.

The code does exactly that — hang the blackboard on the wall:

# ask for a 1GB shared disk
kind: PersistentVolumeClaim
spec:
  resources:
    requests:
      storage: 1Gi
---
# mount it into every container in the pool
volumeMounts:
  - name: workspace-volume
    mountPath: /workspace     # this is the directory the container sees
Enter fullscreen mode Exit fullscreen mode

In K8s a shared disk like this is a persistent volume. The operative word is persistent: it outlives the container, and the next container can mount it.

Then two changes to the work itself. After each page, write it on the board:

for i in range(start_page, total_pages):
    await asyncio.sleep(1.0)                                  # fetch a page
    with open(f"/workspace/{task_id}/checkpoint.json", "w") as f:
        json.dump({"pages_crawled": i + 1, "total": total_pages}, f)   # write immediately
Enter fullscreen mode Exit fullscreen mode

Before starting, read what the last shift left:

start_page = 0
if os.path.exists(ckpt_file):
    start_page = json.load(open(ckpt_file)).get("pages_crawled", 0)
    print(f"checkpoint found, resuming from page {start_page}")
Enter fullscreen mode Exit fullscreen mode

That is all of it. No distributed transactions, no compensation logic — one JSON file.

Now delete the working container on purpose

This step can only be judged by breaking something; reading the code proves nothing.

Halfway through a task, delete the container doing the work:

kubectl delete pod <the-one-currently-working>
Enter fullscreen mode Exit fullscreen mode

Re-dispatch the same task and watch the new container's log:

checkpoint found, resuming from page 17
[worker-b] task-a3f9 progress: 18/30
[worker-b] task-a3f9 progress: 19/30
Enter fullscreen mode Exit fullscreen mode

The first seventeen pages are not redone. The container changed; the work continued where it stood.

One easily-missed consequence: the progress endpoint can now ask any live container, because they all mount the same disk. The state no longer belongs to a particular container.

This is what the often-quoted line actually feels like: containers are cattle, workspaces are pets. Cattle are replaceable; pets have names and need looking after.

What this version still lacks (the honest part)

One, the "automatic recovery" needs a human. The code says so plainly:

# no task_id means a new task; a task_id means resume
Enter fullscreen mode Exit fullscreen mode

In other words, nothing is watching for tasks that stopped. Someone has to notice, then re-send the request with the original task ID. That is not automatic failover, it is "resumable" — and the distance between the two is an entire monitoring and retry mechanism. Closing that gap is what the next two posts are about.

Two, that disk will not mount on three containers in a real cluster. It is requested as ReadWriteOnce, meaning one machine at a time. The local experiment has exactly one machine, so three containers sharing it is fine; on a real multi-machine cluster, containers on the second machine fail to mount. Genuine sharing needs storage that supports multi-machine access — NFS, or a cloud file service.

Three, writing after every page is a trade-off, not a default. Writing more often loses less on a crash and costs more in I/O. In a real system that frequency should be derived from how expensive it is to redo one unit of work.

What this step settles

Once progress leaves the container, the container really is a replaceable executor. But notice what is now missing: nothing in the system knows whether a task has stalled — the dispatcher hands work over and forgets it.

And when the pool is full, it still only knows how to say "no capacity available".

Those two are the same gap: there is no place that remembers what work is still outstanding. The next post fills it — and, as it turns out, that is where I discovered I had not actually built the thing I thought I was building.

In one line: let the state outlive the container, and the container is free to die.

Code: agent-sandbox-oss/lab4.


Originally published on pierceliu.top.

Top comments (13)

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@softverseadmin Hard cap, and make it stop loudly. Summarizing at page boundaries drops the exact detail the resume needs, and it fails silently. A cap that refuses the write and says so is a limit the agent cannot talk its way past.

Collapse
 
softverseadmin profile image
Rabin Adhikari • • Edited

So I actually ran this before replying. Paged job, checkpoint budget of 256 bytes, and the resume needs one exact page token to restart from.

The summarize version lost it. The summary said something like 'processed up to the low forties' and the exact token the resume needed was just gone. Nothing flagged the loss either. Resume would've been guessing.

Yours held up. Checkpoint only carried the state: task id, cursor, last verified output, plus a pointer into the append-only log (offset + hash). 138 bytes. Resume followed the pointer and pulled the exact token back out. And when I stuffed the whole transcript in there — 1,376 bytes — the write refused. Nothing written, run stops. Loud, like you said.

So the working setup is this. Checkpoint holds state only, under a hard cap that refuses the write. The detail never goes in the checkpoint at all — it lives in an append-only log, and all the checkpoint carries is a pointer to it, offset and hash. Resume follows the pointer and pulls the exact detail back out. The cap keeps the checkpoint honest. The log keeps the detail alive.

Collapse
 
mike-johnstone profile image
Mike Johnstone •

I'd add crashes around the two writes to that test. The offset + hash only helps if its log entry survives too. After killing the worker during a log append or checkpoint update, I'd want resume to find either the old complete checkpoint and its log data or the new pair, never a checkpoint pointing at data that didn't survive. That's a separate check from refusing an oversized write.

Thread Thread
 
softverseadmin profile image
Rabin Adhikari •

Yeah, that's the hole in it. I ran the kill between the two writes and it did exactly what you said — checkpoint went out pointing at a log entry that didn't survive, so resume came back with a pointer to nothing.

What held was boringly ordered: log first, fsync it, then the checkpoint with the hash, and resume doesn't trust the pointer unless the hash matches. If it doesn't, it falls back to the last good checkpoint instead of chasing a dead one. So it's not just fsyncing both, it's the order — and the hash is what makes the fallback real instead of hopeful.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

The checkpoint decides what the next container has to read back before it can act, and that read is paid input. So what goes on the shared disk is a cost line as well as a recovery one. The checkpoint should hold the state of the work, not the whole transcript.

Collapse
 
softverseadmin profile image
Rabin Adhikari •

That's the cost line most checkpoint designs hide, Paul — the read-back is paid input, and for an agent it's paid in tokens as well as time. A checkpoint that holds the whole transcript turns every resume into re-reading the entire job.

The split I'd draw is state vs transcript: the checkpoint holds only the minimal resumable state — task id, page cursor, last verified output — and the transcript stays in an append-only log the next container fetches only when it actually needs it. Resume reads a few hundred bytes, not the whole history.

How do you bound it in practice — a hard size cap on the checkpoint, summarise-and-drop at page boundaries, or something else? The cap is where the recovery guarantee and the cost line actually meet.

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

Deleting the working pod on purpose is the honest test here, and the log line resuming from page 17 says more than the code does. Rabin's point is worth keeping next to it: checkpointing decides what survives, not what the next container is allowed to touch. Paul's split between the state of the work and the full transcript is the other half, since every resume pays to read the checkpoint back.

We rebuilt a candidate-enrichment pipeline at Ascendra Ventures around the same idea, for a seed-stage talent platform where any slow step could leave a person in an unclear state. Each person now runs as a durable workflow in Temporal. It claims the person, fetches profile data, processes urgent profiles straight away and batches the rest, waits durably on OpenAI batch jobs, and falls back to polling when webhooks go quiet. The rule we held to was that every candidate either finishes or visibly fails, never sits in the middle. A checkpoint file gets you a long way, and what a workflow engine adds on top is that the waiting itself survives a restart.

What happens today if two containers pick up the same task id after a re-dispatch? Is anything stopping both from writing to the same checkpoint?

Collapse
 
softverseadmin profile image
Rabin Adhikari •

Temporal's the honest version of this if you're already running it — the waiting surviving a restart is the part a file on a shared disk just can't do on its own.

On the duplicate task question, nothing in my setup stops it. Two containers pick up the same task id, both write the same checkpoint, last writer wins and you don't even know it happened. The thing that would stop it is a lease — dispatcher hands the unit out with a token, checkpoint write has to carry it, stale token gets refused. Without that the idempotency key saves the side effect but not the checkpoint itself.

Collapse
 
delayrun profile image
delay.run •

Deleting the container on purpose is the only test that really counts here, nice to see it done instead of argued about.

One thing I'd watch with the JSON file on a shared disk: if the write isn't atomic (temp file, then rename), a container killed mid-write leaves a half-written checkpoint the next shift can't parse. Looking forward to the watcher post.

Collapse
 
softverseadmin profile image
Rabin Adhikari •

Good catch, same thing bit me in the test — killed mid-write on the JSON file directly and resume couldn't even parse it. Temp file plus rename fixed it, resume just saw the old checkpoint and carried on. That's the one I'd actually ship first, it's the cheapest of the three fixes.

Collapse
 
softverseadmin profile image
Rabin Adhikari •

The delete-the-pod test is the right judge — most resilience claims die exactly there. One half I'd add: checkpointing answers what survives the container. It doesn't answer what the next container is allowed to touch. A resumed agent mounting the same PVC with the same identity inherits the full blast radius of the one that crashed — including when the crash was the agent's own doing.

When I set up agent sandboxes I ended up pairing the two: state on a volume so the work survives, but the workload itself inside a gVisor sandbox with a tightly scoped identity, so delete-and-resume is safe to automate, not just possible. Is the PVC fenced per task in your pool, or shared across tasks? A shared blackboard is also a shared contamination path — curious how you fenced it.

The isolation half, for reference: devopsenginer.com/blog/sandbox-ai-...

Collapse
 
izgorodin profile image
Edward Izgorodin •

The resume from page 17 also says when a page runs twice. The checkpoint is written after the page is done, so a kill that lands after the page's work and before the json.dump repeats that page on the next shift. For a crawl that only reads, the repeat costs a second fetch. For an agent step that writes somewhere else, such as opening a pull request, posting a message or charging a card, the next container does it again, and the checkpoint cannot tell, because it records how far the work got, not what the last step already did outside the disk.

The usual fix sits on the side effect rather than on the checkpoint: a key made of the task id and the unit number, which the external write checks before it acts. The same delete-the-pod test would show it, with the kill placed right after a step that writes and a count of the duplicates. Once the next posts make the dispatcher remember outstanding work and re-send a stalled task by itself, will the unit that was in flight run again?

Collapse
 
softverseadmin profile image
Rabin Adhikari •

That's the one that actually bites in practice. Checkpoint tells you how far you got, not what you already did outside the disk. I ran the kill right after a page that writes out — in my test it was opening a PR — and resume ran the page again, second PR, because nothing on the other side knew it'd already happened.

The fix wasn't in the checkpoint at all. It was the key — task id plus unit number, checked by the thing receiving the write before it acts. Same kill, same repeat, one PR.

And on your dispatcher question, yeah — if it re-sends a stalled unit on its own, the one still in flight runs too unless there's a lease on it. Without something fencing the checkpoint write you've just got two containers racing the same unit, and the idempotency key is the only thing stopping the double write.