@softverseadmin Hard cap, and make it stop loudly. Summarizing at page boundaries drops the exact detail the resume needs, and it fails silently. A cap that refuses the write and says so is a limit the agent cannot talk its way past.
DevOps engineer turned project manager in Tokyo. I write practical, honest guides on Kubernetes, GitOps, CI/CD, AWS cost and DevSecOps at DevOps Enginer — real experience, no fluff. New posts weekly.
So I actually ran this before replying. Paged job, checkpoint budget of 256 bytes, and the resume needs one exact page token to restart from.
The summarize version lost it. The summary said something like 'processed up to the low forties' and the exact token the resume needed was just gone. Nothing flagged the loss either. Resume would've been guessing.
Yours held up. Checkpoint only carried the state: task id, cursor, last verified output, plus a pointer into the append-only log (offset + hash). 138 bytes. Resume followed the pointer and pulled the exact token back out. And when I stuffed the whole transcript in there — 1,376 bytes — the write refused. Nothing written, run stops. Loud, like you said.
So the working setup is this. Checkpoint holds state only, under a hard cap that refuses the write. The detail never goes in the checkpoint at all — it lives in an append-only log, and all the checkpoint carries is a pointer to it, offset and hash. Resume follows the pointer and pulls the exact detail back out. The cap keeps the checkpoint honest. The log keeps the detail alive.
I'd add crashes around the two writes to that test. The offset + hash only helps if its log entry survives too. After killing the worker during a log append or checkpoint update, I'd want resume to find either the old complete checkpoint and its log data or the new pair, never a checkpoint pointing at data that didn't survive. That's a separate check from refusing an oversized write.
DevOps engineer turned project manager in Tokyo. I write practical, honest guides on Kubernetes, GitOps, CI/CD, AWS cost and DevSecOps at DevOps Enginer — real experience, no fluff. New posts weekly.
Yeah, that's the hole in it. I ran the kill between the two writes and it did exactly what you said — checkpoint went out pointing at a log entry that didn't survive, so resume came back with a pointer to nothing.
What held was boringly ordered: log first, fsync it, then the checkpoint with the hash, and resume doesn't trust the pointer unless the hash matches. If it doesn't, it falls back to the last good checkpoint instead of chasing a dead one. So it's not just fsyncing both, it's the order — and the hash is what makes the fallback real instead of hopeful.
The checkpoint decides what the next container has to read back before it can act, and that read is paid input. So what goes on the shared disk is a cost line as well as a recovery one. The checkpoint should hold the state of the work, not the whole transcript.
DevOps engineer turned project manager in Tokyo. I write practical, honest guides on Kubernetes, GitOps, CI/CD, AWS cost and DevSecOps at DevOps Enginer — real experience, no fluff. New posts weekly.
That's the cost line most checkpoint designs hide, Paul — the read-back is paid input, and for an agent it's paid in tokens as well as time. A checkpoint that holds the whole transcript turns every resume into re-reading the entire job.
The split I'd draw is state vs transcript: the checkpoint holds only the minimal resumable state — task id, page cursor, last verified output — and the transcript stays in an append-only log the next container fetches only when it actually needs it. Resume reads a few hundred bytes, not the whole history.
How do you bound it in practice — a hard size cap on the checkpoint, summarise-and-drop at page boundaries, or something else? The cap is where the recovery guarantee and the cost line actually meet.
Deleting the working pod on purpose is the honest test here, and the log line resuming from page 17 says more than the code does. Rabin's point is worth keeping next to it: checkpointing decides what survives, not what the next container is allowed to touch. Paul's split between the state of the work and the full transcript is the other half, since every resume pays to read the checkpoint back.
We rebuilt a candidate-enrichment pipeline at Ascendra Ventures around the same idea, for a seed-stage talent platform where any slow step could leave a person in an unclear state. Each person now runs as a durable workflow in Temporal. It claims the person, fetches profile data, processes urgent profiles straight away and batches the rest, waits durably on OpenAI batch jobs, and falls back to polling when webhooks go quiet. The rule we held to was that every candidate either finishes or visibly fails, never sits in the middle. A checkpoint file gets you a long way, and what a workflow engine adds on top is that the waiting itself survives a restart.
What happens today if two containers pick up the same task id after a re-dispatch? Is anything stopping both from writing to the same checkpoint?
DevOps engineer turned project manager in Tokyo. I write practical, honest guides on Kubernetes, GitOps, CI/CD, AWS cost and DevSecOps at DevOps Enginer — real experience, no fluff. New posts weekly.
Temporal's the honest version of this if you're already running it — the waiting surviving a restart is the part a file on a shared disk just can't do on its own.
On the duplicate task question, nothing in my setup stops it. Two containers pick up the same task id, both write the same checkpoint, last writer wins and you don't even know it happened. The thing that would stop it is a lease — dispatcher hands the unit out with a token, checkpoint write has to carry it, stale token gets refused. Without that the idempotency key saves the side effect but not the checkpoint itself.
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
Deleting the container on purpose is the only test that really counts here, nice to see it done instead of argued about.
One thing I'd watch with the JSON file on a shared disk: if the write isn't atomic (temp file, then rename), a container killed mid-write leaves a half-written checkpoint the next shift can't parse. Looking forward to the watcher post.
DevOps engineer turned project manager in Tokyo. I write practical, honest guides on Kubernetes, GitOps, CI/CD, AWS cost and DevSecOps at DevOps Enginer — real experience, no fluff. New posts weekly.
Good catch, same thing bit me in the test — killed mid-write on the JSON file directly and resume couldn't even parse it. Temp file plus rename fixed it, resume just saw the old checkpoint and carried on. That's the one I'd actually ship first, it's the cheapest of the three fixes.
DevOps engineer turned project manager in Tokyo. I write practical, honest guides on Kubernetes, GitOps, CI/CD, AWS cost and DevSecOps at DevOps Enginer — real experience, no fluff. New posts weekly.
The delete-the-pod test is the right judge — most resilience claims die exactly there. One half I'd add: checkpointing answers what survives the container. It doesn't answer what the next container is allowed to touch. A resumed agent mounting the same PVC with the same identity inherits the full blast radius of the one that crashed — including when the crash was the agent's own doing.
When I set up agent sandboxes I ended up pairing the two: state on a volume so the work survives, but the workload itself inside a gVisor sandbox with a tightly scoped identity, so delete-and-resume is safe to automate, not just possible. Is the PVC fenced per task in your pool, or shared across tasks? A shared blackboard is also a shared contamination path — curious how you fenced it.
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
The resume from page 17 also says when a page runs twice. The checkpoint is written after the page is done, so a kill that lands after the page's work and before the json.dump repeats that page on the next shift. For a crawl that only reads, the repeat costs a second fetch. For an agent step that writes somewhere else, such as opening a pull request, posting a message or charging a card, the next container does it again, and the checkpoint cannot tell, because it records how far the work got, not what the last step already did outside the disk.
The usual fix sits on the side effect rather than on the checkpoint: a key made of the task id and the unit number, which the external write checks before it acts. The same delete-the-pod test would show it, with the kill placed right after a step that writes and a count of the duplicates. Once the next posts make the dispatcher remember outstanding work and re-send a stalled task by itself, will the unit that was in flight run again?
DevOps engineer turned project manager in Tokyo. I write practical, honest guides on Kubernetes, GitOps, CI/CD, AWS cost and DevSecOps at DevOps Enginer — real experience, no fluff. New posts weekly.
That's the one that actually bites in practice. Checkpoint tells you how far you got, not what you already did outside the disk. I ran the kill right after a page that writes out — in my test it was opening a PR — and resume ran the page again, second PR, because nothing on the other side knew it'd already happened.
The fix wasn't in the checkpoint at all. It was the key — task id plus unit number, checked by the thing receiving the write before it acts. Same kill, same repeat, one PR.
And on your dispatcher question, yeah — if it re-sends a stalled unit on its own, the one still in flight runs too unless there's a lease on it. Without something fencing the checkpoint write you've just got two containers racing the same unit, and the idempotency key is the only thing stopping the double write.
We're a place where coders share, stay up-to-date and grow their careers.
@softverseadmin Hard cap, and make it stop loudly. Summarizing at page boundaries drops the exact detail the resume needs, and it fails silently. A cap that refuses the write and says so is a limit the agent cannot talk its way past.
So I actually ran this before replying. Paged job, checkpoint budget of 256 bytes, and the resume needs one exact page token to restart from.
The summarize version lost it. The summary said something like 'processed up to the low forties' and the exact token the resume needed was just gone. Nothing flagged the loss either. Resume would've been guessing.
Yours held up. Checkpoint only carried the state: task id, cursor, last verified output, plus a pointer into the append-only log (offset + hash). 138 bytes. Resume followed the pointer and pulled the exact token back out. And when I stuffed the whole transcript in there — 1,376 bytes — the write refused. Nothing written, run stops. Loud, like you said.
So the working setup is this. Checkpoint holds state only, under a hard cap that refuses the write. The detail never goes in the checkpoint at all — it lives in an append-only log, and all the checkpoint carries is a pointer to it, offset and hash. Resume follows the pointer and pulls the exact detail back out. The cap keeps the checkpoint honest. The log keeps the detail alive.
I'd add crashes around the two writes to that test. The offset + hash only helps if its log entry survives too. After killing the worker during a log append or checkpoint update, I'd want resume to find either the old complete checkpoint and its log data or the new pair, never a checkpoint pointing at data that didn't survive. That's a separate check from refusing an oversized write.
Yeah, that's the hole in it. I ran the kill between the two writes and it did exactly what you said — checkpoint went out pointing at a log entry that didn't survive, so resume came back with a pointer to nothing.
What held was boringly ordered: log first, fsync it, then the checkpoint with the hash, and resume doesn't trust the pointer unless the hash matches. If it doesn't, it falls back to the last good checkpoint instead of chasing a dead one. So it's not just fsyncing both, it's the order — and the hash is what makes the fallback real instead of hopeful.
The checkpoint decides what the next container has to read back before it can act, and that read is paid input. So what goes on the shared disk is a cost line as well as a recovery one. The checkpoint should hold the state of the work, not the whole transcript.
That's the cost line most checkpoint designs hide, Paul — the read-back is paid input, and for an agent it's paid in tokens as well as time. A checkpoint that holds the whole transcript turns every resume into re-reading the entire job.
The split I'd draw is state vs transcript: the checkpoint holds only the minimal resumable state — task id, page cursor, last verified output — and the transcript stays in an append-only log the next container fetches only when it actually needs it. Resume reads a few hundred bytes, not the whole history.
How do you bound it in practice — a hard size cap on the checkpoint, summarise-and-drop at page boundaries, or something else? The cap is where the recovery guarantee and the cost line actually meet.
Deleting the working pod on purpose is the honest test here, and the log line resuming from page 17 says more than the code does. Rabin's point is worth keeping next to it: checkpointing decides what survives, not what the next container is allowed to touch. Paul's split between the state of the work and the full transcript is the other half, since every resume pays to read the checkpoint back.
We rebuilt a candidate-enrichment pipeline at Ascendra Ventures around the same idea, for a seed-stage talent platform where any slow step could leave a person in an unclear state. Each person now runs as a durable workflow in Temporal. It claims the person, fetches profile data, processes urgent profiles straight away and batches the rest, waits durably on OpenAI batch jobs, and falls back to polling when webhooks go quiet. The rule we held to was that every candidate either finishes or visibly fails, never sits in the middle. A checkpoint file gets you a long way, and what a workflow engine adds on top is that the waiting itself survives a restart.
What happens today if two containers pick up the same task id after a re-dispatch? Is anything stopping both from writing to the same checkpoint?
Temporal's the honest version of this if you're already running it — the waiting surviving a restart is the part a file on a shared disk just can't do on its own.
On the duplicate task question, nothing in my setup stops it. Two containers pick up the same task id, both write the same checkpoint, last writer wins and you don't even know it happened. The thing that would stop it is a lease — dispatcher hands the unit out with a token, checkpoint write has to carry it, stale token gets refused. Without that the idempotency key saves the side effect but not the checkpoint itself.
Deleting the container on purpose is the only test that really counts here, nice to see it done instead of argued about.
One thing I'd watch with the JSON file on a shared disk: if the write isn't atomic (temp file, then rename), a container killed mid-write leaves a half-written checkpoint the next shift can't parse. Looking forward to the watcher post.
Good catch, same thing bit me in the test — killed mid-write on the JSON file directly and resume couldn't even parse it. Temp file plus rename fixed it, resume just saw the old checkpoint and carried on. That's the one I'd actually ship first, it's the cheapest of the three fixes.
The delete-the-pod test is the right judge — most resilience claims die exactly there. One half I'd add: checkpointing answers what survives the container. It doesn't answer what the next container is allowed to touch. A resumed agent mounting the same PVC with the same identity inherits the full blast radius of the one that crashed — including when the crash was the agent's own doing.
When I set up agent sandboxes I ended up pairing the two: state on a volume so the work survives, but the workload itself inside a gVisor sandbox with a tightly scoped identity, so delete-and-resume is safe to automate, not just possible. Is the PVC fenced per task in your pool, or shared across tasks? A shared blackboard is also a shared contamination path — curious how you fenced it.
The isolation half, for reference: devopsenginer.com/blog/sandbox-ai-...
The resume from page 17 also says when a page runs twice. The checkpoint is written after the page is done, so a kill that lands after the page's work and before the json.dump repeats that page on the next shift. For a crawl that only reads, the repeat costs a second fetch. For an agent step that writes somewhere else, such as opening a pull request, posting a message or charging a card, the next container does it again, and the checkpoint cannot tell, because it records how far the work got, not what the last step already did outside the disk.
The usual fix sits on the side effect rather than on the checkpoint: a key made of the task id and the unit number, which the external write checks before it acts. The same delete-the-pod test would show it, with the kill placed right after a step that writes and a count of the duplicates. Once the next posts make the dispatcher remember outstanding work and re-send a stalled task by itself, will the unit that was in flight run again?
That's the one that actually bites in practice. Checkpoint tells you how far you got, not what you already did outside the disk. I ran the kill right after a page that writes out — in my test it was opening a PR — and resume ran the page again, second PR, because nothing on the other side knew it'd already happened.
The fix wasn't in the checkpoint at all. It was the key — task id plus unit number, checked by the thing receiving the write before it acts. Same kill, same repeat, one PR.
And on your dispatcher question, yeah — if it re-sends a stalled unit on its own, the one still in flight runs too unless there's a lease on it. Without something fencing the checkpoint write you've just got two containers racing the same unit, and the idempotency key is the only thing stopping the double write.