The 7.4% You Don't See: Checkpointing Long LLM Jobs Before They Time Out

Two jobs failed on my agent's VPS on the same day, for two different reasons, and it took me longer than I'd like to admit to notice they were the same bug wearing different clothes.

The agent I run does a mix of unglamorous background work — writing articles, generating demo pages, making paper-trading decisions — by calling an LLM, waiting for a response, and doing something with the output. Nothing exotic. It runs on a single Vultr instance, not behind a queue-and-retry serverless setup, because most of the work is sequential and cheap enough that a VPS with a cron loop is the right amount of infrastructure. Right up until it wasn't.

Failure one: the timeout that ate a full generation

One task needed to write a multi-section HTML demo page. Big prompt, big expected output, one LLM call. It hit my 480-second timeout. Another task, generating a longer article, did the same thing an hour later. Two failures out of 27 jobs that day — 7.4%, which sounds small until you notice both failures were on the largest jobs, and I was actively shifting more of the day's workload toward longer-context tasks. The failure rate on big jobs wasn't 7.4%. It was closer to 100% of anything that pushed the token budget.

The part that stung wasn't the timeout itself — LLM calls time out, that's expected — it was that the entire generation was gone. Five minutes of inference, thrown away, because the process that owned the HTTP connection died with nothing written to disk.

Failure two: the parse that ate a finished article

Separately, a content-writing task logged: "Article generated but JSON parse failed." The LLM had done its job. It returned a full article. My code just couldn't get it out of the response, because I was asking the model to return {"title": ..., "article_markdown": ...} and something — a stray backtick, an unescaped quote, a truncated tail — broke the parse. The generation succeeded. The pipeline still lost it, because the only place that article ever existed was a Python variable that went out of scope when the exception propagated.

Different failure mode, identical shape: expensive work completed, then discarded, because nothing was written to disk until after a step that could fail.

The fix is boring, which is why it works

The instinct when you see a timeout is to add retries. The instinct when you see a parse failure is to make the parser more forgiving. Both are reasonable and I did some of both, but neither addresses the actual problem: the job has no durable state until the very last step succeeds. Retrying a job that has no checkpoint just means paying for the same 480 seconds again.

The actual fix has two parts, and both are just "write to disk earlier than you think you need to."

Part 1 — save the raw output before you trust it.

Before attempting to parse or validate an LLM response, write it verbatim to a fallback path keyed by timestamp:

import json
from datetime import datetime, timezone

def call_llm_with_fallback(prompt, artifacts_dir):
    response_text = llm_client.complete(prompt)

    ts = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
    fallback_path = f"{artifacts_dir}/{ts}_raw_fallback.txt"
    with open(fallback_path, "w") as f:
        f.write(response_text)

    try:
        return json.loads(response_text), None
    except json.JSONDecodeError as e:
        return None, fallback_path

This is maybe six lines. It costs one disk write per LLM call, which is nothing compared to the seconds of inference you just paid for. If the parse fails downstream, the content isn't lost — it's sitting in artifacts/20260813_204701_raw_fallback.txt, recoverable by a human or a repair script. I went from "article generated but lost" to "article generated, parse failed, here's the raw markdown" with no change to the happy path at all.

Part 2 — checkpoint sections, don't checkpoint the whole job.

For the large multi-section generations that were timing out, the fix was to stop asking for one giant completion and start asking for one completion per section, writing each one to a checkpoint file as it lands:

def generate_with_checkpoints(sections, job_id, artifacts_dir):
    checkpoint_path = f"{artifacts_dir}/{job_id}_checkpoint.json"
    state = _load_checkpoint(checkpoint_path)  # {} if none exists

    for section in sections:
        if section["name"] in state:
            continue  # already done, resume past it
        state[section["name"]] = llm_client.complete(section["prompt"])
        with open(checkpoint_path, "w") as f:
            json.dump(state, f)

    return state

If the process dies on section four of six — timeout, crash, OOM, doesn't matter — the next run picks up at section four instead of starting over. This also had a side effect I didn't anticipate: individual section prompts are smaller, so each one is less likely to hit the timeout in the first place. Splitting the job didn't just make failure cheaper, it made failure less frequent.

Why this matters more on a VPS than it would on managed infra

On a platform that gives you automatic retries, dead-letter queues, and step-level state out of the box, some of this comes for free. On a single VPS running a cron loop or a long-lived worker process, you own all of it. That's not a downside — it's usually the right tradeoff for workloads that are sequential, low-volume, and don't need the operational overhead of a queue system — but it means the durability has to be designed in explicitly, at the point where you'd otherwise just trust the next line of code to run.

The pattern generalizes past LLM calls: any job where step N is expensive and step N+1 can fail should write N's output to disk before attempting N+1. It's a five-minute change per job. I made it after losing two jobs in one day; I'd recommend making it before that happens to you, because by the time you notice the failure rate, you've already lost the work that would have told you it was a problem.