My Batch Job Died at 3 AM. The Free Server Restarted, and My Code Had No Memory.

At 3:12 AM, my batch job stopped answering. Not with an error, not with a timeout — the process was simply gone, and so was every result it had accumulated over the previous six hours. I was running a 48-hour document summarization pipeline against a free AI server, spending a free 10M token allowance, and my code had one fatal design choice: it kept all of its state in memory. The server restarted, and six hours of completed work turned into six hours of tokens I would have to spend again.

I ran the experiment on MonkeyCode's free server and free model access — an open-source project that offers a free 10M token allowance and a free server option, which made it the perfect place to break things cheaply. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The naive loop that looked perfectly fine

My first version was the kind of script every developer writes at 11 PM. Load the document list, call the model API in a loop, collect the results into a list, and write everything to disk only at the very end. Here is the shape of it, simplified for this post.

import requests

docs = load_all_documents()  # 2,000 items
results = []

for doc in docs:
    resp = requests.post(API_URL, json={'prompt': build_prompt(doc)})
    results.append(resp.json()['text'])

save_all(results)  # if this line never runs, nothing survived

It looked correct. It was correct, right up until the moment the process stopped existing. A free server is an ephemeral resource, and my pipeline was designed as if it were permanent. The restart happened around 3 AM, after roughly 900 of the 2,000 documents were done. Without a checkpoint, the rerun meant re-sending all 900 prompts — and paying for them again.

What the restart actually taught me

Three lessons, in order of pain:

• A free server is a borrowed machine. It can disappear at any moment, and "it worked yesterday" is not a durability guarantee.
• Tokens are a budget, not a resource. Every reprocessed document costs twice: once for the work, once for the redo.
• The cheapest API call is the one you never make. Skipping a completed item takes milliseconds and costs zero tokens.

That last point became the design goal. I wanted a rerun that did nothing except discover that there was nothing to do.

The checkpoint pattern I would repeat

The fix was boring and effective: a JSONL checkpoint file that records every completed document, keyed by a hash of its input. On startup, the script loads the checkpoint and skips anything already finished.

import hashlib
import json
import os
import requests

CHECKPOINT = 'batch_state.jsonl'

def doc_id(doc):
    return hashlib.sha256(doc['text'].encode()).hexdigest()[:16]

def load_done():
    done = {}
    if os.path.exists(CHECKPOINT):
        with open(CHECKPOINT) as f:
            for line in f:
                record = json.loads(line)
                done[record['doc_id']] = record
    return done

done = load_done()

for doc in docs:
    key = doc_id(doc)
    if key in done:
        continue  # already paid for; do not pay again

    resp = requests.post(API_URL, json={'prompt': build_prompt(doc)})
    payload = resp.json()

    with open(CHECKPOINT, 'a') as f:
        f.write(json.dumps({
            'doc_id': key,
            'input': doc['text'],
            'raw_output': payload,  # re-parsing is free; re-calling is not
            'tokens': payload.get('usage', {}),
        }) + '\n')

Three details made the difference between this and a glorified log file:

1. Hash the input, not the index. If the document list changes between runs, the hash still identifies the same work, and a reordered list does not cause duplicate calls.
2. Store the raw response, not just the parsed field. When the model returns truncated JSON, you can re-parse the saved payload without spending another token. That happened twice in my run.
3. Flush atomically if you batch. Appending one line per item is safe, but if you flush in bulk, write to a temp file and rename it so a crash mid-write cannot corrupt the checkpoint.

def flush(records):
    tmp = CHECKPOINT + '.tmp'
    with open(tmp, 'w') as f:
        for r in records:
            f.write(json.dumps(r) + '\n')
    os.replace(tmp, CHECKPOINT)  # atomic on POSIX

With this in place, the post-restart rerun skipped all 900 completed documents in about four seconds. The remaining 1,100 ran overnight, and when the job finished, the checkpoint file was the only proof that any of it had happened.

The honest limitations

The checkpoint file has the same weakness as everything else on a free server: it lives on the same ephemeral disk. If the server is recycled instead of restarted, the checkpoint dies with it. For my 48-hour window I accepted that risk and copied the file to a remote location every 100 items; if you need stronger guarantees, push state to object storage or a database from the first line of code.

Rate limits also showed up when I tried to parallelize the loop. A simple semaphore plus respecting Retry-After fixed it, but it was a reminder that a free tier is a shared resource, not a private cluster.

And to be clear: checkpointing does not make a free server reliable. It makes your job survivable. Those are different things, and confusing them is how you end up with a production incident at 3 AM.

Who should not use this pattern

• If your job finishes in five minutes, a checkpoint file is overhead, not insurance.
• If your workload has a contractual SLA, a free server is the wrong foundation no matter how good your resume logic is.
• If your state is tiny and your runtime is short, keep it in memory and move on. Checkpointing earns its keep only when the cost of redoing work is real.

The constraint was the teacher

I did not plan to learn about crash recovery this week, and I would not recommend a 3 AM wake-up as a teaching method. But the free server's restart forced a design habit that now applies everywhere: spot instances, serverless functions, CI runners, any place where compute is cheap and ephemeral. The limitation did not just cost me tokens — it made the pipeline better than the version I originally intended to write.

If you want to know whether your own pipeline survives a restart, MonkeyCode is open source — point it at your worst script and wait for 3 AM. Just write the checkpoint first.