nohup Is Not a Checkpoint: An FAQ on Batches That Outlive Your SSH Session

You start a 200-item batch on a free server. You close the laptop. You come back to 40 results and a dead prompt.

Why does this keep happening? Because we repeat five claims about long jobs. Most of them are wrong.

This is a myth-busting FAQ. Each answer has a cheap test you can run. The artifact is a small journal-first runner in plain Python.

The setup: a free box, a batch, and one bad assumption

The job is boring. Read 200 prompts, call a model, write the answer down.

MonkeyCode offers free model access and a free server option, both operator-supplied. That combination is exactly where this failure shows up. Cheap compute plus a remote shell invites long jobs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

So let me be precise about what the box gives you and what it does not.

FAQ: five claims, five corrections

1. "nohup keeps my batch running."

No. nohup only detaches your process from a terminal hangup.

It changes signal delivery. It says nothing about the machine staying up.

If the box is recycled, all PIDs die together. Detaching cannot save you.

Run this test. Start a loop that writes a timestamp every second. Kill your SSH connection, wait ten minutes, reconnect.

nohup bash -c 'while true; do date -u +%FT%TZ >> /tmp/hb.log; sleep 1; done' \
  >/dev/null 2>&1 &
disown

If the file keeps growing, you learned one thing. The box survived ten minutes. That is all you learned.

Corrected model: your process is not your progress. The durable unit is the file you append to.

2. "My client printed the whole answer, so it is saved."

Did you write it to disk before you saw it? That is the only question that matters.

A streamed response lives in a socket buffer and then in your memory. Both vanish on SIGKILL.

Corrected model: a result exists when write() plus fsync() returned. Print statements are not persistence.

That is why the runner below writes one line per item and flushes it.

3. "Resuming means starting over."

Only if you never recorded what finished.

A journal turns resume into a set difference. Read the keys you already have. Skip them. The rest is your queue.

No database. One append-only JSONL file is enough.

4. "Long jobs just need a bigger box or a longer timeout."

Timeouts move the wall. They do not move the crash.

Long-running work is fragile because each unit is large. Shrink the unit and the fragility drops.

Corrected model: durability comes from small, committed steps. Not from generous limits.

5. "A journal makes retries exactly-once."

Careful. It usually gives you at-least-once execution plus dedupe on read.

If you append after the call, a crash between call and append repeats that item. If you append before, a crash loses it.

Pick your failure. For model calls, repeating is usually cheaper than losing.

The artifact: a journal-first batch runner

Standard library only. No frameworks. Read it before you trust it.

The endpoint, model, and key come from environment variables. Nothing is hardcoded.

#!/usr/bin/env python3
"""Journal-first batch runner. One writer. Append, fsync, resume."""
import hashlib, json, os, sys, time, urllib.request
from pathlib import Path

JOURNAL = Path(os.environ.get("JOURNAL", "results.jsonl"))

def key_for(task: dict) -> str:
    raw = json.dumps(task, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def load_done(path: Path) -> set:
    done = set()
    if not path.exists():
        return done
    with path.open("rb") as fh:
        for line in fh:
            line = line.strip()
            if not line:
                continue
            try:
                rec = json.loads(line)
            except json.JSONDecodeError:
                continue  # torn final line after a hard kill
            done.add(rec["key"])
    return done

def append_record(path: Path, rec: dict) -> None:
    with path.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec, sort_keys=True) + "\n")
        fh.flush()
        os.fsync(fh.fileno())

def call_model(prompt: str) -> str:
    # OpenAI-compatible chat endpoint. You supply BASE_URL and MODEL.
    body = json.dumps({
        "model": os.environ["MODEL"],
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        os.environ["BASE_URL"].rstrip("/") + "/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ.get("API_KEY", ""),
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read())
    return payload["choices"][0]["message"]["content"]

def main() -> int:
    src = Path(sys.argv[1])
    tasks = [json.loads(l) for l in src.read_text().splitlines() if l.strip()]
    dry = "--dry-run" in sys.argv
    done = load_done(JOURNAL)
    todo = [t for t in tasks if key_for(t) not in done]
    print(f"{len(tasks)} tasks, {len(done)} journaled, {len(todo)} to run")
    if dry:
        for t in todo[:5]:
            print("would run", key_for(t))
        return 0
    for task in todo:
        k = key_for(task)
        out = call_model(task["prompt"])
        append_record(JOURNAL, {
            "key": k,
            "prompt": task["prompt"],
            "out": out,
            "ts": int(time.time()),
        })
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Step 1: plan before you spend

BASE_URL=... MODEL=... python runner.py tasks.jsonl --dry-run

You get counts and the first five keys. No calls. No surprises.

Step 2: run, then kill it on purpose

Start the batch. After a few records land, send kill -9 to the process group.

BASE_URL=... MODEL=... nohup python runner.py tasks.jsonl >run.log 2>&1 &
kill -9 -$!   # or: pkill -9 -f runner.py

Now rerun the same command. Watch the header line.

Step 3: audit the journal, not the log

jq -r .key results.jsonl | sort | uniq -d | head   # duplicates: expect empty
jq -r .key results.jsonl | sort -u | wc -l         # unique keys: expect 20

Twenty unique keys after a hard kill is the whole point. A torn last line is fine. The loader skips it.

When this is worth the extra file

Situation	Journal?	Why
Single one-shot call	No	Nothing to resume
Batch under ~1 minute	No	Rerun is cheaper than the code
50+ external calls	Yes	Partial progress is real value
Paid per call	Yes	Losing work costs money
Side effects outside the box	Yes, plus dedupe	At-least-once will repeat

If a row says "No", skip this. Simplicity wins.

Limitations, stated plainly

The journal only survives if the disk survives. Copy it off the box.

A git push, an rsync, or an object upload after every N records is enough.

Keep one writer per journal file. Do not lean on O_APPEND for multi-process safety.

fsync per record costs time. Batch it if your item rate is high.

And this pattern does not give you exactly-once. It gives you resumable at-least-once. Design downstream effects accordingly.

Who should not use this

Anyone whose whole job fits in one model call. Stop here.

Anyone with a real job queue and workers. Use that instead.

Anyone who needs strict ordering across items. This runner has none.

The code above is a teaching artifact, not a reviewed library. Check the timeout, error handling, and retry behaviour against your endpoint before a real run.

What changed in my head

I stopped treating the shell session as state. It is a window, not a ledger.

I stopped measuring progress by log lines. I measure it by unique journal keys.

And I stopped asking "how long will this take?". I ask "what happens if it dies at item 41?".

Try the kill -9 test once. If your rerun starts at item 1, you found your bug.

If you want to run that test without a bill, MonkeyCode's free model access and free server option are one place to start. Point BASE_URL and MODEL at whatever endpoint you actually use.