My Nightly Job Ran at 3 PM. Then the Disk Forgot, and the Model Returned Prose.

You know the feeling when a scheduled job runs at the wrong time and the only clue is a suspiciously quiet log file? I spent 48 hours running a small summarization job on a free server with a free model endpoint. Every failure taught me something I wish I had known on day one, and none of it was where I expected it.

The job itself was boring on purpose, and that was exactly the point. It pulled a short feed, asked a free model to summarize each item as JSON, and wrote the results to a local file. Nothing critical, nothing customer-facing, just a nightly experiment that needed to survive without supervision. These are the field notes: what I tried, what broke, and what I would repeat.

The setup I started with

I used MonkeyCode's free model access for the summarization calls and its free server option to host the scheduler. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The first version was naive and happy:

# nightly_summary.py — version one, naive and happy
import json
import time
from pathlib import Path

def summarize(feed_item: dict) -> dict:
    prompt = "Summarize this item as JSON with keys: title, summary, confidence."
    response = model_call(prompt, feed_item["text"])  # free model endpoint
    return json.loads(response)  # this line will hurt later

def main():
    items = fetch_feed()
    results = []
    for item in items:
        results.append(summarize(item))
        time.sleep(2)  # be polite to the free endpoint
    Path("results.json").write_text(json.dumps(results, indent=2))

if __name__ == "__main__":
    main()

Note that model_call and fetch_feed are pseudocode for the free model endpoint and the feed source; everything else is real. I scheduled the script with cron at 02:00, added a log line, and went to bed. The first morning, the log said the job ran at 15:03.

What broke, in the order it broke

1. The server clock was a liar

The free server had been suspended overnight, and when it resumed, its clock was still living in the afternoon. Cron does not care about your timezone expectations; it just fires when the system clock says so. I confirmed the drift with date -u and saw a gap of more than twelve hours, which explained the silent 15:03 run.

The fix was not clever, but it was honest about what a suspended server can do to a schedule. I added a clock check that compares local time against an HTTP Date header, which needs no NTP client and no special permissions:

# clock_check.py — refuse to run if the clock is lying
import email.utils
import time
import urllib.request

def clock_drift_seconds() -> float:
    with urllib.request.urlopen("https://example.com", timeout=5) as resp:
        header = resp.headers.get("Date")
    server_time = email.utils.parsedate_to_datetime(header).timestamp()
    return abs(time.time() - server_time)

def main():
    drift = clock_drift_seconds()
    print(f"clock drift: {drift:.1f}s")
    if drift > 60:
        raise SystemExit("clock too far off; refusing to run")

if __name__ == "__main__":
    main()

One HTTP request per run is cheap insurance, and it turned a confusing log into an obvious one.

2. The disk forgot my checkpoints

I assumed RAM was the only thing that could vanish on a restart, and the next morning proved me wrong. The server had recycled overnight, and my checkpoint file was gone along with the results. On this free tier, the writable disk is ephemeral too, so the home directory behaved exactly like /tmp.

I tried writing to a different folder, then to a different path, and finally stopped pretending. The fix that stuck was pushing a checkpoint to a remote sink after every item, so a restart only cost me the current item:

# checkpoint.py — write progress somewhere the server cannot forget
import json
import urllib.request

def save_checkpoint(remote_url: str, state: dict) -> None:
    payload = json.dumps(state).encode()
    req = urllib.request.Request(
        remote_url, data=payload, method="PUT",
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        if resp.status != 200:
            raise RuntimeError(f"checkpoint failed: {resp.status}")

Point that URL at object storage or your own tiny endpoint; the important part is that it lives outside the server's filesystem.

3. The model returned prose instead of JSON

The prompt asked for JSON, and the free model returned a markdown block with a sentence before it and a sentence after it. json.loads exploded on the first item, and the whole job died before it processed anything. A stricter prompt helped sometimes, but not reliably, so I stopped negotiating with the model.

The fix that worked was a tolerant parser that finds the first fenced code block, extracts it, parses it, and validates the required keys:

# tolerant_json.py — survive prose, find the JSON, validate it
import json
import re

def extract_json_block(text: str) -> dict:
    fenced = re.search(r"```

(?:json)?\s*(.*?)\s*

```", text, re.DOTALL)
    candidate = fenced.group(1) if fenced else text
    data = json.loads(candidate)  # still raises if there is no JSON at all
    required = {"title", "summary", "confidence"}
    missing = required - set(data.keys())
    if missing:
        raise ValueError(f"missing keys: {missing}")
    return data

The parser still fails loudly when there is no JSON, which is exactly what I want; silent corruption is worse than a visible crash.

What I would repeat

• The clock check, before every run. It costs one HTTP request, and it saved me from a week of confusing logs.
• Remote checkpoints after every item. Losing one item is annoying; losing the whole batch is demoralizing.
• The tolerant JSON parser. Free models are chatty, and a parser that expects perfection will fail more often than the model does.
• Raw output logging. I started saving the model's full response to a separate file, and every post-mortem got faster.

Who should not use this approach

If your job needs guaranteed delivery, strict latency, or persistent local disk, this stack is not for you. Free servers recycle without warning, free model endpoints can be slow or rate-limited, and my setup trades reliability for cost. Use this pattern for experiments, prototypes, and non-critical batch work.

For anything customer-facing, put a real queue, a real database, and a real SLA behind it.

The one thing I would tell my past self

The failures were never where I expected them. I assumed the model would be the flaky part, and it was the server clock, the ephemeral disk, and the chatty output that actually broke the job. If you are running a similar experiment on free infrastructure, which failure surprised you first?