48-Hour Field Notes: The Box Recycled My Worker and the Jobs Stayed running

What do you actually own when the worker that runs your jobs lives on a machine you do not control? I spent 48 hours answering that with a small Python runner, a SQLite file, and a habit of killing my own processes on purpose. Three things broke. Two of them were my fault, and one was a clock I had never questioned.

The 48-hour setup

I wanted the smallest possible "prompt in, patch out" triage worker: submit a job, call a model endpoint, store the answer, mark the job done. For the generation step I pointed the runner at MonkeyCode's free model access, and I ran the worker on a free server option instead of my laptop, because the whole experiment was about what happens when the box under me disappears. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free endpoint as a black box that can stall, truncate, or return a different shape than yesterday, and verify current availability and limits yourself before you build a habit on top of it. The runner below does not depend on any specific provider; it depends on the contract you enforce at the boundary. Let me show you the three failures in order, because the fixes only make sense once you see what I misread.

Failure 1: the truncated stream looked like a successful answer

My first decoder appended every chunk it received and returned the joined text when the loop ended. Can you guess what happened when the connection closed halfway through? The loop ended, the function returned, and the job was cached as done with a sentence that stopped mid-word. The bug was not the network; it was that "the loop ended" and "the answer finished" were the same condition in my code.

The fix is to require a terminal signal that only a complete response can produce. If your provider streams SSE with a sentinel, look for it; if it returns a single body, validate the finish reason field instead.

import json

class TruncatedStream(RuntimeError):
    def __init__(self, chunks):
        super().__init__(f"stream ended after {chunks} chunks without a terminal signal")
        self.chunks = chunks

def read_answer(resp, is_terminal):
    """resp: line iterator from your HTTP client. is_terminal: callable(event) -> bool."""
    events, finished = [], False
    for raw in resp:
        if not raw.strip():
            continue
        event = json.loads(raw)
        if is_terminal(event):
            finished = True
            break
        events.append(event)
    if not finished:
        raise TruncatedStream(len(events))
    return "".join(e.get("text", "") for e in events)

Two rules came out of this. First, never write a partial answer to a cache or a job row, since a wrong cached answer outlives the incident. Second, record len(events) on failure, because a stream that dies at chunk 3 and one that dies at chunk 300 are different bugs. And no, retrying the same request blindly is not the fix when the upstream is the one cutting you off.

Failure 2: the worker vanished and the lease never came back

The second failure was state, not text. My runner marked a job running in SQLite and only returned it to queued in a finally block. Then the box recycled the process and there was no finally to run, so every in-flight job stayed running forever, and my reaper did not exist yet. Does that sound like an infrastructure problem? It is a data model problem wearing a costume.

The replacement was a lease: claim a job with an expiry, heartbeat while you work, and let any other process reclaim expired rows. Here is the claim statement, which needs SQLite 3.35 or newer for RETURNING.

CREATE TABLE jobs (
  id INTEGER PRIMARY KEY,
  idempotency_key TEXT NOT NULL UNIQUE,
  state TEXT NOT NULL DEFAULT 'queued',
  attempt INTEGER NOT NULL DEFAULT 0,
  owner TEXT,
  lease_until REAL,
  created_at REAL NOT NULL,
  updated_at REAL NOT NULL,
  result TEXT
);

UPDATE jobs
SET state = 'running', attempt = attempt + 1,
    owner = :owner, lease_until = :now + :lease, updated_at = :now
WHERE id = (
  SELECT id FROM jobs
  WHERE state = 'queued' OR (state = 'running' AND lease_until < :now)
  ORDER BY created_at LIMIT 1
)
RETURNING id, attempt, idempotency_key;

The reaper is one statement that runs on every tick, including in the process that just started:

UPDATE jobs
SET state = CASE WHEN attempt >= :max_attempts THEN 'failed_dead' ELSE 'queued' END,
    owner = NULL, lease_until = NULL, updated_at = :now
WHERE state = 'running' AND lease_until < :now;

A heartbeat every few seconds extends the lease only for the current owner, so a slow worker is not stolen from mid-run. Keep the lease comfortably longer than your p99 job time, or you will pay for the same work twice.

Failure 3: my expiry math trusted the wrong clock

Expiry logic compares two timestamps, so which clock produced each one matters. I wrote time.time() on the box, compared it against a timestamp I generated locally, and my lease looked expired the moment it was created. Here is the cheapest skew probe I know, using the Date header that HTTP responses already carry.

import datetime, email.utils, requests

def skew_seconds(url="https://example.com"):
    r = requests.head(url, timeout=5)
    server = email.utils.parsedate_to_datetime(r.headers["Date"])
    local = datetime.datetime.now(datetime.timezone.utc)
    return (local - server).total_seconds()

Be honest about the resolution here: HTTP defines the Date header with one-second granularity, and the measurement includes round-trip time, so anything under about two seconds is noise. That is still enough to catch the offset I had, but if you need tighter guarantees, sync the clock properly and monitor the offset instead of inferring it from HTTP.

The test plan I would repeat

Numbers are boring until they save you a weekend, so here is the sequence I would run again before trusting any worker. The point is to make each failure happen on purpose, while you are watching.

1. Queue 20 jobs that each sleep three seconds, then kill -9 the worker at a random moment and start it again. Assert that no job stays running past lease + reaper interval.
2. Point the generator at a stub endpoint that closes the socket after N chunks, and assert you get TruncatedStream and zero rows written to result.
3. Submit the same idempotency_key twice and assert one row, one model call, one answer.
4. Log skew_seconds() hourly for 48 hours against a host you trust, and confirm your lease margin covers the worst value you saw.

What I would not repeat, and who should skip this

I would not run a reaper that only lives inside the worker process, because the worker is exactly the thing that dies. I would not set max_attempts to one, since a truncated stream is not a permanent failure and you will discard recoverable work. And I would not use a shared SQLite file across many writers, because the single-writer model turns a busy queue into lock contention.

If your jobs send emails, move money, or call a non-idempotent third-party API, this design is not for you until you add a real idempotency guard on the side effect itself. If you have sub-second latency targets or a managed queue already in place, the lease-and-reap loop is extra machinery you do not need. And if you cannot tolerate a job being attempted twice, at-least-once semantics will eventually find you.

The decision table I keep next to the code

Symptom	Likely cause	First check	Fix
Job stuck in running	worker died without releasing	lease_until < now with no heartbeats	lease expiry plus reaper
Short answer, valid JSON	stream cut at a chunk boundary	terminal signal missing	require terminal signal, never cache partials
Lease expires instantly	clock offset between hosts	skew probe against Date	single time source, wider margin

None of these fixes are exotic, and that is the point: the interesting part of running work on a box you do not own is deciding what your code promises when the box goes away. If you point your own runner at a free endpoint and run the truncated-stream stub, I would like to know whether your provider signaled completion with a sentinel or a finish reason, because that single choice changes the decoder you should be writing.