Idempotent by Default: State Machines for Background Tasks in Python

The Real Cost of "It Works on My Machine"

Background tasks get a design pass no other code gets. A web endpoint is reviewed, its input is validated, its failure path is traced. A cron job that writes rows to a table is the exception — its default is agreed-on-paper and its failure behavior is an afterthought.

Most Python background work begins life as a tight loop: fetch an id, do the work, mark it done, repeat. That shape is honest about the happy path and dishonest about everything else. When the process dies between "fetched" and "marked done", the loop runs the same record a second time on restart. When a worker reconnects after the database hiccups, it does not know which step committed already, so it replays the whole batch. When a second instance starts because the first one is running slow, both lay claim to the same row. The codebase is exactly correct in the junction where a crash is most likely, which is also the single moment it can least afford to be wrong.

This article argues that the walk-way of disaster is not the crash. It is the scattered. When the question "has this unit been processed" lives across twenty loose branches, every crash demands a fresh re-accounting by hand. The stable fix is to stop treating the task body as the unit of progress and to start treating a single field — the task state — as the unit of truth.

State as Data, Not as Control Flow

The happy-path loop keeps exclusivity buried in conditionals: if processed: skip else: process. That arrangement lasts until a third, fourth, or sixteenth branch appears, and the provenance of each row becomes a palimpsest.

The resilient alternative is to model the whole lifecycle as an enumerated value and let the value drive a single dispatch. Every row in the pipeline holds exactly one state at a time. Every handler is a pure function that returns the next state rather than mutating in place. "Is this processed?" becomes a query on one cell of one table, not a tour of twelve branches.

from enum import Enum

class TaskState(str, Enum):
    NEW      = "new"
    ACQUIRED = "acquired"
    WORKING  = "working"
    DONE     = "done"

EDGES = {
    TaskState.NEW:      (TaskState.ACQUIRED,),
    TaskState.ACQUIRED: (TaskState.WORKING,),
    TaskState.WORKING:  (TaskState.DONE,),
    TaskState.DONE:     (),
}

def step(state, next_state):
    if next_state not in EDGES[state]:
        raise InvalidTransition(state, next_state)
    return next_state

The enum is not decoration; it is the invariant. Any call that would push a row from WORKING straight to NEW is rejected before it can corrupt history, and the rejection is a loud exception rather than a silent break.

Why Guarded Transitions Beat Comments

A comment is a summary of what an author meant that day, not a statement about what the machine can permit. When every rule is enumerated in the code, no one has to reconstruct a parallel mental model from prose. Because the transition map is data, it can be checked at import time with a short reachability walk: assert every declared state appears in at least one edge's source, and every edge's destination is a declared state.

That little ceremony catches the two most expensive bugs in practice: dead states that no edge can reach, and final states that somehow still idle out to WORKING. Codified against a pipeline that had grown to six states across three modules, the traversal caught a ghost transition in a hot loop that let a DONE row be resurrected to FAILED and double-processed. A human review had skimmed past it for three code reviews because the path was scattered across files.

Transactions Decide the Crash Fate

A state column is only as trustworthy as the write that sets it. The classic break: a worker sets its own state to DONE, the process crashes between the write and the commit, and the row shows WORKING again. A redeployed worker reads WORKING, re-locks the row, and reprocesses work whose side effects already landed.

Closing that window demands two tool. First shrink it: the commit that flips the state must be the last write in the transaction, and it must carry the computed body along with it. When the commit returns, DONE is true and no body is half-written. Second, make the fallback harmless.

Make Every Side Effect Idempotent

Idempotency is the patient brother of the state machine: when a handler executes twice, it must produce the same observable outcome as when it executes once. A delete is idempotent. An append is not.

A remote queue is the place where this airs first, because a claim followed by a response timeout does not tell you whether the message arrived. The only safe assumption is that it did. So the emphasis is on making the failure cheap, not on asking the network to be reliable:

def run_once(key, action):
    token = redis.set(key, "1", nx=True, ex=3600)
    if token:
        action()
    else:
        record_skip(key)

Whichever worker wins the set operations is the one allowed to run; any race that loses simply observes the key and exits. Redelivery becomes harmless by construction, and the cost of scheduling an exactly-once ledger is deferred to a token expiry, which is a fair trade for most jobs.

A Linchpin Worth a Transaction

Pulling the patterns together, one worker reads a batch, transitions each row through the guard, and commits body and state together:

UPDATE jobs
SET    state = 'done',
       body  = ?
WHERE  id = ? AND state = 'working' AND lock_token = ?

The WHERE clause is an optimistic lock. When another worker has already advanced the row, the update changes zero rows and the caller can ignore it. The body survives exactly if and only if the state writes, because they land in the same transaction. The read happens under the same idea, and the guarded step function is the bridge that keeps the state machine a coherent reason.

• A claim with SELECT ... FOR UPDATE locks the row, advances it to WORKING in the same transaction, and holds the whole unit for the step.
• A token claim instead stamps a short-lived token and checks it only at the transition point.

Both return a worker that is honest under four failures: process death mid-transaction, duplicate delivery from transport, a second instance claiming the same row, and a database-down enough-to-retry case.

async def drain(db, batch_size):
    async with db.cursor() as cur:
        rows = await cur.fetch_many(batch_size)
        for row in rows:
            body = await compute(row)
            await cur.execute(COMMIT_SQL, (_next(), body, row["id"],
                                            row["state"], row["ack"]))
    await db.commit()

The loop never claims more responsibility than it holds. When a crash arrives the transaction restores the boundary; on the next spin the same row is offered to a worker as if fresh, and the idempotent side effect means the transport re-runs no damage.

Retries That Only Move Forward

Failures need a rule, not a mood. A FAILED row must not silently flip back to NEW. It should pass through a retry hop that carries a counter, and that counter must be capped so the job cannot bounce between two states forever. A bounded retry loop terminates in bounded time.

def next_backoff(attempt, base=3, cap=300):
    return min(cap, base * (2 ** attempt))

A task that exhausts max_attempts becomes DEAD, and a human reads it as real: either the body genuinely cannot be satisfied, or the handler is swallowing a reproducible exception. Telling those apart is a one-line log; the point is that the writer stopped trusting the clock and made the decision explicit.

Murder by Kill at Each Window

A state machine is proven by trauma, not by reading. Folder-deploy the pipeline and kill the process at each of the four crash windows, then assert the visible state is the one you expect. A tiny property test sweeps reachable state space with random seeds and confirms that no reachable DONE state still issues an outgoing edge.

def test_no_done_edges(states):
    assert EDGES[TaskState.DONE] == ()

def test_all_states_reachable():
    seen = { TaskState.NEW }
    queue = [ TaskState.NEW ]
    while queue:
        s = queue.pop()
        for nxt in EDGES[s]:
            if nxt not in seen:
                seen.add(nxt)
                queue.append(nxt)
    assert seen == set(TaskState)

These two tests are cheap, deterministic, and catch the two highest-regret bug classes: a state that nothing can reach, and a DONE state that still nominates a successor. Their value is not in elegant grace; it is in letting every later schema migration state plainly "the new edge I am about to cut over cannot take the road to DONE on the way."

The Payoff Outside the Happy Path

Framing a single job route as a state machine costs about forty lines on a new task and a few hours on a growing one. The dividend compounds quietly. The cron job that once redelivered a low on the fifty-fifth-minute retry now records the failed once. The writer that once leaked a dropped row into the done pile now co-commits body and state. The on-call runbook stops asking "did that batch run twice?" and starts asking "which transition aborted, and under what token?" — a question with an actual answer.

None of this is novel research. It is moving fifteen years of database and queue discipline into the half-remembered containers that hobby scripts and hustling data teams actually run. The one fact that makes it worth doing is that a single unglamorous background writer holds the same failure domain as a bank ledger above water: a success that was inadvertently duplicated is a worse outcome than a loss that was observed and reported. A guarded enum, a shared transaction, and an idempotent write turn that worker from a "maybe" into a "certainly-not-twice", and that certainty is worth the forty-line investment on every deployment.

---

Originally published on Dispatch.