State Machines & Durable Execution with just Postgres

I spent a weekend learning every component I'd need to build a durable state machine from scratch. Queues, checkpoints, recovery logic, lease management, idempotency keys, backoff strategies, the lot. I was sketching out the architecture for a walkthrough post, convinced that durable execution meant either assembling it yourself or buying into someone else's distributed system. I had pages of notes. Diagrams. A half-written Python prototype that handled polling and crashes, complete with a janky requeue mechanism for orphaned tasks that I already knew would race under load. Then I found Absurd, and realised someone had already built exactly what I was mapping out — except they had done it inside PostgreSQL, using stored procedures and native tables, and the result was so obvious in retrospect that I felt slightly foolish for not seeing it sooner.

The database you already run is the only orchestrator you need for durable execution, and the specialised platforms exist to solve a scale problem most teams do not have.

The Lie of Stateless Logic

We design backend systems as if execution is atomic and infrastructure is polite. The gap between that assumption and production reality is where user data disappears.

I learnt this the hard way on a sign-up flow. A container restart mid-deployment killed the process halfway through creating an account. The user record had been written to the users table. The payment intent was stuck in "processing" because the Stripe API call had fired but the callback handler never ran. The welcome email never sent. When I traced it the next morning, I found three half-created accounts, two duplicate emails from users who'd hit "sign up" again after hearing nothing, and a trail of application logs that simply stopped mid-line. The database had partial state. The email provider had no record. The users had no confirmation. And the application itself had no memory of any of it because the execution state had lived in a process that no longer existed.

The fix my team reached for was familiar. Add retries. Make the steps idempotent. Wrap the whole thing in a transaction and hope the database commit finishes before the pod dies. But transactions do not span external API calls. Idempotency keys expire. Retries without state are just gambling with different odds. We treated the symptoms because treating the cause would mean admitting that our architecture was built on a fiction: the fiction that a function call is a single uninterrupted breath. In production it is interrupted constantly. Deployments roll. Nodes evict. Networks partition. The state you need to survive these interruptions is not in your application code. It is in the execution state itself — the exact line you reached, the result you obtained, the decision you made — and if you haven't made that explicit and durable, you do not have it at all. You have hope dressed up as architecture.

The worst part is that this pattern is not rare. It is the default. Every framework that promises "scale out of the box" teaches you to keep your functions stateless and your data in the database. But it never teaches you to keep your execution state there too. So you end up with a database full of business data and a fleet of processes full of implicit, ephemeral, unrecoverable execution state. When the process dies, the business data is safe but the execution is nonsense. The database is durable. The application is not. And the gap between them is where your users suffer.

The Orchestration Tax

The standard response to this fragility is to outsource the problem. Buy a platform. Let AWS Step Functions, Temporal, or Airflow handle the state machine for you. The promise is durability without the thinking. The reality is that you have traded one failure mode for another, and most teams pay the tax before they need the benefit.

My engineering lead said something like: "Step Functions is the safe choice. AWS manages it." So I opened the pricing calculator. A three-step sign-up flow — validate, process, confirm — costs per state transition. Every retry, every pause, every wake-up is a billable event. The console shows you a pretty graph of your execution, but when something sticks at 02:00, that graph is the last thing you need. You need the raw state. You need to know which checkpoint failed and why. And the managed abstraction has deliberately made that opaque because its business model is based on you not needing to look inside. You are paying for a black box that bills you by the glance. The JSON DSL compounds it — you are asked to express business logic in a format designed for machine consumption, then charged for the privilege of executing it. A simple loop becomes a nested state machine. Error handling is a separate branch.

Temporal is more honest — it gives you the code — but it also gives you a cluster to run, a database to back it, and a set of failure modes that now belong to you. Airflow brings its own scheduler, its own workers, its own dependency hell. You are now running a distributed system to manage a state machine, when the database you already run is itself a state machine.

The pattern is not new. Telephone exchanges in the 1920s used discrete states and explicit transitions. TCP handles packet loss by recording progress and resuming from the last acknowledged byte. The idea of durable execution is ancient. What is modern is the insistence that implementing it requires infrastructure you do not already have.

Yes, these platforms handle scale and cross-team coordination that a single Postgres instance cannot. But most workflows are not at that scale. The tax is paid from day one: the DSL to learn, the cluster to operate, the pricing model that penalises you for every transition in a long-running flow. The smaller your team, the larger the burden of the orchestrator relative to your actual work. A team of five should not be running a Temporal cluster. Step Functions is a plane. Temporal is a plane. Absurd is a bicycle. Most of your journeys are short enough that the bicycle is faster, cheaper, and more fun.

The Postgres State Machine

Absurd proves that the only part of the stack that needs to be durable is the database, and it already is.

The "Just Use Postgres" meme has circulated for years. Absurd makes it concrete. It pushes workflow state, queues, and task coordination directly into native PostgreSQL tables using stored procedures. Each queue provisions its own set of tables — t_ for tasks, r_ for runs, c_ for checkpoints — all native rows you can query with psql when you need to. Workers are thin stateless processes. They poll for tasks via SKIP LOCKED, acquire a time-limited database lease, execute business logic, and write checkpoints back. The lease is just a timestamp on the row. If a worker crashes before writing the next checkpoint, the timestamp expires, the row unlocks, and another worker picks it up on the next poll. No coordinator. No push pipeline. Just Postgres and a polling loop.

Before any of that runs, you provision the queue. One command, or a Liquibase changeset if you prefer migrations tracked in source control:

absurdctl create-queue signup --storage-mode partitioned

That creates the t_signup, r_signup, c_signup tables and registers the queue's retention policy. Everything from here lives in the database.

The worker is a Python process, thin enough to read in one screen:

import logging
from datetime import datetime, timezone
from absurd_sdk import Absurd

logger = logging.getLogger("absurd-worker")
app = Absurd(queue_name="signup")

@app.register_task("provision-user", default_max_attempts=5)
def provision_user(params, ctx):

    # Step 1: Create user record
    def create_user_record():
        return {
            "user_id": params["user_id"],
            "email": params["email"],
            "created_at": datetime.now(timezone.utc).isoformat(),
        }

    user = ctx.step("create-user-record", create_user_record)

    # Step 2: Simulate a transient failure so the retry behavior is visible
    outage = ctx.begin_step("demo-transient-outage")
    if not outage.done:
        ctx.complete_step(outage, {"simulated": True})
        raise RuntimeError("temporary email provider outage")

    # Step 3: Send activation email
    def send_activation_email():
        return {"sent": True, "provider": "demo-mail", "to": user["email"]}

    delivery = ctx.step("send-activation-email", send_activation_email)

    # Step 4: Suspend until the activation event arrives
    activation = ctx.await_event(
        f"user-activated:{user['user_id']}",
        timeout=3600,
    )

    return {"user_id": user["user_id"], "status": "active", "activated_at": activation["activated_at"]}

if __name__ == "__main__":
    app.start_worker(worker_id="worker-1", concurrency=4)

Each ctx.step() call is a checkpoint. The function inside only runs if that checkpoint does not already exist in c_signup. The deliberate failure in step 2 is the thing worth watching closely. When the worker crashes after completing step 1, the next run reads the cached result for create-user-record, skips the side effect entirely, and continues from send-activation-email. You can see this in the logs directly:

# Attempt 1 — creates the user record, then fails
[019ee555] creating user record for alice
[019ee555] simulating a temporary email provider outage

# Attempt 2 — replays the checkpoint, skips step 1, continues from step 3
[019ee555] sending activation email to [e-mail]
[019ee555] waiting for user-activated:alice

Step 1 did not run twice. No duplicate record. No duplicate email. The database remembered what the application had forgotten.

The await_event call suspends the task — releasing the worker's lease — until the client emits user-activated:<id>. At that point, a new run is scheduled and the task completes. The Bun client that triggers all of this is equally thin:

import { Absurd } from "absurd-sdk";

const app = new Absurd({ queueName: "signup" });

Bun.serve({
  port: 3000,
  routes: {
    "/users": {
      POST: async (req) => {
        const body = await req.json();
        const spawned = await app.spawn("provision-user", body, {
          queue: "signup",
        });
        return Response.json(spawned, { status: 202 });
      },
    },
    "/users/:userId/activate": {
      POST: async (req) => {
        const { userId } = req.params;
        await app.emitEvent(`user-activated:${userId}`, {
          activated_at: new Date().toISOString(),
        });
        return Response.json({ emitted: `user-activated:${userId}` });
      },
    },
  },
});

Here is the full sequence of what happens between them:

sequenceDiagram
    autonumber
    participant Client as App / Scheduler
    participant DB as PostgreSQL (Absurd Engine)
    participant Worker as Pull Worker

    Client->>DB: Spawn Task (Payload + Idempotency Key)
    Note over DB: Validates key, writes row to t_<queue>

    Worker->>DB: Poll Task via SKIP LOCKED
    DB-->>Worker: Lock acquired, lease assigned
    Note over DB: Writes attempt tracking to r_<queue>

    Worker->>DB: Write Step Checkpoint
    Note over DB: Persists value to c_<queue>, extends lease

    Note over Worker: Crash / Network Fault (Lease Timeout)

    Worker->>DB: Re-Poll Task via SKIP LOCKED
    Note over DB: Increments attempt in r_<queue>
    Worker->>DB: Read Checkpoints
    DB-->>Worker: Cached Step Results
    Note over Worker: Skips executed side-effects

    Worker->>DB: Complete Task (Final Result)
    Note over DB: Updates state in t_<queue>

When I simulated a container restart with this setup, the task resumed from the last completed checkpoint. No half-created accounts. No stuck payment intents. The difference is not incremental — it is architectural. Before, the durability of the flow was an emergent property of multiple systems behaving correctly together: the application server, the message queue, the database, the orchestrator. Now the durability is intrinsic to the storage. The queue is a table. The checkpoint is a row. The lease is a timestamp. There is no emergent behaviour to debug when things go wrong. There is only PostgreSQL, and you already know how to debug that.

The Visibility Problem

Trading away the orchestrator is only half the battle. You cannot trade away the need to see what is happening.

The primary risk of database-native workflows is the black box. When your state machine lives in PostgreSQL tables, you need a way to inspect it without writing raw SQL at 02:00 while a production task is stuck. Habitat is the UI Absurd provides for this: a read-only web dashboard that connects directly to Postgres and shows queues with task counts, per-task run history with each attempt's status and error, checkpoint JSON for every completed step, and event payloads with timestamps. You can see exactly which step a stuck task reached and what it returned, without reconstructing it from CLI output. It is not a luxury. It is the difference between a database trick and a system you can actually operate.

Without it, you are left with psql and a growing set of internal queries that someone wrote once and no one dares change. Habitat removes that friction. It runs as a single Go binary with an embedded frontend, pointed at the same Postgres instance:

./habitat run -db-name mydb -db-user postgres -db-password postgres
# navigate to http://localhost:7890

Or drop it into Docker Compose alongside the worker and client, and it just works.

But the limitations are real and worth naming. This is single-instance PostgreSQL. It does not have built-in cross-region replication. The operational model shifts from "managed service I pay for" to "system I own and understand." You need to know how PostgreSQL locking works. You need to monitor table bloat and configure pg_cron for cleanup. This is not a managed service. It is a tool that respects your intelligence and demands your attention. That is a trade-off, not a defect.

The honest boundary is this: if you need cross-team coordination at scale, or you are orchestrating workflows across a hundred microservices, Step Functions or Temporal earn their keep honestly. They are not bad tools. They are solving a real problem — just not the one most teams have. For the majority of workflows that live inside a single service boundary, you are paying for complexity you do not use.

The most sophisticated architecture is often the one that removes the most components. The platforms selling you durability are not wrong. They are just solving a problem most teams do not have, at a price most teams should not pay.

---

The full working example — queue setup, Python worker, Bun client, Docker Compose, and Liquibase migrations — is in the Absurd repository. The documentation covers partitioning, cron patterns, rolling deployments, and agent tooling.