The annoying failure is not the one where everything dies before anything important happens. That one is clean. You retry the job, or the user clicks again, or the scheduler runs next minute. Fine.
The real failure is the one where the process dies after step 3.
Step 1 created the customer record. Step 2 reserved something. Step 3 moved money, activated a subscription, sent a document, or called an external provider that will not forget what you asked it to do. Then the worker disappears. Kubernetes evicts the pod. The VM restarts. A deployment kills the process. Someone discovers that "stateless service" was more aspiration than architecture.
Now what? If you restart the job, you might charge twice. If you skip it, the customer is stuck halfway through activation. If you manually repair it, congratulations, you invented an operations queue with anxiety as the UI.
This is the problem durable execution is trying to make boring.
Not easy. Boring.
we used to build this ourselves
In fintech, this shows up everywhere because workflows are longer than a request and more fragile than the happy path diagram.
At a fintech in the Americas, the answer was the one many serious backend teams reached: sagas, Kafka, idempotency keys, state tables, retries, reconciliation jobs, and discipline.
That architecture was not dumb. It was right for its time.
You split a process into steps. Each step publishes an event or writes state. Commands carry idempotency keys. External calls have deduplication at the boundary when possible. Reconciliation jobs scan for weird states later because you know the system will find one. Everyone learns that "at least once" is not a delivery guarantee, it is a personality test.
The hard part was never only writing the saga. The hard part was owning it.
You had to decide what state meant, which failures should pause, which should compensate, and which should page a human. You had to keep idempotency keys flowing across service boundaries. You had to make sure a step that looked atomic in code was atomic in the business.
And because the mechanism was hand-rolled, every team ended up owning some version of the same machinery.
One team had an orchestration table. Another had choreography over Kafka topics. Another had a cron that repaired half-completed rows. All of this worked, until it did not, and then the organizational memory lived in the heads of the people who had been burned before.
This is why DBOS and Temporal are interesting to me. They did not invent the idea that a workflow should survive a crash. They productize the control-plane part we used to rebuild.
what DBOS changes
I first paid attention to DBOS while looking at subscription activation workflows and scheduled sweeps at a bank I work with now in the Gulf. The storage model caught my eye.
DBOS Transact makes workflows durable by checkpointing completed steps in Postgres. If the process crashes, the restarted app can resume from the last completed step.
It also changes the adoption shape. DBOS is not asking every team to operate a separate workflow cluster before they get durability. The open-source Transact libraries are positioned as application-adjacent durable execution built on Postgres-compatible storage. Their material still says the state can live in your application database. DBOS also sells the control-plane layer, with paid plans priced around users, applications, and checkpoint volume.
That matters because the first objection to workflow engines is often operational weight. DBOS is aiming at the space where you want crash recovery, durable queues, scheduled work, and observability without turning every service into research.
The tradeoff is clear. If durable execution metadata is in Postgres, Postgres becomes part of the runtime. That can be a feature or a coupling point. You get simpler ownership and queryability, but you still need to understand retention, migrations, backups, and the blast radius of putting workflow history near application data.
No free lunch. Fewer moving parts.
what Temporal gives you
Temporal has a different shape.
Temporal gives you a durable execution platform where workflows are recorded as event histories. A workflow can run for seconds, days, or years, and the service persists the events needed to recover and continue. Activities run outside deterministic workflow logic. The SDK replays workflow history to rebuild state.
That model is powerful when the workflow is long-lived, crosses services, and needs signals, timers, retries, versioning, and a strong operational surface.
Temporal Cloud pricing also makes the product shape explicit. It is a managed platform priced around Actions, active storage, retained storage, and support. You are buying a durable execution control plane.
That is a reasonable deal for many teams. It is also a real platform decision. Once Temporal is in the middle of a process, someone owns namespaces, workers, task queues, retry policies, versioning, security, and cost.
the hard part stayed yours
Here is the part that people sometimes skip when durable execution gets exciting.
A framework can remember that step 3 completed. It cannot tell you whether step 3 was the right boundary.
If step 3 called a payment provider, did the provider accept an idempotency key? If the network timed out, did the payment happen or not? If the workflow resumes tomorrow, should it retry, query the provider, compensate the customer, or stop for manual review?
Durable execution makes these questions visible. It does not answer them for you.
That visibility is useful. In hand-rolled systems, step boundaries often leak across topic names, retry loops, database rows, and tribal knowledge. In DBOS or Temporal, you are forced to make the workflow shape explicit. You name the step. You decide what gets checkpointed, what is safe to replay, and what must never be replayed.
That is the real design work.
Not "how do I keep a process alive forever?" The question is, "what does it mean for this business action to have happened?"
That is why fintech engineers get twitchy about retries. Retry is not a technical verb when money is involved. It can mean "try the HTTP call again." It can also mean "charge twice."
The boundary matters.
So does ownership. A durable workflow that crosses payments, subscriptions, notifications, and compliance is not owned by the framework. It is owned by a team, or by nobody.
durability became a purchasable property
The wider pattern is the interesting one.
A lot of backend engineering is the history of boring properties becoming products. We used to build deployment scripts. Then deployment became a platform property. We used to hand-roll metrics pipelines. Then observability became a product category. Some teams still build these things, for good reasons. But the default shifted.
Durable execution is going through the same move.
"This multi-step job survives a crash" used to be a badge of engineering maturity buried inside every serious backend system. Now it is increasingly an infrastructure property. You can install a library, run a platform, or pay for a managed service.
That is good. Fewer teams need to rediscover failure modes from first principles. More workflows can have explicit state instead of vibes. Scheduled sweeps, onboarding flows, subscription activation, refunds, identity checks, and cleanup jobs can stop depending on the fantasy that processes die only at convenient times.
But it also raises the bar for engineering judgment.
If durability is easy to add, teams will add it to things that should have stayed simple. If every background job becomes a workflow, you can create a system that is reliable but hard to reason about.
The question is not "should we use DBOS or Temporal for everything?"
The question is "which business processes deserve durable execution, and who owns the semantics when they resume?"
That is a better question than "how many cron retries should we add?"
And it is where the work belongs.
references
- DBOS, Postgres-backed Durable Workflow Execution
- DBOS pricing
- Temporal, Durable Execution Solutions
- Temporal Cloud pricing documentation
To test my projects, I use Railway. If you want $20 USD to get started, use this link.
Top comments (0)