DEV Community

Cover image for Your Workflow Crashed. What Happens to the Steps That Already Ran?
The Unmeshed Team
The Unmeshed Team

Posted on Originally published at unmeshed.io

Your Workflow Crashed. What Happens to the Steps That Already Ran?

#softwareengineering #workflow #automation #architecture

Here's a fun way to ruin someone's afternoon.

A workflow is three steps: charging a customer, updating an order, and sending a confirmation email. Step two finishes. Then the server dies for no reason anyone can explain yet.

Now someone has to answer the actual question. Did the charge go through? Is the order sitting somewhere half saved? Do you retry and risk charging twice, or hold off and risk losing the order completely? Congratulations, you've just met the reason durable execution exists.

Most engineers don't learn this concept from a textbook. They learn it from an incident channel, at the worst possible moment, usually while someone is already asking hard questions in Slack.

This guide skips that part. Here's what durable execution actually means, how it works under the hood, and when you genuinely need it instead of just liking the sound of it.

1. What Durable Execution Actually Means

Durable execution is the guarantee that a workflow survives crashes, retries automatically, and resumes exactly where it left off, without losing state or repeating side effects.

Go back to the order example above. The difference isn't luck. It's whether the engine underneath already knows what happened before the crash.

Comparison of a workflow crash with and without durable execution. Without durable execution, order state becomes unknown after a failure; with durable execution, completed steps are preserved and execution resumes from the failed step.

Without it

  • The process restarts from scratch, so the charge might run twice
  • Nobody's sure what state the order is actually in until someone checks logs by hand

With durable execution

  • The engine replays what already succeeded and skips it automatically
  • Execution picks back up from the exact step that failed, nothing more, nothing less

2. How Durable Execution Works

Strip away the marketing, and it comes down to one core mechanism: journaling.

Diagram showing how durable execution works through journaling: a workflow step is recorded to a log, the process crashes, a new worker picks up execution, completed steps are replayed, and the workflow resumes from the failure point.

  • Every step gets recorded to a persistent log before its result is used anywhere else
  • If the process crashes, a new process picks up the workflow automatically
  • Completed steps replay instantly from the log instead of running again
  • Execution continues from the exact point of failure, not from the beginning

No custom retry logic. No manual state tracking. No scheduler bolted on the side to handle the parts that take days instead of milliseconds.

3. Durable Execution vs. Workflow Orchestration vs. Event-Driven Systems

These three get used interchangeably, and that mix-up has caused more confused architecture diagrams than almost any other reliability concept.

Here's the comparison table:

Approach How work is defined Best for
Durable execution Plain code, with the engine handling retries and state Business logic that needs reliability without giving up code control flow
Workflow orchestration A DSL, visual builder, or rules engine Well-bounded processes, especially ones non-engineers need to see or edit
Event-driven/choreography Services react to events independently Loosely coupled, high-throughput systems with no central coordinator

The short version. It keeps you in code and hands reliability to the engine. Workflow orchestration trades some of that flexibility for a visual or rules-based process.

If you want the deeper breakdown on the event-driven side, we've covered that separately in orchestration versus choreography.

4. The Core Properties of Durable Execution

Temporal, Restate, and every other platform in this space publish their own list of core properties. Strip out the branding and four show up every time.

Diagram illustrating four core properties of durable execution: journaled interactions, automatic retries with replay of completed steps, crash-resistant timers and signals, and recovery by any healthy worker.

  • Every external interaction gets journaled, recorded to a persistent log before its result is used, so the log becomes the single source of truth for what actually happened
  • Failed steps retry automatically, and completed steps never re-run, since their recorded result gets replayed instead
  • Durable timers and signals survive crashes too, so a workflow can wait days or months for a human approval or a scheduled follow-up without holding a process open
  • Any healthy worker can pick up an in-flight execution, so recovery doesn't depend on the original machine coming back

Get those four right and workflow durability stops being a per-project decision; it just becomes the default.

5. When You Need Durable Execution (and When You Don't)

Not every workflow needs it. Here's the honest signal list.

Decision guide showing when durable execution is needed. Use it for workflows with non-repeatable steps, long waits for human input, or expensive partial failures. Skip it for single idempotent operations, simple user retries, or applications without multi-step workflows.

You probably need it if

  • A single step failure can't safely repeat, like charging a card or sending a payment
  • The process needs to wait days, weeks, or months for a human or an external event
  • Losing partial progress on a crash is expensive enough to actually matter
  • You want fault-tolerant execution without hand-writing retry and recovery logic for every workflow

You probably don't if

  • The operation is a single, idempotent call
  • Losing progress just means the user clicks a button again
  • You don't have durable workflows that span more than one step or service

If that sounds like your workflows, see it built in.
Durable retries, state recovery, and resume-where-it-failed come standard in Unmeshed, with no separate engine to run.
Try Unmeshed Free

6. How Unmeshed Handles Durable Execution

Unmeshed handles durable execution with the same core guarantee: automatic retries, state recovery, and steps that resume exactly where they left off.

  • Durable step execution with automatic retries and state recovery built into every workflow, not bolted on afterward
  • Human-in-the-loop steps behave like durable signals; a workflow can wait indefinitely for an approval without holding a process open
  • Full run history works like the journal: every step, retry, and recovery is logged and replayable
  • AI steps, rules, and API calls run in the same durable workflow instead of a separate engine bolted on the side
Build it yourself With Unmeshed
A standalone engine like Temporal, Restate, or DBOS Durability and orchestration in the same engine
A separate system for rules, human approval, AI steps No second system to bolt on for approvals or AI steps
Two systems to keep in sync instead of one

Most teams reach for the first option because it's the default path. Fewer of them actually need to.

If you're comparing standalone engines first, we've laid out Temporal alternatives in more depth, including where each one's durability model differs. And for the deeper case on why purpose-built orchestration beats traditional platforms built for slower workloads, that's covered separately too.

Top comments (0)