When Your Backend Starts Building a Workflow Engine

How ordinary queues, databases, and scheduled jobs gradually become custom orchestration infrastructure

When a worker crashes halfway through a business operation, who is responsible for making sure the remaining work gets completed?

That question became more interesting to me as I spent more time working with backend systems. I began to realize that successfully executing an operation is only part of the challenge. We also need to understand what happens when something fails halfway through.

A database update might succeed, but the service could crash before publishing the next event. A payment might go through, but its response could get lost. A worker might disappear after completing some steps, leaving the rest unfinished.

These scenarios gradually changed how I approached backend architecture. Instead of thinking only about whether an operation would succeed, I started asking how the system would recover without losing progress or repeating something that had already happened.

We already have familiar ways to handle these situations. Databases preserve state, queues distribute background work, and scheduled jobs help recover unfinished operations. For many applications, these are perfectly reasonable solutions.

But as I explored workflows involving multiple services, retries, and longer waiting periods, I became curious about something else: what happens when tracking execution progress and recovering from failures becomes a significant responsibility of the system itself?

That question led me to explore durable workflow execution and the engineering decisions behind systems like Uber Cadence, Temporal, and AWS Step Functions.

What interested me wasn't learning another infrastructure tool. It was understanding why engineers built these systems, what problems their designs solve, and what trade-offs they introduce.

More importantly, when does adopting a dedicated workflow engine make more sense than continuing with the databases, queues, and recovery mechanisms we already have?

That's the architectural question I want to explore in this article.

---

How Ordinary Backend Systems Evolve

Imagine we're building a simple order-processing system. When a customer places an order, the backend reserves inventory, processes the payment, creates a shipment, and sends a confirmation.

Initially, we might handle this through a series of service calls. For a small application, there's nothing wrong with that approach. The logic is straightforward, the execution flow is easy to follow, and we don't need additional infrastructure to coordinate every step.

But eventually, some operations take longer than expected. Payment providers might respond slowly, shipment creation could depend on another service, and we don't want customers waiting until every background operation finishes.

So we introduce a message queue.

Now the API can accept the order, and a background worker can process it asynchronously. We've separated the customer's request from the actual processing, which is a reasonable improvement.

Then another requirement appears: we need to know how far an order has progressed, even if the worker crashes.

We add a status field to our database.

ORDER_CREATED
    ↓
INVENTORY_RESERVED
    ↓
PAYMENT_COMPLETED
    ↓
SHIPMENT_CREATED
    ↓
COMPLETED

If a worker fails, another worker can inspect the stored state and determine what needs to happen next. Of course, we still need safeguards to prevent duplicate processing or repeating operations that may have already succeeded.

So far, we're still using familiar backend components. We have a database for persistent state, a queue for distributing work, and workers to execute operations.

But failures introduce new questions.

What if the database records a successful payment, but the process crashes before publishing the shipment task? What if two workers pick up the same order? What if the payment provider is temporarily unavailable, and we need to retry after several minutes?

We start adding recovery mechanisms. A retry counter, a timestamp for the next attempt, perhaps a scheduled job to find unfinished orders. As multiple workers become involved, we may also need leases to prevent them from processing the same operation concurrently.

A lease is simply temporary ownership of work. If one worker disappears, another can take over after that ownership expires.

Over time, our architecture might look something like this:

For many applications, this architecture is already a reliable and maintainable solution.

The interesting change happens when the requirements keep growing.

An order might need to wait for manual approval. A failed shipment might require a refund and an inventory release. Some operations may need to resume after several days, while others must survive deployments or temporary outages.

Our code is no longer responsible only for performing business operations. It is also responsible for tracking execution progress, deciding what runs next, scheduling retries, recovering interrupted work, and coordinating actions across services.

We may eventually introduce a dedicated coordinator to manage all of this.

And at that point, an interesting question emerges:

Are we still building business logic, or have we started building the infrastructure needed to execute that business logic reliably?

That distinction is where the architectural discussion becomes interesting. Not because our existing approach has suddenly become wrong, but because coordination itself is becoming a responsibility we need to design, operate, and maintain.

---

When Coordination Becomes Infrastructure

At this stage, we already have a system that can persist its state, process background jobs, and recover from failures. So why would we need anything more?

The answer isn't simply that our workflow has more steps or involves more services. A well-designed database state machine with a message queue can handle surprisingly complex operations.

The real question is how much responsibility we're taking on to keep those operations moving correctly.

Consider a payment that fails because the provider is temporarily unavailable. We can schedule a retry. That's straightforward. But what if the payment actually succeeded and the response was lost? Now we need idempotency or reconciliation to determine what happened.

Suppose the payment succeeds, but the worker crashes before scheduling shipment creation. We need a reliable way to ensure that the next task isn't lost. A transactional outbox can solve that problem by recording the state change and the intent to publish work in the same database transaction.

These are established solutions, and they work well.

Now imagine the same workflow needs to wait three days for approval, coordinate several external services, handle cancellation halfway through, and compensate for previously completed operations when something fails permanently.

We need to know not just the current business state, but which operations completed, which are still running, which can safely retry, and what should happen after a failure.

And this information must remain reliable even when workers crash, services restart, or the application is deployed again.

None of these requirements individually demands a workflow engine. The challenge is their combination.

Where does the boundary lie?

There's no fixed number of services, workflow steps, or retries that tells us when to introduce durable orchestration.

I think a more useful way to evaluate the architecture is to ask:

Recovery: Can we reliably resume interrupted operations without complicated manual intervention?

Coordination: Are we repeatedly implementing timers, retries, worker ownership, state transitions, and failure handling?

Visibility: Can we understand where a business process is stuck and what needs to happen next?

Maintenance: Is more engineering effort going into maintaining execution machinery than implementing the actual business workflow?

These questions help us identify when coordination is becoming a system of its own.

And there's an important trade-off here. Introducing a workflow engine doesn't eliminate complexity. It moves much of the generic execution and recovery machinery into a dedicated platform, while adding its own infrastructure, operational costs, and programming constraints.

For a straightforward payment lifecycle, PostgreSQL, a queue, and a reconciliation job may remain the better choice. For a multi-day onboarding process involving several external systems, approvals, timeouts, and compensating actions, maintaining our own coordination infrastructure may become harder to justify.

The point isn't to replace simpler tools as soon as workflows become complicated. It's to recognize when we're repeatedly building infrastructure that a dedicated workflow engine already provides.

This is the architectural problem that systems such as Uber Cadence, Temporal, and AWS Step Functions were designed to address.

Their common idea is surprisingly simple: the lifetime of a business process shouldn't depend on the lifetime of the worker executing it.

The interesting part is how they make that possible.

---

How Durable Execution Actually Works

The core idea behind durable execution is to separate the progress of a business operation from the process executing it.

Instead of depending on a worker's memory to remember everything, systems like Temporal and Uber Cadence maintain a durable execution history. This history records important events, such as when an activity was scheduled, when it completed, what result it returned, and when a timer was created.

Let's return to our order-processing example.

Suppose inventory has been reserved and payment has been completed. Before the workflow can schedule shipment creation, the worker crashes.

In an ordinary application, we would need our own recovery logic to determine the next step. With Temporal, the workflow's execution history has already recorded the completed activities, assuming their results reached the engine successfully.

Now another worker can take over.

### Workflow: order-123

1. Workflow started
2. ReserveInventory scheduled
3. ReserveInventory completed
4. ChargePayment scheduled
5. ChargePayment completed

### Worker A crashes

6. Worker B receives a workflow task
7. Previously recorded results are replayed
8. Shipment creation is scheduled

The interesting part is how Worker B knows where to continue.

Temporal doesn't restore the original JVM thread or recover its memory from a snapshot. Instead, it uses deterministic replay.

The replacement worker runs the workflow code against the recorded execution history. When it encounters an activity that has already completed, the SDK uses the stored result rather than executing that activity again. As the code progresses, it reconstructs the workflow's in-memory state until it reaches the point where new work is required.

This is why Temporal distinguishes between workflows and activities.

A workflow contains the coordination logic: which operation should run, what should happen after it completes, and which conditions determine the next step. Activities perform the actual work, such as calling a payment provider, updating a database, or invoking another service.

That separation makes replay possible. Workflow decisions must remain compatible with the recorded history, while external side effects happen through activities whose outcomes can be persisted.

It also introduces a constraint. If workflow code makes decisions using uncontrolled randomness, system time, or direct external calls, replay might follow a different path. Temporal provides replay-safe mechanisms for these situations, but developers must still design workflow code with determinism in mind.

The same architecture supports long-running waits. A workflow waiting three days for approval doesn't need a worker thread sleeping for three days. The engine records the timer or waiting state and schedules more work when the appropriate event occurs.

But there's an important limitation.

Durable execution does not guarantee that an external operation happens exactly once.

If a payment succeeds but its completion never reaches the workflow engine, the activity may be retried. The payment operation still needs idempotency, and some failures may require reconciliation or compensation.

What the Engine Handles, and What We Still Own

This distinction is important. The engine can preserve execution progress, retry failed activities, and coordinate recovery. But it cannot determine whether an external payment was successfully processed unless that outcome is reported or reconciled. Nor can it decide what a correct business compensation should be. Those responsibilities still belong to the application.

This is the trade-off behind the abstraction. A workflow engine takes responsibility for durable coordination, scheduling, and recovery, but it doesn't eliminate the need to design correct business operations.

And making that infrastructure reliable introduces its own engineering challenges. Recording history is only one part of the problem. The engine must also ensure that state changes and the tasks they create cannot become inconsistent.

That is where the architecture of real workflow platforms becomes particularly interesting.

---

What Production Engineering Teaches Us

Understanding how durable execution works is one thing. Seeing the trade-offs engineers faced while building and adopting these systems makes the architecture much more interesting.

Two examples stood out to me while exploring Uber Cadence and Temporal.

Uber Cadence: When Orchestration Adds Latency

In 2019, Uber published an engineering case study about redesigning its customer-support ticket-routing system using Cadence.

The original routing logic had grown complicated over time. Matching tickets to support agents involved language, location, issue type, priority, and several business-specific rules. As those requirements increased, the logic became scattered across different parts of the codebase, making it harder to test, modify, and debug.

Uber moved toward a workflow-based architecture, separating the routing process into smaller activities. This made the control flow easier to understand and allowed individual business operations to evolve more independently.

But this introduced another problem.

Cadence normally distributes activities through task queues, allowing workers to execute them independently. That flexibility comes with communication and scheduling overhead. Uber reported activity-to-activity latencies of around 100 milliseconds, which mattered for a system expected to assign tickets quickly.

So the team made a deliberate trade-off.

They used local activities for the core routing steps. A local activity is a short-running operation executed inside the same worker process as the workflow, rather than being dispatched through the normal activity task queue.

This avoids the additional scheduling round trip and reduces latency. However, because local activities run in the workflow worker process, they lose the flexibility of being deployed and scaled independently.

For Uber, that trade-off made sense. The core routing operations were latency-sensitive, while extensible scoring modules still benefited from independently hosted activities. So the team used local execution where speed mattered and retained distributed execution where flexibility was important.

The team reported more than a 33% improvement in routing-workflow p99 latency compared with its previous implementation. However, starting the workflow introduced additional latency, so the end-to-end latency observed by dependent services remained roughly comparable to the original system.

That's an important distinction. The improvement wasn't free, and the workflow engine didn't automatically make everything faster.

The architectural lesson is that abstraction boundaries have a cost. Distribution improves flexibility and isolation, but for latency-sensitive operations, keeping selected work local may be the better decision.

Source: Uber Engineering, Improving the User Experience with Uber's Customer Obsession Ticket Routing Workflow and Orchestration Engine (2019)

Temporal: Persisting State Isn't Enough if the Next Task Can Disappear

Temporal's architecture reveals a different kind of challenge.

Suppose a workflow completes payment and decides to schedule shipment creation.

The workflow engine must durably record that decision and ensure the shipment task eventually reaches a worker. If recording the workflow's progress and dispatching the task happen as two unrelated operations, a crash between them can create an inconsistency.

The engine might record that shipment creation was scheduled, but the task itself might never reach a worker.

This is similar to the dual-write problem we encounter when building event-driven applications with databases and message brokers.

Temporal addresses this through a mechanism closely related to the transactional outbox pattern.

When a workflow schedules an activity, Temporal persists the workflow's updated execution state and a transfer task together in the same persistence transaction.

A transfer task is an internal, durable record indicating that work needs to be dispatched. It is not yet the activity task delivered to an application worker.

A background processor reads these transfer tasks and forwards the work to Temporal's Matching service, the component responsible for connecting pending tasks with available workers.

If the transfer fails, the durable record remains available for another attempt. The transfer may happen more than once, so the engine also needs to handle duplicate dispatch safely.

The key idea is that the workflow's updated state and the intent to schedule its next task are committed together. A failure during task delivery no longer means that the intent to perform the work has disappeared.

This is the same principle behind a transactional outbox: persist the state change and the record of future work atomically, then deliver that work asynchronously.

I find this particularly interesting because it shows that a workflow engine has to solve the same distributed-systems problems that application developers face.

A system cannot provide reliable workflow recovery if it can lose the tasks required to continue execution.

Durable execution is therefore more than storing workflow history. Underneath the abstraction, the engine must coordinate state transitions, task scheduling, timers, and recovery while preserving consistency across failures.

Sources: Temporal, Designing a Workflow Engine from First Principles (2021) and Temporal's Workflow Lifecycle Architecture.

What These Decisions Tell Us

Uber's example shows that introducing orchestration can improve how complex logic is organized while creating new performance trade-offs. Temporal's design shows how much consistency machinery is required underneath an abstraction that appears simple to application developers.

Neither system makes distributed failures disappear. Instead, they make particular failure and coordination problems manageable through infrastructure designed to handle them.

That is the value of durable workflow execution, but it also raises the final architectural question: when does adopting that infrastructure become more valuable than continuing to maintain our own coordination mechanisms?

---

When Not to Introduce a Workflow Engine

Consider a payment service that stores transaction state in PostgreSQL, processes asynchronous events through Kafka, and uses a scheduled reconciliation job to recover uncertain payments.

If this architecture already handles failures reliably, provides enough visibility, and remains manageable for the team, what exactly would introducing Temporal improve?

A workflow engine could provide built-in execution history, durable timers, and recovery mechanisms. But it would also introduce another infrastructure dependency with its own operational requirements and failure boundaries.

Self-hosting means maintaining the workflow services and their persistence infrastructure. A managed offering reduces that burden but introduces additional costs and platform dependencies. Code-first engines like Temporal also require developers to understand deterministic replay, workflow versioning, and how code changes affect executions that are still running.

And importantly, none of this removes the need for application-level idempotency, reconciliation, or correct compensation logic.

Start with the Problem, Not the Infrastructure

A single background job may only need a queue and a worker. A straightforward business lifecycle might work well with a database state machine. A scheduled recovery job could be sufficient for handling delayed or incomplete operations.

There's no reason to replace these approaches simply because a more sophisticated abstraction exists.

On the other hand, consider an onboarding process involving multiple external services, human approvals, long waiting periods, and compensating actions. If the team is repeatedly implementing and maintaining recovery logic for these operations, a workflow engine becomes a much more compelling option.

The same consideration is becoming relevant for long-running AI agents, where execution can involve external tools, unreliable dependencies, and human intervention. But even there, the presence of multiple steps doesn't automatically justify a durable execution platform.

The decision ultimately comes down to a trade-off: Is maintaining our own coordination and recovery infrastructure becoming more costly and difficult than adopting a dedicated workflow engine?

The Bigger Architectural Lesson

What I found most interesting while exploring durable execution wasn't a particular Temporal feature or Cadence implementation detail. It was understanding why this infrastructure exists and the engineering decisions required to make it reliable.

Uber's experience shows how orchestration can improve modularity while introducing performance trade-offs. Temporal's architecture reveals how much work happens underneath a seemingly simple promise: that execution can continue despite failures.

Both are reminders that abstractions don't make complexity disappear. They move it to a different layer, where it can be managed more consistently.

For me, that's the real takeaway from studying durable workflow execution.

The goal isn't to end up using a workflow engine. It's to understand who owns the progress of a business operation when something fails, and to choose the simplest architecture that can recover it reliably.

---

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree. ❤️