Building a Workflow Engine in Rust

Because eventually, every backend becomes a machine that waits for something to happen.

There is a strange moment in software engineering when you realize that your application is no longer just an application.

It has become a collection of things waiting for other things.

A user signs up.

Then an email should be sent.

But before that, their account should be verified.

After verification, create a workspace.

Then provision default resources.

Then notify the user.

If something fails, retry.

If the retry fails, wait five minutes.

If it still fails, notify an administrator.

If the administrator approves something, continue.

If the user cancels their subscription halfway through the process, stop everything.

Congratulations.

You are no longer writing CRUD.

You are building a workflow.

And workflows are everywhere.

Stripe processes payment workflows.

GitHub runs CI/CD workflows.

Airbnb coordinates booking workflows.

Uber coordinates ride workflows.

Banks coordinate transaction workflows.

Kubernetes controllers continuously execute reconciliation workflows.

Even your humble "send a welcome email after signup" feature can quietly evolve into a distributed orchestration problem.

This is where workflow engines enter the picture.

And Rust, interestingly, is an incredibly good language for building one.

Not because Rust makes distributed systems magically easy.

It absolutely does not.

But because workflow engines are fundamentally about managing state, transitions, concurrency, failure, and time.

Those are exactly the things Rust forces you to think about carefully.

So in this article, we are going to build a mental model for designing a workflow engine in Rust.

Not a toy match statement pretending to be orchestration.

A real conceptual engine.

Something that understands:

• Workflow definitions
• Tasks
• Dependencies
• State transitions
• Persistence
• Retries
• Timeouts
• Parallel execution
• Failure handling
• Workers
• Scheduling

And most importantly:

How all these pieces fit together without turning your backend into a haunted house.

---

First, What Actually Is a Workflow Engine?

Let's remove the fancy architecture language.

A workflow engine is simply a system that answers this question repeatedly:

What should happen next?

That is it.

Everything else is implementation detail.

Imagine a workflow like this:

          ┌──────────────┐
          │ User Signup  │
          └──────┬───────┘
                 │
                 ▼
        ┌─────────────────┐
        │ Verify Account  │
        └────────┬────────┘
                 │
                 ▼
        ┌─────────────────┐
        │ Create Workspace│
        └────────┬────────┘
                 │
          ┌──────┴──────┐
          ▼             ▼
   ┌─────────────┐ ┌─────────────┐
   │ Send Email  │ │ Create Team │
   └──────┬──────┘ └──────┬──────┘
          │               │
          └───────┬───────┘
                  ▼
          ┌──────────────┐
          │ Workflow Done│
          └──────────────┘

The workflow engine must understand:

1. Which tasks exist.
2. Which tasks depend on other tasks.
3. Which tasks are ready to run.
4. Which tasks are currently running.
5. Which tasks failed.
6. Which tasks should be retried.
7. When the entire workflow is complete.

This sounds simple.

Until you add reality.

Because reality has:

• Network failures.
• Database crashes.
• Workers dying.
• Duplicate requests.
• Slow APIs.
• Timeouts.
• Users pressing buttons twice.
• Servers restarting.
• Tasks that take three days.
• Tasks that depend on humans.

And suddenly your beautiful architecture diagram starts looking like this:

              ┌───────────┐
              │  Reality  │
              └─────┬─────┘
                    │
                    ▼
          ┌───────────────────┐
          │ Everything Failed │
          └───────────────────┘

That is why workflow engines exist.

---

The Core Idea: A Workflow Is a State Machine

This is the most important mental model in the entire article.

A workflow engine is basically a very sophisticated state machine.

A task might have states like:

Pending
   │
   ▼
Ready
   │
   ▼
Running
   │
 ┌─┴─────────┐
 ▼           ▼
Succeeded   Failed
                │
                ▼
             Retrying
                │
                ▼
              Ready

In Rust, we can represent this with an enum.

#[derive(Debug, Clone, PartialEq)]
pub enum TaskStatus {
    Pending,
    Ready,
    Running,
    Succeeded,
    Failed,
    Retrying,
    Cancelled,
}

Already Rust is doing something useful.

Instead of representing task states using random strings like:

"running"
"RUNNING"
"Runing"
"currently_running"

we have a closed set of possible states.

The compiler knows them.

Your IDE knows them.

Your future self, six months from now, might even know them.

That is the beauty of modeling systems explicitly.

Now let's define a task.

#[derive(Debug, Clone)]
pub struct Task {
    pub id: String,
    pub name: String,
    pub status: TaskStatus,
    pub dependencies: Vec<String>,
    pub retry_count: u32,
    pub max_retries: u32,
}

This task knows:

• Who it is.
• What it is called.
• Its current state.
• What must finish before it can run.
• How many times it has failed.
• How many retries it is allowed.

That alone is enough to begin building something interesting.

---

The Workflow Is a Graph

This is where things get more interesting.

A workflow is not just a list.

It is usually a graph.

More specifically, in many cases, it is a Directed Acyclic Graph, also known as a DAG.

Consider this:

A ─────► B ─────► D
 \                    ▲
  \                   │
   └────► C ──────────┘

Task D cannot run until both B and C have completed.

This is not naturally represented as:

Vec<Task>

Because order alone is not enough.

Dependencies matter.

So let's create a workflow structure.

use std::collections::HashMap;

pub struct Workflow {
    pub id: String,
    pub tasks: HashMap<String, Task>,
}

Now we can efficiently look up tasks by ID.

The engine can ask:

Does this dependency exist?

Has this task completed?

Which tasks are waiting?

This is one of those places where software engineering becomes graph theory wearing a hoodie.

---

Finding Tasks That Are Ready

The workflow engine's primary job is figuring out which tasks can execute.

A task is ready if:

1. It is pending.
2. All its dependencies have succeeded.

Let's write that.

impl Workflow {
    pub fn ready_tasks(&self) -> Vec<&Task> {
        self.tasks
            .values()
            .filter(|task| {
                task.status == TaskStatus::Pending
                    && task.dependencies.iter().all(|dependency_id| {
                        self.tasks
                            .get(dependency_id)
                            .map(|dependency| {
                                dependency.status == TaskStatus::Succeeded
                            })
                            .unwrap_or(false)
                    })
            })
            .collect()
    }
}

This looks innocent.

But this is the heartbeat of the workflow engine.

Every orchestration system eventually asks some variation of:

Given the current state of the world, what can I execute now?

That question is basically the entire business.

---

But We Need to Separate Definition from Execution

This is a mistake many workflow systems make early.

They mix the workflow blueprint with the current workflow execution.

Those are two different things.

Imagine GitHub Actions.

The workflow definition might say:

build:
test:
deploy:

That definition is static.

But every time someone pushes code, a new workflow execution is created.

So conceptually, we need:

Workflow Definition
        │
        │ starts
        ▼
Workflow Instance
        │
        ▼
Task Instances

Let's model that.

pub struct WorkflowDefinition {
    pub id: String,
    pub name: String,
    pub tasks: Vec<TaskDefinition>,
}

And:

pub struct TaskDefinition {
    pub id: String,
    pub name: String,
    pub dependencies: Vec<String>,
}

Then runtime execution:

pub struct WorkflowInstance {
    pub id: String,
    pub workflow_definition_id: String,
    pub status: WorkflowStatus,
}

With:

pub enum WorkflowStatus {
    Pending,
    Running,
    Succeeded,
    Failed,
    Cancelled,
}

This separation becomes incredibly important later.

Because your workflow definition might exist for years.

But workflow instances are constantly being created and destroyed.

---

Now We Need Workers

A workflow engine doesn't necessarily execute tasks itself.

It coordinates execution.

This distinction matters.

Imagine our engine says:

Task 47 is ready.

Who actually runs Task 47?

A worker.

The architecture might look like this:

                    ┌──────────────────┐
                    │ Workflow Engine  │
                    └────────┬─────────┘
                             │
                    Finds Ready Tasks
                             │
                             ▼
                    ┌──────────────────┐
                    │    Task Queue    │
                    └──────┬───────┬───┘
                           │       │
                    ┌──────▼──┐ ┌──▼───────┐
                    │ Worker  │ │ Worker   │
                    │    1    │ │    2     │
                    └─────────┘ └──────────┘

The workflow engine orchestrates.

Workers execute.

This separation is incredibly powerful.

Because now you can scale workers independently.

Maybe you have:

1 Workflow Engine
10 Workers

Then:

3 Workflow Engines
1,000 Workers

The architecture can evolve without changing what a workflow means.

---

Defining Executable Tasks

We need some way to represent actual work.

Rust gives us traits.

use async_trait::async_trait;

#[async_trait]
pub trait WorkflowTask: Send + Sync {
    async fn execute(&self) -> Result<(), WorkflowError>;
}

Then we can implement tasks.

pub struct SendWelcomeEmail;

#[async_trait]
impl WorkflowTask for SendWelcomeEmail {
    async fn execute(&self) -> Result<(), WorkflowError> {
        println!("Sending welcome email...");

        Ok(())
    }
}

Another task:

pub struct CreateWorkspace;

#[async_trait]
impl WorkflowTask for CreateWorkspace {
    async fn execute(&self) -> Result<(), WorkflowError> {
        println!("Creating workspace...");

        Ok(())
    }
}

Now our engine can execute different pieces of work through a common interface.

This is the beginning of extensibility.

---

The Problem With In-Memory Workflow Engines

Let's say we build this:

let mut workflow = Workflow::new();

workflow.start();

workflow.execute();

Everything works.

You are happy.

You write a LinkedIn post.

"Built a workflow engine in Rust."

Then the server restarts.

Everything disappears.

Your workflow was halfway through processing a payment.

Gone.

A worker was sending an important email.

Gone.

A task was waiting for approval.

Gone.

This is why serious workflow engines must persist state.

The database is not just storage.

It becomes part of your orchestration system.

You need tables conceptually like:

workflow_definitions

workflow_instances

task_instances

task_attempts

A task_instances table might contain:

id
workflow_instance_id
task_definition_id
status
retry_count
started_at
completed_at

Now when the server restarts, you can reconstruct reality.

And this is a profound concept:

A workflow engine should not trust memory as the source of truth.

Memory is temporary.

Processes die.

Machines restart.

Containers disappear.

The source of truth must survive the process.

---

Persistence Changes Everything

Once state is stored in a database, your workflow engine can do something powerful.

Recovery.

Suppose the database says:

Task A → Succeeded
Task B → Running
Task C → Pending

But the worker running Task B died.

What now?

The workflow engine must eventually detect:

This task claims to be running, but nobody is actually running it.

This introduces something called a lease.

When a worker picks up a task:

Task: Running

Worker ID: worker-17

Lease expires: 10:45:00

If the worker does not renew the lease:

10:45:00 passes

The engine can assume the worker is dead.

Then:

Running
   │
Lease Expired
   │
   ▼
Retrying
   │
   ▼
Ready

Another worker can pick it up.

This is how distributed systems survive disappearing machines.

And machines disappear all the time.

They do not even say goodbye.

---

Retries: The Feature That Can Destroy Your System

Retries sound harmless.

Something fails.

Try again.

But retries are dangerous.

Imagine an external API goes down.

You have:

10,000 failed tasks

Every task retries immediately.

Now:

10,000 requests

hit the already broken API.

Excellent.

You have turned failure into a denial-of-service attack.

This is why we use exponential backoff.

Something like:

1 second
2 seconds
4 seconds
8 seconds
16 seconds

Mathematically:

delay = base * 2^attempt

In Rust:

use std::time::Duration;

fn retry_delay(attempt: u32) -> Duration {
    let seconds = 2_u64.pow(attempt);

    Duration::from_secs(seconds)
}

But production systems usually add jitter.

Why?

Because if every task retries at exactly the same time:

10,000 tasks
↓
retry at 10:00:00
↓
💥

Jitter randomizes retry times.

Conceptually:

delay = exponential_backoff + random_jitter

This is one of those tiny engineering details that separates:

"It works."

from:

"It works when everything is on fire."

---

Time Is One of the Hardest Parts of Workflow Engines

Developers often think about computation.

But workflow engines think about time.

Tasks might run:

• Immediately.
• In five minutes.
• Tomorrow.
• Every Monday.
• After another task.
• After a human approves something.
• Until a timeout occurs.

So your engine needs a scheduler.

A simple scheduler might ask the database:

SELECT *
FROM task_instances
WHERE status = 'pending'
AND scheduled_at <= NOW();

Then those tasks become eligible for execution.

Conceptually:

                ┌───────────────┐
                │    Scheduler  │
                └───────┬───────┘
                        │
                        ▼
                Find Due Tasks
                        │
                        ▼
                 Put in Queue
                        │
                        ▼
                     Workers

The scheduler does not need to be complicated initially.

A loop is enough.

loop {
    find_due_tasks().await?;

    tokio::time::sleep(
        std::time::Duration::from_secs(1)
    ).await;
}

People sometimes underestimate boring loops.

But a surprising number of distributed systems are basically:

loop {
    check_the_database();
}

With better marketing.

---

Concurrency Is Where Rust Starts Feeling Very Natural

Workflows often contain parallel tasks.

For example:

              ┌──► Send Email ──┐
Create User ──┤                  ├──► Complete
              └──► Create Team ─┘

After creating the user, both tasks can execute simultaneously.

Tokio gives us tools for concurrent execution.

Conceptually:

let email_task = send_email();
let team_task = create_team();

tokio::join!(
    email_task,
    team_task
);

But in a real distributed workflow engine, concurrency is usually managed through workers.

Instead of:

one Rust process doing everything

you get:

Worker A → Send Email

Worker B → Create Team

This is better for scalability.

And it also introduces a fascinating truth:

Parallelism is easy to draw and difficult to make correct.

Because now you must think about:

• Race conditions.
• Duplicate execution.
• Resource contention.
• Idempotency.
• Ordering.

Welcome to distributed systems.

The coffee is cold, but the bugs are fresh.

---

The Most Important Word: Idempotency

Suppose your worker executes:

Charge Customer $100

The payment succeeds.

But the worker crashes before recording success.

The workflow engine thinks:

Task failed.
Retry.

Now the customer gets charged again.

You are no longer building a workflow engine.

You are building a criminal enterprise.

This is why tasks must be idempotent.

An idempotent operation can be safely repeated.

For example:

Create user with ID abc123

If the user already exists:

Success.

Instead of:

Duplicate user.
Everything is broken.

For payments, you use idempotency keys.

payment_workflow_123_task_charge_customer

The payment provider remembers:

I have already processed this request.

Retries become safe.

This is one of the fundamental principles of workflow design:

Assume every task might execute more than once.

Because in distributed systems, exactly-once execution is often more of a dream than a guarantee.

What you usually build instead is:

At-least-once execution
+
Idempotent tasks
=
Safe enough reality

---

Designing the Workflow State Transition System

We should never allow random state changes.

This should not be possible:

Pending → Succeeded

without execution.

Or:

Failed → Running

without retry logic.

Instead, transitions should be controlled.

impl TaskStatus {
    pub fn can_transition_to(&self, next: &TaskStatus) -> bool {
        matches!(
            (self, next),

            (TaskStatus::Pending, TaskStatus::Ready)
                | (TaskStatus::Ready, TaskStatus::Running)
                | (TaskStatus::Running, TaskStatus::Succeeded)
                | (TaskStatus::Running, TaskStatus::Failed)
                | (TaskStatus::Failed, TaskStatus::Retrying)
                | (TaskStatus::Retrying, TaskStatus::Ready)
        )
    }
}

Now the system has rules.

And rules are important.

Because without rules, your workflow engine eventually becomes:

if status == "something" {
    do_the_thing();
}

Then six months later:

if status == "something_else" {
    maybe_do_another_thing();
}

Then eventually:

if status != "something_weird" && status != "something_else" {

And nobody knows what the system actually does anymore.

Explicit transitions preserve sanity.

---

The Database Race Condition

Imagine two workers.

Both ask:

Give me a ready task.

Both receive:

Task 42

Both execute it.

This is bad.

So task claiming must be atomic.

Conceptually:

UPDATE task_instances
SET status = 'running',
    worker_id = 'worker-1'
WHERE id = 42
AND status = 'ready';

Only one worker should successfully update the row.

The second worker sees:

0 rows affected

Which means:

Someone else got there first.

This pattern is incredibly important.

Never assume:

SELECT task

then

UPDATE task

is safe.

Between those operations, another worker can interfere.

Distributed systems are basically millions of tiny moments where two computers say:

I thought I had it.

Atomic operations prevent that.

---

A Basic Engine Architecture

At this point, our system starts looking like this:

                         ┌──────────────────┐
                         │ Workflow API     │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │ Workflow Engine  │
                         └────────┬─────────┘
                                  │
              ┌───────────────────┼───────────────────┐
              │                   │                   │
              ▼                   ▼                   ▼
       ┌────────────┐      ┌────────────┐      ┌────────────┐
       │ Scheduler  │      │ State Mgr  │      │ Retry Mgr  │
       └─────┬──────┘      └────────────┘      └────────────┘
             │
             ▼
       ┌────────────┐
       │ Task Queue │
       └─────┬──────┘
             │
      ┌──────┼──────┐
      ▼      ▼      ▼
   Worker  Worker  Worker
      │      │      │
      └──────┼──────┘
             ▼
       ┌────────────┐
       │ Database   │
       └────────────┘

Each component has a different responsibility.

This is important.

The scheduler should not be sending emails.

The worker should not be deciding the global workflow state.

The API should not directly manipulate internal task state.

Separation of responsibility is not about making your architecture diagram look impressive.

It is about preventing one component from becoming God.

---

Human Tasks Are Where Workflows Become Really Interesting

Not every task is:

Run code.

Sometimes the workflow says:

Wait for manager approval.

That might take:

• Five minutes.
• Five hours.
• Five days.

You cannot keep a Rust future running for five days.

That would be ridiculous.

Instead, the workflow persists its state.

Task Status: WaitingForApproval

Then the process can completely restart.

The workflow still exists.

Later, an API request arrives:

POST /workflow/123/approve

The engine updates:

WaitingForApproval
        │
        ▼
Approved
        │
        ▼
Continue Workflow

This is where workflow engines become incredibly powerful.

They coordinate not only software.

They coordinate time and humans.

---

Compensation: Undoing the Past

Now we reach one of the most fascinating concepts in workflow systems.

Suppose you have:

Reserve Flight
        │
        ▼
Reserve Hotel
        │
        ▼
Charge Customer

The hotel reservation fails.

What do you do?

You cannot simply roll back the entire internet.

The flight reservation may have already happened.

So you need a compensation action.

Reserve Flight
        │
        ▼
Reserve Hotel ❌
        │
        ▼
Cancel Flight Reservation

This is called a Saga pattern.

Instead of database transactions:

BEGIN
COMMIT
ROLLBACK

you have distributed compensations:

Do A

Do B

Do C

If C fails:
    Undo B
    Undo A

In a workflow definition:

pub struct TaskDefinition {
    pub id: String,
    pub compensation_task: Option<String>,
}

This becomes incredibly useful for:

• Payments.
• Bookings.
• Provisioning infrastructure.
• Inventory systems.
• Distributed transactions.

The important thing to understand is:

You cannot always undo the world.

Sometimes compensation is not a perfect rollback.

If you sent an email:

"Welcome to our platform!"

You cannot unsend it from someone's brain.

The best compensation might be:

Send another email explaining the mistake.

Distributed systems are wonderfully human like that.

Sometimes you cannot erase mistakes.

You can only make another action.

---

Observability: How Do You Know Your Engine Is Alive?

Imagine your workflow engine processes:

1 million workflows

Someone asks:

Why didn't my workflow finish?

You need answers.

A good workflow engine should expose:

Workflow ID

Current Status

Current Tasks

Task Attempts

Errors

Execution Duration

Worker ID

For example:

Workflow: signup-9832

✓ Create User
✓ Create Workspace
✓ Send Verification Email

✗ Provision Analytics

Retrying in: 32 seconds
Attempt: 3/5

This is why observability is not optional.

You need:

• Logs.
• Metrics.
• Traces.

Metrics might include:

workflows_started_total

workflows_completed_total

workflows_failed_total

tasks_running

task_retry_total

average_task_duration

Logs might tell you:

Task 72 started by worker-9

Tracing might show you the entire journey:

API Request
    ↓
Workflow Created
    ↓
Task Scheduled
    ↓
Worker Claimed Task
    ↓
External API Called
    ↓
Task Completed

When things break at scale, observability becomes your flashlight.

Because distributed systems are usually very dark places.

---

Where Rust Really Helps

Now let's talk about the language.

Why Rust?

The first reason is correctness.

Workflow engines are state-heavy systems.

Rust makes illegal states harder to represent.

For example:

enum WorkflowStatus {
    Running,
    Completed,
    Failed,
}

You cannot accidentally create:

"completing_but_also_failed_and_maybe_running"

The type system forces structure.

Then there is concurrency.

Rust's ownership model prevents many data races before your code runs.

When multiple workers share state, this matters enormously.

You start using structures like:

Arc<Mutex<T>>

or asynchronous equivalents.

Rust forces you to explicitly answer:

Who owns this?

Who can mutate this?

Who shares this?

Those questions are annoying when you are writing code quickly.

They are extremely useful when you are designing systems that process money.

Rust also gives you performance.

A workflow engine may execute:

• Thousands of tasks.
• Millions of events.
• Hundreds of concurrent workers.

You want efficient resource usage.

Rust gives you:

Native performance
+
Async execution
+
Low memory overhead

But the biggest benefit might actually be psychological.

Rust makes you think about failure paths.

And workflow engines are fundamentally about failure paths.

---

A Minimal Rust Workflow Engine

Let's bring some ideas together.

First, our statuses:

#[derive(Debug, Clone, PartialEq)]
pub enum TaskStatus {
    Pending,
    Running,
    Succeeded,
    Failed,
}

A task:

#[derive(Debug)]
pub struct Task {
    pub id: String,
    pub dependencies: Vec<String>,
    pub status: TaskStatus,
}

A workflow:

use std::collections::HashMap;

pub struct Workflow {
    pub tasks: HashMap<String, Task>,
}

Finding ready tasks:

impl Workflow {
    pub fn ready_tasks(&self) -> Vec<String> {
        self.tasks
            .iter()
            .filter(|(_, task)| {
                task.status == TaskStatus::Pending
                    && task.dependencies.iter().all(|dep| {
                        self.tasks
                            .get(dep)
                            .map(|task| {
                                task.status == TaskStatus::Succeeded
                            })
                            .unwrap_or(false)
                    })
            })
            .map(|(id, _)| id.clone())
            .collect()
    }
}

Executing:

impl Workflow {
    pub fn complete_task(&mut self, task_id: &str) {
        if let Some(task) = self.tasks.get_mut(task_id) {
            task.status = TaskStatus::Succeeded;
        }
    }
}

Then the engine loop becomes conceptually:

loop {
    let ready = workflow.ready_tasks();

    if ready.is_empty() {
        break;
    }

    for task_id in ready {
        println!("Executing task: {}", task_id);

        workflow.complete_task(&task_id);
    }
}

Obviously, this is not production-ready.

But the fundamental idea is already there.

The engine repeatedly:

1. Looks at state.
2. Determines what is allowed to happen.
3. Executes work.
4. Changes state.
5. Repeats.

That is a workflow engine.

Everything else is about surviving reality.

---

Building It in Layers

If I were building a real workflow engine in Rust from scratch, I would not start with Kubernetes.

I would build it in layers.

Phase One: In-Memory Engine

Build:

• Workflow definitions.
• Tasks.
• Dependencies.
• DAG execution.
• State transitions.

Learn the model first.

Phase Two: Persistence

Add:

• PostgreSQL or SQLite.
• Workflow instances.
• Task instances.
• Recovery after restart.

Phase Three: Workers

Separate:

Engine

from:

Task execution

Add worker registration and task claiming.

Phase Four: Retries

Add:

• Maximum attempts.
• Exponential backoff.
• Jitter.
• Dead-letter states.

Phase Five: Scheduling

Support:

• Delayed tasks.
• Scheduled workflows.
• Cron-like execution.

Phase Six: Distributed Reliability

Add:

• Leases.
• Heartbeats.
• Worker recovery.
• Atomic task claiming.

Phase Seven: Advanced Features

Add:

• Compensation.
• Human approval.
• Child workflows.
• Workflow versioning.
• Event-driven triggers.

This approach is much better than attempting:

Temporal, but written by me over the weekend.

That sentence has probably destroyed many promising side projects.

---

The Deep Lesson Behind Workflow Engines

Building a workflow engine teaches you something bigger than workflow engines.

It teaches you how software behaves when time exists.

CRUD applications often pretend everything happens instantly.

Request comes in.

Database changes.

Response goes out.

But real systems often look like:

Something happened.

Now wait.

Something else might happen.

If it does, continue.

If it does not, timeout.

If something fails, retry.

If retry fails, compensate.

If a human intervenes, change direction.

If the server crashes, remember everything.

That is much closer to reality.

Workflow engines are software systems that model process over time.

And once you understand that, you start seeing workflows everywhere.

A CI pipeline is a workflow.

A payment is a workflow.

A ride-sharing trip is a workflow.

An e-commerce order is a workflow.

A user onboarding sequence is a workflow.

A bank transfer is a workflow.

Even your own life is suspiciously close to a workflow engine.

Wake up
   │
   ▼
Check phone
   │
   ▼
Regret checking phone
   │
   ▼
Drink coffee
   │
   ▼
Try to be productive
   │
   ▼
Unexpected error
   │
   ▼
Retry tomorrow

With exponential backoff, hopefully.

---

Final Thoughts

A workflow engine looks complicated because it sits at the intersection of several difficult areas:

• Graph theory.
• State machines.
• Distributed systems.
• Databases.
• Concurrency.
• Scheduling.
• Fault tolerance.

But the core idea remains beautifully simple.

Ask:

What happened?

Then:

What is allowed to happen next?

Then:

Make it happen.

Then:

Remember what happened.

Then repeat.

Rust is an excellent language for this kind of system because it encourages explicitness.

Explicit states.

Explicit ownership.

Explicit concurrency.

Explicit error handling.

And workflow engines desperately need explicitness.

Because ambiguity is expensive when your system is coordinating thousands of tasks across multiple machines.

The real challenge is not writing a function that executes a task.

The challenge is answering:

What happens when the task executes twice?

What happens when the worker dies?

What happens when the database restarts?

What happens when the network disappears?

What happens when the API succeeds but your worker crashes before recording success?

Those questions are the real workflow engine.

The happy path is easy.

The unhappy paths are where architecture is born.

And that might be the most interesting thing about building one in Rust.

You start with:

task.execute().await?;

And somewhere along the way, you accidentally find yourself thinking about distributed leases, idempotency, compensation transactions, graph scheduling, and whether time itself should be persisted in PostgreSQL.

That is software engineering.

You begin by trying to automate a few steps.

Then you discover you are negotiating with reality.

And reality, unfortunately, does not implement Clone.