DEV Community: Marc Duiker

The Three Failures Every Fault Tolerant AI Agent Must Handle

Marc Duiker — Wed, 14 Oct 2026 13:32:46 +0000

"Make the agent fault tolerant" sounds like one task. It is three. Your agent can fail loudly, with an error you can see and retry. It can fail silently, when a side effect fires and the process dies before anything records it. And it can fail by waiting, when an approval takes two days and the process holding that wait gets restarted on day one. Each class needs a different mechanism, and most agent frameworks only ship the first. That is why demos look fault tolerant and production agents send the same email twice or lose an approval.

Why fault tolerance is three assignments

When you move an agent from prototype to production, "make it fault tolerant" lands in the backlog as one ticket. You handle the failures you've seen in testing, and those are almost always the loud ones: a tool returns a 500, the model API times out, the LLM sends back malformed JSON.

The other failures rarely show up on your laptop. They show up during a deploy, when the orchestrator kills a pod for using too much memory, when the cloud provider reclaims a spot instance, or when a run sits waiting for a human for three days. Fault tolerance for an agent covers three distinct classes:

Visible failures, where something errors and you know about it.
Invisible failures, where work happens but the record of it is lost.
Long waits, where the agent has to outlive the process running it.

Visible failures and the retry toolbox

A visible failure announces itself. The tool call returns an error, the API times out with a clear exception, the model refuses the request or returns output your parser rejects.

The toolbox for this class is well known:

Retry with exponential backoff, so a rate-limited or briefly unavailable service gets time to recover.
Cap the number of attempts, so a permanent error doesn't turn into an infinite loop that burns tokens.
Dead-letter the requests that keep failing, so a person or a separate process can look at them later.

Almost every agent framework and HTTP client ships some version of this, so it's the part of fault tolerance most teams already have.

Retries have one built-in assumption, though: the failed attempt had no effect. A retry repeats a step; it doesn't know whether the previous attempt already sent the email or charged the card before the error came back. The next class is where it breaks.

Invisible failures and why checkpointing falls short

An invisible failure happens in the gap between doing something and recording that you did it. Your agent calls the payment API and the charge goes through. Before the agent writes "payment done" anywhere, the process disappears. A rolling deploy replaced the pod, the kernel's out-of-memory killer picked your container, or the spot instance was reclaimed. No exception is raised and no retry policy fires, because the code that would catch the error is gone too.

Many agent frameworks answer this with checkpointing. It saves a snapshot of the agent's state after each step, and a restarted agent loads the latest one. If the process dies during a step or right after it, that snapshot is from before the side effect. The restored agent has no idea the charge happened, and it charges again.

Durable execution closes most of that gap by making the record part of execution itself. The engine writes "scheduled" to a journal before a step runs, and "completed" with the result once it returns. On restart it replays the journal, skips every step marked completed, uses the recorded results instead of calling the tools again, and resumes at the first unfinished step. This is the class of failure that separates a demo agent from a production one.

There's still a small window left: the side effect can succeed just before the process dies and before "completed" is written. In that case the engine reruns the step, which is why durable execution engines guarantee that a step runs at least once rather than exactly once. The journal gets you back to the right step. Making the side effect happen once is the job of the step itself: pass an idempotency key to the downstream service so it recognizes the repeat and returns the original result. Dapr Workflow gives every scheduled task a task execution ID that stays the same across retries, so you can use it as that key.

Long waits and durable timers

Agents spend a lot of their lives waiting. A refund over a certain amount needs a manager's approval, which might come in two hours or two days. A procurement agent can't confirm delivery until the container ships on Tuesday.

Processes don't live that long. Over a two-day approval window your agent will probably go through several deploys, a node upgrade or two, and maybe an autoscaler scaling it down overnight. A wait held in process memory, like a sleep call, an in-memory promise or a loop polling an inbox, disappears with the process. Any wait held in memory is a failure scheduled in advance; you just don't know the exact time yet.

Durable execution handles this by moving the wait into the journal as well. A durable timer records "wake me at this time," and an external event records "resume when this approval arrives." While the agent waits, nothing needs to run: the process can stop, restart or move to another node. When the timer fires or the event comes in, the engine loads the journal and continues from the waiting step. Pair the approval event with a timer so that if nobody responds within three days the agent escalates or cancels instead of waiting forever.

The crash test that checks all three

You can check your agent against both of the harder classes with two tests. Both use a force-kill, meaning the process ends right away with no graceful shutdown and no cleanup handlers. For example:

In a container, use docker kill, or delete the pod with a zero grace period in Kubernetes.
On Windows, end the process from Task Manager or with taskkill /F.
On macOS, use Force Quit in Activity Monitor or kill -9.
On Linux, use kill -9 with the process ID.

For the first test, start a run that makes a side-effecting call, such as sending an email or creating a record in another system. Force-kill the process right after that call succeeds, restart it, and let the run finish. Then count the effects in the downstream system. One is the only right answer. If you see two, the step needs an idempotency key.

For the second test, start a run that waits for an approval or a timer. Force-kill the process during the wait, restart it, then send the approval. The agent should pick up the event and continue.

Most teams skip these tests because they tend to fail the first time. That is exactly why you should run them before production does.

Trade-offs and considerations

Durable execution isn't free. Every step writes to the journal before and after it runs, which adds a little latency and storage per step. For steps with side effects or long waits, that cost is almost always worth paying; for a quick in-memory computation it usually isn't.

The orchestration logic also has to be deterministic, because the engine replays it to rebuild state. Keep LLM calls, tool calls and anything else non-deterministic inside activities, and keep the orchestration code limited to deciding what runs next.

Not every agent needs all three layers either. A stateless chat helper that only reads data is fine with the retry toolbox alone.

Summary

Fault tolerance for AI agents is three separate problems. Retries with backoff and dead-lettering handle visible failures. Invisible failures need more than checkpointing: durable execution writes each step to a journal before and after it runs, so recovery resumes at the right step, and an idempotency key makes sure the side effect lands once. Long waits belong in the journal too, as durable timers and external events, so a restart can't erase them. The two force-kill tests tell you which of these your agent already handles. To see these ideas in a small working example, read The Tiniest Durable Agent. To learn durable timers, external events, and recovery hands-on, take the free Dapr Workflow course on Dapr University.

Understanding At-Least-Once Execution and Why It Matters for Workflow & Agent Developers

Marc Duiker — Thu, 08 Oct 2026 12:51:24 +0000

Introduction

Durable execution guarantees that your workflow finishes. It does not guarantee that each step runs only once. Picture a workflow that charges a credit card for 49 EUR. The payment provider accepts the charge, and then the pod is evicted before the workflow records the result. On restart, the workflow sees an unfinished step and charges the card again. Your customer now pays 98 EUR.

If you are new to durable execution, this can feel like a bug in the engine. It is not. Durable execution engines, including Dapr Workflow, run activities at least once, and making a side effect happen exactly once is the job of your activity code. In this post, you'll see why the engine behaves this way, how to spot the steps that need protection, and how to use the task execution ID in a C# activity as an idempotency key.

Durable Execution Guarantees Completion, Not Single Execution

A Dapr Workflow app has two kinds of code. The workflow code decides what happens next: call this activity, wait for that event, branch on a result. The activity code does the actual work that touches the outside world, such as calling a payment provider, writing to a database, or sending an email. The workflow engine persists every step, so when your app crashes it can pick up where it left off instead of starting over. If you want a primer on this model, watch this session on durable execution with Dapr.

There is a catch in "pick up where it left off". The engine can only store that a step happened after the step has happened. Between the moment your activity charges the card and the moment the engine writes down the result, there is always a window. If the process dies inside that window, the engine has no record of the charge, so the only safe choice is to run the activity again.

That is why durable execution gives you completion, not exactly-once execution. Exactly-once side effects come from making each activity idempotent: running it twice with the same input has the same effect as running it once. This is not specific to Dapr. Temporal and Azure Durable Functions make the same trade-off and give the same advice.

How Dapr Workflow Records Progress

Here is the order workflow from the intro, written with the Dapr .NET SDK:

public class OrderWorkflow : Workflow<Order, OrderResult>
{
    public override async Task<OrderResult> RunAsync(WorkflowContext context, Order order)
    {
        var payment = await context.CallActivityAsync<PaymentResult>(
            nameof(ChargeCardActivity),
            new ChargeRequest(order.CustomerId, order.Amount));

        await context.CallActivityAsync(nameof(ShipOrderActivity), order);

        return new OrderResult(payment.PaymentId);
    }
}

The workflow code does not call ChargeCardActivity directly. The orchestration lifecycle describes what happens instead. The Dapr sidecar sends your app an OrchestratorWorkItem over a long-lived GetWorkItems gRPC stream. That work item contains the full history of the workflow instance. The SDK replays your RunAsync method from the top against that history. When it reaches CallActivityAsync, it looks for a recorded result. If there is one, it returns the recorded result without running the activity. If there isn't, it emits a ScheduleTask action and suspends the workflow.

The engine then appends a TaskScheduled event to the history and dispatches the activity. When the activity reports back, the engine records a TaskCompleted (or TaskFailed) event, and the next replay picks up the result.

According to the state and history docs, this history is the source of truth. It is stored in your configured state store as a list of serialized events, with keys like wf-history-<instance_id>-<index>. For the order workflow, the history at the moment of the pod eviction looks like this:

ExecutionStarted   OrderWorkflow
TaskScheduled      ChargeCardActivity
                   <- pod evicted here, card already charged

There is no TaskCompleted for ChargeCardActivity. As far as the engine knows, the charge never finished.

Why Activities Run at Least Once

The activity lifecycle shows where the window sits. When the engine dispatches an activity, the sidecar sends an ActivityWorkItem over the same GetWorkItems stream. The work item carries the activity name, its input, the workflow instance_id, a task_id, a task_execution_id, and a completion_token. The SDK runs your activity code and then calls CompleteActivityTask with the result (or failure_details) and the completion_token, as defined in the execution API.

Your side effect happens while the activity runs. The engine learns about it only when CompleteActivityTask arrives. Anything that breaks the chain between those two points, such as a pod eviction, an out-of-memory kill, a node failure, or a dropped network connection, leaves a TaskScheduled event without a matching TaskCompleted. The activity lifecycle doc names this case directly: activities "might be executed more than once (e.g., if the worker crashes after execution but before reporting completion)".

Retry policies are a second source of repeats. If an activity fails and the workflow has a retry policy, the engine schedules the activity again with the same name and input. A failure from your activity's point of view is not always a failure on the other side. If the HTTP call to the payment provider times out, your activity throws, yet the provider may have processed the charge anyway.

The third case is a slow worker. If a worker takes too long and the task is handed to another worker, the original one can still finish. The sidecar uses the completion_token to ignore the late response, but by then the side effect has already happened.

In all three cases, the engine protects its own bookkeeping, not your side effects. That is why at-least-once is the norm for durable execution engines.

How to Spot the Steps That Need Idempotency

Not every activity needs extra work. For each activity in your workflow, ask one question: if this runs twice with the same input, is the outcome different from running it once?

Activities that usually need protection:

Charging, refunding, or transferring money
Sending emails, text messages, or push notifications
Inserting records with generated IDs, where a second insert creates a duplicate row
Publishing messages to a broker or event stream
Incrementing counters or adjusting stock levels
Calling an LLM tool or external API that acts on another system, such as creating a ticket

Activities that are usually safe already:

Pure computation on the input
Reads, such as GET requests and database queries
Upserts keyed on a natural ID, such as the order ID
Setting a value to a fixed state, such as "set order status to Shipped"

Don't forget compensation activities. If your workflow uses the saga pattern, the refund that undoes a charge can run twice as well, and refunding a customer twice is just as wrong as charging them twice.

Keep this separate from workflow determinism. Determinism is about your workflow code producing the same decisions on every replay, so no DateTime.UtcNow, random numbers, or direct I/O in RunAsync. Idempotency is about your activity code having the same effect when it runs more than once. You need both, but they solve different problems.

Using the Task Execution ID as an Idempotency Key

The standard fix is an idempotency key. You send a key with the request, and the receiving service remembers which keys it has already processed. When the same key arrives a second time, the service returns the original result instead of doing the work again. Most payment providers support this through an Idempotency-Key HTTP header.

The key has to be the same every time the activity runs for the same step, and different for every other step. Since Dapr .NET SDK v1.17, WorkflowActivityContext exposes a TaskExecutionId property that fits this. The ID is created when the workflow schedules the activity and is stored with the TaskScheduled event in the history. When the engine dispatches that scheduled task again after a crash, a pod eviction, or a worker timeout, your activity receives the same ID. Treat it as an opaque string and don't parse it.

Here is the payment activity with the task execution ID passed as the idempotency key:

public class ChargeCardActivity(
    IHttpClientFactory httpClientFactory,
    ILogger<ChargeCardActivity> logger)
    : WorkflowActivity<ChargeRequest, PaymentResult>
{
    public override async Task<PaymentResult> RunAsync(
        WorkflowActivityContext context, ChargeRequest request)
    {
        var idempotencyKey = context.TaskExecutionId;
        logger.LogInformation(
            "Charging {Amount} EUR for {CustomerId} with idempotency key {Key}",
            request.Amount, request.CustomerId, idempotencyKey);

        var httpClient = httpClientFactory.CreateClient("payments");
        using var message = new HttpRequestMessage(HttpMethod.Post, "/v1/payments")
        {
            Content = JsonContent.Create(request)
        };
        message.Headers.Add("Idempotency-Key", idempotencyKey);

        using var response = await httpClient.SendAsync(message);
        response.EnsureSuccessStatusCode();

        return (await response.Content.ReadFromJsonAsync<PaymentResult>())!;
    }
}

Replay the eviction scenario with this version:

The engine dispatches ChargeCardActivity with task execution ID a1b2....
The provider charges 49 EUR and stores the payment under key a1b2....
The pod is evicted before CompleteActivityTask reaches the sidecar.
The workflow restarts, and the engine dispatches the same scheduled task again with task execution ID a1b2....
The provider recognizes the key and returns the original payment without charging the card.
The engine records TaskCompleted, and the workflow moves on to shipping.

Your customer pays 49 EUR. The same approach works for any service that accepts a deduplication key, such as email providers, message brokers that support message deduplication IDs, and your own APIs.

When the Idempotency Key Expires

An idempotency key only helps while the service still remembers it. Many providers clear keys after a fixed period. Stripe, for example, can remove keys once they are at least 24 hours old. Check the documentation of every service you call for its retention period.

Durable workflows can run for days or weeks, and a second dispatch doesn't always follow the first one quickly. An outage, a long retry backoff, or a suspended workflow that resumes later can all push the second attempt past the retention period. At that point the provider treats the request as new. Some services don't support idempotency keys at all.

In those cases you need a record you own: a dedupe table in your own database, keyed on the task execution ID. Before doing the work, the activity checks the table. After doing the work, it stores the result.

public class SendInvoiceActivity(InvoiceDbContext db, IEmailSender emailSender)
    : WorkflowActivity<InvoiceRequest, InvoiceResult>
{
    public override async Task<InvoiceResult> RunAsync(
        WorkflowActivityContext context, InvoiceRequest request)
    {
        var key = context.TaskExecutionId;

        var processed = await db.ProcessedTasks.FindAsync(key);
        if (processed is not null)
        {
            return new InvoiceResult(processed.MessageId);
        }

        var messageId = await emailSender.SendAsync(
            request.To, request.Subject, request.Body);

        db.ProcessedTasks.Add(new ProcessedTask(key, messageId, DateTimeOffset.UtcNow));
        await db.SaveChangesAsync();

        return new InvoiceResult(messageId);
    }
}

Make the task execution ID the primary key of ProcessedTasks. A unique constraint stops two concurrent attempts, such as a slow worker and its replacement, from both recording a result. Keep rows for longer than your longest-running workflow, and clean up older rows with a scheduled job.

This version still has a gap. If the process dies between SendAsync and SaveChangesAsync, the email goes out twice. To close it, insert a Pending row before calling the external service and mark it Completed afterwards. When an attempt finds a Pending row, it knows a previous attempt may have done the work, and it can check with the provider before trying again. Where the provider supports keys, send the task execution ID to it as well: the provider's key covers the short window, and your table covers everything after the key expires. For more on choosing between keys and this kind of reconciliation, see the Diagrid FAQ on idempotency keys and reconciliation logic.

Trade-offs and Considerations

Check how the task execution ID behaves with retry policies before you rely on it. The .NET SDK documentation describes it as stable across retries, while the activity lifecycle protocol doc says it changes every time the engine retries the activity. For the payment timeout case, that difference decides whether the provider sees one charge or two. Test it on your Dapr and SDK versions. If you need one key across all retry attempts, build it in the workflow from the instance ID and a step name, for example $"{context.InstanceId}:charge-card", and pass it as activity input. Use a different step name for every call, including calls inside a loop.

Scope each key to one side effect. If an activity both charges a card and sends a receipt, it needs two keys, such as {TaskExecutionId}:charge and {TaskExecutionId}:receipt. Splitting the work into two activities is usually cleaner.

Idempotency has a cost. A dedupe table adds a database round trip to every call and a cleanup job to your operations work. Spend that effort on the activities from the checklist above, not on every activity.

Don't move side effects into the workflow code to avoid retries. Workflow code replays on every step, so a side effect there runs far more often, and it breaks determinism.

Finally, log the key in every activity. Idempotency stops the duplicate from reaching your customer, and the logs tell you how often it happened.

Summary

Durable execution keeps your workflow moving through crashes and pod evictions, but it does so by running activities at least once. The Dapr workflow protocol makes the reason visible: the engine only knows an activity finished once a TaskCompleted event lands in the history, so any failure in between leads to a second dispatch. The fix belongs in your activities. Find the steps with side effects, pass the task execution ID as an idempotency key to the services that support it, and back it with a dedupe table when the downstream service forgets keys too soon or doesn't support them at all. Your 49 EUR charge then stays 49 EUR.

To build a solid foundation in the Dapr Workflow programming model, including activity retries and error handling, take the free Dapr Workflow track on Dapr University. If you prefer video, this session on durable execution with Dapr covers how replay and history work.

What's New in the Dapr Dev Dashboard for Local Workflow Debugging

Marc Duiker — Mon, 05 Oct 2026 12:39:18 +0000

Introduction

The Diagrid Dapr Dev Dashboard has shipped nine releases since the first announcement, and if you build Dapr Workflow apps on your machine, several of them change how you debug. The Workflows page now loads thousands of instances in milliseconds instead of hanging. A new State page lets you browse, add, and delete the records in your state store. A context-aware Dapr CLI drawer gives you the exact dapr workflow command for the instance you are looking at, with the IDs already filled in. On top of that, you can publish test messages to a topic, check component secret references, and keep the app list and logs tidy as your dev session grows. This post covers the highlights from v0.0.14 to v0.0.22 and shows how each one helps you find out why a workflow behaves the way it does. You can find an overview of the dashboard on the Dapr Dev Dashboard page.

Why it matters

A Dapr Workflow is long-running and event-sourced. Its state lives in your state store as a history of events, so when an instance stalls or returns the wrong output on your machine, the answer is somewhere in that history. Finding it usually means querying the state store by hand, looking up the right CLI flags, or adding log statements and running the workflow again.

The dashboard already took care of the basics. It finds the apps you start with dapr run, .NET Aspire, Docker Compose, or Testcontainers, tails their sidecar and app logs, shows workflow event history, and generates component and resiliency YAML from a form. The recent releases go further into the questions you ask while debugging: what did the workflow write, which instances are stuck, and what do you run to fix them.

What's new

A Workflows page that stays fast on large state stores

Run a load test or leave a fan-out workflow running overnight, and your local state store can easily hold thousands of workflow instances. Earlier versions of the Workflows page slowed to a crawl at that point, because every instance load scanned the store for that instance's keys: a full keyspace SCAN on Redis, a LIKE table scan on the SQL backends.

The dashboard now loads instances the way Dapr loads its own state. It reads the instance's metadata record, builds the history keys from it, and fetches them in one bulk read. The status counts on the tabs are cached per workflow and only reload when that workflow changes, while the workflow detail page always reads fresh data. These are the numbers for 2,000 instances, before and after:

Backend	List page	Status counts, first load	Status counts, refresh
SQLite	341 ms → 10 ms	13.7 s → 0.20 s	15.9 s → 0.03 s
Redis	702 ms → 39 ms	28.6 s → 1.28 s	29.4 s → 0.10 s
PostgreSQL	386 ms → 18 ms	14.9 s → 0.17 s	14.7 s → 0.02 s
MongoDB	200 ms → 7 ms	8.1 s → 0.28 s	8.1 s → 0.08 s

The page also behaves better while it loads. When you change a status tab, search, or page, the current rows stay on screen with Updating… in the pager until the new results arrive. The tabs show … until their counts are in, instead of a misleading 0.

Browse and edit state with the new State page

Workflow activities often read and write application state, and when a workflow produces the wrong result, you want to see what is actually in the store. The new State page lists the records in any connected state store, with the key, app prefix, value preview, size, version (etag), and TTL for each one. You can switch between stores from the page header and expand a row to see the full value, with a copy button. The dashboard opens Redis, PostgreSQL, SQLite, and MongoDB stores directly.

To find a record, filter by app prefix or search by key. If Dapr stored a value base64-encoded, flip Decode base64 to read it. Workflow history and actor state share the keyspace with your app's records, so they are hidden by default. Turn on Show internal keys when you want to look at the raw workflow keys.

You can also change state. + New record writes a record under <app-id>||<key>, the same key shape Dapr uses, which is handy for seeding data a workflow reads. An existing key is refused unless you tick Overwrite. You can delete records one at a time or select several, with a confirmation step before anything is removed.

Two limits are worth knowing. There is no "last modified" column, because Dapr state components do not expose a modification timestamp. And a store the dashboard cannot open directly, such as an in-memory store inside a Testcontainers app, cannot be browsed, since Dapr's state API has no way to list keys.

Get the right Dapr CLI command for every page

The dashboard is great for looking around, but sometimes you want the terminal, for example to put a command in a script or rerun it after a restart. A collapsible Dapr CLI drawer on the right edge of the dashboard now shows the Dapr CLI commands that match the page you are on.

On the workflow views, that means dapr workflow list, history, suspend, resume, terminate, and purge, with the app ID and workflow instance ID from the current view already filled in. Every command has its own copy button, so you do not have to look up flags or paste IDs by hand.

Other pages get their own set: dapr list on the applications overview, dapr stop --app-id <id> on an app's detail page, dapr scheduler commands on actors, and dapr publish on subscriptions. The commands target self-hosted mode, so you will not find the Kubernetes -k variants here.

Publish test messages from the Subscriptions page

If a message on a topic starts your workflow, testing it used to mean writing a small client or reaching for curl. Each row on the Subscriptions page now has a Publish button that sends a real message to the topic through the app's own sidecar. You edit the JSON or text payload, choose a content type, and optionally set ttlInSeconds or rawPayload under Advanced.

When the publish succeeds, you get a direct link to the app's logs, so you can watch the message get handled. If daprd rejects it, for example because the pub/sub component is unknown, the error shows up inline in the dialog.

The page also tells you more about each subscription. A Type column shows whether it is declarative or programmatic, and subscriptions with several routing rules expand to show each rule's match expression and path.

See which component secrets resolve

Your workflow state store is a Dapr component, and if its password comes from a secret store, one typo in a secretKeyRef is enough to break it. Until now, a component with a missing secret looked exactly like a working one. The dashboard now resolves secretKeyRef values against the secretstores.local.file and secretstores.local.env stores the same way daprd does, including nested keys and custom separators. envRef is supported too.

A component's detail page has a new Secret references panel. For each field it shows the store, the key, and a status, and when resolution fails, the exact file path it tried. Values stay masked; Reveal shows one field at a time and masks it again after 30 seconds. The Components list marks any component with an unresolved reference with a warning sign. When a state store cannot connect because of a secret, the State page names that secret instead of showing a vague connection error.

Non-local secret stores, such as HashiCorp Vault or Azure Key Vault, are listed but never read.

Cleaner logs and app list during long dev sessions

On the Logs page, choosing what to tail is now a single Target dropdown, grouped into Applications and Control plane. When you view an app, a compact daprd | app toggle lets you show the sidecar logs, the app logs, or both side by side.

After a few rounds of starting and stopping apps, the Applications list fills up with stopped entries. Clear inactive removes all fully stopped apps in one click, and each stopped row has its own Remove action. Removing an app only hides it from the list; if it starts again, it reappears. Restart and Start now only appear for Docker Compose apps, where they work reliably. For dapr run apps you get Stop, with a hint to restart from your terminal.

How to get started

The dashboard is a single self-contained binary with no runtime dependencies, available for macOS and Linux (amd64 and arm64) and Windows (amd64). Install it with the one-liner for your OS.

macOS or Linux:

curl -sSL https://raw.githubusercontent.com/diagridio/dev-dashboard/main/scripts/install.sh | sh

Windows (PowerShell):

iwr -useb https://raw.githubusercontent.com/diagridio/dev-dashboard/main/scripts/install.ps1 | iex

Start the dashboard next to your running Dapr apps:

diagrid-dev-dashboard

It serves the UI on http://localhost:9090 and opens your browser. If no Dapr apps are running yet, the Applications page points you to the Dapr Quickstarts so you have something to try it on. The GitHub repo has the full README and the release notes for every version.

Summary

Most of the recent work on the Dapr Dev Dashboard targets the questions you ask while debugging a workflow on your machine: what state did it write, which instances are stuck, and which command gets them moving again. A faster Workflows page, the new State page, and the Dapr CLI drawer answer those questions from your browser. You can also publish a test message from the subscription it targets, and spot a missing secret before it shows up as a vague connection error.

Install the dashboard with the one-liner for your OS, go install, or the container image from the GitHub repo, and run diagrid-dev-dashboard next to your Dapr apps. For an overview of everything it can do, visit the Dapr Dev Dashboard page.

Once the dashboard is open, try the Konami code: type ↑ ↑ ↓ ↓ ← → ← → B A in your browser and see what appears. There is a fresh challenge every day, and you can share your best run with an execution ID so others can watch it back.

The Tiniest Durable Agent I Can Write

Marc Duiker — Mon, 28 Sep 2026 07:11:56 +0000

"Durable execution" sounds like something you adopt after a six-week platform evaluation. It isn't. Here's an agent that survives being killed, in roughly twenty lines of Python.

The trick is that durability isn't a feature you code – it's a property of where the agent's loop runs. With Dapr Agents, the loop runs as a durable workflow, so every LLM call and tool call is journaled automatically:

from dapr_agents import DurableAgent, tool

@tool
def search_flights(destination: str) -> list:
    """Search available flights."""
    return flights_api.search(destination)

@tool
def book_flight(flight_id: str) -> str:
    """Book a flight. Real money involved."""
    return flights_api.book(flight_id)

agent = DurableAgent(
    name="TravelBuddy",
    role="Travel planning assistant",
    instructions=["Find options first. Confirm before booking."],
    tools=[search_flights, book_flight],
)

agent.start()

That's the whole thing. Now the fun part – try to break it:

Ask it to plan a trip. Watch it call search_flights.
Kill the process while it's mid-task. Not gracefully, kill the entire process.
Start it again.

The agent picks up the same task where it left off.
The search results it already gathered come back from the workflow's execution history instead of being fetched (and paid for) again. And if the crash happened around book_flight, the journal knows whether that step confirmed – so recovery doesn't book a second seat. LLM calls are slow, costly, and non-deterministic; re-running everything after a crash burns money and can change the answer. Replaying from history does neither.

Under the hood this is a durable workflow – the same durable execution machinery you'd use for order processing or payment flows, wrapped around an agent loop. No extra database to run: state goes to whatever Dapr state store you configure (Redis locally, Postgres or others in production).

Honest caveats, as always: the agent's reasoning is only as repeatable as your model, tools with side effects still deserve idempotency keys for the truly paranoid path, and you should read up on workflow versioning before changing an agent that has long-running tasks in flight.

The longer version of this, with the full runnable sample, lives in The Tiniest Durable Agent. If you want the structured path, the Dapr Agents track at Dapr University is free, and the quickstarts get you a running example in a few minutes.

Your LangGraph Agent Survived the Demo. Will It Survive a Deploy?

Marc Duiker — Fri, 25 Sep 2026 05:54:28 +0000

The LangGraph checkpointer is good. I want to say that first, because what follows reads like a complaint and it isn't one. Thread-level persistence, time travel, human-in-the-loop pauses: those work, and they work well.

There's one failure mode it doesn't cover, and it's the one you hit in production. The process dies after a side effect but before the checkpoint that records it.

What's the difference between checkpointing and durable execution?

A checkpoint is a snapshot of agent state, written once a step has finished. Durable execution keeps a journal of every step as it runs, and the runtime, not your code, decides what completed.

Here is where that matters. A graph node calls an external API. It files a Jira ticket, or it charges a card. The call succeeds. Then the pod catches a SIGKILL during a rolling deploy, before the checkpoint write lands.

A checkpointer restores the last checkpoint it has, which is the one from before the node ran. So the agent runs the node again, and now there are two tickets and two charges.

Durable execution sees an activity in the journal that started and never confirmed, and it can tell you that's what it is. The steps that did confirm don't run twice. The workflow resumes at the one that didn't.

That's the part of AI workflow orchestration nobody demos. Drawing the boxes and arrows is easy. The runtime underneath has to answer "what already happened?" at any point, and that is a harder thing to build than it looks.

Keeping LangGraph, adding durability

You don't have to pick one. LangGraph stays the brain, and each side-effecting step moves inside a durable activity.

# before: the node does the side effect directly
def create_ticket_node(state):
    ticket = jira.create(state["summary"])   # unprotected side effect
    return {"ticket_id": ticket.id}

# after: the node schedules a durable activity
def create_ticket_node(state):
    ticket_id = yield ctx.call_activity(create_ticket, input=state["summary"])
    return {"ticket_id": ticket_id}

Same graph, same prompts. The difference is that every step now gets journaled, retried with backoff, and resumed after a crash rather than replayed from scratch. That applies to your LLM calls too, so a crash doesn't re-spend the tokens you already burned on steps one through six.

Caveat where it's due, because durable execution asks for things too. Your workflow code has to be deterministic, which rules out random branching outside activities. And you need to think about versioning before you change a workflow that has instances in flight. So there's a cost. I'd pay it the moment an agent starts touching anything with an invoice attached.

We've written up running LangGraph in production with durable execution at more length, and there's a longer argument for why checkpoints aren't durable execution if you want to know more of the details.

Run the Test You'd Rather Skip: Kill Your Agent Mid-Payment

Marc Duiker — Mon, 21 Sep 2026 08:37:29 +0000

Your agent calls a payment API. The process dies right after the charge succeeds and before anything gets written down. What happens when it restarts?

If the answer is "it retries from the top," you have a problem. "Retry the tool call" sounds harmless until the tool issues a refund, or a charge. Then you have billed a customer twice, and no amount of prompt engineering will explain that to them.

Most agent code handles the happy path and the visible error path, the one where the API returns a 500 and you try again. The path nobody tests is the invisible one, where the side effect succeeded and then the process disappeared before recording it. Deployments cause this. So do OOM kills, and spot instances do it on a schedule.

What durable execution does

Durable execution records every completed step in a durable log as the workflow runs. When the process comes back, the runtime replays the log: steps that already completed return their recorded results instantly, and execution resumes at the first step that never finished. Your charge happens once, no matter how many times the process dies.

With Dapr Workflow in Python:

import dapr.ext.workflow as wf

wfr = wf.WorkflowRuntime()

@wfr.activity(name="charge_card")
def charge_card(ctx: wf.WorkflowActivityContext, order):
    # the side effect lives in an activity
    return payments.charge(order["id"], order["amount"], idempotency_key=order["id"])

@wfr.activity(name="send_receipt")
def send_receipt(ctx: wf.WorkflowActivityContext, charge):
    return mailer.send(charge["customer"], charge["receipt_url"])

@wfr.workflow(name="order_flow")
def order_flow(ctx: wf.DaprWorkflowContext, order):
    charge = yield ctx.call_activity(charge_card, input=order)
    receipt = yield ctx.call_activity(send_receipt, input=charge)
    return receipt

if __name__ == "__main__":
    wfr.start()
    client = wf.DaprWorkflowClient()
    instance_id = client.schedule_new_workflow(
        order_flow, input={"id": "order-1", "amount": 4200}
    )
    client.wait_for_workflow_completion(instance_id)
    wfr.shutdown()

Now run the test you would rather skip:

Start the workflow.
Kill the process the moment charge_card completes. It has to be an unclean kill, so the app gets no chance to shut down gracefully: kill -9 <pid> on macOS or Linux, Stop-Process -Id <pid> -Force in PowerShell on Windows, or docker kill <container> if you are running it in a container.
Restart with only wfr.start() running, and watch the workflow resume at send_receipt, with the charge result restored from history instead of re-executed.

Durable execution needs idempotent tools

One caveat, and it is the reason step 2 is interesting. The replay only skips a step once that step's completion has been written down. If the process dies in the gap between the charge returning and the completion being recorded, the activity runs again on recovery. So if your payment provider cannot tell two identical requests apart, pass an idempotency key too. Idempotency means a request can be repeated safely without repeating its effect, and that is the half durable execution cannot do for you.

If you want to go deeper, read this post about The Tiniest Durable Agent, and there is a free, self-paced Dapr Workflow course at Dapr University that covers chaining, fan-out/fan-in, durable timers, and human-in-the-loop in depth.

For a developer the benefit is code you don't have to write. The workflow above reads top to bottom like ordinary Python, and the state machine, checkpoint table and "did I already charge this card?" column that would normally sit behind it are abstracted away, because the runtime manages that state instead. Recovery stops being something you design per tool call. When a run does go wrong, the history shows which step executed and what it returned, so you debug from a record rather than a guess.

So: fail the action at the worst possible moment, then check whether your system recovered and whether you can prove what it did. It takes about ten minutes, and you would rather learn the answer now than from the customer you charged twice.

10 LangGraph Alternatives for Building Agentic Workflows in 2026

Marc Duiker — Sat, 08 Aug 2026 11:00:00 +0000

LangGraph is a good answer when your agent naturally looks like a graph. It gets harder to defend when your team does not want that graph to become the shape of the whole application.

None of the alternatives below are LangGraph clones. Each one asks you to think about agents differently.

If you want a smaller abstraction

The OpenAI Agents SDK gives you agents, tools, handoffs, guardrails, sessions, and tracing, without asking you to draw an explicit graph for every application.
Pydantic AI is worth a look if you write typed Python services. You get structured outputs, validation, and dependency injection.
Strands Agents runs a model-driven loop with tools and keeps the developer experience short.

If you want multi-agent concepts first

CrewAI models role-based agents, tasks, and crews. When collaboration is the main thing you are describing, that can read better than wiring nodes together.
Microsoft Agent Framework deserves a place on the shortlist if your team is Microsoft-oriented and you are building agents and multi-agent workflows across .NET and Python.
Google ADK supports modular agents, tools, and composition, and it lines up with Google's models and cloud.

If you want distributed-systems primitives

Dapr Agents puts agent development on top of Dapr's state, messaging, service invocation, and workflow foundation.
Restate is not only an agent framework. It is a durable application runtime that can wrap your agent logic and your SDKs. Look at it when resilient execution matters more to you than taking on another high-level agent abstraction.

If you want long-horizon or provider-specific tooling

Deep Agents aims at longer and more complex work through planning, sub-agents, and context-management patterns.
The Claude Agent SDK fits teams building tool-rich agents around Claude's ecosystem and its coding-agent patterns.

Framework choice is not runtime choice

Some of these change how you author the agent. Others change how the agent survives failure. The two decisions are related, but they are not the same decision, and it is easy to shop for one while thinking you are solving the other.

If you like LangGraph's programming model and what you actually need is production durability, you may not have to replace LangGraph at all. Diagrid Catalyst, for example, ships a LangGraph runner as part of its bring-your-own-framework approach, and other durable runtimes integrate with agent SDKs too.

My short version:

stay on LangGraph when explicit state and graph control are the strengths you want
pick CrewAI when roles and collaboration match the problem
pick the OpenAI Agents SDK or Pydantic AI when you want fewer orchestration concepts
pick an ecosystem-native framework when organizational alignment matters
add a durable execution layer when the real problem is recovery rather than authoring

Pick the one whose mental model your team can still explain a few months after the demo has become a production service. That test tells you more than how new the framework is.

Inngest or Diagrid Catalyst: From a TypeScript AI Workflow to an Enterprise Agent Platform

Marc Duiker — Fri, 07 Aug 2026 09:00:00 +0000

A five-person team usually picks the infrastructure it never has to think about, and that is the right call. It stops being the right call on the day the first enterprise customer asks where the agent runs and who is allowed to authorize its tools.

At prototype stage, Inngest is easy to understand. You write functions in TypeScript, Python, or Go, split durable work into steps, and let the platform deal with state, retries, flow control, and observability. Its current durable-agent model uses step-level memoization, so completed LLM calls and tool invocations are not repeated during a replay. It can also wait for events and invoke sub-agents.

That suits a web team stretching background jobs into longer AI workflows. The agent stays next to the application code, and there is not much infrastructure to run.

When the decision changes

The architecture conversation usually shifts in stages.

First you ship the feature. A TypeScript workflow calls an LLM, enriches some data, and writes a result. Developer experience decides the winner.

Then you have to survive failure. Tool calls need to resume at the right step. Inngest and Catalyst both handle that, through different execution models.

Then a second and a third agent stack show up. A Python team picks LangGraph, a .NET team picks Microsoft Agent Framework, and another group uses Google ADK. Catalyst's bring-your-own-framework model starts to matter here, because the platform standard can sit below those framework choices instead of competing with them.

Then you go through enterprise review. Customers ask for workload identity, deny-by-default access, private deployment, MCP governance, and evidence of what each agent did. That is where Catalyst's wider governance model can justify running a larger platform.

What I would compare

Retries and dashboards are the least interesting part of this comparison. I would look at:

which languages and agent frameworks are supported
what happens to completed steps and side effects during recovery
who operates the workers, the state, and the execution plane
how agent-to-tool access is authenticated and authorized
which self-hosted, private, or air-gapped topologies exist
how several teams share one platform without sharing permissions

Inngest can stay the best choice for developer-first durable functions, background jobs, and AI workflows, as long as its language and deployment model fit the team. Catalyst fits better once the requirement has grown from "make this function durable" into "run and govern many kinds of agents across the enterprise."

Both mistakes are common, and they are the same mistake pointed in opposite directions: buying the stage-four platform while you are still at stage one, and refusing to reopen a stage-one decision after the organization around it has changed. Let the architecture follow the risk you are actually carrying.

Official sources

Restate or Diagrid Catalyst for Durable Tool Calls?

Marc Duiker — Thu, 06 Aug 2026 08:00:00 +0000

"Retry the tool call" sounds harmless until the tool issues a refund. Then durability, identity, and evidence stop being three concerns owned by three different teams and turn into one incident.

Take a support system built out of several agents. A triage agent classifies the ticket. An account agent retrieves customer state. A policy agent decides whether a refund is allowed. A human approves the recommendation. An action agent calls the refund tool. Then the process loses its connection before the response is recorded.

Retrying is the easy part. What matters is whether the platform and your own code can work out what already happened, without sending a second refund.

Restate: durable code and journaled steps

Restate is a runtime that sits in front of your services and records execution in a journal. Its AI documentation shows LLM calls and tool executions wrapped as durable steps. After a crash, completed results are replayed from the journal and execution continues from the first step that never finished. Restate supports services, virtual objects, and workflows, and runs through Restate Cloud or a server you host yourself.

That is a good model if you want resilient functions and services with built-in state, reliable communication, and durable execution written into ordinary code. It also integrates with agent SDKs rather than asking you to adopt one agent framework.

Catalyst: durable agents plus a shared trust layer

Catalyst also persists workflow activity results and resumes failed executions. Around that it adds App ID-based workload identity, access policies, mTLS-backed communication, MCP governance, and a single operational surface for agents and applications. Diagrid positions verifiable execution, meaning history, lineage, signing, and attestation, as an additional trust layer.

Once money is involved, a security review will want more than proof that the run recovered:

Which workload identity invoked the refund tool?
Which policy authorized the call?
Can you show that the execution history was not altered?
Can you revoke access without editing the agent prompt?
Does the same control apply across cloud, on-premises, and isolated deployments?

Where each one fits

Restate is the better fit when what you need is a lightweight durable runtime for services, stateful entities, workflows, and agents, and you want to express resilience directly in normal application code.

Catalyst is the better fit when durable execution has to arrive together with organization-wide agent identity, policy-controlled tool access, verifiable evidence, and governance across several frameworks and deployment environments.

Neither one takes your application off the hook once the outcome is unknown. A payment or refund integration still needs idempotency keys, reconciliation, and explicit compensation logic. Durable execution preserves progress. It cannot tell you whether an external system committed a request when the response was lost.

So run the test you would rather skip. Fail the action at the worst possible moment, then check whether it recovered, whether you can prove who did what, and whether the same refund can go out twice.

Official sources

Kestra or Diagrid Catalyst for AI Workflow Orchestration?

Marc Duiker — Wed, 05 Aug 2026 09:00:00 +0000

"One orchestrator for everything" sounds like a clean platform strategy. It also packs three different execution problems into one word. A data pipeline, an infrastructure rollout, and an AI agent all have steps in them. That does not mean they want the same authoring model, the same recovery semantics, or the same governance boundary.

That gap is what to look at when you compare Kestra and Diagrid Catalyst. Kestra is a declarative orchestration platform that covers data, infrastructure, business, and AI workflows. Catalyst is a durable execution and governance platform for applications, workflows, agents, and MCP servers, built on Dapr. They overlap, and they start from different places.

Start with the shape of the workload

For scheduled ETL, infrastructure automation, or a business flow that crosses several tools, a declarative control plane is an advantage rather than a tax. Kestra defines workflows in YAML, has a large plugin ecosystem, and manages flows through a UI and Git. Its documentation now also covers AI workflows and AI Agent tasks that can use memory and tools and pick actions at runtime.

The calculation changes when the agent already exists. If it is written in LangGraph, CrewAI, Microsoft Agent Framework, Google ADK, or another supported framework, then rewriting it as a platform-specific flow is a rewrite nobody asked for. Catalyst's pitch is that you keep the framework and wrap durable workflow execution, identity, policy, and observability around it.

"Does it support AI" is the wrong row in the table

Both products support AI use cases, so the answer is yes twice so this question does not bring you closer to the perfect product fit. These questions separate them better:

Is the agent authored inside the orchestration platform, or does it arrive from an external framework?
What is the durable unit: a task, a workflow activity, an agent step, or a tool call?
How does a dynamic decision get reconstructed after a failure?
How are agent-to-tool permissions enforced?
Who runs the control plane, the data plane, the workers, and the state stores?

Those answers shape the developer experience and your Day-2 operations far more than an AI checkbox does.

One platform strategy, more than one abstraction

Kestra is the better fit when teams want declarative orchestration across data, scripts, APIs, and infrastructure, with a central catalog of flows and plugins to look at.

Catalyst is the better fit when the agent frameworks in use are a deliberate developer choice, and the platform team wants durability, identity, and governance to come from one layer without replacing those frameworks.

Running both is a legitimate answer. A general-purpose orchestrator can trigger a durable agent as a bounded service, or a Catalyst workflow can call an external data pipeline. Draw that boundary where ownership and failure semantics put it, and ignore where a vendor would prefer it.

The feature table does not decide this. Ask which execution model makes each of your workloads easier to build, recover, secure, and explain.

Official sources

Orkes Conductor or Diagrid Catalyst for Multi-Agent Workflows?

Marc Duiker — Tue, 04 Aug 2026 12:25:58 +0000

A multi-agent support flow is still a distributed system. You can make it sound less scary by giving the components friendlier names but the timeouts and the duplicate calls remain part of the system, and so does the question of who owned the handoff that failed.

For example; A triage agent reads a ticket, an account agent fetches customer context, a policy agent checks entitlements, and an action agent calls the order service. Somewhere between the policy decision and the refund API, a worker dies.

At 2 a.m. I care about a couple of things: which step completed, whether the refund can safely repeat, who owned the failed handoff, what an operator can do next, and whether the agent was allowed to make that call at all.

Orkes Conductor's operating model

Orkes Conductor coordinates tasks and workers through workflow definitions stored as JSON, and you can author those definitions in code, through the API, or in a visual builder. The documentation covers static and dynamic workflows, human tasks, event-driven flows, AI tasks, agentic orchestration, and MCP exposure. Orkes also positions Conductor as a platform where existing agents and workflows can invoke one another.

If your platform is already organized around microservice workers and explicit task definitions, that model makes sense. The workflow is a visible coordinator, workers can stay polyglot, and retries and state are handled centrally.

Catalyst's operating model

Catalyst builds on Dapr and treats workflows, agents, MCP servers, and services as workloads on one execution and governance plane. Agents can stay in the frameworks they already run in, while durable runners turn reasoning and tool operations into workflow activities. App identities and policies control which workloads may call other applications or MCP servers.

The interesting part is not the diagram. It is what ends up inside a single runtime boundary: recovery, communication, identity, policy, and operational inspection work the same way for agent and non-agent components.

Walk the failure, not the feature list

Run the same scenario on both platforms.

Complete triage and account lookup.
Let the policy agent approve a refund.
Kill the action worker during the refund call.
Restore service and watch what recovers.
Inspect the execution record and the authorization path.

Then ask whether completed steps replay or re-execute, how the refund call is made idempotent, how an operator retries or terminates the run, and how credentials are scoped. A product page that says "durable" will not answer any of that.

Where each may fit

Orkes is a strong candidate if you want Conductor's task-and-worker model stretched across microservices, APIs, humans, and AI, with workflows managed visually and as JSON.

Catalyst is worth evaluating if you already run Dapr, or several agent frameworks, and you want workload identity, policy-controlled communication, durable execution, and deployment governance to come from one platform.

Do not pick on the number of built-in tasks. Pick on the failure contract and the operating contract, because what saves you at 2 a.m. is an on-call engineer who can repair the run without guessing which agent did what.

Official sources

Camunda or Diagrid Catalyst for AI Workflows With Human Approval?

Marc Duiker — Mon, 03 Aug 2026 12:42:17 +0000

In a regulated workflow, "the agent made a reasonable decision" does not count as an audit trail. Someone still has to show who authorized the action, which controls applied, and where a human could have stepped in.

So Camunda and Diagrid Catalyst are not two versions of the same workflow product, and comparing them that way leads to the wrong choice. Camunda starts from end-to-end process orchestration and BPMN. Catalyst starts from durable, governed execution for applications, workflows, agents, and MCP servers. Both address AI now, but they enter the problem at different control boundaries.

Different approaches

Take an insurance claim. Deterministic steps validate coverage and fetch policy data. An AI agent summarizes the evidence and flags anomalies. A human approves or rejects the recommendation. Then a payment service does something that changes state.

Camunda is strong at making that whole process explicit. Its agentic orchestration documentation describes AI agents inside BPMN workflows, next to human tasks, deterministic rules, and AI-driven decisions. That is worth a lot when business and technology teams need one process model they both work from.

Catalyst treats agents as durable workflows and puts the weight on recovery, identity, policy, and observability in the execution layer. That fits better when the AI part is not one boxed task but an agent that may call tools, delegate to other agents, or run in a different environment.

Four questions that decide the fit

Who owns the process model? If business analysts and process owners have to see and change an end-to-end BPMN model, Camunda has the natural advantage. If developers own the agent code in frameworks they already use, a runtime that preserves those frameworks probably fits better.

Where does the non-determinism live? A bounded AI task inside a stable process is a different design problem from an agent that chooses its next step at runtime. Settle that question before you pick the orchestration layer.

What do you have to prove? An audit may want more than a diagram and some logs. It may want execution history, identity, the authorization decisions that were made, lineage, and evidence that the records were not altered afterwards. Diagrid's verifiable execution positioning is worth checking against the specific compliance requirement in front of you rather than accepting it as a generic checkbox.

Where does it have to run? Camunda and Catalyst both offer deployment choices, but the real constraint is which topology your organization has already approved: SaaS, your own infrastructure, private connectivity, or a fully isolated environment. Confirm the exact edition and who is responsible for what.

You may not have to pick one

An enterprise does not always need a single product to own every layer. Camunda can coordinate the formal business process while a durable agent runtime executes a complicated AI task behind a service boundary. It works the other way too. Catalyst workflows can model human approval without bringing BPMN into it, when the process is mostly owned by developers.

Choose Camunda when BPMN, human work, and process transparency are the center of the design. Evaluate Catalyst when agent durability, framework choice, workload identity, tool policy, and governance across environments are what you spend your time on.

Whichever you pick, the useful test is whether the control boundary is obvious afterwards: an auditor can find the evidence, an operator knows what can be resumed, a developer knows where side effects belong, and everyone knows the point at which the machine is allowed to act on its own.