DEV Community: Diagrid

How Do I Stop Prompt Injection From Reaching My Tools?

Josh van Leeuwen — Thu, 22 Oct 2026 11:48:58 +0000

You can't stop untrusted text from reaching a model. If your agent reads public GitHub issues, anyone can put instructions into the text it reads. The model might follow them, even when you've told it not to.

What you can control is what happens afterwards. Which tools the agent can call, what arguments they accept, and whether the model gets to make the decision to execute them at all.

Take an agent that reads GitHub issues and applies labels. It needs to understand the issue, decide on a label and update GitHub. It doesn't need permission to close issues, merge pull requests or read repository secrets.

Here's how I'd build this with Dapr.

Put the agent inside a workflow

I'd use a Dapr Workflow to manage the issue triage process, rather than letting the model decide which actions to take.

The workflow receives a verified GitHub webhook, passes the issue text to an agent and gets back a suggested label. It then checks that label before calling GitHub.

The important part is what the agent doesn't control. It doesn't choose the repository, issue number or which GitHub operation to run. Those come from the workflow and the original verified event.

The model is deciding how to classify the issue, not what it's allowed to do with GitHub.

Treat MCP tools as workflows too

Dapr has an MCPServer resource which lets you declare a connection to an MCP server. When Dapr loads it, it discovers the available tools and registers a durable workflow for each one.

So if our GitHub MCP server exposes add_label and close_issue, Dapr registers workflows named:

dapr.internal.mcp.github.CallTool.add_label
dapr.internal.mcp.github.CallTool.close_issue

The agent doesn't need to manage the MCP connection itself. The application can invoke those workflows, and Dapr handles the MCP call.

For example, the resource on our github-tools application could look like this:

apiVersion: dapr.io/v1alpha1
kind: MCPServer
metadata:
  name: github
spec:
  endpoint:
    streamableHTTP:
      url: https://github-mcp.internal/mcp
scopes:
  - github-tools

The URL and tool names here are examples; you'd use those exposed by your own MCP server.

Now we need to decide which applications can invoke those tool workflows.

Enforce permissions outside the model

A list of permitted tools in a system prompt isn't a security control. The model can be convinced to ignore it.

With Dapr, you can use a WorkflowAccessPolicy to control which App IDs are allowed to schedule particular workflows on another application.

For our example:

apiVersion: dapr.io/v1alpha1
kind: WorkflowAccessPolicy
metadata:
  name: github-tools-policy
  namespace: default
scopes:
  - github-tools
spec:
  rules:
    - callers:
        - appID: triage-workflow
      workflows:
        - name: dapr.internal.mcp.github.CallTool.add_label
          operations: [schedule]

This allows triage-workflow to invoke add_label on github-tools. It doesn't grant access to close_issue, or any of the other tools the MCP server might expose.

The workflow access policy is enforced by Dapr, using the calling application's verified identity. The model doesn't get to override it.

There are two details worth knowing here. You need a policy loaded for deny-by-default behaviour, and the MCP server workflows need to run under a separate App ID. Dapr always allows an application to schedule its own workflows, so you shouldn't rely on this policy to isolate tools running under the same App ID as the caller.

Check the arguments before calling GitHub

Being allowed to call add_label doesn't mean every argument should be accepted.

A poisoned issue could convince the model to suggest a label like security-approved, or try to make it update a different issue.

I'd have the agent return a label, then check it in the workflow before calling GitHub. Something like this, using the Dapr Python Workflow SDK:

import dapr.ext.workflow as wf

wfr = wf.WorkflowRuntime()

ALLOWED_LABELS = {
    "bug", "feature", "docs",
    "question", "needs-triage"
}

@wfr.workflow(name="triage_issue")
def triage_issue(ctx: wf.DaprWorkflowContext, issue: dict):
    # classify_issue is a registered activity that calls the model
    suggestion = yield ctx.call_activity(
        "classify_issue",
        input=issue["body"]
    )

    label = suggestion["label"]

    if label not in ALLOWED_LABELS:
        return {"status": "rejected"}

    return (yield ctx.call_child_workflow(
        workflow="dapr.internal.mcp.github.CallTool.add_label",
        app_id="github-tools",
        input={
            "arguments": {
                "owner": issue["owner"],
                "repo": issue["repo"],
                "issue_number": issue["number"],
                "label": label
            }
        }
    ))

This is the core of the workflow, with the activity implementation and runtime startup left out. The tool arguments will depend on your MCP server.

Notice that the model only supplies the label. The repository and issue number come from the verified webhook that started the workflow.

I'd also validate arguments where the tool runs. Dapr's MCPServer resource supports beforeCallTool hooks, which are themselves workflows. You can use one to reject labels outside the allow-list before the call reaches the MCP server.

That way the check isn't only happening in the application that called the tool.

Use a separate workflow for anything destructive

Adding a label is one thing. Closing an issue, deleting something or deploying a release is another.

For those operations, I'd use a separate workflow that waits for human approval.

Dapr Workflows can wait for an external event, so the workflow can request approval, stop executing and resume when an authorised user responds. There's no need to keep a process running while someone makes the decision.

The approval needs to be verified by your application, not accepted as something the model says it received.

You can then give the review workflow its own App ID and grant it access to close_issue, without giving that permission to the triage workflow.

The same applies to any other tool where a bad decision is difficult to undo.

What this doesn't fix

None of this stops prompt injection from influencing the model. A malicious GitHub issue might still persuade it to suggest the wrong label.

But the model doesn't have permission to close the issue, the workflow won't accept a label outside the allowed set, and the repository and issue number aren't under the model's control.

That's a much smaller problem than giving an agent access to every tool on the MCP server and relying on a system prompt to keep it in line.

You still need to think about prompt injection. You just shouldn't be relying on the model to enforce the limits of what it's allowed to do.

Dapr documentation:

How to Test AI Agent Crash Recovery Locally with Dapr Workflow

Marc Duiker — Tue, 20 Oct 2026 06:33:12 +0000

Introduction

Most advice on testing AI agents is about the answers, such as evals and golden datasets. Crash recovery needs its own test. If your agent runs as a durable workflow, a restart should resume the same run without repeating finished steps or side effects. You can prove that on your laptop, but only if the crash happens at exactly the same place every time, and pressing Ctrl+C half a second late does not give you that. In this post you put the crash in the code, run a pytest integration test against a Dapr Workflow agent, and check the outcome by counting lines in files. For background on how replay skips completed steps, read the durable agents guide.

Prerequisites:

Python 3.10 or later
Docker and the Dapr CLI, initialized with dapr init (tested with Dapr 1.18)
pip install dapr-ext-workflow pytest

The commands in this post run unchanged in bash and in Windows PowerShell, except the manual run in the verification section, which shows both.

Make the side effect countable

The agent triages support tickets in two steps. The summarize activity stands in for your LLM call, and post_comment stands in for the tool that writes to the ticket. Replace both with something you can count: summarize appends the workflow ID to llm_calls.txt, and post_comment appends one line to ledger.txt. One comment is one line, so correct means one line. DATA_DIR lets every test run write to its own folder.

import os
import sys
from pathlib import Path

import dapr.ext.workflow as wf

wfr = wf.WorkflowRuntime()
DATA_DIR = Path(os.getenv("DATA_DIR", "."))


@wfr.workflow(name="triage")
def triage(ctx: wf.DaprWorkflowContext, ticket: dict):
    summary = yield ctx.call_activity(summarize, input=ticket)
    yield ctx.call_activity(post_comment, input=summary)
    return "done"


@wfr.activity(name="summarize")
def summarize(ctx: wf.WorkflowActivityContext, ticket: dict):
    with open(DATA_DIR / "llm_calls.txt", "a") as f:
        f.write(f"{ctx.workflow_id}\n")  # your LLM call goes here
    return f"Summary: {ticket['title']}"


@wfr.activity(name="post_comment")
def post_comment(ctx: wf.WorkflowActivityContext, summary: str):
    with open(DATA_DIR / "ledger.txt", "a") as f:
        f.write(f"{ctx.workflow_id}: {summary}\n")
    return "posted"


if __name__ == "__main__":
    mode, instance_id = sys.argv[1], sys.argv[2]
    wfr.start()
    client = wf.DaprWorkflowClient()
    if mode == "start":
        client.schedule_new_workflow(
            workflow=triage,
            input={"title": "Login page 500s"},
            instance_id=instance_id,
        )
    state = client.wait_for_workflow_completion(instance_id, timeout_in_seconds=60)
    print(f"STATUS={state.runtime_status.name}", flush=True)
    wfr.shutdown()

Save this as agent.py. The start mode schedules a new workflow instance and the resume mode only waits for an existing one, which is what the restart needs.

Add named crash points

You can stop the process with Ctrl+C or kill, but you will press it half a second late, after the step has finished, and the run proves nothing. A crash you cannot place exactly is a crash you cannot repeat, and a test you cannot repeat never makes it into CI.

Instead, add a function that exits the process when the CRASH_AT environment variable names the current step. os._exit(1) skips cleanup and finally blocks, so it behaves like a power cut. Add this below the DATA_DIR line in agent.py:

CRASH_AT = os.getenv("CRASH_AT")


def crash_point(name):
    if CRASH_AT == name:
        print(f"crashing at {name}", flush=True)
        os._exit(1)  # no cleanup, no finally blocks

Then place two crash points in post_comment. The first fires before the write, and the second fires after the write but before the activity returns:

@wfr.activity(name="post_comment")
def post_comment(ctx: wf.WorkflowActivityContext, summary: str):
    crash_point("before_post")
    with open(DATA_DIR / "ledger.txt", "a") as f:
        f.write(f"{ctx.workflow_id}: {summary}\n")
    crash_point("after_post")
    return "posted"

Write the crash recovery test

The test starts the agent under dapr run as a subprocess, once with the crash switch on and once with it off. The first run must print the crash message and must not complete. The second run must report COMPLETED, and the files must hold the expected counts. Save this as test_crash_recovery.py next to agent.py:

import os
import subprocess
import sys
import uuid
from pathlib import Path

import pytest

pytestmark = pytest.mark.integration

AGENT = Path(__file__).parent / "agent.py"


def run_agent(mode, instance_id, app_id, data_dir, crash_at=None):
    env = {**os.environ, "DATA_DIR": str(data_dir)}
    if crash_at:
        env["CRASH_AT"] = crash_at
    return subprocess.run(
        ["dapr", "run", "--app-id", app_id, "--", sys.executable, str(AGENT), mode, instance_id],
        env=env, capture_output=True, text=True, timeout=120,
    )


def count_lines(path):
    return len(path.read_text().splitlines()) if path.exists() else 0


def test_crash_before_post(tmp_path):
    suffix = uuid.uuid4().hex[:8]
    instance_id, app_id = f"ticket-{suffix}", f"triage-{suffix}"

    crashed = run_agent("start", instance_id, app_id, tmp_path, crash_at="before_post")
    assert "crashing at before_post" in crashed.stdout
    assert "STATUS=COMPLETED" not in crashed.stdout

    resumed = run_agent("resume", instance_id, app_id, tmp_path)
    assert "STATUS=COMPLETED" in resumed.stdout, resumed.stdout[-2000:]

    assert count_lines(tmp_path / "ledger.txt") == 1
    assert count_lines(tmp_path / "llm_calls.txt") == 1

Every test generates a new instance ID and app ID, so state left behind by earlier runs cannot make a test pass for the wrong reason. Both runs share the same app ID because the workflow state belongs to that app. Each test also writes to its own tmp_path, so the line counts start at zero.

[pytest]
markers =
    integration: runs the agent under dapr run and needs a local Dapr install

Handle a crash after the side effect

The two crash points show different things.

With before_post, summarize has already completed and been recorded. On restart it is not run again, so llm_calls.txt keeps one line (no second LLM call and no second token bill), and the ledger ends with one line.

With after_post, the comment was written but the activity never returned, so the runtime has no record that it finished. It runs post_comment again and the ledger gets two lines. That is correct behavior for an activity that crashed mid-flight, which is why a side-effecting activity needs a "did I already do this?" check keyed on ctx.workflow_id. Add one to agent.py:

IDEMPOTENT = os.getenv("IDEMPOTENT") == "1"


def already_posted(workflow_id):
    ledger = DATA_DIR / "ledger.txt"
    return ledger.exists() and any(
        line.startswith(f"{workflow_id}:") for line in ledger.read_text().splitlines()
    )


@wfr.activity(name="post_comment")
def post_comment(ctx: wf.WorkflowActivityContext, summary: str):
    if IDEMPOTENT and already_posted(ctx.workflow_id):
        return "already posted"
    crash_point("before_post")
    with open(DATA_DIR / "ledger.txt", "a") as f:
        f.write(f"{ctx.workflow_id}: {summary}\n")
    crash_point("after_post")
    return "posted"

The IDEMPOTENT switch only exists so the test can show both outcomes. In your own agent the check is always on, and the key is usually the idempotency key of the system you write to, such as your ticketing tool.

Now parametrize the test over the three cases. Pass the new idempotent option through run_agent, and replace test_crash_before_post:

def run_agent(mode, instance_id, app_id, data_dir, crash_at=None, idempotent=False):
    env = {**os.environ, "DATA_DIR": str(data_dir), "IDEMPOTENT": "1" if idempotent else "0"}
    if crash_at:
        env["CRASH_AT"] = crash_at
    return subprocess.run(
        ["dapr", "run", "--app-id", app_id, "--", sys.executable, str(AGENT), mode, instance_id],
        env=env, capture_output=True, text=True, timeout=120,
    )


@pytest.mark.parametrize(
    "crash_at, idempotent, expected_comments",
    [
        ("before_post", False, 1),
        ("after_post", False, 2),  # at-least-once: the comment is written twice
        ("after_post", True, 1),
    ],
)
def test_crash_recovery(tmp_path, crash_at, idempotent, expected_comments):
    suffix = uuid.uuid4().hex[:8]
    instance_id, app_id = f"ticket-{suffix}", f"triage-{suffix}"

    crashed = run_agent("start", instance_id, app_id, tmp_path, crash_at, idempotent)
    assert f"crashing at {crash_at}" in crashed.stdout
    assert "STATUS=COMPLETED" not in crashed.stdout

    resumed = run_agent("resume", instance_id, app_id, tmp_path, idempotent=idempotent)
    assert "STATUS=COMPLETED" in resumed.stdout, resumed.stdout[-2000:]

    assert count_lines(tmp_path / "ledger.txt") == expected_comments
    assert count_lines(tmp_path / "llm_calls.txt") == 1  # summarize is never replayed

The second case asserts two lines on purpose. It documents the at-least-once behavior, so a future change to the runtime or to your activity cannot alter it without a failing test.

Verify the test and watch the run

Run the integration tests:

pytest -m integration -v

All three cases should pass in about 20 seconds:

test_crash_recovery.py::test_crash_recovery[before_post-False-1] PASSED
test_crash_recovery.py::test_crash_recovery[after_post-False-2] PASSED
test_crash_recovery.py::test_crash_recovery[after_post-True-1] PASSED

To watch a single run in detail, start the Dapr Dev Dashboard in a second terminal with diagrid-dev-dashboard and open http://localhost:9090. Then crash and restart the agent by hand, using a fresh instance ID. In bash on macOS or Linux:

ID=ticket-$RANDOM
CRASH_AT=before_post dapr run --app-id triage -- python agent.py start $ID
dapr run --app-id triage -- python agent.py resume $ID
wc -l ledger.txt llm_calls.txt

In Windows PowerShell, set the environment variable for the first run and remove it before the restart:

$ID = "ticket-$(Get-Random)"
$env:CRASH_AT = "before_post"
dapr run --app-id triage -- python agent.py start $ID
Remove-Item Env:CRASH_AT
dapr run --app-id triage -- python agent.py resume $ID
(Get-Content ledger.txt | Measure-Object -Line).Lines
(Get-Content llm_calls.txt | Measure-Object -Line).Lines

Select the instance in the dashboard to see its event history: which activity completed before the crash, and which one was scheduled again after the restart. To run the same check in CI, give the runner Docker, run dapr init, and add pytest -m integration next to your evals.

Troubleshooting

If a test passes suspiciously fast, check the instance ID and app ID first. Workflow state persists in your local state store, so resuming an instance that finished in an earlier run reports COMPLETED right away, even when nothing crashed and nothing was written. The unique IDs in the test prevent this. When you run the commands by hand, use a fresh $ID every time.

The dapr run output can show actor retry warnings while the restarted app registers with the placement service. In these runs the workflow still completed, so keep the 60 second wait timeout generous instead of treating those warnings as a failure.

Summary

Crash recovery is now a test that fails when recovery breaks. A named crash point, a restart with the switch off, and a ledger you can count tell you what your agent did after it died. Both crash points should end at one line: before_post because completed steps are never replayed, and after_post because your activity checks whether it already did the work. Add the pytest run to CI next to your evals, and open the Dev Dashboard whenever you want to see the event history behind a result. The durable agents guide explains checkpointing and replay in more detail, and Dapr University has courses on durable execution and Dapr.

The Three Failures Every Fault Tolerant AI Agent Must Handle

Marc Duiker — Wed, 14 Oct 2026 13:32:46 +0000

"Make the agent fault tolerant" sounds like one task. It is three. Your agent can fail loudly, with an error you can see and retry. It can fail silently, when a side effect fires and the process dies before anything records it. And it can fail by waiting, when an approval takes two days and the process holding that wait gets restarted on day one. Each class needs a different mechanism, and most agent frameworks only ship the first. That is why demos look fault tolerant and production agents send the same email twice or lose an approval.

Why fault tolerance is three assignments

When you move an agent from prototype to production, "make it fault tolerant" lands in the backlog as one ticket. You handle the failures you've seen in testing, and those are almost always the loud ones: a tool returns a 500, the model API times out, the LLM sends back malformed JSON.

The other failures rarely show up on your laptop. They show up during a deploy, when the orchestrator kills a pod for using too much memory, when the cloud provider reclaims a spot instance, or when a run sits waiting for a human for three days. Fault tolerance for an agent covers three distinct classes:

Visible failures, where something errors and you know about it.
Invisible failures, where work happens but the record of it is lost.
Long waits, where the agent has to outlive the process running it.

Visible failures and the retry toolbox

A visible failure announces itself. The tool call returns an error, the API times out with a clear exception, the model refuses the request or returns output your parser rejects.

The toolbox for this class is well known:

Retry with exponential backoff, so a rate-limited or briefly unavailable service gets time to recover.
Cap the number of attempts, so a permanent error doesn't turn into an infinite loop that burns tokens.
Dead-letter the requests that keep failing, so a person or a separate process can look at them later.

Almost every agent framework and HTTP client ships some version of this, so it's the part of fault tolerance most teams already have.

Retries have one built-in assumption, though: the failed attempt had no effect. A retry repeats a step; it doesn't know whether the previous attempt already sent the email or charged the card before the error came back. The next class is where it breaks.

Invisible failures and why checkpointing falls short

An invisible failure happens in the gap between doing something and recording that you did it. Your agent calls the payment API and the charge goes through. Before the agent writes "payment done" anywhere, the process disappears. A rolling deploy replaced the pod, the kernel's out-of-memory killer picked your container, or the spot instance was reclaimed. No exception is raised and no retry policy fires, because the code that would catch the error is gone too.

Many agent frameworks answer this with checkpointing. It saves a snapshot of the agent's state after each step, and a restarted agent loads the latest one. If the process dies during a step or right after it, that snapshot is from before the side effect. The restored agent has no idea the charge happened, and it charges again.

Durable execution closes most of that gap by making the record part of execution itself. The engine writes "scheduled" to a journal before a step runs, and "completed" with the result once it returns. On restart it replays the journal, skips every step marked completed, uses the recorded results instead of calling the tools again, and resumes at the first unfinished step. This is the class of failure that separates a demo agent from a production one.

There's still a small window left: the side effect can succeed just before the process dies and before "completed" is written. In that case the engine reruns the step, which is why durable execution engines guarantee that a step runs at least once rather than exactly once. The journal gets you back to the right step. Making the side effect happen once is the job of the step itself: pass an idempotency key to the downstream service so it recognizes the repeat and returns the original result. Dapr Workflow gives every scheduled task a task execution ID that stays the same across retries, so you can use it as that key.

Long waits and durable timers

Agents spend a lot of their lives waiting. A refund over a certain amount needs a manager's approval, which might come in two hours or two days. A procurement agent can't confirm delivery until the container ships on Tuesday.

Processes don't live that long. Over a two-day approval window your agent will probably go through several deploys, a node upgrade or two, and maybe an autoscaler scaling it down overnight. A wait held in process memory, like a sleep call, an in-memory promise or a loop polling an inbox, disappears with the process. Any wait held in memory is a failure scheduled in advance; you just don't know the exact time yet.

Durable execution handles this by moving the wait into the journal as well. A durable timer records "wake me at this time," and an external event records "resume when this approval arrives." While the agent waits, nothing needs to run: the process can stop, restart or move to another node. When the timer fires or the event comes in, the engine loads the journal and continues from the waiting step. Pair the approval event with a timer so that if nobody responds within three days the agent escalates or cancels instead of waiting forever.

The crash test that checks all three

You can check your agent against both of the harder classes with two tests. Both use a force-kill, meaning the process ends right away with no graceful shutdown and no cleanup handlers. For example:

In a container, use docker kill, or delete the pod with a zero grace period in Kubernetes.
On Windows, end the process from Task Manager or with taskkill /F.
On macOS, use Force Quit in Activity Monitor or kill -9.
On Linux, use kill -9 with the process ID.

For the first test, start a run that makes a side-effecting call, such as sending an email or creating a record in another system. Force-kill the process right after that call succeeds, restart it, and let the run finish. Then count the effects in the downstream system. One is the only right answer. If you see two, the step needs an idempotency key.

For the second test, start a run that waits for an approval or a timer. Force-kill the process during the wait, restart it, then send the approval. The agent should pick up the event and continue.

Most teams skip these tests because they tend to fail the first time. That is exactly why you should run them before production does.

Trade-offs and considerations

Durable execution isn't free. Every step writes to the journal before and after it runs, which adds a little latency and storage per step. For steps with side effects or long waits, that cost is almost always worth paying; for a quick in-memory computation it usually isn't.

The orchestration logic also has to be deterministic, because the engine replays it to rebuild state. Keep LLM calls, tool calls and anything else non-deterministic inside activities, and keep the orchestration code limited to deciding what runs next.

Not every agent needs all three layers either. A stateless chat helper that only reads data is fine with the retry toolbox alone.

Summary

Fault tolerance for AI agents is three separate problems. Retries with backoff and dead-lettering handle visible failures. Invisible failures need more than checkpointing: durable execution writes each step to a journal before and after it runs, so recovery resumes at the right step, and an idempotency key makes sure the side effect lands once. Long waits belong in the journal too, as durable timers and external events, so a restart can't erase them. The two force-kill tests tell you which of these your agent already handles. To see these ideas in a small working example, read The Tiniest Durable Agent. To learn durable timers, external events, and recovery hands-on, take the free Dapr Workflow course on Dapr University.

Agents Are For-Loops. That Is Good News for Security.

Josh van Leeuwen — Tue, 13 Oct 2026 09:42:17 +0000

If agents were magic, we couldn’t secure them. They aren’t, so we can.

AI agents are not magic. They are workflows with probabilistic decision-making and real-world privileges. Strip away the framing and you find the classical operator feedback loop you already know from Kubernetes controllers and embedded thermostats: observe, decide, act, repeat. The only new element is that the deciding step is a language model.

This is good news, because it means twenty years of distributed-systems security applies almost unchanged. The model is the part you cannot trust; everything around it is the part you already know how to lock down.

LLMs don’t act. Tools do.

A model emits instructions. It’s the tools in the acting step that translate the model’s intent into real-world behaviour- the database write, the refund, the deployment. Which means the security boundary is not the prompt and never was. It’s the tool layer, and tool access is an authorisation problem we have mature answers for: each agent gets its own verifiable identity, each identity gets explicitly scoped grants, and everything ungrated is denied by default.

Note what this replaces. The common pattern today is one shared API key in an environment variable, used by every agent, with the real policy written in the system prompt as a polite request. A permission that exists only as an instruction to a model is not a permission. And a gateway in front of the tools helps less than it appears: a gateway secures a path, it cannot secure a workload it does not run. If the agent can reach the backing service directly, the gateway is decoration.

What a secured for-loop looks like

Identity issued per agent, short-lived and cryptographically verifiable (SPIFFE gives you the standard). Authorisation scoped per tool, per agent, enforced in the runtime. Deny-all as the starting posture, grants as deliberate exceptions. And a signed execution history, so “what did the agent do” has an answer that survives an audit rather than a log file that anyone could have edited.

There is still a weakness in this system: prompt injection against a tool the agent is legitimately granted. If the agent is allowed to send email and a poisoned document convinces the model to send the wrong one, identity and authorisation did their job and the damage still happened. No infrastructure makes a model trustworthy. What infrastructure can do is keep the set of things a misbehaving model can reach small- which turns a catastrophe into an incident.

The deeper treatments: Agent Identity: The Foundational Layer AI Is Still Missing and Why MCP Gateways Are Not Enough.

Understanding At-Least-Once Execution and Why It Matters for Workflow & Agent Developers

Marc Duiker — Thu, 08 Oct 2026 12:51:24 +0000

Introduction

Durable execution guarantees that your workflow finishes. It does not guarantee that each step runs only once. Picture a workflow that charges a credit card for 49 EUR. The payment provider accepts the charge, and then the pod is evicted before the workflow records the result. On restart, the workflow sees an unfinished step and charges the card again. Your customer now pays 98 EUR.

If you are new to durable execution, this can feel like a bug in the engine. It is not. Durable execution engines, including Dapr Workflow, run activities at least once, and making a side effect happen exactly once is the job of your activity code. In this post, you'll see why the engine behaves this way, how to spot the steps that need protection, and how to use the task execution ID in a C# activity as an idempotency key.

Durable Execution Guarantees Completion, Not Single Execution

A Dapr Workflow app has two kinds of code. The workflow code decides what happens next: call this activity, wait for that event, branch on a result. The activity code does the actual work that touches the outside world, such as calling a payment provider, writing to a database, or sending an email. The workflow engine persists every step, so when your app crashes it can pick up where it left off instead of starting over. If you want a primer on this model, watch this session on durable execution with Dapr.

There is a catch in "pick up where it left off". The engine can only store that a step happened after the step has happened. Between the moment your activity charges the card and the moment the engine writes down the result, there is always a window. If the process dies inside that window, the engine has no record of the charge, so the only safe choice is to run the activity again.

That is why durable execution gives you completion, not exactly-once execution. Exactly-once side effects come from making each activity idempotent: running it twice with the same input has the same effect as running it once. This is not specific to Dapr. Temporal and Azure Durable Functions make the same trade-off and give the same advice.

How Dapr Workflow Records Progress

Here is the order workflow from the intro, written with the Dapr .NET SDK:

public class OrderWorkflow : Workflow<Order, OrderResult>
{
    public override async Task<OrderResult> RunAsync(WorkflowContext context, Order order)
    {
        var payment = await context.CallActivityAsync<PaymentResult>(
            nameof(ChargeCardActivity),
            new ChargeRequest(order.CustomerId, order.Amount));

        await context.CallActivityAsync(nameof(ShipOrderActivity), order);

        return new OrderResult(payment.PaymentId);
    }
}

The workflow code does not call ChargeCardActivity directly. The orchestration lifecycle describes what happens instead. The Dapr sidecar sends your app an OrchestratorWorkItem over a long-lived GetWorkItems gRPC stream. That work item contains the full history of the workflow instance. The SDK replays your RunAsync method from the top against that history. When it reaches CallActivityAsync, it looks for a recorded result. If there is one, it returns the recorded result without running the activity. If there isn't, it emits a ScheduleTask action and suspends the workflow.

The engine then appends a TaskScheduled event to the history and dispatches the activity. When the activity reports back, the engine records a TaskCompleted (or TaskFailed) event, and the next replay picks up the result.

According to the state and history docs, this history is the source of truth. It is stored in your configured state store as a list of serialized events, with keys like wf-history-<instance_id>-<index>. For the order workflow, the history at the moment of the pod eviction looks like this:

ExecutionStarted   OrderWorkflow
TaskScheduled      ChargeCardActivity
                   <- pod evicted here, card already charged

There is no TaskCompleted for ChargeCardActivity. As far as the engine knows, the charge never finished.

Why Activities Run at Least Once

The activity lifecycle shows where the window sits. When the engine dispatches an activity, the sidecar sends an ActivityWorkItem over the same GetWorkItems stream. The work item carries the activity name, its input, the workflow instance_id, a task_id, a task_execution_id, and a completion_token. The SDK runs your activity code and then calls CompleteActivityTask with the result (or failure_details) and the completion_token, as defined in the execution API.

Your side effect happens while the activity runs. The engine learns about it only when CompleteActivityTask arrives. Anything that breaks the chain between those two points, such as a pod eviction, an out-of-memory kill, a node failure, or a dropped network connection, leaves a TaskScheduled event without a matching TaskCompleted. The activity lifecycle doc names this case directly: activities "might be executed more than once (e.g., if the worker crashes after execution but before reporting completion)".

Retry policies are a second source of repeats. If an activity fails and the workflow has a retry policy, the engine schedules the activity again with the same name and input. A failure from your activity's point of view is not always a failure on the other side. If the HTTP call to the payment provider times out, your activity throws, yet the provider may have processed the charge anyway.

The third case is a slow worker. If a worker takes too long and the task is handed to another worker, the original one can still finish. The sidecar uses the completion_token to ignore the late response, but by then the side effect has already happened.

In all three cases, the engine protects its own bookkeeping, not your side effects. That is why at-least-once is the norm for durable execution engines.

How to Spot the Steps That Need Idempotency

Not every activity needs extra work. For each activity in your workflow, ask one question: if this runs twice with the same input, is the outcome different from running it once?

Activities that usually need protection:

Charging, refunding, or transferring money
Sending emails, text messages, or push notifications
Inserting records with generated IDs, where a second insert creates a duplicate row
Publishing messages to a broker or event stream
Incrementing counters or adjusting stock levels
Calling an LLM tool or external API that acts on another system, such as creating a ticket

Activities that are usually safe already:

Pure computation on the input
Reads, such as GET requests and database queries
Upserts keyed on a natural ID, such as the order ID
Setting a value to a fixed state, such as "set order status to Shipped"

Don't forget compensation activities. If your workflow uses the saga pattern, the refund that undoes a charge can run twice as well, and refunding a customer twice is just as wrong as charging them twice.

Keep this separate from workflow determinism. Determinism is about your workflow code producing the same decisions on every replay, so no DateTime.UtcNow, random numbers, or direct I/O in RunAsync. Idempotency is about your activity code having the same effect when it runs more than once. You need both, but they solve different problems.

Using the Task Execution ID as an Idempotency Key

The standard fix is an idempotency key. You send a key with the request, and the receiving service remembers which keys it has already processed. When the same key arrives a second time, the service returns the original result instead of doing the work again. Most payment providers support this through an Idempotency-Key HTTP header.

The key has to be the same every time the activity runs for the same step, and different for every other step. Since Dapr .NET SDK v1.17, WorkflowActivityContext exposes a TaskExecutionId property that fits this. The ID is created when the workflow schedules the activity and is stored with the TaskScheduled event in the history. When the engine dispatches that scheduled task again after a crash, a pod eviction, or a worker timeout, your activity receives the same ID. Treat it as an opaque string and don't parse it.

Here is the payment activity with the task execution ID passed as the idempotency key:

public class ChargeCardActivity(
    IHttpClientFactory httpClientFactory,
    ILogger<ChargeCardActivity> logger)
    : WorkflowActivity<ChargeRequest, PaymentResult>
{
    public override async Task<PaymentResult> RunAsync(
        WorkflowActivityContext context, ChargeRequest request)
    {
        var idempotencyKey = context.TaskExecutionId;
        logger.LogInformation(
            "Charging {Amount} EUR for {CustomerId} with idempotency key {Key}",
            request.Amount, request.CustomerId, idempotencyKey);

        var httpClient = httpClientFactory.CreateClient("payments");
        using var message = new HttpRequestMessage(HttpMethod.Post, "/v1/payments")
        {
            Content = JsonContent.Create(request)
        };
        message.Headers.Add("Idempotency-Key", idempotencyKey);

        using var response = await httpClient.SendAsync(message);
        response.EnsureSuccessStatusCode();

        return (await response.Content.ReadFromJsonAsync<PaymentResult>())!;
    }
}

Replay the eviction scenario with this version:

The engine dispatches ChargeCardActivity with task execution ID a1b2....
The provider charges 49 EUR and stores the payment under key a1b2....
The pod is evicted before CompleteActivityTask reaches the sidecar.
The workflow restarts, and the engine dispatches the same scheduled task again with task execution ID a1b2....
The provider recognizes the key and returns the original payment without charging the card.
The engine records TaskCompleted, and the workflow moves on to shipping.

Your customer pays 49 EUR. The same approach works for any service that accepts a deduplication key, such as email providers, message brokers that support message deduplication IDs, and your own APIs.

When the Idempotency Key Expires

An idempotency key only helps while the service still remembers it. Many providers clear keys after a fixed period. Stripe, for example, can remove keys once they are at least 24 hours old. Check the documentation of every service you call for its retention period.

Durable workflows can run for days or weeks, and a second dispatch doesn't always follow the first one quickly. An outage, a long retry backoff, or a suspended workflow that resumes later can all push the second attempt past the retention period. At that point the provider treats the request as new. Some services don't support idempotency keys at all.

In those cases you need a record you own: a dedupe table in your own database, keyed on the task execution ID. Before doing the work, the activity checks the table. After doing the work, it stores the result.

public class SendInvoiceActivity(InvoiceDbContext db, IEmailSender emailSender)
    : WorkflowActivity<InvoiceRequest, InvoiceResult>
{
    public override async Task<InvoiceResult> RunAsync(
        WorkflowActivityContext context, InvoiceRequest request)
    {
        var key = context.TaskExecutionId;

        var processed = await db.ProcessedTasks.FindAsync(key);
        if (processed is not null)
        {
            return new InvoiceResult(processed.MessageId);
        }

        var messageId = await emailSender.SendAsync(
            request.To, request.Subject, request.Body);

        db.ProcessedTasks.Add(new ProcessedTask(key, messageId, DateTimeOffset.UtcNow));
        await db.SaveChangesAsync();

        return new InvoiceResult(messageId);
    }
}

Make the task execution ID the primary key of ProcessedTasks. A unique constraint stops two concurrent attempts, such as a slow worker and its replacement, from both recording a result. Keep rows for longer than your longest-running workflow, and clean up older rows with a scheduled job.

This version still has a gap. If the process dies between SendAsync and SaveChangesAsync, the email goes out twice. To close it, insert a Pending row before calling the external service and mark it Completed afterwards. When an attempt finds a Pending row, it knows a previous attempt may have done the work, and it can check with the provider before trying again. Where the provider supports keys, send the task execution ID to it as well: the provider's key covers the short window, and your table covers everything after the key expires. For more on choosing between keys and this kind of reconciliation, see the Diagrid FAQ on idempotency keys and reconciliation logic.

Trade-offs and Considerations

Check how the task execution ID behaves with retry policies before you rely on it. The .NET SDK documentation describes it as stable across retries, while the activity lifecycle protocol doc says it changes every time the engine retries the activity. For the payment timeout case, that difference decides whether the provider sees one charge or two. Test it on your Dapr and SDK versions. If you need one key across all retry attempts, build it in the workflow from the instance ID and a step name, for example $"{context.InstanceId}:charge-card", and pass it as activity input. Use a different step name for every call, including calls inside a loop.

Scope each key to one side effect. If an activity both charges a card and sends a receipt, it needs two keys, such as {TaskExecutionId}:charge and {TaskExecutionId}:receipt. Splitting the work into two activities is usually cleaner.

Idempotency has a cost. A dedupe table adds a database round trip to every call and a cleanup job to your operations work. Spend that effort on the activities from the checklist above, not on every activity.

Don't move side effects into the workflow code to avoid retries. Workflow code replays on every step, so a side effect there runs far more often, and it breaks determinism.

Finally, log the key in every activity. Idempotency stops the duplicate from reaching your customer, and the logs tell you how often it happened.

Summary

Durable execution keeps your workflow moving through crashes and pod evictions, but it does so by running activities at least once. The Dapr workflow protocol makes the reason visible: the engine only knows an activity finished once a TaskCompleted event lands in the history, so any failure in between leads to a second dispatch. The fix belongs in your activities. Find the steps with side effects, pass the task execution ID as an idempotency key to the services that support it, and back it with a dedupe table when the downstream service forgets keys too soon or doesn't support them at all. Your 49 EUR charge then stays 49 EUR.

To build a solid foundation in the Dapr Workflow programming model, including activity retries and error handling, take the free Dapr Workflow track on Dapr University. If you prefer video, this session on durable execution with Dapr covers how replay and history work.

What's New in the Dapr Dev Dashboard for Local Workflow Debugging

Marc Duiker — Mon, 05 Oct 2026 12:39:18 +0000

Introduction

The Diagrid Dapr Dev Dashboard has shipped nine releases since the first announcement, and if you build Dapr Workflow apps on your machine, several of them change how you debug. The Workflows page now loads thousands of instances in milliseconds instead of hanging. A new State page lets you browse, add, and delete the records in your state store. A context-aware Dapr CLI drawer gives you the exact dapr workflow command for the instance you are looking at, with the IDs already filled in. On top of that, you can publish test messages to a topic, check component secret references, and keep the app list and logs tidy as your dev session grows. This post covers the highlights from v0.0.14 to v0.0.22 and shows how each one helps you find out why a workflow behaves the way it does. You can find an overview of the dashboard on the Dapr Dev Dashboard page.

Why it matters

A Dapr Workflow is long-running and event-sourced. Its state lives in your state store as a history of events, so when an instance stalls or returns the wrong output on your machine, the answer is somewhere in that history. Finding it usually means querying the state store by hand, looking up the right CLI flags, or adding log statements and running the workflow again.

The dashboard already took care of the basics. It finds the apps you start with dapr run, .NET Aspire, Docker Compose, or Testcontainers, tails their sidecar and app logs, shows workflow event history, and generates component and resiliency YAML from a form. The recent releases go further into the questions you ask while debugging: what did the workflow write, which instances are stuck, and what do you run to fix them.

What's new

A Workflows page that stays fast on large state stores

Run a load test or leave a fan-out workflow running overnight, and your local state store can easily hold thousands of workflow instances. Earlier versions of the Workflows page slowed to a crawl at that point, because every instance load scanned the store for that instance's keys: a full keyspace SCAN on Redis, a LIKE table scan on the SQL backends.

The dashboard now loads instances the way Dapr loads its own state. It reads the instance's metadata record, builds the history keys from it, and fetches them in one bulk read. The status counts on the tabs are cached per workflow and only reload when that workflow changes, while the workflow detail page always reads fresh data. These are the numbers for 2,000 instances, before and after:

Backend	List page	Status counts, first load	Status counts, refresh
SQLite	341 ms → 10 ms	13.7 s → 0.20 s	15.9 s → 0.03 s
Redis	702 ms → 39 ms	28.6 s → 1.28 s	29.4 s → 0.10 s
PostgreSQL	386 ms → 18 ms	14.9 s → 0.17 s	14.7 s → 0.02 s
MongoDB	200 ms → 7 ms	8.1 s → 0.28 s	8.1 s → 0.08 s

The page also behaves better while it loads. When you change a status tab, search, or page, the current rows stay on screen with Updating… in the pager until the new results arrive. The tabs show … until their counts are in, instead of a misleading 0.

Browse and edit state with the new State page

Workflow activities often read and write application state, and when a workflow produces the wrong result, you want to see what is actually in the store. The new State page lists the records in any connected state store, with the key, app prefix, value preview, size, version (etag), and TTL for each one. You can switch between stores from the page header and expand a row to see the full value, with a copy button. The dashboard opens Redis, PostgreSQL, SQLite, and MongoDB stores directly.

To find a record, filter by app prefix or search by key. If Dapr stored a value base64-encoded, flip Decode base64 to read it. Workflow history and actor state share the keyspace with your app's records, so they are hidden by default. Turn on Show internal keys when you want to look at the raw workflow keys.

You can also change state. + New record writes a record under <app-id>||<key>, the same key shape Dapr uses, which is handy for seeding data a workflow reads. An existing key is refused unless you tick Overwrite. You can delete records one at a time or select several, with a confirmation step before anything is removed.

Two limits are worth knowing. There is no "last modified" column, because Dapr state components do not expose a modification timestamp. And a store the dashboard cannot open directly, such as an in-memory store inside a Testcontainers app, cannot be browsed, since Dapr's state API has no way to list keys.

Get the right Dapr CLI command for every page

The dashboard is great for looking around, but sometimes you want the terminal, for example to put a command in a script or rerun it after a restart. A collapsible Dapr CLI drawer on the right edge of the dashboard now shows the Dapr CLI commands that match the page you are on.

On the workflow views, that means dapr workflow list, history, suspend, resume, terminate, and purge, with the app ID and workflow instance ID from the current view already filled in. Every command has its own copy button, so you do not have to look up flags or paste IDs by hand.

Other pages get their own set: dapr list on the applications overview, dapr stop --app-id <id> on an app's detail page, dapr scheduler commands on actors, and dapr publish on subscriptions. The commands target self-hosted mode, so you will not find the Kubernetes -k variants here.

Publish test messages from the Subscriptions page

If a message on a topic starts your workflow, testing it used to mean writing a small client or reaching for curl. Each row on the Subscriptions page now has a Publish button that sends a real message to the topic through the app's own sidecar. You edit the JSON or text payload, choose a content type, and optionally set ttlInSeconds or rawPayload under Advanced.

When the publish succeeds, you get a direct link to the app's logs, so you can watch the message get handled. If daprd rejects it, for example because the pub/sub component is unknown, the error shows up inline in the dialog.

The page also tells you more about each subscription. A Type column shows whether it is declarative or programmatic, and subscriptions with several routing rules expand to show each rule's match expression and path.

See which component secrets resolve

Your workflow state store is a Dapr component, and if its password comes from a secret store, one typo in a secretKeyRef is enough to break it. Until now, a component with a missing secret looked exactly like a working one. The dashboard now resolves secretKeyRef values against the secretstores.local.file and secretstores.local.env stores the same way daprd does, including nested keys and custom separators. envRef is supported too.

A component's detail page has a new Secret references panel. For each field it shows the store, the key, and a status, and when resolution fails, the exact file path it tried. Values stay masked; Reveal shows one field at a time and masks it again after 30 seconds. The Components list marks any component with an unresolved reference with a warning sign. When a state store cannot connect because of a secret, the State page names that secret instead of showing a vague connection error.

Non-local secret stores, such as HashiCorp Vault or Azure Key Vault, are listed but never read.

Cleaner logs and app list during long dev sessions

On the Logs page, choosing what to tail is now a single Target dropdown, grouped into Applications and Control plane. When you view an app, a compact daprd | app toggle lets you show the sidecar logs, the app logs, or both side by side.

After a few rounds of starting and stopping apps, the Applications list fills up with stopped entries. Clear inactive removes all fully stopped apps in one click, and each stopped row has its own Remove action. Removing an app only hides it from the list; if it starts again, it reappears. Restart and Start now only appear for Docker Compose apps, where they work reliably. For dapr run apps you get Stop, with a hint to restart from your terminal.

How to get started

The dashboard is a single self-contained binary with no runtime dependencies, available for macOS and Linux (amd64 and arm64) and Windows (amd64). Install it with the one-liner for your OS.

macOS or Linux:

curl -sSL https://raw.githubusercontent.com/diagridio/dev-dashboard/main/scripts/install.sh | sh

Windows (PowerShell):

iwr -useb https://raw.githubusercontent.com/diagridio/dev-dashboard/main/scripts/install.ps1 | iex

Start the dashboard next to your running Dapr apps:

diagrid-dev-dashboard

It serves the UI on http://localhost:9090 and opens your browser. If no Dapr apps are running yet, the Applications page points you to the Dapr Quickstarts so you have something to try it on. The GitHub repo has the full README and the release notes for every version.

Summary

Most of the recent work on the Dapr Dev Dashboard targets the questions you ask while debugging a workflow on your machine: what state did it write, which instances are stuck, and which command gets them moving again. A faster Workflows page, the new State page, and the Dapr CLI drawer answer those questions from your browser. You can also publish a test message from the subscription it targets, and spot a missing secret before it shows up as a vague connection error.

Install the dashboard with the one-liner for your OS, go install, or the container image from the GitHub repo, and run diagrid-dev-dashboard next to your Dapr apps. For an overview of everything it can do, visit the Dapr Dev Dashboard page.

Once the dashboard is open, try the Konami code: type ↑ ↑ ↓ ↓ ← → ← → B A in your browser and see what appears. There is a fresh challenge every day, and you can share your best run with an execution ID so others can watch it back.

The Tiniest Durable Agent I Can Write

Marc Duiker — Mon, 28 Sep 2026 07:11:56 +0000

"Durable execution" sounds like something you adopt after a six-week platform evaluation. It isn't. Here's an agent that survives being killed, in roughly twenty lines of Python.

The trick is that durability isn't a feature you code – it's a property of where the agent's loop runs. With Dapr Agents, the loop runs as a durable workflow, so every LLM call and tool call is journaled automatically:

from dapr_agents import DurableAgent, tool

@tool
def search_flights(destination: str) -> list:
    """Search available flights."""
    return flights_api.search(destination)

@tool
def book_flight(flight_id: str) -> str:
    """Book a flight. Real money involved."""
    return flights_api.book(flight_id)

agent = DurableAgent(
    name="TravelBuddy",
    role="Travel planning assistant",
    instructions=["Find options first. Confirm before booking."],
    tools=[search_flights, book_flight],
)

agent.start()

That's the whole thing. Now the fun part – try to break it:

Ask it to plan a trip. Watch it call search_flights.
Kill the process while it's mid-task. Not gracefully, kill the entire process.
Start it again.

The agent picks up the same task where it left off.
The search results it already gathered come back from the workflow's execution history instead of being fetched (and paid for) again. And if the crash happened around book_flight, the journal knows whether that step confirmed – so recovery doesn't book a second seat. LLM calls are slow, costly, and non-deterministic; re-running everything after a crash burns money and can change the answer. Replaying from history does neither.

Under the hood this is a durable workflow – the same durable execution machinery you'd use for order processing or payment flows, wrapped around an agent loop. No extra database to run: state goes to whatever Dapr state store you configure (Redis locally, Postgres or others in production).

Honest caveats, as always: the agent's reasoning is only as repeatable as your model, tools with side effects still deserve idempotency keys for the truly paranoid path, and you should read up on workflow versioning before changing an agent that has long-running tasks in flight.

The longer version of this, with the full runnable sample, lives in The Tiniest Durable Agent. If you want the structured path, the Dapr Agents track at Dapr University is free, and the quickstarts get you a running example in a few minutes.

Your LangGraph Agent Survived the Demo. Will It Survive a Deploy?

Marc Duiker — Fri, 25 Sep 2026 05:54:28 +0000

The LangGraph checkpointer is good. I want to say that first, because what follows reads like a complaint and it isn't one. Thread-level persistence, time travel, human-in-the-loop pauses: those work, and they work well.

There's one failure mode it doesn't cover, and it's the one you hit in production. The process dies after a side effect but before the checkpoint that records it.

What's the difference between checkpointing and durable execution?

A checkpoint is a snapshot of agent state, written once a step has finished. Durable execution keeps a journal of every step as it runs, and the runtime, not your code, decides what completed.

Here is where that matters. A graph node calls an external API. It files a Jira ticket, or it charges a card. The call succeeds. Then the pod catches a SIGKILL during a rolling deploy, before the checkpoint write lands.

A checkpointer restores the last checkpoint it has, which is the one from before the node ran. So the agent runs the node again, and now there are two tickets and two charges.

Durable execution sees an activity in the journal that started and never confirmed, and it can tell you that's what it is. The steps that did confirm don't run twice. The workflow resumes at the one that didn't.

That's the part of AI workflow orchestration nobody demos. Drawing the boxes and arrows is easy. The runtime underneath has to answer "what already happened?" at any point, and that is a harder thing to build than it looks.

Keeping LangGraph, adding durability

You don't have to pick one. LangGraph stays the brain, and each side-effecting step moves inside a durable activity.

# before: the node does the side effect directly
def create_ticket_node(state):
    ticket = jira.create(state["summary"])   # unprotected side effect
    return {"ticket_id": ticket.id}

# after: the node schedules a durable activity
def create_ticket_node(state):
    ticket_id = yield ctx.call_activity(create_ticket, input=state["summary"])
    return {"ticket_id": ticket_id}

Same graph, same prompts. The difference is that every step now gets journaled, retried with backoff, and resumed after a crash rather than replayed from scratch. That applies to your LLM calls too, so a crash doesn't re-spend the tokens you already burned on steps one through six.

Caveat where it's due, because durable execution asks for things too. Your workflow code has to be deterministic, which rules out random branching outside activities. And you need to think about versioning before you change a workflow that has instances in flight. So there's a cost. I'd pay it the moment an agent starts touching anything with an invoice attached.

We've written up running LangGraph in production with durable execution at more length, and there's a longer argument for why checkpoints aren't durable execution if you want to know more of the details.

Run the Test You'd Rather Skip: Kill Your Agent Mid-Payment

Marc Duiker — Mon, 21 Sep 2026 08:37:29 +0000

Your agent calls a payment API. The process dies right after the charge succeeds and before anything gets written down. What happens when it restarts?

If the answer is "it retries from the top," you have a problem. "Retry the tool call" sounds harmless until the tool issues a refund, or a charge. Then you have billed a customer twice, and no amount of prompt engineering will explain that to them.

Most agent code handles the happy path and the visible error path, the one where the API returns a 500 and you try again. The path nobody tests is the invisible one, where the side effect succeeded and then the process disappeared before recording it. Deployments cause this. So do OOM kills, and spot instances do it on a schedule.

What durable execution does

Durable execution records every completed step in a durable log as the workflow runs. When the process comes back, the runtime replays the log: steps that already completed return their recorded results instantly, and execution resumes at the first step that never finished. Your charge happens once, no matter how many times the process dies.

With Dapr Workflow in Python:

import dapr.ext.workflow as wf

wfr = wf.WorkflowRuntime()

@wfr.activity(name="charge_card")
def charge_card(ctx: wf.WorkflowActivityContext, order):
    # the side effect lives in an activity
    return payments.charge(order["id"], order["amount"], idempotency_key=order["id"])

@wfr.activity(name="send_receipt")
def send_receipt(ctx: wf.WorkflowActivityContext, charge):
    return mailer.send(charge["customer"], charge["receipt_url"])

@wfr.workflow(name="order_flow")
def order_flow(ctx: wf.DaprWorkflowContext, order):
    charge = yield ctx.call_activity(charge_card, input=order)
    receipt = yield ctx.call_activity(send_receipt, input=charge)
    return receipt

if __name__ == "__main__":
    wfr.start()
    client = wf.DaprWorkflowClient()
    instance_id = client.schedule_new_workflow(
        order_flow, input={"id": "order-1", "amount": 4200}
    )
    client.wait_for_workflow_completion(instance_id)
    wfr.shutdown()

Now run the test you would rather skip:

Start the workflow.
Kill the process the moment charge_card completes. It has to be an unclean kill, so the app gets no chance to shut down gracefully: kill -9 <pid> on macOS or Linux, Stop-Process -Id <pid> -Force in PowerShell on Windows, or docker kill <container> if you are running it in a container.
Restart with only wfr.start() running, and watch the workflow resume at send_receipt, with the charge result restored from history instead of re-executed.

Durable execution needs idempotent tools

One caveat, and it is the reason step 2 is interesting. The replay only skips a step once that step's completion has been written down. If the process dies in the gap between the charge returning and the completion being recorded, the activity runs again on recovery. So if your payment provider cannot tell two identical requests apart, pass an idempotency key too. Idempotency means a request can be repeated safely without repeating its effect, and that is the half durable execution cannot do for you.

If you want to go deeper, read this post about The Tiniest Durable Agent, and there is a free, self-paced Dapr Workflow course at Dapr University that covers chaining, fan-out/fan-in, durable timers, and human-in-the-loop in depth.

For a developer the benefit is code you don't have to write. The workflow above reads top to bottom like ordinary Python, and the state machine, checkpoint table and "did I already charge this card?" column that would normally sit behind it are abstracted away, because the runtime manages that state instead. Recovery stops being something you design per tool call. When a run does go wrong, the history shows which step executed and what it returned, so you debug from a record rather than a guess.

So: fail the action at the worst possible moment, then check whether your system recovered and whether you can prove what it did. It takes about ten minutes, and you would rather learn the answer now than from the customer you charged twice.

10 LangGraph Alternatives for Building Agentic Workflows in 2026

Marc Duiker — Sat, 08 Aug 2026 11:00:00 +0000

LangGraph is a good answer when your agent naturally looks like a graph. It gets harder to defend when your team does not want that graph to become the shape of the whole application.

None of the alternatives below are LangGraph clones. Each one asks you to think about agents differently.

If you want a smaller abstraction

The OpenAI Agents SDK gives you agents, tools, handoffs, guardrails, sessions, and tracing, without asking you to draw an explicit graph for every application.
Pydantic AI is worth a look if you write typed Python services. You get structured outputs, validation, and dependency injection.
Strands Agents runs a model-driven loop with tools and keeps the developer experience short.

If you want multi-agent concepts first

CrewAI models role-based agents, tasks, and crews. When collaboration is the main thing you are describing, that can read better than wiring nodes together.
Microsoft Agent Framework deserves a place on the shortlist if your team is Microsoft-oriented and you are building agents and multi-agent workflows across .NET and Python.
Google ADK supports modular agents, tools, and composition, and it lines up with Google's models and cloud.

If you want distributed-systems primitives

Dapr Agents puts agent development on top of Dapr's state, messaging, service invocation, and workflow foundation.
Restate is not only an agent framework. It is a durable application runtime that can wrap your agent logic and your SDKs. Look at it when resilient execution matters more to you than taking on another high-level agent abstraction.

If you want long-horizon or provider-specific tooling

Deep Agents aims at longer and more complex work through planning, sub-agents, and context-management patterns.
The Claude Agent SDK fits teams building tool-rich agents around Claude's ecosystem and its coding-agent patterns.

Framework choice is not runtime choice

Some of these change how you author the agent. Others change how the agent survives failure. The two decisions are related, but they are not the same decision, and it is easy to shop for one while thinking you are solving the other.

If you like LangGraph's programming model and what you actually need is production durability, you may not have to replace LangGraph at all. Diagrid Catalyst, for example, ships a LangGraph runner as part of its bring-your-own-framework approach, and other durable runtimes integrate with agent SDKs too.

My short version:

stay on LangGraph when explicit state and graph control are the strengths you want
pick CrewAI when roles and collaboration match the problem
pick the OpenAI Agents SDK or Pydantic AI when you want fewer orchestration concepts
pick an ecosystem-native framework when organizational alignment matters
add a durable execution layer when the real problem is recovery rather than authoring

Pick the one whose mental model your team can still explain a few months after the demo has become a production service. That test tells you more than how new the framework is.

Inngest or Diagrid Catalyst: From a TypeScript AI Workflow to an Enterprise Agent Platform

Marc Duiker — Fri, 07 Aug 2026 09:00:00 +0000

A five-person team usually picks the infrastructure it never has to think about, and that is the right call. It stops being the right call on the day the first enterprise customer asks where the agent runs and who is allowed to authorize its tools.

At prototype stage, Inngest is easy to understand. You write functions in TypeScript, Python, or Go, split durable work into steps, and let the platform deal with state, retries, flow control, and observability. Its current durable-agent model uses step-level memoization, so completed LLM calls and tool invocations are not repeated during a replay. It can also wait for events and invoke sub-agents.

That suits a web team stretching background jobs into longer AI workflows. The agent stays next to the application code, and there is not much infrastructure to run.

When the decision changes

The architecture conversation usually shifts in stages.

First you ship the feature. A TypeScript workflow calls an LLM, enriches some data, and writes a result. Developer experience decides the winner.

Then you have to survive failure. Tool calls need to resume at the right step. Inngest and Catalyst both handle that, through different execution models.

Then a second and a third agent stack show up. A Python team picks LangGraph, a .NET team picks Microsoft Agent Framework, and another group uses Google ADK. Catalyst's bring-your-own-framework model starts to matter here, because the platform standard can sit below those framework choices instead of competing with them.

Then you go through enterprise review. Customers ask for workload identity, deny-by-default access, private deployment, MCP governance, and evidence of what each agent did. That is where Catalyst's wider governance model can justify running a larger platform.

What I would compare

Retries and dashboards are the least interesting part of this comparison. I would look at:

which languages and agent frameworks are supported
what happens to completed steps and side effects during recovery
who operates the workers, the state, and the execution plane
how agent-to-tool access is authenticated and authorized
which self-hosted, private, or air-gapped topologies exist
how several teams share one platform without sharing permissions

Inngest can stay the best choice for developer-first durable functions, background jobs, and AI workflows, as long as its language and deployment model fit the team. Catalyst fits better once the requirement has grown from "make this function durable" into "run and govern many kinds of agents across the enterprise."

Both mistakes are common, and they are the same mistake pointed in opposite directions: buying the stage-four platform while you are still at stage one, and refusing to reopen a stage-one decision after the organization around it has changed. Let the architecture follow the risk you are actually carrying.

Official sources

Restate or Diagrid Catalyst for Durable Tool Calls?

Marc Duiker — Thu, 06 Aug 2026 08:00:00 +0000

"Retry the tool call" sounds harmless until the tool issues a refund. Then durability, identity, and evidence stop being three concerns owned by three different teams and turn into one incident.

Take a support system built out of several agents. A triage agent classifies the ticket. An account agent retrieves customer state. A policy agent decides whether a refund is allowed. A human approves the recommendation. An action agent calls the refund tool. Then the process loses its connection before the response is recorded.

Retrying is the easy part. What matters is whether the platform and your own code can work out what already happened, without sending a second refund.

Restate: durable code and journaled steps

Restate is a runtime that sits in front of your services and records execution in a journal. Its AI documentation shows LLM calls and tool executions wrapped as durable steps. After a crash, completed results are replayed from the journal and execution continues from the first step that never finished. Restate supports services, virtual objects, and workflows, and runs through Restate Cloud or a server you host yourself.

That is a good model if you want resilient functions and services with built-in state, reliable communication, and durable execution written into ordinary code. It also integrates with agent SDKs rather than asking you to adopt one agent framework.

Catalyst: durable agents plus a shared trust layer

Catalyst also persists workflow activity results and resumes failed executions. Around that it adds App ID-based workload identity, access policies, mTLS-backed communication, MCP governance, and a single operational surface for agents and applications. Diagrid positions verifiable execution, meaning history, lineage, signing, and attestation, as an additional trust layer.

Once money is involved, a security review will want more than proof that the run recovered:

Which workload identity invoked the refund tool?
Which policy authorized the call?
Can you show that the execution history was not altered?
Can you revoke access without editing the agent prompt?
Does the same control apply across cloud, on-premises, and isolated deployments?

Where each one fits

Restate is the better fit when what you need is a lightweight durable runtime for services, stateful entities, workflows, and agents, and you want to express resilience directly in normal application code.

Catalyst is the better fit when durable execution has to arrive together with organization-wide agent identity, policy-controlled tool access, verifiable evidence, and governance across several frameworks and deployment environments.

Neither one takes your application off the hook once the outcome is unknown. A payment or refund integration still needs idempotency keys, reconciliation, and explicit compensation logic. Durable execution preserves progress. It cannot tell you whether an external system committed a request when the response was lost.

So run the test you would rather skip. Fail the action at the worst possible moment, then check whether it recovered, whether you can prove who did what, and whether the same refund can go out twice.