Run the Test You'd Rather Skip: Kill Your Agent Mid-Payment

Your agent calls a payment API. The process dies right after the charge succeeds and before anything gets written down. What happens when it restarts?

If the answer is "it retries from the top," you have a problem. "Retry the tool call" sounds harmless until the tool issues a refund, or a charge. Then you have billed a customer twice, and no amount of prompt engineering will explain that to them.

Most agent code handles the happy path and the visible error path, the one where the API returns a 500 and you try again. The path nobody tests is the invisible one, where the side effect succeeded and then the process disappeared before recording it. Deployments cause this. So do OOM kills, and spot instances do it on a schedule.

What durable execution does

Durable execution records every completed step in a durable log as the workflow runs. When the process comes back, the runtime replays the log: steps that already completed return their recorded results instantly, and execution resumes at the first step that never finished. Your charge happens once, no matter how many times the process dies.

With Dapr Workflow in Python:

import dapr.ext.workflow as wf

wfr = wf.WorkflowRuntime()

@wfr.activity(name="charge_card")
def charge_card(ctx: wf.WorkflowActivityContext, order):
    # the side effect lives in an activity
    return payments.charge(order["id"], order["amount"], idempotency_key=order["id"])

@wfr.activity(name="send_receipt")
def send_receipt(ctx: wf.WorkflowActivityContext, charge):
    return mailer.send(charge["customer"], charge["receipt_url"])

@wfr.workflow(name="order_flow")
def order_flow(ctx: wf.DaprWorkflowContext, order):
    charge = yield ctx.call_activity(charge_card, input=order)
    receipt = yield ctx.call_activity(send_receipt, input=charge)
    return receipt

if __name__ == "__main__":
    wfr.start()
    client = wf.DaprWorkflowClient()
    instance_id = client.schedule_new_workflow(
        order_flow, input={"id": "order-1", "amount": 4200}
    )
    client.wait_for_workflow_completion(instance_id)
    wfr.shutdown()

Now run the test you would rather skip:

1. Start the workflow.
2. Kill the process the moment charge_card completes. It has to be an unclean kill, so the app gets no chance to shut down gracefully: kill -9 <pid> on macOS or Linux, Stop-Process -Id <pid> -Force in PowerShell on Windows, or docker kill <container> if you are running it in a container.
3. Restart with only wfr.start() running, and watch the workflow resume at send_receipt, with the charge result restored from history instead of re-executed.

Durable execution needs idempotent tools

One caveat, and it is the reason step 2 is interesting. The replay only skips a step once that step's completion has been written down. If the process dies in the gap between the charge returning and the completion being recorded, the activity runs again on recovery. So if your payment provider cannot tell two identical requests apart, pass an idempotency key too. Idempotency means a request can be repeated safely without repeating its effect, and that is the half durable execution cannot do for you.

If you want to go deeper, read this post about The Tiniest Durable Agent, and there is a free, self-paced Dapr Workflow course at Dapr University that covers chaining, fan-out/fan-in, durable timers, and human-in-the-loop in depth.

For a developer the benefit is code you don't have to write. The workflow above reads top to bottom like ordinary Python, and the state machine, checkpoint table and "did I already charge this card?" column that would normally sit behind it are abstracted away, because the runtime manages that state instead. Recovery stops being something you design per tool call. When a run does go wrong, the history shows which step executed and what it returned, so you debug from a record rather than a guess.

So: fail the action at the worst possible moment, then check whether your system recovered and whether you can prove what it did. It takes about ten minutes, and you would rather learn the answer now than from the customer you charged twice.