DEV Community

Neeraj Sharma
Neeraj Sharma

Posted on AI-assisted

Your AI Agent Will Fail Halfway Through a Task. Here Is Durable Execution in 42 Lines of Python.

Durable execution for AI agents

Here is a bug every agent builder hits sooner or later. A support agent picks up a refund ticket. It finds the customer, refunds $40, then the email API times out. The run fails. Your retry logic starts the task again from the top, and the customer gets refunded twice.

The model did nothing wrong. The system around it did.

This week the industry spent a lot of attention on exactly this layer. Microsoft made its Azure Durable Task service available for LangGraph agents, aimed at workflow reliability, and LangChain shipped LangSmith Sandboxes, which run agent code in dedicated microVMs. The common thread: agents now do real work, so they need the same guarantees as any other production workflow.

The core idea is called durable execution, and you can see it in 42 lines of plain Python.

The idea in three rules

1. Every task gets a run id. The support ticket, the order, the lead. One id per piece of work.

2. Every step is checkpointed. After a step succeeds, save its result under (run id, step name). On a rerun, return the saved result instead of doing the step again.

3. Side effects carry an idempotency key. Anything that moves money, sends a message or writes to another system gets a key derived from the run id, and that key is passed to the external API too.

The code

import hashlib, json, sqlite3, sys

db = sqlite3.connect("runs.db")
db.execute("CREATE TABLE IF NOT EXISTS steps (run TEXT, name TEXT, result TEXT, PRIMARY KEY (run, name))")

def step(run_id: str, name: str, fn, *args):
    """Run fn once per (run_id, name). On a rerun, return the saved result instead."""
    row = db.execute("SELECT result FROM steps WHERE run=? AND name=?", (run_id, name)).fetchone()
    if row:
        print(f"  skip  {name} (done earlier)")
        return json.loads(row[0])
    result = fn(*args)
    db.execute("INSERT INTO steps VALUES (?, ?, ?)", (run_id, name, json.dumps(result)))
    db.commit()  # the checkpoint: this step will never run twice for this run
    print(f"  done  {name}")
    return result

# Tools the agent calls. In real life these hit a CRM, Stripe, an email API.
def find_customer(email):
    return {"id": "cus_42", "email": email, "plan": "pro"}

def issue_refund(customer, amount, idempotency_key):
    # Pass the key to the payment API too, so a retried HTTP call cannot refund twice.
    return {"refund_id": "re_" + hashlib.sha1(idempotency_key.encode()).hexdigest()[:8], "amount": amount}

def send_email(customer, refund):
    if "crash" in sys.argv:
        raise ConnectionError("email API timed out")  # fails after the money moved
    return {"sent_to": customer["email"], "refund_id": refund["refund_id"]}

def refund_agent(run_id: str, email: str, amount: int):
    customer = step(run_id, "find_customer", find_customer, email)
    refund = step(run_id, "issue_refund", issue_refund, customer, amount, f"{run_id}:refund")
    receipt = step(run_id, "send_email", send_email, customer, refund)
    return receipt

if __name__ == "__main__":
    run_id = "ticket-1093"  # one id per task, e.g. the support ticket the agent picked up
    try:
        print(refund_agent(run_id, "ana@example.com", 40))
    except ConnectionError as err:
        print(f"  fail  {err}")
Enter fullscreen mode Exit fullscreen mode

Run it once with a simulated crash, then twice more:

$ python durable.py crash
  done  find_customer
  done  issue_refund
  fail  email API timed out

$ python durable.py
  skip  find_customer (done earlier)
  skip  issue_refund (done earlier)
  done  send_email
{'sent_to': 'ana@example.com', 'refund_id': 're_463685a9'}

$ python durable.py
  skip  find_customer (done earlier)
  skip  issue_refund (done earlier)
  skip  send_email (done earlier)
{'sent_to': 'ana@example.com', 'refund_id': 're_463685a9'}
Enter fullscreen mode Exit fullscreen mode

The first run crashed after the money moved. The second run skipped the two finished steps and only sent the email. The third run did nothing new at all. The refund happened exactly once, and the refund id is the same every time because it comes from the run id.

What the small version teaches

  • The step function is the whole trick. Every tool call goes through one place that checks the journal first. Add timing, cost tracking or tracing there once and every agent gets it.
  • Checkpoint after the side effect, and still pass the key. There is a gap between "Stripe refunded" and "we saved that it refunded". If the process dies inside that gap, the step will run again. The idempotency key you pass to the payment API is what makes that second call harmless. Most payment and email providers support one.
  • Save the model's decisions as steps too. If step 2 is "ask the model which plan to offer", wrap that call in step as well. Otherwise a replay asks the model again, gets a different answer, and the rest of the run no longer matches what already happened.
  • Pick the run id carefully. Too broad and two different tasks share a journal. Too narrow, like a random id per attempt, and you get no protection at all. The ticket or order id is usually right.

Three things this toy skips

  • Two workers on the same run. Add a lease: a row that says "worker A owns ticket-1093 until 10:05", so a second worker waits instead of racing.
  • Steps that should time out. A step that hangs forever never fails, so it never gets retried. Give every tool call a timeout.
  • Long waits. If the agent needs a human approval that might take a day, you do not want a process sleeping that long. Save a "waiting" state and resume when the approval arrives.

When to use a framework

The pattern above is fine for a handful of steps. Once you need timers, human-in-the-loop waits, many workers or versioned workflows, use something built for it: Temporal, Azure Durable Task, Restate, Inngest, or LangGraph's own checkpointers. They all implement the same three rules, with more care than a 42-line script can.

The agents that hold up in production are not the ones with the cleverest prompts. They are the ones where a crash halfway through is boring. I wrote about the orchestration side of this in building an AI agent orchestrator with an MCP harness. If you want an agent built that survives real traffic, that is what we do at Axionry.

Sources

Top comments (1)

Collapse
 
osamamalik profile image
Osama Malik •

The checkpoint in step() is written only after fn returns. A crash in the gap after issue_refund has already called the payment API, and before the INSERT, still reruns that step. The idempotency key on the refund covers that retry. send_email does not take a key, so the same gap after a successful send produces a second email. There is also no lock around the SELECT, the call, and the INSERT. Two workers with the same run id can both see no row and both run the side effect. Writing an in progress row (or using BEGIN IMMEDIATE) before calling fn closes that race. The primary key only rejects the second INSERT, which is already too late for the side effect.