DEV Community

Cover image for A Lang Graph Timeout Doesn't Undo Your Charge: The Retry Trap in 1.2
Rozera_ai
Rozera_ai

Posted on Edited on

A Lang Graph Timeout Doesn't Undo Your Charge: The Retry Trap in 1.2

What happens to a payment that succeeds 200 milliseconds after your timeout fires?

Your graph state says nothing happened. Your payment provider says it charged the card. And LangGraph, following its defaults, is about to charge it again.

Per-node timeouts and node-level error handlers arrived in langgraph 1.2 (they require langgraph>=1.2), and most of the attention went to the features themselves. The more interesting part is how a timeout interacts with retries and side effects, because the combination is easy to misconfigure.

The code in this article is illustrative. payments, State and fetch_batches are stand-ins for your own provider client, state schema and data source.

The scenario

A checkout agent has a node that charges the customer. Providers occasionally stall, so you do the responsible thing and cap the node:


from langgraph.types import RetryPolicy, TimeoutPolicy

async def charge_customer(state: State) -> dict:
    charge = await payments.create_charge(
        customer_id=state["customer_id"],
        amount=state["amount"],
    )
    return {"charge_id": charge.id}

builder.add_node(
    "charge_customer",
    charge_customer,
    timeout=TimeoutPolicy(run_timeout=10),
    retry_policy=RetryPolicy(max_attempts=3),
)

Enter fullscreen mode Exit fullscreen mode

Reasonable on its face. Here is what the docs say happens when attempt one takes 11 seconds:

LangGraph raises NodeTimeoutError and clears any writes from that attempt.
NodeTimeoutError is retryable by default, so the retry policy re-runs the node from the top.
If your provider committed the first charge at second 10.2, attempt two charges again.
Why it works this way

LangGraph gives you three mechanisms that compose in a fixed order. An attempt raises an exception, and a timeout is just one more exception. The retry policy decides whether to go again. Only after retries are exhausted does the error_handler run.

State stays correct because checkpoints are written at super-step boundaries, not mid-function, and writes from a failed attempt are discarded. That guarantee covers the graph. It does not cover the outside world. The fault-tolerance docs are explicit that retries give at-least-once semantics for node execution, and that payments, emails and external writes sit outside the checkpoint boundary.

So the timeout does what it says: it clears graph state. It can't clear a charge.

Retries also multiply your worst case. The timeout clock resets on every attempt, so three attempts at a 10-second run_timeout can hold the node for about 30 seconds plus backoff. The defaults are a 0.5 s initial interval, a factor of 2, and jitter. That's my arithmetic, not a documented number, but it's worth checking against your own latency budget.

Making the node safe to repeat

The fix isn't a smarter timeout. It's making repeat execution harmless, and the docs give a specific rule for that: the idempotency key must be derivable from state that is already durable. Generating a UUID inside the node fails, because if the process dies before the node returns, the key was never checkpointed and the retry mints a new one.


from langgraph.errors import NodeError, NodeTimeoutError
from langgraph.types import Command, RetryPolicy, TimeoutPolicy

async def charge_customer(state: State) -> dict:
    key = f"charge:{state['order_id']}"          # stable across retries and resumes
    charge = await payments.create_charge(
        customer_id=state["customer_id"],
        amount=state["amount"],
        idempotency_key=key,
    )
    return {"charge_id": charge.id}

def route_charge_failure(state: State, error: NodeError) -> Command:
    timed_out = isinstance(error.error, NodeTimeoutError)
    return Command(
        update={"charge_outcome": "unknown" if timed_out else "failed"},
        goto="reconcile_payment",
    )

async def reconcile_payment(state: State) -> dict:
    # A missing acknowledgement is "unknown", not "didn't happen".
    status = await payments.lookup(f"charge:{state['order_id']}")
    return {"charge_outcome": status}

builder.add_node(
    "charge_customer",
    charge_customer,
    timeout=TimeoutPolicy(run_timeout=15),
    retry_policy=RetryPolicy(max_attempts=3),
    error_handler=route_charge_failure,
)
builder.add_node("reconcile_payment", reconcile_payment)

Enter fullscreen mode Exit fullscreen mode

Two details matter. The key comes from order_id, which already lives in state. And the error handler doesn't guess. It routes to a node whose only job is to ask the provider what actually happened, which matches the docs' advice to treat missing local state as unknown and query the provider before compensating.

The handler gets NodeError injected by type annotation, so you can tell a timeout from a declined card without parsing strings.

The idle timeout has a blind spot

TimeoutPolicy has two limits. run_timeout is a hard wall-clock cap that never refreshes. idle_timeout resets whenever the node shows progress, and under the default refresh_on="auto" the list of things that count as progress is long: state writes, streamed chunks, child-task scheduling, stream-writer calls, and any LangChain callback event from the node or its descendants, including LLM tokens and tool calls.

That's the right default for a node wrapping an agent loop. It also means a node that is spinning, say retrying a flaky tool in a loop while a sub-chain keeps emitting callbacks, can look healthy to the idle clock indefinitely. If you want "idle" to mean "my code said it's alive", narrow it:


from langgraph.runtime import Runtime

async def ingest(state: State, runtime: Runtime) -> dict:
    async for batch in fetch_batches():
        await process(batch)
        runtime.heartbeat()          # the only thing that resets the clock
    return {"status": "done"}

builder.add_node(
    "ingest",
    ingest,
    timeout=TimeoutPolicy(idle_timeout=30, refresh_on="heartbeat"),
)
Enter fullscreen mode Exit fullscreen mode

runtime.heartbeat() is a no-op outside an idle-timed attempt, so it's safe to leave in unconditionally. If you fan out with Send, you can also override the timeout per push, which is handy for tightening it on individual items.

Trade-offs and edges
Async only. A sync node with timeout= is rejected at compile time, because sync Python can't be safely cancelled in-process. The docs suggest asyncio.to_thread for blocking I/O. Keep in mind, as general Python behaviour rather than anything LangGraph-specific, that cancelling the awaiting coroutine doesn't stop the thread. The blocking call can keep running after the timeout has "fired".
Interrupts bypass all of this. interrupt() raises through GraphBubbleUp, so it skips both retry policies and error handlers. Human-in-the-loop pauses won't trigger your compensation logic.
One handler per node, and if the handler raises, the exception bubbles as if there were no handler.
set_node_defaults isn't inherited by subgraphs. Each graph needs its own, so a timeout you set at the top level won't protect a nested graph.
Idempotency depends on the provider. The key only helps if the service honours it. Check before relying on it, and for effects that can't be made idempotent, the docs suggest moving them out of retryable nodes.
My take

The feature everyone expects from "timeouts" is a stuck agent that gets unstuck. What 1.2 actually ships is a more honest model of failure: the graph is transactional at the super-step level, the world isn't, and the gap between them is your problem. The docs say so plainly, which I appreciate.

I used to file a node timeout under reliability features. I now think of it as a switch that turns slow failures into retries, and retries are only reliable if the work inside them is safe to repeat. Add a timeout to a node with side effects and you've made a decision about duplicates, whether or not you wrote it down.

So the real review question for any node you wrap with timeout= is whether running it twice is harmless. If the answer is "I'd have to check", the timeout isn't ready.

Which of your nodes would do real damage if they ran twice, and do they have a key that survives a crash?

References
LangGraph docs, Fault tolerance (retries, timeouts, error handlers, idempotency, graceful shutdown)
LangGraph docs, Graph API nodes (node timeouts, set_node_defaults)
LangGraph Python changelog (v1.2 release notes)
API reference, NodeTimeoutError

Top comments (4)

Collapse
 
mp3sss31 profile image
mp3sss •

The core trap is that the timeout only reflects the client's wait, not the provider's outcome: NodeTimeoutError clears the in-memory writes and retries, but a charge that committed just after the cutoff still exists on the provider side, so the second attempt duplicates the side effect. The safe fix is to make the charged action idempotent — a stable idempotency key per checkout attempt, plus a read-back query before retrying — rather than letting the retry policy re-run the node blindly. This generalizes to any stateful node: classify side effects that must not auto-retry, and separate 'transient call failure' from 'unknown commit state' the way a payment webhook would.

Collapse
 
rozera_ai profile image
Rozera_ai •

Exactly.. and the webhook comparison is a good one. “Unknown” is a legitimate state, not a failure, so the node shouldn’t pretend to know.

Two refinements. I’d scope the idempotency key to the business intent (the order), not each attempt. If the key changes per attempt, retries stop being deduplicated. And with a working key, a read-back before every retry is mostly redundant. It earns its place after retries are exhausted, where the outcome is truly unknown. A webhook can settle that case asynchronously.

Do you route “unknown” to a reconcile step...or wait for the provider’s webhook?

Collapse
 
zahurul profile image
Zahurul Islam •

This is the trap I wish more framework docs led with. A pattern that has held up for me: split every side-effecting node into "decide" and "commit". The decide node can retry freely because it only writes an intent (with an idempotency key) into state. The commit node is never retried blindly. On timeout it goes to a "reconcile" node that asks the provider whether that key exists. It's one extra node, and it makes the retry policy boring again.

Collapse
 
rozera_ai profile image
Rozera_ai •

Cleanest version of this I’ve seen💯 . The decide node checkpoints the intent, so the idempotency key survives a crash by construction... Its basically an outbox with graph state as the outbox
One edge: in reconcile’s “key not found” branch&& re-running commit is only safe if the provider still honours that key. Some expire them, so I’d store a timestamp with the intent.
On a crash between the provider call and the checkpoint does resume always route through reconcile first?