DEV Community: TechAI Explained

What LangGraph Actually Builds: Runnables, Channels, Bubble-Up, Agents and Deep Agents

TechAI Explained — Sun, 11 Oct 2026 15:17:04 +0000

I spent the weekend digging through LangGraph's source code, trying to understand what actually happens beneath the abstractions. These are my notes, the experiments I ran, and the mental model I ended up with.

The thing that surprised me most is how small the core idea is. A graph, a ReAct agent and a Deep Agent are not three different machines. They are one machine (Pregel) fed three different amounts of configuration.

Everything below was run offline: no network, no live model, a scripted fake chat model. Versions are pinned to langgraph 1.2.12, langchain-core 1.6.6, langchain 1.4.3, deepagents 0.7.20; internals move between releases, so treat the class names as a snapshot. The code is in the samples repo: langgraph_primitives_demo.py produced every printed output in sections 1–6, and structured_cases.py covers section 5c.

Side notes before we start: Python pieces and C# equivalents

If you come from C#, a few Python/LangGraph words show up immediately. They're plain Python or plain LangGraph.

You'll see	What it is	Closest C# idea
`TypedDict`	A dictionary whose keys and value types are declared for type checkers; at run time it's an ordinary `dict`.	A DTO / `record`, except it stays a dictionary.
`Annotated[list, operator.add]`	A type with attached metadata: "this is a `list`, and here's extra info." Python itself ignores the extra part; LangGraph reads it.	A type plus an attribute, e.g. `[Reducer(Add)] List<string> visited`.
`operator.add`	A plain function that does `a + b` (for lists, concatenation).	A method group such as `(a, b) => a + b`, passed as a `Func<T,T,T>`.
`lambda x: x + 1`	An anonymous function. That's what I meant by "unnamed" before: not a keyword of this project, just Python's lambda.	A C# lambda `x => x + 1`; a `Func<int,int>` delegate.
a function passed as a value	Functions are first-class objects, so `add_node("a", a)` just hands over the function.	Passing a delegate or method group.
`invoke`, `stream`	Methods on a runnable.	`Execute` / `IAsyncEnumerable<T>` on an interface.

Two LangGraph words are worth fixing in your head now.

State is the dictionary that flows through the graph. Every node receives it and returns a partial update to it, not a whole new state. If you know C#, think of an immutable record plus with { ... } expressions, applied by the engine for you.

Channel is one named slot of that state, and it owns the rule for how new values are merged in. A state key total becomes a channel called total. The rule matters when several nodes write in the same round: should the new value replace the old one, or be combined with it? In C# terms a channel is roughly a field paired with its own merge function, like Func<T, T, T> merge.

The name is misleading if you know C#. It is not System.Threading.Channels: nothing is queued for a consumer to await, and nothing blocks. It is not Rx's IObservable: nothing pushes values to subscribers. It is not a TPL Dataflow block either, though that is the closest family, because "a node runs when something it depends on was updated" is the same idea as dataflow triggering. The nearest literal picture is a versioned cell in a shared state object: a field whose Update(IEnumerable<T> writes) method decides how writes combine, plus a version number that goes up when it changes. Nodes are woken by comparing those versions, not by receiving messages. (Two exceptions behave a bit more like queues: Topic, used for Send, collects several values; EphemeralValue holds a value for one round only.)

Reducer is that merge function: reducer(old, new) -> combined. You attach one with Annotated. If you've used LINQ's Aggregate, it's the same shape: fold each new value into a running one. operator.add as a reducer on a list means "append the new items to the old items".

With those in hand, here is the smallest interesting graph.

1. The only contract: `Runnable`

A Runnable is anything with invoke, ainvoke, stream, astream, batch and with_config. That's it. In C# you'd call this an interface that everything implements, an IRunnable<TIn, TOut>. The | operator composes two of them (Python lets a class overload |, like a C# operator |):

chain = RunnableLambda(lambda x: x + 1) | RunnableLambda(lambda x: x * 10)
type(chain).__name__   # RunnableSequence
chain.invoke(1)        # 20

Why this matters: a compiled LangGraph is also a Runnable. Its MRO starts CompiledStateGraph → Pregel → PregelProtocol. (MRO is the method resolution order, Python's inheritance chain: CompiledStateGraph : Pregel : PregelProtocol.) So a whole graph can sit inside a chain, or inside a node of another graph. That is how subagents work later.

2. What `compile()` produces

Take the smallest interesting graph: two nodes with a state that has an overwrite field and an append field. A node is just a function that takes the state and returns a partial update:

import operator
from typing import Annotated, TypedDict
from langgraph.graph import StateGraph, START, END

class S(TypedDict):
    total: int                              # bare key: one write per superstep
    visited: Annotated[list, operator.add]  # reducer: appends

def a(s):                               # node "a": doubles n, logs "a"
    return {"total": s["total"] * 2, "visited": ["a"]}

def b(s):                               # node "b": adds 1 to n, logs "b"
    return {"total": s["total"] + 1, "visited": ["b"]}

g = StateGraph(S)
g.add_node("a", a)
g.add_node("b", b)
g.add_edge(START, "a")
g.add_edge("a", "b")
g.add_edge("b", END)
app = g.compile()
app.invoke({"total": 10, "visited": []})        # {'total': 21, 'visited': ['a', 'b']}

StateGraph is only a builder (like a C# ServiceCollection before BuildServiceProvider()). compile() lowers it into primitives. Here is what the demo prints for the compiled object:

Those lines are not part of the graph code above. They come from a small inspection helper I wrote (it's show() in the demo script). It reads the public attributes of the compiled object, app.channels and app.nodes:

# `app` is the compiled graph from above (a Pregel object)
print("channels:", {name: type(ch).__name__ for name, ch in app.channels.items()
                    if not name.startswith("branch:")})    # hide the per-edge channels for brevity
for name, node in app.nodes.items():
    if name == "__start__":
        continue                                            # internal input node, skipped for brevity
    print(f"node {name!r}: triggers={node.triggers} bound={type(node.bound).__name__}"
          f" writers={len(node.writers)}")

which prints:

channels: {'total': 'LastValue', 'visited': 'BinaryOperatorAggregate',
           '__start__': 'EphemeralValue', '__pregel_tasks': 'Topic'}
node 'a': triggers=['branch:to:a'] bound=RunnableCallable writers=2
node 'b': triggers=['branch:to:b'] bound=RunnableCallable writers=1

The lines starting with node describe PregelNode objects, so here is what one is. This is the class from langgraph/pregel/_read.py, trimmed to the fields that matter here (I left out error-handler and subgraph fields, and the docstrings are mine, condensed from the source's):

class PregelNode:
    # A container, not a Runnable. The engine uses it to build a runnable task each time the node fires.

    channels: str | list[str]      # which channels to READ as input. For node "a": ['total', 'visited'],
                                   # so your function receives a dict with those two keys (the `s` in `def a(s)`)
    triggers: list[str]            # which channels WAKE the node: if any is written, the node runs next step.
                                   # For node "a": ['branch:to:a']
    bound: Runnable                # the actual logic: a Runnable wrapping your function, called with the input
    writers: list[Runnable]        # run AFTER `bound`; each takes its output and writes it to channels
                                   # (one stores your return value in state, one wakes the next node)
    mapper: Callable | None        # optional transform of the input before `bound` (None for node "a")

    retry_policy: Sequence[RetryPolicy] | None   # what happens if the node raises (None here)
    cache_policy: CachePolicy | None             # reuse results for identical input (None here)
    timeout: TimeoutPolicy | None                # per-invocation limit (async nodes only)
    tags: Sequence[str] | None                   # tracing labels
    metadata: Mapping[str, Any] | None           # tracing metadata

Note the split between channels (what it reads) and triggers (what wakes it). They are separate on purpose: a node can read many channels but be woken by just one. In C# terms a PregelNode is a record of "my inputs, my wake-up conditions, my handler, my post-processing", much like a message handler registration with a filter, except the messages are channel updates.

How to read the printout, line by line:

'total': 'LastValue': the total field became a channel of type LastValue. It holds one value, and a write replaces it. That is what the plain total: int in S meant.
'visited': 'BinaryOperatorAggregate': the visited field became a channel that combines old and new with your function, here operator.add. That is what Annotated[list, operator.add] meant, and why the list grows.
'__start__': 'EphemeralValue': an internal channel that carries your invoke(...) input into the graph. EphemeralValue means it only lives for one round.
'__pregel_tasks': 'Topic': an internal queue-like channel. It is where Send packets go (section 5c); our graph doesn't use it, but every graph has it.
node 'a': triggers=['branch:to:a']: node a runs when the channel branch:to:a is updated. That channel is what the edge START → a turned into, so there is no edge object, only "wake a when this channel changes".
bound=RunnableCallable: the thing that actually runs. It is a Runnable wrapper around your plain function a.
writers=2 vs writers=1: after a runs, two things write: one stores its returned update into total/visited, and one writes to branch:to:b to wake node b. b is last, so it only has the first writer; its edge goes to END, which wakes nothing.

Note the channel list has no branch:to:a or branch:to:b, because my filter hid them. They exist, one per edge, and the nodes' triggers point at them.

Here is the same transformation as a picture. The first diagram is what you drew with add_edge; the second is what compile() actually builds (I added the __start__ input channel and the two writers per node to the picture, as the printout showed them):

What you wrote (builder):

flowchart LR
   S([START]) --> a --> b --> E([END])

What compile() built (runtime):

flowchart TD
   I["__start__ channel<br/>(your input)"] -->|wakes| CA["channel branch:to:a"]
   CA -->|wakes| NA["node a<br/>reads: total, visited<br/>bound = a(s)<br/>writer1 → total, visited<br/>writer2 → channel branch:to:b"]
   NA -->|wakes| NB["node b<br/>reads: total, visited<br/>bound = b(s)<br/>writer1 → total, visited<br/>(no writer2: its edge goes to END)"]

The arrows between nodes are gone in the second diagram. What remains are boxes (nodes) and named mailboxes (channels): "a" does not know "b" exists, it just drops a message in branch:to:b.

So the mental model is:

Channels hold state (see the side notes above). Each state key becomes a channel whose type is its merge rule: LastValue keeps the latest value across steps but accepts at most one write per superstep (two parallel writers raise InvalidUpdateError, shown in 5c); BinaryOperatorAggregate applies your reducer (operator.add), which is why visited appends instead of replacing. Extra channels carry control flow: __start__ holds the input, and an edge a → b becomes a channel named branch:to:b. The double underscores are only a naming convention for internal channels (Python's __name__ style, not a keyword).
Nodes are PregelNodes. Each has triggers (which channels wake it), channels (which it reads), bound (the actual Runnable, here a wrapper around your function) and writers (what it writes after running).
Edges disappear. There is no edge object at run time. a → b is "a's writer writes to branch:to:b, and b triggers on branch:to:b". get_graph() reconstructs the pretty edge list for drawing, but the engine never walks it.

Conditional edges and Send use the same mechanism: a router Runnable runs as an extra writer and writes to whichever branch:to:X channels it chooses (or sends a task with its own input for map-reduce).

To see it, add a router to the same graph: a now picks b or c with g.add_conditional_edges("a", route, ["b", "c"]), where route(s) returns "b" if s["total"] > 5 else "c". As code (it reuses S, a and b from above, and adds node c and the router):

def c(s):                                # node "c": the alternative path, resets total
    return {"total": 0, "visited": ["c"]}

def route(s):                            # the router: returns the NAME of the next node
    return "b" if s["total"] > 5 else "c"

g = StateGraph(S)
g.add_node("a", a)
g.add_node("b", b)
g.add_node("c", c)
g.add_edge(START, "a")
g.add_conditional_edges("a", route, ["b", "c"])   # replaces add_edge("a", "b")
g.add_edge("b", END)
g.add_edge("c", END)
app = g.compile()
app.invoke({"total": 10, "visited": []})   # {'total': 21, 'visited': ['a', 'b']}  (a: 20 > 5 -> b)
app.invoke({"total": 1,  "visited": []})   # {'total': 0,  'visited': ['a', 'c']}  (a: 2 -> c)

I compiled and ran this, and inspected it the same way:

channels: total=LastValue, visited=BinaryOperatorAggregate, __start__, __pregel_tasks=Topic,
          branch:to:a, branch:to:b, branch:to:c   (the last three are EphemeralValue)
node a: triggers=['branch:to:a']  writers=[state write, _route]
node b: triggers=['branch:to:b']  writers=[state write]
node c: triggers=['branch:to:c']  writers=[state write]

What you wrote:

flowchart LR
   S([START]) --> a
   a --> b --> E1([END])
   a --> c --> E2([END])

What compile() built:

flowchart TD
   NA["node a<br/>writer1: state write → total, visited<br/>writer2: _route(s) ← your router function"]
   NA -->|"router returns 'b'"| CB["writes branch:to:b"] -->|wakes| NB[node b]
   NA -->|"router returns 'c'"| CC["writes branch:to:c"] -->|wakes| NC[node c]

The difference from the linear case is only the second writer of a. Instead of a fixed write to branch:to:b, it is _route, which calls your function and writes to whichever channel the answer names. The other channel is never written, so that node simply doesn't run. With total starting at 10, a makes it 20, the router picks b, and invoke returns {'total': 21, 'visited': ['a', 'b']}; c never fires. Start at 1 and the router picks c instead. (_route is the internal writer's name as I saw it in this version; treat it as an implementation detail.)

3. Pregel: the superstep loop

Pregel is a model for computing over graphs, described in a 2010 Google paper ("Pregel: A System for Large-Scale Graph Processing", SIGMOD 2010). It builds on the older "bulk synchronous parallel" idea. Open-source systems such as Apache Giraph, Spark GraphX and Flink Gelly offer the same vertex-centric model. LangGraph borrows the idea to run your graph (it also names its compiled graph class Pregel, and that is why the module is called pregel). In one sentence: work proceeds in rounds, where every ready node runs in parallel on the same snapshot of the state, and only when all of them finish are their results merged and the next round decided. In C# terms, picture a round of Task.WhenAll(...) over the ready nodes, where every task reads the same snapshot and a barrier merges their results before the next round. (LangGraph's version is an adaptation: Google's Pregel passes messages between graph vertices, while LangGraph's "vertices" are your nodes and the messages are channel writes.) If you work in .NET, you may already have met this algorithm: Microsoft Agent Framework's workflow engine, in both its C# and Python versions, uses what its docs call "a modified Pregel execution model — a Bulk Synchronous Parallel (BSP) approach with superstep-based processing", so its executors run in the same supersteps. See Workflow Builder & Execution, "Execution Model: Supersteps" for details. In langgraph/pregel/_loop.py each round is:

tick: check the stop conditions, then prepare_next_tasks finds every node whose trigger channel was updated since it last ran.
Run those tasks (in parallel if there is more than one). A task reads the channel values from the start of the round and returns writes; nothing it writes is visible to its siblings.
after_tick: apply_writes merges all the writes into the channels through each channel's rule, bumps versions, emits stream output, and (with a checkpointer) saves a checkpoint.

Repeat until no node is triggered. Running the two-node graph with invoke gives {'total': 21, 'visited': ['a', 'b']}, and stream_mode="updates" shows one superstep per node:

[{'a': {'total': 20, 'visited': ['a']}}, {'b': {'total': 21, 'visited': ['b']}}]

The "write then apply at the barrier" split is the single most important idea. It's why parallel branches are safe (they can't see each other), why reducers exist (two branches writing the same key in the same round need an Annotated reducer, otherwise the graph fails), and why a checkpoint between rounds is a consistent snapshot of ordinary channels (the Deep Agents DeltaChannel is the exception, see 5b).

Send: when one node must become many tasks

A normal edge says "after a, run b", once, with the shared state as input. Sometimes you don't know how many tasks you need until runtime: the model asked for three tools, or you have N documents to process. Send(node, arg) covers that. It is a small packet meaning "run this node once, with this private input", and a router can return a whole list of them:

import operator
from typing import Annotated, TypedDict
from langgraph.graph import START, END, StateGraph
from langgraph.types import Send

class S(TypedDict):
    items: list
    out: Annotated[list, operator.add]       # reducer, so parallel writes are appended

def fan(s):                                  # router: one Send per item
    return [Send("work", {"x": i}) for i in s["items"]]

def work(a):                                 # 'a' is the Send's argument, NOT the graph state
    return {"out": [a["x"] * 10]}

g = StateGraph(S)
g.add_node("work", work)
g.add_conditional_edges(START, fan, ["work"])
g.add_edge("work", END)
app = g.compile()

print(app.invoke({"items": [1, 2, 3], "out": []}))

Streaming with stream_mode="debug" and printing each task event shows what the loop scheduled (I ran this):

task work {'x': 1} step 1
task work {'x': 2} step 1
task work {'x': 3} step 1
{'items': [1, 2, 3], 'out': [10, 20, 30]}

One node, three tasks, all in the same superstep, each with its own input. Who does what:

You (or create_agent) write the router. It only returns the Send objects; it doesn't run anything.
The Pregel loop owns the tasks. The router's return value is written to the internal __pregel_tasks channel (the Topic from section 2). At the next barrier the loop reads that channel and, for each packet, builds a task for the named node with the packet's argument as its input (internally prepare_push_task_send, as opposed to the ordinary "triggered by a channel" tasks). The runner then executes those tasks together, like any other superstep.
When: the router runs at the end of one superstep; the Send tasks run in the very next one. They then write their results through the normal channels, which is why the example needs the operator.add reducer: three tasks wrote out in the same round.

The C# picture is await Task.WhenAll(items.Select(i => Work(i))), except the list is decided by a router in the middle of the graph, and each of those calls is a first-class task that gets its own checkpoint writes. This is exactly what create_agent does for parallel tool calls, as the next section shows.

In one sentence: the Pregel loop runs your graph in supersteps, where a superstep is one round in which every ready node runs on the same snapshot of state and all their writes are merged at a barrier before the next round starts.

4. Example: an AI agent is just a graph

Before the harder parts (bubble-up, reducers, durability), here is the payoff of everything so far: an AI agent is expressed as a graph. It sounds like a different species, but create_agent(model, tools=[...]) returns a CompiledStateGraph, the same kind of object as app in section 2. So we can read how it is built, then run it, then look at the graph it produces.

4.1 The code of `create_agent`

create_agent is not magic; it is a function that makes a StateGraph using the same builder calls as section 2 and compiles it. Below is the relevant core of langchain/agents/factory.py (langchain 1.4.3), condensed by me to the path with no middleware and no structured output; the real file is about 2,000 lines, and bodies marked ... are omitted. The node and router functions are shown in full, simplified from model_node, _make_model_to_tools_edge and _make_tools_to_model_edge:

def create_agent(model, tools, ...):
    graph = StateGraph(state_schema=AgentState, ...)   # 'messages' etc. become channels

    # ---- node 1: "model" -- a plain function node ----
    def model_node(state, runtime):
        messages = state["messages"]                    # read the channel
        if system_message:
            messages = [system_message, *messages]
        ai_message = model.bind_tools(tools).invoke(messages)   # call the chat model
        return {"messages": [ai_message]}               # write: append to 'messages'
    graph.add_node("model", RunnableCallable(model_node))

    # ---- node 2: "tools" -- a ToolNode, itself a Runnable ----
    tool_node = ToolNode(tools)    # reads the last AIMessage's tool_calls, runs each tool,
    graph.add_node("tools", tool_node)                  # and writes one ToolMessage per call

    graph.add_edge(START, "model")                      # entry point

    # ---- router after "model": tool calls -> "tools", otherwise finish ----
    def model_to_tools(state):
        last_ai = last_ai_message(state["messages"])
        if not last_ai.tool_calls:                      # classic exit condition of an agent loop
            return END
        pending = [c for c in last_ai.tool_calls if c["id"] not in answered_ids(state)]
        return [Send("tools", [call]) for call in pending]   # one task per pending tool call
    graph.add_conditional_edges("model", model_to_tools, ["tools", END])

    # ---- router after "tools": loop back, unless a return_direct tool ran ----
    def tools_to_model(state):
        if all_executed_tools_are_return_direct(state):
            return END
        return "model"                                  # let the model read the results
    graph.add_conditional_edges("tools", tools_to_model, ["model"])

    return graph.compile(checkpointer=checkpointer, ...)   # same compile() as section 2

Three things are worth noticing, because they connect back to what we already saw:

Nodes are ordinary functions of state. model_node reads the messages channel and returns a partial update that the reducer appends. That is exactly what node a did in section 2. ToolNode is just another Runnable.
The loop is two conditional edges. model_to_tools and tools_to_model are the same kind of router as route in section 2. The cycle model → tools → model is the whole agent loop.
Parallel tool calls are Send. When the model asks for two tools at once, the router returns one Send("tools", [call]) per call, so each runs as its own task in the next superstep (the Send mechanism explained at the end of section 3). In our one-tool run there is just one pending call, so there is one task.

The real function adds more nodes and edges when you pass middleware or a response format (that is what the extra ...middleware.before_agent nodes in the Deep Agents section are), but the skeleton is this.

4.2 Using it

Here is the code from the demo that calls create_agent. To stay offline and deterministic it uses a scripted model, a fake chat model that first asks to call the tool double(21) and then, once it sees the tool's result, answers. (A real agent would pass a real chat model in its place.)

from langchain.agents import create_agent
from langchain_core.language_models import BaseChatModel
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from langchain_core.outputs import ChatGeneration, ChatResult
from langchain_core.tools import tool

@tool                                    # turns a plain function into a tool the model can call
def double(n: int) -> str:
    """Double n."""                      # the docstring is the description the model sees
    return str(n * 2)

class Scripted(BaseChatModel):           # a fake model with a fixed script
    @property
    def _llm_type(self): return "scripted"
    def bind_tools(self, tools, **kw): return self.bind(tools=tools, **kw)
    def _generate(self, messages, stop=None, run_manager=None, **kw):
        done = [m for m in messages if isinstance(m, ToolMessage)]
        if not done:                     # turn 1: no tool result yet, so ask for the tool
            msg = AIMessage(content="", id="ai-1", tool_calls=[
                {"name": "double", "args": {"n": 21}, "id": "call-1", "type": "tool_call"}])
        else:                            # turn 2: tool result is in the history, so answer
            msg = AIMessage(content=f"answer {done[-1].content}", id="ai-2")
        return ChatResult(generations=[ChatGeneration(message=msg)])

agent = create_agent(Scripted(), tools=[double])      # one call builds the whole graph
out = agent.invoke({"messages": [HumanMessage("double 21")]})

Inspecting agent the same way as before prints:

nodes   : ['model', 'tools']
channels: {'messages': 'BinaryOperatorAggregate', 'jump_to': 'EphemeralValue',
           'structured_response': 'LastValue', '__start__': ..., '__pregel_tasks': ...}
node 'model': bound=RunnableCallable
node 'tools': bound=ToolNode
edges   : [('__start__','model'), ('model','__end__',cond), ('model','tools',cond), ('tools','model',cond)]
messages: HumanMessage 'double 21' → AIMessage[tool_call double] → ToolMessage '42' → AIMessage 'answer 42'

4.3 The graph it produced

The graph that create_agent built is small. Asking the compiled object to draw itself (agent.get_graph().draw_mermaid()) gives this, where dashed arrows are conditional edges:

graph TD;
    __start__([__start__]) --> model(model)
    model -.->|tool calls present| tools(tools)
    model -.->|no tool calls| __end__([__end__])
    tools -.-> model

This is what get_graph().draw_mermaid() produced (I ran it), with the two edge labels added by me to explain the routing. Walking it with our run: __start__ puts the HumanMessage into messages and wakes model. The scripted model replies with a tool call, so the router sends the run to tools. tools executes double(21) and appends a ToolMessage('42'), which wakes model again. This time the reply has no tool calls, so the router goes to __end__. That is the messages: line above: two trips through model, one through tools.

That is the whole ReAct loop:

State is a messages channel with an append reducer (so the conversation accumulates, just like visited above).
model is a node that calls the chat model. tools is a ToolNode: a Runnable that reads the last AIMessage's tool_calls, runs each tool and writes ToolMessages.
The loop is a conditional edge out of model: tool calls present → tools, otherwise → __end__. And tools → model is a plain back-edge. A cycle in the graph is the agent loop; the Pregel loop just keeps ticking while something is triggered.

Nothing agent-specific exists in the engine. "Agent" is a graph shape.

5. Graph bubble-up

A quick way to place this next to Send from the previous section: both are things a node or router hands to the loop, but in opposite directions. Send is data going down: a request to create tasks, written to a channel and picked up at the next barrier. Bubble-up is a control signal going up: an exception that stops work and travels from inside a node to the code that owns the run. They are separate mechanisms. They meet in one place: a task created by Send is an ordinary task, so if it calls interrupt() or raises a GraphBubbleUp, that signal bubbles up through the loop like any other (for example, an agent whose tool asks a human for approval).

Some things a node does are not return values. interrupt("approve?") has to stop the whole run, not just the node. LangGraph implements that as an exception, much like throwing a custom exception in C# that a catch further up the stack deliberately handles:

GraphBubbleUp(Exception)
  ├─ GraphDrained        (cooperative drain)
  ├─ GraphInterrupt      (interrupt(), human-in-the-loop)
  │    └─ NodeInterrupt
  └─ ParentCommand       (a subgraph telling its parent what to do)

"Bubble up" means exactly that: the exception is raised inside the node and travels up through the runner. Per _retry.py and _runner.py:

the retry wrapper (run_with_retry) deliberately does not retry a GraphBubbleUp; ordinary exceptions go through your RetryPolicy;
a GraphInterrupt has its pending writes saved to the checkpointer, so the pause survives;
a ParentCommand is handled by the enclosing graph, which is how a Command(graph=Command.PARENT, ...) from a subgraph reaches its parent; a root graph suppresses a GraphInterrupt and returns the interrupt in its result rather than raising.

Command is just a small dataclass of four optional fields: update, goto, resume, graph. You use it in two places, and a different pair of fields matters in each:

Returned from a node, to do "update state and route" in one step: Command(update={"total": 1}, goto="b"). update is applied like a normal node return value, and goto (a node name, a list of names, or Send packets) picks what runs next without needing an edge. graph=Command.PARENT aims it at the enclosing graph instead of the current one (that is the ParentCommand above).
Passed as the input to invoke, to continue a paused run: Command(resume="yes"). Normally invoke takes new state; here it is told "don't start over, the saved run is waiting for an answer, and this is it". resume is the value that the pending interrupt() call returns.

The demo below uses the second form. The first form (update/goto) is shown in section 5c.

The demo shows the whole cycle with an in-memory saver. First the graph, with one node that calls interrupt(). A checkpointer is required, because the pause has to be saved somewhere:

from langgraph.checkpoint.memory import InMemorySaver
from langgraph.types import Command, interrupt

def ask(s):
    answer = interrupt("approve?")       # pauses the whole run here; returns the resume value next time
    return {"visited": [f"answer={answer}"]}

h = StateGraph(S)                        # same state class S as before
h.add_node("ask", ask)
h.add_edge(START, "ask")
h.add_edge("ask", END)
happ = h.compile(checkpointer=InMemorySaver())
cfg = {"configurable": {"thread_id": "t"}}   # identifies the saved conversation/run

print("first call :", happ.invoke({"total": 0, "visited": []}, cfg))   # runs until interrupt(), returns
print("next       :", happ.get_state(cfg).next)                        # which node is waiting
print("resumed    :", happ.invoke(Command(resume="yes"), cfg))         # continue, answering "yes"

which prints (the interrupt id is shortened):

first call : {'total': 0, 'visited': [], '__interrupt__': [Interrupt(value='approve?', id='014f…')]}
next       : ('ask',)
resumed    : {'total': 0, 'visited': ['answer=yes']}

Reading it: the first invoke does not raise. It returns normally, with the pending question under __interrupt__. get_state(cfg).next says node ask is the one waiting. The second invoke is given a Command(resume="yes") instead of new input; it finds the saved checkpoint for the same thread_id and continues.

Note what "resume" really is: the node runs again from the top, and this time interrupt() returns the resume value instead of raising. That re-execution is why side effects before an interrupt() must be idempotent.

6. And Deep Agents

Where it comes from. create_deep_agent lives in the deepagents package, a separate open-source (MIT) project from LangChain, published at langchain-ai/deepagents. Its package metadata describes it as an "agent harness with a built-in filesystem and context management, sub-agent delegation, skills, and long-term memory", and marks it Beta (I used 0.7.20). It is not part of langgraph or langchain; it depends on langchain. Its source imports create_agent from langchain.agents, and its docstring says several arguments are "passed through to create_agent".

Where it sits in the stack. Think of three layers, each built on the one below:

deepagents   create_deep_agent   a ready-made "harness": files, subagents, summarization, skills, memory
   |  calls
langchain    create_agent        the model <-> tools loop, plus the middleware system
   |  builds
langgraph    StateGraph / Pregel the runtime: channels, supersteps, checkpoints

In C# terms: LangGraph is the runtime (like ASP.NET's host and pipeline), create_agent is a minimal app template, and Deep Agents is the opinionated starter kit that pre-installs a set of middleware on that template. Which is why the graph below looks the same as section 4's.

create_deep_agent(model=..., tools=[...]) also returns a CompiledStateGraph, with the same model and tools nodes, plus a middleware node. What the demo printed:

nodes   : ['model', 'tools', 'PatchToolCallsMiddleware.before_agent']
channels: messages: DeltaChannel, files: DeltaChannel,
          _summarization_event, _summarization_session_id, ...
tools the model sees: ['delete','double','edit_file','execute','glob','grep','ls','read_file','task','write_file']
edges   : ... ('model','model',cond), ('model','tools',cond), ('tools','model',cond) ...

So Deep Agents is the agent graph with extra state and extra tools, added by middleware:

Extra channels: files (a virtual file system kept in graph state) and summarization bookkeeping for long conversations. messages is a DeltaChannel here rather than a plain reducer channel, so I wouldn't assume the agent's channel types equal the plain agent's.
Extra tools: ls, read_file, write_file, edit_file, glob, grep, execute from the filesystem middleware, and task. execute is listed to the model, but the default StateBackend has no execute (I checked: hasattr(StateBackend, "execute") is False); it only runs when the backend implements SandboxBackendProtocol, otherwise it returns an error. So no shell access by default, and files is ephemeral graph state.
A middleware hook as a node: PatchToolCallsMiddleware.before_agent runs before the first model call (it repairs dangling tool calls). Only middleware that overrides before_agent, before_model, after_model or after_agent adds nodes (I confirmed M.before_model and M.after_model appear in nodes); middleware that wraps model or tool calls does not.

What is middleware?

Middleware is how create_agent is extended without editing the loop. It is the same idea as ASP.NET middleware or DelegatingHandler: you write a class, hand a list of them to create_agent(..., middleware=[...]), and the loop calls your hooks at defined points. AgentMiddleware has two families of hooks (from langchain/agents/middleware/types.py):

Node hooks, which run between steps: before_agent, before_model, after_model, after_agent. They receive the state and return a state update (or None), and can set jump_to to redirect the loop. Each one you override becomes a real node in the graph.
Wrap hooks, which run around a call: wrap_model_call(request, handler) and wrap_tool_call(request, handler). You call handler(request) to continue, so you can change the request, retry, short-circuit or post-process the response. These live inside the existing model and tools nodes, so they add no nodes.

A middleware can also contribute tools and extra state keys. I ran this to check:

class Audit(AgentMiddleware):
    def before_model(self, state, runtime):             # a node hook
        log.append(f"before_model: {len(state['messages'])} msgs"); return None
    def wrap_model_call(self, request, handler):        # a wrap hook
        log.append("wrap_model_call: enter")
        resp = handler(request)                         # the next layer, ending in the model
        log.append("wrap_model_call: exit"); return resp

agent = create_agent(fake_model, middleware=[Audit()])

nodes: ['Audit.before_model', '__end__', '__start__', 'model']
log  : ['before_model: 1 msgs', 'wrap_model_call: enter', 'wrap_model_call: exit']

Audit.before_model is a new node; wrap_model_call added nothing to the graph but ran around the model call. That is exactly the PatchToolCallsMiddleware.before_agent node in the Deep Agents output above. Deep Agents is mostly a curated middleware list (filesystem, summarization, tool-call patching, optional skills and memory).

What is the state backend?

Deep Agents' file tools (ls, read_file, write_file, edit_file, glob, grep) do not touch the disk directly. They call a backend, an object implementing BackendProtocol (ls, read, write, edit, grep, glob, delete, upload/download). The backend decides where files actually live, so the same agent can run against different storage. It is a repository/strategy interface, like IFileProvider in .NET. The backends in the package:

Backend	Where files live
`StateBackend` (the default)	In the graph state's `files` channel. Ephemeral: persists within a thread through checkpoints, not across threads
`StoreBackend`	A LangGraph `BaseStore`, so persistent across conversations
`FilesystemBackend`	The real disk
`LocalShellBackend`	The real disk plus unrestricted local shell (`execute`)
`BaseSandbox` (and subclasses)	A sandbox that implements `execute()`
`CompositeBackend`	Routes by path prefix to other backends (for example `/memories/` to a store, the rest to state)

Note: this is not the same thing as a LangGraph checkpointer, which is also sometimes called a state backend (section 5b). The checkpointer stores the graph's snapshots; the Deep Agents backend stores the agent's files. With the default StateBackend the two meet: its docstring says reads and writes go through CONFIG_KEY_READ and CONFIG_KEY_SEND, so a file write is just an ordinary channel write to files, applied at the barrier and checkpointed like any other state. That is why files showed up as a channel in the output above, and why the default needs no disk. StateBackend raises if used outside a graph run; to pre-seed files, pass {"files": {...}} to invoke.

Side note: the same pattern in the Squad SDK. Squad's SDK (bradygaster/squad, packages/squad-sdk) has the matching idea in its StorageProvider interface (read, write, append, list, delete, mkdir, rename, stat, createIfAbsent...), with FSStorageProvider (default), InMemoryStorageProvider and SQLiteStorageProvider: a narrow interface, several implementations, one default. It was added by Dina Berry (diberry; #567, #640). Squad also has a separate, higher-level StateBackend (working-tree files, an orphan git branch, or that plus git notes), which is not the same thing: an adapter wraps it as a StorageProvider. Source, pinned to the commit I read: storage-provider.ts, state-backend.ts, State Backends docs. Back to Deep Agents.

What is `DeltaChannel`?

Both messages and files in the Deep Agents graph are DeltaChannels instead of the plain reducer channel. The problem it solves: a normal reducer channel checkpoints its full value every step, so a long conversation or a large virtual file system is copied into every checkpoint. DeltaChannel (beta, langgraph/channels/delta.py) instead stores only a sentinel in ordinary checkpoints and rebuilds the value by replaying the earlier writes through the reducer, writing a full snapshot every so often (every 1000 updates by default). It is event sourcing: the log of writes is the truth, snapshots are an optimization.

The price is in its own docstring: the reducer must be deterministic and batching-invariant (reducer(reducer(s, xs), ys) == reducer(s, xs + ys)), and the checkpoint saver must keep the ancestor writes (get_delta_channel_history). Section 5b covers what this means for checkpoints.

Deep Agents capability inventory

From the installed package layout (deepagents/middleware and deepagents/backends), and what I observed above:

Middleware modules: filesystem, subagents, async subagents, summarization, memory, skills, permissions, rubric, patch-tool-calls, prompt caching, message eviction, overflow clipping, unsupported-content, tool exclusion.
Backends (where files/execute actually go): state (graph state, the default seen in the demo), store, composite, filesystem, local shell, sandbox, LangSmith, context hub.
Only middleware that defines before_*/after_* hooks becomes a graph node (we saw PatchToolCallsMiddleware.before_agent); the others wrap the model or tool call in place, so they don't appear in nodes. I inferred that rule from this one observed node and the module layout, not from reading the middleware loader.
In this default build the model's tools were the seven filesystem/shell tools, task, and my own double. I did not see a todo-list tool, so I won't claim one.

`task` is a graph calling a graph

The subagent mechanism is the most interesting part, and it is ordinary LangGraph. In deepagents/middleware/subagents.py, task is a StructuredTool. Its body prepares a fresh state (for a normal subagent just {"messages": [HumanMessage(description)]} plus non-private state keys), then calls:

result = subagent.invoke(subagent_state, subagent_config)
return _return_command_with_state_update(result, runtime.tool_call_id)

Because a compiled graph is a Runnable (section 1), the subagent is simply another compiled agent invoked from inside a tool call. Its result comes back as a Command(update={..., "messages": [ToolMessage(...)]}), which the parent's ToolNode applies to the parent's channels. The subagent's context window is isolated; only its final message and selected state keys return.

5b. Reducers, channel types and checkpoints, for real

Checkpoints. With a checkpointer, each superstep boundary writes a row. The demo saves the two-node graph in an InMemorySaver and lists it:

step=-1 source=input values={'__start__': {'total': 10, 'visited': []}}
step= 0 source=loop  values={'total': 10, 'visited': []}
step= 1 source=loop  values={'total': 20, 'visited': ['a']}
step= 2 source=loop  values={'total': 21, 'visited': ['a', 'b']}
versions_seen keys: ['__input__', '__start__', 'a', 'b']
channel_versions: {'__start__': '2', 'total': '4', 'visited': '4', 'branch:to:a': '3', 'branch:to:b': '4'}

A checkpoint is a small dict: channel_values, channel_versions, versions_seen, updated_channels, an id, a timestamp and v. Two fields do the scheduling:

channel_versions is a monotonically increasing version per channel (the real strings are zero-padded counters plus a random suffix; I shortened them).
versions_seen[node] records which trigger version each node last consumed. prepare_next_tasks runs a node when a trigger channel's version is newer than what that node has seen. That is the whole "what runs next" rule, and it is why resuming from a checkpoint needs no extra bookkeeping.

Step -1 stores the raw input; steps 0..N are loop supersteps. Writes made by tasks that have run but not yet been applied are stored separately as pending writes against their parent checkpoint (put_writes). That is how a crash in the middle of a parallel superstep doesn't lose the branches that already finished, and how interrupt state survives.

The saver (state backend) contract. BaseCheckpointSaver is the storage seam. The core methods are put (a checkpoint), put_writes (pending writes for a task), get_tuple and list (read), with async twins (aput, aget_tuple, ...), plus thread deletion/pruning helpers and a serde (here JsonPlusSerializer). Rows are addressed by thread_id, checkpoint_ns (the subgraph namespace) and checkpoint_id; a new thread id is a new empty history. I only ran InMemorySaver; I haven't exercised the SQLite or Postgres savers, so I make no claim about them.

Caveat: DeltaChannel. Deep Agents' messages and files use DeltaChannel (beta in langgraph/channels/delta.py). Per my reading of that source, not a run, it checkpoints a sentinel on ordinary steps and rebuilds the value from ancestor writes, with periodic full snapshots, so it needs deterministic reducers and savers that keep ancestor history. "Each checkpoint is a full snapshot" holds only for ordinary channels.

5c. When things get structured or go wrong: conflicts, routing, subgraphs, retry

The earlier sections showed the happy path. This one covers what happens when things go wrong or get more structured: two nodes fight over a key, a node needs to both update state and choose where to go, a graph contains another graph, a node throws, and you want to control how often state is saved or watch it live. I ran every snippet below offline (structured_cases.py in the samples repo); outputs are quoted as seen.

Two parallel nodes write the same key

Remember the superstep rule from section 3: nodes in the same superstep all read the old state, and their writes are applied together at the barrier. So what if two of them write the same key? For a plain field like total: int, the channel is a LastValue, which can hold only one value. "Which of the two writes wins?" has no sane answer, so LangGraph refuses to guess:

class S(TypedDict):
    total: int

g = StateGraph(S)
g.add_node("a", lambda s: {"total": 1})
g.add_node("b", lambda s: {"total": 1})
g.add_edge(START, "a")                           # a and b both start from START,
g.add_edge(START, "b")                           # so they run in the same superstep
g.add_edge("a", END)
g.add_edge("b", END)
g.compile().invoke({"total": 0})

InvalidUpdateError: At key 'total': Can receive only one value per step. Use an Annotated key to handle multiple values.

The error message already contains the fix. If you declare the key as Annotated[list, operator.add], the channel becomes a BinaryOperatorAggregate, which knows how to combine several writes, and the same graph returns {'total': ['a', 'b']}. In C# terms: it is the difference between two threads assigning the same field (a race you must forbid) and two threads calling list.AddRange through a lock (a merge you defined). The reducer is the lock.

`Command`: update state and choose the next node in one return

Normally a node returns a state update, and the edges decide what runs next. Sometimes the node itself knows where to go (a router, an agent deciding to hand off). Returning a Command lets it do both at once, as defined in section 5:

def router(s):
    return Command(update={"out": ["r"]}, goto="t")   # write state AND pick the next node

g = StateGraph(C)                       # C has: total: int, out: Annotated[list, operator.add]
g.add_node("r", router)
g.add_node("t", lambda s: {"out": ["t"]})
g.add_edge(START, "r")
g.add_edge("t", END)                                   # note: no edge from r to t
g.compile().invoke({"total": 0, "out": []})

{'total': 0, 'out': ['r', 't']}

There is no r → t edge in the graph. The goto created that routing at run time, and out shows both nodes ran, in order. The other two uses of Command you already met: resume= to continue after an interrupt (section 5), and graph=Command.PARENT to send the update to the enclosing graph when you are inside a subgraph (next subsection). Send (section 3) is the third routing tool: use goto to pick one next node, Send to start many tasks with different inputs.

A graph inside a graph (subgraphs)

Because a compiled graph is a Runnable (section 1), you can pass it to add_node like any function. The parent treats it as one node; inside, it runs its own supersteps. Its checkpoints live in their own namespace so they don't collide with the parent's, and stream(..., subgraphs=True) lets you see inside:

child = StateGraph(C)                        # C is the same state schema as above
child.add_node("t", lambda s: {"out": ["t"]})
child.add_edge(START, "t")
child.add_edge("t", END)

parent = StateGraph(C)
parent.add_node("child", child.compile())    # a compiled graph used as a node
parent.add_edge(START, "child")
parent.add_edge("child", END)
app = parent.compile(checkpointer=InMemorySaver())

cfg = {"configurable": {"thread_id": "1"}}
for event in app.stream({"total": 0, "out": []}, cfg, subgraphs=True):
    print(event)

(('child:512f5449-103f-f070-0e96-928c949f2c95',), {'t': {'out': ['t']}})
((), {'child': {'total': 0, 'out': ['t']}})

Each event is (path, update). The first has a non-empty path: node t finished inside the child. The second has the empty path (): the parent sees the whole child node finish. The path is the namespace. This is also why the task tool from section 6 is "a graph calling a graph".

Retry: what happens when a node throws

A node that raises normally fails the whole run. Attach a RetryPolicy and the loop reruns just that node instead. One gotcha I hit: the default retry_on deliberately excludes ValueError (programming-error exceptions are not retried by default), so my flaky node needed retry_on=ValueError explicitly:

def flaky(s):
    n["i"] += 1
    if n["i"] < 3: raise ValueError("boom")        # fails twice, then works
    return {"out": ["ok"]}

g.add_node("f", flaky, retry_policy=RetryPolicy(retry_on=ValueError, initial_interval=0.01))

{'total': 0, 'out': ['ok']}      # n["i"] == 3: two failures, third attempt succeeded

The retry is around the node, not the graph, so other nodes' finished work in the same superstep is not repeated. And the control-flow exceptions from section 5 (GraphBubbleUp: interrupts, parent commands) are never retried, because they are not failures. In C# this is Polly's Retry policy wrapped around one call, with the exception filter being retry_on.

Saving less often, and watching live

Two knobs control the cost and visibility of all this machinery:

durability on invoke/stream decides when checkpoints are written: 'sync' (before the next step starts; safest, slowest), 'async' (in the background while the next step runs), or 'exit' (only when the run ends; fastest, but a crash loses the in-flight run). These descriptions are from the parameter's documented meaning, not something I measured. The signature default is None; I did not verify what that resolves to, so I don't state an effective default.
Stream modes choose what stream emits: values (full state after each step), updates (just each node's return, which is what you saw above), checkpoints, tasks, debug, messages (LLM tokens), and custom (whatever your node writes via the stream writer). Same run, different views.

Deep Agents extras (listed, not exercised)

From the module inventory only, I did not run these: skills, memory (prompt injection or a BaseStore), permissions, and interrupt_on human-in-the-loop. Subagents can be any compatible runnable, and task isolates context by default; an experimental fork mode inherits parent state.

Reading state back, and time travel

get_state(cfg) returns a StateSnapshot (values, next, config, metadata, created_at, parent_config, tasks, interrupts), and get_state_history(cfg) walks the parent chain. Time travel is just picking an older snapshot and invoking from it:

state before b = {'total': 20, 'visited': ['a']}  ->  invoke(None, that config) = {'total': 21, 'visited': ['a', 'b']}

The Store is a different thing. A checkpointer is per-thread run state. BaseStore (demo: InMemoryStore) is cross-thread long-term memory addressed by a namespace tuple and a key: store.put(("users","tamir"), "pref", {...}). Deep Agents' memory and store-backed file backends build on this idea; I only verified the plain InMemoryStore here.

A final thought

What stayed with me after reading all this code is how little magic there is. A graph is a set of channels and the nodes that wake when those channels change. Edges are just writes to channels, supersteps give you a clean read, write, merge rhythm, and the occasional special exception or Command is how a node steers the loop from the inside. An agent is that same machine with a model node and a tool node in a cycle, and a Deep Agent is the same machine again, with middleware adding channels, tools and hook nodes. Once I saw it that way, the docs, the stack traces and the checkpoints all started to make sense, and I stopped treating LangGraph as a black box.

If you want to poke at it yourself, everything in this post runs offline, with no API key, in the samples repo. Pin the versions from the top of the post, because the internals move between releases.

Durable Task in the Real World: What Problems Does It Actually Solve?

TechAI Explained — Mon, 28 Sep 2026 19:13:57 +0000

I've never exactly hidden the fact that I think Durable Task is cool.

But "cool" is not a use case.

Some people would probably say a chainsaw is cool too. That doesn't mean every problem needs one. 🪚

After my last couple of posts about Durable Task, I got a comment that made me realize I had skipped an important question:

This all looks interesting, but what are the actual use cases?

Not how Durable Task works. Not what happens during replay or how the scheduler persists state. But the more practical question: what problems does it actually solve?

When should I reach for durable execution instead of regular C#, a queue, a database, or a background worker? And when would adding it just make things more complicated?

Rather than walk through another Hello World orchestration, I want to look at real situations where things get messy, especially when work takes a long time, depends on external systems, or gets interrupted halfway through.

And I also want to be clear about what Durable Task does not solve for you.

We'll look at five examples: invoice approvals, Azure tenant provisioning, subscription payments, breaking news articles arriving through out-of-order webhooks, and an AI agent investigating an incident.

In some of them, Durable Task is a great fit.

In others... maybe put the chainsaw back in the garage. 😄

We'll use the standalone Durable Task .NET SDK v1.26.0 with DTS.

If you want the background first, I covered what happens at a durable await and three less obvious Durable Task capabilities in the previous two posts.

A note on the examples: The snippets below are deliberately focused on orchestration boundaries. Application adapters, DTO definitions, registrations, and some helper implementations are omitted. They illustrate the design rather than constituting a compiled, tested sample. In particular, do not infer exactly-once external effects from durable history.

1. An invoice needs approval. Why should the worker remember the wait?

Consider an invoice-processing service. It extracts a document, validates the result, and either posts the invoice, rejects it, or asks a human to review it. A batch may contain eight invoices, only one of which needs approval. The other seven shouldn't wait for it.

Here's an ordinary C# implementation of the individual processing path:

public static async Task<InvoiceResult> ProcessAsync(
    InvoiceJob invoice, IInvoiceApplication app, CancellationToken ct)
{
    InvoiceCandidate candidate = await app.ExtractInvoiceAsync(invoice, ct);
    InvoiceAssessment assessment = await app.ValidateInvoiceAsync(candidate, ct);

    if (assessment.Check == InvoiceCheck.NeedsReview)
    {
        await app.OpenInvoiceReviewAsync(
            new InvoiceReview(invoice, assessment.PayloadReference), ct);
        return new(invoice.InvoiceKey, "AwaitingReview");
    }

    if (assessment.Check == InvoiceCheck.Rejected)
        return new(invoice.InvoiceKey, "Rejected");

    if (assessment.Check != InvoiceCheck.Valid)
        throw new InvalidOperationException("Unrecognized invoice assessment.");

    AccountingReceipt receipt = await app.SubmitApprovedInvoiceAsync(
        new InvoiceCandidate(invoice, assessment.PayloadReference), ct);
    return new(invoice.InvoiceKey, "Posted", receipt.EntryReference);
}

There's nothing inherently unreliable about this method. OpenInvoiceReviewAsync can persist a review ticket and notification outbox. But when someone approves the invoice tomorrow, another handler must load the right document generation, check the deadline, recover the approved payload, and continue. If the worker dies between extraction and validation, the application must decide whether to extract again or retrieve the saved result.

That continuation protocol is what Durable Task can express directly.

Give each invoice its own durable history

A bounded batch can start independent child orchestrations:

public override async Task<InvoiceResult[]> RunAsync(
    TaskOrchestrationContext context, InvoiceBatch batch)
{
    if (batch.Invoices is null || batch.Invoices.Length is < 1 or > 8
        || batch.Invoices.Select(x => x.InvoiceKey).Distinct().Count()
           != batch.Invoices.Length)
        throw new ArgumentException("Supply 1 to 8 distinct invoices.");

    return await Task.WhenAll(batch.Invoices.Select(invoice =>
        context.CallSubOrchestratorAsync<InvoiceResult>(
            nameof(ProcessInvoice), invoice)));
}

Eight is an application admission limit, not a DTS quota. Larger batches can be paged and subject to tenant/global concurrency limits. A child that completes can post its invoice while another child remains suspended awaiting review. Task.WhenAll waits for every child; it does not undo a sibling's accounting entry if another child fails.

Inside the invoice child, extraction and validation are activities. Once their successful results are recorded in orchestration history, a compatible replay follows the same code path using those recorded results instead of repeating the activities. This is re-execution of orchestrator code against history, not restoration of a suspended CLR stack.

When validation requires review, the child can express the wait:

var review = new InvoiceReview(invoice, assessment.PayloadReference);
await context.CallActivityAsync("OpenInvoiceReview", review);

TimeSpan remaining =
    invoice.ReviewDeadlineUtc - context.CurrentUtcDateTime;

if (remaining > TimeSpan.Zero)
{
    try
    {
        InvoiceReviewHint hint =
            await context.WaitForExternalEvent<InvoiceReviewHint>(
                $"invoice-review:{invoice.ReviewId}", remaining);

        if (hint.ProcessingId != invoice.ProcessingId
            || hint.ReviewId != invoice.ReviewId)
            throw new InvalidOperationException("Review identity mismatch.");
    }
    catch (OperationCanceledException)
    {
        // The durable wait timed out. Read the committed decision below.
    }
}

InvoiceDecision decision =
    await context.CallActivityAsync<InvoiceDecision>(
        "ResolveInvoiceReview", review);

The timer is durable. No thread is occupied while the invoice waits, and a replacement worker can replay the child to the same pending wait. But the event is only a hint. An authenticated endpoint must atomically commit a single terminal review decision to the application's review store before signaling the child. ResolveInvoiceReview reads or closes that authoritative ticket, including when the timer fires. If approval was committed just before the deadline but the hint arrived just after it, the stored decision, not notification timing, determines the outcome.

Only a valid automatic assessment or a committed human approval may proceed to the accounting activity. That activity uses a stable posting key and immutable request fingerprint. If the accounting system accepted the post but the activity's acknowledgement vanished, the integration must reconcile the same posting identity. Durable history cannot promise exactly-once accounting.

This is the part I like. Instead of building separate handlers to remember extraction, validation, the review wait, and posting, I can describe the whole process in one place and let Durable Task remember where it got to. I still need to authorize reviewers, persist their decisions, and make accounting submissions safe to repeat. If my existing continuation handlers already do all of that cleanly, I might not need another runtime.

2. Tenant provisioning: the cloud already has a deployment engine

Now consider onboarding a SaaS tenant. A TenantCreated event means the tenant record was accepted, not that its environment is ready. This design provisions a dedicated network, PostgreSQL server, storage with private access, and Container App; then it configures DNS, checks readiness, and activates the tenant.

The ordinary C# method is easy to read:

public async Task<ProvisionedEnvironment> ProvisionAsync(
    EnvironmentRequest request,
    CancellationToken cancellationToken = default)
{
    NetworkResources network = await azure.CreateVirtualNetworkAsync(
        request, cancellationToken);

    Task<PostgreSqlServerReference> databaseTask =
        azure.CreatePostgreSqlServerAsync(
            request, network.PostgreSqlSubnet, cancellationToken);
    Task<StorageAccountReference> storageTask =
        azure.CreateStorageAccountAsync(
            request, network.StoragePrivateEndpointSubnet,
            cancellationToken);

    await Task.WhenAll(databaseTask, storageTask);

    ContainerAppReference application = await azure.DeployContainerAppAsync(
        request, network.ContainerAppsSubnet,
        await databaseTask, await storageTask, cancellationToken);

    await azure.CreateDnsRecordAsync(
        request, application.Endpoint, cancellationToken);
    await azure.WaitForApplicationReadyAsync(
        application.Endpoint, cancellationToken);

    return new ProvisionedEnvironment(
        request.EnvironmentId, application.Endpoint);
}

The helpers above wrap Bicep/ARM operations and wait for terminal completion or readiness. They're application abstractions, not Azure SDK methods. The parallel modules own disjoint resources and don't compete over subnet updates.

Before adding DTS, ask an uncomfortable question: does ARM already solve this? For a single well-defined deployment, it may. ARM/Bicep already supports dependency ordering, parallel deployment, idempotent reapplication with stable identities, and server-side deployment state. A worker crash doesn't erase an ARM deployment. Querying it and reapplying the same desired state may be sufficient.

The wider onboarding process is different. The application must acknowledge the tenant request, report progress, coordinate readiness with activation, survive external approval steps if any, and decide what to do with a partly provisioned environment. That's where a business-level workflow can add value.

Separate resource readiness from tenant activation

First, the API authorizes and durably admits an immutable onboarding request, then returns 202 Accepted, a request ID, and a status URL. Accepted does not mean ready. The application records the tenant, environment, immutable specification, and ownership generation. Reusing the same idempotency key with the same specification returns the existing request; changing the specification is rejected.

The scheduler handoff is not magically atomic with the application's database. Save the intent to start in an outbox alongside admission; dispatch it and reconcile the workflow identity. Repeatedly scheduling a new instance is not the same as resuming an existing one.

The outer orchestration can then express the business dependency:

public override async Task<TenantActivation> RunAsync(
    TaskOrchestrationContext context, EnvironmentPlan plan)
{
    EnvironmentResult resources =
        await context.CallSubOrchestratorAsync<EnvironmentResult>(
            nameof(ProvisionEnvironment), plan);

    return await context.CallActivityAsync<TenantActivation>(
        "ActivateTenant", new TenantActivationRequest(plan, resources));
}

The child finishes at resource readiness. Activation is a separate, idempotent application activity that revalidates tenant ownership and actual resource readiness. If activation fails, the outer workflow fails visibly without automatically deleting the resources the child provisioned. Repairing activation is a different problem from rolling back provisioning.

Inside the child, stage dependencies form waves: network; database and storage in parallel; application; DNS; readiness. Each stage activity records an intent and starts or reconciles a provider operation. A separate read activity performs one bounded provider status query; the orchestrator schedules a durable timer between polls using context.CurrentUtcDateTime and context.CreateTimer.

DateTime now = context.CurrentUtcDateTime;
if (now >= deadline)
    return Failed("PollingDeadlineExceeded");

TimeSpan delay = TimeSpan.FromSeconds(progress.PollAfterSeconds);
DateTime wakeAt = delay >= deadline - now
    ? deadline
    : now.Add(delay);

await context.CreateTimer(wakeAt, CancellationToken.None);

No HTTP request or Task.Delay runs inside the orchestrator. The activity does the I/O; the orchestration history records the activity result and timer decision. A 45-minute polling budget limits additional polling. It does not cancel an Azure operation that was already accepted.

Follow one failure all the way through

Imagine the database deployment was accepted by Azure, but the worker died before the activity completion reached DTS:

DTS does not have a recorded successful activity result, so activity redelivery is possible.
The redelivered adapter consults the application's operation journal and the stable Azure deployment/resource identity.
If the deployment is still running, it resumes inquiry rather than starting an unrelated database.
Once the provider returns terminal completion, the activity records the result and the child continues toward application deployment.
The tenant is activated only after the resource child returns validated readiness evidence.

Contrast that with a different failure: the activity completion was recorded, and only then the worker crashed. On compatible replay, the orchestrator consumes that result from history without invoking the completed activity again. These are different failure windows; durable execution history solves one, and your adapter's reconciliation protocol solves the other.

Compensation isn't an undo button

A failed workflow doesn't mean Azure stopped working. Imagine that we've started creating a database and a storage account in parallel. Storage provisioning fails, but Azure is still creating the database. If we immediately delete everything we can see, that database might finish provisioning after cleanup has already checked for it. We've now reported failure and cleanup while an orphaned database continues generating charges.

Before cleaning up, the application needs to:

Prevent this failed onboarding attempt from submitting any new provisioning operations.
Check previously submitted operations, including requests whose responses were lost, and reconcile their actual state. Don't assume a timeout or a failed worker stopped Azure.
Delete only resources created exclusively for this attempt, and only after establishing that outstanding operations cannot create them again after deletion.

Automatic cleanup also has a strict boundary: this must be a failed initial onboarding attempt, before tenant activation or customer-data admission. Existing shared resources and active customer data are never cleanup targets. Recheck current ownership and provider identity before each deletion, and delete eligible new resources in reverse dependency order: DNS, application, database/storage, then network.

If we cannot establish what is still running or what we safely own, retain the resources and surface NeedsIntervention for reconciliation rather than guessing. Record both the original provisioning failure and any cleanup failures. Durable Task can coordinate this recovery process, but it cannot decide which Azure resources are safe to delete.

Going deeper: fencing and draining in-flight operations

Two related ideas explain why cleanup is harder than reversing completed steps:

Fencing prevents outdated attempts from making new changes, provided the system receiving those changes enforces the restriction. For example, our application can reject submissions from an older provisioning generation. This does not revoke requests Azure already accepted.

Draining and reconciling in-flight operations means accounting for previously submitted work, including requests whose acknowledgements were lost, before declaring cleanup safe. Some distributed-systems discussions call reaching a state with no relevant outstanding work quiescence (from quiescent, meaning quiet or inactive). Here we're describing a practical cleanup requirement, not the formal concurrency property quiescent consistency and not a guarantee Azure provides automatically.

For background, see Martin Kleppmann's explanation of fencing tokens and Microsoft's ARM deployment cancellation API. Even cancellation may leave resources partially deployed. If we cannot establish that earlier requests can no longer create resources, we retain the environment for reconciliation rather than declaring cleanup complete.

There is one thing to keep in mind here. Task.WhenAll waits for the whole provisioning wave, but if one of those activities throws, execution will not continue to the code that checks the results. In this example I treat expected failures as results and let unexpected failures fail the orchestration normally.

This also matters for cleanup. Waiting for all the provisioning tasks to finish tells me what succeeded and what failed, so I know what needs to be cleaned up. It does not automatically perform that cleanup. I still need to explicitly run the compensation steps for the resources that were already created.

Here, Durable Task is useful for connecting resource provisioning, the long waits, and tenant activation without writing my own continuation state machine. ARM still provisions the resources. My application still needs an outbox, operation identities, safe cleanup, and a way to repair uncertain outcomes. I would not add Durable Task just to rebuild the dependency graph ARM already gives me.

3. A payment timeout is not a decline

Suppose a subscription charge times out. The provider may have accepted the payment before the response disappeared. A naive retry risks charging twice; a naive decline path risks sending a misleading reminder.

A conventional implementation can persist NextCheckUtc, finish its handler, and let a scheduled job inquire later. That's a reasonable architecture. A durable orchestration becomes interesting when the recovery policy also includes due dates, callbacks, bounded reminders, a fixed grace deadline, and transition to the next billing cycle.

Before going further, a billing cycle means one billing period and its associated payment obligation, not one charge attempt or the entire subscription. For a €20 monthly subscription, September and October are separate cycles. Each has its own CycleId, due date, and initial PaymentKey; both belong to the same SubscriptionId. GraceUntilUtc sets the deadline for this cycle’s automated recovery. If a payment times out, the cycle remains the same while the service investigates whether that attempt actually succeeded.

Billing cycle	Due date	Amount	Example state
September 2026	Sep 1	€20	Paid
October 2026	Oct 1	€20	Pending
November 2026	Nov 1	€20	Not started

The distinction matters for ContinueAsNew. Once a paid receipt is confirmed and the billing service authoritatively plans the next cycle, the orchestrator can continue with new cycle input while retaining its instance ID and starting a fresh execution history:

Each confirmed paid cycle closes before ContinueAsNew receives the next cycle.

This is a conceptual lifecycle, not a promise that each month will be charged successfully. An unresolved cycle does not automatically advance, and the billing service, not the orchestrator, decides whether a next cycle is permitted.

Here the billing ledger, not DTS, is authoritative. The billing store binds a cycle ID and stable payment key to the amount and currency. Payment state distinguishes Unknown, Pending, Declined, Paid, and NotCharged, with cancellation and receipt information. The workflow coordinates inquiries and reminders. A separate payment service owns any newly authorized charge after card correction.

The orchestration waits until the cycle's due date with a durable timer, then calls an activity that either initiates the initial payment only when authorized and definitively not submitted, or reconciles an existing paid, pending, or unknown attempt. Its recovery loop checks state and waits for a callback hint or the next six-hour check, whichever comes first:

DateTime now = context.CurrentUtcDateTime;
if (now >= cycle.GraceUntilUtc || check == 15)
    break;

if (!reminderSent && !payment.SubscriptionCancelled
    && payment.State == PaymentState.Declined
    && now >= cycle.DueUtc.AddHours(6))
{
    await context.CallActivityAsync(
        "SendPaymentReminder",
        new PaymentReminder(cycle, $"{cycle.CycleId}:reminder-1"));
    reminderSent = true;
}

now = context.CurrentUtcDateTime;
if (now >= cycle.GraceUntilUtc)
    break;

DateTime wakeAt = now.AddHours(6);
if (wakeAt > cycle.GraceUntilUtc)
    wakeAt = cycle.GraceUntilUtc;

try
{
    BillingHint hint = await context.WaitForExternalEvent<BillingHint>(
        $"billing:{cycle.CycleId}:v{cycle.EventSchema}", wakeAt - now);

    if (hint.CycleId != cycle.CycleId
        || hint.EventSchema != cycle.EventSchema
        || string.IsNullOrWhiteSpace(hint.EventId))
        throw new InvalidOperationException("Billing event identity mismatch.");
}
catch (OperationCanceledException)
{
    // Timer elapsed. Query the billing ledger.
}

payment = await context.CallActivityAsync<PaymentSnapshot>(
    "ReadCyclePayment", cycle);

This excerpt assumes the surrounding orchestrator has validated the cycle, UTC deadlines, grace interval, snapshot identity, and terminal paid/cancellation branches. It also bounds the loop to at most 16 inspected snapshots. Callback handlers authenticate, deduplicate, and persist events in the billing inbox/ledger before sending a hint. Neither a callback nor a payment-method update proves that a charge succeeded.

When a paid receipt is validated, an idempotent activity closes the paid cycle and obtains the authoritative next cycle. Only then does the orchestrator call context.ContinueAsNew(next, preserveUnprocessedEvents: false) with that next-cycle input. That preserves the instance identity while starting a fresh execution history. Here we deliberately discard buffered old hints because the billing facts were persisted before notification; old settlements remain tied to their original billing cycle.

If the grace or check budget expires with an unresolved payment, the application outcome is NeedsReconciliation, tracked by a durable recovery queue. Runtime completion isn't proof of payment. Cancellation stops future collection when recorded but doesn't erase an already accepted charge.

I like being able to read the payment recovery policy as one flow, with its due date, callbacks, reminders, and transition to the next cycle. But the billing ledger still decides whether a payment actually happened. The workflow does not get to guess, charge again, or replace the existing payment safeguards. And if one scheduled job already handles this without much ceremony, I would keep that instead.

4. Webhook bursts: an example where durable state may not be enough

This example is a little different from the previous ones because it is also a good example of where Durable Task does not solve the whole problem.

Imagine a breaking-news publisher. Editors keep updating a developing story as new information arrives, and readers expect the publisher's search results to show the latest published version. Every time an editor publishes a change, the content-management system (CMS) emits a webhook. Our service receives it and updates a separate search index.

For example, an article titled Major storm hits the coast has three published revisions:

Revision 11: the initial report.
Revision 12: updated casualty figures and evacuation information.
Revision 13: a corrected headline and the latest official statement.

The webhook for revision 12 arrives before the delayed notification for 11; another webhook for 12 arrives twice. While our search index is still processing revision 12, revision 13 is published. Worse, two indexing operations can finish in the opposite order from when they started. Without protection, a slow revision-12 update could overwrite revision 13, leaving readers with stale breaking-news results.

Here's the flow. The arrows show possible timing, not a promise of ordering:

Entity counters do not serialize index writes. The destination rejects the late revision-12 update.

The key distinction: the durable entity remembers the newest revision we've heard about. It does not lock the search index or guarantee that only one indexing activity runs. The search destination must atomically reject an older revision if a newer one has already been committed. In the diagram, the delayed revision-12 operation is rejected rather than overwriting revision 13. A separate reliable inbox and follow-up dispatcher ensure that newly published revisions aren't lost between checks.

For this example, assume the CMS supplies strictly increasing positive revision numbers per article generation, immutable content for each published revision, and versioned tombstones for deletion. A recreated article gets a new generation. If the CMS supplies only opaque ETags or webhook arrival timestamps, this particular revision-ordering algorithm does not apply.

A conventional handler can already be correct:

public static async Task HandleAsync(
    DocumentNotice notice, IDocumentProjection projection,
    IProjectionCheckpointStore checkpoints, CancellationToken ct)
{
    long applied = await checkpoints.ReadAppliedAsync(
        notice.DocumentKey, ct);
    if (notice.Revision <= applied)
        return;

    DocumentSnapshot snapshot =
        await projection.ReadDocumentSnapshotAsync(
            new DocumentRead(notice.DocumentKey, notice.Revision), ct);

    ProjectionReceipt receipt =
        await projection.ApplyDocumentRevisionAsync(snapshot, ct);

    await checkpoints.AdvanceAppliedAsync(receipt, ct);
}

This assumes the destination atomically rejects stale revisions and handles duplicates idempotently, while the checkpoint advances monotonically. A lost acknowledgement can be reconciled against the destination. Add a reliable inbox/dispatcher and periodic source reconciliation, and you may already have everything you need.

What does a durable entity change? It can persist the highest observed and applied revisions and serialize updates to those counters:

public sealed class DocumentRevision : TaskEntity<DocumentRevisionState>
{
    public void Observe(long revision)
    {
        if (revision <= 0)
            throw new ArgumentOutOfRangeException(nameof(revision));
        State.LastObserved = Math.Max(State.LastObserved, revision);
    }

    public RevisionProgress Read() =>
        new(State.LastObserved, State.LastApplied);

    public void ConfirmApplied(ProjectionReceipt receipt)
    {
        if (receipt.DocumentKey != Context.Id.Key
            || receipt.ConfirmedRevision <= 0
            || string.IsNullOrWhiteSpace(receipt.ReceiptReference))
            throw new InvalidOperationException("Invalid projection receipt.");

        State.LastApplied = Math.Max(
            State.LastApplied, receipt.ConfirmedRevision);
    }
}

A separate SynchronizeDocument orchestration calls Observe, waits two seconds with a durable timer, reads the entity, fetches an authoritative immutable snapshot through a read activity, applies it through a write activity, and calls ConfirmApplied only after obtaining a valid external receipt. It can make up to four passes, then request follow-up if work remains.

Two different identities, two different guarantees. A durable entity is addressed by its entity name and key. In this example, DocumentRevision plus the article's generation-qualified key identifies one entity. Durable Task processes operations for that same entity instance serially, so simultaneous Observe and ConfirmApplied calls cannot corrupt its counters. This does not mean Durable Task automatically runs only one SynchronizeDocument orchestration per article: by default, separately scheduled orchestrations have separate instance IDs. See Microsoft: Durable entities and orchestration instance identity.

If we want one active synchronization controller per article, we can deliberately adopt the singleton orchestrator pattern: assign a stable orchestration instance ID derived from the article's generation-qualified key, rather than creating a new instance for each webhook. The inbox records every notice; the entity keeps the highest observed revision; the dispatcher starts the controller if needed or signals the existing controller to check again. The inbox must also cover the race between a controller's final read and its completion. A check-then-start request can itself race, so the dispatcher must handle duplicate-start outcomes and reconcile admission rather than assume both callers' success responses mean two controllers started.

A singleton controller still needs reliable admission, follow-up dispatch and the destination's atomic revision check.

A singleton controller avoids concurrent controllers for the same article when admission is implemented correctly, but it is not an exactly-once guarantee for external effects. A timed-out or redelivered activity may have left an indexing request running, and an earlier execution may have submitted work before it stopped. Keep the destination's atomic revision check and reconcile uncertain writes even with a singleton controller. The sequence diagram above intentionally depicts the multiple-controller variant, to show the race that the entity's serialization alone does not prevent.

But here's the catch: serializing entity operations does not serialize the external index writes. Two orchestrations can both read LastObserved = 13 and LastApplied = 12, then fetch and attempt revision 13 concurrently. The two-second pause may even align them. This design does not establish fewer fetches, fewer writes, or true debounce behavior.

Its correctness still depends on a destination-enforced atomic generation/revision fence. A delayed revision-12 write cannot overwrite an already committed revision 13. If the destination cannot enforce that rule, introduce a separately enforced single writer or fenced-writer protocol instead. The entity alone is insufficient.

A final read also races with a newly arriving revision. The application inbox/outbox must durably admit every validated notice and dispatch follow-up work; periodic reconciliation handles missing webhooks. Resetting an entity cannot reset the destination's revision protection.

The entity gives us a durable place to remember the latest article revision, and its operations are serialized for that entity key. That is useful, but it does not magically serialize the external indexing requests. If our existing checkpoint store and dispatcher already coordinate revisions reliably, adding an entity might not buy us much. I would measure actual indexing calls before claiming this approach reduces work.

5. An AI-assisted incident workflow: preserving the exact decision across a crash

I have written before about this kind of AI workflow and why deterministic orchestration matters when models and tools are involved. In that post, I focused mostly on making the workflow predictable. I did not spend nearly enough time on what happens when the process disappears halfway through. That is what I want to explore here.

Consider a production incident assistant: it gathers telemetry, asks a model to propose a remediation, waits for an engineer to approve the specific proposal, runs the approved operation, and verifies the result. The investigation might take minutes; approval might arrive hours later. During that time, the process hosting the assistant may be recycled or deployed.

We need parallel work, durable waits, replay, and external-operation reconciliation. We also face a new problem: an LLM response is not a deterministic function we should casually recompute after recovery. The next response might recommend a different action, use different tool arguments, or refer to different evidence.

Before: an ordinary asynchronous agent pipeline

Suppose the application already has services for telemetry, model inference, approval, remediation, and verification. A simplified ordinary implementation might look like this:

public static async Task<IncidentReport> InvestigateAsync(
    IncidentRequest incident,
    IIncidentApplication app,
    CancellationToken ct)
{
    Task<EvidenceRef> logs = app.CollectLogsAsync(incident, ct);
    Task<EvidenceRef> metrics = app.CollectMetricsAsync(incident, ct);
    await Task.WhenAll(logs, metrics);
    EvidenceBundle evidence = new(await logs, await metrics);

    ProposedRemediation proposal =
        await app.GenerateRemediationAsync(incident, evidence, ct);
    await app.OpenApprovalAsync(proposal, ct);

    // A separate callback must resume the right proposal after approval.
    return new IncidentReport(incident.IncidentId, "AwaitingApproval");
}

This method is not inherently wrong. The application could save a state machine in SQL and continue from an approval callback. But that continuation handler must retrieve the exact evidence and proposal version, check authorization and expiry, and decide whether remediation was already submitted. If the worker crashes after the model generates its proposal but before saving it, generating again is a new decision, not recovery of the same one.

The crucial design decision is to separate the nondeterministic generation step from the deterministic coordination step. The model call belongs inside an activity. The orchestrator receives a stable reference to an immutable proposal, not a mutable chat transcript or a promise that the model will answer identically next time.

After: durable coordination around an agent

For this example, assume the application defines IncidentPlan with a stable incident ID, immutable request generation, approval ID and UTC deadline. EvidenceBundle contains references to saved telemetry, not raw logs. ProposedRemediation contains an immutable proposal ID, evidence references, canonical action parameters, a content fingerprint and policy version. ApprovalDecision names the approved proposal ID and fingerprint. These are application contracts, not types supplied by Durable Task.

The request records below name the payloads passed to activities. ApprovalKind distinguishes Pending, Approved, Rejected, and Expired; only approval can lead to remediation. The activities and remaining DTOs are application code, not a complete executable sample:

public sealed record GenerateProposalRequest(
    IncidentPlan Plan, EvidenceBundle Evidence);
public sealed record RemediationApprovalRequest(
    IncidentPlan Plan, ProposedRemediation Proposal);
public sealed record VerifyRemediationRequest(
    IncidentPlan Plan, RemediationReceipt Receipt);

public sealed class InvestigateIncident
    : TaskOrchestrator<IncidentPlan, IncidentOutcome>
{
    public override async Task<IncidentOutcome> RunAsync(
        TaskOrchestrationContext context, IncidentPlan plan)
    {
        Task<EvidenceRef> logs = context.CallActivityAsync<EvidenceRef>(
            "CollectLogs", plan);
        Task<EvidenceRef> metrics = context.CallActivityAsync<EvidenceRef>(
            "CollectMetrics", plan);

        await Task.WhenAll(logs, metrics);
        EvidenceBundle evidence = new(await logs, await metrics);
        ProposedRemediation proposal =
            await context.CallActivityAsync<ProposedRemediation>(
                "GenerateAndSaveProposal",
                new GenerateProposalRequest(plan, evidence));

        await context.CallActivityAsync(
            "OpenRemediationApproval",
            new RemediationApprovalRequest(plan, proposal));

        TimeSpan remaining =
            plan.ApprovalDeadlineUtc - context.CurrentUtcDateTime;
        if (remaining > TimeSpan.Zero)
        {
            try
            {
                // This event is only a wake-up hint, not authorization.
                ApprovalHint hint =
                    await context.WaitForExternalEvent<ApprovalHint>(
                        $"incident-approval:{plan.ApprovalId}", remaining);
                if (hint.IncidentId != plan.IncidentId ||
                    hint.ApprovalId != plan.ApprovalId)
                    throw new InvalidOperationException(
                        "Approval identity mismatch.");
            }
            catch (OperationCanceledException)
            {
                // Deadline elapsed. Resolve authoritative approval state.
            }
        }

        ApprovalDecision decision =
            await context.CallActivityAsync<ApprovalDecision>(
                "ResolveApproval",
                new RemediationApprovalRequest(plan, proposal));
        if (decision.Kind is ApprovalKind.Rejected or ApprovalKind.Expired)
            return new(plan.IncidentId, decision.Kind.ToString());
        if (decision.Kind != ApprovalKind.Approved)
            throw new InvalidOperationException(
                "Approval is unresolved or unrecognized; reconcile the ticket.");

        // The activity must independently revalidate authorization,
        // proposal fingerprint, current policy and execution eligibility.
        RemediationReceipt receipt =
            await context.CallActivityAsync<RemediationReceipt>(
                "ExecuteApprovedRemediation",
                new ApprovedRemediation(plan, proposal, decision));

        VerificationResult verification =
            await context.CallActivityAsync<VerificationResult>(
                "VerifyRemediation",
                new VerifyRemediationRequest(plan, receipt));
        return new(plan.IncidentId, verification.Status,
            receipt.ReceiptReference);
    }
}

The snippet expresses dependencies, not a security boundary. The authenticated approval endpoint must atomically commit one terminal decision against the immutable proposal ID, fingerprint and deadline before sending its event hint. ResolveApproval reads that committed record or atomically expires a still-pending ticket at its deadline. An unresolved or unrecognized result fails visibly for reconciliation. A hint cannot authorize a tool call, and a late or duplicate hint cannot overwrite a terminal decision. The remediation activity checks the stored decision again and applies any current safety policy before acting. An approved plan that is no longer safe should stop for fresh review, not be executed merely because it appears in orchestration history.

Notice that GenerateAndSaveProposal must use a stable generation identity and reconcile its own saved result if the model responded but the activity acknowledgement was lost. Durable Task will reuse the activity result only after that result was recorded in orchestration history. It cannot guarantee a model is called exactly once in every failure window. Likewise, ExecuteApprovedRemediation needs a stable execution key, an operation journal and a provider-specific read/reconciliation path; blindly retrying a potentially destructive tool is unacceptable.

Follow the failure, not just the happy path

Consider four points at which the worker might disappear:

Interruption	Safe recovery behavior
After telemetry collection was recorded	Replay consumes recorded activity results; it does not collect those results again merely because the worker restarted.
After model generation but before its activity result was recorded	The activity uses the stable generation identity to retrieve the saved proposal or reconcile an uncertain generation; it must not silently substitute a new plan for one already presented.
While waiting for an engineer	The durable wait resumes after restart. Approval authority remains in the application approval store, and the fixed deadline does not move.
After remediation succeeded but before its activity completion was recorded	Redelivery reconciles the same execution key and external receipt. It does not assume the action failed or execute it a second time without evidence.

A useful test deliberately kills the worker at each of those points. Then inspect orchestration history, the proposal and approval records, and the remediation operation journal. The invariant is not “the orchestrator completed.” It is “only the specifically authorized proposal was executed, and uncertain side effects were reconciled before further action.”

So what did we gain?

We now have a workflow that can collect evidence, generate a proposal, wait for approval, execute the approved action, and continue after a worker restart. That is already pretty useful. But notice the boundary: we made the workflow around the AI durable. We did not make the model deterministic, and we did not make every bit of state inside an agent durable. The proposal and approval still need stable identities, and an uncertain tool execution still needs reconciliation.

This is really the foundation of what I want from durable AI systems. It is also why we went further and built Durable Agents and Durable Workflows as extensions to Microsoft Agent Framework (MAF). Instead of having every application build this coordination around its agents, we wanted durability to be part of the way agents and their workflows run.

That deserves its own article. In the next part, I will use this same scenario to show what changes when durability moves from the outer orchestration into the agent and workflow programming model itself.

A final thought

The common thread across these examples is simple. Durable Task can remember what completed, what we are waiting for, and where execution should continue. It cannot tell us whether an external payment really happened, whether an Azure operation is still running, or whether an AI generated action is still safe to execute. Our applications still need to answer those questions.

I would use Durable Task when the code for remembering and resuming a business process starts becoming a system of its own. Not just because I have a few await calls.

Squad: The AI Agent Team You Didn't Know You Needed

TechAI Explained — Fri, 20 Mar 2026 18:14:41 +0000

Squad: The AI Agent Team You Didn't Know You Needed

What if you could type one command and get an entire development team — lead, frontend engineer, backend engineer, tester, and a scribe who tracks every decision — all powered by AI agents?

That's exactly what Squad does.

Wait, Another AI Tool?

I know what you're thinking. "Great, another AI wrapper." Fair. But Squad isn't a chatbot with multiple personalities. Each agent runs in its own context window, reads only its own charter and history, and writes back what it learned. There's no single mega-prompt pretending to be five people.

Here's the mental model: Squad gives you a persistent AI development team through GitHub Copilot. You describe what you're building. Squad proposes specialists. You approve. They start working — in parallel.

You: "Team, build the login page"

  🏗️ Lead — analyzing requirements...          ⎤
  ⚛️ Frontend — building login form...          ⎥ all launched
  🔧 Backend — setting up auth endpoints...     ⎥ in parallel
  🧪 Tester — writing test cases from spec...   ⎥
  📋 Scribe — logging everything...             ⎦

No waiting in queue. No sequential hand-offs. True parallel fan-out.

The Casting System (This Is the Fun Part)

Every Squad team gets character names from fictional universes. Your lead isn't "Agent-1." Your lead is Keaton. Your frontend dev is McManus. Your backend is Verbal. Your tester is Fenster.

Yeah — The Usual Suspects (1995).

const casting = new CastingEngine({
  universe: 'usual-suspects',
  agentCount: 5,
});

const cast = casting.castTeam({
  roles: ['lead', 'frontend', 'backend', 'tester', 'scribe'],
});

// Result:
// { role: 'lead', agentName: 'Keaton' }
// { role: 'frontend', agentName: 'McManus' }
// { role: 'backend', agentName: 'Verbal' }
// { role: 'tester', agentName: 'Fenster' }
// { role: 'scribe', agentName: 'Kobayashi' }

But it's not just The Usual Suspects. Squad ships with themes from Alien, Star Wars, Ocean's Eleven, and more. The names persist across sessions — if Keaton is your lead today, Keaton is your lead next week, next month. You start saying things like "Keaton handles the routing layer" and it just sticks.

Why does this matter beyond the fun factor? Memory. Each agent has a history.md that compounds over time. After a few sessions, Keaton knows your architecture decisions. McManus knows you prefer Tailwind. The agents stop asking questions they've already answered.

How It Actually Works

Getting started is dead simple:

npm install -g @bradygaster/squad-cli
mkdir my-project && cd my-project
git init
squad init

This scaffolds a .squad/ directory in your project:

.squad/
├── team.md              # Who's on the team
├── routing.md           # Who handles what
├── decisions.md         # Shared decision log
├── casting/
│   ├── policy.json      # Which universe, how many agents
│   └── registry.json    # Persistent name mappings
├── agents/
│   ├── Keaton/
│   │   ├── charter.md   # Identity, expertise, voice
│   │   └── history.md   # What Keaton knows about YOUR code
│   ├── McManus/
│   │   ├── charter.md
│   │   └── history.md
│   └── ...
└── log/                 # Searchable session history

You commit this entire folder. Anyone who clones your repo gets the team — with all their accumulated knowledge. It's versioned team memory living in git.

Then fire it up:

copilot --agent squad --yolo

The --yolo flag auto-approves tool calls. Without it, Copilot asks you to confirm each one — and Squad makes a lot of tool calls.

Or use the interactive shell:

squad
# squad > @Keaton, analyze the architecture of this project
# squad > McManus, write a blog post about our new feature
# squad > Build the login page

The coordinator routes messages to the right agents. Multiple agents can work in parallel — you see progress in real-time.

Decisions Are Tracked (Not Lost in Chat)

One of my favorite details: every decision any agent makes gets logged to decisions.md.

const result = await tool.handler({
  author: 'Keaton',
  summary: 'Use PostgreSQL, not MongoDB',
  body: 'We chose PostgreSQL because: (1) transactions, (2) known team expertise, (3) schema flexibility via JSONB.',
  references: ['PRD-5-coordinator', 'architecture-spike'],
});

This isn't just logging — it's a shared brain. Every agent reads decisions.md before starting work. When Keaton decides on PostgreSQL, McManus and Verbal know about it immediately. No Slack thread where someone @-mentioned the wrong channel.

The Safety Net

Squad isn't just "AI agents go brrrr." There's governance:

File-write guards: Agents can only write to paths you allow (src/**, .squad/**, docs/**). Try to write to /etc/passwd? Blocked. Not by a prompt — by code.
PII scrubbing: Email addresses and sensitive data are automatically redacted in agent output.
Reviewer lockout: When the Tester rejects Backend's auth code, Backend literally cannot re-edit that file. Another agent has to fix it. Protocol enforced.
Ask-user rate limiter: Agents get a maximum number of "hey human, what should I do?" calls per session. After that, they decide and move on.

const pipeline = new HookPipeline({
  allowedWritePaths: ['src/**/*.ts', '.squad/**', 'docs/**'],
  scrubPii: true,
  maxAskUserPerSession: 3,
});

This is the hook pipeline — rules that run as code before any tool executes. Not prompt instructions that agents might ignore. Actual guardrails.

What Happens When You Walk Away

Here's the thing that sold me: Squad doesn't need you sitting there.

When agents finish a task, the coordinator chains follow-up work automatically. If you step away for lunch (or, let's be honest, for the day), a breadcrumb trail is waiting:

decisions.md — every decision any agent made
orchestration-log/ — what was spawned, why, and what happened
log/ — full session history, searchable

And there's Ralph — a persistent work monitor that watches for completed tasks, errors, and idle agents. Ralph never sleeps. When you get back, Ralph knows exactly where everything stands.

(Ralph deserves his own article. Stay tuned.)

Is This Actually Useful?

Look, I was skeptical. Multi-agent orchestration sounds like a buzzword salad. But Squad's design makes a few choices that change things:

Agents are files, not services. No Docker containers spinning up. No API keys to manage. Everything is markdown and JSON in your repo.
Knowledge compounds. The more you use Squad, the less time you spend explaining your project.
It's all in git. Fork the repo, you fork the team. That's kind of wild.
Crash recovery is built in. Agent dies mid-work? Sessions are durable. Next run picks up where it left off.

Squad is alpha software — the APIs are still changing. But the core idea — treating AI agents as a persistent, coordinated team rather than disposable chat sessions — feels like it's pointing somewhere important.

Try It

npm install -g @bradygaster/squad-cli
squad init
copilot --agent squad --yolo

Or with npx (no global install):

npx @bradygaster/squad-cli init

Check out the repo: github.com/bradygaster/squad

Eight sample projects, from beginner to advanced. Including casting, governance, streaming, and Docker setups.

This article is by the TechAI Explained team. We explore tools and frameworks that are changing how software gets built.

The Hidden Cost of Not Automating Your Dev Workflow

TechAI Explained — Fri, 20 Mar 2026 10:35:42 +0000

💸 The Hidden Cost of Not Automating Your Dev Workflow

You know you should automate more. You still manually run tests, hand-deploy to staging, and type the same git commands dozens of times daily.

The cost doesn't show up in Jira. But it's there — silently eating your week.

The Numbers

We tracked 8 developers for two weeks:

Manual Task	Weekly Total
Running tests before PR	4.8 hours
Setting up local env	50 min
Manual deployments	45 min
Checking CI in browser	1.7 hours
Copy-pasting configs	25 min

Total: ~8.5 hours/week per developer on automatable tasks. A full working day, gone.

It's Worse Than You Think

Context Switching

Every alt-tab to check CI breaks flow state. Microsoft research: 23 minutes to regain deep focus. At 15 switches/day, you're losing way more than task time.

Human Error

12% of manual deploys had config mistakes
23% of env setups needed retries

Knowledge Silos

Deployment process in someone's head = single point of failure. Hope they don't go on vacation.

Quick Wins (Do These Today)

1. Git Hooks

#!/bin/sh
# .git/hooks/pre-commit
npx prettier --write --staged
npx eslint --fix --staged

2. One-Command Setup

#!/bin/bash
set -e
npm ci
docker compose up -d postgres redis
npm run db:migrate && npm run db:seed
echo "✅ Ready!"

3. Auto Changelog

{ "scripts": { "release": "standard-version" } }

Conventional commits → changelog writes itself.

The ROI

8.5 hours × 8 devs × $75/hour = $5,100/week wasted. That's $265K/year.

Add reduced bugs, faster onboarding → $375K/year in value. Investment? 2-4 weeks of one dev's time.

Start Now

Pick your most annoying manual task. Automate it. Pick the next one. Repeat. Within a month, you'll wonder how you ever worked without it.

Ready to supercharge your workflow? Explore resources on CI/CD automation and agent-driven development to take automation to the next level!

MCP Servers: The Plugin System AI Agents Were Missing

TechAI Explained — Fri, 20 Mar 2026 10:34:54 +0000

🔌 MCP Servers: The Plugin System AI Agents Were Missing

Every AI framework reinvented the same wheel: how to let a model call tools. LangChain had its format, OpenAI had another, Anthropic had yet another.

Then MCP (Model Context Protocol) arrived. It's doing for AI agents what USB did for hardware — one standard plug that works everywhere.

What Is MCP?

An open protocol that standardizes how AI apps connect to tools and data:

AI Host (Claude, Copilot) ◄── MCP ──► MCP Server (any tool/API)

An MCP server exposes:

Tools — Functions the AI can call
Resources — Data the AI can read
Prompts — Reusable prompt templates

Before vs After MCP

Before: Three different formats for the same capability. Maintain three integrations.

After:

from mcp.server import Server

server = Server("my-search-server")

@server.tool("search")
async def search(query: str) -> str:
    """Search the database."""
    return await db.search(query)

# Works with Claude, Copilot, Cursor, and everything else.

Write once, run everywhere. That's the promise, and it delivers.

Build One in 5 Minutes

npm init -y
npm install @modelcontextprotocol/sdk

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";

const server = new McpServer({ name: "project-docs", version: "1.0.0" });

server.tool(
  "search_docs",
  "Search project docs by keyword",
  { query: z.string() },
  async ({ query }) => {
    // Your search logic here
    return { content: [{ type: "text", text: results }] };
  }
);

The Ecosystem (2026)

Category	MCP Servers
Databases	PostgreSQL, MySQL, MongoDB, Redis
Cloud	AWS, Azure, GCP
DevOps	GitHub, GitLab, Jenkins
Monitoring	Datadog, Grafana, PagerDuty

Security First

Least privilege — Only expose what's needed
Read-only by default — Opt-in for writes
Validate inputs — Use Zod schemas
Log everything — Audit every tool invocation

Bottom Line

If you maintain internal tools or APIs, wrapping them in an MCP server is the highest-leverage thing you can do right now. One server. One tool. Watch how it transforms your workflow.

Want deeper technical knowledge? Explore AI-Powered Development resources to understand MCP patterns and production workflows!

Building AI Agent Teams That Actually Ship Code

TechAI Explained — Fri, 20 Mar 2026 10:33:53 +0000

🤖 Building AI Agent Teams That Actually Ship Code

The idea of a single AI assistant is already old news. The real frontier? Teams of AI agents that collaborate, review each other's work, and ship production code.

After six months of building multi-agent workflows, here's what works and what's hype.

Why Single Agents Hit a Ceiling

Copilot, Cursor, Claude — great for autocomplete and boilerplate. But ask one agent to build a complete feature across multiple files with tests? It struggles. The bottleneck: context window management and task decomposition.

The Architecture That Works

We use a three-agent team that mirrors real engineering orgs:

Architect → Implementer → Reviewer
        ↑       ↓            ↓
        └── Shared Context Store ──┘

Architect: Reads the codebase, produces a plan. Never writes code.
Implementer: Codes one file at a time following the plan.
Reviewer: Catches bugs before any human sees the code.

The Secret: Shared Context

The biggest mistake? Treating agents as isolated workers. Without shared context, Agent B has no idea what Agent A decided.

class AgentContext:
    decisions = []      # Why choices were made
    constraints = []    # What must not change
    file_index = {}     # What each file does
    test_results = []   # Latest test outcomes

Every agent gets this context at every step. It's the institutional memory.

Real Results

Our agent team has:

Built a complete REST API (12 endpoints) in 45 minutes
Migrated Express to Fastify with zero test regression
Created a React component library with 20+ components
Fixed 34 bugs in a single afternoon

What Doesn't Work Yet

Let's be honest:

Complex architectural choices → still need humans
Performance optimization → agents write correct but not fast code
Cross-service debugging → agents lose the thread
UI/UX decisions → agents can't judge "good"

Get Started

agents:
  implementer:
    model: claude-sonnet-4
    role: "Write code following the plan"
  reviewer:
    model: claude-sonnet-4
    role: "Review for bugs and correctness"
workflow:
  - implementer: execute_plan
  - reviewer: review_changes
  - if_rejected: goto implementer

Start small with just implementer + reviewer. You'll never go back.

Want the full deep dive? Check out AI-Powered Development resources and keep building with agent teams!

DEV Community: TechAI Explained

What LangGraph Actually Builds: Runnables, Channels, Bubble-Up, Agents and Deep Agents

Side notes before we start: Python pieces and C# equivalents

1. The only contract: Runnable

2. What compile() produces

3. Pregel: the superstep loop

Send: when one node must become many tasks

4. Example: an AI agent is just a graph

4.1 The code of create_agent

4.2 Using it

4.3 The graph it produced

5. Graph bubble-up

6. And Deep Agents

What is middleware?

What is the state backend?

What is DeltaChannel?

Deep Agents capability inventory

task is a graph calling a graph

5b. Reducers, channel types and checkpoints, for real

5c. When things get structured or go wrong: conflicts, routing, subgraphs, retry

Two parallel nodes write the same key

Command: update state and choose the next node in one return

A graph inside a graph (subgraphs)

Retry: what happens when a node throws

Saving less often, and watching live

Deep Agents extras (listed, not exercised)

Reading state back, and time travel

A final thought

Durable Task in the Real World: What Problems Does It Actually Solve?

1. An invoice needs approval. Why should the worker remember the wait?

Give each invoice its own durable history

2. Tenant provisioning: the cloud already has a deployment engine

Separate resource readiness from tenant activation

Follow one failure all the way through

Compensation isn't an undo button

3. A payment timeout is not a decline

4. Webhook bursts: an example where durable state may not be enough

5. An AI-assisted incident workflow: preserving the exact decision across a crash

Before: an ordinary asynchronous agent pipeline

After: durable coordination around an agent

Follow the failure, not just the happy path

So what did we gain?

A final thought

Squad: The AI Agent Team You Didn't Know You Needed

Squad: The AI Agent Team You Didn't Know You Needed

Wait, Another AI Tool?

The Casting System (This Is the Fun Part)

How It Actually Works

Decisions Are Tracked (Not Lost in Chat)

The Safety Net

What Happens When You Walk Away

Is This Actually Useful?

Try It

The Hidden Cost of Not Automating Your Dev Workflow

💸 The Hidden Cost of Not Automating Your Dev Workflow

The Numbers

It's Worse Than You Think

Context Switching

Human Error

Knowledge Silos

Quick Wins (Do These Today)

1. Git Hooks

2. One-Command Setup

3. Auto Changelog

The ROI

Start Now

MCP Servers: The Plugin System AI Agents Were Missing

🔌 MCP Servers: The Plugin System AI Agents Were Missing

What Is MCP?

Before vs After MCP

Build One in 5 Minutes

The Ecosystem (2026)

Security First

Bottom Line

Building AI Agent Teams That Actually Ship Code

🤖 Building AI Agent Teams That Actually Ship Code

Why Single Agents Hit a Ceiling

The Architecture That Works

The Secret: Shared Context

Real Results

1. The only contract: `Runnable`

2. What `compile()` produces

4.1 The code of `create_agent`

What is `DeltaChannel`?

`task` is a graph calling a graph

`Command`: update state and choose the next node in one return