A four-step model pipeline that takes ninety seconds will eventually be interrupted at step three. The interesting question is whether the restart costs you three model calls or zero.
The problem a retry loop does not solve
Chained model calls are expensive, slow and individually unreliable. Extract, then classify, then summarise, then format: each step depends on the last, each can fail on a 429 or a timeout, and the whole chain runs longer than a single HTTP request should.
A retry loop inside one function handles a transient failure of one call. It does nothing about the process dying — a host restart, a scale-in, a deployment, an out-of-memory kill — because the loop’s state lived in memory that no longer exists. The work restarts from step one and you pay for the first three calls twice.
Durable Functions solves specifically that. It persists orchestration state to durable storage at every await point, so a process that dies mid-chain resumes from the last completed step on a different host.
How replay works
The mechanism is worth understanding because every rule below follows from it. An orchestrator function does not run once from start to finish. It runs, hits an await, and stops — its progress written to an append-only history. When the awaited result arrives, the orchestrator is re-executed from the top, and each await it reaches is satisfied instantly from the history rather than re-performed.
So a four-step orchestration executes its own code four times, and only the last step of each pass does any new work. Microsoft’s description is that the framework checkpoints the progress of the instance each time the code awaits, and that a recycled process resumes from the preceding await.
This is why the split between orchestrator and activity is not stylistic. Activities wrap the non-deterministic operations — model calls, tool invocations, HTTP requests — and their results are recorded in the history. Orchestrators define control flow and are replayed. Put a model call directly in an orchestrator and it will be made once per replay pass.
The determinism rules
Orchestrator code must produce the same decisions on every replay given the same history. Three consequences, and they are absolute:
- No current time. A wall-clock read returns a different value on each pass. Use the orchestration context’s deterministic current time, which replays identically.
- No random values, no new GUIDs. Same reason. Generate them in an activity, where the result is recorded.
- No I/O, no network calls, no environment reads. Everything that touches the outside world belongs in an activity. This includes the model call, which is the whole point.
A violation does not fail loudly. It produces an orchestration that works in testing and diverges from its history under load, which surfaces as a non-deterministic-workflow error long after the code was written. Treat the orchestrator as pure control flow and this never arises.
Build the orchestration
The Python v2 programming model declares everything with decorators on a DFApp. Three function kinds: a client that starts things, the orchestrator, and the activities.
import azure.functions as func
import azure.durable_functions as df
from openai import AzureOpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
app = df.DFApp(http_auth_level=func.AuthLevel.FUNCTION)
_token = get_bearer_token_provider(
DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default"
)
_client = AzureOpenAI(
azure_endpoint="https://mg-openai-weu.openai.azure.com/",
azure_ad_token_provider=_token,
api_version="2024-10-21",
)
@app.route(route="summarise")
@app.durable_client_input(client_name="client")
async def start_summarise(req: func.HttpRequest, client) -> func.HttpResponse:
instance_id = await client.start_new("summarise_orchestrator", None, req.get_json())
return client.create_check_status_response(req, instance_id)
@app.orchestration_trigger(context_name="context")
def summarise_orchestrator(context: df.DurableOrchestrationContext):
document = context.get_input()
outline = yield context.call_activity("extract_outline", document)
summary = yield context.call_activity("write_summary", outline)
checked = yield context.call_activity("verify_summary", {
"document": document,
"summary": summary,
})
return checked
@app.activity_trigger(input_name="document")
def extract_outline(document: str) -> str:
response = _client.chat.completions.create(
model="chat-default",
messages=[
{"role": "system", "content": "Return a bulleted outline. No prose."},
{"role": "user", "content": document},
],
max_tokens=500,
)
return response.choices[0].message.content
create_check_status_response returns 202 with a set of URLs for polling status, raising events and terminating the instance. That is the correct shape for this: the caller is told where to look rather than held open for ninety seconds.
Independent steps should not be sequential. Collecting a list of activity tasks without yielding each one and then yielding them together with the context’s task-all combinator fans them out in parallel and fans the results back in — which for five independent model calls turns five round trips into one.
Retries, and the timeout that goes away
Activities get first-class retry policies, which is where 429 handling belongs. Retry configuration lives on the call, not inside the activity:
retry = df.RetryOptions(
first_retry_interval_in_milliseconds=5000,
max_number_of_attempts=5,
)
outline = yield context.call_activity_with_retry("extract_outline", retry, document)
Set the backoff coefficient above 1 so the interval grows rather than hammering a throttled deployment at a fixed rate, and remember that the OpenAI SDK is already retrying twice underneath by default — set max_retries=0 on the client if you want the durable policy to be the only one, or you have multiplied the attempt count without meaning to. That interaction is covered on the 429 page.
The other thing this buys is the removal of a hard limit. A plain HTTP function has a request timeout that varies by hosting plan and that a long model chain will eventually exceed; an orchestration has no such ceiling because it is not one request. Each activity is a short execution, and the orchestrator is asleep between them. The classic Consumption-plan timeout that ends a long-running function mid-call stops being a design constraint.
One thing durable orchestration does not give you is streaming. The whole model is request-response with checkpoints, so a user watching a progress indicator is watching status polls rather than tokens. If the workflow is user-facing, the usual shape is a durable orchestration for the pipeline with a separate streamed call for the final user-visible generation.
Where the state lives, and what it costs
The durability is not free and it is not magic: the history is written to a storage backend, which by default is the function app’s own Azure Storage account, using queues to schedule work and tables to hold the append-only history. Understanding that turns three surprising behaviours into predictable ones.
Everything crossing a boundary is serialised. Inputs and outputs of every activity are JSON-encoded into the history. That means a non-serialisable object fails at the call rather than inside the function, and — more expensively — that a 200 KB document passed into three successive activities is written to storage roughly six times. The fix is the same one used for queue-based systems generally: pass an identifier or a blob reference between steps and let each activity fetch what it needs. Model outputs are the obvious case, because they are large, and they are the thing you were passing along.
Large payloads spill to blobs automatically. When a message exceeds the queue size limit, the framework offloads it to blob storage and passes a pointer. This is correct and invisible, and it is also a round trip per hop that you did not ask for. If a pipeline feels slower than the sum of its model calls, this is the first thing to measure.
History grows, and replay grows with it. Each await adds to the history that every subsequent replay must read. A short chain never notices; a loop that runs a hundred iterations does, and an orchestration intended to run indefinitely will eventually replay itself into uselessness. The documented answer is to restart the orchestration with fresh state at a checkpoint — the eternal orchestration pattern — which truncates the history to the state you chose to carry forward.
Finally, completed instances are retained until something removes them. Purge instance history on a schedule, or the storage account accumulates every run you have ever made along with every document that passed through one, which is a cost problem and, if those documents contained anything sensitive, a retention problem too.
Top comments (0)