Testing Durable Functions and Production Realities

An orchestrator that calls three activities is not a function you can test like any other, and the reason is one word the rest of this article keeps returning to: replay. The runtime re-runs the orchestrator body from the top every time the instance wakes up, rebuilding progress from recorded history, so the question a test has to answer is not "did it return the right value once" but "does it return the right value every time the same code runs against the same history." That reframes two demands at once. A mocked activity has to hand back the same result on every call, and the only way to prove the real replay engine agrees with your mock is to run that engine.

Those two demands split testing into two layers, and the split organizes everything below. Unit tests mock the orchestration context, run in milliseconds, and prove your branch and aggregation logic; no engine runs and nothing replays. Integration tests register the real orchestrator and activities with an in-memory host, so real replay, real serialization, and real activity execution all happen against your code. The root cause underneath both is determinism: an orchestrator body has to make the same decisions on every replay, and your tests either honour that constraint or they quietly lie about whether the workflow is correct.

The code under test is the order workflow from the DurableFunctionsDemo companion sample (isolated worker, .NET 10); the entity examples reuse the DurableEntitiesDemo sample from the previous article. Tests use xUnit and Moq, the stack Microsoft's own Durable testing guidance assumes.

Why replay makes orchestrator tests different

The tests target this orchestrator. It validates an order, creates it, sends a confirmation, and folds a trace id and a start time into the result.

[Function(nameof(OrderOrchestrator))]
public static async Task<string> RunOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var order = context.GetInput<OrderRequest>()!;

    // Replay-safe substitutes: the runtime records these once and replays the
    // recorded value, so every replay sees the same timestamp and the same id.
    DateTime startedUtc = context.CurrentUtcDateTime;
    Guid traceId = context.NewGuid();

    var validated = await context.CallActivityAsync<bool>(
        nameof(ValidateOrderActivity), order);
    if (!validated)
        return "Order validation failed";

    var orderId = await context.CallActivityAsync<string>(
        nameof(CreateOrderActivity), order);

    await context.CallActivityAsync(nameof(SendConfirmationActivity), orderId);

    return $"{orderId} (trace {traceId:N}, started {startedUtc:O})";
}

Two properties of this function decide how it can be tested.

It replays. The runtime re-executes the body from the top after every awaited step completes, and again after a timer fires, an event arrives, or the host recycles. Awaits that already finished return their recorded results instead of running a second time, but the lines between them run again on each pass. The stretch from GetInput to the final return may execute many times over one instance's life.

It must be deterministic. Because every pass has to reach the same decisions, the body cannot read wall-clock time, mint random GUIDs, or touch I/O directly. That is why the sample reaches for context.CurrentUtcDateTime and context.NewGuid() instead of DateTime.UtcNow and Guid.NewGuid(): the runtime records each on the first pass and feeds the recorded value back on every later pass. A raw DateTime.UtcNow would produce a fresh timestamp each replay and diverge the history, at which point the runtime raises NonDeterministicOrchestrationException, though it does not detect every form of divergence.

Both properties land on your tests. Since the body can run more than once, any value a test feeds in has to stay stable across calls; a mock that returns a new GUID or timestamp each time makes the same test pass or fail depending on how many replays happened. And because the return string folds in traceId and startedUtc, a test that stubs those with changing values has nothing stable to assert against. The fix is the discipline the orchestrator already follows: pin every value the body sees to a constant.

Prevention comes before the test. The Durable Functions Roslyn Analyzer flags the common determinism violations (DateTime.Now, Guid.NewGuid(), direct environment or I/O access) at edit time, so a whole class of replay bugs never reaches a test run.

Unit testing orchestrators: replay-safe mocks

A unit test mocks TaskOrchestrationContext, stubs each activity call, invokes the orchestrator method directly, and asserts on what it returns. The happy path runs a valid order all the way to its confirmation.

[Fact]
public async Task Valid_order_runs_to_confirmation()
{
    var order = new OrderRequest(CustomerId: "cust-1", Sku: "sku-9", Quantity: 2);
    var context = new Mock<TaskOrchestrationContext>();

    context.Setup(x => x.GetInput<OrderRequest>()).Returns(order);
    context.Setup(x => x.CurrentUtcDateTime)
        .Returns(new DateTime(2026, 7, 17, 9, 0, 0, DateTimeKind.Utc));
    context.Setup(x => x.NewGuid())
        .Returns(Guid.Parse("00000000-0000-0000-0000-0000000000ab"));

    context.Setup(x => x.CallActivityAsync<bool>(
        It.Is<TaskName>(n => n.Name == nameof(ValidateOrderActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>())).ReturnsAsync(true);
    context.Setup(x => x.CallActivityAsync<string>(
        It.Is<TaskName>(n => n.Name == nameof(CreateOrderActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>())).ReturnsAsync("ORD-cust-1-sku-9");
    context.Setup(x => x.CallActivityAsync(
        It.Is<TaskName>(n => n.Name == nameof(SendConfirmationActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>())).Returns(Task.CompletedTask);

    string result = await OrderOrchestrator.RunOrchestrator(context.Object);

    Assert.StartsWith("ORD-cust-1-sku-9", result);
}

The matcher that trips everyone is the activity name. CallActivityAsync does not take a string; it takes a TaskName struct, and the orchestrator's nameof(ValidateOrderActivity) converts to one implicitly. Write It.IsAny<string>() in the setup and it never matches, the mock returns default (here false), and the order "fails validation" for no visible reason. Match the struct instead: It.Is<TaskName>(n => n.Name == nameof(ValidateOrderActivity)). This is the single most common cause of a Durable orchestrator mock that "returns null." The third argument is TaskOptions; set it up as It.IsAny<TaskOptions>() even for calls that pass no options, because the overload Moq binds against always carries that parameter.

The values you return matter as much as the names you match. ReturnsAsync("ORD-cust-1-sku-9") is replay-safe: the same result on every call. Compare the version that reaches for real non-determinism.

// Flaky: a fresh GUID on every access, so the returned string is never the same twice.
context.Setup(x => x.NewGuid()).Returns(Guid.NewGuid);

Guid.NewGuid here is a method group, so Moq invokes it on each call and hands back a new value. The orchestrator folds that GUID into its return, so the test has no stable string to assert on; worse, the same stub under the real replay engine (the integration host in the next section) records a different history on each pass and breaks the run outright. Pin it to a fixed value and both problems vanish. The rule is the orchestrator's own rule pointed back at the test: every value the body sees has to be a function of its input, not of when or how often it ran.

The second thing this layer buys you is cheap branch coverage. Flip the validation result and assert the workflow short-circuits without ever creating an order.

[Fact]
public async Task Invalid_order_stops_before_creation()
{
    var order = new OrderRequest("cust-1", Sku: "", Quantity: 0);
    var context = new Mock<TaskOrchestrationContext>();
    context.Setup(x => x.GetInput<OrderRequest>()).Returns(order);
    context.Setup(x => x.CurrentUtcDateTime).Returns(DateTime.UnixEpoch);
    context.Setup(x => x.NewGuid()).Returns(Guid.Empty);
    context.Setup(x => x.CallActivityAsync<bool>(
        It.Is<TaskName>(n => n.Name == nameof(ValidateOrderActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>())).ReturnsAsync(false);

    string result = await OrderOrchestrator.RunOrchestrator(context.Object);

    Assert.Equal("Order validation failed", result);
    context.Verify(x => x.CallActivityAsync<string>(
        It.Is<TaskName>(n => n.Name == nameof(CreateOrderActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>()), Times.Never);
}

The Verify(..., Times.Never) is the assertion that earns its keep: it proves the guard actually skipped order creation, rather than the test landing on a matching string by luck.

What this layer cannot do is run the real thing. The mocked context never replays, never serializes OrderRequest into history, never executes an activity. A test can pass here while the deployed orchestrator diverges on replay or fails to round-trip its state. Closing that gap is the job of the in-memory host, and it is the next section.

Testing activities and entities

Activities are where the replay rules stop applying. An activity runs once per invocation, does the real I/O, and has no history to reconstruct, so it tests like any other method: call it, assert on what it returns. The sample's validation activity is a pure function of its input.

[Fact]
public void Validate_rejects_empty_sku()
{
    Assert.False(ValidateOrderActivity.Run(new OrderRequest("cust-1", "", 1)));
    Assert.True(ValidateOrderActivity.Run(new OrderRequest("cust-1", "sku-9", 1)));
}

No mock, no context, no harness. That is the reward for pushing logic into activities: the moment code leaves the orchestrator it becomes ordinary and testable.

Real activities rarely stay pure. They call a database, an HTTP API, a blob container, and that dependency is exactly what you mock. Inject it through the constructor of an activity class (the isolated worker supports standard DI on [Function] classes) and hand the test a fake. The sample registers IPaymentGateway in Program.cs, and the worker resolves it into the activity's constructor.

public class PaymentActivities(IPaymentGateway gateway)
{
    [Function(nameof(ChargeCard))]
    public Task<bool> ChargeCard([ActivityTrigger] OrderRequest order)
        => gateway.ChargeAsync(order.CustomerId, order.Quantity);
}

[Fact]
public async Task ChargeCard_delegates_to_gateway()
{
    var gateway = new Mock<IPaymentGateway>();
    gateway.Setup(g => g.ChargeAsync("cust-1", 2)).ReturnsAsync(true);

    var activities = new PaymentActivities(gateway.Object);

    Assert.True(await activities.ChargeCard(new OrderRequest("cust-1", "sku-9", 2)));
}

Entities test differently again, and the difference is worth pausing on. An entity is never replayed, so its operations run arbitrary code, which makes the logic itself the most ordinary thing in this article to check. The catch is access. On a class-based TaskEntity<TState>, the State property is protected (verified against Microsoft.DurableTask.Abstractions 1.24.1), so a test cannot reach in and seed it before calling an operation. A small test-only subclass exposes the seam.

private sealed class TestableCounter : Counter
{
    // Counter.State is protected; widen it to the test so we can seed prior state.
    public int Seed { get => State; set => State = value; }
}

[Fact]
public void Add_accumulates_onto_existing_state()
{
    var counter = new TestableCounter { Seed = 7 };
    counter.Add(5);
    Assert.Equal(12, counter.Seed);
}

Counter.Add is the previous article's public void Add(int amount) => this.State += amount;; the subclass only widens State for the test.

That same test names a design fact worth asserting on directly: Add is not idempotent. Run it twice and you get 12 then 17, not 12 both times. Entity delivery is at-least-once, so an accumulating operation that runs twice on a redelivery double-counts. An operation you can prove idempotent (Reset, or a SetQuantity(x) that assigns instead of adds) survives redelivery unchanged, and that property deserves a test of its own.

What none of these unit tests touch is serialization. Setting Seed in memory never round-trips State through System.Text.Json, so the case-sensitivity and dictionary-key-casing traps from the previous article stay invisible here. Proving the state survives a real round-trip needs the in-memory host, which runs the actual serializer against your entity. That is where the next section goes.

Integration testing with DurableTaskTestHost

The mocked-context test proved the branch logic and never ran the engine. This test does the opposite: it starts the real orchestration engine in-process, schedules the workflow, and waits for it to finish. DurableTaskTestHost (from Microsoft.DurableTask.InProcessTestHost) hosts that engine in memory, with no Azure Storage, no emulator, and no sidecar to install.

[Fact]
public async Task Valid_order_completes_with_order_id()
{
    await using DurableTaskTestHost host = await DurableTaskTestHost.StartAsync(tasks =>
    {
        // Register the real orchestrator. The typed-input overload deserializes the
        // scheduled input to OrderRequest, so context.GetInput<OrderRequest>() inside
        // the orchestrator round-trips through the real serializer, exactly as in production.
        tasks.AddOrchestratorFunc<OrderRequest, string>(
            nameof(OrderOrchestrator),
            (ctx, _) => OrderOrchestrator.RunOrchestrator(ctx));

        // Register the production activity logic by name so CallActivityAsync executes it.
        tasks.AddActivityFunc<OrderRequest, bool>(
            nameof(ValidateOrderActivity), (_, order) => ValidateOrderActivity.Run(order));
        tasks.AddActivityFunc<OrderRequest, string>(
            nameof(CreateOrderActivity), (_, order) => CreateOrderActivity.Run(order));
        tasks.AddActivityFunc<string>(
            nameof(SendConfirmationActivity), (_, _) => { /* side-effect only */ });
    });

    string instanceId = await host.Client.ScheduleNewOrchestrationInstanceAsync(
        nameof(OrderOrchestrator), new OrderRequest("cust-1", "sku-9", 2));

    OrchestrationMetadata result = await host.Client.WaitForInstanceCompletionAsync(
        instanceId, getInputsAndOutputs: true);

    Assert.Equal(OrchestrationRuntimeStatus.Completed, result.RuntimeStatus);
    Assert.StartsWith("ORD-cust-1-sku-9", result.ReadOutputAs<string>());
}

The shape is register, schedule, wait, assert. StartAsync takes a registration callback where you add the production orchestrator and activities; the standalone SDK registers them by name with AddOrchestratorFunc/AddActivityFunc (the Azure Functions worker discovers [Function] methods for you at runtime, so this manual step is the test-side equivalent). host.Client is a full DurableTaskClient, the same type the production HTTP starter uses, so scheduling, status queries, terminate, and raise-event all work here too. WaitForInstanceCompletionAsync(getInputsAndOutputs: true) blocks until the instance reaches a terminal state and pulls the output back so ReadOutputAs<string>() can deserialize it.

This is the layer that earns its runtime. The typed registration forces OrderRequest through System.Text.Json on the way in, GetInput<OrderRequest>() deserializes it on the way out, and if a [JsonPropertyName] or a case mismatch broke that round-trip the orchestration would land in Failed, not Completed. That is exactly the class of bug the mocked-context test in the previous section cannot see, because its GetInput handed back the in-memory object it was told to return.

The failure path is one assertion away. Pull that registration block into a StartHostAsync() helper so both tests share it, then schedule an order with an empty SKU: the real ValidateOrderActivity returns false, the orchestrator short-circuits, and the instance completes with the validation message rather than throwing.

[Fact]
public async Task Invalid_order_completes_with_validation_failure()
{
    await using DurableTaskTestHost host = await StartHostAsync();

    string instanceId = await host.Client.ScheduleNewOrchestrationInstanceAsync(
        nameof(OrderOrchestrator), new OrderRequest("cust-1", "", 0));

    OrchestrationMetadata result = await host.Client.WaitForInstanceCompletionAsync(
        instanceId, getInputsAndOutputs: true);

    Assert.Equal(OrchestrationRuntimeStatus.Completed, result.RuntimeStatus);
    Assert.Equal("Order validation failed", result.ReadOutputAs<string>());
}

Two layers, not two choices. Unit tests stay for speed and branch coverage; DurableTaskTestHost covers whole-workflow replay and serialization. One honest caveat on lineage: the classic DTFx LocalOrchestrationService (in DurableTask.Emulator) offered a similar in-memory host for the older API, but it is legacy. New isolated-worker code targets DurableTaskTestHost.

Storage in production: Azure Storage vs Netherite

The tests above ran against an in-memory engine. Production runs against a storage provider, and the provider you pick decides both your throughput ceiling and your bill. The question is narrower than "which is faster": it is at what volume the cheaper-per-event provider is worth its fixed cost floor.

Start with where the money goes on the default provider. Azure Storage drives execution through queues, records status and history in Tables, and distributes instances across workers with blobs and blob leases. Payloads over roughly 45 KB serialized are compressed and offloaded to a blob in <taskhub>-largemessages, with a reference left in the row. The cost mechanic is the part worth internalizing: every orchestration event is Storage transactions. Each activity call is a queue put and a queue get; each state change is a Table write. A three-activity orchestration generates well over a dozen billable transactions, and a fan-out that schedules two hundred activities multiplies that. At low volume the transaction bill is noise. At high fan-out it becomes the dominant line item, and per-transaction billing is what accumulates. For performance-sensitive apps, give Durable its own storage account rather than sharing AzureWebJobsStorage, and prefer a general-purpose v1 account (v2 can bill Durable's transaction-heavy pattern at a higher rate).

Netherite attacks that transaction bill directly. Built by Microsoft Research for high throughput, it can raise the ceiling by more than an order of magnitude. It does so by changing the storage shape: an Azure Event Hubs namespace moves messages between partitions, and a Storage account holds durable state through FASTER logs. Many small Storage transactions become batched Event Hubs traffic plus fewer, larger Storage operations. That trade buys better price-performance at scale, but it adds a fixed Event Hubs cost floor (the provisioned throughput units) that a low-volume app pays whether or not it uses the capacity.

The constraints are the part not to bury. Netherite is retiring: support ends March 31, 2028. It is not supported on the Flex Consumption plan, and it does not support identity-based connections, so a managed-identity requirement rules it out. Switching providers does not migrate task-hub data; you start from a fresh, empty task hub. For those reasons Microsoft steers new high-throughput work toward the Durable Task Scheduler, a managed, push-based gRPC backend with the highest throughput of the three, a built-in instance dashboard, managed-identity support, and no storage account to babysit.

The decision rule is measure, do not guess. Netherite pays off only above the throughput threshold where its cheaper per-event handling outweighs the Event Hubs floor; below that line Azure Storage is both cheaper and simpler. There is no published ops-per-second figure to copy, because the crossover depends on your fan-out width, payload sizes, and event rate. Benchmark your own workload before switching an existing app, and for anything greenfield, evaluate the Durable Task Scheduler rather than adopting a backend with a retirement date.

Monitoring in Application Insights

A test proves the workflow was correct when you shipped it. Monitoring answers the harder production question: which of ten thousand running instances failed last night, and where. Application Insights is the recommended surface, because the Durable extension already emits a tracking event for every lifecycle transition (scheduled, started, awaited, completed, failed), and those events land in the traces table where KQL can slice them.

The first instinct in production is usually the wrong one: find every failed orchestration in a window. Each tracking event carries the orchestration's state in a custom dimension, so the filter is a single predicate.

traces
| where timestamp > ago(24h)
| where customDimensions["prop__runtimeStatus"] == "Failed"
| project timestamp,
          instanceId = customDimensions["prop__instanceId"],
          functionName = customDimensions["prop__functionName"],
          reason = customDimensions["prop__reason"]
| order by timestamp desc

Once you have an instance ID, the next query reconstructs what that one instance actually did. This is where a Durable-specific trap shows up. Because the orchestrator replays, the same log line appears many times, so a raw query returns the history several times over. Two dimensions clean it up: isReplay filters out the replayed passes, and ordering by sequenceNumber after timestamp puts the events in execution order.

traces
| where customDimensions["prop__instanceId"] == "the-instance-id"
| where customDimensions["prop__isReplay"] == "False"
| project timestamp,
          functionType = customDimensions["prop__functionType"],
          functionName = customDimensions["prop__functionName"],
          state = customDimensions["prop__state"],
          sequenceNumber = toint(customDimensions["prop__sequenceNumber"])
| order by timestamp asc, sequenceNumber asc

Sort by timestamp first and sequenceNumber second, in that order. sequenceNumber resets to zero when the host restarts, so it disambiguates events inside one host lifetime but cannot order events across a restart on its own.

A few configuration facts change what these queries can see. The relevant log category is Host.Triggers.DurableTask in host.json; its default level emits non-replay tracking events, and raising it to Warning trims volume while logReplayEvents: true turns the replay noise back on only when you are debugging a replay. Inputs and outputs are not logged by default, for cost and PII reasons; only byte counts appear unless you opt in with traceInputsAndOutputs. The Functions runtime also samples telemetry by default, which can silently drop lifecycle events during a burst, so tune sampling in host.json if instances go missing from your results.

For latency rather than failure, turn on distributed tracing V2: set distributedTracingEnabled: true and version: "V2" under durableTask.tracing in host.json (isolated worker needs the extension at v1.4.0 or later). App Insights then draws an end-to-end Gantt chart in Transaction Search that correlates the orchestration with each activity and sub-orchestration, which is the fastest way to spot the one slow step in a long workflow.

The thread back to the rest of this article is the log line inside the orchestrator. Because the body replays, a plain logger.LogInformation(...) fires on every pass and floods the trace with duplicates. context.CreateReplaySafeLogger("Name") suppresses the replayed emissions so each line logs once. The honest limit: it suppresses replay duplicates, not duplicates from a full re-execution (a host restart or an expired queue visibility timeout runs the body again from scratch, and those lines log again). Alongside App Insights, the Durable Functions Monitor gives a UI over instance state, and the Roslyn Analyzer remains the cheapest observability of all: it catches the determinism bug at edit time, before there is anything to monitor.

Conclusion

One word carried this whole article: replay. It is why a mocked activity has to return the same value on every call, why the mocked-context test can prove branch logic but never catches a serialization bug, and why the in-memory host has to exist at all. Testing Durable code is not one activity but a stack: fast mocked-context units for branch coverage, DurableTaskTestHost for the replay and serialization the mocks cannot exercise, and plain method calls for activities and entities where replay never applied. Production sits on top of that stack, where the storage provider decides your bill and App Insights tracking events decide how fast you find the instance that broke.

This was the last of five. The series started with triggers and the isolated worker, moved through orchestration patterns and fan-out, spent an article on entities and their at-least-once delivery, and ends here on proving the whole thing works before and after it ships. The connective tissue was always determinism: every constraint, from context.NewGuid() to the replay-safe logger, exists so the runtime can re-run your code and get the same answer.

When an orchestrator misbehaves in production, which do you reach for first: an App Insights tracking-event query to find where it failed, or a local DurableTaskTestHost run to reproduce the replay on your machine?