Microsoft.Extensions.AI’s New Failover Feature Has a Streaming Blind Spot

I saw the announcement the same afternoon it went up on the .NET blog: Microsoft.Extensions.AI 10.9.0 ships built-in routing and failover. I’d been maintaining a hand-rolled Polly wrapper around IChatClient for about four months at that point, the kind of thing that starts as fifteen lines and grows a switch statement every time a provider has a bad day. My first reaction reading "Routing and Failover for Microsoft.Extensions.AI" was relief bordering on excitement. Four new experimental types, RoutingChatClient, SemanticRoutingChatClient, FailoverChatClient, and OrderedFailoverChatClient, all sitting behind the MEAI001 experimental diagnostic. I closed a dozen browser tabs of my own retry logic and started ripping code out that same evening.

Then I actually wired it into the one endpoint in our app that streams tokens to the browser, and I found the thing the announcement mentions in a single sentence and then moves past: once a token has left the server and reached the caller, failover cannot undo that. It’s not a caveat you can shrug off if your product streams anything, and if you’re building chat UIs in 2026 you almost certainly stream. This is the writeup of what shipped, what it’s genuinely good at, and the specific place where I’d tell you to slow down before you point it at a production streaming endpoint.

What actually shipped in 10.9.0

Before the streaming problem, credit where it’s due, because the routing story here is legitimately well designed. Microsoft.Extensions.AI 10.9.0 adds four IChatClient decorators, all marked experimental with MEAI001, meaning you'll need #pragma warning disable MEAI001 or the equivalent project property until Microsoft graduates them out of preview.

dotnet add package Microsoft.Extensions.AI --version 10.9.0

RoutingChatClient, the base you build on

RoutingChatClient is the abstract root. It wraps a set of candidate IChatClient instances and picks one per request by overriding SelectClientAsync. The simplest form doesn't even need a subclass, there's a static Create factory that takes a delegate:

using Microsoft.Extensions.AI;
#pragma warning disable MEAI001
IChatClient router = RoutingChatClient.Create((context, ct) =>
    new ValueTask<IChatClient>(
        IsComplexRequest(context) ? powerfulClient : cheapClient));
static bool IsComplexRequest(RoutingContext context) =>
    context.Messages.Sum(m => m.Text?.Length ?? 0) > 4000;

That’s the whole surface for the simple case: examine the RoutingContext (which gives you the messages and the ChatOptions for the incoming call), return the client that should handle it. For anything more involved than a length check, you subclass instead:

public sealed class TierRouter : RoutingChatClient
{
    private readonly IChatClient _gpt5Mini;
    private readonly IChatClient _gpt5;
    public TierRouter(IChatClient gpt5Mini, IChatClient gpt5)
    {
        _gpt5Mini = gpt5Mini;
        _gpt5 = gpt5;
    }
    protected override ValueTask<IChatClient> SelectClientAsync(
        RoutingContext context, CancellationToken cancellationToken)
    {
        bool needsReasoning = context.Messages
            .Any(m => m.Text?.Contains("step by step", StringComparison.OrdinalIgnoreCase) == true);
        return new ValueTask<IChatClient>(needsReasoning ? _gpt5 : _gpt5Mini);
    }
}

SemanticRoutingChatClient, routing by meaning instead of keywords

The keyword check above is fragile, and the team clearly knew it, because SemanticRoutingChatClient routes by embedding similarity instead. You give it example utterances per client and an IEmbeddingGenerator, and it picks whichever client's examples are closest to the incoming message:

using Microsoft.Extensions.AI;
#pragma warning disable MEAI001
IEmbeddingGenerator<string, Embedding<float>> embeddingGenerator =
    new AzureOpenAIClient(new Uri(endpoint), new ApiKeyCredential(key))
        .GetEmbeddingClient("text-embedding-3-small")
        .AsIEmbeddingGenerator();
IChatClient router = new SemanticRoutingChatClient(
    embeddingGenerator,
    clientProfiles: new Dictionary<IChatClient, IReadOnlyList<string>>
    {
        [codingClient] = ["write code", "fix this bug", "refactor this function"],
        [creativeClient] = ["write a story", "brainstorm names", "generate a poem"],
        [supportClient] = ["I want a refund", "my order didn't arrive", "cancel my subscription"],
    },
    defaultClient: generalClient,
    scoreThreshold: 0.3f);

I tried this against about sixty real support transcripts we had lying around from an old ticket export, and it correctly routed roughly 54 of them to the support client on the first pass with no prompt engineering. The scoreThreshold parameter matters more than the docs make it sound. At 0.3 I got the 54, dropping to 0.2 pulled in a handful of false positives where casual chit-chat with the word "cancel" in it got routed to support instead of general.

FailoverChatClient and OrderedFailoverChatClient

This is the part I actually came for. FailoverChatClient is an abstract RoutingChatClient subclass that adds retry semantics: if a selected client throws or times out, it calls SelectClientAsync again and tries the next one, up to MaximumAttemptsPerRequest. OrderedFailoverChatClient is the concrete implementation most people will reach for first, it just walks a ranked list:

using Microsoft.Extensions.AI;
#pragma warning disable MEAI001
IChatClient primary = new ChatClientBuilder(openAiClient).Build();
IChatClient backup = new ChatClientBuilder(azureOpenAiClient).Build();
IChatClient lastResort = new ChatClientBuilder(anthropicClient).Build();
IChatClient failover = new OrderedFailoverChatClient(
    [primary, backup, lastResort]);
ChatResponse response = await failover.GetResponseAsync(
    "Summarize this quarterly report in three bullet points.");

Call GetResponseAsync, and if primary throws (rate limited, 500, connection reset, whatever), OrderedFailoverChatClient transparently retries against backup, then lastResort, before it ever surfaces an exception to you. There's a hook, OnRoutingUpdateAsync, that fires after every attempt with a FailoverChatClientAttempt record carrying the duration, any exception, and a ResponseCompleted flag, which is exactly where I put my logging:

public sealed class LoggingFailoverClient : OrderedFailoverChatClient
{
    private readonly ILogger _logger;
    public LoggingFailoverClient(IReadOnlyList<IChatClient> clients, ILogger logger)
        : base(clients) => _logger = logger;
    protected override ValueTask OnRoutingUpdateAsync(
        RoutingContext context, FailoverChatClientAttempt attempt,
        bool isTerminal, CancellationToken cancellationToken)
    {
        _logger.LogInformation(
            "Failover attempt: duration={DurationMs}ms completed={Completed} terminal={Terminal} exception={Exception}",
            attempt.Duration.TotalMilliseconds, attempt.ResponseCompleted, isTerminal,
            attempt.Exception?.GetType().Name);
        return base.OnRoutingUpdateAsync(context, attempt, isTerminal, cancellationToken);
    }
}

Sticky sessions with IDistributedCache

One more pattern worth stealing directly from the announcement: pinning a conversation to whichever route it started on, so a multi-turn chat doesn’t bounce between providers mid-conversation and lose context or tone. The trick is a custom FailoverChatClient that reads and writes a route name through IDistributedCache, keyed by a session id you pass through ChatOptions.AdditionalProperties:

using Microsoft.Extensions.AI;
using Microsoft.Extensions.Caching.Distributed;
using System.Collections.Concurrent;
#pragma warning disable MEAI001
public sealed class StickyRouter : FailoverChatClient
{
    private readonly IReadOnlyDictionary<string, IChatClient> _routes;
    private readonly ConcurrentDictionary<RoutingContext, string> _pending = new();
    private readonly IDistributedCache _cache;
    public StickyRouter(IReadOnlyDictionary<string, IChatClient> routes, IDistributedCache cache)
    {
        _routes = routes;
        _cache = cache;
        MaximumAttemptsPerRequest = 1;
    }
    protected override async ValueTask<IChatClient> SelectClientAsync(
        RoutingContext context, CancellationToken cancellationToken)
    {
        string route = await _cache.GetStringAsync(CacheKey(context), cancellationToken)
            ?? "fast";
        _pending[context] = route;
        return _routes[route];
    }
    protected override async ValueTask OnRoutingUpdateAsync(
        RoutingContext context, FailoverChatClientAttempt attempt,
        bool isTerminal, CancellationToken cancellationToken)
    {
        if (_pending.TryRemove(context, out string? route) && attempt.ResponseCompleted)
        {
            await _cache.SetStringAsync(CacheKey(context), route, cancellationToken);
        }
    }
    private static string CacheKey(RoutingContext context) =>
        context.ChatOptions?.AdditionalProperties?.TryGetValue("routing-session-id", out string? id) == true
            ? $"chat-route:{id}"
            : throw new InvalidOperationException("A routing session ID is required for sticky routing.");
}

We use Redis in production for IDistributedCache, but this works with any backend, including the in-memory implementation for local testing. Register a session id per conversation and every subsequent turn lands on the same client, unless that client fails, at which point it falls through to the next one and the sticky value updates.

All of this, the routing, the semantic matching, the sticky sessions, is genuinely good work, and I don’t want the rest of this article to read as a takedown. It isn’t. But the announcement post spends one sentence on streaming, and that sentence deserves a lot more attention than it got.

The sentence I almost skipped past

Here’s the line, close to verbatim from the announcement: after output starts flowing to the caller, failure becomes terminal, there’s no mid-stream recovery. I read that the first time and mentally filed it under “reasonable limitation, makes sense, moving on.” It took building against it to understand that “no mid-stream recovery” doesn’t mean “failover politely declines to help.” It means something closer to: if you don’t handle this yourself, your application will silently stitch together the first half of one AI-generated response with the second half of a completely different one, and hand the result to your user as if it were coherent.

That’s not a crash. It’s not an exception you catch. It’s wrong output that looks plausible enough that nobody notices until a support ticket comes in asking why the assistant contradicted itself mid-sentence.

Where it actually breaks

Non-streaming calls are safe by construction. GetResponseAsync waits for the entire response before giving you anything, so if the primary client throws at any point, OrderedFailoverChatClient has full latitude to retry against the next client and nothing has escaped to the caller yet. The commit point simply doesn't exist for non-streaming calls, because nothing is committed until the whole response is in hand.

Streaming is a different contract entirely. GetStreamingResponseAsync returns an IAsyncEnumerable, and the moment your code does anything observable with the first update, forwards it over a SignalR hub, writes it to an SSE response stream, appends it to a UI buffer, that update has left the building. There is no version of OrderedFailoverChatClient that can reach into your SignalR hub and retract a token it already sent.

Here’s a stripped-down version of the endpoint that taught me this the hard way. It’s a tool-calling assistant that streams a JSON payload describing line items for an order:

app.MapGet("/api/chat/stream", async (HttpContext http, IChatClient failoverClient) =>
{
    http.Response.ContentType = "text/event-stream";
    var messages = new List<ChatMessage>
    {
        new(ChatRole.User, "List three follow-up tasks for this support ticket as JSON.")
    };
    await foreach (var update in failoverClient.GetStreamingResponseAsync(messages))
    {
        // The moment this write happens, the token is out of our hands.
        await http.Response.WriteAsync($"data: {update.Text}\n\n");
        await http.Response.Body.FlushAsync();
    }
});

Say the primary client streams the first eleven tokens of a JSON array, something like:

[{"task": "Escalate to billing", "priority": "high"

and then the connection drops, the provider returns a 503, or the stream just stalls past your timeout. FailoverChatClient does exactly what it's designed to do: it selects the next client and starts a brand new generation from scratch. That new generation has no idea the first eleven tokens ever existed. It might produce:

Here are three follow-up tasks for this ticket:
1. Escalate to billing team
2. Confirm customer contact information
3. Schedule a callback within 24 hours

Your SSE stream, as received by the browser, now contains the concatenation of both:

[{"task": "Escalate to billing", "priority": "high"Here are three follow-up tasks for this ticket:
1. Escalate to billing team
...

That’s not valid JSON, it’s not valid prose, and your frontend’s incremental JSON parser (if you built one, and if you’re streaming structured output you probably did) either throws or silently produces garbage. Nobody paged you. The failover succeeded, from OrderedFailoverChatClient's point of view, because it did retry and it did eventually produce a complete, well-formed response from the backup provider. It's just that half of a different, already-abandoned response reached the user first.

Defensive patterns that actually work

I landed on two approaches, and which one you want depends on how much latency you can spend to buy correctness back.

Pattern one: a commit-delay buffer

Instead of forwarding every update the instant it arrives, hold the first N updates (or the first few hundred characters, whichever you hit first) in memory before you write anything to the caller. If the primary client fails inside that buffering window, nothing has escaped yet, and FailoverChatClient can retry cleanly. Once the buffer threshold passes, you commit to streaming the rest live, on the theory that a stream healthy enough to survive its first few hundred tokens is unlikely to die mid-stream, and even if it does, you accept the smaller risk in exchange for perceived responsiveness.

public static async IAsyncEnumerable<ChatResponseUpdate> BufferedStreamAsync(
    IChatClient client,
    IEnumerable<ChatMessage> messages,
    int bufferThresholdChars = 300,
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    var buffered = new List<ChatResponseUpdate>();
    int charCount = 0;
    bool committed = false;
    await foreach (var update in client.GetStreamingResponseAsync(messages, cancellationToken: cancellationToken))
    {
        if (!committed)
        {
            buffered.Add(update);
            charCount += update.Text?.Length ?? 0;
            if (charCount < bufferThresholdChars)
            {
                continue;
            }
            committed = true;
            foreach (var pending in buffered)
            {
                yield return pending;
            }
            buffered.Clear();
            continue;
        }
        yield return update;
    }
    // Short response that never crossed the threshold, flush whatever we have.
    foreach (var pending in buffered)
    {
        yield return pending;
    }
}

This doesn’t eliminate the blind spot, it shrinks the window where it can happen. For a lot of use cases, shrinking a failure window that used to span an entire response down to the first few hundred characters is a genuinely good tradeoff. It’s not a tradeoff I’d make blind, which is why the checklist below exists.

Pattern two: non-streaming for failover-critical paths, fake the stream yourself

For anything where correctness genuinely can’t tolerate a stitched response (structured tool output, anything a downstream system parses, financial or medical content, contract text), I stopped trying to make streaming failover-safe and instead did the failover-safe thing first, then simulated streaming on top of it:

app.MapGet("/api/chat/safe-stream", async (HttpContext http, IChatClient failoverClient) =>
{
    http.Response.ContentType = "text/event-stream";
    var messages = new List<ChatMessage>
    {
        new(ChatRole.User, "List three follow-up tasks for this support ticket as JSON.")
    };
    // Full failover protection: nothing reaches the caller until we have
    // one complete, internally consistent response.
    ChatResponse response = await failoverClient.GetResponseAsync(messages);
    // Now replay it to the client in chunks so the UI still feels live.
    string text = response.Text;
    const int chunkSize = 24;
    for (int i = 0; i < text.Length; i += chunkSize)
    {
        string chunk = text.Substring(i, Math.Min(chunkSize, text.Length - i));
        await http.Response.WriteAsync($"data: {chunk}\n\n");
        await http.Response.Body.FlushAsync();
        await Task.Delay(15);
    }
});

You lose true time-to-first-token, the user waits for the whole generation before seeing anything move, but you get the entire failover guarantee OrderedFailoverChatClient was built to provide, and the simulated chunking keeps the UI from feeling frozen. I use this pattern specifically for the endpoints that produce structured JSON our own code parses, and the honest, token-by-token streaming path for anything that's pure prose a human is just reading as it arrives, where a stitched response is jarring but not corrupting.

A telemetry and testing checklist before you ship this

I built this list after the JSON-stitching bug, not before, which is exactly why I’m handing it to you now instead of after you find your own version of it.

+---------------------------------------------+------------------------------------------+
| Scenario to test | What to verify |
+---------------------------------------------+------------------------------------------+
| Primary fails before first token | Failover succeeds silently, caller sees |
| | one clean response, zero visible retries |
+---------------------------------------------+------------------------------------------+
| Primary fails after partial JSON emitted | Confirm this produces a broken response, |
| | then confirm your buffering or non-stream |
| | fallback actually prevents it |
+---------------------------------------------+------------------------------------------+
| Network drop mid-stream (not a clean 5xx) | Client timeout triggers failover instead |
| | of hanging indefinitely on a dead socket |
+---------------------------------------------+------------------------------------------+
| Failover during a tool-call argument stream | Downstream parser rejects or safely |
| | recovers, never silently accepts garbage |
+---------------------------------------------+------------------------------------------+
| Concurrent requests during a provider outage | Failover doesn't thundering-herd your |
| | backup provider into its own rate limit |
+---------------------------------------------+------------------------------------------+
| Sticky session client goes unhealthy | Session correctly re-pins to a new client, |
| | doesn't keep retrying a dead one forever |
+---------------------------------------------+------------------------------------------+

Metrics worth putting on a dashboard, not just in a log line:

+--------------------------------+------------------------------------------------+
| Metric | Why it matters |
+--------------------------------+------------------------------------------------+
| time_to_first_token_ms | Buffering strategies push this up, know your |
| | baseline before you tune bufferThresholdChars |
+--------------------------------+------------------------------------------------+
| failover_attempts_total | Tag by from_client and to_client, a spike tells |
| | you a provider is degrading before your users do |
+--------------------------------+------------------------------------------------+
| streaming_response_completed | Ratio of streams that reach natural end vs get |
| | cut off, your best proxy for how often the blind |
| | spot is actually being hit in production |
+--------------------------------+------------------------------------------------+
| stitched_response_suspected | A counter you build yourself, increment it when a |
| | failover event fires after any bytes were already |
| | written to the response stream |
+--------------------------------+------------------------------------------------+

And log, per request, at minimum: which client handled each attempt, how many characters or tokens had already been flushed to the caller at the moment of failure, and whether the eventual response came from a single client or more than one. That last field is the one that would have caught our bug in minutes instead of a support ticket.

Trying this without a second cloud provider

You don’t need two paid API accounts to test any of this. Microsoft.Extensions.AI.Ollama gives you an IChatClient implementation that talks to a local Ollama instance, which makes a perfectly good stand-in backup provider for development and for the failure-injection tests in the checklist above.

# Install Ollama, then pull a small model to act as your backup
ollama pull llama3.2
ollama serve

dotnet add package Microsoft.Extensions.AI.Ollama --prerelease

using Microsoft.Extensions.AI;
#pragma warning disable MEAI001
IChatClient primary = new ChatClientBuilder(cloudClient).Build();
IChatClient localBackup = new OllamaChatClient(
    new Uri("http://localhost:11434"),
    modelId: "llama3.2");
IChatClient failover = new OrderedFailoverChatClient([primary, localBackup]);
// Force a failover locally by pointing "primary" at a bad endpoint
// or stopping your mock server mid-response, then watch localBackup
// pick up the request and confirm your buffering strategy behaves.
ChatResponse response = await failover.GetResponseAsync(
    "Summarize this quarterly report in three bullet points.");
Console.WriteLine(response.Text);

I keep a small console harness around that swaps primary for an HttpClient-backed fake that I can tell to hang, return a 503, or drop the connection after N bytes, specifically so I can run the "primary fails after partial JSON emitted" row from the checklist above on my laptop, offline, before it ever touches a real provider bill. If you don't have a second cloud account to test failover against, this local Ollama path is not a downgrade from testing against real infrastructure, it's honestly a better place to reproduce mid-stream failures on purpose, because you control exactly when the primary dies.

Where I landed

I’m still using the new routing and failover types, all four of them, and I think the team shipped something genuinely useful. OrderedFailoverChatClient alone replaced about 200 lines of Polly policies I'd rather not maintain. But I stream by default now only for prose the user is reading live, and I run structured, parseable, or downstream-consumed output through the non-streaming path with simulated chunking, because a correctness bug in a JSON payload is a much worse Monday than a slightly less snappy time-to-first-token. The announcement got me excited for the right reasons. It just didn't spend enough words on the one sentence that mattered most.

Tags: dotnet, csharp, microsoft-extensions-ai, ai, llm, streaming, resilience