Microsoft.Extensions.AI

A deep-dive walkthrough of Microsoft.Extensions.AI — the modern, provider-neutral .NET abstraction layer for AI services — covering IChatClient and IEmbeddingGenerator, how AI service abstractions let you swap providers, dependency injection registration, the middleware pipeline (logging, caching, telemetry, function invocation, and custom middleware), tool/function calling with AIFunction, streaming, structured output, and integrating it all into ASP.NET Core.

---

Table of Contents

1. Introduction
2. Overview: What It Is and Why It Exists
3. IChatClient
4. IEmbeddingGenerator
5. AI Service Abstractions
6. Dependency Injection
7. Middleware
8. Tool / Function Calling
9. Streaming
10. Integration with ASP.NET Core
11. Testing and Evaluation
12. Microsoft.Extensions.AI vs. the Alternatives
13. Common Pitfalls
14. Quick Reference Table
15. Conclusion

---

Introduction

Every AI provider ships its own SDK with its own types: different message classes, different option names, different streaming shapes, different ways to define tools. Code written directly against one SDK is welded to that vendor, hard to unit-test, and re-implements the same cross-cutting concerns — logging, caching, telemetry, retries, the tool-calling loop — for every project. .NET already solved this class of problem for logging (ILogger), configuration (IConfiguration), HTTP (HttpClient + handlers), and caching (IDistributedCache): define a small interface, let providers implement it, and compose behavior with middleware. Microsoft.Extensions.AI (MEAI) does the same for AI.

IChatClient client = new OpenAIClient(apiKey)
    .GetChatClient("gpt-4o-mini")
    .AsIChatClient();                                   // adapt a provider SDK to the shared interface

ChatResponse response = await client.GetResponseAsync("Explain dependency injection in one sentence.");
Console.WriteLine(response.Text);

Swap that first expression for Azure OpenAI, Ollama, or any other IChatClient, and the second statement doesn't change. That's the whole idea; the rest of this guide is about composing behavior around it (middleware), registering it properly (DI), and operating it in ASP.NET Core.

A note on versions. MEAI went through a preview period in which method names changed (for example, CompleteAsync became GetResponseAsync, and AsChatClient became AsIChatClient), and it continues to gain features. Older blog posts often use the old names. The samples here target the current shape of the API; compile against the version you install and check the release notes. Model names (gpt-4o-mini and similar) are placeholders — keep them in configuration.

---

1. Overview: What It Is and Why It Exists

Two packages, two roles

Microsoft.Extensions.AI.Abstractions   The CONTRACT: interfaces and types — IChatClient, IEmbeddingGenerator,
                                       ChatMessage, ChatOptions, AIFunction, and so on. Library authors and
                                       providers reference only this, so it stays small and stable.

Microsoft.Extensions.AI                The BATTERIES: ChatClientBuilder, and the middleware — logging, distributed
                                       caching, OpenTelemetry, automatic function invocation — plus DI
                                       registration helpers. Applications reference this.

Provider integrations then implement the interfaces:

OpenAI / Azure OpenAI       Microsoft.Extensions.AI.OpenAI (or Azure.AI.OpenAI) — adapters via .AsIChatClient()
                            and .AsIEmbeddingGenerator()
Ollama (local models)       OllamaSharp's client implements IChatClient directly
Azure AI Inference / GitHub Models   Microsoft.Extensions.AI.AzureAIInference
Others                      A growing list of community and vendor packages

What the abstraction gives you

PORTABILITY       Swap providers or models by changing registration, not calling code.
COMPOSABILITY     Add logging, caching, telemetry, rate limiting, and tool invocation as middleware —
                  once, in one place, for every call.
TESTABILITY       Depend on an interface; substitute a fake in unit tests (Section 10).
A COMMON LANGUAGE Library authors can write AI-using libraries that work with any provider.
ECOSYSTEM         Higher-level frameworks — Semantic Kernel and the Microsoft Agent Framework — build on these
                  abstractions, so what you learn here carries over, and you can mix them.

What it deliberately is not

It is NOT an orchestration framework. It has no prompt-template engine, plugin system, memory abstraction, or
agent loop (those are Semantic Kernel / Agent Framework territory — Section 11). It is a thin, stable foundation
of exchange types and a middleware pipeline. Use it when you want control and minimal dependencies.

The layered picture

 Your application code  ──>  IChatClient (the same interface at every layer)
                                 │
                       ┌─────────┴──────────────────────────────┐
                       │  Middleware pipeline (DelegatingChatClient chain)
                       │    logging → telemetry → caching → function invocation → (your own) ...
                       └─────────┬──────────────────────────────┘
                                 │
                       Provider adapter (OpenAI / Azure OpenAI / Ollama / ...)
                                 │
                            The model service

---

2. IChatClient

The core interface

public interface IChatClient : IDisposable
{
    Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default);

    IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default);

    object? GetService(Type serviceType, object? serviceKey = null);
}

Three members: get a full response, get a streamed response, and ask the client for services it exposes (metadata, or the underlying native client). Everything else is extension methods and types around these.

A single prompt, and a full conversation

// Convenience overload: a single user message
ChatResponse quick = await client.GetResponseAsync("Name three C# 12 features.");

// A real conversation: roles and messages
List<ChatMessage> history =
[
    new(ChatRole.System, "You are a concise C# tutor."),
    new(ChatRole.User,   "What is a record?"),
];

ChatResponse response = await client.GetResponseAsync(history);
Console.WriteLine(response.Text);

history.AddRange(response.Messages);                       // keep the assistant's reply in the history
history.Add(new(ChatRole.User, "How is it different from a class?"));
response = await client.GetResponseAsync(history);

Roles: ChatRole.System | User | Assistant | Tool
The model is STATELESS — you hold and resend the history each turn, exactly as in AI APIs with .NET, Section 4.
(Some providers offer service-managed conversation state via ChatOptions.ConversationId; the default model
is "you resend everything.")

ChatResponse: what you get back

string text            = response.Text;                      // the combined text of the assistant message(s)
IList<ChatMessage> msgs = response.Messages;                  // may include tool-call and tool-result messages
UsageDetails? usage    = response.Usage;                     // InputTokenCount, OutputTokenCount, TotalTokenCount
ChatFinishReason? why  = response.FinishReason;              // Stop, Length, ToolCalls, ContentFilter...
string? model          = response.ModelId;

As with every LLM API: log Usage for cost tracking and check FinishReason — a Length finish means the answer was truncated (see AI APIs with .NET, Sections 4 and 8).

ChatOptions: provider-neutral model parameters

var options = new ChatOptions
{
    Temperature = 0.2f,
    MaxOutputTokens = 500,
    TopP = 0.9f,
    StopSequences = ["END_OF_ANSWER"],
    Tools = [ /* AIFunctions — Section 7 */ ],
    ToolMode = ChatToolMode.Auto,                            // Auto | RequireAny | None
    ResponseFormat = ChatResponseFormat.Json,                // or a JSON schema (see structured output below)
    ModelId = "gpt-4o",                                      // override the client's default model for this call
};

ChatResponse response = await client.GetResponseAsync(history, options);

The options are a common denominator: properties a provider doesn't support are ignored or mapped as closely as the adapter can. For provider-specific features, use the escape hatches below. As always, parameter meanings are in AI APIs with .NET, Section 9 — and reasoning models may restrict some of them.

Multimodal input

var message = new ChatMessage(ChatRole.User,
[
    new TextContent("What is in this image?"),
    new DataContent(imageBytes, "image/png"),                // inline bytes
    // new UriContent(new Uri("https://example.com/photo.jpg"), "image/jpeg")   // or a URL
]);

ChatResponse response = await client.GetResponseAsync([message]);

A ChatMessage holds a list of content items — text, data, URIs, function calls and results, usage, reasoning — which is how one message type covers text, images, and tool traffic across providers (subject to what the model supports).

Structured output

public record WeatherReport(string City, double TemperatureC, string Conditions);

ChatResponse<WeatherReport> response = await client.GetResponseAsync<WeatherReport>(
    "Give a plausible weather report for Pune.");

WeatherReport report = response.Result;                      // deserialized, typed

GetResponseAsync<T> derives a JSON Schema from your type, asks the model to conform to it (using schema-constrained output where the provider supports it), and deserializes the result. As always: a schema guarantees shape, not truth — validate the values (see AI APIs with .NET, Section 7), and use TryGetResult if you want to handle malformed output without an exception.

Metadata and escape hatches

ChatClientMetadata? meta = client.GetService<ChatClientMetadata>();
Console.WriteLine($"{meta?.ProviderName} / {meta?.DefaultModelId}");

// Reach the native provider client when you need a provider-specific feature
OpenAI.Chat.ChatClient? native = client.GetService<OpenAI.Chat.ChatClient>();

// Or pass native options through ChatOptions
var options = new ChatOptions
{
    RawRepresentationFactory = _ => new OpenAI.Chat.ChatCompletionOptions { /* provider-specific settings */ }
};

GetService walks through the middleware pipeline down to the provider, so you can always get at the underlying client. The abstraction is a convenience, not a cage.

Exceptions are not unified

MEAI standardizes the REQUEST/RESPONSE shapes, not the errors. An OpenAI-backed client throws the OpenAI SDK's
exception type (ClientResultException), an Azure AI Inference client throws its own, an Ollama client throws
HttpRequestException. If you need provider-independent error handling (retries, circuit breakers, friendly
messages), implement it in middleware or at one boundary in your code (Section 6) rather than scattering
provider-specific catch blocks.

---

3. IEmbeddingGenerator

The embedding counterpart

public interface IEmbeddingGenerator<in TInput, TEmbedding> : IDisposable
    where TEmbedding : Embedding
{
    Task<GeneratedEmbeddings<TEmbedding>> GenerateAsync(
        IEnumerable<TInput> values, EmbeddingGenerationOptions? options = null, CancellationToken cancellationToken = default);

    object? GetService(Type serviceType, object? serviceKey = null);
}

For text embeddings the concrete type is IEmbeddingGenerator<string, Embedding<float>>. (Embeddings and their uses are the subject of this series' Embeddings & Vector Search guide.)

Creating one

// OpenAI
IEmbeddingGenerator<string, Embedding<float>> generator =
    new OpenAIClient(apiKey).GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator();

// Azure OpenAI — the DEPLOYMENT name, with keyless Entra ID auth
IEmbeddingGenerator<string, Embedding<float>> azureGenerator =
    new AzureOpenAIClient(new Uri(endpoint), new DefaultAzureCredential())
        .GetEmbeddingClient("my-embedding-deployment")
        .AsIEmbeddingGenerator();

Using it

// One text -> one vector
ReadOnlyMemory<float> vector = await generator.GenerateVectorAsync("How do I reset my password?");

// Many texts in one request (much more efficient for ingestion)
GeneratedEmbeddings<Embedding<float>> embeddings = await generator.GenerateAsync(chunks);
for (int i = 0; i < chunks.Count; i++)
    Store(chunks[i], embeddings[i].Vector);

// Shorter vectors, where the model supports it
var options = new EmbeddingGenerationOptions { Dimensions = 512 };
var small = await generator.GenerateVectorAsync(text, options);

// Compare two vectors
float similarity = TensorPrimitives.CosineSimilarity(a.Span, b.Span);     // System.Numerics.Tensors

Metadata and middleware

var meta = generator.GetService<EmbeddingGeneratorMetadata>();
// provider name, default model, and default dimension count — handy for sizing a vector index

// Embedding generators have their own builder and middleware, just like chat clients
builder.Services.AddEmbeddingGenerator(sp =>
        new OpenAIClient(apiKey).GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator())
    .UseDistributedCache()                                    // cache embeddings of repeated text
    .UseLogging()
    .UseOpenTelemetry();

Caching embeddings is one of the best uses of middleware: embedding the same chunk or the same popular query
repeatedly is pure waste. Keep the rules from the vector search guide: ONE embedding model for documents and
queries; record the model with stored vectors; embeddings derived from sensitive text are sensitive data.

---

4. AI Service Abstractions

Same code, different providers

// The only provider-specific lines are the construction of the inner client
IChatClient openAi = new OpenAIClient(openAiKey).GetChatClient("gpt-4o-mini").AsIChatClient();

IChatClient azure  = new AzureOpenAIClient(new Uri(endpoint), new DefaultAzureCredential())
                         .GetChatClient("my-gpt-deployment").AsIChatClient();

IChatClient local  = new OllamaApiClient(new Uri("http://localhost:11434"), "llama3.2");     // OllamaSharp

// Everything downstream depends only on IChatClient
async Task<string> SummarizeAsync(IChatClient chat, string text)
    => (await chat.GetResponseAsync($"Summarize in two sentences:\n\n{text}")).Text;

Why this matters in practice

DEVELOP LOCALLY, DEPLOY TO THE CLOUD   Run a local model through Ollama while developing (free, offline), and use
                                       Azure OpenAI in production — selected by configuration, same code.

AVOID LOCK-IN                          Prices, quality, and availability change fast. Moving to a different provider
                                       or model is a registration change, not a rewrite.

MODEL ROUTING                          Register several clients — a small, cheap one and a strong one — and choose
                                       per task (keyed services, Section 5), as recommended in LLM Application
                                       Development, Section 9.

FALLBACKS                              Wrap a primary and a secondary client in middleware that fails over when
                                       the first is unavailable.

LIBRARIES                              Write reusable AI-using code that accepts an IChatClient and works for any host.

The honest limits of the abstraction

COMMON DENOMINATOR      Features that exist on only one provider (a specific built-in tool, a proprietary option)
                        may need the escape hatches (GetService<T>, RawRepresentationFactory) — which then
                        reintroduce provider coupling, so isolate them behind your own interface.

BEHAVIOR DIFFERS        Different models follow instructions, call tools, and format JSON differently. The
                        interface being identical does NOT mean the results are. Re-run your evaluation set when you
                        switch models (see RAG, Section 12).

ERRORS AREN'T UNIFIED   See Section 2.

NEW FEATURES LAG        New provider capabilities arrive in the native SDK first and in the abstraction later.

Treat MEAI as the thing that makes the common 90% portable and testable, while accepting that the last 10% sometimes needs native code.

---

5. Dependency Injection

Register the client in the container

// Program.cs
builder.Services.AddChatClient(sp =>
        new OpenAIClient(builder.Configuration["OpenAI:ApiKey"]!)
            .GetChatClient(builder.Configuration["OpenAI:Model"]!)
            .AsIChatClient())
    .UseLogging()
    .UseOpenTelemetry(sourceName: "MyApp.AI")
    .UseFunctionInvocation();

AddChatClient registers an IChatClient and returns a ChatClientBuilder, onto which you chain middleware (Section 6). Anything in your application can now take an IChatClient in its constructor:

public class SummaryService(IChatClient chat)
{
    public async Task<string> SummarizeAsync(string text, CancellationToken ct) =>
        (await chat.GetResponseAsync($"Summarize in two sentences:\n\n{text}", cancellationToken: ct)).Text;
}

builder.Services.AddScoped<SummaryService>();

Lifetime

The default is SINGLETON, which is right for most cases: provider clients are designed to be created once and
reused (they manage connections and are thread-safe). Don't register a new chat client per request — you lose
connection reuse. Per-request VARIATION belongs in ChatOptions and in the messages you pass, not in a new client.
You can pass a different ServiceLifetime to AddChatClient when you genuinely need scoped behavior.

Configuration and secrets

public class AiOptions
{
    public string Model { get; set; } = "";
    public string Endpoint { get; set; } = "";
    public int MaxOutputTokens { get; set; } = 600;
}

builder.Services.Configure<AiOptions>(builder.Configuration.GetSection("AI"));

- Model names and deployment names live in CONFIGURATION, never in code.
- Secrets: dotnet user-secrets in development; Key Vault or your platform's secret store in production;
  on Azure prefer KEYLESS auth (DefaultAzureCredential with a managed identity) so there is nothing to leak or rotate.
- Never put a provider key in a client app — clients call YOUR backend.

Multiple clients with keyed services

builder.Services.AddKeyedChatClient("fast",
    new OpenAIClient(apiKey).GetChatClient("gpt-4o-mini").AsIChatClient());

builder.Services.AddKeyedChatClient("smart",
    new OpenAIClient(apiKey).GetChatClient("gpt-4o").AsIChatClient());

// Consumers pick by key
public class TriageService([FromKeyedServices("fast")] IChatClient cheap,
                           [FromKeyedServices("smart")] IChatClient strong) { /* route by task difficulty */ }

(Keyed registration builds on .NET's keyed-services support. Each keyed client can have its own middleware pipeline.)

Embedding generators register the same way

builder.Services.AddEmbeddingGenerator(sp =>
        new OpenAIClient(apiKey).GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator())
    .UseDistributedCache()
    .UseLogging();

public class IndexService(IEmbeddingGenerator<string, Embedding<float>> embeddings) { /* ... */ }

Without the host: build the pipeline manually

IChatClient client = new ChatClientBuilder(innerClient)
    .UseLogging(loggerFactory)
    .UseFunctionInvocation()
    .Build(serviceProvider);          // the service provider is optional; some middleware resolves services from it

AddChatClient(...) is just this builder plus container registration. Console apps and tests can use the builder directly.

---

6. Middleware

Cross-cutting concerns, composed — the same idea as ASP.NET Core middleware and HttpClient handlers

A middleware component is an IChatClient that wraps another IChatClient, doing something before and/or after delegating to it. Because every layer implements the same interface, they stack freely and your application code can't tell how many layers sit underneath.

 call  ──> [Logging] ──> [OpenTelemetry] ──> [Caching] ──> [Function invocation] ──> [Provider client] ──> model
 reply <──          <──                 <──            <──                       <──

The base class is DelegatingChatClient, which forwards everything to the inner client; you override only what you need.

The built-in middleware

builder.Services.AddChatClient(innerClient)
    .UseLogging()                                            // logs requests/responses via ILogger
    .UseOpenTelemetry(sourceName: "MyApp.AI",                // traces + metrics (GenAI semantic conventions)
        configure: c => c.EnableSensitiveData = false)       // don't put prompts/responses in telemetry by default
    .UseDistributedCache()                                   // caches responses in an IDistributedCache
    .UseFunctionInvocation();                                // runs the tool-calling loop automatically (Section 7)

LOGGING            Structured logs of calls. Be deliberate about log level: at verbose levels it can record
                   prompts and responses, which may contain personal or confidential data.

OPENTELEMETRY      Distributed traces and metrics following the GenAI semantic conventions: model, token usage,
                   duration, errors. Prompt/response CONTENT capture is opt-in ("sensitive data") — keep it off
                   unless you've decided where that data may go. Add the source to your tracing setup:
                   tracing.AddSource("MyApp.AI").

DISTRIBUTED CACHE  Stores responses keyed by the messages and options. Excellent for deterministic, repeated,
                   non-personal requests (FAQ answers, classification of identical text). Dangerous for
                   personalized or sensitive prompts, and at high temperature a "cache hit" returns the same
                   answer every time. Use with an IDistributedCache such as Redis or the in-memory cache.

FUNCTION INVOCATION  The automatic tool loop (Section 7).

CONFIGURE OPTIONS  Apply default ChatOptions (for example a default temperature) to every call that doesn't
                   set its own — a clean way to centralize defaults.

Ordering matters

Middleware added FIRST is OUTERMOST (it sees the call first and the response last). Later additions sit closer
to the provider. Two consequences worth knowing:

1. FunctionInvokingChatClient calls the client BENEATH it once per tool-calling round trip. Middleware placed
   INSIDE it (added after it) sees EVERY model call; middleware OUTSIDE it (added before it) sees only the single
   overall request.

     .UseOpenTelemetry()       // outermost: one span for the whole user request
     .UseDistributedCache()    // cache FINAL answers — a hit skips the model AND the tools entirely
     .UseFunctionInvocation()  // the tool loop
     .UseRateLimiting(limiter) // innermost: counts EVERY model call, including the extra tool-round trips
     .UseLogging()             // innermost logging: shows each round trip separately

2. Caching outside the tool loop caches the final answer; caching inside it would cache individual model steps.
   Think about which you want — and about whether a cached answer that involved a live tool call (current order
   status) is still valid.

Writing your own middleware: rate limiting

using System.Threading.RateLimiting;

public sealed class RateLimitingChatClient(IChatClient inner, RateLimiter limiter) : DelegatingChatClient(inner)
{
    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        using RateLimitLease lease = await limiter.AcquireAsync(1, cancellationToken);
        if (!lease.IsAcquired)
            throw new InvalidOperationException("AI request rate limit exceeded.");

        return await base.GetResponseAsync(messages, options, cancellationToken);
    }

    public override async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        using RateLimitLease lease = await limiter.AcquireAsync(1, cancellationToken);
        if (!lease.IsAcquired)
            throw new InvalidOperationException("AI request rate limit exceeded.");

        await foreach (var update in base.GetStreamingResponseAsync(messages, options, cancellationToken))
            yield return update;
    }
}

// A friendly extension method, like the built-in Use* methods
public static class RateLimitingChatClientExtensions
{
    public static ChatClientBuilder UseRateLimiting(this ChatClientBuilder builder, RateLimiter limiter) =>
        builder.Use(inner => new RateLimitingChatClient(inner, limiter));
}

Override both methods: a middleware that handles only GetResponseAsync silently skips streaming calls. This one applies the client-side throttling described in AI APIs with .NET, Section 11, in exactly one place for every caller.

Other custom middleware worth writing

PROVIDER-NEUTRAL RETRY / CIRCUIT BREAKER   Catch the provider's transient exceptions and retry with jittered backoff
                                           (see AI APIs with .NET, Section 10) — or integrate Polly. Because errors
                                           aren't unified, this is where you translate provider-specific exceptions.
FAILOVER                                   Try a primary client; on failure, fall back to a secondary provider/model.
TOKEN AND COST ACCOUNTING                  Read response.Usage after each call and record cost per user/feature/tenant.
INPUT GUARDS                               Reject oversized prompts, strip secrets, or run a moderation check before
                                           the request leaves your process.
PII REDACTION                              Scrub personal data from messages on the way out (and from responses).
AUDIT LOGGING                              Record who called which model with how many tokens, for compliance.
DEFAULTS AND POLICY                        Enforce a MaxOutputTokens cap and allowed models for every call.

Because each concern is a separate, testable class, you can add, remove, or reorder them without touching business code — the same payoff ASP.NET Core middleware provides.

---

7. Tool / Function Calling

Describing C# methods as tools: AIFunction

using System.ComponentModel;
using Microsoft.Extensions.AI;

[Description("Gets the current weather for a city.")]
static string GetWeather([Description("City name, for example Pune")] string city)
    => $"It is 31°C and partly cloudy in {city}.";

AIFunction weatherTool = AIFunctionFactory.Create(GetWeather);

AIFunctionFactory.Create turns an ordinary delegate into an AIFunction: it takes the name from the method, the description from [Description], and a JSON schema for the parameters from the C# signature. Those descriptions are what the model reads to decide when and how to call the function — write them like documentation (see Semantic Kernel, Section 3, for plugin-design guidance that applies equally here).

Offer tools, and let the middleware run the loop

IChatClient client = new ChatClientBuilder(innerClient)
    .UseFunctionInvocation()                                 // turns on the automatic tool-calling loop
    .Build();

var options = new ChatOptions { Tools = [weatherTool] };

ChatResponse response = await client.GetResponseAsync("Do I need an umbrella in Pune today?", options);
Console.WriteLine(response.Text);

Without UseFunctionInvocation(), the model's reply would contain a request to call GetWeather (a FunctionCallContent), and you would run it and send the result back — the manual loop from AI APIs with .NET, Section 6. With the middleware, it happens for you:

1. The model replies with FunctionCallContent("GetWeather", { city: "Pune" }).
2. FunctionInvokingChatClient finds the matching AIFunction, deserializes the arguments, and invokes it.
3. It appends a FunctionResultContent message and calls the model again.
4. It repeats until the model returns a normal answer — or a limit is hit.

Configure the loop

.UseFunctionInvocation(configure: c =>
{
    c.MaximumIterationsPerRequest = 5;        // HARD cap on tool-calling rounds — a generous default exists; set your own
    c.AllowConcurrentInvocation = false;      // run multiple requested calls one at a time (or true to parallelize)
    c.IncludeDetailedErrors = false;          // don't leak exception details back into the model's context
    c.TerminateOnUnknownCalls = false;        // what to do when the model asks for a tool that doesn't exist
})

Always set an iteration cap: a confused model can keep requesting tools and burning tokens (see LLM Application Development, Section 11).

Tools that use dependency injection and per-request state

public class OrderTools(IOrderService orders, ICurrentUser user)
{
    [Description("Gets the status of one of the CURRENT customer's orders by order number.")]
    public async Task<string> GetOrderStatusAsync([Description("The order number")] int orderNumber, CancellationToken ct)
    {
        var order = await orders.GetForUserAsync(user.Id, orderNumber, ct);       // authorization in CODE, from the authenticated user
        return order is null ? $"No order {orderNumber} found." : $"Order {order.Number}: {order.Status}.";
    }
}

// Per request: create the tool bound to THIS user
var tools = new OrderTools(orderService, currentUser);
var options = new ChatOptions { Tools = [AIFunctionFactory.Create(tools.GetOrderStatusAsync, name: "get_order_status")] };

Instance methods work, so tools can close over services and — critically — over the authenticated user's identity. The model never supplies "whose data am I acting on"; your code binds that from authentication (the same rule as RAG, Section 8, and Semantic Kernel, Section 3).

Intercepting each call: audit and approval

.UseFunctionInvocation(configure: c =>
{
    c.FunctionInvoker = async (context, cancellationToken) =>
    {
        string name = context.Function.Name;
        logger.LogInformation("Tool call: {Tool} with {Args}", name, context.Arguments);

        if (SensitiveTools.Contains(name) && !await approvals.IsApprovedAsync(name, context.Arguments, cancellationToken))
            return "The user did not approve this action.";                  // the real function never runs

        return await context.Function.InvokeAsync(context.Arguments, cancellationToken);
    };
})

The invoker hook is the place for audit logging, authorization checks, and human-in-the-loop approval for irreversible actions. (Newer MEAI versions also include built-in support for approval-required functions that surface an approval request to the caller — check the current documentation for your version.)

Other tool sources

MCP SERVERS     The C# Model Context Protocol SDK exposes remote MCP tools as AIFunction-compatible objects, so
                tools from the MCP ecosystem drop straight into ChatOptions.Tools.
HOSTED TOOLS    Provider-run tools (web search, code interpreter, file search, hosted MCP) are represented as
                special tool types that only some providers support — useful, but they reintroduce provider coupling.
NO-AUTO MODE    Skip UseFunctionInvocation and handle FunctionCallContent yourself when you want full manual control.

Safety rules (unchanged from every tool-calling system)

- The model PROPOSES calls; your code decides what runs. Validate arguments and authorize in the function.
- Use the invoker hook or approvals for irreversible actions (cancel, send, delete, pay).
- Cap iterations; set token and time budgets.
- Offer ONLY the tools a request needs — fewer tools means fewer mistakes, less risk, fewer input tokens.
- Treat tool OUTPUT as untrusted: a function returning web content or customer text can carry a
  prompt-injection attempt back into the model's context.
- Return SMALL, useful results and clear error text the model can act on.

---

8. Streaming

Stream tokens as they're generated

await foreach (ChatResponseUpdate update in client.GetStreamingResponseAsync(history, options, ct))
{
    Console.Write(update.Text);                              // a small fragment of the reply
}

Streaming makes long answers feel fast, because the first words appear almost immediately (it doesn't change the token count or cost). GetStreamingResponseAsync is on the same IChatClient interface, so every middleware layer supports it.

Keeping the full response while streaming

var updates = new List<ChatResponseUpdate>();

await foreach (var update in client.GetStreamingResponseAsync(history, options, ct))
{
    updates.Add(update);
    await WriteToClientAsync(update.Text);                   // show it to the user as it arrives
}

ChatResponse full = updates.ToChatResponse();                // assemble the complete response (text, usage, tool traffic)
history.AddRange(full.Messages);                             // save it to the conversation history

You need the assembled response to persist the assistant's message into the history and to read Usage for cost tracking — streamed updates are fragments, and the usage information arrives in a final update.

Streaming with tool calls

With UseFunctionInvocation(), streaming and tools work together transparently: when the model streams a
tool-call request, the middleware assembles it, invokes the function, calls the model again, and continues
streaming the final answer — the updates you iterate include text and (if you look for them) the
function-call and function-result content. You don't reassemble fragmented tool-call arguments yourself.

Streaming to a browser: Server-Sent Events

app.MapPost("/chat/stream", async (ChatRequest req, IChatClient chat, HttpContext http, CancellationToken ct) =>
{
    http.Response.ContentType = "text/event-stream";
    http.Response.Headers.CacheControl = "no-cache";

    List<ChatMessage> messages = [new(ChatRole.System, "You are a helpful assistant."), new(ChatRole.User, req.Message)];

    await foreach (var update in chat.GetStreamingResponseAsync(messages, new ChatOptions { MaxOutputTokens = 600 }, ct))
    {
        if (string.IsNullOrEmpty(update.Text)) continue;

        // JSON-encode each fragment so newlines and special characters cannot break the SSE framing
        await http.Response.WriteAsync($"data: {JsonSerializer.Serialize(update.Text)}\n\n", ct);
        await http.Response.Body.FlushAsync(ct);
    }

    await http.Response.WriteAsync("data: [DONE]\n\n", ct);
});

public record ChatRequest(string Message);

(Recent ASP.NET Core versions include built-in server-sent-events result helpers that make this shorter; the manual form above works on any version.)

Streaming rules (same as in AI APIs with .NET, Section 5)

- FLUSH after each chunk, or the server buffers and the user sees nothing until the end.
- Pass the CancellationToken through, so a closed tab stops generation (and spending).
- ERRORS CAN OCCUR MID-STREAM, after HTTP 200 and some text were sent — you can't change the status code any more;
  emit an error event the client understands, and design the UI for partial answers.
- Middleware must override GetStreamingResponseAsync as well as GetResponseAsync (Section 6) — a custom middleware
  that forgets the streaming path silently skips its logic for streamed calls.
- Caching streamed responses stores and replays the sequence of updates; make sure your cache and privacy rules
  fit that.

---

9. Integration with ASP.NET Core

A complete minimal setup

using Microsoft.Extensions.AI;
using OpenAI;

var builder = WebApplication.CreateBuilder(args);

// 1. Telemetry
builder.Services.AddOpenTelemetry()
    .WithTracing(t => t.AddSource("MyApp.AI").AddAspNetCoreInstrumentation().AddOtlpExporter())
    .WithMetrics(m => m.AddMeter("MyApp.AI").AddOtlpExporter());

// 2. Cache for AI responses (use Redis in production)
builder.Services.AddDistributedMemoryCache();

// 3. The chat client and its middleware pipeline
builder.Services.AddChatClient(sp =>
        new OpenAIClient(builder.Configuration["OpenAI:ApiKey"]!)
            .GetChatClient(builder.Configuration["OpenAI:Model"]!)
            .AsIChatClient())
    .UseOpenTelemetry(sourceName: "MyApp.AI")
    .UseDistributedCache()
    .UseFunctionInvocation(configure: c => c.MaximumIterationsPerRequest = 5)
    .UseLogging();

// 4. Embeddings, if you use retrieval
builder.Services.AddEmbeddingGenerator(sp =>
        new OpenAIClient(builder.Configuration["OpenAI:ApiKey"]!)
            .GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator())
    .UseDistributedCache();

// 5. Protect your own endpoint — it's both a reliability measure and a COST-control measure
builder.Services.AddRateLimiter(o =>
{
    o.AddPolicy("ai", ctx => RateLimitPartition.GetFixedWindowLimiter(
        ctx.User.Identity?.Name ?? ctx.Connection.RemoteIpAddress?.ToString() ?? "anonymous",
        _ => new FixedWindowRateLimiterOptions { PermitLimit = 20, Window = TimeSpan.FromMinutes(1) }));
    o.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
});

builder.Services.AddAuthentication().AddJwtBearer();
builder.Services.AddAuthorization();

var app = builder.Build();
app.UseAuthentication();
app.UseAuthorization();
app.UseRateLimiter();

A request/response endpoint

app.MapPost("/summarize", async (SummarizeRequest req, IChatClient chat, CancellationToken ct) =>
{
    // Cap what callers can send: an endpoint that forwards arbitrary-length text to a paid model invites abuse
    if (string.IsNullOrWhiteSpace(req.Text) || req.Text.Length > 20_000)
        return Results.BadRequest("Text is required and must be under 20,000 characters.");

    var options = new ChatOptions { Temperature = 0.2f, MaxOutputTokens = 300 };

    ChatResponse response = await chat.GetResponseAsync(
        [
            new(ChatRole.System, "Summarize the user's text in at most three sentences."),
            new(ChatRole.User, req.Text)
        ],
        options, ct);

    return response.FinishReason == ChatFinishReason.Length
        ? Results.Problem("The summary was truncated.")
        : Results.Ok(new { summary = response.Text, tokens = response.Usage?.TotalTokenCount });
})
.RequireAuthorization()
.RequireRateLimiting("ai");

public record SummarizeRequest(string Text);

In a minimal API, a CancellationToken parameter is bound to the request-aborted token, so a disconnected client cancels the model call. Add the streaming endpoint from Section 8 the same way.

A tool-using chat endpoint with per-request identity

app.MapPost("/chat", async (ChatRequest req, IChatClient chat, ICurrentUser user, IOrderService orders,
                            IConversationStore store, CancellationToken ct) =>
{
    // Tools bound to THIS authenticated user — the model never chooses whose data to touch
    var orderTools = new OrderTools(orders, user);
    var options = new ChatOptions
    {
        MaxOutputTokens = 600,
        Tools = [AIFunctionFactory.Create(orderTools.GetOrderStatusAsync, name: "get_order_status")]
    };

    List<ChatMessage> history = await store.LoadAsync(req.ConversationId, user.Id, ct)
        ?? [new(ChatRole.System, "You are a concise, friendly support assistant for ACME.")];

    history.Add(new(ChatRole.User, req.Message));
    history = TrimToBudget(history);                         // bound the history (see AI APIs with .NET, Section 8)

    ChatResponse response = await chat.GetResponseAsync(history, options, ct);   // the pipeline runs the tool loop

    history.AddRange(response.Messages);
    await store.SaveAsync(req.ConversationId, user.Id, history, ct);

    return Results.Ok(new { reply = response.Text });
})
.RequireAuthorization()
.RequireRateLimiting("ai");

Persisting history: ChatMessage is serializable with System.Text.Json using AIJsonUtilities.DefaultOptions, so you
can store conversations in a database or cache keyed by conversation ID (and user ID — never trust a conversation
ID alone to authorize access). Don't keep history in server memory if you scale out or restart.

Operational checklist

SECRETS            user-secrets in dev; Key Vault or keyless managed identity in production.
OBSERVABILITY      UseOpenTelemetry + UseLogging; leave sensitive-data capture OFF unless deliberately decided.
RESILIENCE         Timeouts, retries with jittered backoff, and circuit breaking — in middleware or at the HttpClient
                   level; provider SDKs also have their own retry settings (AI APIs with .NET, Section 10).
COST               Log response.Usage per feature/user; cap MaxOutputTokens and tool iterations; cache deterministic
                   requests; route easy work to a cheap keyed client.
SECURITY           Authorize in tool code from the authenticated user; validate tool arguments; treat tool output and
                   user text as untrusted; per-user rate limits; input size caps.
HEALTH             A lightweight health check (e.g. a tiny model call or a provider reachability check) on a schedule,
                   not on every probe — probes that call a paid model add cost.

---

10. Testing and Evaluation

Depend on the interface — fake it in tests

Because everything depends on IChatClient, unit tests don't need network access, API keys, or money:

public sealed class FakeChatClient(string reply) : IChatClient
{
    public List<IList<ChatMessage>> Calls { get; } = [];

    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        Calls.Add(messages.ToList());                         // record what was sent so tests can assert on it
        return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, reply)));
    }

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        await Task.CompletedTask;
        yield return new ChatResponseUpdate(ChatRole.Assistant, reply);
    }

    public object? GetService(Type serviceType, object? serviceKey = null) => null;
    public void Dispose() { }
}

[Fact]
public async Task Summarize_sends_text_and_returns_reply()
{
    var fake = new FakeChatClient("A short summary.");
    var service = new SummaryService(fake);

    string result = await service.SummarizeAsync("A very long article...", CancellationToken.None);

    Assert.Equal("A short summary.", result);
    Assert.Contains(fake.Calls[0], m => m.Text.Contains("A very long article"));
}

What to test where

UNIT TESTS (fake client)     Your orchestration logic: prompt assembly, history trimming, tool wiring, error handling,
                             citation validation, what gets sent to the model.
TOOL TESTS                   Tools are ordinary C# methods — test them directly, including authorization and
                             argument-validation edge cases.
MIDDLEWARE TESTS             Custom middleware wrapped around a fake inner client: does the rate limiter throttle? does
                             the redactor scrub? does the streaming path behave the same as the non-streaming one?
EVALUATION (real model)      Because outputs are probabilistic, quality needs a labeled test set run against the real
                             model: faithfulness, relevance, correct refusals (see RAG, Section 12). The
                             Microsoft.Extensions.AI.Evaluation libraries provide evaluators and reporting for this —
                             check the current packages.

Unit tests with fakes prove your code works; evaluations with the real model prove your prompts and pipeline produce good answers. You need both, and neither replaces the other.

---

11. Microsoft.Extensions.AI vs. the Alternatives

Where it sits

RAW PROVIDER SDK          Full provider-specific control, no portability, you build everything yourself.

Microsoft.Extensions.AI   A thin, stable, provider-neutral foundation: IChatClient, IEmbeddingGenerator,
                          middleware, tool abstractions. Minimal dependencies, maximum control.

SEMANTIC KERNEL           Orchestration built on top: plugins, prompt templates, filters, memory/vector stores,
                          agents, multi-agent orchestration. (This series' Semantic Kernel guide.)

MICROSOFT AGENT           Agent- and workflow-focused framework from the Semantic Kernel and AutoGen teams, also
FRAMEWORK                 built on these abstractions. Check its current maturity for new agent-centric work.

Choosing

A handful of model calls, simple tools, a strong preference for few dependencies, or you're writing a library
   -> Microsoft.Extensions.AI. It's often all you need, and it composes with everything above it.

A plugin-based assistant, reusable prompt assets, rich governance filters, agents
   -> Semantic Kernel (or the Agent Framework), which sit on the same abstractions — so starting with MEAI
      doesn't strand you; you can adopt a framework later and keep your IChatClient code.

Provider-specific features that the abstraction doesn't expose
   -> use the native SDK for those calls (via GetService<T> or directly), isolated behind your own interface.

The principle

Start with the thinnest layer that does the job. MEAI gives you portability, testability, and cross-cutting middleware for almost no cost — and because the larger frameworks build on it, it is the safest foundation to start from, whatever you add later.

---

12. Common Pitfalls

Pitfall	Why it hurts	Better approach
Following preview-era tutorials	Method and type names changed (CompleteAsync → GetResponseAsync, AsChatClient → AsIChatClient); old samples don't compile	Match samples to your installed version; read release notes (Introduction)
Writing against a provider SDK in business code	Lock-in and hard-to-test code	Depend on IChatClient; keep provider construction in one registration spot (Section 4)
Creating a chat client per request	Loses connection reuse; wasteful	Register once (singleton) and vary behavior with ChatOptions and messages (Section 5)
Hard-coding model names and keys	Rotations and model changes require redeploys; leaks	Configuration, secret stores, keyless auth on Azure (Section 5)
Forgetting UseFunctionInvocation()	The model's tool-call request comes back as content and nothing runs	Add the middleware, or handle FunctionCallContent manually (Section 7)
No iteration cap on the tool loop	A confused model loops, burning tokens and money	Set MaximumIterationsPerRequest; add token and time budgets (Section 7)
Model-supplied identity in tool parameters	The model (or an injected prompt) can act as someone else	Bind the authenticated user into the tool instance; authorize in code (Section 7)
Auto-running irreversible tools	Mistakes or injection cancel orders or send email	Approval logic in the FunctionInvoker hook or built-in approval support (Section 7)
Middleware ordering by accident	Caches the wrong thing; counts only one model call per tool round; logs the wrong layer	Remember first-added is outermost; place per-call concerns inside the tool loop (Section 6)
Custom middleware that overrides only GetResponseAsync	Streaming calls silently bypass it	Override both response methods (Sections 6, 8)
Caching personalized or sensitive prompts	Leaks one user's answers to another; stale tool-based answers	Cache only deterministic, non-personal requests; scope keys carefully (Section 6)
Capturing prompts in telemetry by default	Personal or confidential data lands in logs and traces	Keep sensitive-data capture off unless deliberately decided (Section 6)
Ignoring Usage and FinishReason	No cost visibility; truncated answers go unnoticed	Log usage; handle Length finishes (Section 2)
Assuming identical behavior across providers	The interface is the same; models and options are not	Re-run your evaluation set when switching models (Section 4)
Scattering provider-specific catch blocks	Errors aren't unified; code becomes provider-coupled	Translate exceptions in one middleware/boundary (Sections 2, 6)
Holding conversation history in server memory	Lost on restart; breaks when scaled out	Persist per conversation and user; bound the history (Section 9)
Trusting a conversation ID alone	Users can read each other's conversations	Key storage by both conversation and authenticated user (Section 9)
Unbounded request size and rate	One user or bug can exhaust your quota and budget	Input size caps and per-user rate limits on your endpoints (Section 9)
Only unit tests with fakes	They prove your code, not your prompts' quality	Add an evaluation set against the real model (Section 10)

---

Quick Reference Table

Concept	API	Purpose
Chat client	IChatClient	Provider-neutral chat completion interface
Single call	await client.GetResponseAsync("prompt")	One user message, full response
Conversation	GetResponseAsync(List<ChatMessage>, ChatOptions?)	Roles: System, User, Assistant, Tool
Response	ChatResponse → .Text, .Messages, .Usage, .FinishReason	Result, tokens, why it stopped
Options	ChatOptions { Temperature, MaxOutputTokens, Tools, ToolMode, ... }	Provider-neutral parameters
Structured output	GetResponseAsync<T>(...) → .Result	Typed result from a schema derived from T
Streaming	GetStreamingResponseAsync(...) → ChatResponseUpdate	Incremental output
Assemble stream	updates.ToChatResponse()	Full response from fragments (history, usage)
Escape hatches	GetService<T>(), ChatOptions.RawRepresentationFactory	Reach native client and options
Embeddings	IEmbeddingGenerator<string, Embedding<float>>	Provider-neutral embedding generation
One vector	await generator.GenerateVectorAsync(text)	ReadOnlyMemory<float>
Batch embeddings	await generator.GenerateAsync(texts)	Efficient ingestion
Adapt a provider	.AsIChatClient() / .AsIEmbeddingGenerator()	Wrap OpenAI/Azure OpenAI SDK clients
DI registration	services.AddChatClient(...) / AddEmbeddingGenerator(...)	Register and get a builder for middleware
Keyed clients	AddKeyedChatClient("fast", ...) + [FromKeyedServices]	Multiple models, routing
Manual pipeline	new ChatClientBuilder(inner).Use...().Build()	Compose without the host
Built-in middleware	UseLogging, UseOpenTelemetry, UseDistributedCache, UseFunctionInvocation	Cross-cutting behavior
Custom middleware	Subclass DelegatingChatClient; builder.Use(inner => ...)	Rate limiting, retry, redaction, accounting
Tool definition	AIFunctionFactory.Create(method) + [Description]	C# method → model-callable tool
Tool loop	UseFunctionInvocation(configure: c => c.MaximumIterationsPerRequest = 5)	Automatic tool calling, with a cap
Call interception	FunctionInvoker hook	Audit, authorization, approval
Testing	Fake IChatClient; Microsoft.Extensions.AI.Evaluation	Unit-test code; evaluate quality

---

Conclusion

Microsoft.Extensions.AI does for AI what ILogger, IConfiguration, and HttpClient handlers did for their domains: a small, stable set of interfaces — IChatClient and IEmbeddingGenerator — that providers implement and applications depend on, plus a middleware pipeline that lets you add logging, telemetry, caching, rate limiting, and the tool-calling loop once, in one place, for every call. Registered through dependency injection and configured from IConfiguration, it makes AI features behave like the rest of an ASP.NET Core application: swappable providers, testable with fakes, observable through OpenTelemetry, and governed by the same cross-cutting machinery you already use.

Used well, it is also deliberately modest. It standardizes the common shapes — messages, options, responses, tool definitions — but not provider behavior or provider errors, so you still evaluate when you change models, translate exceptions in one place, and reach for the native SDK through the escape hatches when you need a provider-specific feature. And it gives you no prompt engine, memory, or agent loop; those belong to Semantic Kernel and the Microsoft Agent Framework, which are built on top of these same abstractions. That is precisely why it is the right place to start: you get portability, testability, and clean cross-cutting behavior for almost no cost, you can adopt a larger framework later without rewriting your IChatClient code, and everything the earlier guides in this series insisted on — bounding loops, authorizing in code, treating model and tool output as untrusted, capping cost, and measuring quality — slots naturally into a middleware pipeline where it is written once and applied everywhere.

---

Found this useful? Feel free to star the repo, open an issue with corrections, or share the "we switched providers by changing one line of registration — and then discovered the evaluation set was the part that actually mattered" story that made the case for both the abstraction and the tests click.