ChatClient Middleware vs Agent Middleware in Microsoft Agent Framework: What I Learned After…

ChatClient Middleware vs Agent Middleware in Microsoft Agent Framework: What I Learned After Actually Building the Thing

I ran into Jesse Liberty’s post on ChatClient middleware versus Agent middleware a couple of weeks ago, right when I was wiring up logging and tool-call tracking for a small research agent of my own. It’s a short, clean piece and it gets the core idea right: there are two different interception points in Microsoft Agent Framework, one at the model call level and one at the agent run level. But it stops right where things get interesting. It shows three code snippets, names the two layers, and calls it done.

I wanted more. I wanted to know why there are actually four middleware types, not two. I wanted to know what happens when you stack five of them and one of them throws. I wanted to know if this even works if you don’t have an Azure subscription, because I don’t always want to burn API credits just to test a logging wrapper. So I spent a few days pulling apart the Microsoft Learn docs, reading the actual DelegatingChatClient source, and rebuilding the example with a local model running through Ollama instead of Azure AI Foundry. This is the writeup of what I found, including the parts that tripped me up.

If you’re building anything non-trivial with Microsoft Agent Framework (MAF from here on, since I’m not typing that out fifty times), this distinction is not academic. Get it wrong and you’ll either drown in log noise from every retry the framework does under the hood, or you’ll have a rate limiter that silently does nothing the moment someone calls your agent with streaming enabled. I hit both of those. More on that later.

Why the two-layer thing confused me at first

Here’s the mental trap I fell into immediately: I assumed “agent” and “chat client” were basically synonyms, since in a lot of toy examples an agent is just “a chat client with a system prompt and some tools.” So why would you need middleware in two different places?

The answer, once it clicked, is that an agent run and a model call are not the same event, and they don’t happen in a one-to-one relationship. A single call to agent.RunAsync() might trigger the model three, four, or more times if there's a tool-calling loop involved. The agent asks the model something, the model asks for a tool, the agent runs the tool, the agent sends the result back to the model, the model asks for another tool, and so on until the model finally produces a plain-text answer.

If your middleware sits at the IChatClient level, it sees every one of those individual round trips to the model. That's exactly what you want if you're debugging prompt construction, counting tokens per call, or caching identical requests. If your middleware sits at the agent level, it sees the whole thing as one unit of work: one user message in, one final response out, regardless of how many model calls or tool calls happened underneath. That's what you want for session bookkeeping, user-facing latency metrics, or deciding whether to allow the run at all before it starts.

Once I had that framing, the rest of the API surface stopped feeling arbitrary.

The four middleware types (not two)

Liberty’s article frames this as a two-way split: ChatClient middleware and Agent middleware, with function-invocation middleware mentioned almost as an afterthought inside the agent example. Going through the actual Microsoft Learn concepts page, the framework documents four distinct extension points, and they map to four different moments in the request lifecycle.

+----------------------------+------------------------------------------+---------------------------------------+
| Middleware Type | Intercepts | Typical Use |
+----------------------------+------------------------------------------+---------------------------------------+
| IChatClient middleware | A single call into the model provider | Logging raw prompts, caching, retries, |
| | (one GetResponseAsync/GetStreaming call) | rate limiting, redaction |
+----------------------------+------------------------------------------+---------------------------------------+
| Agent run middleware | One full agent.RunAsync() invocation | Session tracking, auth checks, overall |
| | (may contain several model calls) | latency, blocking a request outright |
+----------------------------+------------------------------------------+---------------------------------------+
| Agent run streaming | One full agent.RunStreamingAsync() call | Same as above, but for the streaming |
| middleware | | code path, which is a separate hook |
+----------------------------+------------------------------------------+---------------------------------------+
| Function calling middleware | A single tool/function invocation | Argument validation, per-tool logging, |
| | requested by the model | approval gates, mocking tools in tests |
+----------------------------+------------------------------------------+---------------------------------------+

That fourth row matters a lot more than the original article suggests. Function calling middleware is not a variant of agent middleware, it’s its own hook, and it fires once per tool call, which in a multi-step agent run could be zero times or could be a dozen times. If you only instrument agent-run middleware, you get one log line for the whole run. If you only instrument function-calling middleware, you never find out how long the model itself took to think between tool calls. You usually want both, layered.

Layer one: IChatClient middleware, properly explained

The IChatClient interface is the lowest common denominator in Microsoft.Extensions.AI. Anything that can turn a list of chat messages into a response implements it: Azure OpenAI, plain OpenAI, Anthropic, a local Ollama model, or a fake test double you write yourself. Middleware at this layer wraps one IChatClient around another, using the decorator pattern through a base class called DelegatingChatClient.

DelegatingChatClient lives in Microsoft.Extensions.AI.Abstractions and does exactly one job: it holds a reference to an inner IChatClient and forwards every call to it unless you override something. The built-in middleware classes that ship with the framework, things like LoggingChatClient, CachingChatClient, and FunctionInvokingChatClient, are all just subclasses of it. That was a genuinely useful thing to learn, because it means writing your own middleware is not some special framework trick, it's plain inheritance.

Here’s the fluent way to compose it, which is what most people reach for first:

using Microsoft.Extensions.AI;

IChatClient client = new OpenAIChatClient(
        new OpenAI.OpenAIClient(openAiApiKey),
        "gpt-4o")
    .AsBuilder()
    .UseLogging(loggerFactory) // outermost: sees every attempt, including retries
    .UseDistributedCache(cache) // serves repeat prompts without hitting the model
    .UseFunctionInvocation() // innermost: actually runs the tool-calling loop
    .Build();

Order is not cosmetic here, and this is the first place I made a mistake worth mentioning. I originally put UseDistributedCache before UseLogging in the chain, thinking it didn't matter since both were "just wrappers." It matters a lot. Each Use...() call wraps everything registered before it, so the last one you call ends up closest to the model, and the first one you call ends up as the outermost layer that sees a call first and finishes last. If logging sits inside the cache layer, cache hits never touch your logger, because the cache short-circuits the call before it ever reaches the inner client. I only noticed because my request counts in the logs didn't match my actual OpenAI billing, and it took an embarrassingly long time to realize the cache was the reason.

Once I understood the ordering rule, the fix was trivial: logging goes outermost so it sees everything, caching goes after it so cache hits are still logged, and function invocation goes innermost since it’s the thing that actually talks to the model.

If you need something the built-ins don’t cover, you extend DelegatingChatClient directly. I ended up writing a rate limiter, since none of the built-in middleware does that out of the box:

using System.Collections.Concurrent;
using Microsoft.Extensions.AI;

public class RateLimitingChatClient : DelegatingChatClient
{
    private readonly int _maxRequestsPerMinute;
    private readonly ConcurrentQueue<DateTimeOffset> _requestTimestamps = new();
    public RateLimitingChatClient(IChatClient innerClient, int maxRequestsPerMinute)
        : base(innerClient)
    {
        _maxRequestsPerMinute = maxRequestsPerMinute;
    }
    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        EnforceRateLimit();
        return await base.GetResponseAsync(messages, options, cancellationToken);
    }
    public override async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        EnforceRateLimit();
        await foreach (var update in base.GetStreamingResponseAsync(messages, options, cancellationToken))
        {
            yield return update;
        }
    }
    private void EnforceRateLimit()
    {
        var now = DateTimeOffset.UtcNow;
        var windowStart = now.AddMinutes(-1);
        while (_requestTimestamps.TryPeek(out var oldest) && oldest < windowStart)
            _requestTimestamps.TryDequeue(out _);
        if (_requestTimestamps.Count >= _maxRequestsPerMinute)
            throw new InvalidOperationException(
                $"Rate limit exceeded, max {_maxRequestsPerMinute} requests per minute.");
        _requestTimestamps.Enqueue(now);
    }
}

Notice I overrode both GetResponseAsync and GetStreamingResponseAsync. That's the second mistake I made the first time around: I only overrode the non-streaming method, tested it with a plain RunAsync call, saw it working, and moved on. Then a teammate wired the same client into a streaming UI and the rate limiter simply never fired, because the streaming path is a completely separate method on the interface and DelegatingChatClient does not automatically route one through the other. If you only override half the interface, the other half quietly bypasses your middleware. This is called out as a known gotcha in the community writeups I found, and I can confirm from experience that it is exactly as annoying as it sounds.

Layer two: Agent middleware, all three of its flavors

Agent middleware wraps the AIAgent itself rather than the underlying model client, and it's registered the same way, through AsBuilder() and Use(...), just on the agent instead of the chat client.

async Task<AgentResponse> LogAgentRun(
    IEnumerable<ChatMessage> messages,
    AgentSession? session,
    AgentRunOptions? options,
    AIAgent innerAgent,
    CancellationToken cancellationToken)
{
    var start = DateTimeOffset.UtcNow;
    var response = await innerAgent.RunAsync(messages, session, options, cancellationToken)
        .ConfigureAwait(false);
    var elapsed = DateTimeOffset.UtcNow - start;
    Console.WriteLine($"Run completed in {elapsed.TotalMilliseconds}ms, " +
                       $"{response.Messages.Count} messages returned.");
    return response;
}
async IAsyncEnumerable<AgentResponseUpdate> LogAgentRunStreaming(
    IEnumerable<ChatMessage> messages,
    AgentSession? session,
    AgentRunOptions? options,
    AIAgent innerAgent,
    [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken)
{
    var updates = new List<AgentResponseUpdate>();
    await foreach (var update in innerAgent.RunStreamingAsync(messages, session, options, cancellationToken))
    {
        updates.Add(update);
        yield return update;
    }
    Console.WriteLine($"Streaming run completed, {updates.ToAgentResponse().Messages.Count} messages.");
}
var agent = new ChatClientAgent(chatClient, instructions: "You are a helpful research assistant.")
    .AsBuilder()
        .Use(runFunc: LogAgentRun, runStreamingFunc: LogAgentRunStreaming)
    .Build();

Notice the docs explicitly recommend registering both runFunc and runStreamingFunc. If you only give it the non-streaming version, the framework will use that same delegate for streaming calls too, which usually means your streaming responses get buffered into a single blob before your middleware sees them, defeating the purpose of streaming in the first place. Same lesson as the ChatClient layer, just at a different altitude: cover both code paths or accept that one of them is going to behave unexpectedly.

Function calling middleware is the third flavor, and it’s the one Liberty’s article actually demonstrates, just without naming it as its own category:

async ValueTask<object?> LogFunctionCalls(
    AIAgent agent,
    FunctionInvocationContext context,
    Func<FunctionInvocationContext, CancellationToken, ValueTask<object?>> next,
    CancellationToken cancellationToken)
{
    var start = DateTimeOffset.UtcNow;
    Console.WriteLine($"Invoking tool '{context.Function.Name}' with arguments {context.Arguments}");
var result = await next(context, cancellationToken);
    Console.WriteLine($"Tool '{context.Function.Name}' returned in " +
                       $"{(DateTimeOffset.UtcNow - start).TotalMilliseconds}ms");
    return result;
}
var agentWithToolLogging = agent
    .AsBuilder()
        .Use(LogFunctionCalls)
    .Build();

One detail I like a lot: FunctionInvocationContext has a Terminate flag. Setting it to true inside your middleware stops the tool-calling loop right there, which is a clean way to build a hard stop, say, a tool that touches billing data and requires a human approval step before the agent is allowed to keep going. That's a much better pattern than throwing an exception and hoping something upstream catches it gracefully.

Something worth flagging if you work across languages: MAF ships SDKs for .NET, Python, and Go, and the Python side expresses the exact same four concepts through async context managers instead of delegates. If your team is polyglot, or you’re just reading Python examples in the docs while writing C#, it helps to see them side by side:

+---------------------------+---------------------------------------------+
| .NET | Python equivalent |
+---------------------------+---------------------------------------------+
| Use(runFunc, runStreamFunc)| middleware=[AgentMiddleware subclass or a |
| on AgentBuilder | plain async function decorated with |
| | @agent_middleware] |
+---------------------------+---------------------------------------------+
| Use(functionMiddleware) | @function_middleware decorator, or a class |
| | implementing process(context, call_next) |
+---------------------------+---------------------------------------------+
| Use(getResponseFunc, ...) | @chat_middleware decorator on a ChatClient |
| on ChatClientBuilder | |
+---------------------------+---------------------------------------------+
| context.Terminate = true | raise MiddlewareTermination(result=...) |
| on FunctionInvocationContext| after setting context.result |
+---------------------------+---------------------------------------------+

The Python API also exposes a slightly richer context object than what I initially expected, with fields like function_invocation_kwargs for passing tenant IDs or request metadata down into tool calls without threading them through every function signature by hand. I don't think the .NET side has a direct equivalent yet, and honestly I miss it when I switch back.

How the layers actually nest when you stack everything

This is the part neither the original article nor most of the blog posts I found actually draw out, and it’s the part I most wanted a clear picture of before shipping anything. If you register two agent-level middlewares and then call an agent that also has a run-level override, they nest like Russian dolls, outermost registered first:

Agent middleware A1
  -> Agent middleware A2
    -> Run-level middleware R1
      -> Run-level middleware R2
        -> the actual agent logic (which may call ChatClient middleware
           and function-calling middleware internally, once per model
           round trip and once per tool call, respectively)
        <- R2 post-processing
      <- R1 post-processing
    <- A2 post-processing
  <- A1 post-processing

Every middleware function gets a chance to run code before calling next() and after it returns, so you effectively get pre- and post-hooks for free just by structuring your function that way. This is why a lot of the logging examples print something, call next() or await call_next(), then print again: the second print only fires once everything inside has finished, so you get accurate timing without a separate stopwatch mechanism if you don't want one.

The practical takeaway I took from mapping this out: put your broadest, most expensive checks (auth, rate limiting, blocking on sensitive content) as the outermost agent middleware, so you reject bad requests before any model calls or tool calls happen at all. Put your narrowest, cheapest instrumentation (per-tool timing, per-model-call token counts) as close to the actual work as possible. Mixing that up means you pay for an expensive model call before your cheap security check ever runs, which is both a cost problem and, depending on what you’re guarding against, a real security problem.

Building it for real, with a local model instead of a paid API

Every example I found online, including Liberty’s, assumes you have an Azure AI Foundry project and a deployment name ready to go. That’s a reasonable assumption for a lot of readers, but I wanted to actually run this on my machine without touching a billing dashboard, so I swapped in Ollama.

The key fact that makes this easy: OllamaApiClient from the OllamaSharp package implements IChatClient directly, from Microsoft.Extensions.AI.Abstractions, and from IEmbeddingGenerator for embeddings too. That means everything above, AsBuilder(), DelegatingChatClient, UseLogging(), all of it, works completely unchanged. You are not building a second code path for local development, you're swapping one constructor call.

First, pull a model and start Ollama:

# install Ollama if you haven't: https://ollama.com/download
ollama pull llama3.2
ollama serve

Then the C# side:

// dotnet add package OllamaSharp
// dotnet add package Microsoft.Extensions.AI.Abstractions
// dotnet add package Microsoft.Extensions.Logging.Console

using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging;
using OllamaSharp;
using var loggerFactory = LoggerFactory.Create(b => b.AddConsole().SetMinimumLevel(LogLevel.Debug));
IChatClient baseClient = new OllamaApiClient(new Uri("http://localhost:11434"), "llama3.2");
IChatClient instrumentedClient = baseClient
    .AsBuilder()
    .UseLogging(loggerFactory)
    .UseFunctionInvocation()
    .Build();
var agent = new ChatClientAgent(
    instrumentedClient,
    instructions: "You are a concise research assistant. Use tools when helpful.");
var agentWithToolLogging = agent
    .AsBuilder()
    .Use(LogFunctionCalls) // the function-calling middleware from earlier
    .Build();
var response = await agentWithToolLogging.RunAsync("Summarize the tradeoffs of REST versus gRPC in three bullet points.");
Console.WriteLine(response.Text);

That’s the entire swap. No Azure credential, no deployment name, no API key. If you want the Docker route instead of a native Ollama install, this is the same setup wrapped in a container:

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
volumes:
  ollama_data:

docker compose up -d
docker exec -it $(docker compose ps -q ollama) ollama pull llama3.2

Point the same OllamaApiClient at http://localhost:11434 and the C# code above doesn't change at all. I've been using this setup for iterating on middleware logic specifically because it's fast to reset (delete the container, pull again, done) and I don't have to think about token costs while I'm debugging a logging format string for the tenth time.

One honest caveat: smaller local models are noticeably worse at reliable, structured tool calling than GPT-4-class models, so if your middleware testing depends heavily on multi-step tool-calling loops, expect some flakiness that has nothing to do with your code. I still think it’s the right default for developing and testing the middleware plumbing itself, then switching to a hosted model for the final quality pass on actual agent behavior.

The gotchas, collected in one place

I scattered a few of these through the sections above, but they’re worth restating together since they’re the actual reason to read past the “here’s the two layers” summary you’ll find elsewhere.

+-------------------------------------------+---------------------------------------------------+
| Gotcha | What actually happens |
+-------------------------------------------+---------------------------------------------------+
| Only overriding GetResponseAsync | Streaming calls silently skip your middleware |
| on a custom DelegatingChatClient | entirely, since it's a different interface method |
+-------------------------------------------+---------------------------------------------------+
| Registering UseLogging() after | Cache hits never reach the logger, so your request |
| UseDistributedCache() in the chain | counts undercount actual traffic |
+-------------------------------------------+---------------------------------------------------+
| Only providing runFunc to agent middleware | Streaming agent runs get coerced through the non- |
| Use(), skipping runStreamingFunc | streaming path, breaking incremental output |
+-------------------------------------------+---------------------------------------------------+
| Assuming distributed cache entries expire | They don't, by default, unless you configure cache |
| | entry options yourself; stale answers can persist |
| | indefinitely |
+-------------------------------------------+---------------------------------------------------+
| Treating function-calling middleware as | It fires per tool call, not per run, so a single run |
| equivalent to agent-run middleware | with three tool calls triggers it three separate times|
+-------------------------------------------+---------------------------------------------------+

None of these are bugs in the framework. They’re all consequences of a fairly elegant decorator-pattern design that assumes you understand which method you’re overriding and which order your wrappers run in. Once you internalize the onion model from the section above, all five of these stop being surprising.

Why this is worth getting right now, specifically

I’ll admit part of why I went down this rabbit hole is timing. Microsoft shipped Agent Framework 1.0 as production-ready back in April, folding in ideas from both Semantic Kernel and AutoGen into one supported SDK with a long-term support commitment. That alone made it worth taking seriously instead of treating it as another preview API that might get renamed in six months.

Then, just this month, Microsoft moved the Agent Harness and Foundry Hosted Agents to general availability. The harness is the actual runtime that executes agents in production, and it bakes in function invocation, per-call history persistence, context compaction, tool approval, and built-in OpenTelemetry as standard behavior rather than things you bolt on yourself. That last part is directly relevant to everything in this article: the middleware layers we’ve been building by hand for logging and tracing are exactly the kind of thing the harness now handles natively, and it reportedly routes that telemetry into the same OpenTelemetry traces and dashboards as everything else in a Foundry deployment, including third-party coding agents. There’s also a built-in safety net worth knowing about: the harness will halt its own loop after 40 round trips and return a limit-reached message rather than assuming your own middleware caught every runaway loop.

That doesn’t make custom middleware pointless, not even close. You still want your own logic for things specific to your domain: redacting customer PII before it reaches a log sink, enforcing a business-specific rate limit, blocking specific tool calls based on a user’s role. But it does mean the generic cross-cutting concerns, plain logging and basic tracing chief among them, are increasingly something you get by turning on the harness rather than something you write yourself. I’d rather know that going in than spend another afternoon rebuilding a wheel the platform already ships.

Where I landed

If I’m being honest about what changed in my own agent code after this investigation: I moved my security and auth checks to the outermost agent-run middleware, where they belong, since they should reject a bad request before any model or tool gets touched. I kept per-tool logging as function-calling middleware, since that’s genuinely the only place that granularity lives. I moved distributed caching to sit inside logging rather than outside it, which fixed the undercounting bug I mentioned earlier. And for local development, I stopped reaching for an Azure deployment by default and just run everything against Ollama first, promoting to a hosted model only once the middleware chain itself is behaving the way I expect.

The two-layer framing in the original piece is a fine starting point, and if all you need is “where do I put my logging,” it’ll get you there. But the real distinction that will save you debugging time is between per-model-call scope and per-run scope, layered with per-tool-call scope inside that, all nested in a predictable, overridable order. Once that clicked for me, writing correct middleware stopped being guesswork and started being straightforward composition, which is honestly what middleware is supposed to feel like in the first place.

Tags: microsoft-agent-framework, dotnet, csharp, ai-agents, middleware, ollama, chatclient, software-architecture