4 Pipeline Orders That Quietly Double Your LLM Bill

#dotnet #csharp #ai #aspnet

You ship a "cost control" PR. Grafana looks quieter. Finance still spikes. Nine times out of ten the middleware was correct — the order was not.

Microsoft.Extensions.AI lets you stack IChatClient layers the way ASP.NET stacks HTTP middleware. Caching, GenAI spans, and a hard token budget only work if the request walks through them in a defendable sequence. This post is the short order checklist. The complete offline-tested project (DistributedCachingChatClient, OpenTelemetryChatClient, TokenBudgetChatClient, rate limit, 56 tests) lives in Tech Skill Builder — $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10.

Mistake 1: Putting the budget outside the cache

A cache hit still "spends" tokens on the ledger. You celebrate fewer provider calls while the budget burns twice for one real completion.

Shape that works: OpenTelemetry → cache → budget → rate limit → provider.

IChatClient client = new ChatClientBuilder(inner)
    .UseOpenTelemetry(sourceName: "LlmPipeline")
    .UseDistributedCache(cache)
    .UseTokenBudget(ledger, budgetOptions)   // INSIDE the cache
    .UseConcurrencyRateLimit(limiter)
    .Build();

First Use is outermost. Hits never reach the ledger. Put the budget first and you invent a second invoice.

Mistake 2: Burying OpenTelemetry under the cache

Ops only sees misses. The happiest path — the request you did not pay for — disappears from traces. A week later someone "proves" caching does nothing because APM never counted the hits.

Keep OTel outside. Hits become short spans; misses include provider time. Leave EnableSensitiveData false unless you have a written, time-bounded reason — prompts in a shared APM tenant are a privacy incident that started as "temporary debug."

Mistake 3: Caching live tool results like FAQ answers

Balances, inventory, ticket state. You freeze yesterday's number, then blame the model for "hallucinating." Cache keys also care about more than the prompt string: Temperature, ModelId, and max output tokens participate. Drift defaults between environments and the cache goes colder than the slide deck promised.

Rule of thumb: FAQ-style prompts without live tools → cache the final answer. "What's my balance?" → do not.

JSON round-trip is imperfect (RawRepresentation dropped; AdditionalProperties come back as JsonElement). Design cacheable payloads around message text and strongly typed options — not a custom POCO stuffed in additional properties.

Mistake 4: Soft-deleting the budget because "it blocked the demo"

Caching reduces spend. It does not cap a viral prompt, a buggy while, or a free-tier tenant. A small DelegatingChatClient that reserves before the call and reconciles with Usage (or characters÷4 offline) maps exhaustion to HTTP 429 with remaining/requested/max. Keep the anti-pattern test (budget outside cache) in CI — cheaper than a Slack thread at month-end.

// sketch — full implementation in the complete project
protected override async Task<ChatResponse> GetResponseAsync(
    IEnumerable<ChatMessage> messages, ChatOptions? options, CancellationToken ct)
{
    var estimate = Estimate(messages, options);
    if (!_ledger.TryReserve(estimate))
        throw new TokenBudgetExceededException(/* remaining, requested, max */);

    var response = await base.GetResponseAsync(messages, options, ct);
    _ledger.Reconcile(estimate, response.Usage); // or char÷4 fallback
    return response;
}

Quick order table

Order (outer → inner)	What you observe
Cache outside budget	Hits skip the ledger (good)
Budget outside cache	Hits still charge (bad)
OTel outside cache	Hits show as fast spans (good)
OTel inside cache	Hits invisible
Rate limit inside cache	Hits skip the limiter (good)

What to build next

Wire one shared pipeline for API, worker, and CLI. Use Redis for IDistributedCache once you have more than one pod. Alert on GenAI usage and budget 429s, not only HTTP 5xx. Pin MEAI / OpenTelemetry versions — GenAI semconv is still draft.

Want the runnable .NET 10 solution with an offline fake client, CLI miss-then-hit demo, and the order anti-pattern tests? Grab the complete project in Tech Skill Builder — $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10.

DEV Community