Where Does the Cache Go? 5 Ordering Questions for an IEmbeddingGenerator Pipeline in .NET 10

The first version of embedding code in most .NET apps is one line: call the provider, get floats. The second version is where the interesting decisions are. You add a cache because the same product description gets embedded over and over. You add a rate limiter because the provider started returning 429s. You add tracing because nobody can tell why search is slow. And then you have to decide which of those wraps which.

That decision is easy to skip, because every order compiles and every order "works". But the order changes what you pay for, what shows up in your traces, and which bad request gets stopped before it costs anything. This post goes through the five ordering questions I had to answer while building an embedding pipeline on Microsoft.Extensions.AI 10.11 in .NET 10. The project behind it has 26 tests (20 unit, 6 integration), all running offline against a deterministic fake generator with no API key. The complete project is in Tech Skill Builder.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10.

The one rule you need first

Microsoft.Extensions.AI gives embeddings the same middleware model as chat. You start from an inner IEmbeddingGenerator<string, Embedding<float>> and wrap it with EmbeddingGeneratorBuilder. Each layer is a DelegatingEmbeddingGenerator that does its work and calls the next one.

The rule: the first Use call is the outermost layer. A request goes through the layers top to bottom, and the provider is at the bottom. Here's the order I settled on:

InputSize -> RateLimit -> DistributedCache -> OpenTelemetry -> BatchChunk -> provider (or offline fake)

In code it reads almost like that diagram:

return new EmbeddingGeneratorBuilder<string, Embedding<float>>(inner)
    .Use(g => new InputSizeLimitingEmbeddingGenerator(g, options.MaxInputChars, options.TruncateOversizedInputs))
    .Use(g => new RateLimitingEmbeddingGenerator(g, limiter, ownsLimiter))
    .UseDistributedCache(cache)
    .UseOpenTelemetry(loggerFactory, options.OtelSourceName)
    .Use(g => new BatchChunkingEmbeddingGenerator(g, options.MaxBatchSize))
    .Build();
// limiter creation and option validation: full EmbeddingPipelineFactory in the complete project

Now the questions behind it.

Question 1: Should the input guard run before or after the cache?

Before. Outermost, actually.

A user pastes 200 KB of log output into a search box. You don't want that string hashed into a cache key, counted by your limiter, traced, or sent to a provider. You want it rejected immediately with a clear message, and mapped to a 400 at the API edge.

The guard can also truncate instead of reject, but I keep reject as the default. Truncating quietly changes what the text means, and the caller never finds out. I only turn truncation on when I control the corpus.

if (text.Length <= MaxInputChars) { prepared.Add(text); continue; }
if (TruncateOversized) { prepared.Add(text[..MaxInputChars]); continue; }
throw new ArgumentException($"Input length {text.Length} exceeds the maximum of {MaxInputChars} characters.", nameof(values));
// loop over the batch and the call to base.GenerateAsync: full middleware in the complete project

One trap if you do truncate: GenerateAndZipAsync pairs each embedding with the original input you passed in, not the string your middleware rewrote. So a test that checks zipped[i].Value to prove truncation will mislead you. Compare the vector with the embedding of the truncated text instead. That's how the project's truncation test works.

Question 2: Should telemetry see cache hits?

This one has no universal answer, so pick one and write it down.

• Cache outside telemetry (my choice): a cache hit returns before it reaches UseOpenTelemetry, so your traces show only real provider work. The spans answer "what are we paying for, and how slow is the provider?"
• Telemetry outside cache: every call gets a span, hits included. The spans answer "how often is embedding called at all?"

UseOpenTelemetry follows the GenAI semantic conventions, which are still a draft and can change. Give it an ActivitySource name you control and register that name on your tracer provider, or you'll get nothing in your exporter and spend a while wondering why.

The test that matters here isn't about spans, it's about the cache actually working. Embed the same string twice through the full pipeline and assert the inner generator wasn't called again:

await pipeline.GenerateAsync(["cache-me-please"]);
var afterFirst = inner.CallCount;
await pipeline.GenerateAsync(["cache-me-please"]);
Assert.Equal(afterFirst, inner.CallCount);
// options and MemoryDistributedCache setup: full test in the complete project

MemoryDistributedCache is enough for tests. In production you point the same IDistributedCache at Redis and the pipeline code doesn't change.

Question 3: Should cache hits take a rate-limit permit?

Ideally, no. A cache hit costs you nothing at the provider, so it shouldn't wait in line behind real calls. That argues for putting the limiter inside the cache.

I'll be honest about the project here: its factory puts the limiter just outside the cache, as you can see in the diagram above. So cache hits do take a permit. With a ConcurrencyLimiter the permit is released as soon as the hit returns, so in practice it's cheap, but it does mean a burst of hits can compete with misses for the same permits. If you'd rather hits skip the limiter completely, move the RateLimit line below UseDistributedCache. I didn't test that order in the project, so treat it as a change you'd verify yourself.

Whichever side you put it on, make the limiter fail loudly when it can't get a lease:

using var lease = await rateLimiter.AcquireAsync(permitCount: 1, cancellationToken);
if (!lease.IsAcquired)
    throw new InvalidOperationException("Unable to acquire embedding rate-limiter lease.");
// then base.GenerateAsync, plus limiter ownership in Dispose: full version in the complete project

One more detail that bit me: if the middleware disposes the limiter, it has to own it. When a test passes in a shared limiter, the middleware must not dispose it, or the next test gets a disposed limiter. The project makes this explicit with an ownsLimiter flag.

Question 4: Why does batch chunking go at the very bottom?

Providers cap how many inputs one embedding request can carry. A chunking layer splits a big GenerateAsync call into batches of MaxBatchSize and puts the results back together in order. With MaxBatchSize = 2, five inputs become three inner calls (2 + 2 + 1), and a unit test checks both the count and the order.

It goes right above the provider for two reasons:

1. The cache stays per item. The caching layer looks up each input separately, so only the misses go further down. If you chunked above the cache, you'd be splitting batches that might be mostly cache hits anyway.
2. The provider never sees an oversized batch, no matter which code path built it.

And since chunking sits inside telemetry, you get one span for the batch of misses rather than one per chunk. If you want per-chunk spans, swap those two.

Inside the chunker, clone the options for each chunk:

var chunkOptions = options?.Clone(); // generators may mutate the options instance
var part = await base.GenerateAsync(chunk, chunkOptions, cancellationToken);
combined.AddRange(part);
// slicing, cancellation checks and merging token usage: full BatchChunkingEmbeddingGenerator in the complete project

That Clone() isn't paranoia. Implementations are allowed to change the EmbeddingGenerationOptions you pass in, so one shared instance across chunks or parallel calls is a race waiting to happen.

Question 5: What sits at the bottom in CI?

A fake, and a deterministic one. Calling a paid embedding API from unit tests is slow, costs money, and makes assertions flaky. Random vectors are no better, since you can't assert anything about them.

The project's offline generator is a hashing trick over tokens and word pairs: each one is hashed into a fixed number of dimensions (64 by default), then the vector is L2-normalized. Same text, same vector, on every machine and every run. That's what makes "the second call didn't reach the provider" a test you can trust.

The trap: don't use string.GetHashCode for this. In modern .NET it's randomized per process, so your "deterministic" fake gives different vectors on every test run. Use a stable hash like FNV-1a.

// FNV-1a: stable across processes, unlike string.GetHashCode
var h = StableHash(token);
var sign = (h & 1) == 0 ? 1f : -1f;
vector[(int)(h % (uint)vector.Length)] += sign * weight;
// second hash bucket, bigrams, the noise floor and L2 normalization:
// full OfflineEmbeddingGenerator in the complete project

Keep the fake even after you add a real provider. Ordering bugs show up there first, in milliseconds.

A quick review checklist

When I review embedding code now, I ask:

• [ ] Is there one pipeline, built in one place, that every call site uses?
• [ ] Is the input guard the outermost layer, and does it reject by default?
• [ ] Is it written down whether telemetry sees cache hits?
• [ ] Is the limiter on the side of the cache you actually intended?
• [ ] Is batch chunking next to the provider, with options cloned per chunk?
• [ ] Does a test prove that a repeated input doesn't reach the provider?
• [ ] Is the CI fake deterministic across processes?

What I didn't verify

All 26 tests run against the offline fake, and the API's integration tests use WebApplicationFactory with the same offline pipeline, so no real provider was called. The fake matches tokens, not meaning, so the small catalog search in the project shows the pipeline working end to end, not real semantic quality. Production would also mean Redis behind IDistributedCache, which the project doesn't run.

Get the complete project

The snippets above are the shape of it. The complete, working .NET 10 solution is in Tech Skill Builder: the Core library with the three custom middleware types (input-size guard, rate limiter, batch chunker), the pipeline factory and DI registration, the deterministic offline generator, a small in-memory cosine catalog search, an ASP.NET Core minimal API (/api/health, /api/embed, /api/search) with FluentValidation and an OpenTelemetry console exporter, a CLI demo, and all 26 tests, plus a step-by-step PDF guide. No API key needed.

Tech Skill Builder gives you complete, working, tested .NET projects with full source code and step-by-step PDF guides, with new ones added regularly. Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. Join Tech Skill Builder

How do you order your embedding layers, and do your traces include cache hits or not?