4 HybridCache Gotchas I Only Found When I Tested My Own Sample

My "under 5 ms" demo was running without an L2, hung on Redis and served stale data. I didn't notice any of it until I wrote tests.

Recently I published Serve API Responses in Under 5ms: HybridCache in .NET. The demo worked, the numbers looked great, and I was pretty happy with it.

Then I decided the repo deserved real tests, a CI pipeline with Redis and proper benchmarks instead of a curl script. I expected that to take an evening and confirm what I already knew.

It didn't. I found four problems. One was a plain bug in my code. The other three were HybridCache behaviours I simply hadn't understood. I never noticed any of them while clicking around by hand, and every one of them would have bitten me in production.

All the fixes are in the companion repo, and each one has a test that fails against the old code.

1. AddDistributedMemoryCache() doesn't give you an L2 My Program.cs looked like most HybridCache samples you'll find online:

builder.Services.AddDistributedMemoryCache(); // my "L2"
builder.Services.AddHybridCache();

My README even said AddDistributedMemoryCache() was "used as the L2 store". The logic seemed fine to me. HybridCache uses whatever IDistributedCache you register, so this should give me two levels of cache without running Redis on my laptop.

So I wrote a test for it. Two HybridCache instances play two pods. Each has its own L1, and they share one L2:

var sharedL2 = new MemoryDistributedCache(Options.Create(new MemoryDistributedCacheOptions()));
var podA = CreateHybridCache(sharedL2);
var podB = CreateHybridCache(sharedL2);

await podA.GetOrCreateAsync(key, Factory); // miss: factory runs, fills L1(A) and L2
await podA.GetOrCreateAsync(key, Factory); // L1(A) hit
await podB.GetOrCreateAsync(key, Factory); // L1(B) is empty, so this should come from L2

Assert.Equal(1, factoryCalls); // Actual: 2

The factory ran twice. Pod B found nothing in L2, because nothing had ever been written there.

I didn't quite believe it, so I went and read the source. It's right there in DefaultHybridCache's constructor:

// ignore L2 if it is really just the same L1, wrapped
if (_backendCache is not null
    && _backendCache.GetType() == typeof(MemoryDistributedCache)
    && _localCache.GetType() == typeof(MemoryCache))
{
    _backendCache = null;
}

When you think about it, it's a reasonable optimisation. MemoryDistributedCache lives in the same process, so using it as L2 would just mean serializing a second copy of L1. HybridCache notices and switches L2 off.

The awkward part for me: my sample had been running with L1 only the whole time. The L1 → L2 → factory flow I walked through in the article never actually happened unless you ran Redis.

If you want a real L2, use an out-of-process store like Redis, SQL Server or Postgres. If you only want an in-process L2 for tests, wrap MemoryDistributedCache in a small pass-through class. The check compares exact types, so the wrapper counts as a real L2:

public class PassThroughDistributedCache(IDistributedCache inner) : IDistributedCache
{
    public byte[]? Get(string key) => inner.Get(key);
    public Task<byte[]?> GetAsync(string key, CancellationToken token = default) => inner.GetAsync(key, token);
    // Set, SetAsync, Refresh, RefreshAsync, Remove and RemoveAsync delegate the same way
}

The official HybridCache docs didn't mention this, so I sent a pull request to add a note. Microsoft merged it, and it's now live on Microsoft Learn.

Hopefully it saves someone else the head-scratching.

1. Redis and HybridCache deadlocked on the first request Fine, I thought. I'll wire up Redis properly.

var redis = builder.Configuration.GetConnectionString("Redis");
if (!string.IsNullOrWhiteSpace(redis))
{
    builder.Services.AddStackExchangeRedisCache(o => o.Configuration = redis);
}
builder.Services.AddHybridCache();

My Redis tests went green. Then I started the actual API against Redis, sent GET /products, and... nothing. The app had started normally. The connection was accepted. The request just sat there, with no exception, no log line and no timeout.

The tests had passed for a boring reason. They created the cache with new RedisCache(...), while the API used AddStackExchangeRedisCache(...). That one difference was the whole problem.

I took a stack dump with dotnet-stack, and the loop was easy to spot:

RedisCacheImpl..ctor(IOptions<RedisCacheOptions>, IServiceProvider)
  RedisCacheImpl.IsHybridCacheDefined(IServiceProvider)
    DefaultHybridCache..ctor(IOptions<HybridCacheOptions>, IServiceProvider)
      RedisCacheImpl..ctor(...)
        RedisCacheImpl.IsHybridCacheDefined(...)
          DefaultHybridCache..ctor(...)
            ...

Building HybridCache resolves the IDistributedCache. In Microsoft.Extensions.Caching.StackExchangeRedis 9.0.3, the Redis cache's constructor checks whether HybridCache is registered by... resolving HybridCache. Two singletons, each waiting for the other to finish being created. Forever.

It's a known bug, already reported and fixed upstream in dotnet/aspnetcore#60894. Upgrading the Redis package sorted it out. I confirmed that 9.0.20 works; the issue will tell you the first patched version.

What I took away from this one:

If you use HybridCache with AddStackExchangeRedisCache, check which Redis package version you're on.
Test the app the way it actually runs. My tests built RedisCache by hand and never went near the DI path that deadlocked. The regression test I added runs the whole API through WebApplicationFactory with Redis configured, and fails if GET /products takes longer than 10 seconds.
When something hangs and logs nothing, take a stack dump before you start guessing. One command took me from "is Redis broken?" to the exact constructor cycle.

1. The L2 write can land after GetOrCreateAsync returns With Redis fixed, I pointed my two-pod test from gotcha 1 at real Redis. It still failed. Pod B called its factory again.

When I checked Redis afterwards, the keys were there. They just weren't there yet when pod B asked. In my tests, HybridCache handed the value back before its Redis write had finished. My in-memory wrapper wrote synchronously, so it had hidden this completely.

For the test, the fix was to wait for the entry to show up in L2 before asking the second instance:

// HybridCache can return the value before its L2 write completes,
// so wait for the entry to land in L2 before asking the second instance.
var deadline = DateTime.UtcNow.AddSeconds(5);
while (await sharedL2.GetAsync(key) is null && DateTime.UtcNow < deadline)
    await Task.Delay(10);

The bigger lesson is about production. If two instances ask for the same key at almost the same moment, both can miss L2 and both can run the factory. Stampede protection works inside one process. It doesn't cover your whole fleet.

So make your factories safe to run more than once, and cheap enough that it doesn't matter much when they do. If something really must run only once across all your instances, you need a distributed lock or a different design, because HybridCache won't do that for you. And in tests, don't assume an L2 write is visible straight away.

1. Tag invalidation only clears what you actually tagged This one's entirely my fault, but I suspect it's a common mistake.

My original service tagged the list with products, and each item with products plus product:{id}:

// list
cache.GetOrCreateAsync("products:all", ..., tags: ["products"]);
// item
cache.GetOrCreateAsync($"product:{id}", ..., tags: ["products", $"product:{id}"]);

// update
await cache.RemoveByTagAsync($"product:{id}"); // "invalidates both the item and any list"

Look at that comment. It's wrong. The list never had the product:{id} tag, so updating product 1 cleared the cached item while GET /products kept showing the old name for up to five minutes. My curl script never caught it, because after an update it only checked the item, never the list.

The fix was to give each kind of entry its own tag, and clear every affected tag on every write:

public const string ListTag = "products:list";
public static string ItemTag(int id) => $"product:{id}";

// update and delete: the item and the list both contain the old values
await cache.RemoveByTagAsync([ListTag, ItemTag(id)], ct);

// create: the list is stale, and an earlier GET for this id may have cached "not found"
await cache.RemoveByTagAsync([ListTag, ItemTag(created.Id)], ct);

That last line also handles a sneaky case. GetOrCreateAsync will happily cache null, so calling GET /products/5 before product 5 exists caches "not found" until it expires.

The habit I've picked up: for every write, list every cached view that shows the changed data, and check that each one carries a tag you clear. Then write the test from the reader's side. Update something, then read every endpoint that displays it.

One more thing you'll notice with Redis as L2 is keys like MSFT_HCTproducts:list sitting next to your data. That's HybridCache keeping track of tag invalidations, so don't let a clean-up script delete them.

So, is it still under 5 ms?
Yes, and this time I have numbers instead of a curl script. BenchmarkDotNet on .NET 9, with Redis 7 running in Docker:

Served by                                   Mean
L1 hit (in-process)                       0.88 µs
L2 hit (Redis + deserialize)               288 µs
Miss (factory, 30 ms simulated database)  31.1 ms

Over real HTTP on localhost, a cached GET /products took about 1.5 ms, compared with about 127 ms for the first, uncached call. In production you'll add your network round-trip on top. The benchmarks are in the repo if you want to run them on your own hardware.

What I'd tell myself before writing the first article
Check that each cache layer works, not just that the endpoint is fast. A fast response proves something was cached. It doesn't tell you which cache, or whether the other one is even switched on.
Test the app the way it really runs. My component tests were green while the real setup deadlocked.
Treat stampede protection as per-process. Across instances, assume the factory can run more than once.
Test invalidation from the reader's side.
The full sample is on GitHub, with the tests, the benchmarks and a CI pipeline that runs everything against Redis: paulap887/HybridCache.

Have you run into HybridCache surprises of your own? Tell me in the comments. I'd really like to know what I've missed.

Originally published on Level Up Coding.