Redis went down on a Tuesday morning, and my login page went down with it.
Sit with that sentence for a second, because it describes something absurd. A cache is, by definition, optional infrastructure. Its entire job is to make things faster when it’s there and cost nothing when it isn’t. And yet there I was, staring at a 504 Gateway Timeout on the first login of the day, in an ERP suite I build for SMBs, because a box whose only purpose was to save milliseconds was now spending twelve full seconds failing to do it.
The postmortem took ten minutes. The lessons took longer, and they reshaped how I think about caching entirely: where it lives, how it fails, and who is responsible for cleaning it up.
Caching doesn’t belong in handlers
Let me rewind to before the incident, because the first mistake predates it.
The system is CQRS over MediatR — every read is a query object, every write is a command, and each has exactly one handler. The naive way to cache is the way everyone starts: inline, inside the handler.
public async Task<DashboardDto> Handle(GetDashboardQuery request, CancellationToken ct)
{
var cached = await _cache.GetAsync<DashboardDto>("dashboard");
if (cached is not null)
{
return cached;
}
var dto = await BuildDashboardAsync(ct);
await _cache.SetAsync("dashboard", dto, TimeSpan.FromMinutes(5));
return dto;
}
Multiply this by every cacheable query and you get the usual disease: the same seven lines pasted into thirty handlers, each copy slightly different. One forgets the TTL. One builds the key differently. One caches a null and serves it for five minutes. Business logic and infrastructure plumbing braided together in every file.
But MediatR already gives you the perfect seam. Every request flows through a pipeline of behaviors — logging, validation, authorization — before reaching its handler. Cache-aside is just one more ring in that pipeline:
public sealed class CachingBehavior<TRequest, TResponse>(
ICacheService cache,
ILogger<CachingBehavior<TRequest, TResponse>> logger)
: IPipelineBehavior<TRequest, TResponse> where TRequest : notnull
{
public async Task<TResponse> Handle(
TRequest request,
RequestHandlerDelegate<TResponse> next,
CancellationToken ct)
{
var attribute = typeof(TRequest).GetCustomAttribute<CacheQueryAttribute>();
if (attribute is null)
{
return await next(); // not cacheable — pass straight through
}
var key = CacheKey.For(request);
var cached = await cache.GetAsync<TResponse>(key, ct);
if (cached is not null)
{
return cached;
}
var response = await next();
await cache.SetAsync(key, response, attribute.Duration, ct);
return response;
}
}
The handler goes back to doing exactly one thing — building the dashboard. Opting into caching becomes a declaration on the request type , not code in the handler:
[CacheQuery(durationMs: 300_000)] // 5 minutes
public sealed record GetDashboardQuery : IRequest<DashboardDto>;
This is the same philosophy I keep coming back to: behavior should emerge from types, not from discipline. A reviewer doesn’t have to read the handler to know the query is cached — it’s in the signature. And the cache key derives mechanically from the request type and its property values, so two queries with the same parameters share an entry and nobody ever fat-fingers a key string again.
For months, this was the whole story, and it was a good one. Then Tuesday happened.
The day the cache became a dependency
Here’s the anatomy of the incident. Redis was unreachable — a networking hiccup between containers, nothing exotic. The first login of the morning triggered the user-profile flow: one cache Get (miss the profile), then a database load, then a cache Set (store it).
StackExchange.Redis, with default settings, doesn’t fail fast when the server is gone. It waits out a connect timeout — about five seconds. So:
GetAsync("profile:...") → 5s hang → exception → fall through to DB
SetAsync("profile:...") → 5s hang → exception → swallowed
plus retries and a second lookup on the same path...
Total: ~12 seconds. The reverse proxy gave up at 10. → 504.
Read that timeline again and notice the perverse part: the database was fine. The data was right there, reachable in single-digit milliseconds. The only thing standing between the user and their profile was the system’s insistence on first asking a dead cache — twice — and waiting politely each time.
That’s when the framing snapped into place for me. My caching layer had quietly crossed a line: it was no longer an optimization, it was a dependency. If a component’s failure can take your feature down, it’s on the critical path, whether you meant to put it there or not. A cache on the critical path is a bug in your architecture.
The requirement, stated properly: a cache failure must degrade to the database, never to an error. Slow is acceptable. Down is not.
Degrade, don’t die
Try/catch around every Redis call gets you halfway — you stop erroring, but you still pay the five-second timeout on every operation until Redis recovers. Ten cache touches in a request pipeline is fifty seconds of accumulated waiting. You’ve converted an outage into a tarpit.
The missing piece is memory: the caching layer needs to remember that Redis is down and stop asking. That’s a circuit breaker, and it doesn’t need a library — it needs a timestamp and a flag. I wrapped it in a decorator that fronts Redis with an in-memory fallback:
public sealed class ResilientCacheService(
RedisCacheService redis,
MemoryCacheService memory,
TimeProvider clock,
ILogger<ResilientCacheService> logger) : ICacheService
{
private static readonly TimeSpan Cooldown = TimeSpan.FromSeconds(30);
private DateTimeOffset _retryAfter = DateTimeOffset.MinValue;
public async Task<T?> GetAsync<T>(string key, CancellationToken ct)
{
if (CircuitIsOpen())
{
return await memory.GetAsync<T>(key, ct); // don't even ask Redis
}
try
{
var value = await redis.GetAsync<T>(key, ct);
CloseCircuit();
return value;
}
catch (Exception ex) when (ex is RedisException or TimeoutException)
{
OpenCircuit(ex);
return await memory.GetAsync<T>(key, ct);
}
}
private bool CircuitIsOpen() => clock.GetUtcNow() < _retryAfter;
private void OpenCircuit(Exception ex)
{
_retryAfter = clock.GetUtcNow() + Cooldown;
logger.LogWarning(ex,
"Redis unavailable — circuit open for {Cooldown}, serving from memory", Cooldown);
}
private void CloseCircuit()
{
if (_retryAfter > DateTimeOffset.MinValue)
{
_retryAfter = DateTimeOffset.MinValue;
logger.LogInformation("Redis recovered — circuit closed");
}
}
}
The state machine is the classic three states, compressed. Closed : everything goes to Redis. Open : after one failure, Redis is skipped entirely for thirty seconds — every operation goes straight to the in-memory cache at in-process speed. Half-open : when the cooldown lapses, the next operation is allowed through as a probe; success closes the circuit and logs the recovery, failure re-opens it for another thirty.
The same Tuesday, replayed with this in place:
GetAsync → 5s timeout → circuit OPENS → memory fallback
SetAsync → 0 ms (circuit open, Redis skipped) → memory
every subsequent op for 30s → 0 ms
One five-second toll for the first request that discovers the outage. Everything after is effectively free. Users see a slightly slow page once, then nothing at all — while the logs quietly tell me Redis is gone, which is my problem to fix on my schedule, not the user’s.
Why thirty seconds? Long enough not to hammer a Redis instance that’s mid-restart; short enough that recovery is picked up almost immediately. And the trade-offs are ones I accept with eyes open: the in-memory fallback is per-process, so during an outage each app instance has its own little cache with no shared state — fine for reference data and profiles. And when Redis comes back, memory may briefly disagree with it; TTLs wash that out on their own. The one thing I didn’t build is proactive health probing to avoid that first five-second toll. It’s the kind of complexity that earns its keep at a scale I don’t operate at yet, and pretending otherwise would be cosplay.
Invalidation, or: commands know what they dirty
The famous hard problem. Cache-aside with TTLs is honest about reads, but a five-minute TTL means up to five minutes of lying after a write. Update a product’s price, refresh the list, see the old price — that’s not a stale cache, that’s a bug report.
The pipeline gives you the answer for free, because in CQRS the writes are objects too. Every mutation is a command flowing through the same pipeline. So invalidation becomes the mirror image of caching: queries declare what they cache, commands declare what they dirty.
[InvalidateCache(nameof(GetProductQuery), nameof(GetProductListQuery))]
public sealed record UpdateProductCommand(string Id, /* ... */) : IRequest<string>;
A MediatR post-processor does the sweep. Post-processors only run after the handler completes successfully — which is exactly the semantics invalidation needs. A command that throws changed nothing, so it should evict nothing:
public sealed class CacheInvalidationPostProcessor<TRequest, TResponse>(
ICacheService cache) : IRequestPostProcessor<TRequest, TResponse>
where TRequest : notnull
{
public async Task Process(TRequest request, TResponse response, CancellationToken ct)
{
var attribute = typeof(TRequest).GetCustomAttribute<InvalidateCacheAttribute>();
if (attribute is null)
{
return;
}
foreach (var queryName in attribute.QueryNames)
{
await cache.RemoveByPrefixAsync(CacheKey.PrefixFor(queryName), ct);
}
}
}
What I like most is where the knowledge lives. The relationship “updating a product stales the product queries” is a domain fact , and now it’s written on the command — the one type whose author necessarily knows it — instead of being scattered across handlers or, worse, living in someone’s head. When a new query over the same data appears, its name gets added to the relevant commands’ attributes, and the code review diff makes the coupling visible. It’s not automatic dependency tracking; it’s declared dependency tracking. In my experience the declaration is the valuable part — the moment where someone has to think, “what does this write invalidate?” — and automating it away would remove the thinking, not just the typing.
And yes — eviction goes through the same ICacheService, so when the circuit is open it degrades too: memory entries are evicted, the unreachable Redis entries simply age out on TTL. Degraded mode is a little more stale and a little less shared. It is never broken.
The takeaway
None of this is novel machinery. Cache-aside is decades old, circuit breakers are a named pattern, MediatR behaviors are documented. What took me an outage to actually internalize is a definitional point: a cache that can take your feature down is not a cache. The moment its failure propagates to the user, you’ve silently promoted an optimization into a dependency — and you’ll discover the promotion the way I did, through a 504 on a Tuesday morning.
The craft, as usual, is in the disappearing act. A query opts into caching with one attribute; a command declares what it dirties with another; the pipeline does the reading, writing, and evicting; and a small decorator makes the whole thing shrug off a dead Redis with nothing but a warning in the logs. The handlers know none of it. That’s the shape I aim for now in anything cross-cutting: make the right behavior emerge from a declaration on a type, make failure degrade instead of cascade — and then stop thinking about it, because the next Tuesday is already handled.
Top comments (0)