The Retry Storm Problem: Why Your ASP.NET Core API Needs Idempotency Keys

The Retry Storm Problem: How Idempotency Keys Save Your ASP.NET Core API

Introduction

Every .NET developer knows the frustration of a failed request followed by an automatic retry. But there's a subtler danger: when those retries hit side-effecting endpoints (payment processing, email dispatch, inventory updates), you can end up with duplicate charges, duplicate notifications, or corrupted state. This is the "retry storm" problem—and it's solved with idempotency keys.

The Problem

Consider a mobile client that experiences a brief network glitch while placing an order. The HTTP request times out, the client library retries, and because our API doesn't track whether an operation has already been executed, both attempts succeed. The result: the payment gateway charges the card twice, the notification service sends two emails, and inventory is decremented twice.

This isn't a bug in your code—it's a gap in the contract between client and server. The client assumes retries are safe; the server treats them as independent calls.

The Solution: Idempotency Keys

An idempotency key is a unique identifier (typically a UUID) generated by the client for a logical operation. Before processing, the server checks if that key has already been handled. If so, it returns the cached response. If not, it processes the request and stores the response for future replays.

Where to Store the Key

For high-throughput scenarios, use a distributed cache like Redis. The key lives in memory across instances, giving you sub-millisecond lookups. For lower-volume systems, in-memory caching works fine too.

Implementation Pattern

Below is a simplified ASP.NET Core 8 middleware that implements the core idea:

public class IdempotencyMiddleware
{
    private readonly IDistributedCache _cache;

    public async Task InvokeAsync(Context context)
    {
        var key = context.Request.Headers["Idempotency-Key"]
            .FirstOrDefault();

        if (string.IsNullOrEmpty(key)) return;

        // Check if we've already processed this key
        var cached = await _cache.GetStringAsync($"idem:{key}");
        if (!string.IsNullOrEmpty(cached))
        {
            context.Response.StatusCode = 200;
            context.Response.ContentType = "application/json";
            await context.Response.WriteAsync(cached);
            return;
        }

        // Process the request (your existing logic here)
        var result = await ExecuteBusinessLogic(context);

        // Cache the successful response for future replays
        await _cache.SetStringAsync(
            $"idem:{key}", 
            result, 
            new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(24) }
        );
    }
}

Key Design Decisions

1. Client Responsibility: Generate the key. Make it unique per logical operation (e.g., order placement, payment authorization).
2. Server-Side Storage: Always persist the response after successful processing. This ensures that even if the client forgets to send the key, subsequent retries still get the correct outcome.
3. Response Caching: Store the full response (status code, body, headers) so replays return exactly what the first call returned—no need to re-execute business logic.

When to Use Idempotency Keys

• Financial transactions: Payments, billing, subscription management.
• State-changing operations: Creating orders, updating inventory, modifying records.
• External integrations: Calling third-party APIs that also require idempotency (Stripe, Twilio, etc.).

Avoid over-engineering for read-only endpoints. They don't benefit from idempotency since reading the same resource twice is harmless.

Conclusion

Adding idempotency keys is a small investment—one extra cache lookup—that pays dividends in reliability. It transforms retries from a source of bugs into a feature: the client can safely retry without fear of unintended side effects. Start small, test the edge cases thoroughly, and your API will handle network flakiness gracefully.

---