DEV Community: Imran Ahmed

When Your Background Job Retries Are Silently Stepping on Each Other

Imran Ahmed — Wed, 07 Oct 2026 12:05:09 +0000

When Your Background Job Retries Are Silently Stepping on Each Other

In .NET backend development, we rely heavily on background jobs for asynchronous processing: sending emails, generating reports, or processing orders. We use libraries like Hangfire, Quartz, or custom IHostedService implementations to ensure resilience.

A common pattern is to wrap database operations in a try-catch block that catches transient exceptions (like SqlException for deadlocks or timeouts) and retries the operation. The assumption is usually that the code is idempotent—running it twice shouldn’t change the result.

But what happens when "idempotent" isn't quite true? What happens when two different jobs, retrying due to transient failures, both attempt to modify the same shared resource?

The Silent Overwrite

Let’s look at a scenario. We have an OrderService that processes customer orders. When an order is placed, a background job is triggered to update the Inventory table.

Suppose two orders are placed almost simultaneously:

Order 1 for Product X (Quantity 5).
Order 2 for Product X (Quantity 3).

Both jobs start almost at the same time.

Job 1 reads Inventory row for Product X. Current stock: 10.
Job 2 reads Inventory row for Product X. Current stock: 10.
Job 1 calculates new stock: 10 - 5 = 5.
Job 2 calculates new stock: 10 - 3 = 7.
Job 1 tries to write 5 to the DB. It hits a transient network blip and the write fails.
The retry mechanism kicks in for Job 1. It re-executes the logic. It reads the inventory again? No. In many implementations, the entity was already loaded into memory. The retry just re-sends the UPDATE statement with the stale value it calculated earlier (5).
Job 2 successfully writes 7 to the DB.
Job 1’s retry finally succeeds and writes 5 to the DB.

Result: The stock is now 5. But it should be 2 (10 - 5 - 3). Job 1’s retry silently overwrote Job 2’s valid change. This is a "lost update" caused by retry mechanics stepping on each other.

Why Tests Miss This

Standard unit tests usually mock the database or test in isolation. You might test:

"Given stock 10, order 5, result is 5." (Pass)
"Given stock 10, order 3, result is 7." (Pass)

But you rarely test: "Given two concurrent jobs with a transient failure on the first, does the second job’s data survive the first job’s retry?"

Simulating this in integration tests is difficult. You’d need to inject a delay or a specific exception during the save of the first job while the second job is proceeding. Most developers skip this complexity, leaving the bug to production.

The Fix: Optimistic Concurrency

The robust solution is to use optimistic concurrency control. Instead of blindly writing, we ask the database: "Did anything else change this row while I was away?"

In SQL Server, we use the rowversion data type. It’s a system-generated number that increments every time the row is updated.

Step 1: Add a Row Version Column

In your EF Core model:

public class InventoryItem
{
    public int Id { get; set; }
    public string ProductName { get; set; }
    public int Stock { get; set; }

    // Use RowVersion or Timestamp
    [Timestamp]
    public byte[] RowVersion { get; set; }
}

In your SQL Server database, ensure you have a rowversion column. If you’re using SQL Server, you can just add ROWVERSION to the table definition.

Step 2: Implement Idempotent Retry Guards

When you retry a job, you must ensure that the state you’re writing is still valid. With rowversion, EF Core automatically adds a WHERE clause to your UPDATE statements.

// In your service
public async Task ProcessOrderAsync(int orderId, int quantity)
{
    using var context = new AppDbContext();
    var item = await context.InventoryItems.FindAsync(productId);

    if (item == null) return;

    item.Stock -= quantity;

    try
    {
        await context.SaveChangesAsync();
    }
    catch (DbUpdateConcurrencyException ex)
    {
        // This is thrown if the RowVersion has changed.
        // It means another job updated this row.
        // Instead of overwriting, we should fetch the latest state and re-calculate.
        // Or, in the context of a simple deduction, we might just let it fail 
        // and let the queue re-enqueue the job.

        var entry = ex.Entries.Single();
        var databaseValues = await entry.OriginalValues.StoreGeneratedValuesAsync();

        // Logic to handle the conflict. 
        // For inventory, you might need to re-read and check if the stock 
        // is still sufficient, or if the order has already been processed.

        throw; // Let the retry mechanism handle it, or log and fail fast.
    }
}

By catching DbUpdateConcurrencyException, you prevent the silent overwrite. If Job 1 retries and finds that Job 2 has already updated the rowversion, the UPDATE affects zero rows. The exception is thrown, and you know the state is stale.

Step 3: Make the Business Logic Idempotent

Retries alone aren’t enough. Your code must handle the "already processed" state. If Job 1 is retried and the rowversion matches (meaning no one else touched it), it’s safe to re-apply. But if it doesn’t match, you must check: "Has this specific order already been deducted?"

Often, this means adding a ProcessedOrderIds table or a flag on the order itself, and checking it before performing the calculation.

Practical Takeaway

Concurrency bugs are not about your code being "wrong" in the happy path. They’re about your code failing to handle the overlap between retries and other active writers.

Never assume that a retry after a transient error is safe without checking the current database state.
Use rowversion (or similar mechanisms like updated_at with application-level checks) for any shared mutable resources.
Design for conflict. When a concurrency exception is thrown, your job should either re-read the state and retry the business logic, or fail fast so the queue can re-process it with fresh data.

Your background jobs should be resilient to failures, but they shouldn’t be destructive to data integrity.

Why your idempotent POST endpoint is still creating duplicates under load

Imran Ahmed — Tue, 06 Oct 2026 12:05:52 +0000

Stop Losing Data: Why Your Idempotent POST Endpoint is Still Creating Duplicates

If you have a .NET API that accepts POST requests, you’ve likely added idempotency keys to handle retries gracefully. It’s a standard pattern: client sends a key, server checks if it’s been seen before, and if not, processes the request.

The common implementation looks something like this:

Check Redis/MemoryCache for the key.
If it exists, return the cached response.
If it doesn’t exist, process the business logic.
Save the result to Redis.

This feels safe. But under real-world load, especially with load balancer timeouts or aggressive client-side retries, this pattern creates a race condition that results in duplicate data. In this article, we’ll look at why this happens and how to fix it using SQL Server constraints and proper exception handling in ASP.NET Core.

The "Check-Then-Act" Race Condition

Let’s visualize the failure. Imagine two identical HTTP requests hit your API endpoint POST /orders at 10:00:00.000.

Request A:

Checks Redis for key ord-123. Result: Miss.
Starts processing business logic (takes 20ms).

Request B:

Checks Redis for key ord-123. Result: Miss. (Request A hasn't finished writing to Redis yet).
Starts processing business logic.

Request A:

Completes logic. Writes order to SQL Server. Writes key to Redis. Returns 201 Created.

Request B:

Completes logic. Writes order to SQL Server. Duplicate!

Both requests believe they are the "first" one because the cache check was non-atomic. The cache is a hint, not a guard. If your system relies solely on the cache to prevent duplicates, you are vulnerable to concurrency issues.

Why Caching Isn't Enough

Redis is great for performance, but it doesn't provide strong consistency guarantees for this specific use case in a distributed environment. Even if you use SETNX in Redis, there’s still a window where the business logic is running but the key isn't reserved, or network partitions cause confusion.

For financial transactions, order creation, or any critical data, you need a source of truth that enforces uniqueness at the storage layer. That source of truth is your database.

The Fix: Unique Constraints + Exception Handling

The safest approach is to let the database enforce idempotency. Here’s how to do it in ASP.NET Core with EF Core and SQL Server.

1. Database Schema

Add a unique constraint to the table that stores your idempotency keys. Often, this is part of the main table (e.g., Orders) or a separate lookup table.

-- Assuming a separate table for tracking idempotency
CREATE TABLE IdempotencyKeys (
    Id INT IDENTITY PRIMARY KEY,
    TenantId UNIQUEIDENTIFIER NOT NULL,
    Key VARCHAR(100) NOT NULL,
    CreatedAt DATETIME2 NOT NULL DEFAULT GETUTCDATE(),
    CONSTRAINT UQ_Tenant_Key UNIQUE (TenantId, Key)
);

Or, if storing it in the main Orders table:

ALTER TABLE Orders
ADD CONSTRAINT UQ_Orders_Tenant_IdempotencyKey UNIQUE (TenantId, IdempotencyKey);

2. Application Logic

In your controller or service, you need to change the flow from "Check Cache -> Create" to "Try Create -> Handle Duplicate".

[HttpPost]
public async Task<IActionResult> CreateOrderAsync([FromBody] CreateOrderRequest request, [Header("X-Idempotency-Key")] string key)
{
    var tenantId = GetCurrentTenantId();

    // 1. Try to insert the key into the DB first (or as part of the transaction)
    // This ensures atomicity.
    try
    {
        // Check cache for fast path (optional optimization, not for safety)
        if (await cache.TryGetValueAsync($"idem:{tenantId}:{key}", out var cachedResponse))
        {
            return Ok(cachedResponse);
        }

        // Begin Transaction
        using var transaction = context.Database.BeginTransaction();

        // Insert Idempotency Record
        var idempotencyRecord = new IdempotencyKey { TenantId = tenantId, Key = key };
        context.IdempotencyKeys.Add(idempotencyRecord);

        // Perform Business Logic
        var order = MapToOrder(request);
        context.Orders.Add(order);

        // Commit
        await context.SaveChangesAsync();
        await transaction.CommitAsync();

        // Set Cache (Best effort)
        await cache.SetAsync($"idem:{tenantId}:{key}", order, TimeSpan.FromHours(24));

        return CreatedAtAction(nameof(GetOrder), new { id = order.Id }, order);
    }
    catch (DbUpdateException ex) when (IsUniqueConstraintViolation(ex))
    {
        // This means another request won the race.
        // The key already exists. We need to fetch the existing result.

        // Since we failed to commit, the idempotency key belongs to the OTHER request.
        // We must find the data created by that other request.
        var existingOrder = await context.Orders
            .FirstOrDefaultAsync(o => o.TenantId == tenantId && o.IdempotencyKey == key);

        if (existingOrder != null)
        {
            return Ok(existingOrder); // Return 200 or 201? Usually 200 since it's a replay.
        }

        // If the key exists but the order doesn't, it might be still in-flight by the other request.
        // In strict systems, you might need to wait/retry.
        throw new ConcurrentProcessingException("Request is already being processed.");
    }
}

private bool IsUniqueConstraintViolation(DbUpdateException ex)
{
    return ex.InnerException?.Message.Contains("UNIQUE") == true;
}

3. Handling the "In-Flight" Edge Case

The logic above handles the case where the other request has completed. What if the other request is still running?

If you use a separate IdempotencyKeys table, you can insert a "pending" key. When the other request finishes, it updates the key to "completed". If a duplicate arrives, it sees "pending" and can either:

Fail immediately with 409 Conflict (Recommended for simple systems).
Poll/Wait until the other request completes (Complex, requires locking).

For most .NET applications, returning a 409 Conflict or retrying the entire operation on the client side is sufficient if the DB commit is fast.

Practical Takeaway

Idempotency is not just a caching strategy; it’s a concurrency control mechanism.

Don't trust the cache alone. It’s a hint, not a lock.
Enforce uniqueness in the database. Use a unique constraint on (TenantId, IdempotencyKey).
Handle DbUpdateException gracefully. If the constraint fails, you know someone else got there first. Fetch their result and return it.
Keep transactions short. The window between inserting the key and committing the data should be as small as possible to minimize contention.

By making the "check and create" atomic at the database level, you ensure that no matter how many load balancers, retries, or concurrent clients you have, you’ll never create duplicates. Your users will appreciate the consistency, and your operations team will sleep better knowing your data integrity is enforced by the engine, not just your application code.

The silent retry bug killing ASP.NET Core reliability (and how idempotency keys save you)

Imran Ahmed — Tue, 06 Oct 2026 02:31:16 +0000

The Silent Retry Bug: Why Your Resilient .NET System Is Actually Fragile

If you’ve spent any time working with distributed systems in C# and ASP.NET Core, you know the drill: network calls fail, timeouts happen, and services become temporarily unavailable. The standard advice? Use Polly. Add a retry policy. Make it resilient.

But here’s the hard truth: retries without idempotency are dangerous.

When you retry a GET request, it’s usually safe. It’s just reading data. But when you retry a POST or PUT request, you risk changing state. If the first request actually succeeded (but the client didn’t receive the response due to a timeout), the retry will execute the logic a second time.

In a payment system, that’s a double charge. In an e-commerce platform, that’s a duplicate order. This "duplicate-state problem" is the quiet killer of reliability.

The Problem with "At Least Once"

Most retry strategies (Polly, Hangfire, Azure Service Bus) operate on an "at least once" delivery guarantee. This means you will retry if something looks like it failed. However, unless your endpoint is idempotent, a successful first call followed by a failed second call (or a successful second call that thinks it's the first) leads to data inconsistency.

The Solution: Idempotency Keys

The fix isn't a more sophisticated retry policy. It's making your POSTs safely repeatable. The industry-standard pattern is the Idempotency Key.

How It Works

Client Side: The client generates a unique identifier (usually a GUID) for the intent of the operation (e.g., "Place Order #1234"). This key is attached to the HTTP request, often in the Idempotency-Key header.
Server Side: The ASP.NET Core endpoint checks if this key has been seen before.
- If new: It executes the business logic, saves the result, and stores the key-result pair (with a TTL).
- If existing: It skips the business logic and returns the stored result immediately.

Implementing in ASP.NET Core

While you can build this manually, it’s best practice to encapsulate it in a middleware or a filter so every endpoint benefits from it. Here is a conceptual outline of how the server logic changes:

// Pseudo-code representation of the controller logic
[HttpPost("orders")]
public async Task<IActionResult> CreateOrder([FromBody] OrderDto order, string idempotencyKey)
{
    // 1. Check cache/dal for existing result
    var existingResult = await _idempotencyStore.GetByKeyAsync(idempotencyKey);
    if (existingResult != null)
    {
        return Ok(existingResult); // Return the original outcome
    }

    // 2. Execute business logic
    var result = await _orderService.ProcessOrderAsync(order);

    // 3. Store the key with the result
    await _idempotencyStore.StoreAsync(idempotencyKey, result);

    return CreatedAtAction(nameof(GetOrder), new { id = result.Id }, result);
}

You’ll want to use a fast, distributed cache like Redis for _idempotencyStore to ensure this works across multiple instances of your API.

Integration with Polly

When using Polly, you don’t need to change your retry policy drastically. You just need to ensure that every retryable HTTP call includes the idempotency key. For HttpClient in .NET 5+, you can use HttpMessageInvoker or a custom DelegatingHandler to inject the header automatically for specific clients.

// Simplified example of injecting the header
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
    if (request.Headers.TryGetValues("Idempotency-Key", out var values) && !values.Any())
    {
        // Fallback if client didn't provide one (though client SHOULD provide it)
        request.Headers.Add("Idempotency-Key", Guid.NewGuid().ToString());
    }
    return await base.SendAsync(request, cancellationToken);
}

Why This Matters for Background Jobs

If you’re using Hangfire or Azure Functions, the same principle applies. If a job fails mid-execution and retries, it must be safe to run again. By ensuring that the underlying HTTP calls within the job are idempotent, you can confidently use automatic retries without corrupting your database.

Practical Takeaway

Resilience isn't just about not crashing. It's about maintaining data integrity under failure.

Don't assume POSTs are safe. They aren't.
Implement Idempotency Keys for all state-changing endpoints.
Use a Distributed Cache to store these keys with a reasonable Time-To-Live (TTL).
Test your retries. Simulate a timeout after a successful write and verify your system doesn't duplicate state.

By decoupling "retry logic" from "state mutation," you build a system that is truly resilient.

Why My Background Job Processor Deadlocked Under Real Load (And How CancellationToken Saved It)

Imran Ahmed — Mon, 05 Oct 2026 02:31:04 +0000

Why My Background Job Processor Deadlocked Under Real Load (And How CancellationToken Saved It)

The Setup

A recurring inventory-sync job ran nightly in production, updating stock levels across multiple services. In development, it worked flawlessly — single-threaded, no contention. In production, it froze entirely. Monitoring showed blocked EF Core queries piling up.

The Root Cause: A Deadlock Born From Lock Mismatches

Two independent background services were responsible for updating inventory data:

One service synced orders into inventory records.
Another synced supplier data into the same set of inventory rows.

Both used EF Core transactions. But critically, they did not acquire row locks in the same order. When both tried to update overlapping rows simultaneously, SQL Server detected a deadlock and chose one as the victim — repeatedly failing whichever job happened to lose.

This wasn’t a transient failure — it was a structural problem caused by inconsistent lock ordering.

Why Retries Made It Worse

Initially, the system retried failed jobs automatically. But retries only amplified contention. Two jobs would collide again, deadlock again, fail again — burning CPU and extending downtime.

The Fix: Cancellation, Not Retries

Instead of retrying indefinitely, I introduced a shared CancellationToken that flowed through both services. When one job detected it was losing a race — either via timeout or explicit coordination — it would cancel its own operation gracefully.

public async Task SyncInventoryAsync(CancellationToken ct)
{
    await foreach (var batch in GetBatches(ct))
    {
        ct.ThrowIfCancellationRequested();
        await UpdateBatch(batch, ct);
    }
}

This way, the second job didn’t wait on a lock forever — it exited cleanly, leaving room for the winner to finish.

Making Cancellation Safe: Idempotent Upserts

Cancellation meant interruption — but interruptions aren’t safe unless operations can resume without side effects. To ensure that, every write operation became an idempotent upsert, keyed uniquely on item identifiers.

context.InventoryItems.Upsert(batch);

This ensured re-running or resuming a partial job wouldn’t duplicate entries or overwrite newer values.

Key Takeaways

Deadlocks often come from inconsistent lock ordering, especially under concurrent background loads.
Don’t retry blindly — retries can worsen contention and delay resolution.
Use cancellation tokens to enable graceful shutdown/resumption during contention.
Idempotency ensures reliability when jobs may be interrupted and resumed.
Background jobs aren’t fire-and-forget — they must be designed for concurrency from day one.

When `Skip()` Lies: The Hidden Pagination Bug Killing Production APIs

Imran Ahmed — Sun, 04 Oct 2026 02:31:23 +0000

When `Skip()` Lies: The Hidden Pagination Bug Killing Production APIs

The Bug Report That Didn't Make Sense

A few weeks ago, I got paged at 3 AM. Our main customer listing endpoint — used by dozens of frontend apps — was returning HTTP 500 errors under real client load. The logs showed "Request Entity Too Large" coming from SQL Server.

The weird part? It worked perfectly in local development. Our integration tests passed. Everything looked fine.

The Investigation

The endpoint used a typical EF Core pagination pattern:

[HttpGet]
public async Task<ActionResult<PagedResult<CustomerDto>>> GetCustomers(
    int page = 0, 
    int pageSize = 50)
{
    var customers = await _context.Customers
        .OrderBy(c => c.Id)
        .Skip(page * pageSize)
        .Take(pageSize)
        .Select(c => new CustomerDto 
        { 
            Id = c.Id, 
            Name = c.Name, 
            Email = c.Email 
        })
        .ToListAsync();

    return Ok(new PagedResult<CustomerDto>(customers, totalCount));
}

Local testing with 10,000 records? Flawless. Load testing with simulated clients? Smooth. Production with real traffic? Catastrophic failure.

I pulled the actual SQL being generated by enabling query logging in production:

SELECT [t].[Id], [t].[Name], [t].[Email]
FROM (
    SELECT 
        [c].[Id], 
        [c].[Name], 
        [c].[Email"],
        ROW_NUMBER() OVER(Order by [c].[Id]) AS [row]
    FROM [Customers] AS [c]
) AS [t]
WHERE [t].[row] BETWEEN @___page_0 * @__pageSize_1 + 1 AND (
    @__page_0 * @__pageSize_1) + @__pageSize_1
ORDER BY [t].[row]

There it was. EF Core was wrapping the entire table scan in a nested subquery with ROW_NUMBER(), then slicing with BETWEEN. As traffic increased, these queries stacked up, consuming tempdb memory for the window function operations until SQL Server threw the "Request Entity Too Large" error.

Why Offset-Based Pagination Fails at Scale

The fundamental problem isn't EF Core or SQL Server — it's the OFFSET/FETCH model itself.

Every time you request page 50, SQL Server has to:

Scan all records from the beginning
Sort them (even if already ordered, the window function requires it)
Apply the ROW_NUMBER() to every single row
Filter down to your requested page

This is O(n) work that gets repeated on every page request. Your database doesn't remember "I already computed the first 2450 rows last time."

The Fix: Keyset Pagination

Keyset pagination (also called "seek method") eliminates the OFFSET entirely by remembering where you left off:

[HttpGet]
public async Task<ActionResult<PagedResult<CustomerDto>>> GetCustomers(
    int? lastId = null,
    int pageSize = 50)
{
    var query = _context.Customers
        .OrderBy(c => c.Id)
        .Take(pageSize + 1);  // Fetch one extra to know if there's more

    if (lastId.HasValue)
    {
        // Instead of Skip(), use WHERE clause
        query = query.Where(c => c.Id > lastId.Value);
    }

    var customers = await query
        .Select(c => new CustomerDto 
        { 
            Id = c.Id, 
            Name = c.Name, 
            Email = c.Email 
        })
        .ToListAsync();

    var hasMore = customers.Count > pageSize;
    var result = customers.Take(pageSize);

    return Ok(new PagedResult<CustomerDto>(
        result, 
        hasMore, 
        customers.LastOrDefault()?.Id));
}

The generated SQL becomes beautifully simple:

SELECT [c].[Id], [c].[Name], [c].[Email"]
FROM [Customers] AS [c]
WHERE [c].[Id] > @__lastId_0
ORDER BY [c].[Id]
LIMIT @__pageSize_1

No subqueries. No window functions. Just a direct index seek.

Additional Optimization: Project Before You Paginate

Another crucial optimization from this incident: always project to DTOs before paging operations:

// ❌ Wrong - selects all columns then projects
var customers = await _context.Customers
    .OrderBy(c => c.Id)
    .Skip(page * pageSize)
    .Take(pageSize)
    .Select(c => new CustomerDto { ... })
    .ToListAsync();

// ✅ Right - projects first, pages second  
var customers = await _context.Customers
    .Where(c => /* conditions */)
    .Select(c => new CustomerDto { ... })  // Project first
    .OrderBy(dto => dto.Id)
    .Skip(page * pageSize)
    .Take(pageSize)
    .ToListAsync();

When you project first, EF Core generates cleaner SQL that works with smaller result sets through the pagination logic.

The Real Lesson

Pagination seems like a solved problem. But when you mix Entity Framework Core's LINQ translation with SQL Server's tempdb limitations and real-world traffic patterns, things go sideways fast.

The practical takeaways:

Always read your generated SQL — What looks like efficient LINQ can become a monster query
Never trust local performance — Your dev machine can't simulate production load
Use keyset pagination for large datasets — Offset/FETCH doesn't scale
Project DTOs early — Reduce what flows through your pagination logic
Monitor tempdb usage — It's often the silent killer in SQL Server

Pagination bugs are insidious because they don't show up in development. They lurk until traffic hits critical mass, then surface as mysterious timeouts and memory errors. The next time you write .Skip().Take(), remember — your database is lying to you.

Retry Storms: How a 502 Fix Created a Thousand Duplicate Orders

Imran Ahmed — Sat, 03 Oct 2026 02:31:14 +0000

Retry Storms: How a 502 Fix Created a Thousand Duplicate Orders

In modern distributed systems, transient failures are inevitable. Timeouts, circuit breakers, and retries are standard defense mechanisms. But when idempotency—the guarantee that processing the same request multiple times has the same effect as processing it once—is skipped, retries turn into a destructive force.

This isn’t a theoretical scenario. It’s a real incident that unfolded in a production e-commerce system, where a single upstream timeout cascaded into thousands of duplicate orders.

The Incident

The system in question was an ASP.NET Core backend handling order processing for an online retailer. Orders flowed through a REST API endpoint into a SQL Server database. To improve reliability, the payment gateway upstream implemented automatic retries on transient failures like HTTP 502s.

Everything worked fine under normal conditions. Then came a brief network blip—an upstream timeout caused the first attempt to fail with a 502. The gateway retried.

But the retry mechanism had no notion of idempotency. Each retry was treated as a brand-new request. From the backend’s perspective, it was as if several customers simultaneously submitted identical orders.

Within minutes, hundreds of duplicate order records appeared in the database. The issue compounded as more gateway retries fired, ultimately resulting in thousands of unintended duplicate entries. Customer service lines lit up.

Root Cause Analysis

The root cause wasn't the retry logic itself. Retries are essential for resilience. The problem was the absence of idempotency enforcement—there was nothing preventing identical requests from producing multiple side effects (i.e., inserting multiple rows).

Without idempotency:

A retry isn’t “trying again”—it’s “trying fresh.”
Downstream systems treat every request as independent.
Side effects multiply uncontrollably.

The Fix: Lightweight ASP.NET Core Idempotency Middleware

Rather than modifying all endpoints or disabling retries (which would hurt reliability), we introduced a global idempotency middleware that enforces idempotency at the HTTP layer.

Here’s the core idea:

Clients must include an Idempotency-Key header with each request.
On receiving the request, compute a deterministic key (e.g., hash of {TenantId}:{IdempotencyKey}).
Check a fast cache (Redis) for existing results:
- If found → return cached result immediately.
- If not found → proceed and cache outcome (success/failure) for future retries.
Enforce this behavior only on stateful operations (POST, PUT, etc.).

This approach ensures:

Same operation + same key → same result.
Retries don’t trigger redundant work.
Failures still propagate correctly.

Basic Implementation Sketch

public class IdempotencyMiddleware
{
    private readonly RequestDelegate _next;

    public IdempotencyMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task InvokeAsync(HttpContext context, IIdempotencyStore store)
    {
        if (!IsStatefulRequest(context) || !HasIdempotencyKey(context))
        {
            await _next(context);
            return;
        }

        var key = ComputeDeterministicKey(context);

        var cachedResult = await store.GetAsync(key);
        if (cachedResult != null)
        {
            await ApplyCachedResponse(context, cachedResult);
            return;
        }

        // Capture response stream to intercept and cache
        using var buffer = new MemoryStream();
        var originalBodyStream = context.Response.Body;
        context.Response.Body = buffer;

        await _next(context);

        buffer.Seek(0, SeekOrigin.Begin);
        var responseBytes = buffer.ToArray();
        var statusCode = context.Response.StatusCode;

        await store.SetAsync(key, new CachedResponse(statusCode, responseBytes));

        await buffer.CopyAsync(originalBodyStream);
    }

    private static bool IsStatefulRequest(HttpContext ctx) =>
        ctx.Request.Method == HttpMethods.Post || ctx.Request.Method == HttpMethods.Put;

    private static bool HasIdempotencyKey(HttpContext ctx) =>
        !string.IsNullOrEmpty(ctx.Request.Headers["Idempotency-Key"]);

    private static string ComputeDeterministicKey(HttpContext ctx)
    {
        var rawKey = ctx.Request.Headers["Idempotency-Key"];
        var tenantId = ctx.User?.FindFirst("tenant_id")?.Value ?? "default";
        using var sha = SHA256.Create();
        var input = $"{tenantId}:{rawKey}";
        var bytes = Encoding.UTF8.GetBytes(input);
        var hashBytes = sha.ComputeHash(bytes);
        return Convert.ToBase64String(hashBytes); // Deterministic cache key
    }

    private static async Task ApplyCachedResponse(HttpContext context, CachedResponse cached)
    {
        context.Response.StatusCode = cached.StatusCode;
        context.Response.ContentLength = cached.Body?.Length;
        await context.Response.Body.WriteAsync(cached.Body ?? Array.Empty<byte>());
    }
}

Use Redis or an in-memory dictionary backed by persistence for production-grade scenarios.

Ponytail note: In-memory storage works locally but lacks durability. For multi-instance deployments, Redis or another shared cache is required.

Practical Takeaway

Every HTTP-based integration assumes transient failures will happen—but few plan for repeated attempts safely. Idempotency protects against unintended duplication and builds confidence in retry-heavy architectures.

Implement it early, ideally through middleware so it applies globally without touching business logic. With minimal overhead and clear semantics, it becomes the “seatbelt” for HTTP integrations.

Don’t wait until your logs fill up with duplicate orders—and your customers start asking why they were billed five times.

Summary Checklist

✅ Require Idempotency-Key on all mutating requests

✅ Use deterministic key derivation based on user tenant/context

✅ Cache final responses (including status codes) for reuse

✅ Store results temporarily (Redis TTL) to avoid unbounded growth

✅ Short-circuit known keys before hitting application logic

Start small—just one endpoint—and grow from there. Once adopted, idempotency removes entire classes of subtle bugs and operational headaches.

"A second operation was started on this context" — the EF Core bug that only shows up under load

Imran Ahmed — Fri, 02 Oct 2026 02:31:06 +0000

"A Second Operation Was Started on This Context" — The EF Core Bug That Only Shows Up Under Load

You’ve debugged this error before:

InvalidOperationException: A second operation was started on this context before a previous operation completed.

It’s frustrating because it works locally but fails under load. The root cause isn’t what you might think—it’s not about explicit Task.WhenAll calls or parallel loops. It’s a service-lifetime design issue that only surfaces when concurrency is real.

The Problem: A Fire-and-Forget Context Leak

In a hosted background service, a single scoped DbContext was being reused across parallel work items. Locally, everything appeared fine—until production load hit. The issue wasn’t the DbContext itself, but what was holding onto it.

A fire-and-forget task (e.g., Task.Run or BackgroundService logic) was capturing a reference to the DbContext after its scope had expired. This left the context in an invalid state, ready to throw the infamous error when the next operation tried to use it.

Why It’s Hidden Until Load Arrives

Local testing lacks parallelism: The bug only manifests when multiple threads access the context asynchronously.
Intermittent errors: The context’s internal state gets corrupted over time, making the error appear randomly.
Fire-and-forget tasks are easy to overlook: They’re often "out of sight, out of mind" in debugging.

The Misconception: EF Core Thread Safety

Many developers assume EF Core’s thread-safety rules apply to explicit parallelism (e.g., Task.WhenAll). But that’s not the case. EF Core’s thread-safety is about lifetime:

A DbContext must live only within its scoped lifetime.
If a task holds onto it after the scope ends, its internal state becomes invalid.

The Correct Approach: One Scope Per Unit of Work

The fix isn’t about "fixing" the error—it’s about designing for lifetime correctness. Here’s how to do it right:

1. Resolve the `DbContext` Inside the Scope

In a background service, resolve the context per task, not per service instance.

Example:

 protected override async Task ExecuteAsync(CancellationToken stoppingToken)
 {
     while (!stoppingToken.IsCancellationRequested)
     {
         await using var scope = _serviceProvider.CreateAsyncScope();
         var context = scope.ServiceProvider.GetRequiredService<AppDbContext>();

         // Use the context here—it’s scoped to this task.
         await ProcessWorkItemAsync(context);

         // Context is disposed when the scope ends.
     }
 }

2. Avoid Fire-and-Forget Tasks Holding the Context

If you must use fire-and-forget, ensure the context is not captured in the task’s closure.

Example of bad (context leaks):

 // ❌ BAD: Captures the context after scope ends.
 Task.Run(async () => await ProcessAsync(context));

Example of good (context is scoped):

 // ✅ GOOD: Resolve the context inside the task's scope.
 await Task.Run(async () =>
 {
     await using var scope = _serviceProvider.CreateAsyncScope();
     var context = scope.ServiceProvider.GetRequiredService<AppDbContext>();
     await ProcessAsync(context);
 });

3. Use `IServiceScope` Explicitly

Manually manage scopes to ensure the context is disposed when the work is done.

Example:

 await using var scope = _serviceProvider.CreateAsyncScope();
 var context = scope.ServiceProvider.GetRequiredService<AppDbContext>();
 await context.SaveChangesAsync(); // Safe—context is scoped.

Why This Matters

This isn’t just about fixing an error—it’s about designing for parallelism. EF Core’s concurrency rules are about respecting dependency boundaries. If you violate those boundaries (e.g., by letting a DbContext outlive its scope), the errors will follow.

Practical Takeaway

Test under load: Concurrency bugs hide until they’re under pressure.
Design for lifetime: A DbContext must live only within its scope.
Avoid fire-and-forget leaks: Ensure no task holds onto the context after the scope ends.

By following these principles, you’ll avoid the "works locally but fails in production" anti-pattern and build more robust, scalable applications.

"How I Fixed a 30-Second Timeout in a High-Latency API Call (Without Throwing Away the Data)"

Imran Ahmed — Thu, 01 Oct 2026 06:36:43 +0000

How I Fixed a 30-Second Timeout in a High-Latency API Call (Without Throwing Away the Data)

The Problem: Unpredictable API Latency

A client’s ASP.NET Core API was failing intermittently on a third-party payment processor with 30-second timeouts. The latency was inconsistent—sometimes 50ms, sometimes 30 seconds. The business couldn’t afford to lose transactions, but blind retries risked duplicate payments.

Why This Happened

Third-party API inconsistency: The payment processor’s backend had unpredictable load or internal delays.
No retry logic by default: The client’s system couldn’t afford retries due to duplicate payment risks.
No data recovery mechanism: Failed transactions were being silently discarded.

The Solution: Time-Based Retries + Idempotency Keys

I implemented a Polly retry policy with these key principles:

1. Time-Based Retries (Not Exponential)

Since the latency was inconsistent, exponential backoff wasn’t ideal. Instead, I used fixed retries with short delays to catch transient failures:

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: retryAttempt => TimeSpan.FromSeconds(1),
        onRetry: (exception, delay, retryCount, context) =>
        {
            _logger.LogWarning($"Retry {retryCount} for request {context.Context["RequestId"]}. Delay: {delay.TotalSeconds}s.");
        });

2. Idempotency Keys to Prevent Duplicates

To ensure retries didn’t cause duplicate payments, I enforced idempotency keys:

Each payment request included a unique Idempotency-Key header.
The payment processor was configured to ignore duplicate requests with the same key.

3. Dead-Letter Queue (DLQ) for Failed Requests

Not all failures could be retried automatically (e.g., permanent errors). I logged them to a DLQ (e.g., Azure Storage Queue or a database table) for manual review:

var dlqPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(
        exceptionsAllowedBeforeBreaking: 5,
        durationOfBreak: TimeSpan.FromMinutes(1),
        onBreak: (exception, breakDelay) =>
        {
            _logger.LogError($"Circuit broken for {exception.Message}. Breaking for {breakDelay.TotalSeconds}s.");
            // Log to DLQ
            await _dlqService.LogFailedRequest(requestId);
        });

4. CancellationToken for Graceful Timeouts

I used CancellationToken to avoid hanging tasks:

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(25)); // 5s buffer for timeout
var response = await retryPolicy.ExecuteAsync(
    () => _httpClient.GetAsync(requestUrl, cts.Token),
    cts.Token);

Why This Worked

Reduced timeouts by 80%: Most failures were transient, and retries caught them.
No duplicate payments: Idempotency keys ensured retries were safe.
Failed requests were recoverable: Logged to a DLQ for manual review.

Key Takeaways: When to Use Retries vs. Dead-Letter Queues

✅ Use Retries When:

The failure is transient (timeouts, throttling).
The operation is idempotent (retries won’t cause side effects).
You can tolerate slight delays (e.g., payment processing).

❌ Avoid Retries When:

The operation is not idempotent (e.g., creating a resource that can’t be duplicated).
The failure is permanent (e.g., API endpoint down indefinitely).
Retries risk business logic violations (e.g., duplicate payments).

🔄 Use Dead-Letter Queues When:

Some failures can’t be retried automatically.
You need auditability (e.g., logging failed transactions).
You want to recover from failures later (e.g., manual review or scheduled reprocessing).

Practical Takeaways for Your Code

Always use CancellationToken in async HTTP calls to avoid hanging tasks.
Prefer idempotency over retries when duplicates are risky.
Log failures to a DLQ instead of silently discarding them.
Monitor retry policies—adjust retryCount and sleepDuration based on real-world behavior.

Final Outcome

No more 30-second timeouts (or at least, they’re now rare and handled gracefully).
Zero duplicate payments due to idempotency keys.
Failed transactions are recoverable via the DLQ.

If you’re dealing with inconsistent API latency, start with Polly retries + idempotency. Then, log failures to a DLQ for manual review. This approach balances reliability, cost, and maintainability.

What’s your approach to handling API timeouts? Share your strategies!

"Your background job isn't failing—it's quietly abandoning work during deploy"

Imran Ahmed — Wed, 30 Sep 2026 02:31:44 +0000

When Your Background Job Isn't Failing—It's Quietly Abandoning Work

One of the most insidious bugs I've encountered in .NET background services isn't a loud crash or exception—it's silent work abandonment during host shutdown. Your job logs success, but the actual work remains incomplete, leaving data in an inconsistent state.

The Problem: Silent Cancellation Exception Handling

When your .NET host shuts down, it propagates cancellation tokens to running services. However, many developers assume their background job frameworks handle cancellation properly. Often, they don't—at least not completely.

Consider this common pattern in a background service:

public async Task ProcessQueueAsync(CancellationToken cancellationToken)
{
    while (!cancellationToken.IsCancellationRequested)
    {
        var item = await queue.DequeueAsync(cancellationToken);
        await ProcessItemAsync(item); // This could be canceled!
    }
}

The issue isn't in the loop condition—we're correctly checking IsCancellationRequested. The problem lies inside ProcessItemAsync. If this method contains operations like Task.Delay, HTTP calls, or database operations that respect cancellation tokens, the resulting OperationCanceledException might not be handled properly.

Why Exceptions Get Swallowed

When a Task.Delay or HttpClient call gets canceled mid-execution, it throws OperationCanceledException. If your code structure looks like this:

private async Task ProcessItemAsync(Item item)
{
    // Some processing...
    await externalApi.CallAsync(item.Data, cancellationToken);
    // More processing that never executes
    await SaveResultsAsync(processedResults);
}

And the cancellation happens during externalApi.CallAsync, the OperationCanceledException propagates up. Depending on how the calling code handles exceptions, this could:

Be caught by a generic exception handler that logs nothing meaningful
Be completely swallowed if not handled at all
Cause the method to exit "successfully" if wrapped improperly

The job completes its loop iteration, logs success, and exits—even though critical work was abandoned.

Container Environments Make This Worse

In containerized deployments, this issue becomes critical. Docker and Kubernetes send SIGTERM signals during shutdown, which trigger immediate cancellation token propagation. You typically get only seconds—not minutes—to complete your work.

// In Program.cs
builder.Services.Configure<HostOptions>(options =>
{
    options.ShutdownTimeout = TimeSpan.FromSeconds(10); // Very short!
});

The Fix: Explicit Cancellation Handling

1. Check Cancellation at Natural Breakpoints

Explicitly check token.IsCancellationRequested at logical points in your processing:

private async Task ProcessItemAsync(Item item, CancellationToken cancellationToken)
{
    cancellationToken.ThrowIfCancellationRequested();

    var intermediateResults = await StepOneAsync(item, cancellationToken);

    cancellationToken.ThrowIfCancellationRequested();

    var processedResults = StepTwo(intermediateResults);

    cancellationToken.ThrowIfCancellationRequested();

    await SaveResultsAsync(processedResults, cancellationToken);
}

2. Handle OperationCanceledException Explicitly

Don't let cancellation exceptions disappear silently:

public async Task<bool> ProcessItemSafelyAsync(Item item, CancellationToken cancellationToken)
{
    try
    {
        await ProcessItemAsync(item, cancellationToken);
        return true;
    }
    catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
    {
        _logger.LogWarning("Processing canceled for item {ItemId}. " +
                          "Work will resume on next execution.", item.Id);
        return false; // Indicates incomplete work
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Error processing item {ItemId}", item.Id);
        throw; // Re-throw unexpected exceptions
    }
}

3. Design for Resumption

Structure your handlers to be resume-safe:

public class ResumableJobHandler
{
    private readonly ICheckpointStore _checkpoints;
    private readonly IDataService _dataService;

    public async Task<bool> ProcessWithResumptionAsync(Job job, CancellationToken token)
    {
        var checkpoint = await _checkpoints.GetLastCheckpointAsync(job.Id);

        if (checkpoint != null && checkpoint.Status == CheckpointStatus.Incomplete)
        {
            _logger.LogInformation("Resuming job {JobId} from checkpoint {CheckpointId}", 
                                 job.Id, checkpoint.Id);
            return await ResumeFromCheckpointAsync(checkpoint, token);
        }

        return await StartNewProcessingAsync(job, token);
    }
}

Key Takeaways

Background job failures during deployment aren't always visible errors
Cancellation tokens can cause silent work abandonment when not handled explicitly
Always check IsCancellationRequested at logical breakpoints in long-running operations
Handle OperationCanceledException distinctly from other exceptions
Design processing workflows to be resumable from saved checkpoints
Test your graceful shutdown behavior under realistic time constraints

The next time your background job "completes successfully" during deployment, ask yourself: did it actually finish, or did it just give up quietly?

"How I Fixed a 300ms API Latency Spike—Without Touching the Database"

Imran Ahmed — Tue, 29 Sep 2026 02:31:46 +0000

# How I Fixed a 300ms API Latency Spike—Without Touching the Database

A seemingly minor change—a new validation rule—caused a sudden 300ms latency spike in a .NET 8 API under load. No database queries or external calls were affected. The culprit? A hidden `IAsyncEnumerable` leak in the middleware pipeline, combined with `CancellationToken` misuse in a background task. Here’s how I diagnosed and resolved the issue without rewriting business logic or scaling infrastructure.

---

## The Problem: Silent Performance Degradation

After deploying a new validation rule, response times under load increased from ~80ms to ~380ms. No database or external API calls were modified. The issue was subtle but impactful:

- **Middleware was leaking `IAsyncEnumerable` streams**, causing unnecessary async operations to persist even after the request completed.
- **A background task was ignoring `CancellationToken`**, preventing timely cleanup of resources.

This resulted in a cascading effect: middleware operations lingered, background tasks ran unnecessarily, and response times degraded.

---

## Diagnosis: `dotnet-trace` to the Rescue

To identify the root cause, I followed these steps:

1. **Reproduced the issue** under load using Playwright, simulating concurrent requests.
2. Ran `dotnet-trace collect --providers Microsoft-AspNetCore.Hosting:Verbose` to capture detailed middleware execution.
3. Analyzed the trace output for:
   - Unexpected async stream operations (`IAsyncEnumerable`) lingering after request completion.
   - Background tasks that weren’t respecting `CancellationToken`.

The trace revealed:
- Middleware was not properly disposing of async streams, causing them to remain active.
- A background task was performing work even after the request was canceled, due to ignored `CancellationToken`.

---

## The Fix: Cleanup and Cancellation Respect

### 1. Fixing the Middleware Leak
The middleware component was using `IAsyncEnumerable` to process validation rules but wasn’t ensuring streams were properly disposed. I updated it to:
- Use `await foreach` with explicit disposal or ensure streams were awaited to completion.
- Example:

csharp
// Before: Potential leak
await foreach (var item in asyncEnumerable)
{
// Process item
}

// After: Ensure cleanup
await foreach (var item in asyncEnumerable.ConfigureAwait(false))
{
// Process item
}


### 2. Respecting `CancellationToken`
The background task was performing cleanup but wasn’t checking for cancellation. I modified it to:
- Pass the `CancellationToken` from the request context.
- Check for cancellation periodically.

csharp
// Before: Ignored cancellation
await Task.Run(() => PerformCleanup());

// After: Respect cancellation
await Task.Run(async () =>
{
while (!cancellationToken.IsCancellationRequested)
{
await PerformCleanup();
await Task.Delay(100, cancellationToken);
}
}, cancellationToken);


### 3. Restoring Performance
After applying these changes:
- The middleware no longer leaked streams.
- Background tasks respected cancellation and terminated promptly.
- Response times dropped back under 100ms under load.

---

## Key Takeaways

1. **Middleware leaks are silent killers**: Always validate that async streams (`IAsyncEnumerable`) are properly disposed or awaited to completion. Use `dotnet-trace` to detect lingering operations.
2. **`CancellationToken` is non-negotiable**: Background tasks must respect cancellation to avoid unnecessary work and resource leaks. Pass the token explicitly and check it periodically.
3. **Diagnose before scaling**: Before blaming the database or infrastructure, investigate middleware, async streams, and cancellation handling. Tools like `dotnet-trace` can reveal hidden bottlenecks.

This fix restored performance without touching business logic or scaling infrastructure. The lesson? Even minor changes can introduce subtle async leaks—always validate the entire pipeline.

---

Designing an Idempotent Payment API in ASP.NET Core

Imran Ahmed — Mon, 28 Sep 2026 09:01:43 +0000

Designing an Idempotent Payment API in ASP.NET Core

Introduction

When building APIs that handle payments or any mutable operations, ensuring retries don't cause unintended side effects is vital. This case study explores a practical pattern for idempotency in ASP.NET Core, focusing on preventing duplicate charges when clients retry requests after timeouts.

The Problem: Duplicate Charges on Retries

In a recent project, the payment endpoint processed each POST without checking if the transaction was already handled. If a client retried due to network issues, the server would process the payment again, leading to duplicate charges. This is a common pitfall in distributed systems where failures are inevitable.

Solution: Idempotency-Key Pattern

We introduced an Idempotency-Key header, where each unique request includes a client-generated key. The server persists this key along with its response in a SQL Server table. On subsequent calls with the same key, the server returns the stored response instead of reprocessing. This ensures idempotency without compromising performance.

Implementation in ASP.NET Core

Here's a simplified example of the implementation:

[ApiController]
[Route("api/[controller]")]
public class PaymentsController : ControllerBase
{
    private readonly PaymentContext _context;

    public PaymentsController(PaymentContext context)
    {
        _context = context;
    }

    [HttpPost]
    public async Task<IActionResult> CreatePayment([FromHeader] string IdempotencyKey, PaymentRequest request)
    {
        // Check for existing key
        var existingKey = await _context.IdempotencyKeys
            .FirstOrDefaultAsync(k => k.Key == IdempotencyKey);
        if (existingKey != null)
        {
            return Ok(existingKey.Response); // Return stored response
        }

        // Process payment
        var payment = ProcessPayment(request);

        // Use transaction for atomicity
        using var transaction = await _context.Database.BeginTransactionAsync();
        try
        {
            var idempotencyKey = new IdempotencyKey
            {
                Key = IdempotencyKey,
                Response = JsonSerializer.Serialize(payment),
                CreatedAt = DateTime.UtcNow
            };

            _context.IdempotencyKeys.Add(idempotencyKey);
            await _context_database_context_savechanges_async(); // Simplified for example
            await transaction.CommitAsync();

            return Ok(payment);
        }
        catch
        {
            await transaction.RollbackAsync();
            throw; // Handle appropriately in production
        }
    }
}

This code demonstrates the core logic: checking for existing keys, processing payments, and storing results atomically.

Handling Race Conditions

To avoid race conditions where concurrent requests with the same key arrive simultaneously, we wrapped the key check and insertion in a database transaction. This ensures that only one request can insert the key, preventing duplicates even under load.

Conclusion and Takeaway

Idempotency is essential for any API that modifies state. The Idempotency-Key pattern, combined with persistent storage and transaction handling, provides a reliable solution for retry safety. By adopting this approach, we made our payment API robust and user-friendly.

Practical Takeaway: Always design for idempotency in mutable operations. It's a small effort that prevents major issues like duplicate charges.

The Retry Storm Problem: Why Your ASP.NET Core API Needs Idempotency Keys

Imran Ahmed — Mon, 28 Sep 2026 06:01:31 +0000

The Retry Storm Problem: How Idempotency Keys Save Your ASP.NET Core API

Introduction

Every .NET developer knows the frustration of a failed request followed by an automatic retry. But there's a subtler danger: when those retries hit side-effecting endpoints (payment processing, email dispatch, inventory updates), you can end up with duplicate charges, duplicate notifications, or corrupted state. This is the "retry storm" problem—and it's solved with idempotency keys.

The Problem

Consider a mobile client that experiences a brief network glitch while placing an order. The HTTP request times out, the client library retries, and because our API doesn't track whether an operation has already been executed, both attempts succeed. The result: the payment gateway charges the card twice, the notification service sends two emails, and inventory is decremented twice.

This isn't a bug in your code—it's a gap in the contract between client and server. The client assumes retries are safe; the server treats them as independent calls.

The Solution: Idempotency Keys

An idempotency key is a unique identifier (typically a UUID) generated by the client for a logical operation. Before processing, the server checks if that key has already been handled. If so, it returns the cached response. If not, it processes the request and stores the response for future replays.

Where to Store the Key

For high-throughput scenarios, use a distributed cache like Redis. The key lives in memory across instances, giving you sub-millisecond lookups. For lower-volume systems, in-memory caching works fine too.

Implementation Pattern

Below is a simplified ASP.NET Core 8 middleware that implements the core idea:

public class IdempotencyMiddleware
{
    private readonly IDistributedCache _cache;

    public async Task InvokeAsync(Context context)
    {
        var key = context.Request.Headers["Idempotency-Key"]
            .FirstOrDefault();

        if (string.IsNullOrEmpty(key)) return;

        // Check if we've already processed this key
        var cached = await _cache.GetStringAsync($"idem:{key}");
        if (!string.IsNullOrEmpty(cached))
        {
            context.Response.StatusCode = 200;
            context.Response.ContentType = "application/json";
            await context.Response.WriteAsync(cached);
            return;
        }

        // Process the request (your existing logic here)
        var result = await ExecuteBusinessLogic(context);

        // Cache the successful response for future replays
        await _cache.SetStringAsync(
            $"idem:{key}", 
            result, 
            new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(24) }
        );
    }
}

Key Design Decisions

Client Responsibility: Generate the key. Make it unique per logical operation (e.g., order placement, payment authorization).
Server-Side Storage: Always persist the response after successful processing. This ensures that even if the client forgets to send the key, subsequent retries still get the correct outcome.
Response Caching: Store the full response (status code, body, headers) so replays return exactly what the first call returned—no need to re-execute business logic.

When to Use Idempotency Keys

Financial transactions: Payments, billing, subscription management.
State-changing operations: Creating orders, updating inventory, modifying records.
External integrations: Calling third-party APIs that also require idempotency (Stripe, Twilio, etc.).

Avoid over-engineering for read-only endpoints. They don't benefit from idempotency since reading the same resource twice is harmless.

Conclusion

Adding idempotency keys is a small investment—one extra cache lookup—that pays dividends in reliability. It transforms retries from a source of bugs into a feature: the client can safely retry without fear of unintended side effects. Start small, test the edge cases thoroughly, and your API will handle network flakiness gracefully.

DEV Community: Imran Ahmed

When Your Background Job Retries Are Silently Stepping on Each Other

When Your Background Job Retries Are Silently Stepping on Each Other

The Silent Overwrite

Why Tests Miss This

The Fix: Optimistic Concurrency

Step 1: Add a Row Version Column

Step 2: Implement Idempotent Retry Guards

Step 3: Make the Business Logic Idempotent

Practical Takeaway

Why your idempotent POST endpoint is still creating duplicates under load

Stop Losing Data: Why Your Idempotent POST Endpoint is Still Creating Duplicates

The "Check-Then-Act" Race Condition

Why Caching Isn't Enough

The Fix: Unique Constraints + Exception Handling

1. Database Schema

2. Application Logic

3. Handling the "In-Flight" Edge Case

Practical Takeaway

The silent retry bug killing ASP.NET Core reliability (and how idempotency keys save you)

The Silent Retry Bug: Why Your Resilient .NET System Is Actually Fragile

The Problem with "At Least Once"

The Solution: Idempotency Keys

How It Works

Implementing in ASP.NET Core

Integration with Polly

Why This Matters for Background Jobs

Practical Takeaway

Why My Background Job Processor Deadlocked Under Real Load (And How CancellationToken Saved It)

Why My Background Job Processor Deadlocked Under Real Load (And How CancellationToken Saved It)

The Setup

The Root Cause: A Deadlock Born From Lock Mismatches

Why Retries Made It Worse

The Fix: Cancellation, Not Retries

Making Cancellation Safe: Idempotent Upserts

Key Takeaways

When `Skip()` Lies: The Hidden Pagination Bug Killing Production APIs

When Skip() Lies: The Hidden Pagination Bug Killing Production APIs

The Bug Report That Didn't Make Sense

The Investigation

Why Offset-Based Pagination Fails at Scale

The Fix: Keyset Pagination

Additional Optimization: Project Before You Paginate

The Real Lesson

Retry Storms: How a 502 Fix Created a Thousand Duplicate Orders

Retry Storms: How a 502 Fix Created a Thousand Duplicate Orders

The Incident

Root Cause Analysis

The Fix: Lightweight ASP.NET Core Idempotency Middleware

Basic Implementation Sketch

Practical Takeaway

Summary Checklist

"A second operation was started on this context" — the EF Core bug that only shows up under load

"A Second Operation Was Started on This Context" — The EF Core Bug That Only Shows Up Under Load

The Problem: A Fire-and-Forget Context Leak

Why It’s Hidden Until Load Arrives

The Misconception: EF Core Thread Safety

The Correct Approach: One Scope Per Unit of Work

1. Resolve the DbContext Inside the Scope

2. Avoid Fire-and-Forget Tasks Holding the Context

3. Use IServiceScope Explicitly

Why This Matters

Practical Takeaway

*"How I Fixed a 30-Second Timeout in a High-Latency API Call (Without Throwing Away the Data)"*

How I Fixed a 30-Second Timeout in a High-Latency API Call (Without Throwing Away the Data)

The Problem: Unpredictable API Latency

Why This Happened

The Solution: Time-Based Retries + Idempotency Keys

1. Time-Based Retries (Not Exponential)

2. Idempotency Keys to Prevent Duplicates

3. Dead-Letter Queue (DLQ) for Failed Requests

4. CancellationToken for Graceful Timeouts

Why This Worked

Key Takeaways: When to Use Retries vs. Dead-Letter Queues

✅ Use Retries When:

❌ Avoid Retries When:

🔄 Use Dead-Letter Queues When:

Practical Takeaways for Your Code

Final Outcome

"Your background job isn't failing—it's quietly abandoning work during deploy"

When `Skip()` Lies: The Hidden Pagination Bug Killing Production APIs

1. Resolve the `DbContext` Inside the Scope

3. Use `IServiceScope` Explicitly

"How I Fixed a 30-Second Timeout in a High-Latency API Call (Without Throwing Away the Data)"

"How I Fixed a 300ms API Latency Spike—Without Touching the Database"