Racing Background Jobs: The Distributed Lock Gap in ASP.NET Core

Introduction

Background jobs are a common pattern in modern .NET applications—processing messages from a queue, writing to a database, sending emails, or invoking external APIs. When you run a single instance, the job is naturally serialized. As soon as you scale out by adding more instances (e.g., Kubernetes pods, Azure App Service slots, or multiple VMs), the guarantee of single execution disappears.

If two instances pick up the same message at the same time, the work can be performed twice. This leads to:

• Duplicate rows in a table
• Inconsistent state in event‑sourced systems
• Over‑charged customers or double‑sent emails

The problem often stays hidden until production traffic spikes, making it a classic “silent failure.”

Why the Race Happens

Consider a background service that listens to an Azure Service Bus queue:

// Pseudo‑code
while (true)
{
    var message = await queueReceiver.ReceiveAsync();
    ProcessMessage(message);
}

When you scale to two pods, both may receive the same ReceiveAsync call before the first pod finishes ProcessMessage. The race window is tiny, but under load it becomes frequent.

The Two‑Layer Pattern

To eliminate the race, combine distributed locking with idempotent processing.

1️⃣ Distributed Lock

A lock guarantees that only one process holds the exclusive right to run the critical section. In .NET, a lightweight way to implement this is with Redis.

public class RedisDistributedLock : IDisposable
{
    private readonly ConnectionMultiplexer _redis;
    private readonly IDatabase _db;
    private readonly string _lockKey;
    private readonly TimeSpan _ttl;
    private bool _expired;

    public RedisDistributedLock(string connectionString, string lockKey, TimeSpan ttl = default)
    {
        _redis = ConnectionMultiplexer.Connect(connectionString);
        _db = _redis.GetDatabase();
        _lockKey = lockKey;
        _ttl = ttl == default ? TimeSpan.FromSeconds(30) : ttl;
    }

    public bool TryAcquire(TimeSpan waitTime)
    {
        var lockToken = Guid.NewGuid().ToString();
        var lockExpire = DateTime.UtcNow.Add(_ttl);
        var acquired = _db.LockAsync(_lockKey, lockToken, lockExpire, waitTime);
        if (acquired) _expired = false;
        return acquired;
    }

    public void Release()
    {
        _db.LockRelease(_lockKey, Guid.NewGuid().ToString());
    }

    public void Dispose() => _redis.Close();
}

Why Redis?

• Fast, in‑memory operations → low latency.
• Built‑in SET with NX (set if not exists) and PX (expire) primitives make the lock implementation simple and safe.

Lock Renewal: In long‑running jobs, renew the lock before its TTL expires to avoid accidental release.

2️⃣ Idempotent Processing

Even if the lock is momentarily lost (e.g., network partition), you need a second line of defense. Tag each message with a correlation ID (e.g., the message’s MessageId or a GUID you generate).

public class IdempotentHandler
{
    private readonly IDbContext _db;

    public IdempotentHandler(IDbContext db) => _db = db;

    public async Task<bool> HasBeenProcessedAsync(string correlationId)
    {
        return await _db.ProcessingLog
            .AnyAsync(p => p.CorrelationId == correlationId && p.Status == "Completed");
    }

    public async Task MarkAsProcessedAsync(string correlationId)
    {
        var log = new ProcessingLog { CorrelationId = correlationId, Status = "Completed", ProcessedAt = DateTime.UtcNow };
        _db.ProcessingLog.Add(log);
        await _db.SaveChangesAsync();
    }
}

Workflow:

1. Acquire the distributed lock.
2. Check HasBeenProcessedAsync. If already done, release the lock and exit.
3. Perform the actual work.
4. Call MarkAsProcessedAsync to record success.

If the lock expires mid‑execution, the idempotency check ensures the work is not duplicated.

Implementing the Pattern

Using Redis

var lock = new RedisDistributedLock(connectionString: "redis://localhost:6379", lockKey: "job:process:123");
if (!lock.TryAcquire(TimeSpan.FromSeconds(10)))
{
    // Another instance holds the lock – skip this iteration
    return;
}

// Critical section
await ProcessMessageAsync(message);
await MarkAsProcessedAsync(message.CorrelationId);

Using a Database Advisory Lock (SQL Server)

using (var tran = await _db.Database.BeginTransactionAsync())
{
    var lockKey = $"JobLock:{message.CorrelationId}";
    var lockCmd = _db.Database.ExecuteSqlInterpolated(
        $"SELECT GET_LOCK(@key) FROM (SELECT @key = {lockKey}) AS t");
    if (lockCmd.ExecuteScalar<bool>())
    {
        try
        {
            // Do work
            await _db.ProcessingLog.AddAsync(new ProcessingLog { CorrelationId = message.CorrelationId, Status = "Completed" });
            await _db.SaveChangesAsync();
        }
        finally
        {
            _db.Database.ExecuteSqlInterpolated($"RELEASE_LOCK(@key)").ExecuteNonQuery();
            await tran.CommitAsync();
        }
    }
}

Practical Takeaway

• Lock first, then idempotent check – the lock prevents concurrent execution, while idempotency protects against lock loss or renewal failures.
• Choose the right backing store – Redis is ideal for fast, low‑latency locks; a DB‑backed advisory lock works when you already have a transactional database and want to avoid an extra service.
• Make the correlation ID immutable – use the message’s system‑generated ID or a GUID you create at enqueue time; never rely on mutable fields that could change.
• Renew locks for long jobs – implement a background renewal task or use a “renewable” lock pattern to avoid accidental timeout.

By adopting this two‑layer approach, you eliminate the hidden race condition that can cause duplicate side‑effects in any scaled‑out ASP.NET Core background service. Your jobs become predictable, safe, and easier to debug in production.

---