Two Agents, One Task: The Race Condition Hiding in Your Multi-Agent System

nischita sadanand — Wed, 17 Jun 2026 00:26:38 +0000

Description

The race condition and the vanishing-task bug that show up the moment you run more than one AI agent against a shared queue — and the decades-old fix that's quietly missing from modern agent tooling.

You wire up a few agents to chew through a queue of work. It runs. You ship it. Then one day the same email goes out twice, or a task you know you queued never finishes, and there's no error anywhere to explain it. Welcome to the two hardest bugs in distributed work — and they show up the moment you run more than one
agent against a shared queue.

This is a walkthrough of those two bugs, why the obvious fixes don't work, and the small, boring mechanism that does. The mechanism isn't new — it's how databases and job queues have worked for decades — but it's quietly missing from a lot of the agent tooling being written right now, because that tooling is mostly built by people thinking about prompts, not about what happens when a process dies.

Bug 1: the same task runs twice

Here's the naive way every multi-agent system starts:

pythontask = queue.get_next_pending()   # read a pending task
mark_in_progress(task)            # claim it
do_the_work(task)

Looks fine. It is fine — with one agent. With two, picture the timeline:

Agent A runs get_next_pending() and reads task 42.
Agent B runs get_next_pending() one microsecond later and also reads task 42, because A hasn't marked it yet.
Both mark it in progress. Both do the work.

The work runs twice. If the work is "call the payments API" or "send the email," you now have a duplicate charge or a duplicate message, and nothing logged an error, because from each agent's point of view nothing went wrong. The root cause is that "read a pending task" and "mark it taken" are two
separate operations, and anything can happen between them. This is a classic race condition, and you cannot fix it by being careful in application code. No amount of "check again before marking" closes the gap — it just makes the window smaller. The fix has to make read-and-mark a single, indivisible step.

Bug 2: the task that silently vanishes

Suppose you fix bug 1 and tasks are claimed cleanly. Now an agent claims task 42, marks it in progress, and then — mid-work — the process is OOM-killed, or the LLM call hangs forever, or the container restarts.

Task 42 is now marked "in progress." Forever. No agent is progressing it. It will never be completed, and no one will tell you. The work is simply gone.

The instinct is: "I'll add a sweeper that retries tasks stuck in progress too long." But how long is too long? You genuinely cannot tell, from the outside, the difference between an agent that died and an agent that's just slow on a hard task. Retry too eagerly, and you've reintroduced bug 1 — now the slow-but-alive agent and the retry agent both run task 42. Retry too late, or never, and you're back to vanishing tasks. No fixed timeout's correct.

The mechanism: leases with heartbeats

The fix for both bugs is the same idea, and it's the same one a locker that auto-releases an unclaimed reservation uses, or that a DHCP server uses to hand out IP addresses: don't grant ownership forever. Grant a lease — ownership for a bounded window that the holder must actively renew.

Three pieces:

Atomic claim. Claiming a task — moving it from "pending" to "leased by me" — happens as one indivisible operation. In the implementation, this lives in a single Redis Lua script, because Redis runs a Lua script start-to-finish with nothing else interleaved:

lua
local task_id = redis.call('LPOP', KEYS[1])   -- take from pending
if not task_id then return nil end
redis.call('SET', 'task:' .. task_id .. ':token', ARGV[1]) -- stamp owner
redis.call('ZADD', KEYS[2], ARGV[2], task_id)     -- record expiry
return {task_id, ...}

The LPOP and the ownership stamp can't be split apart, so two agents physically
cannot receive the same task. Bug 1, closed — not made less likely, closed.

Heartbeats distinguish slow from dead. A leased task carries an expiry time. While an agent works, it periodically renews the lease (a heartbeat), pushing the expiry forward. A slow-but-alive agent keeps heartbeating and keeps its task. A dead agent stops heartbeating — and that, finally, is a reliable signal of death, because it's the agent actively asserting "still alive" rather than us guessing from the outside.
The reaper requeues expired leases. A background loop finds leases whose expiry has passed and returns those tasks to the pending pool. The crashed agent's task becomes claimable again, automatically, and gets picked up by a healthy agent. Bug 2, closed.

The ownership token matters for the edge case: when a dead agent's task is reaped and reassigned, the original agent might briefly come back to life and try to complete "its" task. The completion is rejected because its token no longer matches the current owner — so a zombie can't clobber the agent that legitimately took over.

One design decision worth naming

When a task is reaped, does it go to the back of the pending queue or the front? The implementation puts it at the front. The reasoning: a task whose agent just died has already waited; sending it to the back of a long queue delays it twice. Prompt retry of already-attempted work is usually what you want. It's a small choice, but it's the kind of thing that separates a library someone thought about from one that just wraps a queue.

Why this is worth a library

None of these mechanisms are novel — leasing is decades old. The point is that they're easy to get subtly wrong and quietly missing from a lot of current agent frameworks, which handle the happy path and stop. The hard part isn't the idea; it's making the claim genuinely atomic, getting the token check right so zombies can't double-complete, and proving it under contention. (The
accompanying implementation ships a test that fires 20 workers at 200 tasks and asserts every task is claimed exactly once — that test is the real deliverable; the rest is plumbing.)

If you run multiple agents against shared work, you will hit both of these bugs. Better to borrow a forty-year-old solution than to rediscover it in production at 2am.

Implementation is open source: agent-lease on GitHub.
Corrections and counterpoints welcome — especially if you've solved this differently. How are you handling crash recovery in your agent systems?

DEV Community: nischita sadanand

Two Agents, One Task: The Race Condition Hiding in Your Multi-Agent System