How I solved the "crashed worker" problem in Go (Lease Fencing)

Om singh Chouhan — Tue, 03 Nov 2026 18:02:50 +0000

Handling Stale Workers in Go: Why I Chose Lease Fencing Over Distributed Locks

When building background workers that process tasks—sending webhooks, processing images, or syncing data—the hardest problem isn't the happy path.

It's this:

What happens when a worker crashes in the middle of a task?

I've been building Relay, an open-source webhook delivery service written in Go and backed by MongoDB.

A webhook delivery is essentially a background job: we send an HTTP request to a destination, wait for a response, and persist the result.

The naive approach to worker crashes is simple: retry the job.

But what if a worker dies while the HTTP request is still in flight?

You're left with some difficult distributed systems questions:

Did the request reach the downstream server?
Did the downstream server actually process it?
If we retry, could we trigger the same side effect twice—for example, charging a credit card twice?
If we don't retry, could the event be lost permanently?

There is no reliable way for the delivery engine to know exactly what happened after a connection is interrupted.

For Relay, I chose at-least-once delivery. This means a delivery can happen more than once, so downstream consumers should use idempotency keys to protect against duplicate side effects.

But that decision introduced another problem I needed to solve: stale workers.

The Stale Worker Problem

Imagine the following sequence:

Worker A claims a delivery job and receives a lease valid for 30 seconds.
Worker A sends the HTTP request to the downstream server.
The downstream server processes the request successfully.
Before Worker A can persist the successful result, its process experiences a long pause—perhaps due to a runtime pause, VM suspension, or network issue.
The 30-second lease expires.
Worker B detects the expired lease, claims the job, and sends the HTTP request again.
Worker A eventually resumes and attempts to write its result to the database.

Now imagine Worker A writes an outdated result after Worker B has already reclaimed the job.

Without additional protection, Worker A could overwrite newer delivery state.

This is the stale worker problem: a worker can continue executing even after it has lost ownership of the job.

A lease expiration doesn't physically stop a worker. It only means the system is allowed to give the job to someone else.

The Solution: Lease Fencing

To solve this, I used a technique called lease fencing.

Instead of treating a lease as a simple boolean lock, the database assigns an incrementing integer called a fencing token every time a job is claimed.

The token represents the ownership generation of that job.

Here's how it works:

Worker A claims the job. The database assigns lease_token = 1.
Worker A stores token 1 in memory.
Its lease expires.
Worker B claims the same job. The database atomically increments the token to 2.
Worker B now owns the latest lease generation.
Worker A wakes up and tries to persist its result using token 1.
The database rejects the update because the stored token is now 2.

Worker A has been fenced out.

Its result cannot overwrite the state written by the newer owner.

What this looks like in Go

Using the MongoDB Go driver, a worker can include its fencing token in every state-changing update:

filter := bson.M{
    "_id":         deliveryID,
    "lease_token": myToken,
}

update := bson.M{
    "$set": bson.M{
        "status": "delivered",
    },
}

result, err := collection.UpdateOne(ctx, filter, update)
if err != nil {
    return fmt.Errorf("update delivery: %w", err)
}

if result.MatchedCount == 0 {
    // We no longer own this delivery.
    // Another worker has reclaimed it.
    // Drop the stale result.
    return nil
}

The important part isn't the update itself.

It's the condition:

"lease_token": myToken

The database only accepts the update if the worker's token still matches the current token.

This comparison and update happen atomically as part of the database operation.

The rule is simple: a worker can only modify delivery state while its fencing token is still current.

The same ownership check must apply to every state-changing operation—not just successful delivery updates.

Why Not Just Use Distributed Locks Like Redis?

Distributed locks are useful, but a lock with a time-to-live (TTL) doesn't automatically solve the stale worker problem.

Consider this:

Worker A acquires a Redis lock.
Worker A pauses longer than the lock TTL.
The lock expires.
Worker B acquires the lock.
Worker A resumes and continues executing.

Both workers may now believe they can perform work.

A lock tells a worker whether it acquired ownership at some point. It doesn't necessarily prevent an old owner from writing stale state after ownership changes.

Lease fencing addresses this at the persistence layer.

The database checks the fencing token when accepting a state change, rather than trusting that the worker still owns the lease.

That is the important distinction: ownership must be enforced where state is written.

The Trade-off

There is an interesting trade-off here.

Suppose Worker A successfully delivers a webhook, but its lease expires before it can save the success result.

Worker B reclaims the delivery and sends the webhook again.

Worker A eventually resumes, but its completion update is rejected because its fencing token is stale.

The system may therefore lose Worker A's success signal and perform another delivery attempt.

That is intentional.

With at-least-once delivery, duplicate attempts are possible. Downstream systems should use stable event IDs or idempotency keys to prevent duplicate side effects.

I would rather retry a webhook than allow an outdated worker to corrupt the delivery engine's state machine.

Delivery duplication is an accepted trade-off. State corruption is not.

Building This Into Relay

I've been building Relay to explore these infrastructure problems in a reusable, open-source project.

The goal is to provide a reliable webhook delivery engine without having to rebuild the same background-processing infrastructure for every application.

Some of the areas I'm working on include:

Bounded concurrent workers
At-least-once webhook delivery
Lease-based job ownership
Fencing tokens to reject stale worker updates
Exponential backoff and automatic retries
Delivery state management and observability

I'm also building out the test coverage and benchmarking harness to understand how the system behaves under concurrency and failure conditions.

If you're interested in contributing, I've curated a list of good first issues, including documentation and testing tasks that should take roughly 1–4 hours.

I'd Love Your Feedback

I'm particularly interested in how other engineers approach worker ownership and failure recovery.

A few questions:

Do you use lease fencing, visibility timeouts, or another ownership mechanism?
How do you prevent stale workers from writing outdated state?
How do you test these failure scenarios under concurrency?
What trade-offs have you made between delivery guarantees and operational complexity?

If you're working with Go, distributed systems, or background job infrastructure, I'd love to hear how you approach this problem.

Repository: github.com/rajputomsingh/Relay

Thanks for reading!

DEV Community: Om singh Chouhan