The Retry Trap: Why I Added Exponential Backoff to My Job Queue

Did you know that retrying when a job fails can be more harmful than you anticipate? The first instinct for every developer is to retry when a job has failed and that is usually recommended, right? I encountered this problem while building a distributed job queue. My first retry implementation did the obvious thing, and the obvious thing is a trap.

When you build a job queue, some jobs will fail, and often you can't just let them stay failed. A job that generates a PDF or sends an email is supposed to complete. So it is fair to retry a couple of times to see if these failures can be resolved. A job may fail for many reasons and it is important to know the 'why'. This helps developers narrow down and solve the issue. Job failures can be categorized into Transient and Permanent failures.

Transient failures as the name suggests are temporary failures that are self-correcting over time. They are usually caused by network instability, server timeouts, rate limit hit on an API or resource shortages. Permanent failures however do not self-correct over time, they fail always until manually corrected. Some of the causes of permanent failures include, malformed payload, a bug in a handler or bad credentials.

These two failure types mean we have to be deliberate about how we retry. In my job queue, after a producer has published a job (which gets stored in a PostgreSQL DB), a worker picks it up executes the job and updates the status of the job. The important part for this article is when the job fails and is to be retried. Below is a function used in the worker to process jobs.

export const processNextJob = async (workerId:number) => {
    const pool = getPool()
    const client = await pool.connect()

    await client.query('BEGIN')
    const result = await client.query(
        `
        SELECT * FROM jobs
        WHERE status = 'pending'
        AND schedule_at <= NOW()
        ORDER BY priority  DESC, created_at  ASC
        LIMIT 1
        FOR UPDATE SKIP LOCKED
        `
    )

    const job = result.rows[0]
    try {

        if (job){
            console.log(`Worker ${workerId} processing job ${job.id}`)
            // update job
            await client.query(
                `
                UPDATE jobs 
                SET status = $1
                WHERE id=$2
                `,
                [JobStatus.PROCESSING, job.id]
            )

           jobTypeHandler(job)

           // complete job
           await client.query(
            `
            UPDATE jobs 
            SET status = $1
            WHERE id=$2
            `,
            [JobStatus.COMPLETED, job.id]
        )

           await client.query('COMMIT')
        }

        if(!job){
            await client.query('ROLLBACK')
        }

    } catch (error) {
        await client.query('ROLLBACK')
        const newRetries = job.retries + 1

        if(newRetries <= job.max_retries){
            await client.query(
                `
                UPDATE jobs 
                SET status = $1, retries = $2
                WHERE id=$3
                `,
                [JobStatus.PENDING, newRetries, job.id]
            )
        } else {
            await client.query(
                `
                UPDATE jobs 
                SET status = $1
                WHERE id=$2
                `,
                [JobStatus.FAILED, job.id]
            )
        }

    } finally{
        client.release()
    }
}

Let's take a closer look at the retry logic in the catch block:

catch (error) {
        await client.query('ROLLBACK')
        const newRetries = job.retries + 1

        if(newRetries <= job.max_retries){
            await client.query(
                `
                UPDATE jobs 
                SET status = $1, retries = $2
                WHERE id=$3
                `,
                [JobStatus.PENDING, newRetries, job.id]
            )
        } else {
            await client.query(
                `
                UPDATE jobs 
                SET status = $1
                WHERE id=$2
                `,
                [JobStatus.FAILED, job.id]
            )
        }

    } 

The failed job gets a maximum amount of times it can be retried before the status of the job is set to failed permanently. This retries correctly. But it has a hidden problem.

Let's use an example to understand this better. Imagine the job calls an external API and the API is currently down. What happens here is the worker retries instantly, it fails instantly, retries again and again until max retries is reached. So you'll be firing repeated requests per second at a service that's already struggling and if many jobs are failing at once, every worker is doing this simultaneously. You won't just be failing to help, you might even be the reason the API stays down.

Immediate retries are especially bad for transient failures since those are exactly the cases where the service might recover if you'd just given it room, but instead you're crowding it. Here's how we can solve it:

Add an exponential delay to the job after every fail

const delaySeconds = 2 ** newRetries 

Doing this grows the delay per new retry: 2s, 4s, 8s, 16s. Each retry waits twice as long as the last

And schedule it for the next job processing

 UPDATE jobs 
                SET status = $1, retries = $2, schedule_at = NOW() + (interval '1 second' * $3)
                WHERE id=$4
                `,
                [JobStatus.PENDING, newRetries, delaySeconds, job.id]

So this is how the full code will look

await client.query('ROLLBACK')
        const newRetries = job.retries + 1
        const delaySeconds = 2 ** newRetries // exponential backoff
if(newRetries <= job.max_retries){
            await client.query(
                `
                UPDATE jobs 
                SET status = $1, retries = $2, schedule_at = NOW() + (interval '1 second' * $3)
                WHERE id=$4
                `,
                [JobStatus.PENDING, newRetries, delaySeconds, job.id]
            )
        } else {
            await client.query(
                `
                UPDATE jobs 
                SET status = $1
                WHERE id=$2
                `,
                [JobStatus.FAILED, job.id]
            )
        }

With every job fail, we grow the delay exponentially with each retry and set its scheduled time interval with it.

Here is a database query result of jobs for sending emails showing the backoff taking effect:

id	status	retries	created_at	schedule_at
341	failed	3	2026-07-10 16:13:15.488721	2026-07-10 16:13:30.048009
339	failed	3	2026-07-10 16:13:15.481891	2026-07-10 16:13:29.934821
337	failed	3	2026-07-10 16:13:15.477258	2026-07-10 16:13:29.610338
335	failed	3	2026-07-10 16:13:15.47414	2026-07-10 16:13:29.93442
333	failed	3	2026-07-10 16:13:15.460275	2026-07-10 16:13:29.610181

To verify if this db results show the backoff effect, recall that we exponentially increase delay 2^newRetries so 2s, 4s, 8s is what we get for the delays per retry until max retry which is 3.

Take job 341. Subtract created_at from the final schedule_at and you get roughly 14 seconds, which is exactly 2 + 4 + 8, the three backoff delays adding up. Without backoff, all three retries would have fired within milliseconds.

One improvement I'd add is jitter, a small random offset so that jobs failing together don't all retry at the same instant. Without jitter, if fifty jobs fail at the same moment, they all wait exactly 2 seconds and then all retry at the same instant, recreating the stampede. A small random offset scatters them.

const delaySeconds = (2 ** newRetries) + Math.random()

Retries aren't free, and the gap between a retry that helps and one that hurts is just a little bit of waiting.