DEV Community: Theophilus Frimpong

You Might Not Need Kafka: Building a Job Queue with PostgreSQL

Theophilus Frimpong — Sat, 25 Jul 2026 00:00:48 +0000

It's easy to reach for the popular tool before asking what your system actually needs. For job queueing, the usual advice is to use RabbitMQ or Kafka. The underlying burden of using these tools will be additional processes to deploy, monitor and reason about. But what if using your existing database is possible? I built a job queueing system with a PostgreSQL database that cleared the bar without adding infrastructure. The next question will be, does this solution meet the criteria of a job queue?

A job needs a few things to execute properly within a system. It needs to be persistent, surviving unforeseen crashes. Jobs must not be processed more than once by workers; one job should be processed once by one worker. Also, a job's state has to be tracked through every step. A job state must show when it's pending, completed or failed. That's the bar any solution must clear.

With the Postgres approach, persistence comes free. Jobs live in a table so when a worker dies mid-job, the job still exists in a row in the db. Whereas with in-memory queues, a crash loses everything still in memory. A broker like RabbitMQ has to be configured for persistence and if configured wrongly, jobs get lost. A database however is fundamentally built for durability.

Now, let's say three workers poll the queue at the same instant and run the same query. They'll see the same pending job at the top and nothing stops them all from grabbing it. If that job is a payment, the customer gets charged three times for one service. All the workers successfully process the job with no indication of an error or alerts. This is the requirement that seems to demand a real message broker, and it's exactly where people assume a database can't compete. It can. Postgres has a specific tool for exactly this.

The SQL clause FOR UPDATE is used to lock rows. This can be called on a job when a worker picks it up to process. By default other workers will get blocked during this process, they'll wait for the lock to release before processing available rows. This kills concurrency. The SKIP LOCKED modifier changes that. It allows workers to step over locked rows and take the next available one. An important note to make is to start a database transaction before you run your query. FOR UPDATE only holds the lock for the life of the transaction. Outside a transaction, the lock releases immediately and the whole guarantee evaporates. Here's the query inside its transaction:

BEGIN;

SELECT * FROM jobs
WHERE status = 'pending'
  AND schedule_at <= NOW()
ORDER BY priority DESC, created_at ASC
LIMIT 1
FOR UPDATE SKIP LOCKED;

UPDATE jobs SET status = 'processing' WHERE id = $1;

COMMIT;

Now when worker 1 locks job 5, workers 2 and 3 hit the same query, see job 5 is locked, skip it and take jobs 6 and 7. Three workers, three different jobs, no collision.

After running 100 jobs across 3 workers, here's what each worker picked up:

Worker 1 processing job 543
Worker 2 processing job 545
Worker 3 processing job 546
Worker 1 processing job 550
Worker 2 processing job 556
Worker 3 processing job 558
....

Total jobs processed:  100
Unique job IDs:        100
Duplicates:            0
Worker 1: 34   Worker 2: 33   Worker 3: 33

Every job ID appears exactly once across all three workers — no job was ever picked up twice

SELECT status, COUNT(*) FROM jobs GROUP BY status;

  status   | count
-----------+-------
 completed |   100

One row, all completed — nothing stuck in processing, which is the signature of a worker that died holding a job.

Evidently, PostgreSQL can be used to simplify the daunting task real message queuing systems can bring. But every solution still has trade-offs. First of the trade-offs is the throughput ceiling from polling. Workers poll the database at a specified interval for jobs to run even when the queue is completely empty. All workers keep asking forever. This results in wasted work where workers run pointless queries at specified intervals. Also it can cause a delay in job processing. If a job gets added one millisecond after a worker checks, that job sits untouched until the next poll. Although a weakness, Postgres has a feature called LISTEN/NOTIFY that can push notifications, however it's more complex to wire up and has its own limits.

Also there's no native pub/sub fanout. The use of SKIP LOCKED ensures that one job goes to one worker but some situations may need one event to reach several different systems. For example: an order is placed and email, inventory and analytics each need their own copy of that event.

Another trade-off worth mentioning is one I encountered while building my job queue. Long-held connections. Postgres pool holds 10-20 connections and workers borrow connections to communicate with the database, then release afterwards. If a job calls a slow external API and takes 30 seconds, that database connection sits reserved and idle for 30 seconds doing nothing, helping nobody and unavailable to anyone else. An easy fix is to commit the transaction immediately after marking the job status to processing, this releases the lock and frees the connection back to the pool. Let the worker run the job with no transaction open, then open a second short transaction to update the job to completed. A job with a processing status won't be touched by other workers so the current worker won't be competing with any other to process it.

None of these are dealbreakers for most systems.

For most applications operating at normal scale the infrastructure you already run is enough, and reaching for Kafka before you need it is solving a problem you don't have yet.

Thanks for reading to this point. If you enjoyed this article you can find a related job queue post I made on exponential backoff here

The Retry Trap: Why I Added Exponential Backoff to My Job Queue

Theophilus Frimpong — Fri, 10 Jul 2026 17:14:48 +0000

Did you know that retrying when a job fails can be more harmful than you anticipate? The first instinct for every developer is to retry when a job has failed and that is usually recommended, right? I encountered this problem while building a distributed job queue. My first retry implementation did the obvious thing, and the obvious thing is a trap.

When you build a job queue, some jobs will fail, and often you can't just let them stay failed. A job that generates a PDF or sends an email is supposed to complete. So it is fair to retry a couple of times to see if these failures can be resolved. A job may fail for many reasons and it is important to know the 'why'. This helps developers narrow down and solve the issue. Job failures can be categorized into Transient and Permanent failures.

Transient failures as the name suggests are temporary failures that are self-correcting over time. They are usually caused by network instability, server timeouts, rate limit hit on an API or resource shortages. Permanent failures however do not self-correct over time, they fail always until manually corrected. Some of the causes of permanent failures include, malformed payload, a bug in a handler or bad credentials.

These two failure types mean we have to be deliberate about how we retry. In my job queue, after a producer has published a job (which gets stored in a PostgreSQL DB), a worker picks it up executes the job and updates the status of the job. The important part for this article is when the job fails and is to be retried. Below is a function used in the worker to process jobs.

export const processNextJob = async (workerId:number) => {
    const pool = getPool()
    const client = await pool.connect()

    await client.query('BEGIN')
    const result = await client.query(
        `
        SELECT * FROM jobs
        WHERE status = 'pending'
        AND schedule_at <= NOW()
        ORDER BY priority  DESC, created_at  ASC
        LIMIT 1
        FOR UPDATE SKIP LOCKED
        `
    )

    const job = result.rows[0]
    try {

        if (job){
            console.log(`Worker ${workerId} processing job ${job.id}`)
            // update job
            await client.query(
                `
                UPDATE jobs 
                SET status = $1
                WHERE id=$2
                `,
                [JobStatus.PROCESSING, job.id]
            )

           jobTypeHandler(job)

           // complete job
           await client.query(
            `
            UPDATE jobs 
            SET status = $1
            WHERE id=$2
            `,
            [JobStatus.COMPLETED, job.id]
        )

           await client.query('COMMIT')
        }

        if(!job){
            await client.query('ROLLBACK')
        }

    } catch (error) {
        await client.query('ROLLBACK')
        const newRetries = job.retries + 1

        if(newRetries <= job.max_retries){
            await client.query(
                `
                UPDATE jobs 
                SET status = $1, retries = $2
                WHERE id=$3
                `,
                [JobStatus.PENDING, newRetries, job.id]
            )
        } else {
            await client.query(
                `
                UPDATE jobs 
                SET status = $1
                WHERE id=$2
                `,
                [JobStatus.FAILED, job.id]
            )
        }

    } finally{
        client.release()
    }
}

Let's take a closer look at the retry logic in the catch block:

catch (error) {
        await client.query('ROLLBACK')
        const newRetries = job.retries + 1

        if(newRetries <= job.max_retries){
            await client.query(
                `
                UPDATE jobs 
                SET status = $1, retries = $2
                WHERE id=$3
                `,
                [JobStatus.PENDING, newRetries, job.id]
            )
        } else {
            await client.query(
                `
                UPDATE jobs 
                SET status = $1
                WHERE id=$2
                `,
                [JobStatus.FAILED, job.id]
            )
        }

    }

The failed job gets a maximum amount of times it can be retried before the status of the job is set to failed permanently. This retries correctly. But it has a hidden problem.

Let's use an example to understand this better. Imagine the job calls an external API and the API is currently down. What happens here is the worker retries instantly, it fails instantly, retries again and again until max retries is reached. So you'll be firing repeated requests per second at a service that's already struggling and if many jobs are failing at once, every worker is doing this simultaneously. You won't just be failing to help, you might even be the reason the API stays down.

Immediate retries are especially bad for transient failures since those are exactly the cases where the service might recover if you'd just given it room, but instead you're crowding it. Here's how we can solve it:

Add an exponential delay to the job after every fail

const delaySeconds = 2 ** newRetries

Doing this grows the delay per new retry: 2s, 4s, 8s, 16s. Each retry waits twice as long as the last

And schedule it for the next job processing

 UPDATE jobs 
                SET status = $1, retries = $2, schedule_at = NOW() + (interval '1 second' * $3)
                WHERE id=$4
                `,
                [JobStatus.PENDING, newRetries, delaySeconds, job.id]

So this is how the full code will look

await client.query('ROLLBACK')
        const newRetries = job.retries + 1
        const delaySeconds = 2 ** newRetries // exponential backoff
if(newRetries <= job.max_retries){
            await client.query(
                `
                UPDATE jobs 
                SET status = $1, retries = $2, schedule_at = NOW() + (interval '1 second' * $3)
                WHERE id=$4
                `,
                [JobStatus.PENDING, newRetries, delaySeconds, job.id]
            )
        } else {
            await client.query(
                `
                UPDATE jobs 
                SET status = $1
                WHERE id=$2
                `,
                [JobStatus.FAILED, job.id]
            )
        }

With every job fail, we grow the delay exponentially with each retry and set its scheduled time interval with it.

Here is a database query result of jobs for sending emails showing the backoff taking effect:

id	status	retries	created_at	schedule_at
341	failed	3	2026-07-10 16:13:15.488721	2026-07-10 16:13:30.048009
339	failed	3	2026-07-10 16:13:15.481891	2026-07-10 16:13:29.934821
337	failed	3	2026-07-10 16:13:15.477258	2026-07-10 16:13:29.610338
335	failed	3	2026-07-10 16:13:15.47414	2026-07-10 16:13:29.93442
333	failed	3	2026-07-10 16:13:15.460275	2026-07-10 16:13:29.610181

To verify if this db results show the backoff effect, recall that we exponentially increase delay 2^newRetries so 2s, 4s, 8s is what we get for the delays per retry until max retry which is 3.

Take job 341. Subtract created_at from the final schedule_at and you get roughly 14 seconds, which is exactly 2 + 4 + 8, the three backoff delays adding up. Without backoff, all three retries would have fired within milliseconds.

One improvement I'd add is jitter, a small random offset so that jobs failing together don't all retry at the same instant. Without jitter, if fifty jobs fail at the same moment, they all wait exactly 2 seconds and then all retry at the same instant, recreating the stampede. A small random offset scatters them.

const delaySeconds = (2 ** newRetries) + Math.random()

Retries aren't free, and the gap between a retry that helps and one that hurts is just a little bit of waiting.