How to Build a PostgreSQL Backed Job Queue in Go

timothy ogbemudia — Sun, 31 May 2026 13:37:34 +0000

When you build a web application, not every task should happen inside the user's request.

Some work is slow. Some work can fail. Some work should happen later. Sending emails, resizing images, processing webhooks, generating reports, and retrying third party APIs are all good examples.

These tasks are usually handled by a background job system.

In this article, we will use a Go project I built called Swig as a practical example of how a PostgreSQL backed job queue can work.

The goal is not only to understand Swig. The goal is also to learn Go concepts that show up in real backend systems:

Interfaces
Goroutines
Contexts
Transactions
JSON serialization
Driver abstraction
Graceful shutdown
PostgreSQL row locks
PostgreSQL LISTEN/NOTIFY
PostgreSQL advisory locks

By the end, you should understand how a background job queue can be built with Go and PostgreSQL, and why PostgreSQL is more than just a place to store rows.

What Is a Job Queue?

A job queue is a system that stores work to be done later.

Your application adds a job to the queue. A worker takes a job from the queue and runs it.

For example, when a user signs up, your application might create the user immediately, then add a job like this:

{
  "kind": "send_welcome_email",
  "payload": {
    "to": "user@example.com",
    "subject": "Welcome!"
  }
}

A background worker later picks up that job and sends the email.

This keeps the user request fast. The signup route does not need to wait for the email provider before returning a response.

A job queue usually needs to answer a few important questions:

Where are jobs stored?
How do workers find jobs?
How do we stop two workers from processing the same job?
How do we retry failed jobs?
How do we shut workers down safely?
How do we keep job creation consistent with application data?

Swig answers those questions with Go and PostgreSQL.

Why Use PostgreSQL for a Queue?

Many job queues use Redis, RabbitMQ, SQS, Kafka, or another dedicated queueing system. Those are all useful tools.

But many applications already depend on PostgreSQL. If your app already has Postgres, you may not want to operate another service just to run background jobs.

PostgreSQL gives us several features that are surprisingly useful for queues:

Tables for durable job storage
Transactions for atomic writes
Row locks for safe concurrent processing
SKIP LOCKED for letting workers claim different jobs
LISTEN/NOTIFY for waking workers when new jobs arrive
Advisory locks for leader election
JSONB for flexible job payloads

Swig is a Go job queue that uses these PostgreSQL features directly.

The tradeoff is important. A PostgreSQL backed queue is not trying to replace Kafka for event streaming or RabbitMQ for complex routing. It is trying to make common application background jobs simple, reliable, and easy to operate.

Swig's Basic Architecture

At a high level, Swig has five parts:

A swig_jobs table in PostgreSQL
Go workers that process jobs
A worker registry that maps job names to worker types
A driver layer that supports both pgx and database/sql
A leader loop for shared maintenance work

The basic flow looks like this:

Your app calls AddJob.
Swig serializes the job payload to JSON.
Swig inserts a row into swig_jobs.
PostgreSQL sends a notification that a job was created.
A Go worker wakes up and tries to claim one pending job.
PostgreSQL row locks ensure only one worker claims that row.
The worker runs the job.
Swig marks the job as completed or failed.

That sounds simple, but the details matter.

The hard parts are concurrency, failure, connection lifecycle, and shutdown. That is where Go and PostgreSQL work together.

Representing Jobs in PostgreSQL

A simplified version of Swig's job table looks like this:

CREATE TABLE swig_jobs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  kind TEXT NOT NULL,
  queue TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  priority INTEGER NOT NULL DEFAULT 0,
  attempts INTEGER NOT NULL DEFAULT 0,
  max_attempts INTEGER NOT NULL DEFAULT 3,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  scheduled_for TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  instance_id UUID,
  worker_id UUID,
  locked_at TIMESTAMPTZ,
  last_error TEXT,
  last_error_at TIMESTAMPTZ
);

Each row is one job.

The important columns are:

kind: the type of job, such as send_email
queue: the queue the job belongs to
payload: the JSON data needed to run the job
status: whether the job is pending, processing, completed, or failed
priority: the job's priority inside the queue
attempts: how many times the job has been tried
scheduled_for: when the job is allowed to run
instance_id: which Swig instance claimed the job
worker_id: which worker claim is processing the job
locked_at: when the job was claimed
last_error: the last error returned by the worker

The table is the source of truth.

That is a key design point. PostgreSQL notifications can wake workers, but the notifications are not the durable queue. The rows in swig_jobs are the durable queue.

Defining a Worker in Go

In Swig, a worker is a Go type that knows how to process one kind of job.

Here is a simple email worker:

type EmailWorker struct {
    To      string `json:"to"`
    Subject string `json:"subject"`
    Body    string `json:"body"`
}

func (w *EmailWorker) JobName() string {
    return "send_email"
}

func (w *EmailWorker) Process(ctx context.Context) error {
    fmt.Printf("Sending email to %s with subject %s\n", w.To, w.Subject)
    return nil
}

There are two important methods:

JobName tells Swig what kind of job this worker handles.
Process contains the actual work.

The struct fields are also the job arguments.

When you enqueue an EmailWorker, Swig serializes the struct into JSON and stores it in PostgreSQL. Later, a worker claims the row, unmarshals the JSON back into an EmailWorker, and calls Process.

This is a simple API, but it teaches an important Go lesson: your data model and your behavior can live together in a small type.

Go Concept: Interfaces

Go interfaces describe behavior.

Swig does not need to know the exact concrete type of every worker. It only needs to know that a worker can provide a job name and process a job.

Conceptually, that behavior looks like this:

type Worker interface {
    JobName() string
    Process(context.Context) error
}

If a type has those methods, it can behave like a worker.

In Go, a type does not need to explicitly declare that it implements an interface. If the methods match, the type satisfies the interface.

This is one of the reasons interfaces are so useful in Go. They let you design around behavior instead of inheritance.

Registering Workers Without Sharing State

Swig has a worker registry. The registry maps a job name to a worker type.

For example:

registry := workers.NewWorkerRegistry()
registry.RegisterWorker(&EmailWorker{})

Later, when a job row says kind = 'send_email', Swig can look up the registered worker and run it.

There is a subtle concurrency issue here.

If the registry stored the exact &EmailWorker{} pointer and reused it for every job, multiple goroutines could unmarshal payloads into the same Go value at the same time.

That would be shared mutable state, and shared mutable state is where many concurrency bugs come from.

Swig now avoids that by using a factory approach internally. Registration captures the worker type, and each claimed job gets a fresh worker instance before JSON is unmarshaled.

The API stays simple:

registry.RegisterWorker(&EmailWorker{})

But internally, Swig can create a new EmailWorker for each job.

This is a useful Go pattern. You can keep the public API simple while making the internal lifecycle safer.

Adding a Job

Here is what adding a job looks like from the user side:

err := swigClient.AddJob(ctx, &EmailWorker{
    To:      "user@example.com",
    Subject: "Welcome!",
    Body:    "Thanks for signing up.",
})
if err != nil {
    return err
}

Inside Swig, the process is roughly:

argsJSON, err := json.Marshal(workerWithArgs)
if err != nil {
    return err
}

_, err = db.ExecContext(ctx, `
    INSERT INTO swig_jobs (kind, queue, payload, priority, scheduled_for, status)
    VALUES ($1, $2, $3, $4, $5, 'pending')
`, jobName, queue, argsJSON, priority, runAt)

This uses Go's standard encoding/json package.

The worker struct becomes JSON. PostgreSQL stores that JSON in a JSONB column. Later, a worker unmarshals it back into a fresh worker instance.

This is a common pattern in job systems. The job payload must cross a process boundary, so it needs a serializable format.

Transactional Job Enqueueing

One of the best reasons to use PostgreSQL for jobs is transactional enqueueing.

Imagine a user signs up. You want to:

Insert the user.
Queue a welcome email.

If those happen separately, you can get inconsistent states:

The user is created, but the email job is not queued.
The email job is queued, but the user is not created.

With a transaction, both succeed or both fail.

tx, err := pool.Begin(ctx)
if err != nil {
    return err
}
defer tx.Rollback(ctx)

_, err = tx.Exec(ctx, `
    INSERT INTO users (email)
    VALUES ($1)
`, email)
if err != nil {
    return err
}

err = swigClient.AddJobWithTx(ctx, tx, &EmailWorker{
    To:      email,
    Subject: "Welcome!",
    Body:    "Thanks for joining.",
})
if err != nil {
    return err
}

return tx.Commit(ctx)

This is powerful because the application data and the background job commit together.

If the transaction rolls back, the user is not created and the job is not queued.

If the transaction commits, both are visible.

This is much harder to guarantee when your database and queue are separate systems.

Go Concept: Context

Most Swig methods accept context.Context.

For example:

func (s *Swig) AddJob(ctx context.Context, worker interface{}) error

Contexts are used for cancellation, timeouts, and deadlines.

That matters in a job queue because many operations can block:

Database queries
Waiting for notifications
Processing long jobs
Graceful shutdown

For example, you can create a timeout:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

If the database operation takes longer than five seconds, the context can cancel it.

In Go services, passing context through your call stack is a standard habit. It gives callers a way to say, "This work is no longer needed."

Swig uses contexts for worker lifecycles too. When Stop is called, Swig cancels worker contexts so blocked database calls and notification waits can return.

The Hard Part: Multiple Workers

A queue gets interesting when many workers run at the same time.

Imagine three workers all asking PostgreSQL for the next pending job. We do not want all three workers to process the same job.

A naive approach might be:

SELECT id FROM swig_jobs
WHERE status = 'pending'
LIMIT 1;

Then the worker updates the job:

UPDATE swig_jobs
SET status = 'processing'
WHERE id = $1;

This has a race condition. Two workers can select the same job before either one updates it.

Swig avoids this with PostgreSQL row locks.

PostgreSQL Concept: `FOR UPDATE SKIP LOCKED`

PostgreSQL can lock rows selected inside a transaction.

The key phrase is:

FOR UPDATE SKIP LOCKED

FOR UPDATE means "lock this row because I plan to update it."

SKIP LOCKED means "if another worker already locked a row, skip it and find another one."

This is perfect for a queue.

Worker A locks job 1. Worker B skips job 1 and locks job 2. Worker C skips jobs 1 and 2 and locks job 3.

No central coordinator is needed.

A helpful Go analogy is sync.Mutex.

A Go mutex protects shared memory inside one process. PostgreSQL row locks protect shared rows across many transactions, processes, and machines.

So SKIP LOCKED is not literally a Go mutex, but it plays a similar coordination role at the database level.

Atomic Job Claiming

Swig uses an atomic update pattern:

UPDATE swig_jobs
SET status = 'processing',
    instance_id = $1,
    worker_id = $2,
    locked_at = NOW(),
    attempts = attempts + 1
WHERE id = (
  SELECT id
  FROM swig_jobs
  WHERE status = 'pending'
    AND scheduled_for <= NOW()
  ORDER BY priority DESC, created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, kind, payload;

This query does several things at once:

Finds a pending job.
Skips jobs already locked by other workers.
Marks the selected job as processing.
Records which worker claimed it.
Returns the job data to Go.

That atomic claim is the heart of the queue.

The important part is that the selection and state change happen together. Workers are not doing a separate SELECT and then hoping the later UPDATE is still safe.

Go Concept: Goroutines

Swig starts worker loops as goroutines.

In Go, a goroutine is a lightweight concurrent function.

go s.startWorker(ctx, queueType)

A queue can start several workers:

for i := 0; i < maxWorkers; i++ {
    go s.startWorker(ctx, queueType)
}

Each worker runs independently. PostgreSQL coordinates which job each worker gets.

This is a nice division of responsibility:

Go handles concurrency with goroutines.
PostgreSQL handles safe job claiming with locks.

This is one of the reasons Go works well for background processing. You can express concurrency directly, while still relying on the database for cross-process coordination.

Go Concept: WaitGroup and Graceful Shutdown

When a service shuts down, it should wait for workers to finish cleanly.

Go's sync.WaitGroup helps with this.

The pattern looks like this:

var wg sync.WaitGroup

wg.Add(1)
go func() {
    defer wg.Done()
    processJobs()
}()

wg.Wait()

Swig uses a WaitGroup to track active worker goroutines.

When Stop is called, Swig cancels worker contexts, closes the shutdown signal, waits for workers, cleans up jobs owned by the instance, and releases leader resources.

Swig also uses sync.Once to make shutdown idempotent.

That means calling Stop more than once should not panic because of a double channel close.

This is a small detail, but it matters. Shutdown paths are often where production systems behave differently from happy path demos.

Waking Workers with `LISTEN/NOTIFY`

If workers constantly query the database for jobs, they waste resources when the queue is empty.

PostgreSQL has a feature called LISTEN/NOTIFY.

A connection can listen on a channel:

LISTEN swig_jobs;

Another session can send a notification:

NOTIFY swig_jobs, '{"id":"job-id"}';

Swig creates a trigger so PostgreSQL sends a notification after a job is inserted:

CREATE TRIGGER swig_jobs_notify_trigger
AFTER INSERT ON swig_jobs
FOR EACH ROW
EXECUTE FUNCTION notify_job_created();

This means workers can sleep when there is no work and wake when a new job arrives.

But there is an important PostgreSQL detail: LISTEN is session-scoped.

That means a worker must wait for notifications on the same database session that executed LISTEN.

This matters when you use a connection pool. If LISTEN happens on one pooled connection, but WaitForNotification happens on another pooled connection, the worker may not receive the notification.

Swig handles this by creating a dedicated listener for each worker.

The listener owns one database session. That same session runs LISTEN, waits for notifications, and gets closed during shutdown.

This is a great example of a backend engineering lesson: abstractions like connection pools are useful, but some database features still depend on the lifecycle of a specific connection.

Leader Election with Advisory Locks

Some queue maintenance tasks should only be done by one instance.

For example:

Retrying failed jobs
Recovering stale processing jobs
Cleaning old job history

Swig uses PostgreSQL advisory locks for leader election:

SELECT pg_try_advisory_lock($1);

If the result is true, that Swig instance becomes the leader.

Advisory locks are application-defined locks managed by PostgreSQL. They are useful because you can coordinate distributed processes without adding a separate lock service.

But advisory locks are also session-scoped.

That means the lock is held by a specific PostgreSQL session. If that session ends, PostgreSQL releases the lock.

Swig uses a dedicated advisory-lock connection for leadership. The leader runs maintenance work while that connection remains open.

When Swig shuts down, it releases the advisory lock from the same session that acquired it.

If an instance does not become leader, it keeps retrying in the background. If the current leader dies and PostgreSQL releases the lock, another Swig instance can acquire leadership later.

That gives Swig a simple failover model without ZooKeeper, etcd, or a separate coordinator.

Go Concept: Driver Abstraction

Swig supports both pgx and database/sql.

pgx is a PostgreSQL focused driver. database/sql is Go's standard database abstraction.

Instead of writing the queue twice, Swig defines a driver interface.

A simplified version looks like this:

type Driver interface {
    Exec(ctx context.Context, sql string, args ...interface{}) error
    Query(ctx context.Context, sql string, args ...interface{}) (Rows, error)
    QueryRow(ctx context.Context, sql string, args ...interface{}) Row
    WithTx(ctx context.Context, fn func(tx Transaction) error) error
    NewListener(ctx context.Context, channel string) (Listener, error)
    TryAdvisoryLock(ctx context.Context, lockID int64) (AdvisoryLock, bool, error)
}

The driver interface hides the concrete database library from the queue logic.

The pgx implementation can use pgxpool. The database/sql implementation can use *sql.DB and lib/pq.

The core queue code only depends on behavior:

Can you execute SQL?
Can you run a transaction?
Can you create a listener?
Can you try to acquire an advisory lock?

This is a common Go design:

Define the behavior your core package needs.
Write small adapters for concrete dependencies.
Keep the core logic independent from specific libraries.

Interfaces are especially useful at boundaries. Database drivers, HTTP clients, storage backends, and queues are all good places to use them.

Handling Failed Jobs

Jobs can fail.

An email provider might be down. A network request might timeout. A third party API might rate limit you.

When a worker returns an error, Swig records the error and either retries the job or marks it as failed.

A simplified version looks like this:

UPDATE swig_jobs
SET status = CASE
    WHEN attempts >= max_attempts THEN 'failed'
    ELSE 'pending'
  END,
  last_error = $2,
  last_error_at = NOW()
WHERE id = $1;

This gives the queue a memory of what happened.

Failed jobs can be retried with backoff. Jobs that exceed their maximum attempts can stay failed for inspection.

In production systems, you often add more around this:

Dead letter queues
Metrics
Dashboards
Alerts
Manual retry tools

But the foundation is simple: store the attempt count and error state in the job row.

Delivery Semantics: Be Careful with "Exactly Once"

It is tempting to say a job queue processes jobs exactly once.

In distributed systems, that is a dangerous claim.

Swig prevents multiple workers from claiming the same pending job at the same time. That is important.

But imagine this:

Worker sends an email.
Worker crashes before marking the job completed.
The job is retried.
The email might be sent again.

This means the safer language is:

Swig provides atomic claiming and at-least-once processing.

Because jobs can be retried, workers should be idempotent.

Idempotent means running the same operation more than once has the same result as running it once.

For example, an email job could use an idempotency key so the email provider does not send duplicates.

This is one of the most important lessons in queue design. The queue can reduce duplicate processing, but workers still need to handle retries safely.

What This Project Teaches About Go

Swig is a useful learning project because it touches many parts of Go backend development.

It teaches interfaces because the database layer supports multiple drivers.

It teaches goroutines because workers run concurrently.

It teaches contexts because database calls, workers, and shutdown need cancellation.

It teaches transactions because enqueueing jobs can be atomic with application writes.

It teaches JSON serialization because job arguments need to cross a process boundary.

It teaches error handling because failed jobs need retry behavior.

It teaches lifecycle management because listeners, advisory locks, and worker goroutines all need to be opened and closed carefully.

Most importantly, it teaches that backend systems are not just about the happy path.

The interesting work is in concurrency, failure, connection ownership, shutdown, and consistency.

Conclusion

A PostgreSQL backed queue is not the right answer for every system.

If you need massive event streaming, Kafka may be a better fit. If you need a dedicated broker with routing features, RabbitMQ may be better. If you need a simple fast queue, Redis may be a good choice.

But for many Go applications, PostgreSQL is already there.

Using it for background jobs can be practical, simple, and powerful.

Swig shows how far you can get with a small Go API and a few PostgreSQL features:

Store jobs in a table.
Claim jobs with FOR UPDATE SKIP LOCKED.
Wake workers with dedicated LISTEN/NOTIFY sessions.
Coordinate leadership with dedicated advisory-lock sessions.
Use transactions to keep app data and jobs consistent.
Use goroutines and contexts to manage worker lifecycles.

That combination makes a great project for learning Go, PostgreSQL, and distributed systems fundamentals.

DEV Community: timothy ogbemudia

How to Build a PostgreSQL Backed Job Queue in Go

What Is a Job Queue?

Why Use PostgreSQL for a Queue?

Swig's Basic Architecture

Representing Jobs in PostgreSQL

Defining a Worker in Go

Go Concept: Interfaces

Registering Workers Without Sharing State

Adding a Job

Transactional Job Enqueueing

Go Concept: Context

The Hard Part: Multiple Workers

PostgreSQL Concept: FOR UPDATE SKIP LOCKED

Atomic Job Claiming

Go Concept: Goroutines

Go Concept: WaitGroup and Graceful Shutdown

Waking Workers with LISTEN/NOTIFY

Leader Election with Advisory Locks

Go Concept: Driver Abstraction

Handling Failed Jobs

Delivery Semantics: Be Careful with "Exactly Once"

What This Project Teaches About Go

Conclusion

PostgreSQL Concept: `FOR UPDATE SKIP LOCKED`

Waking Workers with `LISTEN/NOTIFY`