A Postgres table with SELECT ... FOR UPDATE SKIP LOCKED is a correct, durable job queue, and for most teams it stays correct well past the throughput they actually have. You move off it for two reasons only: sustained enqueue rates high enough that queue churn hurts your primary database, or a need for fan-out to multiple independent consumers. "It feels wrong to use the database" is not one of the reasons.
I've run all three — a Postgres queue for a scheduling app, Redis Streams for an ingestion pipeline, SQS for anything crossing an account boundary. What follows is the failure mode each one actually produced, because that's the part the "5 ways to build a queue" posts skip.
Why does a naive Postgres queue lose or double-process jobs?
The broken version everyone writes first looks like this:
-- DON'T: two workers can read the same row
SELECT id, payload FROM jobs WHERE status = 'queued' ORDER BY run_at LIMIT 1;
UPDATE jobs SET status = 'running' WHERE id = $1;
Between the SELECT and the UPDATE, another worker runs the same SELECT. Both get the job. If you "fix" it by wrapping the select in FOR UPDATE without SKIP LOCKED, you get the opposite symptom: every worker blocks on the same hot row and your throughput collapses to one job at a time, which shows up in pg_stat_activity as a pile of workers in Lock wait events.
The correct claim is a single statement:
UPDATE jobs
SET status = 'running',
locked_at = now(),
attempts = attempts + 1
WHERE id = (
SELECT id FROM jobs
WHERE status = 'queued' AND run_at <= now()
ORDER BY run_at
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING id, payload, attempts;
SKIP LOCKED (Postgres 9.5 and later) tells the subselect to step over rows another transaction already holds instead of waiting for them. Each worker gets a different job, atomically, with no advisory-lock bookkeeping.
That handles concurrency. It does not handle a worker that gets OOM-killed halfway through a job — the row sits in running forever. You need a reaper:
UPDATE jobs
SET status = 'queued', locked_at = NULL
WHERE status = 'running'
AND locked_at < now() - interval '10 minutes'
AND attempts < 5;
That interval is a contract: your jobs must either finish inside it or heartbeat locked_at while they run. Pick it deliberately rather than copying ten minutes from a blog post.
The other thing that bites is dead tuples. A queue table is the highest-churn table you own — every job is an insert, one or more updates, and eventually a delete. Autovacuum's default scale factor is proportional to table size, which is wrong for a table that turns over completely every hour. Set it per-table:
ALTER TABLE jobs SET (
autovacuum_vacuum_scale_factor = 0.0,
autovacuum_vacuum_threshold = 1000,
autovacuum_analyze_scale_factor = 0.0,
autovacuum_analyze_threshold = 1000
);
Without this, index bloat makes your "instant" dequeue query slowly get slower over weeks, and the symptom — a queue that was fine in month one and mysteriously laggy in month three — looks nothing like its cause.
A Postgres queue's real operational cost is not the dequeue query; it's autovacuum tuning on a high-churn table.
When is Redis actually the better queue?
Redis is the right answer when you need low-latency dequeue at high rates and you can tolerate the durability model. The trap is which Redis primitive you pick.
BRPOP is at-most-once. The moment Redis hands the item to your worker, it's gone from the server. Worker crashes, job is gone, no trace. This is fine for cache warming and catastrophic for payment webhooks, and the API gives you no hint about which situation you're in.
The reliable patterns are BLMOVE (Redis 6.2+, replacing the deprecated BRPOPLPUSH) into a per-worker processing list, or Redis Streams with consumer groups:
import redis
r = redis.Redis(decode_responses=True)
# Create the group once; MKSTREAM makes the stream if it doesn't exist yet.
try:
r.xgroup_create("jobs", "workers", id="0", mkstream=True)
except redis.ResponseError as e:
if "BUSYGROUP" not in str(e):
raise
while True:
resp = r.xreadgroup("workers", "worker-1", {"jobs": ">"}, count=1, block=5000)
if not resp:
continue
_, entries = resp[0]
for entry_id, fields in entries:
try:
handle(fields)
r.xack("jobs", "workers", entry_id) # only now is it done
except Exception:
pass # stays pending; XAUTOCLAIM will hand it to another worker
The unacknowledged entry stays in the group's pending list, and XAUTOCLAIM reassigns it after an idle threshold. That's the same reaper concept as the Postgres locked_at sweep, just built in.
The durability caveat is real and worth stating plainly: with the default appendfsync everysec, a hard crash can lose about a second of writes. Managed Redis with AOF enabled and replication narrows the window but doesn't close it. If you want Redis Streams with the operational surface managed for you, Upstash is the one that fits a serverless worker fleet, since it bills per request and doesn't hold a connection per worker.
Choose Redis for dequeue latency, and only with Streams or BLMOVE — BRPOP quietly makes your queue at-most-once.
What does SQS give you that neither gives?
SQS's value is that it is not your infrastructure. No vacuum tuning, no failover, no capacity planning, and a dead-letter queue you configure instead of build. It's the default when a queue crosses a service or account boundary.
The two things that surprise people:
Visibility timeout is not a lock, it's a timer. If your job takes longer than the timeout, SQS re-delivers it to another consumer while the first is still working. You must either set the timeout above your worst-case duration or heartbeat with ChangeMessageVisibility:
import boto3
sqs = boto3.client("sqs")
resp = sqs.receive_message(QueueUrl=QUEUE_URL, WaitTimeSeconds=20, MaxNumberOfMessages=1)
for msg in resp.get("Messages", []):
handle_with_heartbeat(
msg["Body"],
extend=lambda: sqs.change_message_visibility(
QueueUrl=QUEUE_URL,
ReceiptHandle=msg["ReceiptHandle"],
VisibilityTimeout=120,
),
)
sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])
Note WaitTimeSeconds=20 — long polling. Leaving it at zero is the single most common way people turn an idle SQS queue into a surprising bill, because empty receives are still billable requests.
And standard queues are at-least-once with best-effort ordering. Duplicates are a documented property, not a bug. FIFO queues give ordering and deduplication within a message group, at lower throughput per group. Either way, your handler has to be idempotent — which is true of all three options here, so treat it as a fixed cost rather than a differentiator.
SQS trades local latency and cheap introspection for someone else being on call for the queue itself.
Decision table
| Postgres + SKIP LOCKED | Redis Streams | SQS | |
|---|---|---|---|
| Durability | Same as your DB (WAL, PITR) | Config-dependent; sub-second loss window on crash | Managed, replicated |
| Delivery | At-least-once | At-least-once with XACK | At-least-once (standard) |
| Transactional with app writes | Yes — same commit | No | No |
| Ops burden | Autovacuum + reaper | Memory + persistence config | Effectively none |
| Debugging | SELECT * FROM jobs |
XPENDING, XINFO
|
Console + CloudWatch, no ad-hoc query |
| Cost shape | Free-ish; costs you DB headroom | Instance or per-request | Per request |
| Breaks down when | Enqueue churn competes with app traffic | You need durable-by-default | You need transactional enqueue or sub-10ms latency |
The row that decides it most often is "transactional with app writes." If enqueueing a job must be atomic with the row that caused it, Postgres wins outright — everything else needs an outbox table, at which point you've built a Postgres queue anyway and added a second system.
FAQ
Can Postgres handle a job queue in production?
Yes. With FOR UPDATE SKIP LOCKED, a partial index on pending rows, and per-table autovacuum tuning, a single Postgres instance handles job rates well beyond what most applications produce. The limit you hit first is usually contention with your application's own queries on the same instance, not the queue mechanics.
Is Redis a reliable message queue?
Only with Streams and consumer groups, or BLMOVE into a processing list. BRPOP and LPOP delete the item at delivery, so a worker crash loses the job. Even with Streams, Redis persistence is configurable and can lose a small window of writes on a hard crash.
What's the difference between SQS visibility timeout and a lock?
A lock is held until released; a visibility timeout expires on a schedule regardless of whether your worker is still running. If processing exceeds the timeout, SQS delivers the same message to another consumer, so long-running jobs must extend the timeout with ChangeMessageVisibility.
Bottom line
Start with Postgres if your jobs are enqueued by the same application that owns the database — the transactional guarantee is worth more than anything the alternatives offer, and one fewer system to operate is a real feature. Move to Redis Streams when dequeue latency or enqueue volume starts showing up in your database's wait events, and accept the persistence trade-off explicitly rather than by default. Reach for SQS when the queue spans services, teams, or AWS accounts, or when nobody on the team wants to own queue infrastructure. All three demand idempotent handlers, so build that first and the migration between them stays cheap.
Top comments (0)