- Book: Event-Driven Architecture Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Postgres delivers a NOTIFY to whoever is listening at that instant. During a deploy of your order worker, for the few seconds between the old process stopping and the new one booting, that is nobody. The API keeps sending NOTIFY new_order, '{"id": ...}' for every checkout, and those orders never get processed. Nothing in the logs says so.
That is the trap with Postgres LISTEN/NOTIFY. It ships with the database and it's fast, so it looks like a free message queue. It isn't one. Think of it as a doorbell that tells whoever happens to be home that something changed. The fix: jobs live in a table and NOTIFY only wakes the worker. A slow poll plus FOR UPDATE SKIP LOCKED makes a missed ring harmless.
You'll build that worker in TypeScript with node-postgres, run two copies, kill one with kill -9 in the middle of a job, and check the table.
Why NOTIFY is not a queue
Read the NOTIFY docs with queue semantics in mind and four things jump out.
It only reaches sessions that are listening right now. NOTIFY sends the event "to each client application that has previously executed LISTEN" on that channel. A worker that is restarting or stuck reconnecting is not one of them, and the docs describe no replay for latecomers.
It is a broadcast. Every listener gets every notification. Run three workers that do the work named in the payload and each job runs three times. A queue gives each message to one consumer.
The payload is small. "In the default configuration it must be shorter than 8000 bytes," and the docs themselves suggest putting larger data "in a database table and send the key of the record."
The commit path has a global lock. In March 2025 Recall.ai had three outages in four days and traced them to AccessExclusiveLock on object 0 of class 1262 of database 0, a lock that a transaction which issued NOTIFY takes during COMMIT. Because it sits on "database 0", it covers the whole instance, so under many concurrent writers the commits line up one behind another.
Listeners add their own cost. Christophe Pettus measured it in October 2026: through PostgreSQL 18, each notifying commit also signals every listener in the database. In his test, fifty idle listeners on an unrelated channel cut throughput from about 22,000 to about 3,700 transactions per second. PostgreSQL 19 wakes only the listeners on the notified channel plus any that have fallen far behind. It has been at release candidate since October 15, and Pettus tested a beta. Recall.ai has since added an update to its post pointing at the core commit behind that change and saying it "has eliminated the bottleneck." DBOS, which got NOTIFY up to 60K stream writes per second by batching, reads the same change more narrowly: it "does not remove the global lock." By DBOS's account, wake-ups got cheaper and the commit-time lock stayed.
None of this makes NOTIFY useless. Treat it as a latency optimisation. Notice the design DBOS landed on: notifications to wake readers, plus a low-frequency poll in case a notification is lost. That's the doorbell pattern, and it's what you'll build here.
The schema: the table is the queue
Start a throwaway Postgres with the official Docker image:
docker run --name jobs-pg -e POSTGRES_PASSWORD=pg \
-p 5432:5432 -d postgres:18
export PGUSER=postgres PGPASSWORD=pg PGDATABASE=postgres
docker exec -it jobs-pg psql -U postgres
node-postgres reads the same PG* environment variables as psql and defaults to port 5432 on the local machine, so the worker needs no connection string. In the psql session, create the table, an index for the hot path, and a trigger that rings the bell:
CREATE TABLE jobs (
id bigserial PRIMARY KEY,
payload jsonb NOT NULL,
status text NOT NULL DEFAULT 'pending',
attempts int NOT NULL DEFAULT 0,
run_at timestamptz NOT NULL DEFAULT now(),
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX jobs_ready ON jobs (run_at)
WHERE status = 'pending';
CREATE FUNCTION jobs_ring() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('jobs', '');
RETURN NULL;
END $$ LANGUAGE plpgsql;
CREATE TRIGGER jobs_ring AFTER INSERT ON jobs
FOR EACH STATEMENT EXECUTE FUNCTION jobs_ring();
Three choices in there are deliberate.
The payload is an empty string. The worker never reads it, because the row already carries the data. That keeps you miles away from the 8000-byte limit and means a notification can't disagree with the table.
The trigger is FOR EACH STATEMENT, so an INSERT ... SELECT of 500 jobs rings once, not 500 times. The docs also say identical notifications on the same channel within one transaction get folded into one, so even several insert statements in one transaction ring once.
The trigger fires inside the producer's transaction. NOTIFY is only delivered on commit, so a rolled-back insert never rings and a worker never wakes up to a row it can't see.
The worker
Install the driver and tsx to run TypeScript directly. The worker uses top-level await, so mark the package as an ES module:
npm init -y
npm pkg set type=module
npm install pg
npm install -D tsx typescript @types/pg
The worker is one file, worker.ts, built in three parts. Start it with the config, the two connections and a stand-in handler:
import pg, { type PoolClient } from "pg";
const POLL_MS = 5_000;
const MAX_ATTEMPTS = 5;
type Job = { id: string; payload: unknown; attempts: number };
// both read PGUSER, PGPASSWORD, PGDATABASE from the environment
const pool = new pg.Pool({ max: 2 });
const listener = new pg.Client();
let running = true;
let rung = false;
let wake = () => {};
async function handle(job: Job): Promise<void> {
await new Promise((r) => setTimeout(r, 500)); // fake work
console.log(`pid ${process.pid} did job ${job.id}`);
}
The pool has two connections for claiming jobs. The listener is a separate, long-lived Client, because a LISTEN belongs to the session that ran it. Append the claim logic:
async function claimWith(db: PoolClient): Promise<boolean> {
await db.query("BEGIN");
const { rows } = await db.query<Job>(
`SELECT id, payload, attempts FROM jobs
WHERE status = 'pending' AND run_at <= now()
ORDER BY run_at
LIMIT 1
FOR UPDATE SKIP LOCKED`,
);
const job = rows[0];
if (!job) {
await db.query("COMMIT");
return false;
}
let status = "done";
try {
await handle(job);
} catch (err) {
console.error(`job ${job.id} failed:`, err);
const last = job.attempts + 1 >= MAX_ATTEMPTS;
status = last ? "failed" : "pending";
}
await db.query(
`UPDATE jobs
SET status = $2,
attempts = attempts + 1,
run_at = now() + make_interval(secs => 2 ^ attempts)
WHERE id = $1`,
[job.id, status],
);
await db.query("COMMIT");
return true;
}
async function claimOne(): Promise<boolean> {
const db = await pool.connect();
try {
const worked = await claimWith(db);
db.release();
return worked;
} catch (err) {
db.release(true); // closing it makes Postgres roll back
throw err;
}
}
Each claimOne call opens a transaction, locks one ready row with FOR UPDATE SKIP LOCKED, runs the handler, marks the row, and commits. SKIP LOCKED is what lets two workers share the table: rows another worker has locked are skipped instead of waited on. The SELECT docs call this "not suitable for general purpose work" but fine "to avoid lock contention with multiple consumers accessing a queue-like table", which is this table.
A thrown handler error puts the job back to pending with run_at pushed out by 2^attempts seconds. After five attempts it parks in failed for a human to look at.
If a query fails mid-job, release(true) destroys the connection instead of returning it to the pool, so Postgres rolls back and the row unlocks. The error then escapes the main loop and the process exits.
Last part: the wait, the signal handlers and the main loop.
function waitForBell(ms: number): Promise<void> {
if (rung) return Promise.resolve();
return new Promise((resolve) => {
const timer = setTimeout(resolve, ms);
wake = () => {
clearTimeout(timer);
resolve();
};
});
}
function stop() {
running = false;
wake();
}
listener.on("notification", () => {
rung = true;
wake();
});
listener.on("error", (err) => {
console.error("listener lost, exiting:", err.message);
process.exit(1); // let the supervisor restart us
});
pool.on("error", (err) => {
console.error("idle client:", err.message);
});
process.on("SIGINT", stop);
process.on("SIGTERM", stop);
await listener.connect();
await listener.query("LISTEN jobs"); // listen first, then look
while (running) {
rung = false;
while (running && (await claimOne())) {}
if (running) await waitForBell(POLL_MS);
}
await listener.end();
await pool.end();
Listen, then look. The worker runs LISTEN before its first drain. The LISTEN docs recommend exactly this order: register and commit first, then "inspect the database state", then rely on notifications for later changes. Do it the other way round and a job inserted between your check and your LISTEN sits there until the next poll.
Then it drains until empty: the inner loop calls claimOne as long as it returns true.
The rung flag closes the gap. The flag resets before each drain. If a notification lands while the worker is busy, rung flips to true and waitForBell returns at once, so the worker drains again instead of sleeping on a fresh job.
The poll covers everything the bell misses. After POLL_MS with no ring, the worker drains anyway. That catches rings sent while it was down, retries whose run_at is in the future (nothing rings for those), and any notification lost for any other reason. The bell decides how fast you react. The poll decides whether you're correct.
Crashes are boring on purpose. If the listener connection dies, or a claim query fails, the process exits and whatever supervises it starts a new one, which drains the backlog on boot.
Kill it mid-stream
Open two terminals with the three PG* variables exported, plus your psql session. In each terminal, start a worker:
npx tsx worker.ts
In psql, queue 50 jobs in one statement:
INSERT INTO jobs (payload)
SELECT jsonb_build_object('n', g)
FROM generate_series(1, 50) AS g;
Both workers wake on the single ring and start printing lines with different pids, since SKIP LOCKED hands each worker different rows. While they're going, find one worker's pid in its output and kill it hard:
kill -9 <pid>
The worker never gets to COMMIT. The OS closes its socket, Postgres rolls back the open transaction, the row lock goes away, and the job is pending again. The surviving worker picks it up on its next claim.
Now the scenario from the opening. Stop the surviving worker with Ctrl-C (it finishes its current job, then exits), insert another 50 rows while no worker is running, and start one worker again. Nobody heard those rings. The new worker drains all 50 on boot anyway, because its first move is to look at the table. Then check:
SELECT status, count(*) FROM jobs GROUP BY status;
Every row should be done. In production, if a row sits in pending with a run_at well in the past, either no worker is running or the ones you have are stuck. That's a query you can alert on. You can't alert on a NOTIFY nobody received.
What this design does not promise
It is at-least-once, not exactly-once. If the worker dies after handle did its side effect but before COMMIT, the job runs again. Make the handler idempotent: key external calls on the job id.
The transaction stays open for the whole job. That's fine for jobs measured in seconds. For long jobs it holds a connection per worker, and if the machine vanishes rather than the process, Postgres only notices the dead client when the socket finally errors, which depends on settings like tcp_keepalives_idle. For long jobs, claim with a lease and a heartbeat instead. This earlier post builds that version, with fencing tokens.
The listener needs a real session. PgBouncer lists LISTEN as never supported in transaction pooling, so point the listener Client at Postgres directly or at a session-mode pool.
The global commit lock still applies to every transaction that inserts a job. If job inserts are a big share of a busy database's writes, you can delete the trigger and run on the poll alone. Because the table is the source of truth, that's a one-line DROP TRIGGER, and the worker keeps running on the poll with up to POLL_MS of extra latency.
If this was useful
The doorbell-plus-mailbox split is one case of a rule the Event-Driven Architecture Pocket Guide keeps coming back to: the event says something happened, durable state says what. The book covers the same split for the outbox pattern, sagas and CQRS, plus the traps around retries, ordering and duplicate delivery that this worker only starts to handle.
Top comments (1)
The kill -9 walkthrough is convincing for the happy case, but one consequence of keeping the handler inside the claim transaction is worth spelling out. The attempts increment and the backoff on run_at only get written in the UPDATE right before COMMIT, so a job that crashes the process (OOM, native addon segfault, kill from the supervisor) rolls back with the rest and comes back with attempts unchanged and no delay. A job that throws is capped at MAX_ATTEMPTS, but a job that takes the worker down is not, so a poison payload can restart-loop workers indefinitely and the failed bucket never sees it.
A hung handler has the mirror problem: it isn't dead, so Postgres never rolls anything back, and the row stays locked and a pool connection stays idle in transaction. I'd probably bump attempts in a short separate transaction at claim time, or at least set idle_in_transaction_session_timeout, so a stuck worker eventually releases the row. Curious whether you'd go that way or move to a claimed_at lease instead.