DEV Community

Cover image for PortQueue: background jobs on the database you already have
Gherciu Gheorghe
Gherciu Gheorghe

Posted on Edited on Originally published at episclera.github.io

PortQueue: background jobs on the database you already have

Almost every app needs to do work in the background at some point: build a sales report, resize an upload, call a slow API, send a receipt. You don't want the user waiting for any of that, so you put it on a queue and let a worker handle it later.

The usual advice is to add a separate queue server. But your data already lives in a database you're paying for and running anyway. Most databases can push changes in real time and scale well, so why add another service just for jobs?

PortQueue takes a simpler route: your jobs live in the database you already run. PostgreSQL, Oracle, MySQL, PGLite or SQLite all work. There's nothing extra to install on your servers, and it's free and open source (MIT).

What it looks like

You set up your queue and say what to do with each job, then add jobs from anywhere in your app:

import { PortQueue } from '@portqueue/core';
import { PostgresqlDriver } from '@portqueue/core/drivers/postgresql';

const pq = new PortQueue({
  driver: new PostgresqlDriver({ connectionString: process.env.DATABASE_URL }),
});
await pq.init();

// What should happen to each job on the "reports" queue.
pq.createQueue({
  name: 'reports',
  handler: async (job) => {
    await buildReport(job.data);
  },
});

// Somewhere in your app: "build the sales report when you can".
await pq.addJob({ queue: 'reports', data: { kind: 'sales' } });
Enter fullscreen mode Exit fullscreen mode

That's all it takes to get started. Here's what you get on top of it.

Is it fast?

Yes, and that's checked continuously. Every change to PortQueue runs a head-to-head benchmark before it ships. Here it is against pg-boss, a popular PostgreSQL-based queue, on the same database (PGLite, Apple M1 Max, Node 24, median of 3 runs):

What we measured PortQueue pg-boss
Adding jobs one at a time 2,294 jobs/s 1,141 jobs/s ~2ร— faster
Adding jobs in batches 76,938 jobs/s 41,952 jobs/s ~1.8ร— faster
Processing jobs 20,131 jobs/s 17,441 jobs/s ~15% faster
Time until a new job starts (median) 1.2 ms 2.3 ms ~2ร— quicker

In plain terms, a single server can take in tens of thousands of jobs a second, and a new job usually starts about a millisecond after you add it.

The scenarios are open and you can run them yourself. The benchmarks page explains exactly how each one is measured.

Features

๐Ÿ—„๏ธ Works with the database you already have

PortQueue runs on PostgreSQL, Oracle, MySQL, PGLite and SQLite, and your code is the same on all of them. Moving from SQLite on your laptop to PostgreSQL in production means changing one line.

Your queue also gets everything your database already has: backups, monitoring, access control. If you want to know what's stuck, it's just a table you can look at.

๐Ÿ”’ Jobs are saved together with your data

This one saves you from a whole class of quiet bugs. Imagine a customer places an order and you queue a "send receipt" job. If your app crashes in between, you end up with an order and no receipt, or a receipt for an order that never got saved.

Because PortQueue lives in your database, the job can be saved in the same step as the order. Either both are saved or neither is. It plugs into the tools you probably use already: Prisma, Drizzle, Knex, Kysely, TypeORM, Sequelize and MikroORM.

๐Ÿ“ˆ Scale by adding servers

More work coming in? Start another copy of your app. That's the whole scaling story. There's no setup and no "main" server to keep alive, and two servers never run the same job twice.

Some jobs need a particular kind of machine, like a video export that wants lots of memory. You can label your servers by what they have (ram-8gb, gpu, a region) and send those jobs only to matching servers. If one of those machines goes away, any other server with the same label picks up the work.

๐Ÿ” Retries when things fail

APIs time out and networks blip. When a job fails, PortQueue tries again, waiting a bit longer each time, up to a limit you choose. If a server crashes in the middle of a job, another one notices and picks the job back up, so nothing quietly gets lost.

โฐ Delayed jobs, repeating jobs and cron

The same addJob call handles timing too:

  • "Do this in 24 hours", for things like a report that should wait until the day's numbers are in.
  • "Do this every 30 seconds", for things like health checks.
  • "Every weekday at 8:00 New York time", using cron. Time zones and daylight saving are handled for you.

If your app was down when a scheduled job was due, it runs once when you're back. It doesn't fire a burst of missed runs all at once.

๐Ÿšฆ Control how fast work happens

You can set how many jobs a server runs at once, cap how many run per minute (handy when a third-party API has rate limits), and have a server pause taking new work when it's running low on CPU or memory.

๐Ÿงน Cancel, dedupe and clean up

  • Cancel a job that's no longer needed.
  • Add the same job twice by accident? Give it a key and PortQueue keeps just one.
  • Set how long finished jobs are kept and PortQueue deletes old ones for you, so the table doesn't grow forever.

๐Ÿ“Š A dashboard you host yourself

@portqueue/dashboard adds a dashboard to the web server you already have (Express, Fastify, Koa, NestJS or plain Node). You can see every queue, find any job, retry or cancel it, and keep an eye on your servers. Your data never goes to a third-party service.

The overview shows what's running, waiting, failed and done across all your queues:

PortQueue dashboard overview: activity cards above a table of queues with per-status counts

Click into a queue to search its jobs and retry, cancel or remove them, one at a time or in bulk. You can also add a job straight from the browser:

A queue's detail page: status cards above the paged, searchable jobs table

Open any job to see exactly what happened: what it was given, what it returned, why it failed and when each step took place:

A job's detail page: the full record, its payload and return value, and the lifecycle timeline

Your repeating jobs are all in one list, with when each runs next and a switch to turn it off:

The schedules page: every recurring schedule with its pattern and next run

The servers page shows each of your servers, how busy it is, and lets you pause any one of them without touching the rest:

The instances page: every keyed server with its heartbeat, CPU and heap usage

๐Ÿ’ป A command-line tool too

Like the terminal better? @portqueue/cli gives you the same controls: live stats, retrying a job, setting up the database, and even serving the dashboard without changing your app:

portqueue stats --watch --connection-string "$DATABASE_URL"
portqueue dashboard --root-user admin --root-password "$DASH_PASSWORD" --connection-string "$DATABASE_URL"
Enter fullscreen mode Exit fullscreen mode

๐Ÿงฉ Fits your TypeScript code

It's written in TypeScript, so your job data keeps its type from the moment you add a job to the moment it's processed. It also has events and hooks if you want to send metrics or traces to your monitoring tools.

Get started

npm install @portqueue/core pg   # or: oracledb, mysql2, @electric-sql/pglite, better-sqlite3
Enter fullscreen mode Exit fullscreen mode

If you've ever added Redis to your stack just to build reports in the background, give PortQueue a try. Issues, ideas and PRs are all welcome. Which database would you run it on? Let me know in the comments ๐Ÿ‘‡

Top comments (4)

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

The benchmark table is the part that matters here. If the queue lives in Postgres anyway, the extra broker is the thing to justify.

Collapse
 
gherciu profile image
Gherciu Gheorghe • • Edited

Exactly.
If Postgres is already there, backed up and monitored, a separate broker has to earn its place. That's why PortQueue is just a library: it runs in your backend code and uses your existing database, so there's nothing else to deploy. Looking at the benchmarks, the throughput you get straight from the database covers what the vast majority of projects will ever need. That's especially true early on, when adding Redis means one more thing to run before you even know you need it. You can always bring in a broker later if you outgrow it.

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

that tracks. the throughput point is the one that decides it for most projects, and a library riding on the database they already run is a smaller bet than a new broker.

Collapse
 
corinabotico profile image
corinabotico •

awesome!