DEV Community

Kareem Ashraf
Kareem Ashraf

Posted on

Stop Over-Engineering Your Node.js Background Jobs

#node #npm #redis #opensource

Stop Installing Redis Just to Run Background Jobs

Most Node.js apps reach for Redis the moment they need background jobs. But before you add that dependency, it's worth understanding what's actually happening under the hood — because it's more than most people realise.

Every enqueue is a network call. Your app serializes the job payload, opens a TCP connection (or borrows one from the pool), performs a TLS handshake if the connection is remote, sends the command, and waits for an acknowledgement. On a local network that's a few milliseconds. On a managed Redis instance, it's more. It adds up.

Durability is not free. By default, Redis is an in-memory store. To make jobs survive a restart you need to configure AOF (Append Only File) or RDB snapshots — which means tuning fsync behaviour, understanding the tradeoffs between always, everysec, and no, and accepting that in some failure modes you still lose the last second of writes. Getting this right for a job queue is non-trivial.

A single instance is a single point of failure. If Redis goes down — the process crashes, the container restarts, the managed service has an outage — your queue stops. To harden that you need Redis Sentinel or Cluster, which introduces leader election, replication lag, and split-brain scenarios. For a single-server app, you now have more infrastructure complexity than the application itself.

CPU-heavy jobs make it worse. If you're running jobs like PDF generation, image processing, or report building directly in Node.js, they block the event loop — meaning your HTTP server stalls while a job is crunching. The standard fix is worker_threads, but wiring that up manually with a pool, job dispatch, and error propagation is a lot of boilerplate on top of an already-complex setup.

Observability requires extra plumbing. Background jobs are notoriously invisible. When you use Redis, getting visibility into queue health usually means installing a secondary dashboard UI service, manually exporting metrics to a collector, or writing custom polling scripts just to find out why a job failed or how long it’s been sitting in the backlog.

I built lite-q to eliminate all of this for the case where it shouldn't be this hard. It's a persistent task queue for Node.js backed entirely by SQLite:

Zero Infrastructure: Jobs are written to disk atomically, survive crashes, and retry with exponential backoff. No TCP round-trips, no TLS overhead, and no external process to keep alive.

Built-in Thread Isolation: CPU-bound work automatically runs in a managed worker thread pool, keeping your event loop completely free.

Instant Observability: It exposes standard Prometheus metrics out of the box. You get a clean endpoint to track pending backlogs, failure rates, handler durations, and cron reliability instantly—no extra microservices required.

No TLS overhead. No replication to configure. No external process to keep alive. Just a file on disk and a single npm install.

npm install @km-dev/lite-q
Enter fullscreen mode Exit fullscreen mode

What problems does it solve?

1. Background jobs without infrastructure

Register a handler, enqueue a job, done. Everything runs in-process and persists to a local SQLite file.

import { LiteQ } from '@km-dev/lite-q';

const queue = new LiteQ({ storagePath: './jobs.db' });

const sendEmail = queue.register<{ to: string; subject: string }>(
  'send-email',
  async (job) => {
    await mailer.send(job.data.to, job.data.subject);
  }
);

await queue.start();
await sendEmail({ to: 'user@example.com', subject: 'Welcome!' });
Enter fullscreen mode Exit fullscreen mode

2. Jobs survive crashes and restarts

Jobs are written to SQLite before they run. If your app crashes mid-job, it picks back up on the next restart — nothing silently disappears.

3. CPU-heavy work off the main thread — with a real worker pool

In Node.js, anything CPU-intensive (PDF generation, image processing, report building, data crunching) blocks the event loop. Every millisecond your code is crunching numbers is a millisecond your server can't handle incoming requests.

lite-q solves this by routing CPU jobs to a generic worker thread pool. Just pass a file path instead of a callback — lite-q detects the difference, spins up workers, and keeps your main thread free.

const generatePdf = queue.register('generate-pdf', './workers/pdf-worker.js');

await generatePdf({ orderId: 'ord_123' });
Enter fullscreen mode Exit fullscreen mode 
// workers/pdf-worker.js — runs in an isolated thread, no worker_threads boilerplate
export default async function (job) {
  const url = await buildAndUploadPdf(job.data);
  return { url };
}
Enter fullscreen mode Exit fullscreen mode

The pool manages itself: idle workers are reused, new ones are spawned up to maxWorkers, and excess threads above minWorkers are trimmed automatically. I/O jobs and CPU jobs use completely separate concurrency controls — they never block each other.

const queue = new LiteQ({
  storagePath: './jobs.db',
  concurrency: 4,   // max concurrent I/O jobs on the main thread
  minWorkers: 2,    // keep 2 threads warm and ready
  maxWorkers: 8,    // scale up to 8 under load
});
Enter fullscreen mode Exit fullscreen mode

No worker_threads API. No manual thread lifecycle. Just write the handler, register it with a path, and the pool handles the rest.

4. Recurring jobs (cron) — persistent and observable

Schedules are stored in SQLite, so they survive restarts. Each run gets its own execution record, so you have full history.

queue.cron('cleanup-sessions', '0 0 * * *', async (job) => {
  await db.deleteExpiredSessions(job.data.batchSize);
}, { payload: { batchSize: 500 } });
Enter fullscreen mode Exit fullscreen mode

5. Delayed jobs, retries, and priority 

// Run 1 hour from now
await checkTrialExpiry({ userId: 'usr_9011' }, { delay: 60 * 60 * 1000 });

// Custom retry config with exponential backoff
await syncLedger({ transactionId: 'ch_3Mv1' }, { maxRetries: 5 });

// High-priority job — runs before lower-priority ones
await sendAlert(data, { priority: 100 });
Enter fullscreen mode Exit fullscreen mode

6. Prometheus metrics out of the box

Background jobs are invisible by default. lite-q exposes a /metrics endpoint you can scrape with Prometheus to track pending backlogs, failure rates, handler durations, and cron reliability.

app.get('/metrics', async (_req, res) => {
  res.type('text/plain; version=0.0.4; charset=utf-8');
  res.send(await queue.metrics({ windowMs: 7 * 24 * 60 * 60 * 1000 }));
});
Enter fullscreen mode Exit fullscreen mode

7. Full control over your data — it stays on your machine

Because lite-q uses SQLite, every job, every cron execution, every retry record lives in a plain file on your disk. You can inspect it, query it, back it up, or move it — no external service owns your data.

You also get full visibility through queue.stats() and Prometheus metrics, so you always know exactly what's in the queue, what's running, and what failed — without tailing logs or guessing.

const stats = await queue.stats();
// { pending: 3, processing: 1, completed: 142, failed: 2, total: 148 }
Enter fullscreen mode Exit fullscreen mode

How it compares

lite-q BullMQ Bee-Queue
Infrastructure required None Redis Redis
Persistent jobs ✅ ✅ ❌
Survives crashes ✅ ✅ ❌
CPU thread isolation ✅ ❌ ❌
Cron / scheduled jobs ✅ ✅ ❌
Prometheus metrics ✅ ❌ ❌
TypeScript built-in ✅ ✅ ❌
Multi-machine workers ❌ ✅ ✅

If you need workers across multiple machines, use BullMQ. If you're on a single server and don't want to operate Redis, lite-q is the right tool.

When to use it

  • Side projects and internal tools that don't need a distributed queue
  • Apps already using SQLite that want to avoid adding another service
  • Any single-server Node.js deployment that needs reliable background work

📦 npm: npmjs.com/package/@km-dev/lite-q

💻 GitHub: github.com/iikareem/liteQ

Would love feedback — especially if you run into edge cases or want something on the roadmap.

Top comments (0)