Stop Installing Redis Just to Run Background Jobs
Most Node.js apps reach for Redis the moment they need background jobs. But before you add that dependency, it's worth understanding what's actually happening under the hood — because it's more than most people realise.
Every enqueue is a network call. Your app serializes the job payload, opens a TCP connection (or borrows one from the pool), performs a TLS handshake if the connection is remote, sends the command, and waits for an acknowledgement. On a local network that's a few milliseconds. On a managed Redis instance, it's more. It adds up.
Durability is not free. By default, Redis is an in-memory store. To make jobs survive a restart you need to configure AOF (Append Only File) or RDB snapshots — which means tuning fsync behaviour, understanding the tradeoffs between always, everysec, and no, and accepting that in some failure modes you still lose the last second of writes. Getting this right for a job queue is non-trivial.
A single instance is a single point of failure. If Redis goes down — the process crashes, the container restarts, the managed service has an outage — your queue stops. To harden that you need Redis Sentinel or Cluster, which introduces leader election, replication lag, and split-brain scenarios. For a single-server app, you now have more infrastructure complexity than the application itself.
CPU-heavy jobs make it worse. If you're running jobs like PDF generation, image processing, or report building directly in Node.js, they block the event loop — meaning your HTTP server stalls while a job is crunching. The standard fix is worker_threads, but wiring that up manually with a pool, job dispatch, and error propagation is a lot of boilerplate on top of an already-complex setup.
Observability requires extra plumbing. Background jobs are notoriously invisible. When you use Redis, getting visibility into queue health usually means installing a secondary dashboard UI service, manually exporting metrics to a collector, or writing custom polling scripts just to find out why a job failed or how long it’s been sitting in the backlog.
I built lite-q to eliminate all of this for the case where it shouldn't be this hard. It's a persistent task queue for Node.js backed entirely by SQLite:
Zero Infrastructure: Jobs are written to disk atomically, survive crashes, and retry with exponential backoff. No TCP round-trips, no TLS overhead, and no external process to keep alive.
Built-in Thread Isolation: CPU-bound work automatically runs in a managed worker thread pool, keeping your event loop completely free.
Instant Observability: It exposes standard Prometheus metrics out of the box. You get a clean endpoint to track pending backlogs, failure rates, handler durations, and cron reliability instantly—no extra microservices required.
No TLS overhead. No replication to configure. No external process to keep alive. Just a file on disk and a single npm install.
npm install @km-dev/lite-q
What problems does it solve?
1. Background jobs without infrastructure
Register a handler, enqueue a job, done. Everything runs in-process and persists to a local SQLite file.
import { LiteQ } from '@km-dev/lite-q';
const queue = new LiteQ({ storagePath: './jobs.db' });
const sendEmail = queue.register<{ to: string; subject: string }>(
'send-email',
async (job) => {
await mailer.send(job.data.to, job.data.subject);
}
);
await queue.start();
await sendEmail({ to: 'user@example.com', subject: 'Welcome!' });
2. Jobs survive crashes and restarts
Jobs are written to SQLite before they run. If your app crashes mid-job, it picks back up on the next restart — nothing silently disappears.
3. CPU-heavy work off the main thread — with a real worker pool
In Node.js, anything CPU-intensive (PDF generation, image processing, report building, data crunching) blocks the event loop. Every millisecond your code is crunching numbers is a millisecond your server can't handle incoming requests.
lite-q solves this by routing CPU jobs to a generic worker thread pool. Just pass a file path instead of a callback — lite-q detects the difference, spins up workers, and keeps your main thread free.
const generatePdf = queue.register('generate-pdf', './workers/pdf-worker.js');
await generatePdf({ orderId: 'ord_123' });
// workers/pdf-worker.js — runs in an isolated thread, no worker_threads boilerplate
export default async function (job) {
const url = await buildAndUploadPdf(job.data);
return { url };
}
The pool manages itself: idle workers are reused, new ones are spawned up to maxWorkers, and excess threads above minWorkers are trimmed automatically. I/O jobs and CPU jobs use completely separate concurrency controls — they never block each other.
const queue = new LiteQ({
storagePath: './jobs.db',
concurrency: 4, // max concurrent I/O jobs on the main thread
minWorkers: 2, // keep 2 threads warm and ready
maxWorkers: 8, // scale up to 8 under load
});
No worker_threads API. No manual thread lifecycle. Just write the handler, register it with a path, and the pool handles the rest.
4. Recurring jobs (cron) — persistent and observable
Schedules are stored in SQLite, so they survive restarts. Each run gets its own execution record, so you have full history.
queue.cron('cleanup-sessions', '0 0 * * *', async (job) => {
await db.deleteExpiredSessions(job.data.batchSize);
}, { payload: { batchSize: 500 } });
5. Delayed jobs, retries, and priority
// Run 1 hour from now
await checkTrialExpiry({ userId: 'usr_9011' }, { delay: 60 * 60 * 1000 });
// Custom retry config with exponential backoff
await syncLedger({ transactionId: 'ch_3Mv1' }, { maxRetries: 5 });
// High-priority job — runs before lower-priority ones
await sendAlert(data, { priority: 100 });
6. Prometheus metrics out of the box
Background jobs are invisible by default. lite-q exposes a /metrics endpoint you can scrape with Prometheus to track pending backlogs, failure rates, handler durations, and cron reliability.
app.get('/metrics', async (_req, res) => {
res.type('text/plain; version=0.0.4; charset=utf-8');
res.send(await queue.metrics({ windowMs: 7 * 24 * 60 * 60 * 1000 }));
});
7. Full control over your data — it stays on your machine
Because lite-q uses SQLite, every job, every cron execution, every retry record lives in a plain file on your disk. You can inspect it, query it, back it up, or move it — no external service owns your data.
You also get full visibility through queue.stats() and Prometheus metrics, so you always know exactly what's in the queue, what's running, and what failed — without tailing logs or guessing.
const stats = await queue.stats();
// { pending: 3, processing: 1, completed: 142, failed: 2, total: 148 }
How it compares
| lite-q | BullMQ | Bee-Queue | |
|---|---|---|---|
| Infrastructure required | None | Redis | Redis |
| Persistent jobs | ✅ | ✅ | ❌ |
| Survives crashes | ✅ | ✅ | ❌ |
| CPU thread isolation | ✅ | ❌ | ❌ |
| Cron / scheduled jobs | ✅ | ✅ | ❌ |
| Prometheus metrics | ✅ | ❌ | ❌ |
| TypeScript built-in | ✅ | ✅ | ❌ |
| Multi-machine workers | ❌ | ✅ | ✅ |
If you need workers across multiple machines, use BullMQ. If you're on a single server and don't want to operate Redis, lite-q is the right tool.
When to use it
- Side projects and internal tools that don't need a distributed queue
- Apps already using SQLite that want to avoid adding another service
- Any single-server Node.js deployment that needs reliable background work
📦 npm: npmjs.com/package/@km-dev/lite-q
💻 GitHub: github.com/iikareem/liteQ
Would love feedback — especially if you run into edge cases or want something on the roadmap.
Top comments (4)
The "don't reach for Redis by default" instinct is right, and the SQLite durability angle is a nice hook. The thing I'd want nailed before swapping in any at-least-once queue, this one included: what's the contract when a job crashes after its side effect but before it's marked done? Your durability story says it picks back up on the next restart, which is exactly when send-email runs a second time and the user gets two welcomes. That's not a bug in lite-q, it's just the nature of at-least-once, but it means the handler has to be idempotent and that lands on the caller. Do you give handlers anything to dedup on, a stable job id they can key against, or is exactly-once effect left entirely to the user?
Good question @mickyarun - and you’re naming the real tradeoff, not a bug in the queue.
lite-q is at-least-once by design. If the process dies after the side effect ran but before the success write lands in SQLite, that job comes back as pending after jobTimeout and can run again. So the handler must be idempotent. Exactly-once effect is the caller’s job; the queue only guarantees the work isn’t silently lost.
What we give you: every handler gets a stable job.id (UUID). That ID is assigned at enqueue and stays the same across retries, so you can key dedup against it — pass it to the provider as an idempotency key, or check a local “already processed” store before doing the side effect.
queue.register('send-email', async (job) => {await emailProvider.send(job.data.to, job.data.templateId, {
idempotencyKey: job.id, // same id on retry
});
});
So: durability + restart recovery are on lite-q; “don’t send two welcomes” is on the handler, with job.id as the stable handle to make that practical. Same pattern as BullMQ / most durable queues — we just call it out explicitly because with SQLite people sometimes expect crash recovery to also mean exactly-once.
Happy to dig into edge cases (cron overlap, crash mid-retry, etc.) if useful.
Calling out that durability isn't free with Redis is the part people skip right past. Folks forget they're one AOF setting away from losing the last second of jobs, and "in-memory by default" catches a lot of teams off guard. Quick question on the SQLite side though. With everything writing to one file, how does it hold up when a burst of enqueues and the worker's status updates all want the write lock at once? SQLite serializes writes, so I'm curious whether WAL mode was enough or if you had to batch the enqueues to keep the event loop from stalling under load. That's the one number I'd want before swapping Redis out on a busy single server.