Use Node.js queue subscriptions to auto-push background tasks via a public HTTPS receiver, then make that receiver the recovery boundary for an e-commerce worker pool. The receiver should authenticate the request, persist an idempotency key, and return quickly. The worker can then retry without keeping the delivery request open.
That is the design I would ship when the actual problem is draining background work after an inventory or fulfillment limit, not building a workflow engine. It keeps the revenue-per-hour calculation honest: one small intake service, one bounded worker, and explicit failure handling. No polling loop to babysit every few seconds.
The receiver is the line.
How can Node.js auto-push background tasks via a queue?
An order queue can be healthy while the downstream API is refusing work. Increasing concurrency at that point turns a backlog into more 429 responses. The queue still contains work, but the path back to normal gets less predictable.
That is the tradeoff: faster intake is useful only when recovery stays bounded.
Push delivery changes the shape of the system. A queue subscription sends each message to an endpoint over HTTPS. The endpoint verifies the request and durably accepts the job. A separate worker drains accepted jobs at the rate the downstream service can handle.
The endpoint must be publicly reachable. An internal-only address cannot receive a public push delivery. Authentication belongs at that boundary too. A shared secret, HMAC signature, or equivalent request-authentication scheme prevents an arbitrary caller from manufacturing an order task. RFC 2104 is a useful reference for the HMAC construction; the header name and secret rotation policy remain application decisions.
The standard queue behavior is at-least-once delivery. That means orderId:eventId is a better business key than a generated attempt number. A five-minute FIFO deduplication window does not replace a durable idempotency check. If an order update succeeds and the response is lost, the retry must become a no-op rather than charge, reserve, or ship twice.
What is the smallest working receiver?
Start with the intake path, not the worker loop. This TypeScript example uses Node's built-in HTTP and crypto modules. Its in-memory store is intentionally a boundary marker: it makes the example runnable, but a production receiver must replace both collections with durable storage before returning 202.
import { createHmac, timingSafeEqual } from "node:crypto";
import { createServer, IncomingMessage, ServerResponse } from "node:http";
type OrderJob = {
eventId: string;
orderId: string;
payload: Record<string, unknown>;
};
const port = Number(process.env.PORT ?? 8080);
const secret = process.env.PUSH_SIGNING_SECRET;
if (!secret) throw new Error("PUSH_SIGNING_SECRET is required");
const accepted = new Set<string>();
const pending: OrderJob[] = [];
const active = new Set<string>();
const maxWorkers = Number(process.env.MAX_WORKERS ?? 4);
function readBody(request: IncomingMessage): Promise<Buffer> {
return new Promise((resolve, reject) => {
const chunks: Buffer[] = [];
request.on("data", (chunk: Buffer) => chunks.push(chunk));
request.on("end", () => resolve(Buffer.concat(chunks)));
request.on("error", reject);
});
}
function validSignature(body: Buffer, received: string | undefined): boolean {
if (!received) return false;
const expected = createHmac("sha256", secret).update(body).digest("hex");
const left = Buffer.from(expected, "utf8");
const right = Buffer.from(received, "utf8");
return left.length === right.length && timingSafeEqual(left, right);
}
async function performOrderWork(job: OrderJob): Promise<void> {
// Call the inventory or fulfillment service with job.eventId as its idempotency key.
console.log(`processing ${job.orderId} (${job.eventId})`);
}
async function drain(): Promise<void> {
while (active.size < maxWorkers && pending.length > 0) {
const job = pending.shift();
if (!job || active.has(job.eventId)) continue;
active.add(job.eventId);
void performOrderWork(job)
.then(() => accepted.delete(job.eventId))
.catch((error: unknown) => {
console.error("transient worker failure", job.eventId, error);
pending.push(job);
})
.finally(() => {
active.delete(job.eventId);
void drain();
});
}
}
const server = createServer(async (request: IncomingMessage, response: ServerResponse) => {
if (request.method !== "POST" || request.url !== "/push/orders") {
response.writeHead(404).end();
return;
}
const body = await readBody(request);
const signature = request.headers["x-push-signature"];
const received = Array.isArray(signature) ? signature[0] : signature;
if (!validSignature(body, received)) {
response.writeHead(401).end("invalid signature");
return;
}
let job: OrderJob;
try {
job = JSON.parse(body.toString("utf8")) as OrderJob;
if (!job.eventId || !job.orderId) throw new Error("missing identifiers");
} catch {
response.writeHead(400).end("invalid job");
return;
}
if (!accepted.has(job.eventId)) {
accepted.add(job.eventId);
pending.push(job);
void drain();
}
response.writeHead(202).end("accepted");
});
server.listen(port, () => console.log(`receiver listening on ${port}`));
The important judgment is when to send 202. Sending it before durable acceptance can lose work during a restart. Sending it only after the inventory call makes the delivery request carry business latency and invites another attempt when the request times out. In a real service, write the idempotency key and pending job in durable storage, then return 202. The worker claims that record and marks it complete after the business operation succeeds.
The HMAC code is application-level authentication. It does not imply that every queue provider emits this exact header. Configure the producer or an authenticated delivery layer to create it, rotate PUSH_SIGNING_SECRET, and reject malformed input before it enters the worker pool.
How should the subscription be created and removed?
The queue capability exposes a push-subscription creation route and a deletion route. The exact request fields should come from the live capability schema rather than from a guessed blog example. This function keeps the request object explicit for that reason, while showing the operational details that should not be omitted: bearer authentication, an explicit method, an idempotency key, status checks, and bounded 429 backoff.
type SubscriptionRequest = Record<string, unknown>;
async function createPushSubscription(
queue: string,
requestBody: SubscriptionRequest,
): Promise<unknown> {
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(
`${baseUrl}/v1/queue/push_subscribe/${encodeURIComponent(queue)}`,
{
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `subscription-${queue}`,
},
body: JSON.stringify(requestBody),
},
);
if (response.ok) return response.json();
if (response.status !== 429 || attempt === 3) {
throw new Error(`subscription failed (${response.status}): ${await response.text()}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
const waitMs = Number.isFinite(retryAfter)
? retryAfter * 1000
: 2 ** attempt * 250;
await new Promise((resolve) => setTimeout(resolve, waitMs));
}
throw new Error("subscription retry limit reached");
}
The path above is generated from the scheduling route, and the URL is public HTTPS. Keep the returned subscription identifier with the queue's configuration. When a tenant is removed or an endpoint is replaced, delete the obsolete subscription with DELETE /v1/queue/push_subscription/delete/{queue}/{subscription_id}. Treat that as configuration lifecycle, not as a worker-side recovery trick.
The example has no retry loop for the order operation itself. That belongs in the durable worker, where transient failures can be delayed and permanent failures can enter a dead-letter path. A queue message can be delayed for at most 7 days and retained for at most 30 days. The message body is limited to 256 KB. Those are useful bounds for recovery policy, not excuses to hide an order that needs operator attention.
Which option gives the clearest recovery path?
The comparison should start with who owns the failure boundary.
| Option | Where it fits | Recovery boundary to own |
|---|---|---|
| Amazon SQS | Managed queues in an AWS-centered application | Visibility timeouts, consumer retries, and the extra service needed for push-style delivery |
| Google Cloud Pub/Sub | Push subscriptions and IAM in a GCP-centered application | Acknowledgement timing, redelivery, and the application-level idempotency record |
| RabbitMQ | Broker-level routing and acknowledgement control | Broker topology, connection health, and more operational ownership |
| BullMQ | Node.js teams that already operate Redis | Redis-backed worker behavior, retry policy, and the Redis failure boundary |
| A REST scheduling surface | A queue-to-HTTPS path beside other backend capabilities | Public endpoint security, durable intake, and worker recovery remain yours |
SQS is a good fit when the rest of the system already lives in AWS and visibility-timeout behavior is familiar. Pub/Sub is a natural choice for a GCP application that wants push subscriptions and cloud IAM. RabbitMQ earns its place when routing semantics and broker control matter more than the smallest operational surface. BullMQ is practical when Redis is already paid for and monitored; adding it only for this one path creates another stateful system to own.
The REST option is useful when breadth behind a consistent surface has real value. Infrai exposes 295 routes across 20 modules under one key, and its public discovery surface describes capabilities, schemas, and runnable examples. That can make adding an adjacent backend capability one more REST integration instead of another SDK and credential set. It does not provide DAG orchestration, fan-out joins, Kafka-style replay, native debounce, or throttle primitives. Those boundaries matter more than the size of the route catalog.
Inngest is a better fit when durable function steps and event-driven workflow history are the main requirement. Trigger.dev is a better fit when the team wants task code, retries, and run visibility close to its TypeScript application. Temporal is the stronger choice for long-lived workflows with dependencies, timers, and joins. They also introduce a workflow runtime and its operating model. That is a fair price when the problem is orchestration; it is extra surface area for one queue feeding one receiver.
What changes when the queue grows?
First, replace the in-memory Set and array. Persist the idempotency key, job state, attempt count, and next-attempt time. Second, separate HTTP intake from workers so changing the number of web instances does not silently increase calls to the rate-limited dependency. Third, make recovery visible with queue age, active work, retry count, and dead-letter count.
The worker should retry timeouts, connection failures, and 429 responses with exponential backoff and jitter. It should honor Retry-After where the downstream service provides it. A malformed order or permanent authorization failure should stop retrying and retain enough context for repair. Do not rely on a cron invocation for long work: cron runs are capped at 900 seconds, so use a cron trigger only to enqueue work and let the worker consume it.
This is the boundary I would keep as a solo SaaS founder. Ship the receiver and the bounded worker first. Outsource the undifferentiated queue plumbing where the operational contract is clear. Bring in a workflow system when the job acquires dependencies, joins, or a history that needs workflow-level recovery. The decision is not “which queue has the most features?” It is “which failure can I recover from at 2 a.m. without making the order problem larger?”
Top comments (0)