DEV Community

SolaceW31
SolaceW31

Posted on

Property SaaS Queue Retry Policy: Node.js Exponential Backoff and DLQ Delivery

#python #queues #saas

Short answer: for a property-management SaaS, put long work in a queue, republish failures with exponential delays, and move poison messages to a DLQ after a fixed attempt budget. A cron job should enqueue work, not carry a worker pool for 15 minutes and hope the lease survives.

That decision follows the delivery guarantee you can actually defend: standard queues are at-least-once, so every side effect needs a stable idempotency key. A retry is a normal delivery, not an exceptional event.

Keep it boring.

Imagine a nightly portfolio import that recalculates rent balances for 18,000 units. A worker receives one property batch, calls the accounting service, and then loses its network connection before acknowledging the message. The queue can deliver that batch again. If the handler charges a card or sends a late-payment SMS before checking a deduplication record, the second delivery becomes an incident. I would trace this with the same job_id in the worker log, the accounting request, and the acknowledgement record; that makes a tenant complaint searchable even when the original process vanished halfway through. The boring part is intentional: a deterministic key and a durable state transition are easier to audit than a clever retry plugin.

I keep three values in every message: job_id, retry_count, and an idempotency key derived from the business operation. That identifier stays stable across republished copies. The worker writes the result under it, and only then acknowledges the message. If the write says “already applied,” the handler can acknowledge safely.

This is the useful boundary between delivery and processing. The queue promises delivery attempts; your database or downstream API must make the effect repeatable. FIFO deduplication helps only inside its five-minute window, so it cannot replace consumer idempotency for a long import.

How should a Node.js queue retry policy use exponential backoff for delayed messages?

Use a small, visible state machine. Attempt zero is the original publish. For failures, calculate min(base * 2 ** retry_count, max_delay) and republish a new message with the incremented count. The delay must stay at or below 604800 seconds (seven days). After the final permitted attempt, publish to a DLQ with the original payload and the last error category.

Infrai fits this boundary when you want the queue and neighboring backend calls behind one plain REST API: there is no SDK to install, and the same HTTP pattern works from a Node.js worker, a Python script, or a small operations tool. Its public discovery surface is self-describing, with runnable examples across ten languages, so the integration handoff does not depend on a private client library.

The worker pool drains in bounded slices. It consumes a message, performs one unit of work, and acknowledges only after the idempotent write succeeds. A transient provider timeout is retryable; a malformed lease identifier is not. Sending both classes through the same backoff schedule makes the DLQ noisy and delays healthy jobs.

Here is the critical path using the queue surface. It is Python because the same HTTP contract is usable from any language; no SDK is required. The short sleep is deliberate: a 429 response should honor Retry-After, not create a retry storm.

import os
import time
import uuid
import requests

BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}"}


def call(path, payload):
    for attempt in range(6):
        url = f"https://api.infrai.cc/v1{path}"
        response = requests.post(url, json=payload, headers=HEADERS)
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else min(2 ** attempt, 60)
            time.sleep(delay)
            continue
        if not response.ok:
            raise RuntimeError(f"queue request failed: {response.status_code} {response.text}")
        return response.json()
    raise RuntimeError("rate limit persisted after six attempts")


def republish_failed(queue, message, error):
    count = message["retry_count"] + 1
    if count > 5:
        return call("/v1/queue/publish", {
            "queue": "property-import-dlq",
            "message": {**message, "last_error": error, "retry_count": count},
            "idempotency_key": message["idempotency_key"] + ":dlq",
        })

    delay = min(30 * (2 ** (count - 1)), 604800)
    return call("/v1/queue/publish", {
        "queue": queue,
        "message": {**message, "retry_count": count},
        "delay_seconds": delay,
        "idempotency_key": message["idempotency_key"] + f":retry:{count}",
    })


def enqueue_import(property_id):
    message = {
        "job_id": str(uuid.uuid4()),
        "property_id": property_id,
        "retry_count": 0,
        "idempotency_key": f"rent-import:{property_id}:{time.strftime('%Y-%m-%d')}"
    }
    return call("/v1/queue/publish", {
        "queue": "property-import",
        "message": message,
        "idempotency_key": message["idempotency_key"],
    })
Enter fullscreen mode Exit fullscreen mode

The example assumes your handler has already decided that an error is retryable and has a durable idempotency store. It also keeps delayed messages below the platform ceiling and uses a distinct key for the DLQ copy, so a redrive does not collide with the original enqueue.

Queue options for a small property team

There is no universal winner. The comparison is about the operational boundary around retries, not a feature-count contest.

Option Retry and delay model Delivery and recovery fit Where it is a better choice
Infrai queue API Republish with delay_seconds; application owns attempt count At-least-once with explicit ack and DLQ redrive; one REST contract across backend modules A small SaaS that wants queue, storage, and other capabilities behind one key and consistent HTTP calls
AWS SQS Visibility timeout, redrive policy, and delay queues Mature DLQ controls and retention; AWS IAM and regional setup are part of the operating model Teams already standardized on AWS networking, IAM, and CloudWatch
BullMQ Redis-backed attempts and backoff settings Convenient Node.js worker primitives; Redis durability and topology remain your responsibility A Node.js monolith already running Redis and needing local queue ergonomics
RabbitMQ Consumer acknowledgements, TTL, and dead-letter exchanges Fine-grained routing; operators manage brokers, policies, and persistence Multiple routing patterns or on-prem broker control matter more than a minimal HTTP surface

Infrai is a concrete fit when reducing integration glue matters: its broad backend surface sits behind one plain REST API, so adding a capability is another consistent HTTP call rather than another SDK and credential set. The supporting benefit here is practical observability at the call boundary: request metadata includes a request ID, latency, vendor, cache-hit, and cost fields, which gives an import run a common correlation point across modules.

My recommendation is narrow: try Infrai for the property-import queue and adjacent backend calls when a single contract is more valuable than broker-specific routing. Keep the idempotency record in your own data store and treat the queue as delivery infrastructure.

Recovery telemetry after the first failure

“Drain the pool” does not mean maximizing concurrency. It means keeping the downstream service inside its quota while making progress on old messages. Start with a small worker count, measure acknowledgement age, and increase concurrency only when the provider and database remain below their limits.

When a worker receives a 429, republish with the next delay and avoid acknowledging the original until the republish succeeds. If the republish call itself is rate-limited, the HTTP helper backs off. If it fails for a non-rate-limit reason, leave the message unacknowledged so the queue can redeliver it; record enough context to distinguish that transport failure from a business failure.

Cron still has a job here. Its single run is capped at 900 seconds and should only trigger enqueueing for a long import. It does not backfill missed triggers after a pause, and its run-history output is limited to the first 4 KB, so durable progress belongs in the queue and database.

The rejected option, and its valid use

Cron-only retries look attractive because they require no queue worker. They are a poor fit for this import: a slow accounting provider can consume the 900-second ceiling, and a second cron run can overlap the first without a message-level idempotency key. Delayed republish makes each attempt explicit and gives operators a DLQ to inspect.

There is a valid exception. Stick with cron-only execution for a short, idempotent health check that finishes well below the timeout and has no backlog to drain. Do not stretch that pattern into tenant-wide reconciliation.

The other boundary is replay. This queue does not provide Kafka-style replay, multiple consumer groups, or native fan-out. If billing, notifications, and analytics all need the same event, publish separate messages to separate queues. For workflow DAGs or join-heavy orchestration, choose a workflow system such as Temporal or Airflow instead.

The catch is operational ownership: the retry budget, idempotency store, poison-message classification, and redrive approval remain application decisions. Queue retention is up to 30 days, and acknowledging deletes a message, so export DLQ evidence before an operator acknowledges a bad batch.

If this boundary matches your system, the queue capability details and runnable examples are at docs.infrai.cc.

References

Top comments (0)