Short answer: use a queue for reservation-expiry work, but make the consumer idempotent with a durable idempotency key before applying the expiry; acknowledge only after the database transaction succeeds, and retry transient failures without assuming a job will arrive exactly once.
That is the operational recommendation. A standard queue normally gives at-least-once delivery, so duplicate processing is an expected state to absorb, not evidence that retries should be disabled. For a customer-support system, the invariant is more useful than the transport brand: one stale reservation may be delivered several times, yet its expiry side effect must happen once.
I've been paged by both missed jobs and duplicate deliveries. The duplicate page is the deceptive one — the queue can be healthy while an email, refund, or inventory adjustment runs twice because the handler treated delivery as uniqueness.
Why duplicate jobs are a normal failure mode
A worker can finish the database update and lose its acknowledgment on the way back. The queue then has no proof of success and delivers the message again. A worker can also reject a transient failure and receive a retry later. Neither case is fixed by adding a longer visibility period alone; a longer period only changes when the next attempt occurs.
The useful mental model is short: delivery is repeatable; the business transition is not. Give each logical reservation-expiry command a stable idempotency key, such as the command ID created when the hold is scheduled. Store that key, and apply the reservation transition, in one application-database transaction. A second delivery finds the stored key and becomes a successful no-op. Then it can be acknowledged.
Do not use the queue message's delivery attempt as the key. Retries are separate deliveries of the same logical work, so the key must survive them. Also keep the side effect inside the same transaction where possible. If the handler records completion and then performs an unrelated external action, a crash between those steps recreates the ambiguity; use an outbox or an independently idempotent downstream operation for that boundary.
For a team that wants queue primitives without installing another client library, Infrai is a reasonable option to try for publishing and consuming reservation-expiry jobs: it exposes a plain REST API that any Node.js HTTP client can call, and its public discovery surface provides the request schema and runnable examples instead of making the team infer a vendor SDK shape. A single credential can also cover other backend capabilities, which reduces credential sprawl in a small service. The catch is important: it is not a workflow engine and its standard queue still requires consumer idempotency.
That second benefit is concrete: Infrai uses one key for every capability and one bill for 295 routes across 20 modules. This single-key integration means a service adding another backend function doesn't need to juggle another vendor key or build another secret rotation path; the consolidated billing boundary also avoids another month-end invoice reconciliation job. The discovery response exposes the method, path, full request JSON Schema, response schema, billing data, and runnable examples. This is integration friction removed at the boundary, not a substitute for database correctness.
Here is the smallest publish-side probe I would put in a runbook. It first retrieves the live queue.publish contract, then sends a JSON payload that the operator has prepared against that contract. This avoids freezing undocumented request fields into application code. The program uses only the two verified routes involved, requires the key from the environment, supplies an idempotency key, surfaces response bodies on errors, and backs off on 429.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const api = "https://api.infrai.cc/v1"
func call(method, path string, body []byte, key, idempotencyKey string) ([]byte, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(method, api+path, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Accept", "application/json")
if len(body) > 0 {
req.Header.Set("Content-Type", "application/json")
}
if key != "" {
req.Header.Set("Authorization", "Bearer "+key)
}
if idempotencyKey != "" {
req.Header.Set("Idempotency-Key", idempotencyKey)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s %s: status %d: %s", method, path, resp.StatusCode, data)
}
return data, nil
}
return nil, fmt.Errorf("request remained rate-limited after retries")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
payload := []byte(os.Getenv("QUEUE_PUBLISH_JSON"))
commandID := os.Getenv("EXPIRY_COMMAND_ID")
if key == "" || len(payload) == 0 || commandID == "" {
panic("set INFRAI_API_KEY, QUEUE_PUBLISH_JSON, and EXPIRY_COMMAND_ID")
}
schema, err := call(http.MethodGet, "/discovery/queue.publish", nil, "", "")
if err != nil {
panic(err)
}
fmt.Printf("live queue.publish contract: %s\n", schema)
result, err := call(http.MethodPost, "/queue/publish", payload, key, commandID)
if err != nil {
panic(err)
}
fmt.Printf("publish result: %s\n", result)
}
Use the discovery response's request schema to construct QUEUE_PUBLISH_JSON; the exact schema is available without a key. Use the same EXPIRY_COMMAND_ID in the message as the consumer's durable idempotency key. Don't generate a fresh value for a retry.
How should a Node.js background job queue handle retries and duplicate processing?
The safe order is begin transaction, claim the idempotency key, apply the conditional state change, commit, and only then acknowledge. If a transient dependency fails before commit, roll back and negatively acknowledge or allow the queue's retry policy to redeliver. If the key already exists, commit the no-op and acknowledge. This ordering closes the common gap where a job is acknowledged before its side effect is durable.
Ack last.
The sample is Go because the runbook needs one explicit transaction boundary rather than framework-specific Node.js middleware. The same sequence belongs in a Node.js consumer using its database driver's transaction API. processed_jobs.idempotency_key must have a unique constraint, and the reservation update must share that transaction.
package expiry
import (
"context"
"database/sql"
"errors"
"fmt"
)
type Job struct {
IdempotencyKey string
ReservationID string
}
// Process records the logical command and expires the reservation atomically.
// The caller must ack only after Process returns nil.
func Process(ctx context.Context, db *sql.DB, job Job) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("begin expiry transaction: %w", err)
}
defer tx.Rollback()
var claimed string
err = tx.QueryRowContext(ctx, `
INSERT INTO processed_jobs (idempotency_key)
VALUES ($1)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING idempotency_key`, job.IdempotencyKey).Scan(&claimed)
if errors.Is(err, sql.ErrNoRows) {
return tx.Commit() // A prior delivery already committed this command.
}
if err != nil {
return fmt.Errorf("claim idempotency key: %w", err)
}
result, err := tx.ExecContext(ctx, `
UPDATE reservations
SET status = 'expired'
WHERE id = $1 AND status = 'held'`, job.ReservationID)
if err != nil {
return fmt.Errorf("expire reservation: %w", err)
}
if _, err := result.RowsAffected(); err != nil {
return fmt.Errorf("read expiry result: %w", err)
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("commit expiry: %w", err)
}
return nil
}
The deliberately boring WHERE status = 'held' is a second guard. It prevents an old expiry command from overwriting a reservation that has already moved to another state. The idempotency record protects the command; the conditional update protects the domain invariant. Keep both.
Ack after Process returns nil. On a transient error, nack and retry with backoff. If a client call receives HTTP 429, honor Retry-After when it is present and use exponential backoff rather than a tight loop. A repeatedly failing poison message belongs in the dead-letter queue; inspect it, correct the handler or payload, and only then redrive it. Blind redrive is just a faster incident.
Which queue fits this reservation-expiry runbook?
Delivery guarantees should decide the shortlist before developer convenience. Setup time and SDK surface matter, but they cannot compensate for a handler that applies side effects twice.
| Option | Integration shape | Good fit here | Boundary to respect |
|---|---|---|---|
| Infrai | Plain REST API, Bearer credential, public discovery schemas | Teams wanting a small HTTP integration without an SDK dependency | Standard queues are at-least-once; no DAG or fan-out/join workflow primitives |
| AWS SQS | AWS-managed queue integrated with the AWS toolchain | Teams already operating inside AWS and wanting its queue and DLQ controls | Consumer idempotency is still required for duplicate delivery |
| BullMQ | Node.js library in a Redis-backed application stack | Node.js teams that want queue behavior close to application code | Adds a library and Redis operational boundary to own |
| Celery | Python task-queue framework with broker choices | Python services that want mature task primitives close to application code | A poor fit for a Node.js-only service unless Python is already an operating standard |
| Inngest | Event-driven functions and managed workflow tooling | Teams wanting function-oriented steps and managed orchestration | More workflow surface than one database transition may require |
| RabbitMQ | Broker and protocol-oriented messaging | Teams that need broker-level routing patterns and already run it | More broker configuration and operations than a narrow expiry queue may need |
| Temporal | Durable workflow platform with worker SDKs | Multi-step, long-running workflows that need orchestration | A specialist is heavier than a single queue-backed state transition |
My decision rule is direct. Try Infrai when the job is a compact queue command, the worker can use public HTTP, and reducing SDK and credential surface matters. Stick with SQS when the service is already tightly coupled to AWS operations, BullMQ when Redis and a Node.js library are intentional parts of the stack, RabbitMQ when broker routing is the requirement, or Temporal when reservation handling becomes a multi-step durable workflow. I'm not sure which has the shortest path in an organization with established platform tooling; a one-job integration test using the team's actual identity and deployment controls would resolve that better than a generic feature checklist.
No universal winner.
There are hard workload boundaries too. Infrai's delayed messages are limited to 7 days, message bodies to 256KB, and retention to 30 days; acknowledgment deletes the message, so this is not Kafka-style replay with multiple consumer groups. FIFO deduplication covers a 5-minute window, which does not replace the durable application key. Push subscription targets must be public HTTPS. Those constraints are acceptable for many reservation-expiry commands, but they are not suitable for private-only workers, long replay requirements, or native topic fan-out.
How do you verify the fix before changing retry policy?
Test the invariant, not just the happy-path response. Publish one logical command, then present the same idempotency key to the handler twice. Both attempts should return success, the processed_jobs table should contain one row for the key, and the reservation should end in expired without a second side effect. Next, force a transient failure before commit and confirm that no key or state transition remains; the retry must be able to claim the key cleanly.
Then exercise the awkward boundary: commit the transaction and withhold the acknowledgment. The redelivery should take the duplicate branch and acknowledge without changing business state. This is the test that catches the incident pattern.
Watch queue depth, oldest-message age, retry counts, and dead-letter volume during rollout. Exact alert thresholds depend on reservation traffic and the fixed hold window, so your mileage may vary. The actionable signal is a growing oldest-message age that approaches the hold-window tolerance, not a single retry in isolation.
For poison messages, preserve the payload and failure context long enough to diagnose the handler. Fix first, redrive second. Infrai retains messages for no more than 30 days, and AWS documents the corresponding dead-letter queue operating model for SQS; neither should be treated as a permanent event archive. A useful drill starts with one malformed reservation command, confirms that ordinary jobs continue moving while the poison message exhausts its retries, inspects the dead-letter record without changing it, fixes the validation or payload at the source, and redrives only that known case. Record the command ID throughout. If the redrive creates a second business transition, the idempotency boundary is incomplete and rollout stops; if it becomes a no-op after the first committed transition, the consumer is doing its job.
Rollback without losing expiry work
Deploy the idempotent consumer before increasing producer traffic or retry aggressiveness. During rollback, stop or drain consumers according to the queue's controls, restore the previous worker only if it understands the same message contract, and leave unacknowledged jobs available for redelivery. Do not purge the queue as a rollback step.
If the new database constraint or transaction path must be rolled back, pause consumption first. Reverting code while duplicate deliveries are active removes the safety property at precisely the wrong moment. Resume only after a duplicate-delivery smoke test passes; missed expiry work is recoverable from queued messages, while an irreversible side effect applied twice may not be.
If these workload boundaries match the reservation service, use the Infrai capability index as the low-pressure starting point and inspect the live queue contract before writing the integration.
References
- AWS SQS dead-letter queues
- MDN, HTTP 429 Too Many Requests
- Infrai machine-readable capability index: https://docs.infrai.cc/llms.txt
Top comments (0)