Short answer: enqueue one idempotent cleanup job per property record, let a background worker enforce the downstream rate limit, and acknowledge each job only after its result is durable.
This is the decision rule I use for periodic property-management cleanup: expired lease records, stale import rows, or email work should be a collection of independently retryable jobs. A batch is a transport optimization. It is not a transaction.
I've been paged by missed jobs and duplicate deliveries. The useful lesson is not a preferred queue product. It is that the producer owns admission, while the worker owns correctness. Those are different failure domains.
What does a Node.js worker queue need for batch background processing?
Start with a stable identity. For a property cleanup, use the cleanup run ID plus the property ID and operation name. Store that identity beside the business result, protected by a uniqueness constraint. A retry on another worker should discover the completed result and become a no-op before repeating the external side effect.
Keep the batch boundary on the enqueue request. The worker should still see one logical item per property, email, or import row. If record 17 is rate limited, records 1 through 16 should not be replayed just because they were published in the same request. This also gives an operator a useful answer to “what happened?”: each item has its own attempt, result, and acknowledgement state.
The queue does not know the quota of a property-management API, mail provider, or import endpoint. A worker must bound both concurrency and request starts. Read Retry-After when the dependency supplies it; otherwise use exponential backoff with jitter. A 429 is backpressure, not a reason to tight-loop.
Ack last.
The normal path is: claim the stable identity, perform the side effect, commit the result, then acknowledge the message. If the process exits between the side effect and the acknowledgement, delivery may happen again. That is expected. The durable identity is what makes the second delivery safe.
Why do batch jobs duplicate work during cleanup?
There are two common boundaries, and confusing them creates the incident. The first is publish acceptance: did the queue receive the work? The second is business completion: did the external system accept the cleanup? A successful publish says nothing about the second answer.
Consider a run that contains 200 stale property records. The producer times out after the service has accepted the request, so the scheduler cannot tell whether it should retry. If it retries the whole batch, some records now have two messages before any worker has started. Later, a worker completes 87 records and is terminated before acknowledging them. Those 87 can be delivered again while the remaining 113 are still waiting. A second worker may then see a property whose lease record was already cleaned, an email that was already submitted, or an import row that was already updated. Without a durable idempotency key, the cleanup repeats and can trigger another downstream quota response; with one, the second attempt can record “already complete” and acknowledge safely. During review, each transition should answer a separate question: was the item published, claimed, side-effected, marked complete, and acknowledged? A batch-level status cannot provide that accounting.
This is why a batch-level completion flag is weak evidence. It cannot distinguish a lost response, an accepted publish, a failed downstream call, and an acknowledgement that never reached the queue. Per-item state can.
The same rule applies to emails and imports, though the side effect differs. An import might update a row; an email might create a provider delivery. The stable key and the “result before ack” ordering remain the invariant. I’m not sure what quota the downstream service assigns, so the current service documentation and response headers should determine the limiter rather than a guessed constant.
A Go worker path that makes retries boring
The following queue-neutral example keeps the important control flow visible. The memory store is only a compact demonstration; a deployed worker needs a database-backed record with a unique key and a clear state transition.
package main
import (
"context"
"errors"
"fmt"
"sync"
"time"
)
var errRateLimited = errors.New("downstream returned 429")
type Job struct {
ID string
Attempt int
}
type Store struct {
mu sync.Mutex
done map[string]bool
}
func (s *Store) RunOnce(job Job, sideEffect func() error) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.done[job.ID] {
return nil
}
if err := sideEffect(); err != nil {
return err
}
s.done[job.ID] = true
return nil
}
func process(ctx context.Context, jobs <-chan Job, store *Store) error {
limiter := time.NewTicker(200 * time.Millisecond)
defer limiter.Stop()
for job := range jobs {
select {
case <-ctx.Done():
return ctx.Err()
case <-limiter.C:
}
err := store.RunOnce(job, func() error {
fmt.Printf("clean property record %s\n", job.ID)
return nil
})
if errors.Is(err, errRateLimited) {
delay := time.Second << min(job.Attempt, 5)
fmt.Printf("retry %s after %s\n", job.ID, delay)
continue // Leave the item unacknowledged for the queue's retry path.
}
if err != nil {
return fmt.Errorf("process %s: %w", job.ID, err)
}
fmt.Printf("ack %s\n", job.ID)
}
return nil
}
func main() {
jobs := make(chan Job, 2)
jobs <- Job{ID: "cleanup-run-42-property-17"}
jobs <- Job{ID: "cleanup-run-42-property-17"}
close(jobs)
store := &Store{done: make(map[string]bool)}
if err := process(context.Background(), jobs, store); err != nil {
panic(err)
}
}
The repeated property ID is deliberate. The second delivery reaches the identity check and does not repeat the simulated cleanup. In production, do not hold a database lock across a slow network call as this small example does; claim work with a state machine, make the external request idempotent where possible, and make the completion write conditional. The exact transaction depends on the database and side effect, but the contract should be tested under a crash between each pair of steps.
A rate limiter should control starts, and a separate concurrency limit should control in-flight calls. They solve different overload problems. Add metrics for queue age, attempt count, rate-limit responses, successful no-ops, and time from side effect to acknowledgement. A dashboard that reports only published count can look healthy while cleanup is falling behind.
For a small team, this checklist is enough to make the contract testable:
| Check | Passing behavior |
|---|---|
| Duplicate delivery | The second job ID produces no second side effect |
| Worker crash | The item remains retryable until its durable result exists |
| Rate limit | The worker delays or requeues instead of acknowledging success |
| Publish timeout | A retry does not create a second business effect |
Which scheduling boundary fits the cleanup?
Use a scheduler to create bounded work, not to perform the entire cleanup inside a web request. A scheduled HTTP trigger should hand off a run ID and a cursor; workers then process records independently. This keeps request time separate from the duration of the import or cleanup and gives retries an explicit place to live.
The catch is that this pattern is not suitable when all records must commit atomically. Independently acknowledged jobs provide isolation, not a distributed transaction. Use a transactional design or an orchestrator with compensating actions when partial completion is unacceptable. Increasing the batch size cannot create atomicity.
It is also a poor fit for exact wall-clock guarantees, indefinite replay, or a workflow with many joins and dependencies. Choose a scheduler and queue whose documented semantics match those requirements. For a periodic cleanup with bounded records, at-least-once delivery plus durable idempotency is usually the simpler contract to operate.
The runbook I would hand to on-call
Before enabling a new cleanup, verify that the job identity is deterministic, unique, and present in logs. Test a publish timeout, a worker restart after the side effect, a downstream 429, and a duplicate delivery. The expected result is one business effect, a later retry, and an acknowledgement only after the durable result exists.
Keep the batch size independent from worker concurrency. Tune the former for publish overhead and the latter for dependency capacity. Alert on oldest queue age and repeated attempts, then inspect downstream response headers before raising throughput. Don't use a larger batch to hide a slow consumer.
The recommendation changes when the business operation needs all-or-nothing behavior, a long-lived workflow, or replayable event history. Stick with a workflow engine or event-stream design in those cases. For ordinary property cleanup, the boring rule wins: batch the transport, isolate each job, throttle at the worker, persist idempotency, and ack last.
Top comments (0)