Short answer: consume cleanup work with a bounded worker, make the operation idempotent, acknowledge only after durable success, and send repeatedly failing messages to a dead-letter queue. The least complex design is usually a long-running worker process, not a web request stretched until a timeout gives up.
For an education platform, a useful message might say that tenant school-42 has 18,240 expired upload records to remove. The web application records that work and returns. A worker reads it later, processes a bounded batch, and leaves a precise trail of what happened. Recovery is the design target: when the worker disappears between deleting records and acknowledging the message, the next delivery must be safe.
That is the whole contract.
How should I consume a cleanup job from a message queue in Node.js?
A queue consumer does not magically turn a distributed operation into exactly-once execution. Treat delivery as at-least-once unless the queue's documented semantics and your application design prove otherwise. A message can be delivered again after a timeout, a process crash, or a lost acknowledgement. That is why the cleanup command needs a stable operation key, such as tenant_id + cleanup_window, and a database record that can say pending, running, completed, or failed.
The acknowledgement boundary matters more than the polling loop. Acknowledge after the database transaction and the external object deletion have reached the state your system considers durable. Acknowledge before that point and a crash can lose work. Acknowledge much later without extending the visibility or lease window and the queue can redeliver work while the first worker is still running. Both outcomes create operational work.
Consider the awkward middle case. The worker receives a cleanup for school-42, marks its operation record as running, deletes one batch of 500 rows, and loses its process before it can acknowledge the message. The queue quite correctly makes the message available again. On the second delivery, the operation record cannot be treated as a reason to skip everything: the first process may have committed only part of the work. The cleanup query must therefore be conditional, the cursor must advance only after a durable batch commit, and the operation state must be recoverable after a restart. A record saying running is not proof that the work finished; it is a prompt to inspect the batch state and continue. This is why I keep the operation key separate from the queue message ID. Redelivery creates a new delivery event, while the operation key identifies the business action that must converge to one result. That small distinction is more useful during an incident than another layer of worker abstraction.
I start capacity planning with the worst useful number, not the average. If one cleanup takes 90 seconds, five workers can complete about 20 such jobs in 10 minutes before overhead; that is a planning estimate, not a throughput guarantee. Set concurrency from database connections, object-store request limits, and the queue's visibility timeout, then test the failure path. A worker count that looks efficient on a dashboard can starve interactive traffic.
The alert-to-action trace
The page should fire when the cleanup backlog threatens its service objective, not when a worker happens to crash. On-call should see the oldest message age, visible backlog, in-flight work, acknowledgement failures, dead-letter count, and the ratio of successful to retried jobs. One alert should name the tenant or workload class when that information is available; “queue unhealthy” is a starting point, not a diagnosis.
Work backward from that page. The queue must expose backlog and age metrics, the worker must emit a job identifier and attempt number, and the cleanup store must record a state transition with a timestamp. Correlate those fields in logs. A trace that starts at the enqueue operation and ends at acknowledgement is useful only if it also shows the database and object-store calls that determine whether acknowledgement was safe.
This is the instrumentation change that catches the important failure earlier: record started_at, finished_at, attempt, and outcome for every operation, then alert on oldest age against the cleanup SLO. For example, a 15-minute objective might page when the oldest pending cleanup exceeds 12 minutes for two consecutive evaluation periods, while a dead-letter message creates a separate, lower-volume operational signal. The exact threshold belongs to the workload's recovery objective.
A low threshold has a price. A short burst after a school import can produce a false page, and repeated false pages train the on-call engineer to ignore the real backlog. A high threshold hides a saturated worker until recovery is already late. I would rather tune the threshold from observed arrival and service rates than copy a round number from another queue.
A consumer that can be restarted
The following worker sketch keeps the queue interface generic. Its important behavior is visible: claim the operation, do bounded work, classify the result, and acknowledge only on success or a deliberate terminal decision. The queue adapter supplies the actual receive, visibility-extension, and acknowledgement semantics.
package main
import (
"context"
"errors"
"log"
"time"
)
type Message struct {
ID string
Body []byte
Attempt int
}
// Queue is intentionally small; its implementation owns provider-specific details.
type Queue interface {
Receive(context.Context) (Message, error)
Ack(context.Context, Message) error
Retry(context.Context, Message, time.Duration) error
DeadLetter(context.Context, Message, string) error
}
func consume(ctx context.Context, q Queue, cleanup func(context.Context, Message) error) {
for ctx.Err() == nil {
msg, err := q.Receive(ctx)
if err != nil {
if ctx.Err() != nil {
return
}
log.Printf("receive failed: %v", err)
time.Sleep(time.Second)
continue
}
err = cleanup(ctx, msg)
switch {
case err == nil:
if ackErr := q.Ack(ctx, msg); ackErr != nil {
log.Printf("ack failed message=%s: %v", msg.ID, ackErr)
}
case errors.Is(err, context.Canceled):
return
case msg.Attempt >= 5:
if dlqErr := q.DeadLetter(ctx, msg, err.Error()); dlqErr != nil {
log.Printf("dead-letter failed message=%s: %v", msg.ID, dlqErr)
}
default:
if retryErr := q.Retry(ctx, msg, 30*time.Second); retryErr != nil {
log.Printf("retry failed message=%s: %v", msg.ID, retryErr)
}
}
}
}
The five-attempt value is an application policy in this example, not a universal setting. A transient object-store timeout may deserve another attempt; a malformed tenant identifier will not improve with repetition. Classify errors explicitly, cap the work per message, and preserve the original reason when moving a message to the dead-letter queue. The dead-letter queue is a recovery surface, not a trash can: operators need enough context to replay or correct the job.
Graceful shutdown is part of the same contract. Stop receiving new messages, let active work finish within a bounded deadline, and allow an unfinished message to become visible again according to the queue's lease rules. Do not report a clean shutdown while a goroutine is still deleting records in the background.
Idempotency is the recovery mechanism
A cleanup operation should be safe when run twice. For database rows, use a condition such as deleted_at IS NULL and record the operation key in a unique table before doing expensive work. For object deletion, treat an already-absent object as the desired end state if the storage contract permits it. The consumer should distinguish “already complete” from “failed halfway through.” Both can be safe to acknowledge, but they deserve different metrics.
Bounded batches make recovery legible. A message that asks a worker to delete 18,240 records in one transaction can hold locks too long and exceed the visibility window. A message that carries a cursor or partition key lets the worker commit smaller units and enqueue the next unit only after the current one is durable. The trade-off is more messages and more state. In an edtech system, that is usually a reasonable price for keeping school-facing requests responsive during term-end cleanup.
Test the transitions, not only the happy path: crash after the database commit and before acknowledgement; timeout while the lease is active; duplicate delivery; poison payload; dead-letter failure; and a restart with half a batch complete. A useful test assertion is that the final dataset is correct and the operation record has one terminal outcome, even if the message was delivered multiple times.
Buy or build the recovery surface
The decision is less about which queue has the nicest client library and more about which team will own recovery at 03:00. A managed queue can reduce the infrastructure work around durability, leases, and dead-letter routing, while a self-hosted broker can offer control over placement and integration at the cost of operating those guarantees. The application still owns idempotency and error classification in either case.
| Concern | Managed queue | Self-hosted broker |
|---|---|---|
| Queue durability and upgrades | Provider-operated boundaries; verify the service contract | Team-operated storage, upgrades, and failure recovery |
| Lease and acknowledgement behavior | Use documented delivery and visibility semantics | Configure and test broker-specific consumer behavior |
| On-call load | More operational work is delegated, but integration alerts remain yours | Broker health joins the application on-call surface |
| Lock-in | API and delivery semantics can shape the worker | More control, with ownership of the whole failure domain |
| Recovery workflow | Build replay, inspection, and access controls around the queue | Build the same workflow plus broker administration |
I would choose the boundary that leaves the smallest unowned recovery gap. A team that cannot staff broker upgrades should count that work as capacity, even when the software itself is available. A team with strict placement requirements should count the ongoing operational burden rather than treating self-hosting as free control.
The decision rule
For periodic edtech cleanup, start with one durable operation record, one bounded worker pool, explicit acknowledgement rules, and a dead-letter path that an operator can inspect. Define the cleanup SLO in terms of oldest work age, then size concurrency against the database and storage limits that actually constrain recovery.
The web request should enqueue intent and return. The worker should make progress that can be repeated. The alert should tell the on-call engineer what is stuck and why. When those three boundaries are clear, the queue becomes a scheduling tool rather than a hidden place where failed work disappears.
Keep the recovery path boring.
Top comments (0)