Short answer: use a scheduler only to create a bounded cleanup run, then let a durable background job queue give each cleanup unit its own retry and dead-letter boundary. For a customer-support system reconciling payment-provider records overnight, this is the simplest arrangement that preserves an audit trail without making one malformed record repeat the entire run. The deciding constraint is latency versus cost: more independent work units reduce the time lost to one failure, but they add queue operations, worker capacity, and operational state.
This is an architecture decision record, not a library recommendation. The names in a queue configuration matter less than the invariants around them.
The decision and its failure boundary
The scheduler owns time. It should create a run identifier, establish the cutoff timestamp, and publish bounded commands. It should not perform every deletion or reconciliation itself. A worker owns one command at a time, records the durable result, and acknowledges the message only after the application transaction commits. A dead-letter queue owns neither business truth nor automatic forgiveness; it is an inspection boundary for work that has exhausted its normal retry policy.
For example, a nightly support job might identify payment records whose reconciliation window has closed and publish commands such as reconcile:2026-08-10:ticket-1842. The command can carry the record identifier, cutoff, attempt policy, and schema version. It should not carry a full customer transcript, a payment instrument, or an access token. Keeping sensitive material out of messages narrows the compliance surface and makes dead-letter retention easier to reason about.
The invariants are more important than the schedule expression:
- Every command has a stable application-level identity.
- Replaying that identity produces the same durable outcome as processing it once.
- The audit record and the cleanup or reconciliation result commit before acknowledgement.
- A retry preserves the original identity and records an attempt, rather than creating a new obligation.
- A dead-letter item contains enough bounded context for triage without becoming a copy of the underlying customer data.
Exactly once is an application outcome, not a delivery guarantee. A queue can deliver a command again after a worker commits but before its acknowledgement reaches the broker. The worker therefore needs an idempotency record, usually keyed by the run identifier plus the cleanup subject. A second delivery can observe the committed result and finish as a successful no-op. A design that acknowledges first has the opposite failure mode: a process exit can make unfinished work disappear.
That distinction is easy to lose in a cost discussion. A single large message may reduce per-message overhead, but it couples every record to the slowest or most defective record. A small message can improve parallelism and reduce the retry blast radius, but it increases queue traffic and database contention. The right unit is the smallest unit for which the business can state a meaningful, idempotent result.
Keep it small.
For this decision, the queue is not a good fit if you need a dependency graph, human approval between steps, or a transaction spanning several external systems. Use a workflow engine or a database-backed process with explicit state in those cases; a pile of retry handlers is not orchestration.
What should a background job queue do for scheduled cleanup retries?
It should make the lifecycle observable: scheduled, published, received, committed, acknowledged, retried, and dead-lettered. Those states let an operator answer two different questions: did the scheduler create the obligation, and did the worker complete it? A run marked complete merely because the scheduler returned successfully is not sufficient evidence.
The retry policy should distinguish transient from permanent failures. A temporary provider timeout, rate limit, or database connection interruption may merit exponential backoff with a cap. An invalid record shape, an expired reference, or a rejected business invariant should not be retried forever. After a bounded number of attempts, the command moves to the dead-letter queue with its stable identity intact. Triage can then correct data, change the command, or explicitly abandon it under the retention policy.
Don't hide the decision in a generic catch block.
Do not make the scheduler the retry database. If the scheduler repeats a whole nightly batch because one worker failed, healthy records pay for the failure and the resulting audit trail cannot clearly separate an omitted run from a repeated item. The queue is useful precisely because it gives the application a smaller failure boundary.
A reconciliation run also needs a completeness record. Persist the expected run and its cutoff before publishing work, then reconcile the set of published and terminal commands against that expectation. Scheduled triggers can be delayed or skipped by their hosting system, and a job log is not automatically a ledger. For payment-adjacent data, the audit record should state what was compared, which cutoff applied, which source identifiers were involved, and who or what initiated the redrive. In a long overnight run, this record is what lets an on-call engineer distinguish an empty result from a missing page of work, a distinction that is otherwise easy to erase when the scheduler reports only process success.
The security boundary belongs here too. If a cleanup worker accepts a URL from a record and fetches it, validate the destination against an explicit allowlist and follow SSRF guidance; a retry mechanism must not turn an attacker-controlled URL into repeated internal network access. The OWASP guidance is especially relevant when dead-letter correction involves re-reading stored data rather than merely replaying a fixed command.
I am not sure a queue is necessary for every scheduled cleanup. If one short transaction can safely repeat the entire run and no individual item needs separate recovery, a direct scheduled process is cheaper to operate. Your mileage may vary with volume and provider latency. The resolving test is whether one item can fail while its neighbors should still complete. If yes, the batch is already the wrong retry boundary.
Comparing the practical setup choices
| Setup | Latency profile | Cost and ownership | Failure boundary | Use it when |
|---|---|---|---|---|
| One scheduled process | Low startup overhead; latency grows with the whole batch | Few moving parts; the process owns retries and state | Entire run or manually coded sub-batch | Cleanup is short, repeatable, and whole-run retry is acceptable |
| Scheduler plus durable queue | Work can run in parallel; queue and worker capacity must be tuned | Queue and worker operations add cost; state is explicit | Individual command | A poison record must not delay healthy records |
| Database-backed work table | Often predictable for modest volume; polling adds delay | Uses existing database, but polling and locking are application work | Claimed row or batch | The team needs durable claims and already operates a relational database |
| Workflow engine | Suitable for long, dependent sequences; orchestration adds scheduling latency | Highest control-plane complexity | Activity or workflow step | The cleanup has joins, compensation, timers, or human approval |
The middle option is the default decision in this ADR because it balances a bounded nightly reconciliation with per-record recovery. It is not automatically the lowest-cost option. At low volume, a work table or direct process may be preferable; at high volume, queue fan-out can expose database hot spots that require partitioning and worker limits.
Measure the axis that actually matters. Record time from the scheduled cutoff to the last committed command, not merely worker runtime. Record queue depth, retry age, dead-letter age, duplicate deliveries, and database lock time. A cheaper queue that leaves the support team waiting until morning is not cheaper in the operational sense; a faster design that creates unreviewed duplicate reconciliation events is not acceptable either.
The critical path in Go
The code below shows the ordering, using generic interfaces so the business transaction remains visible. The queue is at-least-once in this model, so the idempotency store is part of the application contract. Commit must durably include the cleanup result and audit entry before Ack runs.
package main
import (
"context"
"errors"
"fmt"
"time"
)
type Command struct {
ID string
Subject string
Cutoff time.Time
}
type Queue interface {
Receive(context.Context) (Command, error)
Ack(context.Context, string) error
Retry(context.Context, string, time.Duration) error
DeadLetter(context.Context, string, string) error
}
type Store interface {
Begin(context.Context) (Tx, error)
}
type Tx interface {
AlreadyCommitted(string) (bool, error)
ApplyCleanup(context.Context, Command) error
AppendAudit(context.Context, Command, string) error
Commit() error
Rollback() error
}
func process(ctx context.Context, queue Queue, store Store, cmd Command) error {
tx, err := store.Begin(ctx)
if err != nil {
return err
}
committed, err := tx.AlreadyCommitted(cmd.ID)
if err != nil {
_ = tx.Rollback()
return err
}
if committed {
_ = tx.Rollback()
return queue.Ack(ctx, cmd.ID)
}
if err := tx.ApplyCleanup(ctx, cmd); err != nil {
_ = tx.Rollback()
if errors.Is(err, ErrPermanent) {
return queue.DeadLetter(ctx, cmd.ID, "permanent validation failure")
}
return queue.Retry(ctx, cmd.ID, 30*time.Second)
}
if err := tx.AppendAudit(ctx, cmd, "committed"); err != nil {
_ = tx.Rollback()
return queue.Retry(ctx, cmd.ID, 30*time.Second)
}
if err := tx.Commit(); err != nil {
return err
}
return queue.Ack(ctx, cmd.ID)
}
var ErrPermanent = errors.New("permanent command failure")
func main() {
fmt.Println("worker wiring is application-specific")
}
One subtle case deserves emphasis. Suppose Commit succeeds and the process exits before Ack. The command is delivered again; AlreadyCommitted prevents another side effect, and the second acknowledgement closes delivery. Suppose the acknowledgement succeeds first and the process exits before Commit. The queue has lost the obligation. The order is therefore a correctness rule, not a style preference.
In production, make the retry decision from typed error classes, cap total age as well as attempt count, and alert on dead-letter age rather than only dead-letter count. A command that fails once at 02:00 and remains unreviewed at 09:00 is a different operational risk from one that is retried successfully at 02:01.
The rejected simple setup, and when it is right
The rejected design is one scheduled Node.js process that scans all eligible rows, performs cleanup inline, and retries the complete scan after an error. It has a valid use case: a small dataset, a short runtime, no per-record intervention, and a transaction whose repeat behavior is demonstrably idempotent. In that case, introducing a queue can create more state and more monitoring than the problem warrants.
That is the boundary.
It becomes unsuitable when support records have different failure causes, when the payment provider imposes variable latency, when one malformed record can poison a batch, or when the team must redrive one item with an auditable reason. The queue setup is justified by that failure boundary, not by fashion. Keep the direct process until the evidence says the boundary must become finer.
Top comments (0)