Active customer digest delivery should use bounded message retries around an idempotent cleanup operation, with the retry decision made from the failure class rather than from a fixed attempt count.
TL;DR: For a weekly e-commerce digest, enqueue one durable work item per customer and give it a stable operation key. The worker may retry transient delivery failures, but the cleanup record must be claimed and completed atomically. A retry that sends twice is a data-integrity bug, not a scheduling feature.
The useful design has three gates: eligibility, delivery, and cleanup. Each gate records state; each state transition can be resumed. Before copying the pattern, measure duplicate sends, stale jobs, retry latency, dead-letter volume, and the fraction of failures that are genuinely transient. Those numbers decide the bounds.
Why is weekly cleanup harder than a cron expression?
A cron entry can express a time. It cannot express ownership of a message after a worker has accepted it, or tell you if a timeout happened before or after the remote mail service committed the send. That gap is where scheduled cleanup becomes a distributed-systems problem.
For this scenario, the source table contains customers who opted into a weekly digest, plus generated rows that should be removed after the digest window closes. The scheduler creates a run for the period, then creates one work item per active customer. The work item contains a customer identifier, period identifier, and operation key such as digest:2026-W42:customer-1842. It does not contain a mutable attempt number as its identity.
I initially treated the scheduler and worker as one concern. That made the happy path look clean: run a scheduled function, loop over customers, send, delete. The failure path was the real design. A process can disappear after the provider accepts a request, and a queue can deliver the same message again after a visibility timeout. The loop has no honest way to distinguish those cases.
The scheduler should therefore do a small amount of work: create the run, enqueue durable items, and record that enqueueing happened. The queued task owns retries. Cleanup is a separate state transition after the send result is known. This separation also gives an evaluation harness three observable boundaries instead of one opaque function.
The queue is not the ledger.
How should I configure queued cleanup tasks when message retries repeat failed executions?
The three gates
At the eligibility gate, select only active customers for the requested period and freeze the selection. A customer becoming inactive halfway through the run should not silently change the meaning of an already-created digest. Store the run identifier and selection timestamp.
At the delivery gate, classify results. A connection reset, a temporary overload response, or a transport timeout can be retried with exponential backoff and jitter. Invalid recipient data, an explicit permanent rejection, and a schema error should be recorded for inspection and should not spin forever. The worker must acknowledge a message only after it has recorded a terminal result or a retry schedule.
At the cleanup gate, delete or archive only rows linked to the same operation key and only after delivery has reached the state your business rule accepts. A cleanup query without that key is too broad. A delete performed before delivery is confirmed makes recovery depend on reconstruction.
Here is a compact Python shape for the decision boundary. The queue and mail client are deliberately generic; the important part is the transaction and the stable key.
from dataclasses import dataclass
from enum import Enum
class Outcome(Enum):
SENT = "sent"
RETRY = "retry"
PERMANENT_FAILURE = "permanent_failure"
@dataclass(frozen=True)
class DigestJob:
customer_id: str
period: str
operation_key: str
def handle(job: DigestJob, store, mailer) -> Outcome:
with store.transaction():
state = store.get_delivery(job.operation_key)
if state in {"sent", "cleaned"}:
return Outcome.SENT
store.claim_delivery(job.operation_key, job.customer_id, job.period)
try:
mailer.send_digest(job.customer_id, job.period, idempotency_key=job.operation_key)
except TemporaryDeliveryError:
store.record_retry(job.operation_key)
return Outcome.RETRY
except PermanentDeliveryError:
store.record_failure(job.operation_key)
return Outcome.PERMANENT_FAILURE
with store.transaction():
store.mark_sent(job.operation_key)
store.remove_generated_rows(job.customer_id, job.period, job.operation_key)
return Outcome.SENT
The external send still needs a duplicate-control contract. A local database lock cannot undo a remote side effect. If the delivery endpoint does not support an idempotency key, use an outbox or delivery ledger and define what a timeout means operationally. In some systems, the honest result is unknown, followed by reconciliation, rather than an automatic resend.
That distinction changes the retry policy.
Consider the awkward boundary in detail. The worker sends the digest, receives no response within its timeout, and loses its lease a moment later. The remote service may have accepted the request, while the local delivery row still says claimed. Retrying immediately can produce two messages; refusing to retry can leave the customer without a digest. The durable answer is to preserve the same operation key, mark the result as unknown, and let reconciliation ask the delivery ledger or provider for a definitive result before cleanup. If no such query exists, the system should expose the ambiguity to an operator and keep the generated rows until a bounded retention rule resolves it. This is slower than a blind resend, but it makes the risk explicit and keeps the cleanup decision reversible.
What should a retry actually repeat?
A retry should repeat a bounded attempt, not the entire weekly run. The message payload should be immutable enough that a second worker sees the same customer, period, and operation key. The attempt metadata belongs to the queue or delivery ledger. Configure the task so that its execution identity remains stable while its attempt count changes; otherwise the retry can look like a new cleanup request and bypass the duplicate check.
Use a retry schedule with a maximum age and a maximum count. The exact values are workload decisions, not universal constants. A weekly digest may tolerate minutes of delay for a transient transport failure, while a stale cleanup job from a previous period may have no business value after reconciliation.
Jitter matters because thousands of customers can fail together after a regional network event. Without it, all messages wake at the same backoff boundary and recreate the load spike. Keep the delay calculation testable and cap it.
import random
def retry_delay(attempt: int, base_seconds: float = 30.0, cap_seconds: float = 1800.0) -> float:
exponential = min(cap_seconds, base_seconds * (2 ** max(0, attempt - 1)))
return random.uniform(exponential * 0.5, exponential)
The dead-letter path is part of the feature. Include the operation key, customer id, period, last error class, attempt count, and timestamps. Do not put private digest contents into a broadly visible queue log. A replay tool should require an explicit period and a bounded set of keys; replaying every failed message is how an incident becomes a second incident.
A small evaluation harness catches the expensive mistakes
I prefer testing this as a state machine before connecting it to a real scheduler. The cases that matter are awkward: the worker crashes immediately after the send returns; the same message arrives twice; the cleanup transaction commits and the acknowledgement is lost; the remote endpoint times out after accepting the request.
For each case, assert an invariant rather than a particular call count. There must be at most one completed cleanup for an operation key. There must be a visible terminal or retry state. A timeout must not be silently converted into not sent. These checks are more useful than a happy-path test that only asserts one call to send_digest.
def test_duplicate_delivery_does_not_clean_twice(fake_store, fake_mailer):
job = DigestJob(
customer_id="customer-1842",
period="2026-W42",
operation_key="digest:2026-W42:customer-1842",
)
assert handle(job, fake_store, fake_mailer) is Outcome.SENT
assert handle(job, fake_store, fake_mailer) is Outcome.SENT
assert fake_mailer.send_count(job.operation_key) == 1
assert fake_store.cleanup_count(job.operation_key) == 1
Run the harness with fault injection at each boundary. Record the number of sends, cleanup attempts, retry delays, and unresolved operations. I would also run a small load test with a fixed customer set and a fixed failure rate, then vary the visibility timeout and worker concurrency independently. That exposes queue settings that look harmless at low volume.
Choosing the operating boundary
A host scheduler is adequate when the job can be rerun safely, the run is short, and a separate durable ledger handles work ownership. A queue-backed worker is the better boundary when per-customer work can outlive the scheduler invocation or needs independent backoff. A workflow engine becomes useful when the process contains several durable waits and human or service-level decisions. The choice follows state and recovery needs, not the label attached to the scheduler.
Keep observability tied to the operation key. Useful metrics include eligible customers per run, enqueue-to-claim delay, retry rate by error class, age of the oldest pending item, duplicate-send detections, and cleanup lag. Logs should make one customer-period traceable without making customer content the log payload.
Do not hide policy inside a generic retry=True setting. Write down which failures are retryable, how long a message remains useful, what an unknown delivery result triggers, and who can replay a dead-letter item. Those rules are the real configuration.
The final decision is simple: schedule the creation of bounded work, retry only failures that can plausibly recover, and make both delivery and cleanup idempotent. Then test the crash points that the happy path avoids. A weekly digest is reliable when a duplicate message is boring and a timeout is visible.
Top comments (0)