Short answer: for a marketplace's weekly digest, let the scheduler identify due active customers, publish small queue messages, and let separate email and SMS workers enforce concurrency and provider pacing. A batch publisher should create work quickly; it should not wait for the whole delivery backlog. This keeps latency measurable without turning a provider limit into a duplicate-send incident.
I've been paged by missed jobs and duplicate deliveries. The uncomfortable part is that both can come from the same rushed fix: a retry that publishes again, or a worker pool enlarged until the queue looks healthy while the downstream channel starts refusing work. The useful boundary is simple: claim once, publish once per logical digest, and acknowledge only after delivery is accepted.
The failure was in the handoff, not the cron expression
Imagine Monday's digest run for active marketplace customers. The scheduler finds records whose due time has passed and claims a bounded page. Each record becomes a message with a stable digest ID, customer ID, channel, and template ID. The publisher returns. Email and SMS workers then take over. The important detail is what happens when the boundary is unclear: a database claim can succeed while the publish request times out, or the queue can accept a message while the publisher sees no response. In the first case, a record stuck in claimed must be visible to reconciliation; in the second, the retry must carry the same digest ID so a consumer can suppress the duplicate. A worker that receives the same message twice should check delivery state before sending, and a provider response such as HTTP 429 should affect pacing rather than mark the customer as permanently failed. I keep those transitions in separate fields because an on-call engineer needs to answer three different questions quickly: did we select the customer, did we enqueue the work, and did the channel accept it? Combining them into one sent boolean hides the recovery path and turns a routine timeout into an argument about whether to run the whole batch again.
That sequence gives operators three separate things to inspect: what the database claimed, what the queue accepted, and what the channel provider accepted. Without those states, a slow provider looks like a scheduler failure and a publisher retry looks like a customer decision to receive the digest twice.
The queue message should be a pointer, not the entire rendered message. Keep the delivery state and audit detail in the application store. The consumer can load the current template and customer preferences, but it must retain the same logical ID when it retries. A duplicate delivery is an input to the design.
Short queue. Long patience.
For the marketplace scenario, the main decision axis is latency versus cost. A large worker pool can reduce queue age until the provider's quota becomes the actual bottleneck. More workers then add contention, retry traffic, and operational noise without improving customer-visible completion time. A small pool with a channel-specific pacer may take longer, but its behavior is easier to explain during an on-call shift.
How should a Node.js queue batch worker pace email and SMS reminders?
Even if the application is written in Node.js, the control loop has four independent controls: batch size, worker concurrency, request pacing, and retry delay. Do not collapse them into one concurrency setting. Concurrency limits simultaneous requests; it does not represent a per-second, per-minute, per-recipient, or account-wide provider limit.
Use separate lanes when email and SMS have different quotas, credentials, payloads, or retry policies. A channel worker should acquire its pacing slot before making a request. On a rate-limit response, honor the provider's Retry-After value when one is supplied. Otherwise, exponential backoff gives repeated attempts room to spread out instead of forming a synchronized retry wave. The backoff idea is also useful for transient network failures, but permanent validation failures should go to a reviewable failure state rather than retry forever.
The publisher needs the same discipline. A scheduled run should claim only a bounded page of due records, with an overlap-safe state transition. If the run is interrupted after claiming but before publishing, the record needs a recoverable state and a later reconciliation pass. If the publish call times out after the queue accepted the message, retrying without a stable deduplication key can create a second digest. Exactly-once delivery is not a reasonable assumption for this path.
Here is the worker-shaped part of that contract. It is Go because the important behavior is the protocol, not a framework-specific queue API. The same state transitions fit a Node.js worker.
package main
import (
"context"
"fmt"
"net/http"
"strconv"
"sync"
"time"
)
type Digest struct {
ID string
Customer string
Channel string
}
func send(ctx context.Context, client *http.Client, endpoint string, d Digest) error {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, nil)
if err != nil {
return err
}
req.Header.Set("Idempotency-Key", d.ID)
resp, err := client.Do(req)
if err != nil {
return err
}
resp.Body.Close()
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("delivery status %d", resp.StatusCode)
}
delay := time.Duration(1<<attempt) * 200 * time.Millisecond
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
case <-ctx.Done():
return ctx.Err()
}
}
return fmt.Errorf("delivery retry budget exhausted")
}
func run(ctx context.Context, jobs <-chan Digest, endpoint string, concurrency int) {
var workers sync.WaitGroup
for i := 0; i < concurrency; i++ {
workers.Add(1)
go func() {
defer workers.Done()
pacer := time.NewTicker(250 * time.Millisecond)
defer pacer.Stop()
for d := range jobs {
<-pacer.C
if err := send(ctx, http.DefaultClient, endpoint, d); err != nil {
// Leave the queue message eligible for its retry policy.
fmt.Printf("nack %s: %v\n", d.ID, err)
continue
}
fmt.Printf("ack %s\n", d.ID)
}
}()
}
workers.Wait()
}
func main() {
jobs := make(chan Digest)
go func() {
defer close(jobs)
jobs <- Digest{ID: "digest-2026-08-10-customer-17", Customer: "customer-17", Channel: "email"}
}()
run(context.Background(), jobs, "https://provider.invalid/send", 2)
}
The example deliberately leaves queue acknowledgement abstract. nack here means the queue's configured retry path should retain the message; it is not a command every queue implements with that spelling. In a real adapter, read and record the response body before classifying a non-success response, and keep the idempotency key stable across every attempt. The database row should move to delivered only after the worker has a successful provider result.
What should operators measure before raising worker concurrency?
Start with a small, explicit runbook. Record the age of the oldest queued digest, claimed-but-unpublished records, publish latency, delivery latency by channel, rate-limit responses, retry count, and duplicate-suppression count. Also record the number of active customers selected by each run. A queue depth graph alone cannot tell you whether customers are receiving messages or whether workers are repeatedly reprocessing the same ones.
Alert on symptoms that map to an action. Rising queue age with no rate-limit responses suggests publisher or worker capacity. Rising 429 responses suggest pacing or quota pressure. A growing claimed-but-unpublished set suggests the handoff needs reconciliation. A rise in duplicate-suppression events is a delivery-control problem, even if the provider reports success.
Run a dry schedule against a fixture containing two customers, one email digest, one SMS digest, an already-delivered digest, and a record whose due time is just past the scan boundary. Verify that the claim is repeatable, that a second publisher run does not create a new logical ID, and that a retry does not acknowledge early. Test clock movement too; cron is a trigger, not the source of truth for due work.
The practical tuning rule is to raise concurrency only when queue age is the limiting signal and channel responses remain below the documented limit. If rate-limit responses rise, reduce request rate or split the lane. I'm not sure which provider limit will bind first for a new account, and your mileage may vary; the delivery log and the provider's current policy should settle that question.
Which trade-off fits a weekly marketplace digest?
The design choice should follow the failure you can afford.
| Approach | Useful when | Cost or limitation |
|---|---|---|
| Scheduler sends directly | The audience is tiny and a missed run is easy to repair | A slow provider ties up the scheduler and makes retries harder to isolate. |
| Scheduler plus one shared worker pool | Both channels have similar pacing and one operational lane is enough | One channel can consume capacity needed by the other. |
| Scheduler plus channel-specific workers | Email and SMS have different limits or incident procedures | More queues, dashboards, and deployment configuration must stay consistent. |
| Durable workflow engine | A digest is one step in a long approval or compensation process | More workflow state and operational concepts than a bounded publish-and-send loop needs. |
The catch is that channel-specific workers are not suitable when the team cannot operate separate retry policies and alerts. Stick with a shared pool when the traffic is small and the limits are genuinely aligned. Move to a workflow engine when delivery has dependencies, human approval, or compensation; adding hidden orchestration to a cron worker makes recovery harder to reason about.
Cost matters, but it follows the control model. A smaller pool can be cheaper to run, yet it is a poor choice if the weekly digest has a customer-facing deadline that it cannot meet. A larger pool is wasteful when downstream limits dominate. Measure the completion target first, then choose the least concurrency that meets it without converting retries into a second traffic spike.
Where does this scheduling pattern stop being enough?
It does not solve every scheduling problem. A cron trigger can start a scan, but the application database must decide what is due, what was claimed, and what was delivered. A paused trigger should not silently erase work; the next scan needs a due-time query and reconciliation for records left in an intermediate state.
It also is not a replay system or a general workflow graph. If the product needs fan-out and join semantics, long-lived timers, or a durable history of every workflow transition, choose a tool designed for those requirements. Keep the digest path as a bounded dispatcher until those requirements are real.
The invariant is the part worth carrying forward: one stable logical ID, one owner for pacing, and one explicit acknowledgement point. Those three decisions prevent a queue from hiding the most expensive kind of failure: a customer receiving the same reminder twice while the dashboard says everything is healthy.
Top comments (0)