Short answer: use a short daily trigger plus a durable recipient-level sending queue; choose pull workers for a private sender and a public HTTPS push subscriber only when the endpoint and handoff can be operated as a real production boundary.
For a media service sending a daily payment-reconciliation report, put the provider's rate limit in the consumer and keep a stable idempotency key for every report delivery. This keeps latency and cost visible: more workers reduce drain time, but they also consume the provider's sending budget faster and increase the chance of throttling. The queue is the pacing boundary, while the report ledger remains the accounting boundary.
Keep the trigger small. The schedule should identify the reconciliation date, write one work item per recipient, and finish. The queue then carries the slow part.
What should a daily payment report queue do before it sends?
Start with the delivery contract, not the cron expression. A report run needs a durable identity such as report date + account + recipient + template version. The producer can safely publish that identity more than once only when the consumer can recognize it. At-least-once delivery is a normal queue contract; an acknowledgement is not proof that the email provider accepted the message.
For a payment reconciliation report, the work record should make these states distinguishable: pending, eligible to send, retryable, sent, and terminal failure. That small state model answers the first on-call question: which report is late, and why? Queue depth alone cannot answer it. A queue may be draining while one old item is stuck behind a bad provider response.
The report body should not be the queue's source of truth. Store the rendered artifact and reconciliation result in durable application storage, then put a compact reference and the delivery identity in the message. This also makes a deliberate replay possible without recomputing financial data during an incident.
The consumer must claim the identity before making a provider call, then record the provider-facing result. A process can stop after the provider accepts a message but before the local record says sent. On restart, reconcile by the same idempotency identity before attempting another send. The exact provider mechanism varies, so I'm not sure every provider can offer the same lookup operation; where it cannot, the application still needs a policy for that uncertain interval instead of silently sending twice.
That's the boundary.
How do rate limits change the queue and public HTTPS design?
Pacing belongs at the boundary that knows the sending budget. A pull worker can wait for its local token budget, send one item, and schedule a retry after a 429 response. A push subscriber receives work over a public HTTPS endpoint, so it needs authentication, request validation, a bounded handler, and a handoff to durable state. The HTTP handler should acknowledge only after the work has been durably accepted; doing so earlier turns an endpoint timeout or process exit into lost work.
The retry delay should honor Retry-After when the provider supplies it. Without that signal, exponential backoff avoids a tight retry loop. Add a cap and jitter so a large retry wave does not become a synchronized second outage. A retry is still a new attempt at the same delivery identity, not a new email.
| Delivery shape | Operational fit | Main trade-off |
|---|---|---|
| Pull workers | Sender stays on a private network and can poll for work | The team owns worker capacity and its pacing loop |
| Public HTTPS push subscriber | The delivery system can reach an authenticated public endpoint | The handler and durable handoff become an inbound security boundary |
| Workflow engine | Report delivery has dependent steps, timers, or human approval | More coordination machinery for a simple enqueue-and-drain job |
| Event log | Independent consumers need retained history and replay | Retention and replay semantics are heavier than acknowledged work |
| Simple scheduled loop | Very small recipient set and a generous deadline | A provider throttle can hold the whole run open and complicate recovery |
The table's last row is a useful warning. A queue earns its keep when the sending window can outlast the scheduler's execution window.
Here is the core policy in Go. The queue client and provider client are intentionally small interfaces: the operational rule matters more than a particular broker or email SDK.
package delivery
import (
"context"
"errors"
"math/rand"
"time"
)
var ErrRateLimited = errors.New("provider rate limit")
type Item struct {
Identity string
}
type Sender interface {
Send(context.Context, Item) error
}
type ClaimStore interface {
Claim(context.Context, string) (bool, error)
MarkSent(context.Context, string) error
}
func Deliver(ctx context.Context, item Item, store ClaimStore, sender Sender) error {
claimed, err := store.Claim(ctx, item.Identity)
if err != nil {
return err
}
if !claimed {
return nil
}
if err := sender.Send(ctx, item); err != nil {
return err
}
return store.MarkSent(ctx, item.Identity)
}
func Backoff(attempt int, retryAfter time.Duration) time.Duration {
if retryAfter > 0 {
return retryAfter
}
base := 1 << min(attempt, 6)
return time.Duration(base)*time.Second +
time.Duration(rand.Int63n(int64(time.Second)))
}
func min(a, b int) int {
if a < b {
return a
}
return b
}
This claim-before-send sequence is not a magic exactly-once guarantee. It is a place to define the uncertain outcome and connect it to reconciliation. The provider-facing idempotency facility, if available, should use the same identity. If the endpoint is push-based, return an HTTP success only after the item is in durable queue storage; if the handoff fails, let the delivery system retry the request.
Which latency and cost signals should decide capacity?
Measure the deadline, not just throughput. For each report date, record the age of the oldest pending item, enqueue-to-send latency, provider response classes, retry count, and terminal outcome count. Also record the number of accepted sends separately from the number of queued items. Those measurements show whether a slower run is caused by insufficient workers, a provider window, or a report-generation problem.
The capacity decision is a budget exercise. If the provider permits a fixed number of sends in a time window, set the consumer pace below that ceiling and reserve headroom for retries. Adding workers should increase concurrency only until the pace budget is reached. Past that point, it raises contention and throttling without improving completion time.
For a nightly reconciliation, a longer drain time may be acceptable if it stays inside the report's delivery deadline. A shorter drain time may be worth the extra infrastructure when recipients need the report before a morning editorial or finance workflow. Cost is part of the decision, but it should be compared with the operational cost of a backlog, duplicate emails, and manual reconciliation, not used as a standalone quality claim.
What belongs in the runbook when delivery goes wrong?
Test the boring path first: one report date, a small recipient set, one deliberate duplicate identity, and a provider test response that supplies Retry-After. Verify that the duplicate produces one provider attempt, that the retry waits, and that the scheduled kickoff remains short. Then test a worker exit between provider acceptance and MarkSent; the recovery result must follow the documented uncertain-outcome policy.
Alert on oldest-work age and delivery-deadline risk. Use queue depth as context. A single old item deserves attention even when the total backlog is falling.
Rollback starts by pausing new scheduled kickoffs. Reduce consumer capacity or stop consumers while operators inspect pending and retryable items. Do not purge first: acknowledged work is no longer available from the queue, and a paused schedule does not recreate missed triggers. Resume at a conservative pace after the provider budget and the report ledger have been checked. Republish only the chosen report date and original delivery identities.
The catch is that this design is intentionally a delivery queue, not a workflow engine or an event-history system. It is not suitable when the report requires many dependent steps, long-lived human approval, independent replay for several consumer groups, or a private-only receiving network that cannot expose an authenticated HTTPS endpoint. In those cases, choose infrastructure designed for that boundary and keep the same identity, pacing, and verification rules.
Top comments (0)