Short answer: make retries durable, bounded, and idempotent
For outbound customer-support webhooks, the simplest scheduler is the one that records intent before waiting, retries from a durable queue, and makes duplicate delivery harmless at the receiver. A daily report email can use the same shape: persist the report job, claim it with a lease, and let the worker retry delivery without assuming that a successful network response means the recipient processed the message.
The deciding constraint is the delivery guarantee. You can usually promise at-least-once delivery with durable state and receiver-side idempotency. You cannot honestly turn an HTTP call into exactly-once processing merely by adding a cron expression. A timeout after the remote server accepts a request is still ambiguous.
No shortcut.
That ambiguity is where support automation becomes operational work. A ticket update may trigger a downstream notification, a case-management record, or a daily digest. Repeating the event can be noisy; dropping it can hide a customer-visible change.
Start small.
Can a SaaS cron scheduler automate daily email without duplicate delivery?
A daily schedule solves timing, not delivery semantics. The useful question is whether the scheduler can preserve intent across a worker restart, expose retry age, and give the receiver a stable key for deduplication. If it cannot, minimal setup has moved complexity into incident response.
What does a retry scheduler actually guarantee?
Start with an explicit contract. For one event, the sender stores an immutable event ID, the destination, the payload version, the attempt count, and the next eligible time. The scheduler selects due work. The delivery worker sends it with the event ID as an idempotency key. The receiver persists that key before applying the side effect, or uses an equivalent atomic operation.
This is at-least-once delivery with deduplication at the consumer. It is a useful, defensible contract. The sender may transmit the same event more than once, but the receiver applies the event once per key. A receiver that cannot deduplicate needs a weaker application-level promise, regardless of how polished the scheduler looks.
I start with a 99.9% monthly delivery SLO for accepted events, then define the measurement carefully: accepted means durably recorded, delivered means the receiver returned an accepted response, and applied means the receiver confirmed its own idempotent write. Those are three different timestamps. Blurring them makes an attractive dashboard and a bad incident review.
A five-minute retry interval is not a reliability policy. It is one point in a policy that also needs a maximum age, a backoff function, a jitter rule, and a dead-letter path.
That distinction matters.
The safe implementation path
The minimum useful state machine is small: pending, leased, delivered, and dead-lettered. A lease has an expiry so a worker that loses its process cannot hold the job forever. A delivery attempt is an observation, not proof of final state; a timeout should return the job to retryable state after the lease expires.
Here is the important boundary in Go. The queue and database interfaces are intentionally generic because the guarantee belongs to the state transitions, not to a scheduler brand.
type Delivery struct {
EventID string
Attempt int
NextTryAt time.Time
LeaseUntil time.Time
Status string
}
func nextDelay(attempt int, base, cap time.Duration) time.Duration {
if attempt < 1 {
attempt = 1
}
d := base * time.Duration(1<<(attempt-1))
if d > cap {
d = cap
}
return d
}
The production version must add bounded randomness to the delay, because a shared outage can otherwise release a large batch at the same instant. It must also make the claim operation atomic. Two workers may read the same due row; only one should acquire its lease.
The receiver contract deserves equal attention. Send Event-ID and an idempotency key, retain the key for at least the sender's maximum retry age, and make the key cover the business operation rather than a worker attempt. If the key expires after 24 hours but the sender retries for 72 hours, duplicates are expected by design.
Secrets are part of this path. Keep signing keys and delivery credentials outside job payloads, rotate them through a documented process, and avoid logging them while recording enough metadata to investigate a failed attempt. The OWASP key-management guidance is a useful baseline for that boundary.
How should capacity and failure be planned?
Capacity planning starts with arrivals, not with the cron expression. Let R be accepted events per second, A the average attempts per event, and L the average seconds a delivery occupies a worker. Required worker concurrency is approximately R * A * L, with headroom for the retry burst. The retry multiplier is the part people omit.
For a daily report, a schedule may create a sharp cohort at 08:00. For support webhooks, an upstream incident can create the same shape across every tenant. I would reserve capacity for the retry cohort separately from ordinary traffic, then enforce per-destination and per-tenant limits so one failing endpoint cannot consume the whole worker pool.
The failure pattern is easy to underestimate. Suppose a support platform accepts 20 events per second and a destination is unavailable for 10 minutes. The first wave is already 12,000 events; if each event gets three attempts, the worker pool must absorb the original traffic plus the retry work after the destination returns. A scheduler that reports only its current timer count hides that queue mathematics. I want the retry cohort visible, bounded, and attributable to a destination before I decide how much spare capacity to buy or build.
The useful signals are age of the oldest pending event, due queue depth, delivery latency, retry ratio, lease expiry count, and dead-letter growth. Alert on the SLO error budget and on queue age, not only on process health. A green worker process with a queue that is six hours old is an outage wearing a green badge.
There is a deliberate trade-off here: longer retention improves recovery and forensic value, while increasing storage, replay risk, and privacy exposure. Set retention from the business recovery window, encrypt sensitive payloads, and prefer storing a reference to a report or event body when reconstruction is reliable.
A managed scheduler can remove patching, leader election, and basic operational toil. A self-hosted queue can preserve portability and give the platform team direct control over retention, isolation, and replay. Neither choice supplies receiver-side idempotency, and neither choice resolves an ambiguous timeout. That is the boundary I would put in the architecture review.
| Decision factor | Managed scheduler | Self-hosted scheduler |
|---|---|---|
| Initial setup | Usually shorter; operational defaults are prebuilt | Longer; the team owns deployment and upgrades |
| On-call load | Less infrastructure ownership, with external limits to understand | More failure modes inside the team's scope |
| Portability | API, event model, and retention semantics can create lock-in | More control over interfaces and storage choices |
| Delivery guarantee | Still depends on durable state and consumer deduplication | Same requirement, plus queue correctness |
| Capacity control | Often shaped by service quotas and tenant boundaries | Shaped directly by worker and storage capacity |
For a small daily report, I would choose the option whose audit trail, retry controls, and failure notifications the on-call team can actually operate. “Minimal setup” is a valid requirement, but it should be measured in reviewable state transitions and pages at 03:00, not in the number of buttons on day one.
Verify the contract before rollout
Test the ambiguous cases explicitly: response lost after acceptance, receiver returns a 500, receiver is slow, worker dies after claim, clock skew moves a job across a schedule boundary, and the same event is delivered concurrently. The expected result should be written as assertions about event IDs and business side effects, not only HTTP status codes.
Run a replay test with a fixed event ID. Send it twice, then inspect the receiver's durable record and the customer-visible result. Run a lease-expiry test with a worker killed after claim. Finally, force a destination outage long enough to exercise the cap and dead-letter policy. These tests expose more than a successful cron run.
Rollout should be reversible at the policy layer. Pause new scheduling, let in-flight leases expire or finish, preserve the event records, and replay only from an reviewed boundary. Do not delete the queue to stop a retry storm; deletion removes the evidence needed to decide what was delivered.
The decision rule is straightforward: use a scheduler for timing, durable state for intent, leases for worker failure, idempotency for duplicates, and an SLO that distinguishes recorded, accepted, and applied. If a design cannot show those five pieces, it is automating a request, not operating a delivery system.
Top comments (0)