Short answer: let the cron-like scheduler create a small, idempotent sweep command, and let a worker queue own the long-running report job. Keep reservation expiry separate from report rendering. The important boundary is not the timer; it is the retry contract. A repeated trigger must enqueue equivalent work, a repeated worker delivery must not send a second email, and a delayed report must never extend a reservation that has already crossed its fixed hold window.
For a B2B SaaS daily report email, I would set the report's schedule and the reservation's expiry policy independently. A 24-hour hold is a business rule, not a queue timeout. The scheduler can wake every 5 minutes, discover due work, and enqueue a stable job key. Workers can then spend minutes on report generation without making the scheduler wait. This combining approach gives the platform team a tractable SLO: for example, 99.9% of due reports accepted within 10 minutes, with report delivery latency measured separately from trigger latency.
That is the whole boundary.
The incident lesson: the timer was innocent
The bounded failure scenario is familiar: a daily sweep starts while yesterday's report is still rendering. The sweep sees the same account again, submits another job, and a retry after a worker timeout submits a third. Meanwhile, a reservation created for a fixed hold window is still eligible in one code path because the renderer copied an old status into its output. The scheduler did what it was asked to do. The missing invariant was elsewhere.
I treat a trigger as a request to reconcile state, never as proof that work has happened exactly once. That distinction survives process restarts, overlapping sweeps, queue redelivery, and a database failover. The durable record needs a business key such as (account_id, report_date), a reservation version or expiry timestamp, and a send state that can be checked atomically. A queue message is a delivery mechanism; it is not the source of truth.
The preventative path is deliberately boring:
- Read due report rows using a bounded query and claim them with a lease or compare-and-swap update.
- Enqueue a job containing the immutable report key, not a large snapshot of mutable account data.
- Let the worker re-read current state, render, and record an idempotency result before acknowledging delivery.
- Retry transient failures with a bounded policy; send permanent validation failures to an operator-visible dead-letter path.
The same rule applies to reservation expiry. A sweep may issue expire(reservation_id, expected_version). If the row is already expired, the operation succeeds as a no-op. If its version changed, the worker reloads and decides again. It does not resurrect the reservation because an earlier message arrived late.
How should I combine a cron job trigger with a worker-based queue?
A scheduler should own time and discovery. It should not own report computation, outbound HTTP calls, or email delivery. Its run duration should be predictable enough to fit inside the trigger interval, and its query should be resumable when the due set is larger than one process can inspect. A cursor, a partition key, or a database-side claim can provide that boundary; a loop that loads every account into memory cannot.
A worker-based queue owns the expensive and failure-prone part. It can apply concurrency limits, back pressure, timeouts, and per-tenant fairness while the scheduler remains available for the next sweep. I would make the job payload small and versioned. These are report-generation tasks, so the payload should identify work rather than attempt to carry every input needed to perform it:
package reports
import "time"
type ReportJob struct {
AccountID string
ReportDate string
Schema int
CreatedAt time.Time
}
func (j ReportJob) Key() string {
return j.AccountID + ":" + j.ReportDate
}
The identifier is not a magic guarantee. It must be enforced by a unique constraint or an equivalent atomic state transition. If the queue hands the same job to two workers, both may render; only one should be allowed to commit the send decision. Where the email provider cannot offer an idempotency contract, an outbox record and a send ledger reduce duplicates, but they cannot make an external side effect perfectly transactional with the database. That limitation belongs in the design review.
How do retries avoid extending the hold window?
Retry policy needs two clocks. The first is the business deadline: expires_at for the reservation. The second is the operational deadline: how long the report job may be retried before it becomes an incident or a dead-letter item. Mixing them is how a transient queue outage silently changes business behavior.
At each attempt, the worker should compare the current time with the stored expiry and compare the stored version with the job's expected version. The update must be conditional. In SQL terms, the intended shape is an atomic update with predicates for identity, version, and state; the exact syntax depends on the database. A successful update returns one affected row. Zero rows means the worker must inspect current state, not blindly retry the same mutation.
package reservations
import "context"
type Store interface {
ExpireIfDue(ctx context.Context, id string, version int64) (bool, error)
}
func Expire(ctx context.Context, store Store, id string, version int64) error {
changed, err := store.ExpireIfDue(ctx, id, version)
if err != nil {
return err // The queue may retry a transient storage failure.
}
if !changed {
// Already expired, cancelled, or updated: all are terminal here.
return nil
}
return nil
}
This advice does not apply unchanged to a job whose effect is intentionally cumulative, such as appending an audit event for every attempt. There, the attempt identifier belongs in the event identity and deduplication rules. It also does not remove the need for an email provider's delivery semantics; it only prevents the platform from treating an at-least-once queue as exactly-once execution.
Build or buy: compare the operational boundary
The choice is less about the word cron and more about which failure handling the team is prepared to operate. A managed scheduler plus a managed queue can reduce control-plane maintenance, while a self-hosted scheduler and queue can offer more direct control over persistence, topology, and upgrade timing. Neither choice supplies the reservation invariant automatically.
| Decision area | Managed components | Self-hosted components |
|---|---|---|
| On-call load | Less scheduler and broker maintenance; provider limits become part of the design | More patching, capacity planning, failover drills, and operational ownership |
| Retry semantics | Must verify documented delivery, visibility, and redelivery behavior | Can tune behavior deeply, while owning every edge case and upgrade |
| Lock-in | APIs and event formats may become migration constraints | Fewer service-specific dependencies, but more internal platform code |
| Cost model | Often tracks executions, retention, or throughput | Tracks infrastructure, people, and the capacity headroom needed for bursts |
| Team fit | Useful when the team needs to focus on report logic and SLOs | Useful when queue behavior is a core platform capability |
I would reject a design that cannot answer three questions in a review: what happens when the sweep runs twice, what happens when a worker acknowledges after the email call but before recording success, and what happens when a reservation changes between enqueue and execution? The answers should be testable state transitions, not an assumption about a vendor's exactly-once label.
Testing the boundary before production
The highest-value tests are not happy-path timer tests. Freeze time and run two scheduler instances against the same due rows. Deliver one job twice. Kill a worker after the external send and before the database commit. Advance the clock beyond expires_at, then deliver a job created before expiry. Verify that the reservation remains terminal and that the report ledger records one logical send decision.
Load tests should vary the due-set size, worker concurrency, report duration, and retry delay independently. A capacity plan that uses only average report duration will fail during the morning burst; reserve headroom for the longest permitted retry backlog and for database claim contention. Alert on age of the oldest due item, queue depth, retry rate, dead-letter count, duplicate-send conflicts, and the percentage of reservations whose expiry transition misses its SLO.
The practical result is a narrow, defensible architecture: a short periodic reconciliation creates durable work, workers perform slow computation, and every state-changing operation is conditional and repeatable. The scheduler provides time. The database provides truth. The queue provides delivery attempts. Keeping those responsibilities separate is what makes a daily report system recoverable when the boring failures arrive. It also makes capacity planning concrete: the trigger path needs headroom for due-row discovery, while the worker path needs headroom for report duration, retries, and the largest expected burst, and those are different resources with different alerts.
The failure is visible sooner.
Top comments (0)