Treat cron and a message queue as interchangeable wake-up mechanisms, and put the reliability contract in a durable retry ledger that owns event identity, eligibility, and attempt history. For a logistics SaaS that expires capacity reservations after a fixed hold window, this means the database decides whether a reservation may move from held to expired; the scheduler merely asks a worker to check.
That recommendation has a practical consequence: don't choose infrastructure until the same worker can safely receive the same wake-up twice. A queue does not make a non-idempotent state change safe, and cron does not make a late webhook harmless. The deciding constraints are the allowed expiry lag, the credible recovery backlog, and the operational burden the team can carry without weakening its SLO.
Late delivery is normal.
Retry failure modes begin with two clocks and one identity
There are two clocks in this system, and combining them creates most of the trouble. The reservation clock answers when held capacity becomes eligible for release. The delivery clock answers when an already committed reservation.expired event may be sent again. Expiry is a domain transition; webhook retry is transport work. One can succeed while the other waits, so they need separate state and separate observability.
Use an immutable expires_at on the reservation and a versioned event identity in the delivery ledger. An illustrative key such as reservation.expired:rsv_8f31:v3 says which business transition occurred. Retrying that event increments an attempt number and changes next_attempt_at, but it does not mint a new event ID. If confirmation wins the conditional update before expiry, no expiry event is committed. If expiry wins, later scheduler invocations see the completed transition and do nothing. This is where a deceptively small Node.js service can accumulate reliability debt: imagine a reservation eligible at 10:15:00, a worker wake-up at 10:15:02, and an ambiguous webhook timeout after the receiver has applied the payload. Reconstructing the event from mutable reservation data on every attempt can change the payload, signature, or identity, while retrying the committed ledger record preserves the logical event and lets the receiver deduplicate on the stable ID before applying its effect. There is still uncertainty after a network timeout -- nobody can infer the remote commit from silence -- but duplicate processing no longer has to become duplicate business action.
Identity does.
Authentication failures have a separate retry cost
HMAC addresses another boundary. RFC 2104 defines a keyed-hash construction for message authentication; it does not provide replay protection, freshness, or idempotency by itself. Sign the bytes actually delivered, include the stable event identity in the signed payload, and define a receiver policy for acceptable age. Secret rotation must not turn an old logical event into a new one.
The database transaction defines idempotent processing
Make one conditional database transaction the authority for reservation expiry, then append one immutable delivery record as part of that transaction. The Node.js request path may create and confirm reservations, while the worker below is Go because the worker boundary is a language-neutral contract. Cron and queue consumers call the same Process method; neither gets a privileged path around the state check.
package retry
import (
"context"
"time"
)
type WakeUp struct {
ReservationID string
}
type Event struct {
ID string
Payload []byte
Attempt int
NextAttemptAt time.Time
}
type Store interface {
// ExpireAndRecord performs a conditional held-to-expired transition and
// records its stable event in the same transaction.
ExpireAndRecord(ctx context.Context, reservationID string, now time.Time) (bool, error)
ClaimDelivery(ctx context.Context, eventID string, now time.Time) (Event, bool, error)
MarkDelivered(ctx context.Context, eventID string, attempt int, now time.Time) error
ScheduleRetry(ctx context.Context, eventID string, attempt int, next time.Time) error
}
type Sender interface {
Deliver(ctx context.Context, event Event) error
}
type Worker struct {
Store Store
Sender Sender
Now func() time.Time
Delay func(attempt int) time.Duration
}
func (w Worker) Expire(ctx context.Context, wake WakeUp) error {
now := w.Now().UTC()
_, err := w.Store.ExpireAndRecord(ctx, wake.ReservationID, now)
return err
}
func (w Worker) Deliver(ctx context.Context, eventID string) error {
now := w.Now().UTC()
event, claimed, err := w.Store.ClaimDelivery(ctx, eventID, now)
if err != nil || !claimed {
return err
}
if err := w.Sender.Deliver(ctx, event); err != nil {
next := now.Add(w.Delay(event.Attempt))
return w.Store.ScheduleRetry(ctx, event.ID, event.Attempt, next)
}
return w.Store.MarkDelivered(ctx, event.ID, event.Attempt, now)
}
The Node.js and Go worker integration keeps one contract
The terse return from ExpireAndRecord matters. false, nil is an ordinary result when another worker already expired the reservation or the customer confirmed it first. It should contribute to a contention or duplicate-wake metric, not automatically page the on-call engineer. The transaction must also enforce uniqueness for the event identity, because application-level checks alone leave a race between concurrent workers.
ClaimDelivery should only return an event whose next_attempt_at is due and whose delivery lease is available. ScheduleRetry should use the attempt it claimed as a compare-and-set token, so a slow attempt cannot overwrite a newer result. The exact backoff and terminal policy belong to the service contract; I'm not sure what attempt ceiling is right for your recipients, and a review of their error distribution and recovery expectations is what resolves that question. Authentication rejection and an ambiguous timeout should not blindly share one retry policy.
Keep the payload immutable after the event is committed. It's tempting to rebuild it from the latest reservation row, especially when the schema is small, but that turns a transport retry into a fresh statement about business state. Store the canonical delivery bytes or a versioned event document. Then sign that representation at send time under an explicitly identified key, without changing the event's business identity.
How does cron compare with a message queue for delayed webhook recovery?
The cheapest option cannot be selected from a monthly service line. Capacity consumed in the primary database, recovery time after paused workers, on-call familiarity, and the cost of operating another stateful component all belong in the comparison. I use a buy-versus-build table because it forces each attractive feature to meet an operational consequence.
| Decision signal | Cron-triggered due scan | Delayed message queue |
|---|---|---|
| Expiry-lag SLO | Suitable when scan interval plus worst-case catch-up stays inside the objective | Suitable when wake-ups need finer timing or consumer backpressure |
| Recovery model | Due database state is rediscovered by the next bounded scan | Pending delivery needs an explicit redrive and retention policy |
| Primary capacity risk | Repeated indexed scans compete with request traffic | Backlog replay competes for worker and database capacity |
| Duplicate source | Overlapping ticks, expired leases, or repeated scans | Redelivery, expired leases, or ambiguous acknowledgements |
| On-call surface | Scheduler, scanner, database, and workers | Broker policy, consumers, database, and workers |
| Exit condition | Move when measured scan load or oldest-due age exhausts headroom | Stay only while the tighter timing need justifies the extra control plane |
Cron plus a bounded, indexed scan is not suitable when the expiry-lag objective is shorter than the scan and catch-up budget, or when scanning due rows consumes database headroom needed by reservation traffic. A delayed queue is not suitable when the team cannot operate its redrive, retention, and saturation behavior to the same standard as the database path. Stick with the scanner while it meets the measured objective with recovery margin; choose a queue when timing distribution or backlog control requires it, not because “queue” sounds more reliable.
Priority is not delayed delivery. RabbitMQ documents priority queues as queues with multiple internal priority levels and notes that higher priorities have resource and scheduling implications. That mechanism can influence which ready message is consumed first, but the reservation's database timestamp must remain the authority for whether expiry is valid.
Capacity planning should begin with oldest-due age: current database time minus the earliest eligible reservation that has not completed its transition. Depth alone is a weak service signal because future work can make a queue look large while a small set of old reservations violates the objective. Measure claim-to-commit latency, conditional no-op rate, retry age, terminal delivery count, worker saturation, and the primary database's remaining capacity. Then test the recovery cohort created by the longest pause in your deployment and incident model.
No shortcuts.
Migrate from cron to a queue with reversible state
Test the state machine with a controllable clock and barriers around database commits. The useful cases are not “the handler returned success”; they are two expiry workers racing, confirmation racing expiry, a delivery lease ending during a slow request, the receiver applying an event before the sender observes a timeout, and an old retry arriving after recorded success. For every schedule, assert that at most one versioned expiry event exists and that a confirmed reservation never returns to an expired state.
Before switching wake-up mechanisms, shadow the candidate path without side effects. Compare the reservation IDs it would wake against a direct query of due state, then enable a bounded shard and a strict worker concurrency limit. Watch oldest-due age and database headroom during both steady traffic and an induced consumer pause. A clean steady-state graph proves very little about catch-up.
Rollback should stop new claims, preserve leases and ledger rows, and let the previous wake-up adapter rediscover eligible work. Do not delete queued messages or reset attempt counters merely to make a dashboard look clean; the durable state is what lets a different adapter resume without changing business meaning. Since both adapters call the same idempotent core, rollback changes how work is noticed, not what expiry means.
The final selection rule is deliberately plain: pick the least complex wake-up mechanism that meets the measured expiry-lag SLO under recovery load, and reject any design whose correctness depends on exactly-once invocation. Idempotency lives in the transaction and retry ledger. Scheduling only controls when they are revisited.
References
- RFC 2104: HMAC: Keyed-Hashing for Message Authentication: https://www.rfc-editor.org/rfc/rfc2104
- RabbitMQ Priority Queue Support: https://www.rabbitmq.com/docs/priority
Top comments (0)