When a player holds a seat for eight days, the queue is the wrong place to keep the calendar. Short answer: store the due time durably, sweep for work before the broker's maximum delay, and enqueue a small, idempotent notification job. This keeps latency predictable and makes a missed message explainable.
The useful mental model is a relay. A database row is the calendar; a scheduler is the person checking the calendar; the queue is the short running track. Before, one delayed message tried to sleep for the whole hold window and silently hit a seven-day ceiling. After, a sweep turns due rows into ordinary queue messages, each with a traceable attempt and a clear owner.
What does a seven-day delay limit change for game reminders?
A broker delay is a transport feature, not a retention policy. Limits differ, and a message that cannot be scheduled for the requested horizon may be rejected, truncated, or never become visible to a consumer, depending on the system. Treating that behavior as a reminder contract is the first failure mode.
For a reservation service, write expires_at and reminder_at in the same transaction as the hold. Keep a status such as pending, enqueued, or sent, plus an attempt count and a deduplication key. The sweep can then select rows whose reminder time has arrived, even if the hold is longer than the broker's supported delay.
I prefer measuring two clocks: scheduler lateness (enqueued_at - reminder_at) and delivery lateness (delivered_at - reminder_at). They answer different questions. A healthy queue can still hide a scheduler that stopped running, while a fast sweep can feed a backed-up consumer.
Measure it.
How should a reminder system handle delayed queue messages and cron enqueue work?
Use a short, repeatable polling interval that fits the latency budget. A five-minute sweep is a reasonable example for a “remind before expiry” message, but the right interval comes from player experience and load, not from a magic number. Every run claims a bounded batch, commits the claim, and emits an enqueue event. Two workers may race; the claim must be safe.
Here is a compact PostgreSQL pattern using row locks. FOR UPDATE SKIP LOCKED lets concurrent sweepers divide the due work without waiting on each other.
const due = await db.query<{
id: string;
reminder_at: string;
dedupe_key: string;
}>(`
UPDATE reservation_reminders
SET status = 'enqueued', enqueued_at = now(), attempts = attempts + 1
WHERE id IN (
SELECT id
FROM reservation_reminders
WHERE status = 'pending' AND reminder_at <= now()
ORDER BY reminder_at
FOR UPDATE SKIP LOCKED
LIMIT 500
)
RETURNING id, reminder_at, dedupe_key
`);
for (const reminder of due.rows) {
await queue.publish({
type: 'reservation.reminder',
reminderId: reminder.id,
dedupeKey: reminder.dedupe_key,
});
}
The database update and the publish are separate systems, so a crash between them is possible. An outbox table closes that gap: the transaction inserts an outbox record, and a publisher retries records until the queue accepts them. The consumer still checks the dedupe key before sending. Delivery is at-least-once; the player should see one reminder.
Which signals expose a missed scheduled notification?
Logs should carry reservation_id, reminder_id, dedupe_key, and an attempt number. Metrics should include due rows, claimed rows, publish failures, consumer age, and a counter for reminders whose deadline passed without delivery. Alert on age and rate, not only on process health. A cron process that exits cleanly every five minutes can still do zero useful work.
Make the dashboard tell a before/after story: “312 due, 310 enqueued, oldest consumer message 18 seconds” is actionable; “scheduler up” is not. Sample payloads in a staging environment with a clock you can advance. I am not sure a single percentile captures player impact, because launch-day bursts and quiet weekdays have different shapes; keep both tail latency and missed-deadline counts.
| Choice | Best fit | Main cost or risk |
|---|---|---|
| Native delayed message | Horizon and retry semantics match the hold window | Harder to audit long waits |
| Due-date table plus sweep | Auditable reminders with minute-level jitter | Polling and outbox state to operate |
| Dedicated scheduler | Tight deadlines and very high volume | More infrastructure and a new operational boundary |
Where is this cron-and-queue workaround the wrong fit?
The catch is operational complexity. A due-date table, outbox, sweepers, and reconciliation jobs add state to own. If reminders must fire within a few seconds across millions of tenants, a purpose-built scheduler or provider with a longer native horizon may be a better fit. Stick with a native delayed delivery feature when its limit, durability, and retry semantics are explicit and match the business window.
Conversely, do not stretch a queue to be a calendar when the business can tolerate minutes of jitter. A database sweep is often easier to audit and replay, but it costs periodic reads and careful index design. The decision axis is latency versus cost of operation: choose the simplest mechanism that meets the deadline, then instrument the parts that can fail.
A practical runbook for reminders that do not arrive
Start with one reservation ID and follow its timestamps: row creation, due time, claim, outbox publish, queue receipt, consumer attempt, and provider response. Suppose a hold is created at 10:00, its reminder becomes due eight days later at 09:00, and support checks it at 09:17. The investigation shouldn't begin with a guess about the queue. Compare reminder_at with enqueued_at; a missing enqueue timestamp points back to the sweep. If the row is marked enqueued, find the outbox record and its publish timestamp. Then compare queue receipt with the first consumer attempt. This sequence assigns the unexplained 17 minutes to one boundary. It also prevents a risky response: blindly enqueueing the same reminder again before checking whether the consumer already handled it. If the row was never claimed, inspect the sweep schedule and index. If it was claimed but not published, inspect outbox age. If it was delivered twice, inspect the dedupe key at the consumer boundary. Don't stop at “cron ran.” Prove that it moved due work.
Test the seven-day case as a time-compressed integration test, then test a restart during each handoff. Verify that a late reminder is either sent once or marked intentionally skipped, with a reason visible to support. That small audit trail turns “the message never arrived” into a bounded investigation. A useful stress test replays a batch with 500 due rows, pauses the publisher halfway through, restarts it, and compares dedupe keys before and after; this catches the awkward middle state that unit tests rarely model. The runbook is useful only when somebody can answer, from one reservation ID, which handoff happened last and why the next one was safe to retry.
Top comments (0)