DEV Community

FrostY45
FrostY45

Posted on

Draining Game Worker Pools: Node.js Cron, Postgres due_at, Queues, and Idempotency

Short answer: keep reminder time in Postgres, let a minute-level cron enqueue bounded batches, and make workers drain the rate-limited pool with leases, backoff, and a delivery ledger. The recovery rule matters more than the cron expression: a missed tick must leave due rows discoverable, while a repeated delivery must not notify a player twice.

This is a good fit for game backends that send user reminders around events, energy refills, or match windows. The scheduler should find work. It should not perform the provider call. That separation gives an on-call engineer a useful question during an incident: is the reminder due, claimed, queued, retrying, or delivered?

Should a Node.js cron scan Postgres due_at rows for user reminder queue work?

Yes, when the database already owns reminder state and the latency target is measured in minutes. A cron tick is only a wake-up signal. The authoritative query is due_at <= now() plus the application state that says the reminder is still eligible. Do not make the current minute the only search window. A paused process does not magically replay the ticks it missed.

Give the scan a bounded lookback and a batch limit. The lookback should cover the longest routine pause your operations team expects to recover automatically; the batch limit protects the database and the provider when a game launch creates a backlog. Add an index that supports the eligibility query. Rows older than the automatic window need an explicit review path, because “catch up everything” can become its own outage.

The handoff has two separate duplicate risks. Two cron instances can see the same row, and a queue can deliver one message more than once. A short broker deduplication window cannot replace an application-level record. The row claim and its lease belong in one database transaction; the worker’s side-effect guard belongs around the notification call.

Keep it boring.

What does a recoverable due-time handoff look like?

Use a stable reminder ID all the way through the system. A dispatcher claims only rows that are due, incomplete, and not covered by a live lease. It commits that claim before publishing a small queue message. If publishing fails after the commit, the lease expires and the row becomes eligible again. If the dispatcher times out after publishing, a second message is acceptable because the worker checks the delivery ledger.

The worker should acknowledge last. It loads current reminder data, checks a uniqueness key such as (reminder_id, channel), and records the provider acceptance before acknowledging the message. A provider response such as HTTP 429 means the attempt must respect the provider’s retry guidance and rate-limit semantics, not spin in a tight loop. Messages that exhaust the retry policy belong in a dead-letter queue for inspection and deliberate replay; AWS documents this pattern for SQS dead-letter queues.

Consider a game launch where the dispatcher claimed 500 due reminders and then lost its connection while publishing. The database now contains 500 active leases, but the queue contains an unknown subset. The recovery job should wait for the lease deadline, find the still-undelivered rows through the same bounded query, and publish stable reminder IDs again. A worker may receive one of those IDs twice, or it may receive a copy that was already accepted by the provider before the process crashed. In both cases the delivery ledger decides: an existing completed key stops the side effect, while an absent key permits one attempt and records its result. If the provider answers 429, the worker releases pressure through delayed retry rather than treating the reminder as permanently failed. If the retry policy is exhausted, the dead-letter item preserves a reviewable unit of work. This sequence is why the database lease, queue message, provider request, and ledger record must have separate states. Collapsing them into a single “sent” boolean makes the dashboard clean and the incident opaque.

Here is the small decision predicate I would unit-test before wiring up a queue. It is deliberately boring. The production version must evaluate the equivalent predicate and update atomically in Postgres, using row locks or an equivalent concurrency control mechanism.

package main

import (
    "fmt"
    "time"
)

type Reminder struct {
    ID         string
    DueAt      time.Time
    Delivered  bool
    LeaseUntil time.Time
}

func claimable(r Reminder, now time.Time) bool {
    return !r.Delivered && !r.DueAt.After(now) && !r.LeaseUntil.After(now)
}

func deliveryKey(reminderID, channel string) string {
    return "reminder:" + reminderID + ":" + channel
}

func main() {
    now := time.Date(2026, time.August, 10, 12, 0, 0, 0, time.UTC)
    r := Reminder{ID: "match-queue-42", DueAt: now.Add(-time.Minute)}
    fmt.Println(claimable(r, now), deliveryKey(r.ID, "push"))
}
Enter fullscreen mode Exit fullscreen mode

The message should contain the identity and narrow routing data, not a 200 KB rendering payload. Current state stays in the application database. That makes a replay safer: the worker can render from current data while the delivery key remains stable.

How should a rate-limited worker pool handle retries and recovery?

Treat the pool as a budget, not as a target to fill. Bound concurrency per provider or channel, add exponential backoff with jitter, and expose the age of the oldest due reminder. When the provider returns 429, reduce pressure and retry according to its documented limits. A queue depth graph without due-age and retry-age graphs can look healthy while players wait.

I have been paged for missed jobs and duplicate deliveries. The uncomfortable part is that both alerts can fire from the same weak boundary: a scheduler reports success after it woke up, while no one checks whether a delivery record was created. Keep separate counters for claims, publish attempts, provider-accepted attempts, completed ledger rows, expired leases, and dead-letter messages. Three minutes of overdue age is a more actionable signal than “cron ran.”

Drain mode needs an owner and a stop condition. Stop new claims first. Let already-published work drain only if the rate limit and idempotency checks remain intact. Then inspect expired leases and dead-letter items. On resume, the bounded scan should recover eligible rows without changing their IDs. Never manufacture a new identity merely to force a retry.

What should verification and rollback cover before enabling reminders?

Start with a test tenant and a narrow due window. Run two dispatcher instances together and confirm that one eligible row receives one active lease. Publish the same reminder identity twice to a worker with external delivery disabled; the second pass should stop at the ledger. Then test a provider 429 response and verify delayed retry, bounded concurrency, and a visible dead-letter path.

The rollback is procedural. Pause the cron trigger, preserve reminder and ledger rows, and record the oldest overdue due_at value before changing batch size or concurrency. If the incident involves duplicate attempts, keep the provider request IDs and queue message IDs for reconciliation. A green scheduler metric is not a rollback criterion.

The operator needs a decision table, not another timer chart.

Signal Immediate action Recovery evidence
Oldest due row is growing Stop new claims and inspect batch pressure Due age falls without duplicate ledger keys
Lease expiry is rising Check dispatcher and publish latency Reclaimed rows have stable IDs
HTTP 429 volume rises Lower concurrency and honor retry timing Provider acceptance returns without a retry storm
Dead-letter count rises Classify before replaying Replay closes the original delivery key

I’m not sure a one-minute interval is right for every game. Your mileage may vary: a high-volume launch, a second-level notification promise, or a private-only service boundary can require a different scheduler and delivery design. Stick with this pattern when Postgres is the source of truth, minute-level latency is acceptable, and the team needs an auditable recovery loop. Pick a workflow engine for multi-step joins, or a replay-oriented event log when independent consumers need durable history.

References

Top comments (0)