Short answer: make cron admit bounded cleanup jobs to a durable queue, return immediately, and let workers claim idempotent reminder batches; a public HTTPS webhook that stays open while it scans tenants and sends reminders cannot provide a useful delivery guarantee, even when its timeout is 900 seconds.
For a property-management system, the unit of work should be smaller than "run reminders." Use something replayable, such as "process lease-expiry reminders for property 1842 and the 2026-08-14 delivery window." The scheduler may submit that unit more than once. A worker may receive it more than once. The database must turn those repeats into one durable intent and one observable outcome.
That's the invariant.
The familiar failure starts innocently: a Node.js route receives a cron webhook, queries every building, renders messages, calls a delivery provider, and responds after the loop. As the portfolio grows, the request crosses a 900-second deadline. An unreachable public HTTPS endpoint makes the trigger fail earlier, but fixing DNS, TLS, or ingress only reveals the deeper design error: request lifetime has been confused with job lifetime. Extending the deadline buys runway, not correctness.
The incident-shaped failure is an ownership problem
Consider a bounded production scenario rather than an invented outage story. A property manager changes a tenant's move-out date while the nightly cleanup is running. The trigger submits a batch for that property, loses its response, and retries. One worker claims the original batch; another sees the retry. Meanwhile, a pod is terminated after sending a reminder but before recording completion. There are now three ordinary events that can create a duplicate: trigger uncertainty, concurrent claims, and a crash between an external side effect and a local state update.
No timeout setting resolves all three.
Retries are normal.
The scheduler owns when to propose work. The queue owns durable admission and redelivery. The worker owns bounded execution. The database owns deduplication and state transitions. The delivery adapter owns translating a stable reminder ID into an external request. If any one component quietly owns two or three of those responsibilities, the failure modes become hard to name and harder to alert on.
I use a capacity-planning reflex here: estimate the peak number of properties due in one window, multiply by the worst credible reminders per property, and compare that arrival rate with measured worker throughput. The precise throughput is unknown until a load test uses representative rendering and delivery latency; I'm not sure a synthetic no-op provider tells you much beyond queue overhead. The useful question is whether the backlog can drain inside the reminder's freshness budget after losing one worker, not whether a single happy-path run finishes before an HTTP timer expires.
This changes the SLO as well. "Cron returned 200" is an admission signal, not a delivery objective. Track separately whether scheduled batches were admitted on time, whether eligible reminders reached a terminal state inside their window, and whether retries produced duplicate side effects. An HTTP success can coexist with a growing backlog. An HTTP timeout can coexist with successfully admitted work. Conflating those cases trains the on-call engineer to restart jobs that may already be running.
How should a Node.js queue worker fix unreachable public HTTPS reminder endpoints?
First, diagnose reachability as a narrow trigger-plane issue. Resolve the hostname from outside the private network, validate the certificate chain and hostname, verify that ingress permits the scheduler's traffic, and make the handler authenticate before accepting work. Keep its response path short: validate a signed request, derive a deterministic batch key, insert or find the batch, enqueue its identifier, and return. Do not put tenant scans or message delivery on that path.
Then make the schedule recoverable without the webhook. Store the expected schedule window and reconcile it: a small process periodically asks which property-window batches should exist but do not. This closes the gap when an external trigger cannot reach the endpoint. The trigger is now a latency optimization; persisted schedule state and reconciliation provide recovery.
Deadlines are not durability.
In a Node.js deployment, the admission handler can remain in the existing service while workers run as a separate process type. Separation matters operationally. Web replicas can scale on request concurrency, workers can scale on queue age and processing rate, and a slow delivery dependency cannot consume every web connection. It also gives deployment control: drain workers, stop new claims, allow bounded tasks to finish, and then replace the process.
Retries need classification. Retry timeouts, connection loss, and rate limiting with capped exponential backoff plus jitter. Send malformed tenant contact data to a terminal state such as invalid, because another attempt won't repair it. Treat a duplicate batch insert as success by reading the existing record, not as an exceptional page. Use a retry budget so poison work cannot monopolize capacity.
For example, if the trigger's caller reports a timeout at exactly 900 seconds, don't infer that nothing happened. Query by the deterministic batch key. If the batch exists, observe its state; if it does not, reconciliation can create it. This is why a random job ID generated on every request is a trap — each uncertain retry manufactures new work.
The catch is that a queue adds state, operating cost, and an on-call surface. It is not suitable when the cleanup is a tiny, read-only, fully recomputable task whose duplicate execution has no user-visible effect and whose runtime is comfortably bounded. In that case, keep a short scheduled process and record only its last successful window. Do not pay for delivery semantics the job does not need.
Make retries boring with a claim-and-commit state machine
The preventative path begins with a unique reminder key, for example (tenant_id, reminder_kind, due_window). A batch transaction inserts missing reminder intents under that key. Workers claim a limited number of pending rows using a lease with an expiry, then commit the claim before calling the delivery adapter. A crashed worker's lease eventually expires, making the row eligible again. A live worker renews only while it is making progress.
Exactly-once delivery across a database and an external provider is not a promise I would put in an SLO unless both participate in one atomic protocol. Most systems should target at-least-once execution with idempotent effects. Pass the stable reminder key to any downstream operation that accepts an idempotency key. If it does not, record an attempt ID and reconcile ambiguous outcomes before sending again. Your mileage may vary because the decisive fact is the downstream provider's contract, not the queue brand.
The following Go sketch shows the worker boundary. The store implementation is expected to make Claim atomic, enforce uniqueness for the reminder key, and reject a stale lease token during MarkSent. Those are database constraints, not comments that callers may choose to obey.
package reminders
import (
"context"
"errors"
"time"
)
var ErrNoWork = errors.New("no work available")
type Reminder struct {
ID string
LeaseToken string
TenantID string
DeliveryWindow string
Attempt int
}
type Store interface {
Claim(ctx context.Context, leaseFor time.Duration) (Reminder, error)
MarkSent(ctx context.Context, id, leaseToken, receipt string) error
Release(ctx context.Context, id, leaseToken string, retryAt time.Time) error
}
type Sender interface {
Send(ctx context.Context, idempotencyKey string, reminder Reminder) (string, error)
}
type Worker struct {
Store Store
Sender Sender
Now func() time.Time
}
func (w Worker) RunOnce(ctx context.Context) error {
reminder, err := w.Store.Claim(ctx, 45*time.Second)
if errors.Is(err, ErrNoWork) {
return nil
}
if err != nil {
return err
}
receipt, err := w.Sender.Send(ctx, reminder.ID, reminder)
if err != nil {
delay := retryDelay(reminder.Attempt)
return w.Store.Release(ctx, reminder.ID, reminder.LeaseToken, w.Now().Add(delay))
}
return w.Store.MarkSent(ctx, reminder.ID, reminder.LeaseToken, receipt)
}
func retryDelay(attempt int) time.Duration {
if attempt > 6 {
attempt = 6
}
return time.Duration(1<<attempt) * 5 * time.Second
}
Keep the claim batch small enough that a worker can finish well inside the lease. Forty-five seconds in the sketch is an example configuration, not a universal recommendation; derive the real value from the upper tail of delivery latency, then test process termination near the lease boundary. Walk through the ugly timing before shipping: worker A claims a reminder, spends most of the lease waiting on delivery, and loses its lease just as the provider accepts the request; worker B then claims the same row while A tries to commit its receipt. The stale token must prevent A from overwriting B's ownership, but that protection alone cannot retract the first external side effect, so both sends need the same stable idempotency key and the ambiguous attempt needs reconciliation. A lease that is too short creates concurrent attempts. One that is too long delays recovery after a dead worker. Both should be visible through lease-expiry counts and reminder age, and the load test should vary latency near the boundary instead of testing only fast success and total failure.
Race the clock on purpose.
There is also a subtle ordering choice. Per-tenant ordering may matter when a move-out cancellation and its reminder race, but global ordering wastes capacity and increases blast radius. Partition by the smallest business key that truly requires ordering, usually the tenant or property, and let unrelated properties proceed independently. Check current eligibility immediately before delivery so stale queued intent cannot override a later business change.
Buy, build, or keep the scheduler small
The architecture does not require one particular product. The decision is about who carries persistence, redelivery, upgrades, and pager duty. A useful comparison starts with constraints instead of feature counts:
| Option | Good fit | Operational burden | Boundary to test |
|---|---|---|---|
| A Node.js process plus database-backed claims | The team already operates the database and volume is modest | Schema migrations, polling load, lease correctness, worker deploys | Prove concurrent claims and expired-lease recovery under load |
| GitHub Actions schedule plus a short admission call | Repository automation already owns the trigger | Workflow permissions and dependence on an external control plane | Treat schedule delivery as a trigger, then reconcile missed windows |
| RabbitMQ consumers | The team already runs a broker and understands acknowledgements | Broker capacity, consumer tuning, upgrades, and dead-letter policy | Acknowledgement placement must match completed durable work |
| BullMQ workers | The Node.js team already operates its backing services | Queue data lifecycle, worker concurrency, upgrades, and alerting | Test retry classification and deterministic job identity |
RabbitMQ's acknowledgement model makes the central boundary explicit: acknowledgements tell the broker when a delivery can be considered handled, while publisher confirms cover a different direction of responsibility. Put the acknowledgement after the durable outcome that the worker promises. An early acknowledgement turns a crash into lost work; a late or missing acknowledgement permits redelivery, which is why the handler must be idempotent.
GitHub Actions documents scheduled workflow triggers, but using a workflow as the entire reminder engine still couples business completion to an automation run. Keep it only as the clock if that is already an accepted dependency. The same judgment applies to any hosted scheduler: its request proves an attempt to trigger, not completion of every reminder behind that request.
Stick with database claims when the team can prove the locking behavior, backlog queries are cheap, and adding a broker would create more on-call load than it removes. Choose an operated queue when backlog isolation, consumer control, and redelivery are material enough to justify another system. Choose a managed queue when the team values reduced broker operations more than portability, after checking message retention, acknowledgement deadlines, observability, and egress constraints. There is no honest universal winner.
Before deployment, test duplicate admission, two simultaneous claims, termination after send but before commit, lease expiry, invalid recipient data, dependency throttling, and a full trigger outage followed by reconciliation. Release with worker concurrency capped below downstream quotas. Alert on oldest eligible reminder age and terminal failure rate; queue depth alone is a capacity signal, not a user-impact signal. The runbook should answer one dangerous question plainly: after an ambiguous attempt, how does the operator determine whether replay is safe?
The final decision rule is compact. Keep HTTP responsible for authenticated admission, use persisted schedule windows to recover missed triggers, and make every reminder transition replayable. A 900-second deadline then becomes a diagnostic clue rather than the boundary of the job. The queue worker is useful because it moves ownership to durable state, not because background processing makes timeouts disappear.
References
- RabbitMQ, "Consumer Acknowledgements and Publisher Confirms": https://www.rabbitmq.com/docs/confirms
- GitHub Docs, "Events that trigger workflows": https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows
Top comments (0)