Short answer: keep new-order templates in the application repository, enqueue one immutable notification intent per channel, and let separate workers send, poll delivery status, and retry under a defined SLO. A Node.js service can own the event and queue contracts without depending on webhooks; a cron job only schedules due work, while the queue remains the source of operational truth.
For a media marketplace, the deciding constraint is template ownership. A seller notification is part of the order contract: order ID, listing title, locale, and the safe link back to the marketplace must mean the same thing in email and SMS. If those templates can change in a provider dashboard without the application release process, the team has split ownership of a user-visible transaction across two control planes. That makes review, rollback, and incident reconstruction harder.
This is the boring recommendation. Boring is useful on call.
What should a Node.js event notifications service do for transactional email and SMS alerts?
Treat the order event, notification intent, send attempt, and delivery observation as four different records. The order event says what happened. The intent freezes what the seller should receive, including template_version, channel, recipient reference, locale, and an idempotency key such as order_8421:seller_73:email:v4. A send attempt records each provider submission. A delivery observation records the latest provider state and when it was observed.
Do not let an HTTP request that creates an order also wait for two messaging networks. Commit the order and an outbox record in one database transaction, then have a relay publish the intent to a durable queue. The consumer renders an application-owned template and submits it. This pattern closes the awkward gap where an order commits but the process ends before the notification is queued; the relay can safely revisit unpublished outbox rows. The same design works in a Node.js process even though the state-machine example below is Go.
The target is not exactly-once delivery. External delivery cannot be proven from a local commit, and a retry can race a successful submission whose response was lost. The useful target is at-least-once processing with deduplication: preserve a stable idempotency key, store the provider message ID before acknowledging queue work, and make repeated transitions harmless. Define the SLO against what the system can observe, for example the share of accepted new-order intents submitted to a channel within a chosen time budget. Do not label a message "delivered" merely because the provider accepted it.
Capacity planning starts with the burst, not the daily average. If a campaign can create 30,000 orders in 10 minutes and every order produces two channel intents, the queue receives 100 intents per second before retries and status checks. Size worker concurrency against provider quotas, database connection limits, and the oldest-message SLO; then load-test the whole path with rendering and persistence enabled. A worker pool that drains synthetic no-op jobs says almost nothing about this system.
Why polling delivery status needs its own state machine
Polling is a reconciliation loop, not a delayed send call. After submission, persist a normalized state such as submitted, delivered, failed_permanent, or unknown, plus the provider's raw status for diagnosis. Terminal states stop polling. Nonterminal states receive a next_check_at timestamp calculated with bounded backoff and jitter, so a large order burst does not turn into a synchronized status burst a minute later.
Keep uncertainty visible.
A provider's accepted response proves custody, while channel delivery signals have different semantics and may arrive late or remain inconclusive. Email opens are especially weak evidence because Apple Mail Privacy Protection can download remote content in the background, preventing a sender from reliably learning whether the recipient opened the message. For the seller-order workflow, use provider delivery state as transport telemetry and marketplace activity as product telemetry; do not combine either into a claim that a human read the alert.
The retry decision belongs to a classifier. Rate limiting, a network timeout, and a still-pending delivery state can be retried within a bounded policy. A malformed address, an unsubscribed destination where the message is not legally or contractually permitted, or an invalid template input should end automatic retries and enter an explicit review or suppression path. Set a maximum attempt count and an age limit. Without both, a poison message can consume capacity forever.
Consider one concrete sequence. Order 8421 commits at 09:00:00 with email intent e-17 and SMS intent s-18; the outbox relay publishes both, the SMS submission returns a provider message ID, and the email worker loses its connection before it knows whether submission completed. The email job then returns to the queue. A naive consumer creates a second email, while a consumer that reuses the intent's idempotency key can reconcile the ambiguous attempt without changing the seller-facing content. At 09:01 the polling dispatcher finds s-18, but its status is still nonterminal, so it records the observation and advances next_check_at rather than holding a worker. At 09:05 the status becomes terminal and polling ends. Every step is reconstructable from records; none requires the cron process to remember prior work in memory. This is also why template version v4 belongs on the intent rather than in a mutable global setting: if v5 deploys between the ambiguous email attempt and its retry, the seller must not receive two materially different descriptions of the same order.
Retries need a budget.
Here is the core transition shape. The storage and provider interfaces are deliberately generic; production code should make the lease and update atomic in the database.
package notifications
import (
"context"
"errors"
"time"
)
type DeliveryState string
const (
Submitted DeliveryState = "submitted"
Delivered DeliveryState = "delivered"
PermanentFailure DeliveryState = "failed_permanent"
Unknown DeliveryState = "unknown"
)
type Attempt struct {
ID string
ProviderID string
State DeliveryState
PollCount int
NextCheckAt time.Time
FirstSentAt time.Time
}
type StatusClient interface {
Lookup(ctx context.Context, providerID string) (DeliveryState, error)
}
type AttemptStore interface {
Due(ctx context.Context, now time.Time, limit int) ([]Attempt, error)
SaveObservation(ctx context.Context, attempt Attempt) error
}
func Reconcile(ctx context.Context, now time.Time, store AttemptStore, client StatusClient) error {
attempts, err := store.Due(ctx, now, 200)
if err != nil {
return err
}
for _, attempt := range attempts {
state, lookupErr := client.Lookup(ctx, attempt.ProviderID)
if lookupErr != nil {
attempt.State = Unknown
} else {
attempt.State = state
}
attempt.PollCount++
if attempt.State != Delivered && attempt.State != PermanentFailure {
attempt.NextCheckAt = now.Add(nextDelay(attempt.PollCount))
}
if err := store.SaveObservation(ctx, attempt); err != nil {
return errors.New("save delivery observation")
}
}
return nil
}
func nextDelay(pollCount int) time.Duration {
schedule := []time.Duration{time.Minute, 5 * time.Minute, 15 * time.Minute, time.Hour}
if pollCount >= len(schedule) {
return schedule[len(schedule)-1]
}
return schedule[pollCount]
}
The sample uses a batch of 200 and an illustrative schedule, not universal limits. Your mileage may vary: the correct numbers depend on traffic shape, provider quotas, and how quickly a status normally becomes terminal. Measure those distributions before fixing the production schedule.
Run the cron job as a scheduler, not as the queue
A cron job should wake a small dispatcher, claim rows whose next_check_at is due, and place their IDs on the retry or polling queue. It should not scan every historical notification, hold a process open while contacting providers, or encode retry policy in crontab. In a multi-instance deployment, use database leases or an atomic claim so two schedulers cannot select the same due rows without detection. Duplicate queue delivery must still be safe because a lease reduces overlap; it does not replace idempotency.
Keep separate queues for initial sends, status polls, and controlled retries. They have different urgency and failure shapes. A flood of slow delivery checks must not consume every worker needed to notify sellers about fresh orders. Reserve concurrency per workload, cap polling throughput below the external quota, and alert on queue age rather than queue length alone: 10,000 fast jobs may be healthy while 20 jobs stuck beyond the notification SLO are not.
The catch is that polling spends requests even when nothing changes and detects transitions later than a healthy webhook path. It is suitable when webhooks are unavailable, prohibited by the network model, or too expensive to operate across regions. If a provider offers authenticated, replayable webhooks and the team can expose and monitor an ingress, use webhooks for the fast path and retain a slower polling sweep for reconciliation. Stick with polling-only when an inbound endpoint would create more security and on-call burden than its latency benefit justifies.
For US and EU SaaS deployments, avoid casually copying recipient data into a global queue. Put the message body and destination in the region that owns the order; queue an opaque notification ID when possible, and send only the fields required by the selected channel. Retention, consent, suppression, and residency requirements depend on the business and jurisdiction, so legal and security owners must resolve them. An architecture diagram can't.
Choose template ownership before choosing a sender
| Ownership model | Release and rollback | Operational cost | Best fit | Main limitation |
|---|---|---|---|---|
| Application repository | Reviewed with code; immutable version can travel with the intent | Engineers own rendering, previews, and localization tooling | Transactional messages tied closely to domain events | Content edits require the application release path |
| Messaging control plane | Content teams may publish independently | Another permission, audit, and synchronization surface | High-volume editorial iteration with mature governance | Runtime content can drift from the application contract |
| Self-hosted renderer | Full control over data path and rendering | Highest capacity-planning and on-call load | Strict customization or isolation requirements | The team owns scaling, patching, and deliverability integration |
For the marketplace order alert, application ownership is the safer default because the template consumes versioned order fields and must roll back with them. Store the rendered subject and body, or enough immutable inputs to reproduce them under the recorded template version, according to the system's data-retention policy. Test the SMS length and email rendering before release, but also contract-test required variables: a beautiful preview with a missing seller_name is still a failed transaction.
This choice is not suitable when non-engineering teams must change regulated or time-sensitive copy independently several times a day. In that case, use a controlled messaging plane with roles, approval, version history, staged publication, and an application-visible template version. The decision is buy versus build, but the expensive column is usually ongoing ownership: localization review, preview generation, access control, audit history, cache behavior, and the person paged when rendering stops meeting its SLO.
Email authentication also sits outside the template engine. DMARC defines a domain-level policy and reporting mechanism built on SPF and DKIM alignment. Treat authentication records, sending-domain changes, and aggregate reports as production configuration with owners and review; a correct HTML template does not compensate for a broken domain policy.
Verify the rollout and make rollback dull
Start with deterministic tests: the same intent and template version should render the same channel payload, required fields should fail before submission, and duplicate queue deliveries should reuse the same idempotency key. Then run integration tests against a controlled mailbox and phone set, recording the provider message ID and every normalized transition. Do not use open tracking as the email success gate.
Deploy the new template version to a small cohort of seller IDs, while keeping the prior version addressable. Watch submission latency, permanent-failure rate, retry count, poll age, queue age, and the ratio of intents with no terminal observation after the age limit. Break those metrics down by channel and region; a global average can hide an EU backlog behind idle US capacity. Logs should connect order_id, intent_id, attempt_id, template version, and provider message ID, but should avoid raw message bodies and destinations unless access and retention are explicitly justified.
Rollback changes the active template pointer for new intents. Already-created intents keep their recorded version, which preserves auditability and prevents a retry from silently changing the seller's message. Pause a bad cohort, drain or quarantine its unsent intents according to business policy, restore the previous version, and replay only after checking the idempotency boundary. For a delivery-status change, disable the new poller, let leases expire, and resume with the prior classifier; never delete observations merely to make a dashboard green.
One final capacity check matters: prove that initial-send workers still meet their queue-age objective while the poll queue contains the largest plausible nonterminal population. If they share an unbounded pool, the design has no meaningful priority under stress. Fix that before launch.
Top comments (0)