TL;DR: Put email behind a durable, deduplicated alert queue, and make the rollback decision from metrics and traces rather than from an inbox. For a logistics notification service, the smallest useful event has a delivery ID, route, attempt number, release identifier, and reason. Persist it before asking an email worker to send anything.
A failed delivery notification is operationally awkward. The package may still be moving, the customer may still be waiting, and a second email can be as damaging as no email. I treat email as a projection of the error stream, not as the error stream itself. That distinction keeps a mail outage from becoming a second incident and gives a rollback a clean boundary.
Keep it boring.
For a Node.js service, configure the error boundary once and make it emit the same event whether the failure came from a callback, a promise rejection, or a provider timeout. That is the useful part of observability here: one searchable event model across the paths that can lose a delivery, rather than a separate email snippet in every handler. The application can then enqueue a message without knowing which mail transport will eventually deliver it.
What should an error email contain before a rollback?
The message should answer three questions quickly: which delivery is affected, what changed, and whether the failure is growing after the release. A useful event has a stable delivery identifier, notification channel, failure class, attempt number, release ID, and timestamp. Keep customer content out of logs and alerts unless the operation genuinely needs it; a delivery ID is usually enough to find the protected record.
The release ID matters more than a dramatic stack trace. If failures rise only for release-2026-10-30.2, the rollback candidate is obvious. If the same failure appears across releases, reverting code may only hide the symptom. My decision rule is simple: roll back a change when the failure is release-correlated, reversible, and materially affecting delivery completion.
Here is the smallest useful data flow:
type DeliveryFailure = {
eventId: string;
deliveryId: string;
channel: "email" | "sms" | "push";
attempt: number;
reason: string;
releaseId: string;
occurredAt: string;
};
type FailureStore = {
insertOnce(event: DeliveryFailure): Promise<boolean>;
};
type MailQueue = {
enqueue(message: {
dedupeKey: string;
subject: string;
text: string;
}): Promise<void>;
};
async function recordDeliveryFailure(
event: DeliveryFailure,
store: FailureStore,
mailQueue: MailQueue,
): Promise<void> {
const inserted = await store.insertOnce(event);
if (!inserted) return;
await mailQueue.enqueue({
dedupeKey: `${event.deliveryId}:${event.channel}:${event.reason}`,
subject: `Delivery notification failed: ${event.deliveryId}`,
text: [
`Delivery: ${event.deliveryId}`,
`Channel: ${event.channel}`,
`Attempt: ${event.attempt}`,
`Reason: ${event.reason}`,
`Release: ${event.releaseId}`,
`Occurred at: ${event.occurredAt}`,
].join("\n"),
});
}
insertOnce is the important boundary. Retries, process restarts, and duplicate provider callbacks should not create a new incident every time. The queue can retry delivery to the operator, while the stored event remains the canonical record for counting and investigation.
How do you keep email from deciding the incident?
Email is slow, lossy as a workflow, and easy to acknowledge without fixing anything. It is still useful for a small team, but it should be fed by measurements that support an explicit rollback decision. Count failures by reason and channel. Count attempted notifications separately from completed notifications. A ratio such as notification_failures_total / notification_attempts_total is more useful than an unqualified error count because traffic changes during a dispatch peak.
Use metric names that describe the measured thing and keep units in names where they apply. Prometheus recommends consistent naming practices, including a base unit suffix for measurements and a _total suffix for counters. The naming guide is a useful guardrail even when the metrics backend is not Prometheus.
A trace or structured log should carry the same delivery_id, release_id, and event_id as the searchable event fields. Do not put unbounded delivery IDs into metric labels; that creates a cardinality problem and exposes identifiers in a place intended for aggregation. Keep the ID in the event store and trace context instead.
The alert should be grouped. A single broken mail connection can produce thousands of delivery failures, but the operator needs one incident with a count, first-seen time, latest release, and top reason. Grouping by reason and release is usually a better starting point than grouping by delivery ID.
The rollback boundary is a data contract
A rollback is safe only when the notification service can tell old work from new work. Include the release ID in every failure event, and record the last successful notification time. When a rollback begins, freeze deploys, keep consuming the failure queue, and tag events emitted during the transition. Otherwise, the dashboard mixes two versions and makes the rollback look ineffective.
There is a subtle trap here. Reverting the application does not necessarily retry a notification that failed before the revert. Retry policy belongs to the delivery workflow, not to the deployment command. For each failure class, define whether it is retryable, manually reviewable, or terminal. A timeout may be retried with bounded backoff. An invalid destination may need correction. A policy rejection should not loop forever.
The trade-off is deliberate: one extra durable record creates a little more storage and schema work, but it prevents the rollback path from depending on the availability of the email transport. That is a better boundary for a solo team because the same record can feed an investigation, a replay, and a release comparison without asking an operator to reconstruct the incident from messages.
I also separate the operator notification from the customer notification. The operator email can say that a delivery attempt failed and include a runbook link. The customer-facing message needs its own idempotency key and policy. Reusing the same retry loop for both is how an internal alert accidentally becomes a duplicate customer email.
The simplest operational state machine has four states: observed, queued, sent, and suppressed. Suppressed is not the same as lost; it means a deduplication or rate policy intentionally prevented another email. Store the reason for suppression. During an incident, that field often explains why the inbox is quiet while the failure counter is rising.
A practical test before shipping
Test the boundaries that create duplicate or misleading evidence. Send the same event twice and assert that storage accepts it once. Make the queue unavailable after the event is stored and verify that the event remains visible for replay. Restart the worker after it has claimed a message but before it records success. The result should be a retry with the same dedupe key, not a new failure identity.
Then test rollback as an observable operation. Emit failures under one release, deploy an isolated change in staging, and confirm that dashboards can distinguish the releases. Check that the alert groups the burst and that a later successful notification closes the operational signal without deleting the failure history. I care more about this evidence than about a polished email template. That is the trade-off: a little schema work buys a much clearer rollback.
A staging fixture can stay deliberately boring:
const failureFixture: DeliveryFailure = {
eventId: "evt-7f3",
deliveryId: "del-1042",
channel: "email",
attempt: 2,
reason: "provider_timeout",
releaseId: "release-2026-10-30.2",
occurredAt: "2026-10-30T08:15:00.000Z",
};
Specific values expose whether a test checks identity, retry count, and release correlation instead of merely checking that a function returned.
Before production, I want one owner for the event schema, one retention policy for failure records, and one documented rule for when an alert is closed. I also want an inbox test with a non-customer address, a queue depth alert, and a metric for suppressed notifications. Those controls cost little compared with debugging a rollback from screenshots of an inbox.
The final checklist is prose because the order matters. First persist a normalized failure event. Then publish an idempotent mail job. Measure attempts, failures, completions, queue age, and suppressions separately. Correlate all of them with the release ID without using delivery IDs as metric labels. During rollback, preserve the queue and the evidence, and let the retry policy decide which delivery work resumes. If the alert cannot show those facts, it is not ready to drive a rollback.
The least complex reliable design is an event store plus a small mail worker, with metrics and release correlation making the rollback call. It keeps email useful without making email the source of truth, and it leaves room to replace the mail transport later without rewriting error tracking.
Further reading
- Prometheus, “Metric and label naming”: https://prometheus.io/docs/practices/naming/
- OpenTelemetry, “Context propagation”: https://opentelemetry.io/docs/concepts/context-propagation/
- MDN, “HTTP response status codes”: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status
Top comments (0)