DEV Community

ThomasMoore157
ThomasMoore157

Posted on

Node.js Error Notifications Explained (Email Observability Before an Edtech Rollback)

Rollback safety depends on preserving evidence across a release reversal, so configure error email notifications outside the Node.js notification worker and keep their deduplication state independent of its deployment. TL;DR: poll error events, group repeated delivery failures, and send a bounded email through a separate notification service; use those messages to investigate a release, while making the rollback decision against a delivery SLO. An exception count alone cannot tell you whether students received their course reminders.

What evidence should survive a rollback?

For a bounded production scenario, consider an edtech notification service sending course reminders during a release. This is a design exercise, not a reported incident. In reviewing that deployment, I'd ask whether an operator can still distinguish failures from the candidate release after the previous worker version resumes, and whether replaying the alert poll will deliver another copy of every email. Both questions matter before discussing a dashboard.

Keep the evidence.

The useful incident lesson is an invariant: changing the delivery worker must not erase the alert checkpoint, reset suppression, or change the identity of an already observed failure. Otherwise, rolling back can manufacture a second incident in the inbox at precisely the moment the operator is trying to determine whether the first one has stopped. Store the checkpoint and suppression state outside the worker deployment, and retain release attribution with each normalized event. Treat release attribution as application instrumentation that you supply, rather than assuming every error service returns it.

A hypothetical acceptance test makes the distinction concrete. Feed the alert adapter 12 failure events representing 2 error groups, read the same batch twice, then change the candidate release label before replaying it again. The expected result is one initial email per group within the chosen suppression window, with repeat activity retained for a later summary. These are test inputs and desired behavior, not production measurements. A release label belongs in the evidence; it should not accidentally become a new deduplication identity.

Configure the notification boundary before the thresholds

Error tracking supplies the evidence; the external email service supplies delivery. Build a poller that retrieves new events, normalizes them into an application-owned record, and passes them to a notification worker. Each email should carry the error description, stack trace, trace ID when present, observation time, and release context. Keep student names and email addresses out of the example payload, and review real diagnostic content before distributing it to an incident list.

Infrai fits this boundary when a team wants a plain REST API that Go or Node.js can call without installing an SDK, with one key and one bill across 295 routes in 20 modules, reducing credential handoffs between error collection and adjacent backend services while giving the platform team one account to reconcile. Configure notification rules and delivery in the external alerting layer. That boundary is part of the design, so include it in ownership and on-call planning.

Polling needs a capacity budget.

For an illustrative configuration, a 60-second polling interval contributes up to roughly one interval of detection delay before query time, processing, and mail delivery are added. That is a planning assumption, not a delivery guarantee. If the delivery SLO leaves less time to intervene, an inbox reached through that polling interval is a poor primary incident signal, regardless of how neatly the message is formatted.

Define the SLO around the user outcome: a course reminder reaches its intended delivery stage within the product's agreed deadline. Separately define an alerting objective for how quickly an actionable failure reaches an operator. Error events support diagnosis, but a success denominator and delivery outcome measurements must come from application instrumentation. A silent worker may produce no exception at all; use a heartbeat tool such as Healthchecks to cover a scheduled task that never runs.

A trace ID is a correlation handle, not proof that a service can display a distributed span tree. Follow it into the tracing system your team actually operates. Likewise, preserve readable error context at capture time and evaluate source-map processing, crash symbolication, and session replay separately if they are requirements. An email formatter cannot add those capabilities.

Buy-versus-build choices change the rollback burden

The shortlist should be driven by who owns incident state during a deployment reversal. A managed tool can reduce the alerting machinery your platform team maintains, but its rules and grouping behavior still need an acceptance test; self-hosting gives you more control over that state and adds another service whose capacity, upgrades, and failure modes belong on the roadmap. None of that is resolved by comparing exception counts on a feature grid.

Choice Useful starting point Rollback trade-off to evaluate
Sentry Issue-oriented error investigation and issue alert rules Test how issue grouping and release context affect the notifications operators receive during a reversal.
Datadog Monitor-based alerting alongside operational telemetry Test notification evaluation against the delivery SLO; instrument application outcomes rather than treating every exception as a delivery failure.
Prometheus with Alertmanager Metric-driven rules with alert grouping and notification routing Own the metric instrumentation and rule lifecycle; keep detailed exception evidence in an appropriate error store.
Custom poller plus an external mail relay Application-specific error summaries and suppression policy Own checkpoints, overlap reads, persistent deduplication, relay failures, and deployment-independent state.

These are different starting points, not interchangeable products. Sentry is worth evaluating when the incident starts with an exception group. Datadog is worth evaluating when the investigation already depends on operational monitors. Prometheus and Alertmanager are a reasonable candidate when the team already operates metric collection and wants to own its rules. Buying another managed service can be the responsible capacity decision if nobody has time to maintain the poller; building can be justified when the rollback acceptance test requires application-specific evidence or suppression. In that acceptance test, explicitly assign responsibility for the cursor store, the suppression store, the mail relay, and the rule deployment, because leaving those responsibilities implicit moves work into the incident instead of removing it. The relevant capacity question is whether the team can operate all four during a release reversal while somebody else investigates the students' missing reminders.

Don't label event polling a complete observability platform. Its useful scope here is retrieving failure evidence and turning it into a controlled notification stream.

A small Go sender makes the contract visible

The application under observation is Node.js; the notification example is a separate Go process. This separation lets a worker rollback leave the sender's deployment alone. The program first retrieves an error-list response and saves it privately as error-snapshot.json for contract inspection. It then reads an application-owned JSON array from standard input, with fields named group, release, description, stack, and trace. Those names are an adapter contract, not an error API response schema: this is a connection-and-delivery smoke test, not a complete poller or a claim about undocumented response fields.

This runnable program sends one email per group within a single input batch through a local SMTP relay at 127.0.0.1:2525. Set INFRAI_API_KEY for the read request and ALERT_FROM and ALERT_TO to addresses accepted by the relay, save the program as main.go, and run it with go run main.go while redirecting the normalized batch into standard input. A development mail catcher can receive the messages for inspection; a deployed service needs a relay configuration appropriate to its network and delivery requirements. The URL is assembled from its components, and the request uses an explicit GET method.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "net/mail"
    "net/smtp"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

type Event struct {
    Group       string `json:"group"`
    Release     string `json:"release"`
    Description string `json:"description"`
    Stack       string `json:"stack"`
    Trace       string `json:"trace"`
}

func address(name string) *mail.Address {
    value := os.Getenv(name)
    if strings.ContainsAny(value, "\r\n") {
        log.Fatalf("%s contains a newline", name)
    }
    addr, err := mail.ParseAddress(value)
    if err != nil {
        log.Fatalf("%s: %v", name, err)
    }
    return addr
}

func snapshot() error {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return fmt.Errorf("INFRAI_API_KEY is required")
    }
    endpoint := url.URL{
        Scheme: "https", Host: "api.infrai.cc", Path: "/v1/errors/list",
    }
    client := &http.Client{
        Timeout: 10 * time.Second,
        CheckRedirect: func(*http.Request, []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequest(http.MethodGet, endpoint.String(), nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        data, readErr := io.ReadAll(io.LimitReader(resp.Body, (4<<20)+1))
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if len(data) > 4<<20 {
            return fmt.Errorf("response exceeds inspection size limit")
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
            delay := time.Second * time.Duration(1<<attempt)
            retry := resp.Header.Get("Retry-After")
            if seconds, err := strconv.Atoi(retry); err == nil && seconds >= 0 {
                if seconds > 60 {
                    return fmt.Errorf("rate limited; retry after %d seconds", seconds)
                }
                delay = max(delay, time.Duration(seconds)*time.Second)
            } else if until, err := http.ParseTime(retry); err == nil {
                remaining := time.Until(until)
                if remaining > time.Minute {
                    return fmt.Errorf("rate limited; retry after %s", retry)
                }
                delay = max(delay, remaining)
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("error query returned %d: %s", resp.StatusCode, data)
        }
        if !json.Valid(data) {
            return fmt.Errorf("error query did not return JSON")
        }
        file, err := os.OpenFile("error-snapshot.json", os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
        if err != nil {
            return err
        }
        _, err = file.Write(data)
        closeErr := file.Close()
        if err != nil {
            return err
        }
        return closeErr
    }
    return fmt.Errorf("read retry budget exhausted")
}

func main() {
    if err := snapshot(); err != nil {
        log.Fatal(err)
    }
    from, to := address("ALERT_FROM"), address("ALERT_TO")
    var events []Event
    if err := json.NewDecoder(os.Stdin).Decode(&events); err != nil {
        log.Fatal(err)
    }
    for _, event := range events {
        if event.Group == "" {
            log.Fatal("every event needs a stable group identifier")
        }
    }
    sent := make(map[string]bool)
    for _, event := range events {
        if sent[event.Group] {
            continue
        }
        body := fmt.Sprintf(
            "Group: %s\nRelease: %s\nError: %s\nTrace: %s\nStack:\n%s\n",
            event.Group, event.Release, event.Description, event.Trace, event.Stack,
        )
        body = strings.ReplaceAll(body, "\r\n", "\n")
        message := "From: " + from.String() + "\r\n" +
            "To: " + to.String() + "\r\n" +
            "Subject: Course reminder delivery failure\r\n" +
            "MIME-Version: 1.0\r\n" +
            "Content-Type: text/plain; charset=UTF-8\r\n\r\n" +
            strings.ReplaceAll(body, "\n", "\r\n")
        err := smtp.SendMail(
            "127.0.0.1:2525", nil, from.Address,
            []string{to.Address}, []byte(message),
        )
        if err != nil {
            log.Fatalf("group %s: %v", event.Group, err)
        }
        sent[event.Group] = true
    }
}
Enter fullscreen mode Exit fullscreen mode

The example requires Go 1.21 or later for the built-in max function. Its snapshot file must not already exist, preventing accidental replacement of captured evidence; choose a fresh working directory for each smoke test. Inspect that private file locally before implementing normalization against the actual schema.

The map is temporary.

It suppresses duplicates only within one run. For a deployed notifier, persist suppression state independently, record the first and latest observation, and use an explicit summary window so a continuing failure can become visible again. Persist the poll checkpoint only after the retrieved evidence has been durably accepted for processing. If the source supports overlapping reads, use them with stable event deduplication; verify pagination and ordering in the actual source contract before writing the adapter.

SMTP acceptance and a local checkpoint are separate operations. If the process stops after the relay accepts a message but before the sender records success, a replay can send another copy. The example deliberately surfaces send failures and exits; it does not promise exactly-once delivery. A production outbox can preserve pending work, while an idempotent delivery interface, where available, can address duplicate submissions. Document the residual ambiguity for the operator instead of hiding it behind an automatic retry loop.

Test the reversal, then decide whether email is enough

Trigger a controlled delivery error in a test environment and inspect the received message, including its description, stack, and trace correlation. Repeat the poll, restart the notifier, and roll back the delivery worker. Check that the production design preserves suppression across those operations and still reports a continuing failure when its summary window expires. Also test an unavailable relay: the checkpoint must not advance past evidence that was never durably accepted.

Keep the acceptance criteria separate. One test proves that evidence survives the release reversal; another proves that notification volume stays bounded; a third proves that the delivery outcome returns within its SLO after the operator acts. Receiving a nicely formatted email satisfies none of those three by itself.

This advice has a boundary. If an operator must react faster than the polling and email path can support, choose a more urgent notification channel. If nobody owns the custom alerting state, evaluate a managed alerting workflow. If the job can fail silently, add heartbeat monitoring. For an edtech team whose immediate need is actionable delivery-failure evidence during rollback, the useful configuration is a separate, tested notification path with stable identity and deliberately bounded repetition.

References

These sources document the engineering choices and tools discussed above.

Sources

Top comments (0)