FerdinandBlake3517

Posted on Nov 5

Node.js Error Tracking Library — Integrate Practices for Notification Cost Attribution

#node #observability #errortracking #notifications

A useful Node.js error-tracking integration for a notification service starts with one accounting rule: attribute cost to the delivery attempt, while grouping errors by the stable failure cause. That preserves the difference between one bad destination and ten retry attempts, and it keeps an observability bill from becoming a second source of confusion.

Short answer: capture a normalized error event at the boundary where the notification attempt is created, attach a redacted operation and delivery identity, and keep retry state as structured fields. Store the full event only for a bounded retention window. Keep a small aggregate keyed by failure class, channel, and tenant so an engineer can answer both questions: what failed, and what did it cost?

The hard part is deciding what counts as an event. A thrown exception is an implementation detail. A delivery attempt is an operational fact.

That distinction is the whole design.

Start with the bill, then choose retention

For a developer-tools notification service, the dominant cost is often repeated data rather than the initial error. A single SMS or email can produce an accepted attempt, a provider timeout, a retry, and a final permanent failure. If each record includes the request body, stack trace, user address, and every response, the same business event gets copied several times. That may help a short incident investigation, but it is a poor default retention model.

I start with three counters per tenant and channel: attempts, transient failures, and permanent failures. Each counter is keyed by a stable failure class such as timeout, rate_limited, invalid_destination, or unknown. The aggregate is cheap to retain and useful for cost attribution. It also prevents a common mistake: interpreting a high exception count as a high sending cost when a single malformed address is being retried by policy.

The detailed error event still matters. I retain it for the period needed to investigate delivery behavior, with payload fields removed or hashed according to the data policy. After that, the aggregate remains while the raw event expires. This is a deliberate trade: a late investigation may lose the exact stack and response body, but it does not lose the number of attempts, the tenant that incurred them, or the failure class that drove them.

A practical event shape looks like this:

from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class DeliveryFailure:
    notification_id: str
    attempt: int
    tenant_id: str
    channel: str
    failure_class: str
    retryable: bool
    elapsed_ms: int
    error_type: str
    trace_id: str | None


def to_log_fields(failure: DeliveryFailure) -> dict[str, Any]:
    return {
        "event_name": "notification.delivery.failure",
        "notification_id": failure.notification_id,
        "attempt": failure.attempt,
        "tenant_id": failure.tenant_id,
        "channel": failure.channel,
        "failure_class": failure.failure_class,
        "retryable": failure.retryable,
        "elapsed_ms": failure.elapsed_ms,
        "error_type": failure.error_type,
        "trace_id": failure.trace_id,
    }

The notification identifier supports reconstruction without making the message body part of the event. attempt is essential: without it, a retry storm looks like many independent notifications. A cost ledger can then charge or report per attempt while the incident view groups all attempts under one notification.

How should a Node.js library integrate error tracking practices?

The integration point belongs around the delivery adapter, not around every try statement in the application. In Node.js, rejected promises can cross several layers before they reach a worker boundary. Reporting them at the adapter gives the event a channel, attempt number, elapsed time, and classification at the same place. Reporting them higher up usually loses at least one of those fields.

Classification should be explicit and boring. A timeout can be retryable. An invalid destination normally is not. A rate limit may be retryable under a policy that includes a delay, while an authentication failure should page an operator instead of multiplying attempts. The classifier should consume a small adapter error contract rather than parse arbitrary message text.

from collections.abc import Mapping


def classify_delivery_error(error: Mapping[str, object]) -> tuple[str, bool]:
    code = error.get("code")
    status = error.get("status")

    if code in {"ETIMEDOUT", "ECONNRESET"}:
        return "timeout", True
    if status == 429:
        return "rate_limited", True
    if status in {400, 422}:
        return "invalid_destination", False
    if status in {401, 403}:
        return "authentication", False
    return "unknown", False

This is intentionally not a list of provider routes. It is a boundary contract. The concrete adapter can map its response into code and status, while the rest of the service observes the same dimensions across email, SMS, and OTP.

The Node.js side should also preserve the original error as the cause when it adds context. That keeps stack information available to the error tracker without replacing the useful low-level failure with a generic DeliveryError. Do not log the destination address or message content by default. Those fields are tempting during debugging and expensive from a privacy and retention standpoint.

This design has a boundary: it is a poor fit when an audit requires reconstructing every message body and provider response indefinitely. The trade-off is deliberate: a separate restricted archive with a documented legal retention policy is needed in that case; error tracking should not become that archive by accident.

How do you keep retries from corrupting cost attribution?

A retry is a state transition, not a new notification. Give every notification one durable identifier and every attempt a monotonically increasing number. The worker writes the attempt record before dispatch, then updates it with the result. If the process dies after dispatch and before the update, the next worker must treat the outcome as uncertain. It should not silently convert that uncertainty into a successful send.

This is where idempotency and observability meet. The error event needs a deduplication key such as (notification_id, attempt), while the cost ledger needs an append-only attempt fact or an equivalent idempotent upsert. A tracker that deduplicates every identical exception globally can hide real retry volume; a ledger that counts every log line can inflate it. Keep those concerns separate.

Here is the small piece of accounting I want near the worker boundary. It does not decide retry policy; it records the result that policy produced.

from collections import Counter


def update_cost_counters(
    counters: Counter[tuple[str, str, str]],
    tenant_id: str,
    channel: str,
    failure_class: str,
) -> None:
    counters[(tenant_id, channel, failure_class)] += 1


def attempt_cost_key(tenant_id: str, channel: str, attempt: int) -> tuple[str, str, int]:
    return tenant_id, channel, attempt

The real implementation must enforce uniqueness in durable storage. An in-memory Counter is only a readable model of the dimensions. It is not a ledger, and it cannot survive a worker restart. That distinction is easy to miss in a library example because the code looks complete before deployment.

Keep this part plain. The counter is a model, not the accounting system.

For structured logs, use a schema that your collector can forward as logs or map into an error signal. OpenTelemetry describes logs as records with a timestamp, body, attributes, and context that can correlate them with traces. The standard gives you transport-neutral vocabulary; it does not choose your retention period, retry semantics, or cost model. Those remain application decisions.

What should the integration test before production?

Test the failure matrix, not just the happy path. At minimum, inject a timeout, a rate-limit response, an invalid destination, a worker restart after dispatch, and a duplicate delivery callback. Assert the resulting failure class, retryable flag, attempt number, and cost key. Also assert that sensitive payload fields are absent.

The most valuable test I added to this kind of boundary is a replay test: feed the same delivery result twice and verify that the detailed error may be observed twice but the cost fact is counted once for the same attempt. Then feed attempt 2 and verify that it is counted separately. That catches the two opposite bugs that matter here.

Make the test data uneven. Use one notification with one attempt, one with three attempts, and one with an unknown outcome. A perfectly uniform fixture can pass while the aggregation logic is wrong.

In production, alert on ratios and accumulation, not raw exceptions alone. A rising unknown class signals a mapping gap. A high retryable-failure ratio suggests a channel or dependency issue. A growing permanent-failure count for one tenant may indicate bad input or abuse. Each alert should link to the structured event and the aggregate, with access controls appropriate to tenant data.

The decision rule

Use error tracking to preserve diagnostic context, and use a separate, idempotent attempt record to explain delivery cost. Keep the event schema small enough that retention is defensible. Classify at the adapter boundary, correlate with a trace when one exists, and make retry behavior visible through attempt numbers.

I would stop keeping raw response bodies once they no longer change an operational decision. I would keep the failure class, attempt count, tenant, channel, and timestamps longer because those fields answer the questions finance, support, and on-call engineers actually ask. The choice is less glamorous than adding another integration, but it makes the resulting numbers explainable.

DEV Community