DEV Community

ValdemarBlack3817
ValdemarBlack3817

Posted on

Node.js Observability: Set Up Error Tracking with Trace IDs Explained

#node #observability #healthtech #backend

Use one trace context to connect a failed request, its tenant cohort, and the decision to roll an experiment back. Keep the context immutable across the request, attach it to structured error events, and keep cohort labels out of high-cardinality metrics. That combination gives an on-call engineer a useful path from an error to a rollout decision without turning observability data into a second patient-data system.

I build email, SMS, and OTP flows, so I am suspicious of any design that treats delivery failure as a single log line. A timeout may be a dependency problem, a rate-limit response, or a cohort-specific configuration error. The same caution applies to a healthtech experiment: the trace is evidence for a rollback, not proof that every failure has one cause.

The trade-off is real.

What must remain true during a rollback?

The architecture has three invariants. Every inbound request gets a valid correlation context or an explicitly generated one. Every error event carries that context plus a low-cardinality tenant or cohort reference. Finally, the experiment assignment used by the request is recorded with the event, so a later query does not infer treatment from timing or from a mutable tenant table.

The failure boundaries matter more than the logging library. A worker must receive the context when a job is created, a retry must preserve the original trace relationship without pretending it is the same attempt, and an exception handler must record the error before the context is discarded. Redaction belongs at the event boundary. Identifiers such as patient names, message bodies, and one-time passwords do not belong in a trace label.

Design choice Useful property Rollback risk
Request-scoped context Easy to join application errors to one request Lost when work crosses a queue unless propagated explicitly
Explicit event envelope Stable fields for search, retention, and redaction More code at every async boundary
Metric labels for every identifier Fast aggregate dashboards Cardinality grows with tenants, requests, or trace IDs
Trace and log correlation Detailed path plus searchable error record Requires consistent propagation and clock-aware interpretation

The third row is the trap. Prometheus instrumentation guidance warns against labels whose values have high cardinality. A trace ID is excellent for finding one execution and poor as a metric label. Store it in logs or trace events; aggregate metrics by a bounded outcome such as error_type, cohort, or dependency class.

How should Node.js error tracking cross an async boundary?

Treat the trace context as data, not as ambient magic. In Node.js, request-local storage can make handlers convenient, but queue payloads, scheduled jobs, and outbound calls still need an explicit propagation rule. The event envelope should contain the trace ID, the current span ID when available, tenant or cohort key, experiment revision, attempt number, and a normalized error type. It should not contain the full exception payload by default.

Here is the critical path in deliberately small pseudocode. The surrounding application can use its own Node.js framework; the contract is the part worth testing.

Test the boundary.

def handle_request(request, experiment):
    context = read_or_create_trace_context(request.headers)
    assignment = experiment.assign(tenant_id=request.tenant_id)

    try:
        result = run_health_operation(
            tenant_id=request.tenant_id,
            cohort=assignment.cohort,
            trace_id=context.trace_id,
        )
        return result
    except Exception as exc:
        event = {
            "event": "application_error",
            "trace_id": context.trace_id,
            "span_id": context.span_id,
            "tenant_key": request.tenant_key,
            "cohort": assignment.cohort,
            "experiment_revision": experiment.revision,
            "attempt": 1,
            "error_type": normalize_error_type(exc),
        }
        emit_redacted_error(event)
        raise
Enter fullscreen mode Exit fullscreen mode

The important detail is experiment_revision. If the cohort assignment changes while an incident is investigated, a bare tenant ID cannot tell you which behavior the request actually exercised. I also record an attempt number because a retry is a separate observation. Otherwise, a dependency that fails three times looks like three independent tenants in a noisy dashboard.

That distinction saves the rollback review.

For an outbound call, propagate a standards-based trace context and create a child span for the dependency operation. For a queue, put the parent relationship and the business fields in a versioned envelope. On consumption, create a new processing span linked to the enqueue operation. This keeps the causal story readable without pretending that minutes of queue delay happened inside the original HTTP span.

Which option survives a cohort experiment?

There are two reasonable storage shapes. A log-first design writes structured error events and uses traces for timing. A trace-first design makes the trace the primary investigation record and derives error views from it. I prefer the first for rollback decisions in healthtech systems because retention, redaction, and access control for an error event can be reviewed as a bounded contract. Traces remain valuable for dependency timing, but they should not become the only copy of the decision evidence.

The choice is operational, not ideological. If a team has mature trace sampling and a well-defined error-event stream, either shape can work. Sampling is the sharp edge: a sampled-out successful request is usually fine, while a sampled-out error can remove the comparison point needed to judge a rollout. Error events should therefore have an independent retention and delivery policy, with a bounded payload and a clear failure mode when the telemetry sink is unavailable. The business request must not fail merely because telemetry failed. The limitation is that a telemetry pipeline can preserve correlation while still losing events during an outage, so rollback evidence needs an explicit completeness check.

Do not use a dashboard threshold as the rollback algorithm by itself. Compare treatment and control using the same observation window, normalize by eligible requests, and split results by dependency class and cohort. A spike in one tenant can indicate a tenant configuration issue rather than a bad experiment. A spike only after a dependency timeout deserves a different response from a spike in validation failures.

What did I reject, and when is it still valid?

I initially thought putting trace_id into every metric label would make the first incident easier. It feels convenient when searching for one request is the immediate need. It becomes an operational liability once traffic grows: each request creates another time series, dashboards become expensive to query, and the aggregate signal gets harder to read. The valid use case is a short-lived local diagnostic metric with a tightly bounded identifier set, not a production request counter. The trade-off is search convenience today versus a usable aggregate tomorrow.

I also rejected copying raw exception text into the event as the primary field. Exception messages often contain input fragments or dependency responses. Normalize the type and preserve a redacted, access-controlled detail field only when the investigation requires it. That is a compliance boundary, not a formatting preference.

A rollback-safe review should answer four questions: can we identify the exact experiment revision, can we join the error to its trace without exposing sensitive payloads, can we distinguish retries from new requests, and can we compare cohorts without trace-level metric cardinality? If any answer is no, fix the evidence path before trusting the experiment result.

The minimum implementation is therefore modest: propagate a trace context, emit a redacted structured error event, record cohort and revision, and aggregate metrics only on bounded dimensions. In a healthtech rollout, that is more useful than a dense dashboard full of identifiers. It leaves the team with a defensible rollback signal and a smaller blast radius when the next dependency behaves badly.

References

Top comments (0)