Unhandled API Rejections in Pricing Rollouts (Preserving Request Context and Stack Traces)

#observability #node #express

Short answer: ship the new pricing rule behind a flag only after handled request errors, unhandled promise rejections, and uncaught exceptions produce one common event shape; correlate that event with a low-cardinality rollout metric, and make the rollback decision independent of the error-reporting backend.

The page should say which pricing revision failed and whether rollback is still possible. A stack trace by itself can't answer either question. For an education platform changing subscription or course-bundle pricing, the useful unit of evidence is the failed decision: release, flag revision, stable route template, request correlation identifiers, error chain, and outcome, with student and payment data excluded.

This is a rollback design, not a dashboard design. At 03:00, a graph that says “errors are up” transfers the investigation to the person carrying the pager; an event contract tied to the rollout tells automation and the responder what action remains safe.

Which evidence earns a page and triggers rollback?

Page on user-visible failure and rollback risk, not on every exception object. An Express request that reaches centralized error middleware is a handled request failure even when its HTTP response is 500; an exception that escapes the request path is a process-integrity failure. Those cases can share fields, but they shouldn't share urgency or recovery policy. RFC 5424 defines distinct severity levels, and the practical lesson is to document a stable mapping instead of calling every log “error” and every error “critical.”

The pricing flag adds a second dimension: did the candidate rule make the request fail, or did a control request fail for the same unrelated reason? Record pricing_rule_revision and flag_variant on the error event, then expose counters such as pricing_rule_requests_total and pricing_rule_errors_total with bounded labels such as revision, route template, and outcome. Prometheus recommends base units and the _total suffix for accumulating counters. It also warns that every label combination creates another time series, so request IDs, stack traces, email addresses, course IDs, and raw URLs belong in neither metric labels nor page grouping keys.

No page from one sample.

The actual rollback threshold depends on traffic and the harm of a wrong price. I'm not sure a universal percentage exists, because a checkout endpoint with sparse traffic and a read-only catalog endpoint have different evidence requirements. Resolve that uncertainty before launch: declare a minimum sample, a comparison window against the control variant, an absolute failure ceiling, and the owner authorized to disable the flag. If the candidate crosses the agreed boundary while control stays normal, rollback can be automatic. If both move together, disabling the pricing rule may hide the symptom without addressing the cause.

How should a Node.js Express API capture an unhandled exception with request context?

Start request context at the first middleware boundary. Accept a caller's correlation ID only after validating its length and character set, otherwise generate one; attach the route template rather than the raw URL; add the current release and pricing-rule revision; then place that small immutable context in AsyncLocalStorage. Node documents AsyncLocalStorage as the API for keeping data coherent through asynchronous operations. Keep the context deliberately boring. Request bodies, authorization headers, query strings, student names, and payment details create a privacy incident when copied into an error tracker.

Next, make ordinary Express failures converge on one error middleware. Synchronous throws reach Express error handling, and the Express documentation says rejected promises from route handlers call next with the rejection value. The middleware should normalize non-Error rejection values, walk the cause chain, capture the stack at the point where the error was created, add the current async request context, emit exactly one event, and return the response. Put an internal “reported” marker on the normalized error or use one capture boundary; otherwise a route wrapper, error middleware, and process hook can turn one defect into three alerts.

Process-level hooks are different. Node's unhandledRejection event identifies a promise that lacked a handler within a turn of the event loop, while uncaughtExceptionMonitor observes an uncaught exception without changing the default crash behavior. Use those hooks for a final, minimal capture when request middleware cannot own the failure. Don't treat uncaughtException as a route-level recovery mechanism: Node's own guidance warns that resuming normal operation after an uncaught exception is unsafe. Stop accepting work, attempt a bounded telemetry flush, and terminate non-zero so the supervisor can replace the process.

That boundary matters. A process hook might still see request_id through asynchronous context, but it must work when no request exists, so every request field is optional. The event also needs a handled boolean and an origin enum such as express_middleware, unhandled_rejection, or uncaught_exception. OpenTelemetry's exception semantic conventions provide the portable core: exception type, message, and stack trace. W3C Trace Context provides the interoperable traceparent mechanism; store the parsed trace and span identifiers, not a hand-edited copy of an arbitrary header.

Choose the capture boundary by failure consequence

The collector contract below is intentionally small. It doesn't install a reporter in the application and it doesn't care where events are stored. It gives CI a way to reject events that cannot support triage or rollback, while allowing process-level events to omit request fields. In production, perform equivalent validation at the capture boundary and again at ingestion, because telemetry that quietly changes shape is operationally indistinguishable from missing telemetry.

Capture boundary	Evidence it owns	Rollback value	Main limitation
Express error middleware	Route template, request context, response outcome	Attributes a handled request failure to candidate or control	Cannot observe work that escapes the request pipeline
Process event hook	Fatal origin, optional async context, last-chance stack trace	Distinguishes a bad instance from a rule-only failure	Delivery is best effort while the process is terminating
Supervisor-owned collector	Process lifecycle and locally forwarded structured events	Preserves evidence independently of application shutdown	Has less application context unless the event contract supplies it

package errorevent

import (
    "errors"
    "fmt"
)

type Origin string

const (
    ExpressMiddleware  Origin = "express_middleware"
    UnhandledRejection Origin = "unhandled_rejection"
    UncaughtException  Origin = "uncaught_exception"
)

type RequestContext struct {
    RequestID     string `json:"request_id"`
    Method        string `json:"method"`
    RouteTemplate string `json:"route_template"`
    TraceID       string `json:"trace_id,omitempty"`
}

type Event struct {
    SchemaVersion       int             `json:"schema_version"`
    OccurredAt          string          `json:"occurred_at"`
    Service             string          `json:"service"`
    Release             string          `json:"release"`
    Origin              Origin          `json:"origin"`
    Handled             bool            `json:"handled"`
    ExceptionType       string          `json:"exception_type"`
    ExceptionMessage    string          `json:"exception_message"`
    ExceptionStacktrace string          `json:"exception_stacktrace"`
    PricingRuleRevision string          `json:"pricing_rule_revision,omitempty"`
    FlagVariant         string          `json:"flag_variant,omitempty"`
    Request             *RequestContext `json:"request,omitempty"`
}

func (e Event) Validate() error {
    if e.SchemaVersion != 1 {
        return fmt.Errorf("schema_version: expected 1, got %d", e.SchemaVersion)
    }
    if e.OccurredAt == "" || e.Service == "" || e.Release == "" {
        return errors.New("event identity fields are required")
    }
    if e.ExceptionType == "" || e.ExceptionStacktrace == "" {
        return errors.New("exception type and stack trace are required")
    }
    if e.Origin == ExpressMiddleware && e.Request == nil {
        return errors.New("request context is required for middleware failures")
    }
    if e.Request != nil && e.Request.RouteTemplate == "" {
        return errors.New("use a route template, never a raw request URL")
    }
    return nil
}

Version the envelope. Producers may add optional fields without forcing a coordinated release, but changing meaning requires a new schema version and a compatibility test. Also test serialization of nested causes and non-Error promise rejection values. A reporter that handles only throw new Error(...) will look healthy in a tidy unit test and lose the awkward failures that motivated the process hook.

There is a less obvious test: deliberately make the sink unavailable in an isolated test environment and verify that business requests don't wait on it. Run one candidate request that fails in Express middleware, one background promise that rejects after the response, and one child process that throws outside any request; hold the sink unavailable until the bounded queue fills, then restore it. The first event must retain its route template and flag revision, the second must identify unhandled_rejection without borrowing stale context from the completed request, and the third must identify uncaught_exception even though its request object is absent. Meanwhile, successful control requests must keep returning normally. This exercise tests isolation, context lifetime, queue bounds, and schema validation in one sequence, and it exposes duplicate reporting far more reliably than staring at an empty dashboard. Error capture must use bounded queues, bounded payloads, and bounded flush time. Otherwise the observability path becomes part of pricing availability — exactly the coupling a flag was supposed to reduce. The catch is that an in-process queue cannot guarantee delivery after abrupt termination. When loss on hard exit is unacceptable, write structured events to a local out-of-process collector or another supervisor-owned channel; don't promise durability from a dying process.

Prove rollback safety with candidate, control, and missing telemetry

A safe rollout starts before production. Exercise the control and candidate rules with the same boundary cases: missing price inputs, unsupported currency or region, stale rule revision, downstream timeout, and a deliberate rejected promise. The expected assertions are more useful than a screenshot: one client response, one normalized event, the correct variant and revision, a stable route template, a non-empty exception type and stack trace, no restricted request data, and a counter increment with the same bounded dimensions.

Then canary the candidate to a small cohort chosen by the release system, but don't bake a supposedly universal percentage into the runbook. Compare candidate and control using the predeclared sample and time boundaries. A release is healthy only if requests still complete, event ingestion still works, and the candidate's error behavior stays inside the rollback budget. Silence is ambiguous. It can mean no errors, broken capture, a blocked egress path, or a process that died before flushing, so send a synthetic handled failure through the full path and alert separately when its expected event disappears.

Dashboards come last.

For paging, join the metric signal to deploy and flag metadata, then link to grouped events by release, pricing revision, origin, exception type, and route template. Keep the raw stack available for diagnosis but out of the page title. The responder needs “candidate revision exceeded its error budget; control is stable; rollback owner is on call,” not the first line of a minified stack repeated 400 times.

Separate pricing rollback from fatal process recovery

Rollback the flag when the candidate breaches its declared boundary and the control does not. Preserve the failing revision in every event even after the flag changes, because queued requests and delayed telemetry can arrive under the old decision. Continue observing both variants through the drain window, record who changed the flag and when, and verify recovery against the same metric that triggered the rollback. A falling alert count alone isn't proof; grouping or delivery may have changed.

Do not use a feature-flag rollback to keep a process alive after an uncaught exception. These are separate controls. The flag restores the last accepted pricing behavior for new requests; the process supervisor replaces an instance whose state can no longer be trusted. If graceful drain exceeds the bounded deadline, exit anyway. Slow shutdown is a diagnostic clue, not permission to run indefinitely.

This pattern is not suitable when the runtime can disappear without enough time for an in-process flush, when policy forbids stack traces outside the host, or when request volume is too low for a comparative rollback threshold. Use a supervisor-owned local collector for the first case, keep redacted evidence inside the approved boundary for the second, and require an explicit human decision based on deterministic pricing tests for the third. Likewise, stick with simple centralized middleware alone when the service has no background promises and the runtime supervisor already captures fatal stderr reliably; extra process listeners add policy and testing work.

After recovery, write the postmortem around the control failure, not around the graph: which page fired, which invariant distinguished candidate from control, whether one exception became one event, whether the flag changed before user harm expanded, and whether the process terminated as designed. If the answer requires opening five dashboards, the telemetry contract still has work to do.

DEV Community