Pricing Rule Rollouts: A Failure-Containment Plan for Node.js Error Tracking
The least complex safe choice is to treat an uncaught exception as a process-fatal event: record the error synchronously, stop accepting new work, and let a supervisor replace the process. For a logistics pricing rule behind a flag, that gives the rollback path a clear boundary. The flag can return to the previous rule while the unhealthy worker drains or exits.
The tempting alternative is to catch uncaughtException, log it, and keep serving quotes. That creates a process whose memory, sockets, and in-flight state may no longer be trustworthy. Error tracking is useful here because it explains why a worker stopped. It is not a license to continue after an unknown state transition.
The field guide: which response fits the failure?
| Failure shape | Response | Pick this when | What error tracking must preserve |
|---|---|---|---|
| A request-level validation or business error | Return a controlled error and continue | The failure is expected and the process state is known | Rule version, shipment identifier, and a bounded error code |
| A rejected promise with a known boundary | Handle it at that boundary | The operation can be discarded without corrupting shared state | Rejection cause, operation id, and whether a retry was attempted |
| An uncaught exception or unhandled rejection | Capture, stop intake, drain briefly, then exit | The process may be inconsistent | Stack, release, flag state, and shutdown reason |
| A dependency timeout during a rollout | Fail closed or use the last known-good rule | A quote must not silently use a partial calculation | Dependency timing, selected rule, and fallback decision |
This is a decision table, not a promise that every incident has one perfect response. The important split is known failure versus unknown process state.
For a controlled request error, use an ordinary try/catch around the operation that can fail. For a fatal process event, use a last-resort handler only to report and shut down. Node.js documents uncaughtException as a mechanism for synchronous cleanup before termination, and warns against using it as a way to resume normal operation.
What are the best practices for handling uncaught exceptions in error tracking?
A quote worker has three clocks: the request clock, the rollout clock, and the process clock. A request may be retried in milliseconds. A flag may be changed in seconds. A process supervisor may need longer to replace a worker. The error record has to connect those clocks without making the error handler responsible for pricing.
Picture the flow as a line: quote request enters, flag selects rule, calculation emits a quote, telemetry records the decision. An uncaught exception breaks that line after an unknown step. The safe response moves traffic away from that process, marks the rollout suspect, and leaves the pricing decision to the flag controller.
I start with the rollback question: “Can an operator identify the last known-good rule from one event?” If the answer is no, adding another dashboard will not fix the operating model. Put the rule version and flag revision in structured context before the calculation begins. Keep customer data out of the event unless it is explicitly required and protected.
The OpenTelemetry logs model is useful language for this design: a log record carries a body plus attributes and can be associated with trace and span context. In practice, that means the fatal event should be one structured record with stable fields, rather than a stack trace followed by several unrelated strings. That record is the handoff between the dying worker and the operator deciding whether the flag should move back. It should answer what happened, which release was serving the request, which rule revision was selected, and whether intake had already stopped. If those answers are scattered across asynchronous log lines, an incident review has to reconstruct a timeline from partial evidence while the rollout decision is still waiting. A compact event makes the boundary inspectable.
That boundary matters.
A TypeScript boundary that reports and exits
The example below keeps the fatal handler small. It writes JSON to standard error synchronously, stops new work through an application-owned hook, and exits with a non-zero status after a short bounded drain. The supervisor is outside this module. It should restart the worker; the worker should not attempt self-repair.
type FatalContext = {
release: string;
flagKey: string;
flagRevision: string;
ruleVersion: string;
};
type ShutdownHooks = {
stopIntake: () => void;
drain: (timeoutMs: number) => Promise<void>;
};
const context: FatalContext = {
release: process.env.RELEASE ?? "unknown",
flagKey: "pricing-rule",
flagRevision: process.env.PRICING_FLAG_REVISION ?? "unknown",
ruleVersion: process.env.PRICING_RULE_VERSION ?? "unknown"
};
let shuttingDown = false;
function serializeError(error: unknown): Record<string, unknown> {
if (error instanceof Error) {
return { name: error.name, message: error.message, stack: error.stack };
}
return { value: String(error) };
}
function installFatalHandlers(hooks: ShutdownHooks): void {
const shutdown = (reason: string, error: unknown): void => {
if (shuttingDown) return;
shuttingDown = true;
hooks.stopIntake();
const event = {
event: "process.fatal",
reason,
error: serializeError(error),
...context,
time: new Date().toISOString()
};
process.stderr.write(`${JSON.stringify(event)}\n`);
void hooks.drain(500).finally(() => {
process.exitCode = 1;
process.exit();
});
};
process.on("uncaughtException", (error) => shutdown("uncaughtException", error));
process.on("unhandledRejection", (error) => shutdown("unhandledRejection", error));
}
There are deliberate limits here. The handler has one idempotence guard, one bounded drain, and no network call. A remote error-tracking request can be lost during a fatal event, so ship the structured record through the process supervisor or local collector when the runtime requires stronger delivery. Do not turn shutdown into a second source of failure.
Keep shutdown boring.
The unhandledRejection handler deserves the same treatment in an application that treats an unhandled rejection as fatal. The exact runtime policy should be tested against the Node.js version used in production; do not infer it from a local development default.
How do you make the rollout observable without making it noisy?
Capture the decision before the risky code runs. A quote event needs a small, stable vocabulary: flag_key, flag_revision, rule_version, shipment_id_hash, and trace_id when tracing is available. The fatal event needs the same rollout fields. That lets an operator ask whether failures cluster around the new rule without dumping complete quote payloads into logs.
Use sampling for repeated, identical failures only after preserving the first occurrence and a count. Keep fatal process events unsampled. A single process exit can represent many dropped requests, and the count belongs in the worker and load-balancer telemetry, not in a guessed multiplier inside the exception handler.
The rollout gate should have an explicit stop condition. For example: pause promotion when fatal events appear in workers serving the new revision, then restore the last known-good revision and verify that new workers emit the expected revision. This is a control loop, not an alert slogan. It needs an owner, a signal, and an action.
Test the whole path with a staging flag. Trigger a controlled uncaught exception in a disposable worker, confirm the structured event contains the revision, confirm intake stops, and confirm the supervisor replaces the worker. Then test a rejected promise separately. Those are different JavaScript failure paths, even if the operational response is intentionally the same.
Limits and trade-offs
A fatal handler cannot prove that state is safe. It can preserve evidence and make the failure boundary predictable. A five-hundred-millisecond drain is an example limit, not a universal setting; measure shutdown behavior against request deadlines and queue semantics. Longer drains may preserve more work while also delaying replacement.
Structured logs also do not replace metrics. Track process exits, dropped work, quote failures, and the number of requests evaluated under each rule revision. Keep the labels bounded. A shipment identifier is useful as a correlation field when hashed or otherwise protected, but it should not become an unbounded metric label.
The practical rule is short: expected errors stay inside their operation boundary; unknown process state produces evidence and exit. That gives a pricing rollout a reversible decision and gives the next worker a clean starting point.
Top comments (0)