Pricing rule rollouts should be observable as decisions, not just exceptions. Capture a stable request fingerprint, the rule version, flag state, tenant-safe cohort, and trace context at the API boundary; then connect that event to metrics and a replayable timeline. The deciding constraint in healthtech is incident reconstruction: an error report that says 500 is not enough to explain which price was calculated, under which rule, and why the request took that path.
TL;DR: Put structured error events at the boundary of the pricing operation, redact sensitive fields before export, and record the same bounded dimensions in metrics. Keep the event payload small. Keep the decision context durable. A useful record lets an engineer answer three questions without guessing: what failed, who was affected, and which rollout state produced it.
The before-and-after mental model
The weak model is a stack trace with a timestamp. It is quick to emit and hard to use. During an incident, someone has to join that trace to deployment logs, feature-flag audit records, request logs, and a database snapshot. Those sources rarely share the same identifier or clock precision.
The stronger model is a short chain:
request -> pricing decision -> error event -> metric -> trace timeline
The request carries a correlation identifier. The pricing decision records the rule revision and evaluated flag state. The error event records the failure class and a safe cohort key. The metric shows whether failures cluster by endpoint, rule revision, or cohort. The trace supplies timing across the API, flag evaluation, and pricing dependency.
Keep it boring.
That separation matters. OpenTelemetry describes metrics as a signal for measurements collected over time, which makes them useful for rates and distributions rather than for storing a full exception object. Keep high-cardinality context, such as an internal request identifier, in logs or traces. Use bounded dimensions in metrics.
For a healthtech pricing endpoint, the event might contain ruleRevision: "pricing-2026-04", flagState: "new-rule", and cohort: "pilot-a". It should not contain a patient name, diagnosis, full address, payment credential, or an unfiltered request body. The safe context is the context an on-call engineer needs to reconstruct the decision.
What should an API error event contain?
Start at the boundary. Error handling scattered across controllers produces inconsistent records: one path has a status code, another has a message, and a third serializes an entire exception. A single boundary handler can enforce the contract and apply redaction once.
type PricingContext = {
ruleRevision: string;
flagState: 'control' | 'new-rule';
cohort: string;
};
type ErrorEvent = {
eventName: 'pricing_api_error';
requestId: string;
route: string;
method: string;
status: number;
errorType: string;
retryable: boolean;
pricing: PricingContext;
traceId?: string;
};
function makePricingErrorEvent(
error: unknown,
request: { id: string; method: string; route: string; traceId?: string },
pricing: PricingContext
): ErrorEvent {
const knownError = error instanceof PricingDependencyError;
return {
eventName: 'pricing_api_error',
requestId: request.id,
route: request.route,
method: request.method,
status: knownError ? 503 : 500,
errorType: knownError ? 'pricing_dependency_timeout' : 'unexpected_error',
retryable: knownError,
pricing,
traceId: request.traceId
};
}
class PricingDependencyError extends Error {}
The important part is the contract, not the class names. errorType is a bounded vocabulary. ruleRevision is deployable and searchable. flagState describes the evaluated state, not the flag's configured name alone. cohort is retained in the event because it can be necessary for reconstruction, but it should not automatically become a metric label.
Do not log the raw exception message by default. Messages often absorb upstream payloads. Classify known errors, attach a sanitized cause code, and keep the original exception available only in a controlled diagnostic path. A healthtech log pipeline should treat redaction as a data boundary, not a formatting preference.
How do you reconstruct a failed rollout?
Imagine the new rule is enabled for pilot-a. The API returns intermittent 500 responses for one pricing operation. The incident timeline should be readable without opening a patient record.
First, group errors by ruleRevision, flagState, errorType, and route. If failures appear only under pricing-2026-04, the flag state is a strong lead. It is not proof. A dependency timeout could affect both control and new-rule traffic, so compare both populations.
Next, follow one requestId through the trace and structured logs. Confirm that the request reached the intended rule evaluator, that the evaluator returned the recorded state, and that the pricing dependency saw the same trace. A mismatch means the telemetry contract is broken even if the endpoint response is correct.
Finally, compare the error event with the API response policy. A retryable dependency timeout should have a retryable classification and an appropriate response status. An invalid rule should fail closed according to the service's domain policy. The error event documents the decision path; it is not a second source of pricing truth. If the request says flagState: "new-rule" but the trace shows a control calculation, stop investigating the pricing formula first. The telemetry contract has already exposed a rollout problem, and the next check is the boundary where that context was copied or changed.
| Dimension | Question it answers |
|---|---|
ruleRevision |
Did one rule revision introduce the pattern? |
flagState |
Is the pattern limited to the rollout path? |
errorType |
Is this validation, dependency, or unknown failure? |
route |
Is one API operation affected? |
requestId and traceId
|
Can one failure be reconstructed end to end? |
This is where before and after diverge. Before, the on-call engineer searches for a timestamp. After, the engineer tests a hypothesis. That difference reduces the chance of disabling a rollout that was only coincidentally near an unrelated dependency failure.
What will break this design in production?
The first trap is cardinality. A unique user, patient, invoice, or request identifier is valuable in an event and dangerous as a metric label. Metrics aggregate. Logs and traces provide lookup paths. Mixing those jobs creates noisy time series and a dashboard that cannot answer a simpler question.
The second trap is inconsistent flag capture. If the flag is evaluated in one module and errors are recorded in another, pass the evaluated context explicitly. Reading the flag again inside the error handler can produce a different answer after a configuration change. Record the value used by the pricing decision.
The third trap is retry duplication. A client or gateway can retry the same operation. Each attempt may be a real infrastructure event, but the business incident may be one failed operation. Include an attempt number or idempotency reference when the protocol provides one, and decide whether your alert counts attempts or operations.
The fourth trap is over-alerting on raw exception volume. Alert on a bounded signal that represents user impact: an elevated rate of 5xx responses, a sustained retryable dependency error rate, or a rollout cohort crossing a defined failure threshold. The exact threshold belongs to the service's traffic and risk profile; it should not be copied from a generic example.
I initially treated the stack trace as the center of error tracking. The correction was operational, not stylistic: incident reconstruction needs the decision inputs beside the failure classification. The stack trace became supporting evidence instead of the only clue.
Which trade-offs should the team choose?
A full trace on every successful pricing request may be unnecessary, while sampling every failed request can hide the context needed to understand a rare issue. Retain error events that pass the privacy filter, align trace sampling with error status when the tracing system supports it, and sample routine successes according to traffic needs. Document the policy so engineers know why a trace is absent.
There is also a tension between rich context and minimal data. Store a stable cohort key and rule revision, not a copied business object. Keep correlation IDs opaque. Apply retention by signal: high-volume metrics can outlive verbose events, while sensitive event payloads need a shorter and more deliberate retention period.
A small contract test catches regressions. Send one synthetic request through the control path and one through the new-rule path, force a known dependency error, and assert that the emitted event has the expected bounded fields and no prohibited body fields. Run it alongside the pricing service tests. Observability code is production behavior. It deserves a test.
The result is a rollout process with an evidence trail: flag evaluation, pricing revision, request correlation, classified failure, bounded metric, and trace timing. That trail supports a rollback decision without exposing clinical data or turning telemetry into a shadow database.
References
- OpenTelemetry, “Metrics signal concepts”: https://opentelemetry.io/docs/concepts/signals/metrics/
- OpenTelemetry, “Signals”: https://opentelemetry.io/docs/concepts/signals/
- W3C, “Trace Context”: https://www.w3.org/TR/trace-context/
- MDN Web Docs, “HTTP response status codes”: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status
- Node.js documentation, “Diagnostics Channel”: https://nodejs.org/api/diagnostics_channel.html
Top comments (0)