The page says the checkout health check failed. The responder sees a red status tile, a recent deployment, and a new pricing rule behind a flag. Treat this as a hypothetical incident: the immediate problem is deciding whether to disable the rule, while the evidence currently proves only that a check failed. A dashboard full of red does not establish which customers were exposed or whether the rule caused their failures.
Short answer: for HTTP health checks with polling in Node.js, choose a small request interface that lets you set a timeout, inspect the response, and retain the result of every attempt. Add a poller that records timing and prevents overlapping attempts. For incident reconstruction, the useful API is the one that preserves evidence about the request and the pricing decision; a single healthy boolean leaves too much work for the person carrying the pager.
Start with direct polling when one service and one check are enough to answer the operational question. Consider a separate monitoring service when scheduling, shared incident state, or retained history would otherwise become another service your team must operate. Evaluate those responsibilities explicitly. No product comparison can decide who owns a missed poll.
1. What should HTTP health checks record when polling triggers a page?
Ask that before reading the deployment graph. For the pricing rollout, define two different observations: a reachability check against an agreed health endpoint, and a controlled checkout check that exercises the flagged pricing path. Use the former to describe availability of that endpoint; use the latter to test a pricing expectation under a known flag context. Do not let either silently stand in for the other. A successful request to a health endpoint does not, by itself, supply evidence that the pricing calculation was exercised.
The distinction changes the page. A failed reachability check should identify its target and failure category. A pricing check should identify the controlled scenario and the expectation it violated. Keep the alert summary short, but make its underlying attempts accessible so the responder can distinguish a timeout from a returned response whose content failed validation.
Name the observation.
For an illustrative acceptance test, suppose a controlled cart has an expected total of 12000 minor currency units under the enabled rule. That number is a test fixture, not a reported customer result or a recommended business rule. The check should compare the returned total with that fixture under a specified currency and flag context; calling the response successful merely because it arrived would discard the business assertion that makes the test worth running. Use a test path that cannot charge a customer or leave an order behind, and make cleanup behavior part of the test design.
A synthetic cart also has a boundary: it observes that cart. If the rollout targets particular customer cohorts, the responder needs evidence about the relevant cohort rather than an unsupported inference from an unrelated test account. That is a reason to label the scope of the check clearly, not a reason to collect customer identities in every monitoring record.
2. Work backward to the first missing signal
In the hypothetical page, the checkout check is already failing. Work backward through its retained attempts to find the last successful pricing assertion, the first failed assertion, and the flag configuration recorded by the application for those operations. Then compare those observations with the rollout history. The useful question is whether the evidence narrows the investigation enough to justify a reversible mitigation; it cannot establish causality merely because two events appear next to each other on a chart.
A compact evidence contract makes that investigation reviewable:
| Observation | Evidence to retain | Reconstruction question |
|---|---|---|
| Poll attempt | Start time, completion time, outcome, checker identity | Did the checker run and finish? |
| HTTP response | Status when available and validation outcome | Did transport succeed while the assertion failed? |
| Pricing evaluation | Rule revision and evaluated flag variant | Which decision did this operation actually use? |
| Request context | Correlation identifier and test scenario | Can the responder find the related application record? |
Keep the configured rollout state separate from the variant actually evaluated for the operation. The first expresses intent; the second is the evidence needed to examine that particular pricing decision. If the application does not record the evaluated variant, the postmortem should identify the resulting uncertainty rather than fill it with a deployment timestamp.
This is where logs earn their storage. The OpenTelemetry logs concepts documentation describes logs as recordings of events and explains their relationship to other telemetry signals, including correlation with traces through trace and span identifiers. Use that correlation where it exists to connect the failed check with the application operation. A correlation identifier is a route to related evidence; it is not proof that the pricing rule caused the failure.
Prefer bounded summaries over dumping response bodies. An assertion result, scenario identifier, and rule revision are useful during reconstruction. Collecting whole carts adds a data handling burden without automatically answering another incident question. Decide retention and access around the investigation you expect responders to perform.
3. Change the instrumentation before expanding the API
For a Node.js caller, assess the HTTP request interface and the polling loop separately. Require explicit request deadlines, visible status values, deliberate response validation, and distinct outcomes for transport failure and assertion failure. The loop should own scheduling and attempt records. That separation gives the team a small contract to test without turning the application into a monitoring platform. Don't let a final success erase the observations that preceded it.
The following Go helper expresses the classification contract for a separate evidence processor; the Node.js poller should produce equivalent outcomes. The caller supplies its own accepted-status policy and evaluates the pricing assertion only after receiving an acceptable response. This keeps transport failure, response rejection, and an incorrect price distinguishable without coupling the evidence processor to a particular request library.
package evidence
func Classify(transportErr error, statusAccepted, assertionOK bool) string {
if transportErr != nil {
return "transport_failure"
}
if !statusAccepted {
return "response_rejected"
}
if !assertionOK {
return "assertion_failed"
}
return "passed"
}
Use concrete settings in the test plan. For example, a 15 second interval and a 3 second deadline are illustrative inputs for exercising the scheduler, not measured production defaults. Test a response that completes before the deadline, one that exceeds it, and an attempted tick while a previous request remains active. Choose whether to skip or defer that tick, record the choice, and make sure the implementation behaves that way. Otherwise the incident timeline can contain a monitoring gap whose cause is indistinguishable from an application problem.
Silence needs meaning.
Record a completed failed attempt differently from an absent attempt. Include an attempt identifier and enough scheduler state to tell a skipped poll from a failed HTTP request. If the system retries, preserve the initial outcome alongside the retry outcome; replacing the first failure with eventual success would remove evidence that the responder may need. Retry policy also belongs in the load review because every additional attempt is another request to the system under investigation.
Instrument the pricing evaluation in the application as well. Attach the evaluated flag variant, rule revision, controlled scenario identifier, and correlation context to an event describing the decision. Put the corresponding check result in the monitoring record. This is the instrumentation change that turns the page from a color change into a starting point for reconstruction, while keeping responsibility for the actual pricing decision in the service that made it.
Before rollout, exercise the evidence path as part of deployment testing: disable the flag, enable it for the test cohort, introduce an expected assertion failure in an isolated test environment, and verify that a responder can retrieve the matching decision event. Do not stop at verifying that the alert notification arrived. A page that arrives without its supporting evidence has passed a delivery test and failed the operational test.
4. Set the threshold around the rollback decision
Once the records exist, write down what would justify disabling the pricing rule. A failed controlled assertion under the enabled variant can warrant investigation of that rollout, while a timeout with no corresponding application evidence leaves a wider set of explanations. Preserve that distinction in the runbook. The responder should know what the check observed, which mitigation is available, and which additional evidence would change the decision.
Keep the rollout reversible. Test flag disablement as a separate action and verify subsequent pricing evaluations before treating the incident as mitigated. A quiet alert alone is weak closure if the checker itself has stopped running.
Alert thresholds now become an explicit trade-off. Requiring 3 consecutive failed attempts instead of 1 is an illustrative policy choice: it asks for more observations before paging, while allowing additional failed attempts to occur before the condition is met. The trade-off is slower detection in exchange for fewer pages from isolated failures. A single failed pricing assertion might deserve a different response from a single reachability timeout. Choose that policy with the owners of checkout and the rollout, and test notification and recovery behavior against recorded sequences rather than borrowing a threshold from an unrelated service.
The closing postmortem question is what the threshold made the responder do. A threshold that pages on every isolated observation can consume attention and prompt unnecessary flag changes; a threshold that waits too long can postpone a justified mitigation. Count false pages and review their supporting attempts alongside missed or delayed signals. The simplest useful health check API is the one that leaves enough evidence to revise that policy after the page, without making the next responder reconstruct the checker itself.
References
OpenTelemetry's logs concepts documentation supports the discussion of event records and correlation with other signals. The cart value, polling settings, and alert counts above are illustrative design inputs rather than observed results.
Further reading
- OpenTelemetry logs concepts: https://opentelemetry.io/docs/concepts/signals/logs/
Top comments (0)