For error tracking on failed health endpoint checks, probe the checkout path from outside the Node.js service, classify each failure by the stage that stopped progress, and attach bounded ownership fields before storing the event. The deciding constraint is cost attribution: a healthtech team needs to know which service and dependency consumed probe, retry, and response effort without turning every raw fetch error string into a new group.
TL;DR: preserve the original error for investigation, but group on four stable layers: name resolution, connection, response deadline, and application response. Search on a probe ID and low-cardinality ownership fields. Retry only when the operation and budget permit it, and verify the classifier with deterministic failures before it can page anyone.
This is a failure ledger, not an availability score. A checkout can fail while every process is alive, and a green process check can coexist with a broken dependency path. The useful record answers three operational questions: where did progress stop, who owns the next action, and how much work did the failed attempt create?
How should error tracking group failed health endpoint checks?
A Node.js checkout service may report a timeout, a refused connection, or a name-resolution failure during a health check. Those labels describe different stopping points. A timeout means the configured deadline expired before the expected result arrived. A refusal means the target host was reached but no listener accepted the connection at that address and port. A DNS-class failure means the probe could not obtain the address needed for the connection. Collapsing all three into health check failed destroys the first routing decision.
Grouping by the complete error message causes the opposite problem. Messages may contain hosts, ports, wrapped causes, and other changing text. If those values become metric labels or group keys, the number of time series and issue groups can grow with the input. Prometheus explicitly warns against high-cardinality labels and recommends keeping labels bounded. Keep the full message in the searchable event; keep it out of the metric identity.
The application-response layer belongs beside the transport layers. A completed HTTP exchange with a failing status is evidence that DNS and connection setup progressed far enough to receive a response. It should not share a group with a deadline or socket failure. For a checkout path, this distinction also prevents an application owner from being charged operationally for resolver or network work that belongs elsewhere.
Use one event per completed probe attempt, not one event per internal exception. A single failed attempt can produce a wrapped error plus several log lines; counting each artifact inflates both the apparent failure rate and the ingestion cost. The attempt ID is the deduplication boundary.
One attempt, one event.
Define four stable groups and a bounded cost envelope
The grouping key should be boring:
| Layer | Stable group key | Useful search fields | Initial owner |
|---|---|---|---|
| Name resolution | checkout/name_resolution |
probe_id, service, dependency, region
|
DNS or platform |
| Connection | checkout/connection |
probe_id, service, dependency, region
|
network or service runtime |
| Response deadline | checkout/response_deadline |
probe_id, service, dependency, timeout_bucket
|
downstream service |
| Application response | checkout/application_response |
probe_id, service, dependency, status_class
|
application team |
Four groups are enough to start. Do not put a hostname, customer, checkout ID, URL, raw status code, or error text in the group key. Those values can remain searchable event attributes when access controls and retention rules allow them, but they should not define alert identity. Health-related checkout data deserves a separate review: diagnostic records should not acquire patient or payment details merely because those details were present in request context.
Cost attribution needs its own small envelope. Record the accountable service, dependency class, attempt count, elapsed-time bucket, and response-byte bucket. Buckets make aggregation possible without pretending that a probe can calculate an invoice. The attribution is operational: it shows which boundary generated repeated work and where a retry policy increases load. It does not claim exact infrastructure or vendor charges.
Keep dimensions bounded. For example, timeout_bucket might be lt_250ms, 250ms_1s, 1s_3s, or gte_3s; choose boundaries from the service objective and probe deadline, then version them deliberately. Exact duration still belongs in the event body for forensic search.
There is a trade-off here. Coarse groups make paging and trends stable, while detailed events make investigation possible. Trying to make one representation serve both jobs usually produces noisy alerts or unusable evidence. Store both forms and connect them with the same probe ID.
Consider a scheduled probe that starts in one region, resolves the checkout name, connects, and then reaches its deadline while waiting for the endpoint. The terminal group is checkout/response_deadline; the resolver and connection are completed stages, not extra failures. The event retains the exact duration and wrapped cause, while the metric increments once with the service, dependency class, region, and bounded timeout bucket. If policy permits a second attempt, that retry receives a new probe ID plus an attempt number linking it to the scheduled check. Search can reconstruct both attempts. Aggregation can charge two request attempts and their elapsed-time buckets to the dependency while counting one scheduled check as affected. This example is why a generic failed counter is inadequate: it cannot separate affected checks from the work generated by retries, and it gives the responder no first boundary to inspect.
Make classification deterministic at the probe boundary
Classify at the component that owns the deadline and sees the complete attempt. For an external checker aimed at a Node.js endpoint, that component is the probe, not the checkout handler. The handler cannot report a request that never reached it.
The following Go example keeps the wire action small and the output generic. It uses a request-scoped context deadline, preserves the cause for search, and maps failures into the four groups. The errors.Is and errors.As checks avoid depending on display strings.
package probe
import (
"context"
"errors"
"fmt"
"net"
"net/http"
"syscall"
"time"
)
type Result struct {
ProbeID, Group, Service, Dependency string
Duration time.Duration
Status int
Cause error
}
func Check(ctx context.Context, client *http.Client, probeID, endpoint string) Result {
started := time.Now()
result := Result{ProbeID: probeID, Service: "checkout", Dependency: "checkout-api"}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
result.Group, result.Cause = "checkout/application_response", err
return result
}
resp, err := client.Do(req)
result.Duration = time.Since(started)
if err != nil {
result.Group, result.Cause = classify(err), err
return result
}
defer resp.Body.Close()
result.Status = resp.StatusCode
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
result.Group = "checkout/application_response"
result.Cause = fmt.Errorf("unexpected status class: %dxx", resp.StatusCode/100)
return result
}
result.Group = "ok"
return result
}
func classify(err error) string {
if errors.Is(err, context.DeadlineExceeded) {
return "checkout/response_deadline"
}
var dnsErr *net.DNSError
if errors.As(err, &dnsErr) {
return "checkout/name_resolution"
}
if errors.Is(err, syscall.ECONNREFUSED) {
return "checkout/connection"
}
return "checkout/connection"
}
The final fallback is intentionally broad. Unknown transport failures need a stable home, but they also need a review queue. Track the rate of fallback classifications as a metric with no raw-message label. If that rate becomes material, add a tested class based on machine-readable error identity, not a substring observed in one log.
Fallback is debt.
Do not let the probe follow an unlimited chain of work. Give each attempt one deadline. If retries are justified, reserve them inside a total retry budget and add backoff with jitter; the AWS Builders' Library explains why retries can amplify load and why jitter spreads synchronized work. A checkout write must also be idempotent before a retry is considered safe. A health probe should remain read-only.
Search from the failed attempt outward
Start a page investigation with the stable group and a narrow time window. Then search the exact probe_id to retrieve the original cause, duration, target dependency, deployment marker, and region. Next, compare nearby groups: a rise in name-resolution failures across several dependencies points to a different boundary than response deadlines isolated to checkout.
The sequence matters. Group first, pivot second. Free-text search across every error message is slow to reason about during an incident and makes spelling part of the runbook. A saved query should rely on structured equality for the stable fields and use raw text only to inspect the remaining cause.
A practical event has two identities. probe_id identifies this attempt and supports deduplication. group identifies the operational class and supports aggregation. Do not reuse a checkout transaction ID as the metric label; it is high-cardinality, and it may cross a data boundary that the monitoring system was never meant to hold.
For cost review, aggregate attempts and elapsed-time buckets by accountable service and dependency class. Look for retry multiplication: three attempts arising from one scheduled probe represent one observed outage but three units of request work. That difference is exactly why the attempt number belongs in the event.
Short failures count.
A refused connection can return quickly and still deserve urgent attention, while a slow timeout consumes the probe's entire deadline. Duration alone is not severity. Route on layer, scope, and sustained occurrence; use duration to explain resource consumption.
Verify before paging and roll back without losing evidence
Test the classifier with controlled fixtures before deployment. Use a deliberately unresolvable test name for the DNS path, a closed local port for refusal, a test server that waits beyond the deadline for timeout, and a server that returns a non-success status for application response. Assert the stable group, the single emitted attempt, and the absence of unbounded values from metric labels. Do not aim production probes at arbitrary public targets to manufacture failures.
Then run the new classifier in shadow mode. Emit its structured events without changing alerts, compare its groups with the existing probe outcome, and inspect the fallback bucket. The acceptance rule is concrete: every test fixture maps to one expected group, one probe produces one terminal event, and removing the new grouping leaves the existing availability decision unchanged.
Deployment markers should be searchable fields, not new groups. During rollout, compare failure layers before and after the marker and across regions. A simultaneous shift in several unrelated dependencies is evidence to inspect a shared boundary before paging each application team separately.
Rollback must disable routing and aggregation changes without deleting the raw attempt records. Keep the previous alert rule available, stop the new rule first, and leave event capture running while the team determines whether the classifier or the system changed. If the schema itself is wrong, version the classifier and reprocess only when retention policy and event semantics permit it. Never rewrite old evidence silently.
The operational target is modest: a failed checkout probe should create one durable fact, land in one of four bounded groups, and point to one accountable boundary. That gives responders a fast first move and gives cost owners a defensible count of retries and elapsed work, without converting every runtime detail into an alert.
Top comments (0)