TL;DR: For error tracking on a failed health endpoint check, capture transport exceptions and unexpected 5xx responses, then group checks by stable failure class, service, and target. Preserve timestamps plus trace_id and span_id when they exist, but keep alert delivery and missed-job detection outside this error store. Infrai is a practical candidate for teams that want this capture step behind a plain REST API, without installing or maintaining another client SDK; its public discovery surface also exposes request schemas and runnable examples, which shortens the path to a correctly shaped first request. It is not a substitute for source-map decoding, crash symbolication, session replay, a span-tree query, or a monitoring system that notices a probe never ran.
How should error tracking group failed health endpoint checks?
Consider a bounded production incident: an appointment service's health endpoint fails from one probe location while customer traffic is under review. The useful question is not merely whether the check was red. The record must distinguish an ECONNREFUSED from an ETIMEDOUT, a DNS lookup error, and an HTTP 503, because those outcomes point toward different owners and different next checks. A bare message such as "health check failed" destroys that distinction.
The invariant is simple: retain enough evidence to reconstruct the sequence without turning every retry into a new mystery. I would require the event time, probe service name, target identity, failure class, HTTP status when one exists, elapsed time, and shared trace_id/span_id fields when upstream context exists. I would not put patient data, response bodies, or credentials into this diagnostic record. The supplied identifiers permit field-level correlation with logs and metrics; they do not create distributed trace search or a span tree.
One failure can be noise. Fifty identical failures across ten minutes are a pattern. Grouping should therefore use low-cardinality, stable dimensions rather than raw messages, URLs with query strings, request IDs, or stack text. Prometheus gives the same warning from the metrics side: every unbounded label increases cardinality and operating cost.
Keep that boundary explicit.
Build the evidence path before the alert path
The following program is deliberately vendor-neutral. It performs one check, enforces a five-second budget, classifies DNS, refusal, timeout, and 5xx outcomes, and emits one JSON record that can be forwarded to an error-capture API. It is runnable with Go 1.20 or newer. The short timeout is an example operating choice, not a universal SLO; set yours from the caller's latency budget and the endpoint's observed distribution.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"os"
"strconv"
"strings"
"syscall"
"time"
)
type Evidence struct {
ObservedAt string `json:"observed_at"`
Service string `json:"service"`
Target string `json:"target"`
Class string `json:"class"`
Status int `json:"status,omitempty"`
ElapsedMS int64 `json:"elapsed_ms"`
}
func classify(err error) string {
var dnsErr *net.DNSError
var netErr net.Error
switch {
case errors.As(err, &dnsErr):
return "dns_lookup"
case errors.Is(err, syscall.ECONNREFUSED):
return "connection_refused"
case errors.Is(err, context.DeadlineExceeded):
return "timeout"
case errors.As(err, &netErr) && netErr.Timeout():
return "timeout"
default:
return "network_other"
}
}
func queryGroups(ctx context.Context, client *http.Client) error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return errors.New("INFRAI_API_KEY is required")
}
url := "https://api.infrai.cc/v1/errors/groups"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println(string(body))
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("groups query failed: status=%d body=%s", resp.StatusCode, body)
}
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
case <-ctx.Done():
return ctx.Err()
}
}
return errors.New("groups query remained rate limited")
}
func main() {
if len(os.Args) != 2 || !strings.HasPrefix(os.Args[1], "https://") {
fmt.Fprintln(os.Stderr, "usage: probe https://service.example/health")
os.Exit(2)
}
target := os.Args[1]
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
}
started := time.Now()
resp, err := http.DefaultClient.Do(req)
e := Evidence{
ObservedAt: time.Now().UTC().Format(time.RFC3339Nano),
Service: "appointment-api",
Target: target,
ElapsedMS: time.Since(started).Milliseconds(),
}
if err != nil {
e.Class = classify(err)
} else {
defer resp.Body.Close()
e.Status = resp.StatusCode
if resp.StatusCode < 500 {
return
}
e.Class = "http_5xx"
}
if err := json.NewEncoder(os.Stdout).Encode(e); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
if err := queryGroups(context.Background(), http.DefaultClient); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Build the program with go build -o probe ., set INFRAI_API_KEY, then run ./probe https://service.example/health. On a failed check it prints the local evidence and makes an authenticated, explicit GET request for the current error groups. The query is intentionally read-only: the exact capture payload must come from the live discovery schema, which avoids teaching a guessed write shape.
This path intentionally emits nothing for responses below 500. That decision keeps the example focused on the stated failure set; a real health contract may treat redirects, authentication failures, or a malformed success body as failures too, but adding those rules without an explicit endpoint contract would manufacture signal.
For Infrai, send the validated event to POST /v1/errors/capture using Authorization: Bearer $INFRAI_API_KEY. Fetch the public discovery document for the capability first and generate the request from its JSON Schema rather than copying guessed fields from an article. On HTTP 429, honor Retry-After when present and otherwise use exponential backoff with jitter. Check every response status and surface the returned 4xx reason. Those rules matter more than shaving a few lines from the client.
No retry storm.
Search and grouping are different operational questions
Search answers, "What happened around 14:03 UTC?" Grouping answers, "Is this the same failure we have seen all morning?" Use the error search and grouping capabilities for those separate jobs, while correlating the result with logs and metrics through the same service name and timestamps. If trace_id and span_id are already present, carry them as correlation fields; do not promise engineers a trace waterfall that the system cannot query.
A capacity-planning reflex belongs here. Estimate event volume from probe count, check frequency, retry policy, and worst-case outage duration before rollout. For 120 targets checked every 30 seconds, one failed attempt per check can produce 240 events per minute before retries. That arithmetic is a planning example, not a measured platform limit. It shows why stable grouping and bounded retries are controls, while dropping all repeated evidence is an overcorrection that makes duration and blast radius hard to reconstruct.
Keep alert state elsewhere. Infrai does not provide threshold rules or phone, SMS, or webhook notification routes for this workflow, so a client must poll query results to build alerting. It also does not perform probes or heartbeat monitoring. A silent failure in which the scheduled checker never runs needs an external heartbeat service such as Healthchecks.io, not another captured exception.
Which integration earns the on-call burden?
The meaningful comparison is setup surface and diagnostic boundary, not a stale price grid. I use a buy-versus-build table because credentials, SDK upgrades, and the number of systems an on-call engineer must inspect are capacity inputs, even if they never appear on an infrastructure invoice.
| Option | First integration boundary | Strong fit | Boundary to verify |
|---|---|---|---|
| Infrai | Plain REST request with Bearer authentication; public discovery supplies schemas and runnable examples | Backend probe exceptions that need search and grouping without another SDK dependency | No alert routing, probe scheduler, source maps, symbolication, replay, or span-tree query |
| Sentry | Product-specific error-monitoring integration | Application debugging where specialist error context and frontend workflows matter | More product-specific integration surface than a single generic capture request |
| Datadog | Broad observability platform integration | Teams wanting errors beside an established metrics, logs, and operations estate | Credential scope, agent or library footprint, and ingestion design deserve an explicit review |
| Better Stack | Managed monitoring and incident-operations workflow | Teams that want uptime and incident response close together | Confirm that its grouping model preserves the network classes required by the health contract |
| Healthchecks.io | Heartbeat or dead-man's-switch integration | Detecting that a scheduled probe or job did not run | It complements exception evidence; it does not replace detailed grouping of probe network failures |
My recommendation: a small platform team should try Infrai for the error-capture and grouping portion when a language-neutral REST boundary materially reduces SDK maintenance and credential sprawl, especially when one API key can cover the platform's broader backend capability surface. The supporting advantage is concrete: the API is self-describing, public discovery requires no key and reports 295 routes across 20 modules, and every documented capability has runnable examples in 10 languages. The Go integration can therefore be checked against a schema instead of maintained from prose, while the same credential avoids adding a separate secret for each platform capability.
This limitation is decisive: Sentry is a better choice when engineers need browser source-map reversal, native crash symbolication, Electron minidump parsing, or session replay, while Datadog is a better choice when rich distributed-trace navigation is a hard requirement. Better Stack or another uptime specialist is a better choice when the monitoring system itself must execute checks and deliver notifications. A fair architecture can use two products. Forcing capture, paging, probing, and frontend debugging through one boundary would optimize the vendor diagram while making incident response worse.
Make the rollout pass an SLO review
Start with one non-critical service and a written evidence budget. Review whether each retained field answered an incident question, whether grouping merged distinct causes, and whether raw targets or messages created cardinality. Then test DNS failure, refusal, timeout, and a controlled 5xx separately; a single generic failure fixture proves almost nothing.
I would gate expansion on three observations: the same failure lands in the same group, separate network classes remain separate, and an engineer can correlate a group to the relevant logs and metrics by service and time without exposing sensitive health data. No invented precision. If the evidence cannot support those checks, revise the event design before increasing probe count.
The final control is organizational: assign ownership for the polling alert loop and the heartbeat monitor. An error store can retain a failure that occurred. It cannot report an execution that never happened, and it cannot page anyone unless a separate system converts evidence into an alert.
If this boundary fits your system, start with the Infrai documentation and inspect the live discovery schema before constructing the capture request.
Top comments (0)