A health probe for a customer-support AI agent must preserve the failure cause without turning every retry into another page. Short answer: capture fetch exceptions and unexpected 5xx responses as searchable errors, group them by stable operational cause, and correlate them with logs and metrics by service name and timestamp. Error tracking helps explain a failed check; it does not replace alert delivery, distributed tracing, or heartbeat monitoring.
The first question in a review should be blunt: what page fired? If the answer is “one page per failed fetch,” the design records retry volume rather than incidents. If no page fired, an immaculate error dashboard is still a postmortem artifact, not a monitoring system.
Failure taxonomy before tooling
For this workload, start with four classes: ECONNREFUSED, ETIMEDOUT, DNS lookup errors, and unexpected HTTP 5xx responses. Keep the endpoint's service identity and the failure class stable enough to group; keep the timestamp and request-specific detail on the individual event. A refusal generally points the investigation toward reachability or a listener, a timeout toward latency or an exhausted deadline, and a DNS error toward name resolution. A 5xx is different again: the connection worked, but the dependency reported failure.
Do not group on the complete message, a request ID, an AI conversation ID, or the raw URL when it contains variable path segments. Those values fracture one persistent fault into thousands of groups. The inverse mistake is just as bad — grouping every failed probe under health-check-failed erases the distinction that the on-call engineer needs at 3am. Prometheus gives the same warning from the metrics side: every unique label combination creates a new time series, so high-cardinality dimensions need care.
For a customer-support agent loop, use a shared service name across the error event, application logs, and latency or cost metrics. Then a search around the same timestamp can answer whether a slow model or backend call preceded the health failure. trace_id and span_id may be carried as fields, but they do not create a queryable span tree here. That's a correlation aid, not distributed tracing.
One failure is evidence. Repetition is a pattern.
Build a probe that preserves the cause
The probe below is deliberately small and uses Go because the important contract is language-independent: bound the request with a timeout, distinguish transport failures from HTTP failures, and emit a normalized internal event. It does not pretend that the string returned by a network stack is a durable grouping key.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
"net/http"
"os"
"strings"
"syscall"
"time"
)
type ProbeError struct {
Service string `json:"service"`
Endpoint string `json:"endpoint"`
Kind string `json:"kind"`
Message string `json:"message"`
Occurred time.Time `json:"occurred_at"`
StatusCode int `json:"status_code,omitempty"`
}
func classify(err error) string {
var dnsErr *net.DNSError
if errors.As(err, &dnsErr) {
return "dns_lookup"
}
if errors.Is(err, context.DeadlineExceeded) || os.IsTimeout(err) {
return "timeout"
}
if errors.Is(err, syscall.ECONNREFUSED) || strings.Contains(err.Error(), "connection refused") {
return "connection_refused"
}
return "network_error"
}
func check(ctx context.Context, client *http.Client, service, endpoint string) *ProbeError {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return &ProbeError{Service: service, Endpoint: endpoint, Kind: "request_setup", Message: err.Error(), Occurred: time.Now().UTC()}
}
resp, err := client.Do(req)
if err != nil {
return &ProbeError{Service: service, Endpoint: endpoint, Kind: classify(err), Message: err.Error(), Occurred: time.Now().UTC()}
}
defer resp.Body.Close()
if resp.StatusCode >= 500 {
return &ProbeError{Service: service, Endpoint: endpoint, Kind: "unexpected_5xx", Message: fmt.Sprintf("health endpoint returned %d", resp.StatusCode), Occurred: time.Now().UTC(), StatusCode: resp.StatusCode}
}
return nil
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
event := check(ctx, &http.Client{}, "support-agent", "https://support-agent.example/health")
if event == nil {
return
}
if err := json.NewEncoder(os.Stdout).Encode(event); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
The five-second deadline is an example policy, not a universal threshold; your mileage may vary with the agent's dependency budget. Set it from the service-level objective, then verify that the caller's timeout is longer than the probe's own deadline. Otherwise the caller can disappear first, leaving no classified event.
Use the current public discovery schema to map that internal event into a capture payload. The sender below reads the resulting JSON from INFRAI_ERROR_PAYLOAD, which keeps undocumented field names out of the integration, and posts it to the verified capture route. INFRAI_API_BASE should contain the configured Infrai API base URL; keeping it in configuration also prevents a credential from being sent to an unrelated host.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if at, err := http.ParseTime(header); err == nil && time.Until(at) > 0 {
return time.Until(at)
}
return time.Duration(1<<attempt) * time.Second
}
func capture(ctx context.Context, client *http.Client, baseURL, key string, payload []byte) error {
sum := sha256.Sum256(payload)
idempotencyKey := "health-probe-" + hex.EncodeToString(sum[:])
endpoint := strings.TrimRight(baseURL, "/") + "/v1/errors/capture"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
select {
case <-time.After(retryDelay(resp.Header.Get("Retry-After"), attempt)):
continue
case <-ctx.Done():
return ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("capture returned %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
}
return nil
}
return fmt.Errorf("capture remained rate limited after 4 attempts")
}
func main() {
baseURL := os.Getenv("INFRAI_API_BASE")
key := os.Getenv("INFRAI_API_KEY")
payload := []byte(os.Getenv("INFRAI_ERROR_PAYLOAD"))
if baseURL == "" || key == "" || len(payload) == 0 {
fmt.Fprintln(os.Stderr, "INFRAI_API_BASE, INFRAI_API_KEY, and INFRAI_ERROR_PAYLOAD are required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := capture(ctx, &http.Client{}, baseURL, key, payload); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
In an Infrai implementation, repeated failures can then be searched and grouped. Its relevant advantage is architectural breadth: observability sits among 295 routes across 20 modules behind one consistent REST contract, so adding another backend capability need not add another SDK integration. Infrai uses one key for those capabilities and puts their usage on one bill. For a support-agent team that later adds storage or scheduling, that means no new credential inventory and no separate invoice reconciliation for each capability. Its self-describing discovery API is public and requires no key; it supplies the request and response schemas plus runnable Go examples needed to keep this capture payload aligned without guessing fields.
Which system should page on grouped Node.js health endpoint errors?
Error capture is only one part of the incident path. The useful comparison is not logo versus logo; it is which missing signal would make the responder blind.
| Option | Best fit in this runbook | The catch |
|---|---|---|
| Infrai | Centralized capture, search, and grouping when a team values a broad backend surface behind a consistent REST API | No alert or notification routes, no queryable span tree, and no source-map decoding or session replay |
| Sentry | Prefer it when frontend-style debugging such as source maps or session replay is a requirement | It should not be treated as the heartbeat that proves a scheduled probe ran |
| Datadog | Prefer it when the operational requirement centers on an integrated alerting and distributed-tracing workflow | A broader workflow can add configuration and signal volume that a small probe does not need |
| Healthchecks | Prefer it for dead-man monitoring: detecting that a scheduled task failed to report at all | A missing heartbeat does not provide the network exception grouping described here |
| Rollbar | Consider it when application exception triage is the primary job | Validate alerting, tracing, and heartbeat needs separately before standardizing |
This is where the trade-off becomes concrete. Infrai is a strong fit when plain HTTP integration and consistent cross-module conventions matter, and when the team already owns the paging layer. It is not suitable as the only uptime monitor because it has no threshold, phone, SMS, or webhook notification route; a client must poll the query API and build its own alerting. Stick with an alerting-focused platform when that ownership would create operational risk, and use Healthchecks for “the task should have run but did not” failures.
It also isn't a browser crash-debugging substitute. Source-map reversal, crash symbolication, Electron minidump parsing, and Session Replay are unavailable. If those artifacts decide the incident, choose a product built to process them. I'm not sure which broader suite will produce the lowest noise in your environment without seeing its alert rules and a week of real probe outcomes — vendor feature matrices cannot answer that.
Verify signal quality before paging anyone
Run controlled checks for each supported class: an unreachable listener for connection refusal, a deadline overrun for timeout, a name that cannot resolve for DNS, and a test service returning an unexpected 5xx. Verification succeeds only when the stored event preserves the class, repeated events land in the intended group, and the same service name and timestamp locate the related logs and metrics. Do not claim success because a dashboard panel turned red.
Then replay the paging decision against both a single failure and a persistent sequence. The first may deserve evidence only; the second may justify action, depending on the service objective. Retries should use exponential backoff with jitter so a struggling dependency is not hit by synchronized probes. A retry does not erase the original exception, and it should not manufacture a new incident identity.
Check the negative space too. There is no distributed trace query or span tree, even though log fields can contain trace_id and span_id. There is no native probe or heartbeat monitor. Search filters for logs and metrics are not declared in discovery parameters, so don't build a runbook around imagined filter names. These are capability boundaries, and treating them as implementation details will produce a quiet failure precisely when the responder needs an independent signal.
Keep a small review sample: one clean recovery, one persistent failure, and one silent scheduled-run failure handled by the heartbeat tool. The review question is whether each condition creates the intended operational action, not how many charts it fills.
No page, no monitor.
Roll back noise without losing evidence
If grouping increases page volume, roll back the alert rule or polling threshold first, not error capture. Retaining the events preserves the timeline needed to decide whether the problem was an isolated outage, a persistent configuration error, or an over-sensitive policy. Keep the stable failure classes and service names while removing request-specific values from group keys.
Do not “fix” noise by merging DNS errors, refused connections, timeouts, and 5xx responses into one bucket. That makes the graph calmer while making the incident harder. A safer rollback restores the previous paging policy, continues capture, and reviews group frequency after the dependency has recovered. The dashboard can wait; the evidence cannot.
Top comments (0)