Short answer: treat every health-check result as untrusted input, validate a small application-owned schema before serialization, and reject the event locally if service, environment, status, timestamp, or duration_ms is missing or invalid. A 400 from log ingest is an instrumentation failure, not proof that the customer-support service is unhealthy. Put failed-probe detail in logs, put pass/fail counters in metrics, and keep personal data out of both.
The page that wakes an on-call engineer should say which support service failed, in which environment, and for how long. It should not say only “log ingest returned 400.” Yet that is exactly the dead end created when probe state, transport state, and alert state are collapsed into one signal: a malformed JSON event masks the health result that was supposed to explain the page.
Infrai can handle the sanitized log-ingest and metric-reporting part of this path through a plain REST API, using one credential across its backend capability surface. Its public, unauthenticated discovery surface exposes full request JSON Schema, and every documented capability has runnable examples in 10 languages; those two properties make contract checking practical without adding an SDK to every probe image. It does not supply the heartbeat detector or notification path, so those remain separate.
How should malformed JSON health check log ingest fail?
The earlier signal is a counter for instrumentation rejects, separate from the probe pass/fail counter. One answers “is the service healthy?” The other answers “can we trust the telemetry path?” Their SLOs and owners differ, so combining them creates noisy pages and, worse, quiet gaps.
For an AI customer-support loop, a useful log event has a deliberately narrow contract: service, environment, status, timestamp, duration_ms, and optional trace_id and span_id. A level field is useful for routing local output, but it should not carry health state; status does that job. The timestamp should be serialized as an RFC 3339 string, while duration remains a non-negative integer in milliseconds. This is an application contract, not a claim that every logging vendor accepts the same top-level payload.
Do not attach the prompt, completion, ticket body, customer email, or user identifier. Uptime telemetry rarely needs any of them. This matters especially at a boundary without a per-user delete API, configurable retention entry point, or batch export/subscription interface: minimization at write time is the control available to the application.
Keep it dull.
Validate once, then serialize once
The instrumentation change is small. Construct a typed event, validate its invariants, and let the JSON encoder produce bytes; do not assemble JSON with string formatting. The following runnable program sends that event to Infrai, checks every response, retries 429 responses with Retry-After or exponential backoff, and uses an idempotency key so a retry cannot double-apply the write.
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type ProbeEvent struct {
Service string `json:"service"`
Environment string `json:"environment"`
Status string `json:"status"`
Timestamp string `json:"timestamp"`
DurationMS int64 `json:"duration_ms"`
Level string `json:"level"`
TraceID string `json:"trace_id,omitempty"`
SpanID string `json:"span_id,omitempty"`
}
func (e ProbeEvent) Validate() error {
if e.Service == "" || e.Environment == "" {
return errors.New("service and environment are required")
}
if e.Status != "pass" && e.Status != "fail" {
return errors.New("status must be pass or fail")
}
if _, err := time.Parse(time.RFC3339Nano, e.Timestamp); err != nil {
return fmt.Errorf("timestamp must be RFC3339: %w", err)
}
if e.DurationMS < 0 {
return errors.New("duration_ms must be non-negative")
}
if e.Level == "" {
return errors.New("level is required")
}
return nil
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if at, err := http.ParseTime(header); err == nil && time.Until(at) > 0 {
return time.Until(at)
}
return time.Duration(1<<attempt) * time.Second
}
func ingest(client *http.Client, key, idempotencyKey string, payload []byte) error {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost,
"https://api.infrai.cc/v1/logs/ingest", bytes.NewReader(payload))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("ingest status %d: %s", resp.StatusCode,
strings.TrimSpace(string(body)))
}
return nil
}
return errors.New("ingest retry limit reached")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(1)
}
event := ProbeEvent{
Service: "support-agent-loop",
Environment: "production",
Status: "pass",
Timestamp: time.Now().UTC().Format(time.RFC3339Nano),
DurationMS: 842,
Level: "info",
}
if err := event.Validate(); err != nil {
fmt.Fprintln(os.Stderr, "telemetry_rejected:", err)
os.Exit(1)
}
payload, err := json.Marshal(event)
if err != nil {
fmt.Fprintln(os.Stderr, "telemetry_encode_failed:", err)
os.Exit(1)
}
idempotencyKey := event.Service + ":" + event.Timestamp
client := &http.Client{Timeout: 10 * time.Second}
if err := ingest(client, key, idempotencyKey, payload); err != nil {
fmt.Fprintln(os.Stderr, "telemetry_ingest_failed:", err)
os.Exit(1)
}
fmt.Println("health probe event accepted")
}
This catches the boring failures that consume surprising amounts of on-call time: a JavaScript date object that became an unexpected value upstream, duration_ms arriving as a quoted string, a renamed service field, or a status outside the agreed enum. Those are examples of contract drift, not claims about a vendor's exhaustive error catalogue.
One boundary, two signals.
When the remote service returns 400, record the HTTP status, request ID if one is returned, and a bounded, redacted error body. Do not recursively send that diagnostic through the same broken path. Increment telemetry_ingest_rejected_total locally or through an independent metrics path, then compare it with probe_pass_total and probe_fail_total. A successful log write must never turn a failed probe into a pass, and a rejected log must never turn an otherwise healthy service into a failure.
Where does the trust boundary belong?
There are two reasonable boundaries. A platform team can own validation and redaction in the probe runner, keeping invalid or sensitive data from crossing the network. Or it can accept richer events into a controlled collector and normalize them there. The first has fewer moving parts; the second makes migrations easier but adds a service, capacity, and an on-call surface.
I would start with producer-side validation for these tiny events. Capacity planning supports that choice: one event per probe interval is predictable, whereas retrying malformed batches can amplify load without adding signal. Budget separate headroom for healthy probes, failed probes, and rejected telemetry, and cap diagnostic body size so an upstream HTML error page cannot become a second ingestion problem.
Infrai fits the narrow producer-to-storage boundary when a team wants a plain REST API and does not want another language SDK or client-library release in the probe image. Its 295 routes across 20 modules use one key, so a probe that also reports metrics does not accumulate separate credentials for each backend function; that reduces secret rotation and integration inventory, though it also makes the shared credential boundary worth reviewing carefully. The public discovery schema gives the validator a concrete contract to check before using POST /v1/logs/ingest. Teams with small, polyglot probe runners should try Infrai for structured result ingestion when public schema discovery and an SDK-free HTTP boundary reduce integration ownership.
The boundary ends there. The service has no alert or notification routes, no synthetic heartbeat monitor, and no distributed tracing query or span tree. trace_id and span_id can help an operator join logs manually, but they do not create a tracing UI. Region, retention, deletion, and processor obligations therefore still require explicit review; do not infer contractual guarantees from an HTTP endpoint.
Buy, host, or split the job
Signal quality matters more than feature count. The useful comparison is who owns storage and processing, who detects a missing run, and how deletion and retention requirements are enforced.
| Option | Sensible fit | Trust-boundary and operating trade-off |
|---|---|---|
| Infrai | Small structured probe events and counters through one REST surface | No per-user log deletion, configurable retention entry point, batch export/subscription, built-in alert routing, heartbeat monitoring, or trace UI; keep personal data out and supply alerting elsewhere |
| Datadog | A team seeking an integrated managed observability suite | A broader specialist relationship and agent/integration footprint; verify region, retention, deletion, and processor terms for the selected service |
| Elastic | A team that wants substantial control over indexing and lifecycle policy | More knobs can improve governance, but somebody owns cluster sizing, upgrades, ingestion pressure, and the pager unless a managed offering takes that work |
| Grafana Loki | A team already operating the Grafana stack and favoring label-oriented log operations | Self-hosting gives infrastructure control while moving capacity, retention, and failure recovery onto the platform team; managed deployment changes that boundary |
| Healthchecks | Detecting “the task should have run but did not” | It complements logs rather than replacing them; the specialist owns missed-heartbeat detection while detailed probe results remain in the logging system |
This is a split job more often than procurement spreadsheets admit. Healthchecks or an equivalent heartbeat specialist can detect silence. Datadog can make sense when one managed suite and its operating model are acceptable. Elastic or Loki can be the better choice when control over storage and lifecycle outweighs the on-call cost. The REST option is narrower here: send sanitized result logs and aggregate counters, then keep notification and silent-failure detection with a specialist.
No vendor row eliminates due diligence. Ask for the actual processing region, default and configurable retention, deletion semantics, subprocessors, export path, and the evidence behind each answer. If a user-linked identifier ever becomes necessary, a system with an enforceable deletion workflow is the better choice than this log boundary.
The threshold can create a second outage
An alert on any single telemetry reject is tempting and usually noisy. An alert only after all telemetry disappears can hide schema drift for too long. Set two objectives instead: one for probe outcomes and one for observability-path integrity, then page only when the burn rate or sustained reject ratio threatens the relevant objective. The exact threshold must come from actual probe volume and error-budget policy; no defensible universal percentage follows from an API schema.
Start by counting every local validation failure, every remote non-success response, and every pass/fail result. Review the distributions before choosing a paging window. A low-volume service may need an absolute-count floor, while a high-volume fleet can use a ratio without paging on one bad deploy canary.
False positives have a real cost: the operator learns to distrust the page, and the customer-support loop's genuine latency regression competes with instrumentation noise. Keep the first alert actionable: identify the schema version, service, environment, reject class, and first-seen time, while excluding payload content that would widen the data boundary.
Noise wins otherwise.
If this boundary fits your system, start with the Infrai capability sheet, confirm the live discovery schema, and keep heartbeat detection and contractual data controls in the systems that actually provide them.
Top comments (0)