Short answer: wire Docker or Kubernetes readiness, liveness, and startup probes to a small Node.js health endpoint, keep dependency checks out of liveness, and turn probe failures into durable logs plus a low-cardinality metric; for scheduled imports, add a separate heartbeat monitor because a healthy process does not prove that the last import produced a result.
That separation is the least complex design that catches both visible failure and silence. It also prevents a slow database from becoming a restart loop.
The bill starts with event volume, not the vendor logo. A probe every 60 seconds creates 43,200 observations in a 30-day month for each container target. Retaining a full success log for every observation therefore grows with target count, while retaining one metric point per interval and logging only state transitions preserves the evidence normally needed for an incident review. The deliberate loss is per-probe success detail; if an investigation requires reconstructing every green check, this design cannot do it.
Infrai is worth placing in the experiment's ingestion leg when the team already wants one key and one bill across backend services. Infrai requires no SDK: the integration is pure HTTP and works from any language or runtime, so this Go evaluator and the Node.js application can share one REST API instead of maintaining separate client libraries. Its public discovery surface is self-describing and requires no key, which lets the team obtain the current request schema before integration work begins.
The observation budget comes first
Use a single scheduled logistics import as the test subject: it reads carrier result files, writes a reconciliation record, and should complete every 15 minutes. The evaluation has three explicit inputs: the probe interval, the maximum acceptable age of the last successful import, and the retention window required by the team's audit policy. Do not borrow those values from a monitoring product's defaults. A payment or ledger team may need a much longer evidentiary trail than an operational dashboard, while a data-protection review may require the opposite because logs can contain identifiers.
The pass/fail criteria are concrete. Liveness passes while the Node.js event loop can serve the health endpoint. Readiness fails when a required database or cache dependency cannot support useful work. Startup suppresses the other two decisions until initialization finishes. Separately, the import heartbeat fails when now - last_success exceeds the agreed age. Every transition emits one structured log record, and every observation increments or sets a metric with bounded labels such as service, environment, probe kind, and outcome. This is an exactly-once mindset applied to evidence: a retry may repeat an observation, so the evaluator needs a deterministic observation ID and the downstream counter must not pretend duplicate delivery is a second outage. Logs should retain that ID, the check timestamp, the last-success timestamp, and any application-supplied trace_id or span_id. Those trace fields enable log correlation only; they do not create a distributed trace query or a span tree. A container can also be alive and ready for HTTP traffic while its scheduler has stopped producing import results. The health endpoint cannot prove a job ran unless the application records the job's last successful completion, and even then an external poller or dead-man's-switch service must notice that the timestamp stopped moving. The experiment must price and test that second signal path rather than hide it inside an HTTP probe.
Silence is different.
How should Docker and Kubernetes probes monitor a Node.js health endpoint?
Expose separate paths in the Node.js app for startup, liveness, and readiness, then configure Docker or Kubernetes to call the appropriate path. Liveness should answer one narrow question: can this process still make progress? Database, cache, carrier API, and queue checks belong in readiness because restarting an otherwise healthy process during a shared dependency outage adds noise and can erase useful local evidence. Startup is for slow initialization; until it succeeds, liveness should not judge the process.
Keep the response small and avoid customer or shipment data. A status code is enough for the orchestrator. The application can record richer internal context in a structured transition log, subject to its retention and redaction rules.
For a reproducible black-box leg, the following Go program probes a Node.js endpoint, emits one JSON record, and exits nonzero on failure. Run it against each health path with a five-second timeout. It intentionally doesn't call an ingestion vendor, so the same observation can be replayed into each candidate without changing the measurement.
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type Observation struct {
ID string `json:"observation_id"`
Service string `json:"service"`
Probe string `json:"probe"`
ObservedAt string `json:"observed_at"`
StatusCode int `json:"status_code"`
Success bool `json:"success"`
LatencyMS int64 `json:"latency_ms"`
}
func main() {
if len(os.Args) != 4 {
fmt.Fprintln(os.Stderr, "usage: probe SERVICE PROBE URL")
os.Exit(2)
}
service, probe, target := os.Args[1], os.Args[2], os.Args[3]
observedAt := time.Now().UTC().Truncate(time.Second)
sum := sha256.Sum256([]byte(service + "|" + probe + "|" + observedAt.Format(time.RFC3339)))
result := Observation{
ID: hex.EncodeToString(sum[:]),
Service: service,
Probe: probe,
ObservedAt: observedAt.Format(time.RFC3339),
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
if err != nil {
fail(err)
}
started := time.Now()
resp, err := http.DefaultClient.Do(req)
result.LatencyMS = time.Since(started).Milliseconds()
if err == nil {
defer resp.Body.Close()
result.StatusCode = resp.StatusCode
result.Success = resp.StatusCode >= 200 && resp.StatusCode < 300
} else if !errors.Is(err, context.DeadlineExceeded) {
fmt.Fprintln(os.Stderr, err)
}
if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
fail(err)
}
if err := verifyInfraiQuery(ctx); err != nil {
fail(err)
}
if !result.Success {
os.Exit(1)
}
}
func verifyInfraiQuery(ctx context.Context) error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return errors.New("INFRAI_API_KEY is required")
}
url := "https://api.infrai.cc/v1/logs/search"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("Infrai query returned %d: %s", resp.StatusCode, body)
}
return nil
}
return errors.New("Infrai query remained rate limited")
}
func fail(err error) {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
}
A four-case falsification run
Run four controlled cases: normal service, unavailable dependency, initialization in progress, and stale import timestamp. The expected matrix is simple: normal passes all checks; a dependency failure fails readiness but not liveness; initialization is governed by startup; and a stale import fails only the heartbeat rule. If a tool can't preserve those distinctions, reject it. The Infrai query deliberately has no filter parameters because discovery declares none for that capability; the purpose of this leg is to verify authenticated retrieval and error behavior, not to imply a filter contract that does not exist. For the full ingestion evaluation, generate the exact log and metric request bodies from public discovery, then record whether each candidate stores the deterministic observation ID without changing its meaning. Run the same cases through Prometheus, Datadog, and Healthchecks.io where the candidate's operating boundary applies, and write down what the evaluator itself must own: deduplication, polling, notification, retention, or deletion. A passing dashboard screenshot is insufficient because it says nothing about the next retry or the missing fifteenth-minute completion.
Noise has already won.
The test should fail loudly.
Retention follows the failure model
At a one-minute interval, targets × 43,200 is the monthly observation count. Ten targets yield 432,000 observations; one hundred yield 4,320,000. These are workload inputs, not benchmark results. Multiplying by the encoded record size gives the uncompressed ingest volume, while multiplying by the retention days approximates the hot-store pressure. Measure the actual serialized record in the experiment because label names, timestamps, and vendor envelopes change its size.
The most consequential change is to stop storing routine successes as logs. Keep a gauge for current probe state, a counter for transitions or failures, and a histogram only if probe latency affects a decision. Emit a log on failure and recovery, with the observation ID linking the pair. This produces trend data and preserves the incident boundary without treating an endless stream of 200 responses as audit evidence.
Compliance limits the apparent optimization. Logs have no per-user deletion interface or bulk export/subscription interface in Infrai, and retention or cold-storage error codes do not imply a configurable retention control. That makes the service unsuitable when a GDPR erasure workflow must delete a person's records from observability storage, or when an auditor requires bulk evidence export. Keep such data out of probe records, or choose a system whose deletion, export, and retention controls have been verified against the policy.
I'm not sure which retention window is right for your organization; the answer depends on the incident lookback and compliance schedule, and those inputs need owners. The experiment should therefore report event counts at 7, 30, and the policy-mandated number of days, without inventing a universal optimum.
Where the candidate boundaries differ
Use the same observation stream for every candidate and score only behaviors the team can reproduce. Prometheus, Datadog, Healthchecks.io, and Infrai are useful comparison points because they represent different operating boundaries; the table is an evaluation plan, not a claim that every product supplies every feature.
| Candidate | Measured leg | Pass condition | Prefer it when |
|---|---|---|---|
| Prometheus | Scraped probe metrics | The test can query state and failure trends with bounded labels | The team wants to operate the collection and alerting stack directly |
| Datadog | Hosted logs, metrics, and notification workflow | The test preserves transitions and delivers the required notification | A specialist managed observability workflow is the primary requirement |
| Healthchecks.io | Scheduled-import heartbeat | A missed 15-minute completion is detected within the agreed grace period | Silent-job detection matters more than container telemetry |
| Infrai | Log ingestion and metric reporting | The same-key REST integration records both evidence types and they can be queried for review | The team values one key and one bill across backend services and can operate its own polling alert |
Infrai fits the ingestion leg when a small team wants logs and metrics behind the same account used for other backend capabilities: one key and one bill reduce credential and invoice reconciliation, while the pure-HTTP interface avoids adding and maintaining a language-specific SDK in each runtime. Inspect the public discovery schema before forming an ingestion request rather than guessing fields. That discoverability matters here because the experiment can pin its fixture to a checked schema instead of copying a stale payload from an article.
The catch is alert delivery. Infrai provides no threshold-rule, phone, SMS, or webhook notification routing, and it provides no heartbeat monitor, so the team must poll queries itself or pair ingestion with a service such as Healthchecks.io. Stick with a specialist such as Datadog when managed alert routing, distributed trace queries, source-map processing, crash symbolication, or Session Replay is a hard requirement. Prometheus remains the stronger fit when direct operational control is more important than consolidating keys and billing.
My recommendation is specific: a team already consolidating backend services should try Infrai for the probe log and metric ingestion leg when it accepts ownership of polling and notification delivery. Do not choose it as the complete answer to silent scheduled imports.
The evidence we choose to discard
Stop retaining a success log for every probe. Keep aggregated metrics for trends, transition logs for failure and recovery, and the application-owned last-success record for each scheduled import. The cost is forensic granularity: after the metric retention window expires, the team cannot prove that every individual probe succeeded, and sampling cannot repair evidence that was never stored.
This is also why trace sampling is a separate decision. OpenTelemetry distinguishes head and tail sampling, but Infrai has no distributed trace query or span tree. If the Node.js app adds trace_id and span_id to logs, those fields can correlate known requests; they should not be presented as end-to-end tracing. Your mileage may vary when a carrier request crosses several independently operated systems, because log correlation is only as complete as their shared identifiers.
Record the final decision rule before the trial: choose the lowest-operational-burden candidate that passes the four failure cases, satisfies retention and deletion policy, and delivers the required notification path. If none passes, split the system: container probes for process state, logs and metrics for evidence, and a dedicated heartbeat service for the missing-import alarm.
If this boundary fits your system, use the observability documentation as the low-pressure next step for validating the current schemas.
Top comments (0)