A small, explicit polling loop is the safest starting point for HTTP health checks: one request, one deadline, one normalized result, and a decision rule that separates a bad endpoint from a noisy cohort. For a customer-support experiment, I would use that loop to compare tenant cohorts only after the health signal is good enough to trust.
Short answer: use the platform HTTP client available in Node.js, keep the health endpoint narrow, set an AbortController deadline, and poll on a fixed interval with jitter. Record status, latency, timeout, and error class as separate fields. Do not turn a single failed poll into an experiment conclusion. Require a small run of consecutive failures, and roll back the monitoring change independently from the experiment.
This is a runbook, not a vendor recommendation. The important API is the contract around the request.\n\nThe trade-off is deliberate: this pattern is a poor fit for a full user journey, long-running workflow, or event-driven check. Use a synthetic transaction or an event consumer when the question is behavior beyond readiness. A narrow poll is easier to trust, but it cannot prove that every support workflow works.\n\nKeep it boring.
What question should the health check answer?
A health check should answer one operational question. Is this service ready to receive the kind of traffic represented by this check? A route that runs a full customer-support workflow, calls three dependencies, and returns a large diagnostic payload is not a health check anymore. It is a synthetic transaction, with different latency and failure semantics. Keep those signals separate.
For a tenant-cohort experiment, define the unit before writing the poller. A tenant belongs to cohort A or B, and each tenant contributes observations only while its service path is healthy. The numerator might be successful responses from the experiment, while health-check failures and client timeouts belong in an exclusion or data-quality stream. Mixing them makes a weak endpoint look like a treatment effect.
I have been paged for missed jobs and duplicate deliveries, so I treat retry behavior as part of the API design. A retry can create another request, another log record, and sometimes another side effect. A health check must be read-only. The response should expose a small state such as ok, degraded, or failed; the poller should attach its own timestamp and attempt number rather than trusting a remote clock.
The five signals I keep are availability, HTTP status class, latency, timeout count, and error class. They are useful together.\n\nOne endpoint. One deadline. One result. Availability without latency hides saturation. Latency without status hides an error page that returned quickly. A single aggregate score hides both.
Is polling really the simplest API?
Yes, if the requirement is a periodic HTTP observation and the check has no side effect. The simplest interface is a function that accepts a URL and deadline, then returns a typed result. Scheduling is a separate concern. This boundary makes the check testable and prevents a timer callback from becoming an accidental queue.
The question is not whether the loop can make a request. It can.
Measure the edges.
The question is what happens at the edges: a response arrives after the deadline, the process is suspended, a DNS error repeats, or two ticks overlap because the previous request is slow. Those cases decide signal quality.
Here is the shape I use for the request itself. The example is in Go because the operational rules are easier to see without hiding them behind framework middleware; the same contract can sit behind Node.js's standard HTTP client. It performs no retry and no business action.
package health
import (
"context"
"fmt"
"net/http"
"time"
)
type Result struct {
StatusCode int
Latency time.Duration
Outcome string
ErrClass string
}
func Check(ctx context.Context, client *http.Client, url string) Result {
started := time.Now()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return Result{Latency: time.Since(started), Outcome: "failed", ErrClass: "request_build"}
}
resp, err := client.Do(req)
latency := time.Since(started)
if err != nil {
if ctx.Err() != nil {
return Result{Latency: latency, Outcome: "timeout", ErrClass: "deadline"}
}
return Result{Latency: latency, Outcome: "failed", ErrClass: "transport"}
}
defer resp.Body.Close()
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return Result{StatusCode: resp.StatusCode, Latency: latency, Outcome: "ok"}
}
return Result{StatusCode: resp.StatusCode, Latency: latency, Outcome: "failed", ErrClass: "http_status"}
}
func CheckWithDeadline(client *http.Client, url string, limit time.Duration) Result {
ctx, cancel := context.WithTimeout(context.Background(), limit)
defer cancel()
return Check(ctx, client, url)
}
var _ = fmt.Sprintf
The code records a timeout distinctly from a non-2xx response. That distinction matters during cohort analysis: a timeout often points to load or a network path, while a 503 can be an intentional readiness signal. Do not collapse them into false. The practical trap is a mixed interval: tenant A returns 503 in 180 ms because its dependency is deliberately unready, tenant B takes 2.4 seconds and hits the 2-second deadline, and tenant C returns 200 in 90 ms. If those three observations become one failed count, the report loses the reason and the operator cannot tell whether to wait, investigate capacity, or remove a dependency. Keep the records separate, classify the interval as noisy when the evidence is mixed, and make the experiment consumer read that classification before calculating a cohort difference.
How do you stop overlapping polls from corrupting the signal?
Use a fixed cadence with a single in-flight request per target. If one check takes 12 seconds and the interval is 10 seconds, starting a second request at the next tick creates concurrency that the service did not ask for. It also makes the observed latency depend on the monitor's own backlog. Skip the tick, or wait for completion and schedule the next run from the completion point. Choose one policy and emit it as a metric.
A small scheduler can make the policy explicit. Jitter should be bounded and deterministic enough to test; the exact distribution is less important than avoiding a fleet of monitors hitting the same second.
package health
import (
"context"
"math/rand"
"time"
)
func Poll(ctx context.Context, interval, jitter time.Duration, run func() Result, publish func(Result)) {
timer := time.NewTimer(0)
defer timer.Stop()
for {
select {
case <-ctx.Done():
return
case <-timer.C:
result := run()
publish(result)
delay := interval
if jitter > 0 {
delay += time.Duration(rand.Int63n(int64(jitter)))
}
timer.Reset(delay)
}
}
}
This scheduler does not overlap because run completes before the timer is reset.
Never hide a missed tick.
In a production implementation, use a concurrency-safe random source or inject a clock and jitter function so tests can control both. A poller's own failure to schedule is an observable event; count it instead of silently treating the target as healthy.
For customer-support cohorts, I would start with a 30-second interval and a 2-second request deadline, then validate those values against the service's normal response distribution. Those are policy inputs, not universal defaults. A deadline shorter than ordinary tail latency manufactures noise; one longer than the interval invites overlap.
What should be emitted for a useful comparison?
Give every observation a target identifier, tenant cohort, check name, attempt number, timestamp, outcome, status code when present, latency, and error class. Keep tenant identifiers low-cardinality in metrics; put detailed identifiers in logs with access controls. OpenTelemetry describes logs as a signal with structured records and attributes, which is a useful model for preserving detail without turning every possible tenant value into a metric label.
The data pipeline should make exclusion visible. An experiment report can then say, “cohort B had 2.1% timeout observations and was excluded from this interval,” rather than silently dropping rows. Do not infer a customer outcome from availability alone.
package health
import (
"encoding/json"
"os"
"time"
)
type Observation struct {
Check string `json:"check"`
Cohort string `json:"cohort"`
Attempt uint64 `json:"attempt"`
ObservedAt time.Time `json:"observed_at"`
Outcome string `json:"outcome"`
StatusCode int `json:"status_code,omitempty"`
Latency time.Duration `json:"latency"`
ErrClass string `json:"error_class,omitempty"`
}
func WriteObservation(value Observation) error {
encoder := json.NewEncoder(os.Stdout)
return encoder.Encode(value)
}
Idempotency applies to processing these observations too. Give each poll attempt an identifier, or use a stable key made from target, scheduled time, and attempt. A downstream consumer should be able to receive the same observation twice without counting two incidents. This is where a harmless health probe can still create a duplicate-delivery problem if the reporting path is not designed for it.
A practical decision rule is consecutive evidence, not a single threshold crossing. For example, page after three failed observations in a row, recover after two successful observations, and mark a cohort interval “insufficient” when timeout share crosses a threshold chosen from baseline data. The numbers must be reviewed against the service's normal tail latency. They are not a substitute for that baseline.
How do you verify and roll back the monitor?
Test the check with a local server that returns a success, a 503, a delayed response, malformed addressing, and a connection close. Assert the outcome and error class for each case. Add a test where the first poll takes longer than the interval; the scheduler must prove that the second request does not begin before the first finishes. Fake clocks make this deterministic.
Before deployment, compare the new monitor with the existing one for a fixed observation window. This comparison is the long part: inspect request rate, timeout share, duplicate observations, cohort exclusions, and alert transitions together, then review a few raw records to confirm that a delayed response was not mislabeled as a clean failure.\n\nDo not optimize the dashboard first. Look for changes in request rate, timeout share, duplicate observations, and the number of cohort intervals marked insufficient. A quieter dashboard is not proof of a better signal.
Rollback should remove or disable the schedule, not delete the observations already written. Preserve the raw records long enough to explain a cohort decision, then apply the retention policy for that data. If the monitor itself is causing load, reduce its target set and cadence first; do not respond by hiding failures.
The final check is operational: can an on-call engineer tell whether the service failed, the network timed out, the poller overlapped, or the experiment data was excluded? If those states have different fields and different alerts, this small API is doing its job.
Top comments (0)