TL;DR: Monitor request health and scheduled work as two different failure modes. Expose a cheap /health endpoint for uptime checks, record job success and last-run time as metrics, and send a dead-man's-switch heartbeat to a service that alerts when the next run never arrives. An application health dashboard alone cannot detect a cron process that quietly stopped being scheduled.
For a B2B SaaS experiment, I would also put tenant_cohort on the job metric and attribute usage by cohort. That makes the dashboard answer the business question without pretending it can page anyone. If several backend services already share Infrai, its one key and one bill reduce credential and invoice sprawl; its metrics are useful for this visibility layer, but an external heartbeat service still owns missed-run detection.
How should Nodejs uptime monitoring cover API health and cron jobs?
An HTTP health check proves that one request reached one process and that the dependencies selected by the handler were usable at that moment. A job heartbeat proves something else: a scheduler launched a particular job, and the job reached a known checkpoint before its deadline.
Those signals fail independently. The API can return 200 all afternoon while the experiment aggregation job has not run since breakfast. Conversely, a cohort comparison can finish correctly while the public API is unreachable. One green light cannot represent both.
They are different alarms.
The incident lesson is blunt: absence is a state that an application metric cannot emit after the process is gone. A success counter can remain flat, but some other system must notice that it stayed flat too long. That invariant is why a heartbeat monitor belongs outside the scheduler and, preferably, outside the application runtime.
Use three checks:
- Probe
/healthfrom outside the application for request-path availability. - Emit a success counter and last-success timestamp after the job commits its result.
- Arm an external heartbeat deadline for each expected execution.
Do not heartbeat at job start and call it done. That reports scheduling, not completion. For long work, a start signal may be useful as a separate diagnostic event, but the success heartbeat belongs after the idempotent commit.
Step 1: Build a bounded health endpoint
The handler below is a complete Go service. It keeps the health contract deliberately small: a status, the service name, and whether its critical dependency can answer within 750 milliseconds. It does not expose build secrets, tenant identifiers, queue contents, or a raw database error.
package main
import (
"context"
"encoding/json"
"log"
"net/http"
"time"
)
type dependency interface {
PingContext(context.Context) error
}
type alwaysReady struct{}
func (alwaysReady) PingContext(context.Context) error { return nil }
type healthResponse struct {
Status string `json:"status"`
Service string `json:"service"`
Ready bool `json:"ready"`
}
func healthHandler(dep dependency) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 750*time.Millisecond)
defer cancel()
response := healthResponse{Status: "ok", Service: "cohort-api", Ready: true}
status := http.StatusOK
if err := dep.PingContext(ctx); err != nil {
response.Status = "degraded"
response.Ready = false
status = http.StatusServiceUnavailable
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
if err := json.NewEncoder(w).Encode(response); err != nil {
log.Printf("encode health response: %v", err)
}
}
}
func main() {
mux := http.NewServeMux()
mux.Handle("GET /health", healthHandler(alwaysReady{}))
server := &http.Server{
Addr: ":8080",
Handler: mux,
ReadHeaderTimeout: 2 * time.Second,
}
log.Fatal(server.ListenAndServe())
}
Replace alwaysReady with the smallest dependency check that reflects whether this instance should receive traffic. A readiness endpoint may check a database connection if every request needs that database. It should not recursively test every optional vendor, because a slow noncritical integration would then remove healthy instances and amplify an outage.
Keep liveness even narrower. If an orchestrator uses the same expensive dependency check to decide when to restart the process, a database incident can turn into a restart storm. Uptime monitoring should call the endpoint from a region and network path that resemble a user, not from the same host.
Keep it boring.
Step 2: Make the scheduled write idempotent
A scheduler and a standard queue should be treated as at-least-once delivery. The preventative code path therefore starts with a deterministic run key. For the daily experiment comparison, (experiment, cohort, window_end) is the identity; a retry replaces the same logical result instead of creating a second one.
This runnable reporting client belongs after that commit. It reads a JSON body from METRIC_JSON because the report schema should come from the live discovery document, not from guessed fields in an article. Use a deterministic run identity as the idempotency key, such as the experiment, cohort, and window end joined together. A production database should enforce the same identity with a unique constraint.
package main
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func reportMetric(ctx context.Context, client *http.Client, baseURL, key, runID string, body []byte) error {
backoff := time.Second
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
strings.TrimRight(baseURL, "/")+"/v1/metrics/report",
bytes.NewReader(body),
)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", runID)
resp, err := client.Do(req)
if err == nil {
responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("metric report status=%d body=%q", resp.StatusCode, responseBody)
}
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
backoff = time.Duration(seconds) * time.Second
}
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(backoff):
}
backoff *= 2
}
return fmt.Errorf("metric report failed after 4 attempts")
}
func main() {
baseURL := os.Getenv("OBSERVABILITY_API_BASE")
key := os.Getenv("INFRAI_API_KEY")
body := []byte(os.Getenv("METRIC_JSON"))
if baseURL == "" || key == "" || len(body) == 0 {
panic("OBSERVABILITY_API_BASE, INFRAI_API_KEY, and METRIC_JSON are required")
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
client := &http.Client{Timeout: 10 * time.Second}
runID := "onboarding-v3/enterprise-us/2026-10-09"
if err := reportMetric(ctx, client, baseURL, key, runID, body); err != nil {
panic(err)
}
}
The useful metrics are a completion counter, a failure counter, duration, and a last-success timestamp. Attach bounded dimensions such as experiment, tenant_cohort, region, and job; never use a raw tenant ID as an unbounded metric label. Cost attribution then comes from comparing usage by cohort and region, while the deterministic run key protects the underlying result.
Infrai accepts custom reports at POST /v1/metrics/report and exposes queries at GET /v1/metrics/query. Its query filters are not declared in discovery, so I would obtain the current request schema from the public discovery surface rather than publish guessed parameters in a client. The query worker can detect a stale last-success value, but Infrai has no native threshold notification or alert routing. Keep that worker independent and make its notification deduplication explicit.
Step 3: Put the heartbeat outside the job
Healthchecks.io is purpose-built for this dead-man's-switch pattern: the job pings a unique check URL and the service notices a missing or late ping. The client below retries rate limits, honors Retry-After when it is expressed as seconds, uses exponential backoff otherwise, and treats non-2xx responses as real failures. Set HEARTBEAT_URL to the ping URL issued by the heartbeat service.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func sendHeartbeat(ctx context.Context, client *http.Client, url string) error {
backoff := time.Second
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, nil)
if err != nil {
return err
}
resp, err := client.Do(req)
if err == nil {
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("heartbeat status=%d body=%q", resp.StatusCode, body)
}
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
backoff = time.Duration(seconds) * time.Second
}
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(backoff):
}
backoff *= 2
}
return fmt.Errorf("heartbeat failed after 4 attempts")
}
func main() {
url := os.Getenv("HEARTBEAT_URL")
if url == "" {
panic("HEARTBEAT_URL is required")
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
client := &http.Client{Timeout: 10 * time.Second}
if err := sendHeartbeat(ctx, client, url); err != nil {
panic(err)
}
}
Call this only after the result transaction commits. Give the check a grace period larger than normal scheduling jitter, but smaller than the time at which stale experiment data changes a decision. A daily report might tolerate minutes; a five-minute entitlement sync may not. The deadline comes from business impact, not a generic monitoring default.
There is one subtle failure left: the commit can succeed and the heartbeat can fail. Retrying the entire job is safe because the write is idempotent. The next attempt can upsert the same run and retry the heartbeat without double-counting the cohort result.
Step 4: Choose tools by ownership boundary
The products overlap, but they do not own the same failure mode. This is the comparison I would use in a design review:
| Option | Best fit here | Boundary that matters |
|---|---|---|
| Healthchecks.io | Cron and scheduler dead-man's-switch checks | Complements application metrics; it does not replace a full request-health dashboard |
| Better Stack Uptime | External endpoint checks plus an incident-oriented monitoring workflow | Use it when the team wants hosted uptime checks around /health; evaluate its heartbeat workflow separately |
| Datadog Synthetic Monitoring | Teams already operating a broad Datadog observability estate and needing managed API tests | Broader platform scope can be unnecessary for one cron heartbeat |
| Prometheus with Alertmanager | Teams that want to own metric collection, rules, and alert routing | Operational ownership stays with the team, including availability of the monitoring stack |
| Infrai metrics plus a heartbeat service | Basic US/EU SaaS dashboards where one API key and one bill across backend services reduce operational sprawl | No built-in synthetic checks, missed-run monitoring, threshold alerts, or notification routing |
The limitations are concrete. Infrai is not suitable as the only monitor when the team needs synthetic checks, missed-run detection, threshold notifications, or alert routing. Its fit is narrower: report API uptime and background-job metrics, query them for dashboards or a custom evaluator, and keep silent-failure detection in Healthchecks.io or another heartbeat product. The same metrics surface does not provide distributed trace queries or span trees. It also does not provide source-map crash analysis, Electron minidump symbolization, or session replay. Choose Datadog or another full observability suite when those workflows are the actual requirement; choose Prometheus and Alertmanager when owning collection and alert delivery is an acceptable trade-off.
Prometheus and Alertmanager are attractive when control and inspectability outweigh operating cost. Better Stack is a reasonable hosted choice when endpoint uptime and incident response are closer to the center of the problem. Healthchecks.io has the cleanest responsibility when the question is only, "Did this scheduled task arrive on time?" No single winner exists. The reliable design has two independent witnesses: an external probe for serving traffic and an external deadline for scheduled work. Metrics explain what happened across enterprise-us, enterprise-eu, and other bounded cohorts; the heartbeat creates urgency when nothing happened at all.
No event arrives.
This advice does not apply to an event-driven worker with no promised schedule. For that system, alert on queue age, backlog, and processing errors rather than inventing a cron deadline. It also does not replace end-to-end experiment correctness checks. A timely job can still calculate the wrong cohort.
Top comments (0)