Short answer: keep failure detection outside the service, combine an HTTP health probe with application metrics, and use a heartbeat for work that has no request to poll. Alert only after a small, explicit failure window, then test the whole delivery path.
The operational constraint is easy to miss: the thing that reports an outage must not share the outage's failure domain. A check running in the same Node.js process cannot report that process is gone. A metrics scraper inside a partition cannot tell you that users in another region cannot connect. A heartbeat cannot prove an API is serving traffic. These are different observations, and treating them as interchangeable is how quiet failures become morning surprises. I've seen a 503 page arrive only after a queue had been aging for an hour because the team had measured the web port and nothing else.
It fails quietly.
How should Node.js uptime alerts use a health endpoint for failure detection?
Start with two endpoints, even if they are tiny. A liveness endpoint answers whether the process can accept work. A readiness endpoint answers whether this instance should receive work now. Keep the response boring: a successful check returns 200, and a temporarily unusable dependency returns 503. HTTP semantics define 503 Service Unavailable; a Retry-After header can tell a caller when to try again.
Do not put dependency credentials, connection strings, request bodies, or user identifiers in either response. A health route is often reachable from a public monitor, and a verbose JSON dump tends to get copied into logs. OWASP's Logging Cheat Sheet is a useful boundary here: sensitive values do not become safe just because the line is called a diagnostic. When I am unsure whether a field is operationally necessary, I leave it out; your mileage may vary if an internal, authenticated endpoint has a documented debugging contract.
The probe should exercise the smallest useful path. Checking only that a TCP port is open catches a dead process, but it misses an event loop stuck on synchronous work. Checking a full transaction catches more, but it can create writes, consume quota, and fail because of a dependency that is intentionally degraded. Name the check after the contract it verifies, and record the trade-off in the runbook.
How can probes, metrics, and heartbeats cover different failure modes?
Use each signal where it has authority:
| Signal | Good at detecting | It cannot establish | Placement |
|---|---|---|---|
| External HTTP probe | DNS, TLS, routing, process reachability, status-code failures | queue starvation or a wrong-but-healthy response | outside the cluster, preferably from two independent regions such as EU and US |
| Metrics scrape | error rate, latency, queue depth, event-loop lag, last-success age | a service that is unreachable from the scraper's network | inside the private network |
| Heartbeat | a scheduled or queued job that never completes | correctness of every item in the batch | emitted by the job after durable success |
That separation changes the alert policy. A probe can require three consecutive misses; a queue alert should use a sustained age or depth threshold; a heartbeat should expire after the expected schedule plus p99 runtime and a margin. One threshold cannot fit all three.
For a Node.js API, expose a metric for request failures and another for latency buckets. Avoid putting account IDs or full URLs in labels: cardinality grows faster than the incident budget. If the process is alive but its event loop is blocked, event-loop lag and request latency will move before the liveness route necessarily fails. That is a useful early warning, not proof that a customer request failed.
For a worker, send the heartbeat only after the work is durable. A ping at startup proves the scheduler launched, not that the invoice was written or the message acknowledged. Make the work idempotent so a retry can safely send a second heartbeat; duplicate delivery is still a separate problem that needs its own metric and alert.
A small external checker with explicit state
The checker below has one job: turn a sequence of observations into one open incident and one recovery. It uses standard HTTP, so the same binary can run in a small VM, a second cluster, or a CI runner. The >= guard on recovery prevents repeated messages while the service remains down.
package main
import (
"context"
"fmt"
"log"
"net/http"
"os"
"time"
)
const (
interval = 30 * time.Second
requestTimeout = 5 * time.Second
missesToOpen = 3
)
func probe(ctx context.Context, client *http.Client, target string) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
if err != nil {
return err
}
req.Header.Set("User-Agent", "health-probe")
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("unexpected status %d", resp.StatusCode)
}
return nil
}
func main() {
target := os.Getenv("HEALTH_URL")
if target == "" {
log.Fatal("HEALTH_URL is required")
}
client := &http.Client{Timeout: requestTimeout}
misses := 0
open := false
for {
ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
err := probe(ctx, client, target)
cancel()
if err != nil {
misses++
if misses >= missesToOpen && !open {
log.Printf("ALERT down target=%s error=%v", target, err)
open = true
}
} else {
if open {
log.Printf("RECOVERY target=%s", target)
}
misses = 0
open = false
}
time.Sleep(interval)
}
}
The code is intentionally not a paging integration. Send the state transition to the delivery system your team already owns, and attach a runbook URL. Keep the transition state durable if the checker can restart; otherwise a restart during an outage may produce a second opening notification. The monitor itself deserves a heartbeat or a process-level check, too.
What does a useful fallback and verification plan look like?
Two regions reduce dependence on one network path, but they add coordination. Requiring both regions to fail before paging avoids a page caused by one provider's route; requiring either region to fail detects a regional customer impact faster. Choose the rule from your service's traffic policy and write it down. The right answer is not universal.
Test failure deliberately. Point one probe at a closed port, return a controlled 503, and pause a worker after it claims a job but before it records completion. Verify the alert, the deduplication, the recovery message, and the on-call phone. Measure observed detection time instead of trusting the configured interval; connection timeout, retries, and notification latency all add to it.
Respect DO_NOT_TRACK in any CLI that emits telemetry. The convention gives operators a clear opt-out signal. Do not make an opt-out change the correctness of the check itself: delivery and diagnostic telemetry are separate concerns.
There is a catch. A two-region setup and a second delivery channel cost attention, and quorum logic can be excessive for a one-person service with no customer-facing SLA. Stick with one external probe plus a job heartbeat when the consequence of a missed alert is low. Add metrics and independent paths when queue age, data loss, or contractual uptime makes a silent failure expensive. Free or low-cost hosted checks can be a sensible starting point, but a shared monitor should not be your only escape route during an incident affecting that monitor.
Rollback for monitoring is a controlled reduction in noise: disable the new notification route, keep the raw signal, and leave an owner and expiry on the silence. If a check fires twice without a real failure, inspect its contract and vantage point before lowering every threshold. Removing evidence makes the next postmortem harder.
References
- OWASP Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
- Console Do Not Track convention: https://consoledonottrack.com/
- RFC 9110, HTTP Semantics (503 Service Unavailable): https://www.rfc-editor.org/rfc/rfc9110.html
Top comments (0)