Short answer: for a small Node.js SaaS, use one shallow health endpoint probed from both the EU and US, one independently timed receipt for each cron job, and one delivery-outcome signal labeled with the deployed version; page only on sustained user-impacting failure, and keep every check compatible with the previous release so a rollback remains boring.
This is the least complex setup I would trust for a logistics notification service. An all-green process check cannot tell me that delivery-failure alerts stopped reaching operators, while a dashboard full of application counters cannot tell me that a scheduler never started. I distrust both when nobody can answer the first incident question: what page fired?
I've been woken by alerts that meant nothing and missed the one that mattered. At 3 a.m., “the service is unhealthy” is not a diagnosis; “EU delivery notifications have produced no terminal outcome for 12 minutes after release 2026.08.15-3” gives the responder a failure boundary and a safe first move. The invariant is simple — observe execution from outside the code path being observed, and attach enough release context to reverse a change without guessing.
How can Node.js uptime monitoring separate EU/US health endpoints from missed cron runs?
Check three different promises because they fail independently.
The first promise is reachability: can a client in each served region resolve the hostname, negotiate a connection, and receive a timely success response from /healthz? Keep that endpoint shallow. It should prove that this instance can accept work, but it should not synchronously traverse every dependency or send a real notification. A deep dependency chain makes a local probe inherit unrelated latency and turns one damaged subsystem into a fleet-wide restart signal.
The second promise is scheduled execution. A cron process reporting that it is alive proves very little; the useful evidence is a completion receipt emitted after a bounded unit of work. An external timer should compare the last successful receipt with the job's expected interval plus a stated grace period. That timer must live outside the scheduler. Otherwise, the same dead scheduler is responsible for announcing its own absence, which is a neat design until it disappears.
The third promise is the business outcome. For this notification service, count delivery attempts by terminal result such as delivered, provider_rejected, or expired, then retain the deployment version and operating region as low-cardinality dimensions. The result vocabulary belongs to the application contract, not to whichever transport currently carries a message. Prometheus recommends metric names with a single unit and a base-unit suffix where applicable; following that convention keeps queries intelligible during a rollback instead of leaving responders to decode names under pressure.
These promises should not collapse into one boolean. Reachability can stay green while a cron job misses every run. The cron can complete while every notification is rejected. A delivery provider can affect the EU path while US probes still pass. One “uptime” percentage hides precisely the distinction needed to choose between rollback, failover, and waiting for a late job.
The alert-to-rollback integration boundary
Consider a bounded deployment scenario, not a claimed production event. Release A accepts a queue item, sends it, and records a terminal delivery outcome. Release B changes the payload contract. The process remains up, /healthz returns 200, and the scheduler continues to acknowledge its loop, yet terminal outcomes fall silent for newly created items. A monitor that checks only HTTP reachability reports success. A monitor that checks only queue depth may page late or page on an ordinary burst. The signal that matters is the absence or failure ratio of terminal outcomes for Release B, separated by region. The incident clock now has useful marks: the deployment began, Release B first accepted work, the last terminal outcome appeared, the absence window elapsed, and one narrowly scoped page fired. That sequence lets the responder compare the failing cohort with Release A before changing anything. It also guards against a reflexive rollback when the evidence instead points to a single probe location or a late scheduler.
Now add rollback. If Release B writes state that Release A cannot read, a responder cannot safely reverse the deploy even after the alert identifies the release. If Release B renames the cron receipt without a compatibility window, rolling the application back may restore delivery while making the external timer report a missed run. And if the alert rule itself changes in the same atomic step as the application, the new release can erase the evidence used to judge it. This is why rollback safety is an observability property, not merely a deployment feature: the old binary, the new binary, and the monitoring plane need a period in which they understand the same health path, receipt names, and outcome schema.
I would split the rollout into observable phases. First, deploy backward-compatible readers and emit both old and new outcome fields where a schema migration requires it. Next, introduce the new metric or receipt and confirm that both versions produce interpretable signals. Then change alert evaluation. Remove the old field only after the rollback window closes. I don't need a clever dashboard for that sequence; I need an alert annotation that names the check, region, release, threshold window, and runbook action.
Short alerts win.
No pulse, no proof.
The page should identify whether it was fired by an external probe, a missing cron receipt, or an outcome-rate rule. Syslog's severity model is useful discipline here even if the implementation is metrics-based: severity expresses the significance of the event, while facility identifies the source category. Do not map every error log to a page. A single provider_rejected result is evidence for diagnosis; a sustained loss of terminal outcomes across enough traffic is a candidate symptom of user impact.
I'm not sure a universal grace period exists, because job duration, arrival rate, and delivery deadlines vary. The value should come from the service's own timing contract and observed completion distribution, then be tested by delaying a run on purpose. Your mileage may vary. What should not vary is the ownership of the clock: the external evaluator decides that a receipt is late.
Implement the versioned Go signal contract
The application can expose a small, stable probe and send structured observations through generic interfaces. The sample below omits storage and transport choices on purpose; those are adapters. It also avoids putting a remote dependency call inside /healthz.
package monitoring
import (
"context"
"encoding/json"
"net/http"
"time"
)
type Observation struct {
Name string `json:"name"`
Status string `json:"status"`
Region string `json:"region"`
Release string `json:"release"`
Timestamp time.Time `json:"timestamp"`
Labels map[string]string `json:"labels,omitempty"`
}
type Sink interface {
Record(context.Context, Observation) error
}
type Service struct {
Region string
Release string
Sink Sink
}
func (s Service) Healthz(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(map[string]string{
"status": "ok",
"release": s.Release,
})
}
func (s Service) RecordCronCompletion(ctx context.Context, job string, started time.Time) error {
return s.Sink.Record(ctx, Observation{
Name: "cron_completion",
Status: "success",
Region: s.Region,
Release: s.Release,
Timestamp: time.Now().UTC(),
Labels: map[string]string{
"job": job,
"duration": time.Since(started).String(),
},
})
}
func (s Service) RecordDeliveryOutcome(ctx context.Context, result string) error {
return s.Sink.Record(ctx, Observation{
Name: "delivery_outcome",
Status: result,
Region: s.Region,
Release: s.Release,
Timestamp: time.Now().UTC(),
})
}
Production code should bound writes to the observation sink and define what happens when that sink is unavailable; notification delivery must not wait forever for telemetry. The key operational choice is explicit: either return the sink error to a retryable job runner, or record it locally through an independent channel while preserving the delivery result. Which choice fits depends on whether a missing audit record violates the product contract. Hiding the error is not a choice.
The external evaluator can remain tiny. For each job, store its last accepted completion time and reject receipts with unknown job names. Evaluate lateness from a monotonic service-side clock where possible, not from a timestamp supplied by an arbitrary caller. For delivery outcomes, compare like with like: same region, same release cohort, and a window large enough to avoid paging merely because no deliveries were scheduled.
A staging rollback test harness
The useful comparison is not a feature count. It is the action each signal permits when a new release goes bad.
| Check | Detects | Misses | Rollback-safe condition | Page action |
|---|---|---|---|---|
EU and US /healthz probes |
Regional reachability and response timing | Scheduler and delivery semantics | Path and response contract remain valid in both releases | Inspect region boundary before rollback |
| External cron receipt timer | A job that did not complete before its grace period | Incorrect work that still reports completion | Receipt identity survives the rollback window | Retry or rollback according to job idempotency |
| Versioned delivery outcomes | Release-specific notification failure or silence | Failures before an outcome can be emitted | Old and new versions share result semantics | Roll back the affected release cohort |
| Log-derived error alert | Diagnostic events with severity context | Silent absence and unknown denominators | Parsers accept both log schemas | Investigate; page only when tied to impact |
Synthetic probes from the EU and US are worth keeping separate if customers are served from both locations. Do not average them into a global green state. A regional page can point to routing, certificate, or edge behavior; a release-cohort page can point to application change. Those are different first moves, and the monitor should preserve that distinction.
Cost still matters for a small SaaS, but “cheap” is a constraint rather than an architecture. Count probe locations, check frequency, retained event volume, and notification channels against the operating budget. Then test exportability: checks, receipts, and alert definitions should be recoverable in a documented form. A low monthly bill does not compensate for a monitor that cannot preserve history or definitions during a migration.
Budget and capability limits
The catch is that three signal types create more operational surface than a single ping. A pre-revenue service with one region, no background jobs, and no delivery deadline may be better served by one external health probe and a human-readable log until the first real service objective exists. Do not manufacture paging complexity for traffic that cannot establish a meaningful rate.
This design is also not suitable when the cron task is inherently non-idempotent and has no reconciliation mechanism. A missed-run alert without a safe retry rule tells the responder that work is late but offers no responsible action. Fix the job contract first: attach a stable run identifier, make completion durable, and define how duplicate execution is detected.
At the other extreme, high-volume multi-region systems with formal service-level objectives need more than these three checks. They may require burn-rate alerts over multiple windows, trace correlation, regional failover automation, and a tested telemetry pipeline with its own availability target. Stick with that machinery when alert evaluation itself is a critical distributed system. The small setup described here is intentionally bounded.
For the logistics notification service, the final acceptance test is blunt: stop a scheduled run, isolate one probe region, and deploy a version that records a controlled rejected outcome in staging. Each action should fire one distinct signal with the release and region attached; restoring or rolling back should clear it without renaming the evidence. If the test produces three vague pages, the dashboards are not the problem. The contracts are.
References and further reading
- Prometheus, “Metric and label naming”: https://prometheus.io/docs/practices/naming/
- IETF RFC 5424, “The Syslog Protocol”: https://datatracker.ietf.org/doc/html/rfc5424
Top comments (0)