TL;DR: An Express health check endpoint cannot prove that a scheduled logistics import is producing results merely because the process is live or ready. Keep liveness limited to whether the process can serve, make readiness represent admission control, and detect import silence from durable run records plus a metric whose deadline is derived from the schedule. Page only when the deadline is crossed and the evidence is complete; log every state transition so an operator can reconstruct why the alert fired.
The central trade-off is signal quality versus noise. A single /health response is attractive because it is easy to wire into an Express application, but it collapses several independent claims into one bit: the Node.js process is alive, dependencies are usable, the scheduler is advancing, and an import has committed a result. Those claims fail on different clocks. Combining them produces either a permissive check that misses stalled work or a sensitive check that restarts healthy servers during a downstream interruption.
For a scheduled import, success has a stricter meaning than “the callback ran.” A defensible success record says which schedule occurrence ran, which source window it covered, which input identity it observed, and which result was committed. That is an exactly-once mindset applied to evidence: retries may execute more than once, but the audit trail must let reconciliation distinguish one logical occurrence from several attempts.
Should an Express health check endpoint report ready, live, or import progress?
Suppose carrier manifests are due every 15 minutes. The web process may continue answering requests while the scheduler loses leadership, a queue stops delivering, credentials reject new reads, or a worker repeatedly parses the same poison input. Returning 200 from /live is correct in several of those states: killing the HTTP process does not necessarily restore progress, and restart loops can erase useful context.
The inverse mistake is equally noisy. If /live synchronously checks a database, object store, queue, and every carrier connection, one transient dependency failure changes process health into dependency consensus. The check now couples restart policy to remote systems with unrelated recovery behavior. Slow checks also consume the same resources needed for recovery.
Use three contracts instead:
| Signal | Question it answers | Typical evidence | Consumer action |
|---|---|---|---|
| Liveness | Can this process answer at all? | Local event-loop path and internal fatal state | Replace a stuck process |
| Readiness | Should this instance receive new work? | Local admission state and required serving dependencies | Remove it from work assignment |
| Import progress | Did the expected schedule occurrence produce an accepted result? | Durable occurrence, attempt, and commit records | Investigate or page after a deadline |
/health can remain a human-facing summary, but automation should consume the narrower contracts. Avoid making that summary the source of truth. In an Express/Node.js service, /live and /ready should be cheap handlers over already-maintained state; the same separation applies even though the reference implementation below is Go, because the endpoint contract matters more than framework syntax.
Derive the deadline from the work contract
An “import stopped” alert needs a deadline, not a generic error counter. Define an expected occurrence from the configured schedule, then add the allowed arrival delay and the normal processing allowance. For an illustrative 15-minute schedule with 5 minutes of arrival tolerance and 10 minutes of processing allowance, the occurrence becomes late 30 minutes after its nominal boundary. These values are policy choices, not universal constants; historical completion distributions and the business reconciliation window should determine them.
The monitor should compare the current time with the latest committed logical occurrence, not the latest attempt and not the latest log line. An attempt that starts, downloads zero bytes because the upstream file is not yet present, and exits cleanly is operationally different from a committed zero-result import. Decide explicitly whether zero results are valid for each source and schedule. Otherwise a well-behaved empty period looks indistinguishable from lost data.
I would model the evidence with four identifiers: source, occurrence_id, attempt_id, and result_id. occurrence_id is stable across retries. attempt_id changes on every execution. result_id identifies the committed output and is protected by an idempotency constraint. The monitor advances only when the commit transaction records the result and its occurrence together; this prevents a success metric from getting ahead of durable state.
This is the important correction. Counting executions feels sufficient at first, but it answers “did code run?” rather than “is the ledger of expected imports complete?” Reconciliation operates on the latter.
Implement narrow probes and a durable progress check
The following Go example makes the boundary concrete. It deliberately keeps remote calls out of probe handlers and evaluates schedule progress from a repository. A production repository should read the committed occurrence record written by the import transaction.
package health
import (
"context"
"encoding/json"
"net/http"
"sync/atomic"
"time"
)
type Progress struct {
Source string `json:"source"`
OccurrenceID string `json:"occurrence_id"`
CommittedAt time.Time `json:"committed_at"`
NextDeadline time.Time `json:"next_deadline"`
}
type ProgressRepository interface {
LatestCommitted(ctx context.Context, source string) (Progress, error)
}
type Handler struct {
ready atomic.Bool
repo ProgressRepository
now func() time.Time
}
func (h *Handler) Live(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(map[string]string{"status": "alive"})
}
func (h *Handler) Ready(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
if !h.ready.Load() {
w.WriteHeader(http.StatusServiceUnavailable)
_ = json.NewEncoder(w).Encode(map[string]string{"status": "not_ready"})
return
}
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(map[string]string{"status": "ready"})
}
func (h *Handler) ImportProgress(w http.ResponseWriter, r *http.Request) {
p, err := h.repo.LatestCommitted(r.Context(), "carrier_manifest")
w.Header().Set("Content-Type", "application/json")
if err != nil {
w.WriteHeader(http.StatusServiceUnavailable)
_ = json.NewEncoder(w).Encode(map[string]string{"status": "unknown"})
return
}
if h.now().After(p.NextDeadline) {
w.WriteHeader(http.StatusServiceUnavailable)
_ = json.NewEncoder(w).Encode(map[string]any{
"status": "late", "occurrence_id": p.OccurrenceID,
})
return
}
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(map[string]any{
"status": "on_time", "occurrence_id": p.OccurrenceID,
})
}
Do not expose source credentials, object keys containing customer data, raw payloads, or stack traces through these responses. Compliance and data-retention limits belong in the design: probe payloads should contain the minimum operational identifiers, while access-controlled audit storage holds the evidence needed for investigation. The retention period must follow the organization's applicable legal and contractual requirements rather than an arbitrary observability default.
The unknown state deserves separate treatment from late. If the monitor cannot read the evidence repository, it cannot honestly declare that the import failed. Alert routing may still escalate prolonged uncertainty, but the message should say that verification is unavailable. Precision in state names prevents responders from chasing the wrong subsystem.
Make metrics and logs corroborate the same occurrence
Metrics are useful because they aggregate recurring measurements and support time-based evaluation. Export a last-committed timestamp per bounded source, an attempt counter by outcome, and a completion-duration distribution. Do not attach occurrence_id, file name, shipment ID, or attempt_id as metric attributes; those values grow without a stable bound and belong in logs or traces. The OpenTelemetry metrics model distinguishes metric data through names, resource attributes, scope, and data-point attributes, so attribute design is part of the signal contract rather than decoration.
A compact set might be:
-
import.last_committed_timewith asourceattribute from a controlled registry; -
import.attemptswithsourceand a small outcome set such ascommitted,empty_valid,retryable, andrejected; -
import.durationwithsource, measured around the logical processing attempt.
One alert should evaluate lateness from last_committed_time; another, lower-urgency signal can identify repeated rejected attempts before the deadline. Resist paging on every retryable error. Retries are expected control flow until either the deadline is threatened or the retry budget is exhausted.
Logs carry the audit linkage. Emit structured records at occurrence discovery, attempt start, result commit, rejection, and terminal deadline breach. Each record should include the stable occurrence identifier and changing attempt identifier. Severity must reflect operational meaning: RFC 5424 defines ordered severity levels from Emergency through Debug, but a level alone is not an incident policy. A retryable attempt can be informational or warning-level evidence; a deadline breach becomes an error because the promised result is now missing. Keep the original event timestamp distinct from ingestion time so delayed delivery does not rewrite history.
No single signal gets to invent success. The transaction establishes it; the metric summarizes it; the log explains it.
Control noise with explicit state transitions
Alerting directly on now - last_success creates edge cases around startup, paused schedules, maintenance, and sources that have never completed. Represent those conditions. A small state machine is easier to audit than several overlapping threshold rules: on_time moves to at_risk, then to late; a verified commit returns it to on_time; an approved pause suppresses the deadline without forging a success.
Require evidence before notification. The monitor needs the expected occurrence, its deadline, the latest committed occurrence, and the schedule's pause status. Deduplicate notifications on (source, occurrence_id, state), and record the notification decision with its policy version. This does not make message delivery magically exactly once, but it makes repeated delivery idempotent and explainable.
Test the awkward boundaries with a fake clock: one instant before the deadline, exactly at it, and one instant after it. Also test a retry that commits after an earlier attempt failed, a legitimate empty import, a repository read failure, an approved schedule pause, and a process restart between commit and metric export. The invariant is simple: durable commit evidence wins over volatile telemetry.
Roll out in shadow mode first. For one reconciliation cycle, calculate states and write audit records without paging; compare expected occurrences with committed results, classify mismatches, and adjust source-specific tolerances. Then enable notifications for one source, preserve the policy version in every decision record, and expand only after the alert stream agrees with reconciliation. This compact rollout costs time, but it is cheaper than training operators to ignore an alarm whose definition keeps changing.
Top comments (0)