Short answer: give an Express.js service separate, cheap liveness and dependency-aware readiness endpoints, poll readiness from outside the Docker failure domain, and page only on sustained user-visible failure. A rollback should depend on deployment-tagged evidence and a searchable run_id, not on one red health check.
At 03:07, the page says Nightly pipeline unavailable. The container is running, three readiness polls have failed, and the broad 5xx chart started rising ten minutes earlier. None of that yet answers the operational question: should the on-call remove one instance, retry the normalize stage, or roll back the Node.js deployment?
What page fired?
That is the useful starting point. The alert must carry the affected job, the observed failure and duration, the current deployment identifier, and a structured-log search scoped to the failing pipeline run. If it only says service unhealthy, the monitoring system has detected anxiety rather than supplied a decision.
How should an Express production health check endpoint separate readiness from rollback?
No, not by itself. Liveness answers whether the process can still execute its event loop and serve a minimal request. Readiness answers whether this instance should accept new work. Rollback asks a harder question: can the previous release safely read the data and tolerate the side effects produced by the current release? A probe can support that decision, but it cannot prove data compatibility.
Treat those as three separate contracts. A failed readiness check should first stop new work from reaching the affected instance while it remains available for diagnosis. A failed liveness check may justify replacing a process that cannot make progress. A rollback needs additional evidence: a deployment-correlated failure, a known compatibility window, and confirmation that the prior release understands every schema and pipeline state the candidate may have written.
This distinction matters most during a canary. Start the previous release, create a test pipeline run, and record its deployment identifier. Send one reversible unit of work to the candidate. In the test environment, make a required dependency unavailable and verify that the candidate becomes unready while remaining live, receives no new work, and leaves records the previous release can still read. Then route work back. Search by the original run_id and confirm that the older process can continue or reject the unit without corrupting it. If it cannot understand a new stage value, use a forward compatibility change rather than treating a red tile as permission for a blind rollback.
The catch is that dependency-aware readiness is not suitable when the service can still provide useful degraded behavior without that dependency. Keep the instance ready in that case, expose the affected operation as a separate signal, and alert on its user-visible outcome. Use dependency gating when accepting work would merely queue doomed requests or create state that the rollback target cannot consume.
No guesswork.
The integration boundary for readiness dependencies
The Express.js application owns the meaning of both endpoint contracts; Docker or an orchestrator merely consumes them. /health/live can return a small success response containing a status and deployment identifier when the process is responsive. It should avoid remote calls, because coupling liveness to a database or log store turns one dependency interruption into a fleet-wide restart trigger.
/health/ready may evaluate only the dependencies required to accept the next unit of work. Give every check a tight time budget and enforce an overall deadline. A compact response can contain an overall state and named checks such as pipeline_store and log_sink, but it should never expose credentials, internal hostnames, stack traces, or raw dependency errors. Return a success status when the instance may accept work and a non-2xx status when it may not. These route names are examples of a local contract, not a standard imposed on every Express.js service.
Do not turn readiness into a nightly integration suite. Searching an entire data set, running an expensive logs query, or waiting through a dependency's default timeout makes the probe itself a source of load during an impairment. Deeper validation belongs in a scheduled synthetic check. Readiness is admission control.
The deployment review should record the contract in a small decision table:
| Observation | Immediate action | Evidence still needed |
|---|---|---|
| Liveness fails | Replace the stuck process | Whether the release caused the failure |
| Readiness fails | Stop assigning new work to that instance | Dependency scope and deployment correlation |
| Pipeline stage fails | Pause or retry that run under its policy | Whether written state remains rollback-compatible |
| Sustained 5xx increase | Inspect the affected operation and release | Route class, status class, and structured logs |
The application team owns endpoint semantics. The runtime team owns reachability of the published Docker service. The pipeline team owns completion windows and retry policy. One page may combine their evidence, but it still needs one named responder who can stop assignment and evaluate rollback; three teams waiting for another team to interpret unhealthy is not an escalation policy.
How does a test connect production 5xx errors to searchable logs?
The signal that should have fired before the generic readiness page is often the pipeline-stage outcome. In the hypothetical 03:07 case, the useful event carries run_id=nightly-2026-08-18, stage=normalize, deployment_id=api-7f3c, outcome=failed, and a bounded time window. Those are example values, not production measurements. They make the investigation repeatable.
Emit one structured event when a stage starts and another when it finishes. Useful fields include timestamp, severity, service, environment, deployment identifier, run identifier, stage, outcome, duration, and an error class when applicable. An Express error boundary should emit one canonical event for an unhandled request failure and increment the corresponding 5xx metric. Logging the same exception in the route, middleware, and worker inflates apparent severity without adding evidence.
Metrics and logs serve different cardinality budgets. Give the 5xx counter a stable name and bounded labels such as route class and status class. The Prometheus naming guidance recommends one unit per metric, base units, and names whose sum or average remains meaningful; it also notes that every unique label combination creates a new time series. A run_id, raw URL, request ID, or error message therefore belongs in searchable logs, not in metric labels.
Search design comes before the incident. Verify a query that selects the production service, one run_id, non-success outcomes, and a narrow time range, then confirm that it finds both the stage event and any related request failure. I don't trust a dashboard link until the underlying field query works from a fresh session — saved views can conceal stale filters, missing access, or a retention mismatch.
A custom log destination also has a failure policy. The Logback appender documentation is Java-focused, but its boundary is broadly useful: an appender is responsible for delivering logging events to a destination, and append operations are synchronized by default unless an implementation deliberately changes that behavior. For a Node.js pipeline, the equivalent engineering question is still concrete: can log delivery delay request handling, and what happens to diagnostic events under backpressure? The answer depends on the chosen logger and transport, so verify it in that implementation's documentation and load tests rather than assuming the word "async" settles the matter.
One assumption may change during the rehearsal. The team may expect readiness to fail first, then observe that a stage-failure event identifies the exact run while readiness correctly stays healthy because unrelated traffic remains serviceable. Don't expand readiness to cover the entire pipeline. Page on the stage outcome, keep admission semantics narrow, and preserve both signals for the rollback decision.
How can an external alert monitor a Node.js readiness failure?
Run at least one poll outside the Docker host or cluster failure domain. An in-process self-check shows that a process can call itself; it does not exercise the caller's DNS path, routing, TLS termination, or published port. The monitor below is intentionally written in Go even though the target service is Express.js. Runtime independence is part of the test, and the program emits structured events that a separate alert router can consume.
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"time"
)
type probeEvent struct {
Time string `json:"time"`
Target string `json:"target"`
StatusCode int `json:"status_code,omitempty"`
Consecutive int `json:"consecutive_failures"`
Outcome string `json:"outcome"`
}
func probe(client *http.Client, target string) (int, error) {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
if err != nil {
return 0, err
}
resp, err := client.Do(req)
if err != nil {
return 0, err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return resp.StatusCode, fmt.Errorf("readiness returned %d", resp.StatusCode)
}
return resp.StatusCode, nil
}
func emit(event probeEvent) {
_ = json.NewEncoder(os.Stdout).Encode(event)
}
func main() {
target := os.Getenv("READINESS_URL")
if target == "" {
fmt.Fprintln(os.Stderr, "READINESS_URL is required")
os.Exit(2)
}
client := &http.Client{Timeout: 3 * time.Second}
ticker := time.NewTicker(30 * time.Second)
defer ticker.Stop()
consecutive := 0
for {
status, err := probe(client, target)
if err == nil {
consecutive = 0
emit(probeEvent{
Time: time.Now().UTC().Format(time.RFC3339), Target: target,
StatusCode: status, Outcome: "ready",
})
} else {
consecutive++
outcome := "probe_failed"
if consecutive >= 3 {
outcome = "alert"
}
emit(probeEvent{
Time: time.Now().UTC().Format(time.RFC3339), Target: target,
StatusCode: status, Consecutive: consecutive, Outcome: outcome,
})
}
<-ticker.C
}
}
Set READINESS_URL to the externally reachable Express.js readiness endpoint. The surrounding monitoring system should route outcome=alert to the page channel and attach the run_id search when a pipeline event provides one. The monitor waits for three consecutive failed polls in this example. Three is a policy input, not a universal constant; I'm not sure it is appropriate for a particular nightly pipeline until its completion window, normal latency distribution, and tolerated detection delay are measured.
Test the monitor separately from the application. A controlled readiness failure should remove the canary from new assignments while liveness remains successful. A controlled process stall should affect liveness. A stage failure that leaves unrelated API work available should page through the pipeline signal without changing readiness. Finally, verify that every event contains the deployment identifier needed to compare the candidate with the rollback target.
The attention cost of probe thresholds
Walk backward from the page one last time. The on-call needs a failing run and deployment, the alert needs a sustained signal, the signal needs bounded metric labels plus high-cardinality log fields, and those fields must be emitted before the endpoint can summarize anything useful. The instrumentation change is therefore not “add /health/ready.” It is to connect admission state, request failures, pipeline outcomes, and deployment identity without collapsing them into one Boolean.
Polling every 30 seconds and alerting after three failures yields an example detection path, not a service-level objective. A threshold that is too sensitive wakes someone for transient network loss and trains responders to distrust the channel. A threshold that is too slow can let a bad candidate accept more pipeline work, shrinking the safe rollback window as it writes additional state. Measure normal probe latency, pipeline completion behavior, and the time needed to stop assignment; then choose the interval and consecutive-failure count from the action deadline.
There is a real false-positive cost. Each meaningless page consumes attention, but suppressing all probe noise by stretching the window also delays the one decision that protects rollback safety. Prefer multi-signal confirmation when the action is destructive: an external readiness failure can remove an instance quickly, while a rollback page should require deployment correlation or a matching user-visible failure. Keep the two actions separate, rehearse them during a canary, and make the alert say which one it is asking for.
They should not. A probe failure can drive fast, reversible admission control, while a rollback request needs compatibility evidence and deployment correlation. Combining both actions in one alert either slows removal of an unready instance or makes rollback too easy to trigger; separate pages can share the same evidence without pretending they carry the same risk.
Top comments (0)