Short answer: treat startup, readiness, and liveness as three different control signals, and keep notification-delivery outcomes out of all three. A Node.js pod should become ready only when it can accept its assigned work, remain live while its event loop can still make progress, and use a startup probe to protect deliberate initialization. Delivery failures belong in durable attempt records, bounded metrics, and correlated logs. That division detects real customer harm without converting a provider outage into a restart storm.
This is an architecture decision record for a small fintech SaaS notification service that emits payment receipts, transfer alerts, and reconciliation notices. The Node.js app runs in a Docker image under Kubernetes, but the decision is not container-specific: signal quality versus noise means every health monitoring signal must name one failure boundary and one operator action. A failing probe changes traffic or process state, so it is an actuator rather than a dashboard ornament. Metrics and logs observe delivery; probes control the workload.
What must remain true?
The first invariant is auditability. Every accepted notification has a stable notification ID, every delivery attempt has its own attempt ID, and a retry appends an attempt rather than rewriting history. An idempotency key protects the acceptance boundary; it does not prove that an email or message was delivered exactly once. The service may repeat an external request after an ambiguous timeout, so its ledger must preserve what it knows, what it attempted, and what remains uncertain.
The second invariant is that probes describe the local process. Startup answers whether initialization has completed. Readiness answers whether this replica should receive new work now. Liveness answers whether restarting this replica is likely to restore progress. None of those questions is equivalent to "did a downstream notification provider recently reject a request?"
That distinction is decisive. Consider ten replicas consuming durable notification jobs when their shared delivery provider begins timing out. If provider reachability is embedded in liveness, Kubernetes can restart all ten Node.js processes even though each app is scheduling timers, updating attempt state, and responding normally; the replacement Docker containers then meet the same timeout, while the churn obscures whether any attempt received a remote acknowledgement. A readiness dependency on that same provider can instead withdraw all ten replicas, even though they could safely classify attempts and apply bounded backoff. Keep the process alive, stop or throttle intake through readiness only when a replica cannot safely retain work, and record provider outcomes in delivery metrics and logs. The trade-off is explicit: a narrow probe does not claim end-to-end health, so a separate delivery alert is mandatory.
The third invariant is bounded disclosure. Logs for a financial workflow should carry identifiers, state transitions, classifications, and timing, but not message bodies, account details, access tokens, or raw responses that may contain personal data. Retention and deletion policy must account for the right to erasure described by GDPR Article 17; an audit trail is not permission to retain every payload forever.
Failure boundaries follow:
- Process failure: the Node.js event loop cannot make useful progress. Liveness may restart the replica.
- Initialization failure: configuration validation or cache warmup has not reached the declared startup state. Startup remains false.
- Admission failure: the replica cannot safely accept work because its local queue or required internal state is unavailable. Readiness becomes false.
- Delivery failure: a provider rejects, throttles, or ambiguously times out an attempt. The attempt ledger, metrics, and logs report it; probes remain independent.
- Policy outcome: a disabled notification route is intentionally suppressed. A feature toggle changes behavior, but suppression is not infrastructure failure.
How should Node.js readiness, liveness, and startup work on Kubernetes?
A signal is valuable only if an operator can infer the next action without reconstructing its meaning from several dashboards.
| Signal | Question answered | Appropriate reaction | Main noise risk |
|---|---|---|---|
| Startup probe | Has bounded initialization completed? | Delay liveness and readiness evaluation | Treating an unbounded dependency as initialization |
| Readiness probe | Can this replica safely accept assigned work? | Remove it from new traffic | Including every remote dependency |
| Liveness probe | Is a local restart likely to restore progress? | Restart this replica | Restarting healthy processes during shared outages |
| Attempt metric | Which result occurred, by stable class? | Alert on sustained impact | IDs or raw errors used as labels |
| Structured log | What happened to this attempt? | Investigate and reconcile | Sensitive payloads or inconsistent fields |
| Durable attempt row | What is the authoritative history? | Retry, reconcile, or prove disposition | Mutable rows that erase earlier states |
For a simple service, use three local endpoints: /startup, /ready, and /live. That is the complete route set for probe semantics, not a catalogue of operational APIs. Keep responses cheap and deterministic. A handler should not send a test notification, query an external provider, or write to the database merely to prove that it can; those actions introduce side effects and make probe frequency part of production load.
A useful readiness rule is narrower than "all dependencies are green." If an HTTP acceptance path durably commits a notification before acknowledging it, readiness should fail when that commit path cannot preserve the contract. If a worker already owns durable work and a provider is temporarily unavailable, withdrawing the worker may add no safety; bounded backoff and an explicit retry state preserve more information. Acceptance and worker roles should therefore expose readiness independently when ownership differs.
No magic threshold exists. Configure probe periods and failure thresholds from measured initialization and recovery distributions, then test them under CPU throttling, graceful termination, and dependency latency. A copied threshold is an undocumented availability policy.
Measure first.
The critical path as an executable decision
The application can expose the endpoints in Node.js while keeping policy small enough to test as a pure state machine. This Go model is transport-neutral: it documents the contract that the handlers must implement, and prevents a later refactor from quietly making provider health a liveness condition.
package health
import "time"
type State struct {
BootComplete bool
EventLoopProgress time.Time
AcceptanceWritable bool
LocalQueueSaturated bool
}
type Decision struct {
StartupReady bool
Live bool
Ready bool
}
func Evaluate(now time.Time, s State) Decision {
const progressBudget = 10 * time.Second
startup := s.BootComplete
live := now.Sub(s.EventLoopProgress) <= progressBudget
ready := startup && live && s.AcceptanceWritable && !s.LocalQueueSaturated
return Decision{StartupReady: startup, Live: live, Ready: ready}
}
Notice what is absent: provider status. Provider results enter a separate, finite outcome classifier, for example delivered, retryable, permanent_rejection, suppressed, and unknown. The raw provider code belongs in the attempt record or controlled log field, not in a metric label. This produces bounded metric series while retaining evidence for reconciliation.
Exactly-once delivery is the wrong promise at an external boundary where a timeout can hide whether the remote side accepted a request. The achievable discipline is exactly-once accounting: one accepted identity, append-only attempts, deterministic transitions, and idempotent handling wherever the downstream contract supports it. Unknown stays unknown until reconciliation resolves it. Do not relabel ambiguity as failure merely to make a graph look complete.
Each attempt should emit one terminal metric update and one structured terminal log event. Fields should include notification ID, attempt ID, channel, outcome class, duration, and a correlation value carried from acceptance. Avoid logging the message body. A separate accepted counter permits reconciliation over a fixed interval: accepted work should eventually partition into delivered, permanently rejected, suppressed by policy, or still pending. The pending set needs an age distribution because a small, old backlog can be more serious than a larger, fresh one.
Alert on customer impact rather than isolated events. One permanent rejection is usually actionable for one recipient but not an infrastructure incident; a sustained rise in retryable or unknown outcomes across enough traffic may be. Low-volume routes require longer evaluation windows or direct age checks because ratios become unstable when the denominator is tiny.
Signal quality wins.
How should this be tested and deployed?
Test probes as contracts, not endpoint snapshots. Hold initialization incomplete and verify that liveness does not preempt the startup allowance. Make the durable acceptance boundary unwritable and verify readiness withdrawal. Stop the progress marker and verify liveness failure after its budget. A provider-failure test must prove that liveness remains true while attempt outcomes move to retryable or unknown.
Then test termination. On shutdown, readiness should become false before the process stops accepting work; owned attempts must either finish inside the termination budget or return to a durable retryable state. Exercise idempotency with duplicate acceptance requests and repeated worker delivery, including the ambiguous case where a simulated provider accepts a request but the local caller receives a timeout.
Deployment changes deserve the same care. A feature toggle can separate deployment from activation, but Fowler's discussion also shows that toggles introduce decision logic and require deliberate management. Record the evaluated policy version with each suppressed or routed attempt. Remove obsolete toggles, because an unnamed branch in delivery policy becomes an audit question later.
Run load tests with realistic queue depth and CPU limits. Probe handlers compete for the same event loop as application work, so a sound design can still appear dead if budgets ignore scheduling delay. Compare transitions with attempt outcomes: readiness flapping without a change in safe acceptance capacity indicates a noisy condition; worsening delivery outcomes with stable local probes indicates that the boundaries remain separate.
Metrics should use bounded labels, logs should use a documented schema, and the attempt ledger should remain the source for financial reconciliation. Do not ask logs to serve as the ledger. Sampling may be acceptable for diagnostic success logs, but it is incompatible with any claim that logs alone enumerate every accepted notification.
Rejected option, limitations, and a valid boundary
We reject deep probes that synchronously check the database, queue, and notification provider on every liveness request. They combine unrelated ownership domains, create correlated failure, and make restart behavior depend on remote latency. They also confuse evidence: a red liveness result no longer says whether the process is stuck or a provider is unavailable. The limitation of the chosen local-probe design is equally real: it cannot establish that a customer received a notification, detect a semantically malformed message, or validate the full path from queue to provider. Teams that operate a single process without an orchestrator may gain little from three HTTP probe routes; a process supervisor plus external synthetic monitoring can express the same local-versus-end-to-end boundary with less machinery.
A deep synthetic check still has a valid use outside the pod restart loop. Run a controlled canary notification through a non-customer route, at a frequency and retention policy appropriate to its data, then measure its end-to-end state as an external service-level signal. Its failure should page according to sustained impact and should never directly kill application replicas. Synthetic evidence complements real attempt outcomes; it does not replace reconciliation.
This decision leaves one deliberate tension. Readiness can protect acceptance correctness, yet overly broad conditions can remove all capacity at once. Resolve it at the ownership boundary: fail readiness only when this replica cannot honor the contract for newly assigned work, use delivery telemetry for remote outcomes, and preserve every ambiguous attempt for later reconciliation. The result is quieter because each signal has less to say, and more trustworthy because what it says has a precise operational meaning.
Top comments (0)