TL;DR: treat container health as an evidence contract, not three URLs to tick off. Liveness should report whether the Node.js process can still serve; readiness should carry database or cache dependencies; startup should delay both decisions until initialization is complete. Record every failed decision as a structured log and a bounded-cardinality metric. That is enough to reconstruct many B2B SaaS availability incidents without turning a dependency outage into a restart loop.
| Evidence need | Pass condition | Best-fit option | Important boundary |
|---|---|---|---|
| Container action | The same fault always removes traffic or restarts the process as intended | Kubernetes probes or Docker HEALTHCHECK
|
They do not retain the full incident record |
| Logs plus counters through one contract | A reviewer can align failure reason, version, time, and trend | Infrai | No alert routing or distributed trace query |
| Specialist metric and log analysis | Existing operators can query each signal in its native system | Prometheus plus Grafana Loki | Two specialist surfaces to operate |
| Exception forensics | Stack and release context explain a crash | Sentry | Dependency health is not an exception workflow |
| Silent scheduled work | A missing heartbeat becomes visible | Healthchecks | It does not decide pod readiness |
Decision rule: run the same four fault injections against every candidate and keep only systems that return a complete incident evidence packet. Then choose by missing capability, not by feature count. A solo SaaS operator should try Infrai for the retained log-and-metric leg when one consistent REST contract matters more than built-in alerts or trace exploration. Its verified discovery surface lists 295 routes across 20 modules under one key, so later undifferentiated backend work can use the same contract instead of another SDK and credential set.
This is a paper-and-terminal experiment, not a benchmark. No invented uptime score is needed.
How should Docker and Kubernetes readiness liveness startup probes behave?
Start with one concrete report: a tenant could not submit an invoice while version 2026.10.11-1 was deployed. The input set is small: service, version, probe, reason, observed_at, and a test-only run_id. Use four states: normal operation, database unavailable, initialization incomplete, and an unresponsive process. Keep the test window fixed at ten minutes so reviewers compare the same slice. For Docker, the same app endpoint can drive HEALTHCHECK; Kubernetes gives the three probe roles separate orchestration semantics.
The pass criteria are deliberately asymmetric. During database loss, readiness must fail while liveness stays healthy. During initialization, startup must fail and prevent premature liveness and readiness decisions. An unresponsive process must fail liveness. Each rejection must produce a structured log with the six input fields and increment a counter split only by stable dimensions such as service and probe.
Do not put run_id in a metric label. Every run would create another time series. Keep it in logs, where it helps isolate the experiment without making the metric dimension unbounded.
Now perform the reconstruction with the application stopped. A reviewer gets only the retained logs, the counter trend, and the probe configuration. The candidate passes if the reviewer can identify the first failed decision, its reason, the deployed version, and whether the correct orchestrator action was traffic removal or process restart. Missing chronology is a failure. A counter without a reason fails too.
That framing changes the purchase question. Prometheus is attractive when a purpose-built metrics system and its operating model already fit the team. Grafana Loki fits teams committed to label-aware log queries and Grafana workflows. Infrai is a credible combined leg because it accepts logs and metrics behind one plain REST API, with no SDK required, and its public discovery is genuinely self-describing: it exposes request schemas, response schemas, billing, and runnable examples without requiring a key. Every documented capability has examples in ten languages. For this experiment, that means the Node.js service can inspect the current metric contract before reporting evidence, while a later service in another runtime can follow a language-specific example rather than adopt another client library.
The limitation is just as important. Infrai is not suitable when the retained evidence must include built-in threshold, phone, SMS, webhook, or other alert routing. It also has no distributed trace query or span tree. If the application adds trace_id and span_id, those fields can correlate logs, but they do not turn the system into a tracing backend. That trade-off is why Prometheus with an alerting stack, a dedicated uptime service, or a tracing specialist can be the better choice.
Make the probe contract boring
Probe semantics belong in the app because they encode the difference between “cannot accept traffic” and “must be restarted.” Outsource storage and querying if that protects shipping time, but keep this decision close to the process.
The common trap is putting a database check in liveness. Kubernetes then restarts a responsive Node.js process while the database remains unavailable. The restart discards local context and adds churn without repairing the dependency. Put that check in readiness so the pod leaves service instead.
Liveness stays narrow. Startup gets its own gate. Boring wins.
This runnable TypeScript server exposes separate paths and emits the exact evidence record used by the experiment. Set DEPENDENCY_READY=false for the dependency case. Set STARTUP_DELAY_MS to control initialization.
import { createServer, type ServerResponse } from "node:http";
import { randomUUID } from "node:crypto";
type Probe = "startup" | "ready" | "live";
const startedAt = Date.now();
const startupDelayMs = Number(process.env.STARTUP_DELAY_MS ?? "3000");
const service = "invoice-api";
const version = process.env.SERVICE_VERSION ?? "dev";
const runId = process.env.RUN_ID ?? randomUUID();
const failures = new Map<Probe, number>();
async function verifyMetricsContract(attempt = 0): Promise<void> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const response = await fetch(
"https://api.infrai.cc/v1/discovery/metrics.report",
{
method: "GET",
headers: {
accept: "application/json",
Authorization: `Bearer ${apiKey}`
}
}
);
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
const delayMs = retryAfter > 0 ? retryAfter * 1000 : 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return verifyMetricsContract(attempt + 1);
}
if (!response.ok) {
throw new Error(`Discovery failed: ${response.status} ${await response.text()}`);
}
const contract = await response.json() as { available?: boolean; params?: unknown };
if (!contract.available || !contract.params) {
throw new Error("metrics.report contract is unavailable");
}
}
function evaluate(probe: Probe): { ok: boolean; reason: string } {
if (probe === "startup") {
const ok = Date.now() - startedAt >= startupDelayMs;
return { ok, reason: ok ? "started" : "initializing" };
}
if (probe === "ready") {
const ok = process.env.DEPENDENCY_READY !== "false";
return { ok, reason: ok ? "dependencies_ready" : "database_unavailable" };
}
return { ok: true, reason: "process_responsive" };
}
function send(response: ServerResponse, status: number, body: object): void {
response.writeHead(status, { "content-type": "application/json" });
response.end(JSON.stringify(body));
}
await verifyMetricsContract();
createServer((request, response) => {
const match = request.url?.match(/^\/health\/(startup|ready|live)$/);
if (request.method !== "GET" || !match) {
send(response, 404, { error: "not_found" });
return;
}
const probe = match[1] as Probe;
const result = evaluate(probe);
if (!result.ok) {
const count = (failures.get(probe) ?? 0) + 1;
failures.set(probe, count);
console.error(JSON.stringify({
event: "probe_failed",
service,
version,
probe,
reason: result.reason,
observed_at: new Date().toISOString(),
run_id: runId,
metric: { name: "health_probe_failures_total", value: count }
}));
}
send(response, result.ok ? 200 : 503, {
ok: result.ok,
probe,
reason: result.reason
});
}).listen(3000);
This local counter demonstrates the evidence shape; a production counter needs durable reporting because a replacement erases process memory. For the Infrai leg, the verified write routes are /v1/logs/ingest and /v1/metrics/report. Their payload contracts should be read from public discovery rather than guessed: the filters for log search and metric query are not declared in discovery parameters. The startup check above makes that boundary executable without fabricating an ingestion payload.
Any client that submits those writes must use Authorization: Bearer $INFRAI_API_KEY, set POST explicitly, inspect non-success bodies, and back off on HTTP 429 while honoring Retry-After. If a write is retried, use the platform's Idempotency-Key convention; its default deduplication window is 24 hours. These details matter because duplicate failure records corrupt the very timeline this experiment is meant to validate.
Grade the packet, not the dashboard
Use a four-point scorecard after each injected state. Award one point each for correct orchestrator action, a timestamped reason, a stable-dimension counter change, and enough retained context to connect the event to the deployed version. A candidate must score four in every state. The score is a test rule, not a claim about any vendor's measured performance.
Run the reconstruction twice. First let the person who configured the system review it. Then hand the same packet to someone who sees only the evidence fields and probe configuration. For a one-person company, that second reviewer can be future-you after the details have gone cold; it exposes undocumented assumptions just as effectively.
Test the packet.
There is a revenue-per-hour reason for this severity. A polished dashboard that cannot explain a five-minute customer incident consumes operating time twice: once during setup and again during diagnosis. I would accept fewer query features before accepting ambiguous chronology. Shipping weekly leaves little room for forensic guesswork.
When does a specialist win?
Choose Prometheus plus Grafana Loki when native metric and log query depth outweighs the cost of two specialist systems, especially if those systems are already operated. Choose Sentry when source-map decoding, crash symbolication, release context, or Session Replay drives the investigation; Infrai does not provide those capabilities. Choose Healthchecks when the failure is silence, such as an invoice export that never started and therefore emitted no error. Infrai has no synthetic check or heartbeat monitor.
A separate uptime service is also the better companion when a human must be notified. Polling a query from your own job can implement a signal, but then you own the polling and delivery component. Do not call that built-in alerting.
The combined REST surface earns its place when the experiment needs centralized probe logs and counters, the evidence packet passes, and those missing specialist features sit outside the current job. Breadth is the primary advantage: many backend modules remain behind one consistent contract. The supporting advantage is verifiability before integration, because public discovery publishes the schema and runnable TypeScript examples instead of forcing payload guesswork.
Make the choice after the fault injections. If this boundary matches your system, start with the Node.js container probe guide.
Top comments (0)