DEV Community

EllisVance1273
EllisVance1273

Posted on

3 Docker and Kubernetes Readiness Liveness Startup Probes for Node.js Apps

#kubernetes #docker #observability

Wire container probes to a small application-owned health endpoint, keep dependency checks out of liveness, and emit every failed check as both a structured log and a counter metric. The deciding constraint is rollback safety: a telemetry vendor change must not force an application release while a broken deployment is already being reversed.

TL;DR: use three separate signals. Liveness answers “is this process stuck?” Readiness answers “should this instance receive work?” A heartbeat from the nightly customer-support pipeline answers “did the scheduled job run?” They look related on a dashboard, but treating them as one Boolean creates destructive restart loops and hides silent jobs.

How should Docker and Kubernetes use readiness, liveness, and startup probes?

A liveness failure asks the container runtime to restart the process. That is a blunt recovery action, so the check should be blunt too. If the event loop can serve the health request, liveness should normally pass. A database or cache outage belongs in readiness: remove the instance from service without turning a shared dependency incident into a restart storm.

Startup is the temporary exception. A slow initialization path can use a startup probe to hold back liveness and readiness until boot completes. Once startup succeeds, it gets out of the way. No clever scoring model is needed.

Keep it boring.

The nightly pipeline has a different failure mode. It can fail without any container becoming unhealthy because the task never started. A dead-man's-switch service such as Healthchecks.io is a better fit for that missing heartbeat. This distinction matters for a support pipeline: “the worker is alive” does not prove that last night's ticket import finished.

Keep the emitted contract vendor-neutral: JSON logs with stable field names and monotonic counters with stable metric names. Then Prometheus, Datadog, Grafana Cloud, or an API service can sit behind the adapter without changing probe behavior. The contract stays put while the backend moves. That is useful during rollback, when the smallest possible diff wins.

The smallest working implementation

This server uses only Node's built-in modules. It exposes separate startup, liveness, readiness, and metrics endpoints; logs failed probes as one-line JSON; and counts outcomes in a Prometheus-compatible text response. Set DEPENDENCY_URL to a cheap endpoint owned by the dependency you actually need before accepting work.

import { createServer, IncomingMessage, ServerResponse } from "node:http";

type Probe = "startup" | "liveness" | "readiness";
type Outcome = "ok" | "failed";

const startedAt = Date.now();
let startupComplete = false;
const counts: Record<Probe, Record<Outcome, number>> = {
  startup: { ok: 0, failed: 0 },
  liveness: { ok: 0, failed: 0 },
  readiness: { ok: 0, failed: 0 },
};

async function verifyLogSearch(attempt = 0): Promise<unknown> {
  const apiKey = process.env.INFRAI_API_KEY;
  const baseUrl = process.env.OBSERVABILITY_BASE_URL;
  if (!apiKey || !baseUrl) {
    throw new Error("INFRAI_API_KEY and OBSERVABILITY_BASE_URL are required");
  }

  const response = await fetch(`${baseUrl}/logs/search`, {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });
  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return verifyLogSearch(attempt + 1);
  }
  if (!response.ok) {
    throw new Error(`log search failed (${response.status}): ${await response.text()}`);
  }
  return response.json() as Promise<unknown>;
}

function record(probe: Probe, outcome: Outcome, reason?: string): void {
  counts[probe][outcome] += 1;
  if (outcome === "failed") {
    process.stderr.write(`${JSON.stringify({
      level: "warn",
      event: "health_probe_failed",
      probe,
      reason,
      service: "support-nightly-pipeline",
      timestamp: new Date().toISOString(),
    })}\n`);
  }
}

function reply(res: ServerResponse, status: number, body: object): void {
  res.writeHead(status, { "content-type": "application/json" });
  res.end(JSON.stringify(body));
}

async function dependencyReady(): Promise<{ ok: boolean; reason?: string }> {
  const url = process.env.DEPENDENCY_URL;
  if (!url) return { ok: false, reason: "DEPENDENCY_URL is not configured" };

  try {
    const response = await fetch(url, {
      method: "HEAD",
      signal: AbortSignal.timeout(1_500),
    });
    return response.ok
      ? { ok: true }
      : { ok: false, reason: `dependency returned ${response.status}` };
  } catch (error) {
    return {
      ok: false,
      reason: error instanceof Error ? error.message : "dependency check failed",
    };
  }
}

function renderMetrics(): string {
  const lines = [
    "# HELP health_probe_total Application health probe outcomes.",
    "# TYPE health_probe_total counter",
  ];
  for (const probe of Object.keys(counts) as Probe[]) {
    for (const outcome of Object.keys(counts[probe]) as Outcome[]) {
      lines.push(
        `health_probe_total{probe="${probe}",outcome="${outcome}"} ${counts[probe][outcome]}`,
      );
    }
  }
  return `${lines.join("\n")}\n`;
}

async function handle(req: IncomingMessage, res: ServerResponse): Promise<void> {
  if (req.method !== "GET") {
    reply(res, 405, { ok: false, reason: "method_not_allowed" });
    return;
  }

  if (req.url === "/health/live") {
    record("liveness", "ok");
    reply(res, 200, { ok: true, uptime_ms: Date.now() - startedAt });
    return;
  }

  if (req.url === "/health/startup") {
    const outcome: Outcome = startupComplete ? "ok" : "failed";
    record("startup", outcome, startupComplete ? undefined : "initializing");
    reply(res, startupComplete ? 200 : 503, { ok: startupComplete });
    return;
  }

  if (req.url === "/health/ready") {
    const result = await dependencyReady();
    const outcome: Outcome = result.ok ? "ok" : "failed";
    record("readiness", outcome, result.reason);
    reply(res, result.ok ? 200 : 503, result);
    return;
  }

  if (req.url === "/metrics") {
    res.writeHead(200, { "content-type": "text/plain; version=0.0.4" });
    res.end(renderMetrics());
    return;
  }

  reply(res, 404, { ok: false, reason: "not_found" });
}

const server = createServer((req, res) => {
  void handle(req, res).catch((error: unknown) => {
    process.stderr.write(`${JSON.stringify({
      level: "error",
      event: "health_handler_failed",
      message: error instanceof Error ? error.message : "unknown error",
      timestamp: new Date().toISOString(),
    })}\n`);
    reply(res, 500, { ok: false });
  });
});

server.listen(Number(process.env.PORT ?? 3000), () => {
  startupComplete = true;
});

if (process.env.VERIFY_LOG_BACKEND === "1") {
  void verifyLogSearch().then((result) => {
    process.stdout.write(`${JSON.stringify(result)}\n`);
  });
}
Enter fullscreen mode Exit fullscreen mode

The readiness timeout is deliberately 1.5 seconds. It is a configuration choice, not a universal optimum; it must stay below the probe timeout and below the latency your workload can tolerate. Benchmark that path against the real dependency. Do not pile five downstream checks into it just because they fit.

There is one subtle trap here: probe traffic can dominate low-volume logs. This implementation records failures and counts every outcome, which preserves incident evidence without writing a success log every few seconds. The metric gives the trend. The log gives the reason.

One switch, one check.

Container wiring without config sprawl

The deployment mapping is mechanical: point the startup probe at /health/startup, liveness at /health/live, and readiness at /health/ready. Docker has one container health check, so use the liveness endpoint there and keep dependency readiness at the orchestrator or load-balancer boundary. Probe intervals, timeouts, and failure thresholds are operational policy; copy-pasting arbitrary numbers would fake certainty.

For Kubernetes, give startup enough failures to cover the measured worst-case boot time. Then make liveness slower and less sensitive than readiness. A transient database response should stop new work quickly, but it should not erase the process state through a restart.

Test rollback explicitly. Deploy a version whose readiness check fails, verify that it receives no traffic, and confirm the previous ReplicaSet remains ready while the rollout is reversed. Watch the failure counter rise and inspect one structured failure record for the reason. During that test, the old ReplicaSet should keep serving while the new one stays out of rotation; a restart of the new process would prove nothing about dependency recovery and would erase useful state. Then separately force the process to stop answering liveness and verify that the runtime restarts it. Finally, skip one nightly pipeline run and check that the heartbeat monitor notices the absence. Those are three tests because they promise three different recovery actions.

Choosing the monitoring backend

I judge this layer by time-to-first-use and adapter size, not by the longest feature grid. The fair comparison is about ownership boundaries:

Option Best fit here Trade-off to accept
Prometheus plus Alertmanager Scrape the counter locally and own alert routing You operate the collection and alerting path
Grafana Cloud Send metrics and logs to one hosted observability workspace The hosted workspace becomes part of the operational dependency chain
Datadog Use managed monitors alongside a broader hosted telemetry stack Agent and product configuration are more surface area to govern
Healthchecks.io Detect that the nightly job never checked in It complements probes; it does not replace application readiness
Infrai Put 295 routes across 20 modules behind one REST API and one key No alert routing, uptime heartbeat, or distributed trace query is provided, so notifications and missed-job detection stay separate

No row wins every job. Prometheus is attractive when owning the stack is already normal. Datadog or Grafana Cloud fits teams that want a managed suite. Healthchecks.io is the precise extra tool for the nightly pipeline's silent-failure case. The REST option is reasonable when a small, stable adapter and backend replaceability matter most, but its logs only correlate trace_id and span_id if the application supplies those fields; there is no span tree to query.

That limitation changes the design. Keep alert evaluation outside the ingestion adapter, and let the adapter return enough information to log delivery failures. Do not pretend storage is paging. It isn't.

What I would change at scale

First, I would move the in-memory counters to the process's normal metrics client so restarts do not reset the only view of probe history. I would also rate-limit repeated failure logs while preserving a counter for every occurrence. The exact suppression window should come from measured incident traffic, not taste. Next, I would add trace_id and span_id to the structured log context when the application already has them. Sampling guidance matters if traces exist, but IDs in logs are correlation fields, not a substitute for distributed trace storage or a span tree. Finally, I would make the nightly pipeline send a completion heartbeat only after its output is durable. Alerting would live in Alertmanager, a managed monitor, or a polling job, depending on the chosen backend. The rollback rule remains boring and strict: probes decide container state, telemetry records why, and the scheduler heartbeat proves the work happened.

References

Top comments (0)