App Health Endpoint Design: 3 Probes That Keep Logging and Metrics Useful

#observability #kubernetes #node

Short answer: for a Node.js app in Docker or Kubernetes, give startup, readiness, and liveness probes separate meanings, keep routine health traffic out of application logging, and measure state transitions instead of counting every successful check. For a property-management API rolling out a new pricing rule, this preserves useful metrics: whether an instance can calculate rent correctly and accept traffic, without turning each kubelet poll into noise.

Which health signal should control each container decision?

Start with the decision, not the endpoint name.

Signal	Question it answers	Include	Exclude	Action
Startup	Has initialization completed?	Configuration parsing, pricing-rule compilation, required local warm-up	Long-term dependency health	Allow the process more time before other probes apply
Readiness	Can this instance safely receive a new pricing request now?	Ability to serve the active rule version and any required dependency state	Optional analytics and background exports	Remove the pod from Service endpoints
Liveness	Is the process stuck beyond local recovery?	Event-loop progress or another narrow process invariant	Database, cache, and third-party availability	Restart the container

This split is the main noise filter. A downstream dependency becoming unavailable can make a pod unready, but restarting the same healthy process usually doesn't repair that dependency. If the dependency is placed in liveness anyway, every pod can restart together. The health response has then amplified one problem into two: lost capacity plus a restart storm.

The pricing rollout makes readiness more demanding than “the port is open.” Imagine rule version rent-2026-08 is enabled for one building cohort. A newly started instance has loaded configuration but hasn't compiled that version yet. It is alive. It isn't ready. Its startup check should hold back liveness and readiness until initialization finishes; afterward, readiness should stay false until the active rule can be evaluated. That distinction protects tenants from inconsistent quotes while keeping the process lifecycle understandable.

Don't put every dependency in every check.

When should startup, readiness, and liveness probes be picked?

Pick a startup probe when initialization can legitimately take longer than the liveness budget. Kubernetes does not run liveness or readiness probes until the startup probe succeeds. That makes startup the right place for one-time work such as loading and validating the pricing rules. The catch is that an excessively generous startup window delays detection of a process that will never initialize, so derive the window from observed cold-start distributions rather than copying a large value. Pick readiness for conditions that should stop new traffic but may recover without a process restart. A pricing worker that temporarily lacks the active rule snapshot is a clean example: readiness failures remove the pod from matching Service endpoints while the container continues running, creating space for recovery and keeping the action proportional. Pick liveness only for a condition a restart can fix. An event loop that no longer advances qualifies; an unreachable billing database does not, because a fresh process reaches the same database. This is a sharp trade-off — a narrow check can miss some degraded states, while a broad check can cause destructive restarts. Favor the narrow invariant and alert on degradation through metrics.

Keep it narrow.

How should a beginner wire Kubernetes probes to a Node.js health endpoint?

Use distinct paths even if they share one small server. The handlers below avoid network calls in liveness, return 503 when startup or readiness has not been reached, and expose no tenant or property details. 204 is enough for success because the kubelet needs status, not a diagnostic document.

import { createServer, type ServerResponse } from "node:http";
import { monitorEventLoopDelay } from "node:perf_hooks";

let startupComplete = false;
let activeRuleVersion: string | null = null;
let dependencyState: "available" | "unavailable" = "unavailable";

const loopDelay = monitorEventLoopDelay({ resolution: 20 });
loopDelay.enable();

function reply(response: ServerResponse, status: 204 | 503): void {
  response.writeHead(status, { "cache-control": "no-store" });
  response.end();
}

const healthServer = createServer((request, response) => {
  if (request.method !== "GET") {
    response.writeHead(405, { allow: "GET" });
    response.end();
    return;
  }

  if (request.url === "/health/startup") {
    reply(response, startupComplete ? 204 : 503);
    return;
  }

  if (request.url === "/health/ready") {
    const ready =
      startupComplete &&
      activeRuleVersion === "rent-2026-08" &&
      dependencyState === "available";
    reply(response, ready ? 204 : 503);
    return;
  }

  if (request.url === "/health/live") {
    const loopIsResponsive = loopDelay.mean / 1e6 < 250;
    reply(response, loopIsResponsive ? 204 : 503);
    return;
  }

  response.writeHead(404);
  response.end();
});

async function initialize(): Promise<void> {
  // Replace these assignments with validated startup and dependency state.
  activeRuleVersion = "rent-2026-08";
  dependencyState = "available";
  startupComplete = true;
}

healthServer.listen(3001, "0.0.0.0", () => {
  void initialize();
});

The 250 millisecond threshold is an example policy, not a universal Node.js limit. I'm not sure what threshold fits your workload until its event-loop delay distribution is visible under representative traffic; CPU limits, synchronous work, and request latency objectives all change the useful boundary. Tune it from measurements, and require repeated failures through probe configuration so one scheduling spike doesn't restart an otherwise healthy instance.

The corresponding Deployment fragment can stay type-checked in an infrastructure package. Kubernetes calculates the maximum startup allowance from failureThreshold * periodSeconds; here it is 60 seconds. Readiness reacts sooner, while liveness waits through three consecutive failed checks.

type Probe = {
  httpGet: { path: string; port: number };
  periodSeconds: number;
  timeoutSeconds: number;
  failureThreshold: number;
};

const startupProbe: Probe = {
  httpGet: { path: "/health/startup", port: 3001 },
  periodSeconds: 2,
  timeoutSeconds: 1,
  failureThreshold: 30,
};

const readinessProbe: Probe = {
  httpGet: { path: "/health/ready", port: 3001 },
  periodSeconds: 5,
  timeoutSeconds: 1,
  failureThreshold: 2,
};

const livenessProbe: Probe = {
  httpGet: { path: "/health/live", port: 3001 },
  periodSeconds: 10,
  timeoutSeconds: 1,
  failureThreshold: 3,
};

export const containerHealth = {
  startupProbe,
  readinessProbe,
  livenessProbe,
};

There is one important boundary in that first snippet. Readiness reads a locally maintained dependency state instead of making a fresh remote request for every kubelet poll. Update that state from the application's normal connection management or a bounded background check. Otherwise, health traffic can become material load on the very dependency being diagnosed, and a slow dependency can consume every health handler at once.

Keep the health server small, but don't pretend it is a security boundary. Restrict exposure with cluster networking and avoid returning rule versions, property identifiers, credentials, stack traces, or dependency addresses in the response body. Diagnostics belong in authenticated operational surfaces.

Keep probe logging quiet and metrics actionable

Successful probes shouldn't produce one application log per request. At a five-second readiness interval, one pod can receive 17,280 readiness checks per day. Multiply that by replicas and add startup plus liveness traffic: access-log volume grows while operator knowledge stays flat. Filter the three exact health paths in the access logger, then emit a structured log only when the health state changes. Keep unsuccessful transitions, recovery transitions, and configuration changes.

type HealthState = "starting" | "ready" | "unready" | "live";

let previousReadiness: HealthState = "starting";

function recordReadiness(next: HealthState, reason: string): void {
  if (next === previousReadiness) return;

  const event = {
    timestamp: new Date().toISOString(),
    event: "health.readiness.transition",
    previous: previousReadiness,
    next,
    reason,
    ruleVersion: activeRuleVersion,
  };

  process.stdout.write(`${JSON.stringify(event)}\n`);
  previousReadiness = next;
}

function shouldWriteAccessLog(pathname: string): boolean {
  return !new Set([
    "/health/startup",
    "/health/ready",
    "/health/live",
  ]).has(pathname);
}

For metrics, track state and transitions rather than attaching a label for each tenant, property, request, or error message. A gauge such as app_ready{rule_version="rent-2026-08"} can be 0 or 1; a counter such as app_readiness_transitions_total{to="unready",reason="rule_unavailable"} captures churn. Keep reason to a reviewed, finite vocabulary. Raw property IDs create unbounded cardinality and can expose operational data.

Probe metrics also answer a different question from request metrics. Readiness says whether traffic should arrive. HTTP duration and error measurements say what happened after it arrived. During a flagged rollout, compare request outcomes for the enabled and control cohorts using a bounded cohort label, while watching readiness transitions by rule version. Logs explain individual transitions. Traces connect a pricing request to its downstream work. Each signal gets one job.

Sampling needs similar care. OpenTelemetry defines head sampling as a decision made when a trace starts and tail sampling as a decision made after all or most spans are available. Head sampling is simpler and cheaper to operate, but it can discard a rare slow or failed pricing trace before its outcome is known. Tail sampling can retain traces based on outcomes, at the cost of buffering and additional operational complexity. Your mileage may vary: for a small beginner deployment, keep all low-volume health transitions as metrics and logs, then use a conservative head-sampling policy for request traces before adding a tail-sampling collector.

This is the diagram in words: kubelet asks three narrow questions; local state answers immediately; state changes produce one log and one counter increment; pricing requests produce normal latency and outcome telemetry; alerts evaluate sustained user impact, not individual probe failures. Quiet input. Useful output.

Limits and rollout checks

This pattern is not suitable when a separate process supervisor already owns restarts based on a stronger runtime invariant, or when the workload cannot truthfully determine readiness without a remote check. In the latter case, keep that check bounded and cached, and choose a timeout below the probe timeout. For a single-process Docker deployment without Kubernetes Service routing, readiness alone does not remove traffic; the reverse proxy or orchestrator must consume it. Stick with a platform-native health mechanism when it already provides those semantics.

Before enabling the pricing flag, test four transitions: slow startup, loss and recovery of the active rule, event-loop saturation, and a clean rollout from the control cohort. Confirm that startup delays traffic without causing premature liveness checks, unready instances leave rotation, dependency loss does not trigger restarts, and recovery creates one transition event rather than thousands of success logs. Then alert on sustained absence of ready capacity and on pricing request outcomes. A single failed poll is evidence. It isn't an incident by itself.

DEV Community