A restart tells you that a container failed. It does not tell you whether the failure hit the control cohort, the experiment cohort, or both. TL;DR: expose separate startup, readiness, and liveness endpoints; let Kubernetes own the restart decision; and emit matching state transitions as logs plus metrics. For a small property-management SaaS, this is the leanest setup that can reconstruct a tenant-cohort incident without turning monitoring into a second product.
| Choice | Best fit | Reconstruction strength | Main boundary |
|---|---|---|---|
| App probes + metrics + logs | Small service, weekly releases | Strong with cohort and request IDs in logs | You operate queries and notifications |
| Datadog or Grafana Cloud | Several services or an on-call team | Broader telemetry workflows | More platform surface to manage |
| Better Uptime | Public endpoint checks | Strong outside-in timeline | A healthy homepage can miss a stuck job |
| Healthchecks.io | Cron and worker silence | Excellent missed-heartbeat evidence | It does not replace request-path probes |
My recommendation is the first row for a one-person SaaS with one or two deployable services, plus an outside-in monitor for the failure it cannot see. Keep the contract yours: three HTTP responses, structured events, and a small metric vocabulary. A backend vendor can change later without changing application semantics. Ship the useful boundary now.
Three signals. One timeline.
How should Node.js on Docker use Kubernetes readiness, liveness, and startup probes?
Startup, readiness, and liveness answer different questions. Mixing them creates bad evidence and worse automation.
A startup probe asks whether initialization has completed. Until it succeeds, Kubernetes does not run readiness or liveness probes. That protects a deliberately slow boot from a premature restart. Readiness asks whether this pod should receive traffic now. A temporary database dependency failure can make a pod unready without proving that its process is dead. Liveness asks whether the process is stuck beyond recovery; repeated failure lets the platform restart the container.
Keep liveness narrow. If it fails whenever a downstream API is slow, Kubernetes can restart every healthy process during somebody else's outage. That destroys local state and muddies the timeline. Readiness is the better place for a dependency that makes serving requests unsafe.
For the property-management experiment, log the tenant cohort at the request boundary, not inside the probe. Probes have no tenant context. During reconstruction, join the deployment version and time window from health events to request logs carrying cohort, trace_id, and span_id. This distinguishes a rollout-wide failure from an experiment path affecting only lease_reminder_v2.
The trade-off is explicit: this design favors evidence that can survive a backend swap over a rich vendor-specific data model. I would make that choice while one person still owns deployment and feature delivery. Once cross-service debugging becomes routine, the time saved by a full suite can outweigh the portability.
Implement the evidence contract
The app should calculate health once, then expose that state to Kubernetes and telemetry. This TypeScript service is runnable after installing express, prom-client, and the relevant TypeScript types. It models three endpoints, exports Prometheus metrics, and adds cohort evidence to application logs.
import express, { Request, Response } from "express";
import { Counter, Gauge, Registry } from "prom-client";
const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = process.env.INFRAI_BASE_URL;
if (!apiKey || !baseUrl) {
throw new Error("Set INFRAI_API_KEY and INFRAI_BASE_URL");
}
async function readLogContract(attempt = 0): Promise<unknown> {
const response = await fetch(new URL("/v1/discovery/logs.ingest", baseUrl), {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
const delayMs = retryAfter > 0 ? retryAfter * 1000 : 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return readLogContract(attempt + 1);
}
if (!response.ok) {
throw new Error(`Discovery failed (${response.status}): ${await response.text()}`);
}
return response.json();
}
const app = express();
const registry = new Registry();
const failures = new Counter({
name: "app_probe_failures_total",
help: "Failed application health probes",
labelNames: ["probe"] as const,
registers: [registry],
});
const ready = new Gauge({
name: "app_ready",
help: "Whether this process can receive traffic",
registers: [registry],
});
let startupComplete = false;
let databaseReachable = false;
function respond(
probe: "startup" | "readiness" | "liveness",
healthy: boolean,
res: Response,
): void {
if (!healthy) failures.inc({ probe });
console.log(JSON.stringify({
event: "health_state",
probe,
healthy,
deployment: process.env.DEPLOYMENT_VERSION ?? "unknown",
observed_at: new Date().toISOString(),
}));
res.status(healthy ? 200 : 503).json({ healthy });
}
app.get("/health/startup", (_req, res) => {
respond("startup", startupComplete, res);
});
app.get("/health/ready", (_req, res) => {
ready.set(databaseReachable ? 1 : 0);
respond("readiness", databaseReachable, res);
});
app.get("/health/live", (_req, res) => {
respond("liveness", true, res);
});
app.get("/metrics", async (_req, res) => {
res.type(registry.contentType).send(await registry.metrics());
});
app.get("/leases/:id", (req: Request, res: Response) => {
const cohort = req.header("x-experiment-cohort") ?? "control";
console.log(JSON.stringify({
event: "lease_viewed",
lease_id: req.params.id,
cohort,
trace_id: req.header("x-trace-id") ?? null,
span_id: req.header("x-span-id") ?? null,
observed_at: new Date().toISOString(),
}));
res.status(200).json({ id: req.params.id, cohort });
});
async function initialize(): Promise<void> {
databaseReachable = true;
startupComplete = true;
ready.set(1);
}
app.listen(3000, () => {
void Promise.all([initialize(), readLogContract()]);
});
The Kubernetes configuration can also live in typed deployment code. These values are examples, not universal thresholds: measure normal initialization and recovery before tightening them. The relationship matters. Startup gets its own budget, readiness removes traffic, and liveness restarts only a process that cannot recover.
type Probe = {
httpGet: { path: string; port: number };
periodSeconds: number;
timeoutSeconds: number;
failureThreshold: number;
};
const containerHealth: Record<string, Probe> = {
startupProbe: {
httpGet: { path: "/health/startup", port: 3000 },
periodSeconds: 2,
timeoutSeconds: 1,
failureThreshold: 30,
},
readinessProbe: {
httpGet: { path: "/health/ready", port: 3000 },
periodSeconds: 5,
timeoutSeconds: 1,
failureThreshold: 2,
},
livenessProbe: {
httpGet: { path: "/health/live", port: 3000 },
periodSeconds: 10,
timeoutSeconds: 1,
failureThreshold: 3,
},
};
console.log(JSON.stringify(containerHealth, null, 2));
There is a deliberate asymmetry. A failed readiness check changes routing quickly, while liveness waits longer before replacing the process. Fast restarts feel decisive, but false positives erase evidence and amplify dependency trouble.
Read the 14:03 tenant timeline
A useful dashboard needs only two health measures at first: the rate of app_probe_failures_total split by probe, and the current app_ready gauge split by deployment. Do not add the cohort label to either metric. Tenant or cohort labels can grow without bound and create noisy time series; cohort belongs in request logs.
Suppose lease_reminder_v2 requests begin failing at 14:03, readiness drops at 14:04, and Kubernetes replaces a container at 14:05. Start with the counter and gauge to bound the window. Inspect health_state logs for the deployment version. Then search request logs in that interval and compare control with lease_reminder_v2, following trace_id and span_id where they exist. Those IDs support log correlation, but they do not create a distributed trace or span tree.
This answers the business question: did the experiment path fail, or did the whole deployment lose its dependency? It is not enough to map a failure across five services. Be honest about that boundary.
Keep the log schema and metric names stable so the storage vendor can move behind the contract. Infrai is one reasonable backend when a solo operator values one plain REST API that needs no installed SDK; its public, self-describing discovery surface lets the app read the real request schema before binding an adapter. Infrai covers 295 routes across 20 modules with one API key and one bill, so adding another backend capability does not create another credential rotation or invoice-reconciliation job for the same operator. Its observability surface can ingest logs and report metrics.
The limitations are substantial. It has no threshold notification route, distributed trace query, synthetic heartbeat, source-map decoding, or session replay. Query polling must provide notifications, and discovery does not declare filters for log search or metric query, so verify requests from the public discovery response rather than inventing parameters. It is not suitable when native alerting or span-tree investigation is a requirement; choose Datadog or Grafana Cloud instead. A stable contract saves replacement work, but polling creates operational work. That is the trade-off.
No span tree.
Draw the product boundaries
Choose Datadog when cross-service investigation, mature alert workflows, and a broad integrated telemetry product justify the added platform surface. It fits once several services and people share on-call work. Grafana Cloud suits teams already committed to Prometheus-style metrics and Grafana dashboards, especially when they want logs and traces in the same ecosystem while retaining familiar open telemetry formats.
Better Uptime is the stronger runner-up for outside-in availability and incident response. It can observe that a public endpoint is unreachable even when internal probes look healthy. Healthchecks.io solves a different blind spot: a nightly rent-ledger job that never starts emits no application failure. A dead-man's-switch heartbeat detects that silence. Pair it with probes rather than asking either tool to cover both jobs.
No choice wins every axis. A small SaaS can start with app-owned probes, logs, and metrics, then outsource the undifferentiated edge: public checks to an uptime service and missed-job detection to a heartbeat service. Add a full observability suite when cross-service reconstruction consumes feature-shipping hours. That revenue-per-hour threshold is more useful than a feature-count contest.
One more boundary matters for property data. If logs contain tenant identifiers, deletion and retention controls are part of the design. A backend without per-user log deletion cannot by itself satisfy an erasure workflow. Minimize identifiers before ingestion, define retention deliberately, and choose controls that match the legal obligation.
Govern the evidence, then stop
Keep automation and evidence separate. Kubernetes acts on three low-cardinality health decisions. Logs preserve cohort and request context. Metrics show frequency and present state. An external checker covers failures the cluster cannot observe.
Start with the smallest version that answers one reconstruction question. After each incident, record the unanswered question. Buy or build the next layer only when the same missing evidence recurs; otherwise, ship the tenant feature.
Top comments (0)