DEV Community

WindwhisperBoren33
WindwhisperBoren33

Posted on

Express Production Health Check Endpoints in Node.js Docker — Readiness, Liveness, 5xx Logs

#node #observability #docker

TL;DR: Give the marketplace pipeline two small Express endpoints: liveness says the Node.js process can answer, while readiness says this instance can accept a run. Poll through the external route, retain every sanitized 5xx, and keep compact run summaries longer than verbose success events.

The bill is ingestion, indexed fields, retained bytes, query work, and alert evaluation. Start with events per run × encoded bytes × retained runs × storage overhead. In a planning example, not a benchmark, 8 million offers emitting four 700-byte events produce 22.4 GB per run and 672 GB across 30 runs before indexing or replication. One terminal event per offer is 5.6 GB; 10,000 sampled successes at four events add 28 MB. Event elimination moves the dominant term.

Retain evidence in proportion to diagnostic value. Keep failures, rejected batches, and one run summary; sample repetitive successes.

What should an Express production health check endpoint prove about readiness?

/health/live should be shallow. If the event loop answers, return success; a dependency outage should not convert every healthy process into a restart candidate. /health/ready controls work admission. It can require completed startup and loaded configuration, and it should fail before a worker drains so no new batch arrives.

Kubernetes defines liveness, readiness, and startup probes separately. Docker HEALTHCHECK also consumes an application contract; neither system decides what “ready” means for this pipeline. Keep responses bounded and omit credentials, dependency URLs, stack traces, and queue contents.

curl --fail --silent --show-error --max-time 2 https://pipeline.example.test/health/live
Enter fullscreen mode Exit fullscreen mode 
curl --fail --silent --show-error --max-time 2 https://pipeline.example.test/health/ready
Enter fullscreen mode Exit fullscreen mode

Store health state transitions, not every successful poll. Tiny records add up.

Search failures without indexing the marketplace

A useful error record includes timestamp, severity, service, environment, pipeline_run_id, stage, HTTP status, error class, duration, and a trace identifier. Never log tokens, payment data, or raw bodies. Stage and status are bounded dimensions; run IDs, listing IDs, URLs, exception messages, and stack traces can approach one value per event. Store the latter when useful, but do not automatically index them as aggregation labels.

Prometheus documents the analogous metric risk: every unique label combination creates a time series. Logs differ, yet the cardinality budget still matters. Narrow by environment, service, run, stage, and status before searching text. W3C Trace Context supplies interoperable propagation fields.

curl --fail --silent --show-error --max-time 5 --header 'Accept: application/json' 'https://logs.example.test/search?service=catalog-pipeline&run_id=run-2026-09-21&stage=publish&status_class=5xx'
Enter fullscreen mode Exit fullscreen mode

That is an illustrative generic interface. A real search endpoint needs authentication, authorization, time and result limits, and an audit trail.

Attribute cost to each run.

Attach a stable run ID to logs and traces, plus low-cardinality ownership fields such as service and environment. Do not put run or listing IDs into metric labels. Usage accounting can then group bytes by run without parsing messages.

Signal Detailed retention Cost control
Health polls Failed transition metadata Drop repetitive successes
Successful work Stable bounded sample Sample before transport
5xx failures Every sanitized event Bound fields and duration
Completion One run summary Retain compact records longer

Head sampling can discard a trace before its outcome is known. Outcome-aware sampling retains failures but requires buffering and a decision window. For logs, a clear policy is often enough: keep every error, aggregate routine success, and choose a deterministic sample by hashing a run or batch identifier.

External monitoring belongs outside the container.

A self-check does not cover DNS, certificates, gateways, or routing. Keep local liveness for supervision and poll readiness through the path operators depend on. Alert on a short sequence or window, not one miss; state both interval and threshold because three failures five seconds apart imply a different delay from three failures one minute apart.

Keep probe and 5xx alerts separate. A ready worker can reject one record class, while an unready worker may emit no 5xx after traffic stops. Test startup, draining, a sanitized 5xx, timeout, DNS failure, certificate failure, and alert recovery before deployment.

What do we deliberately stop keeping?

Stop retaining a verbose event for every successful transformation. Stop indexing raw URLs, listing IDs, and exception text as universal dimensions. Keep bounded counters, summaries, stable samples, and complete sanitized failures.

The limitation is real: an unsampled success may leave no event-level history when it later proves semantically wrong. This trade-off is unsuitable when regulation or dispute handling requires a complete per-item audit trail; in that case, retain a purpose-built audit record with its own access and deletion policy instead of treating diagnostic logs as the ledger. Mitigate ordinary diagnostic loss with replayable source data where policy permits, deterministic samples, schema versions, and a documented investigation window. The shortest acceptable retention covers alert response, replay, compliance, and the slowest credible investigation. Measure encoded sizes and counts after launch; planning assumptions should not become quotas.

References

Further reading

Top comments (0)