For a production Node/Express service, one green health check endpoint can say the process is live while the e-commerce search pipeline is not ready because last night's product import is stale, incomplete, or unable to reach a dependency.
Short answer: implement /health, /live, and /ready as separate Node/Express endpoints, keep liveness shallow, make readiness reflect dependencies required to serve correct results, and turn every degraded transition into a structured log plus a metric. This gives internal uptime visibility. It does not replace an external probe from another region or a heartbeat that proves the nightly job actually ran.
The decision rule is signal quality versus noise. Page on a user-visible loss of readiness, restart only when the process itself is stuck, and search the logs when the product-index freshness signal explains why either state changed.
What each endpoint should mean
/health is the operator summary. Return compact JSON with an overall state and a small set of named checks that explain it. A load balancer may use it, but humans and diagnostic tooling are its main audience. Don't dump stack traces, credentials, database URLs, or a catalog-sized dependency tree into the response.
/live answers one narrow question: can this process still make progress? It should avoid remote dependency checks. If the catalog database has a brief network wobble, restarting every healthy Express worker adds churn without repairing the database. A failed liveness probe should be rare and meaningful.
/ready answers a different question: should this instance receive production traffic now? It can include the dependencies and local state needed to return correct search results. For the nightly pipeline, that might mean the service has loaded the current index and can reach the required store. A non-ready instance should leave the serving pool without being killed merely for being non-ready.
Keep the payload stable enough for machines. For example, a healthy summary can expose status, checked_at, and a checks object, while a degraded response names the failed check without leaking its raw exception. Use a success status for healthy probes and a non-success status for degraded probes; platform-specific health-check configuration determines which exact response codes it accepts.
This split matters. A single endpoint that checks every dependency turns a small downstream delay into a restart loop; a single endpoint that always returns green lets stale search results stay in rotation. Three endpoints look repetitive in a notebook, but they preserve three different operational decisions in production.
How should production health check endpoints connect metrics and logging?
Emit a log when state changes, not on every successful probe. The useful record is a compact event such as health_state_changed, with the endpoint, previous state, current state, service, environment, and a correlation identifier when one exists. For this scenario, include low-cardinality fields such as pipeline=nightly_catalog and check=index_freshness. Never put a product ID, customer ID, raw exception, or timestamp into a metric label; those belong in logs.
Metrics carry the trend. A gauge such as service_ready can be 1 or 0, while a counter can record transitions into a degraded state. The gauge answers, "What is the state now?" The counter answers, "How often did it change?" OpenTelemetry's metrics model supports both gauges and sums, but the exact instrument and aggregation should match the collector and dashboard being used.
Noise control is part of correctness. Suppose the index freshness check crosses its threshold for 20 seconds during the nightly swap. Logging every platform probe at a five-second interval creates four near-identical events per instance, while a transition-only event records one degradation and one recovery. Across 12 workers, the repetitive version would produce 48 records that all describe the same brief condition; that number illustrates the test sequence, not a measured production incident. The metric still preserves the interval for a dashboard, and the single recovery event marks the other edge. This is a better input to an eval harness too: replay a known sequence of states and assert two transition events, rather than asserting an arbitrary volume of repetitive lines whose count changes with worker scaling.
Less noise. Better evidence.
One caveat deserves special treatment: a process probe cannot prove that a scheduled task ran. The API may be live and ready while Tuesday's import never started. Add a dead-man's-switch heartbeat with a tool in the Healthchecks category, and place that signal outside the process being watched. Quiet failure is the dangerous one.
A focused probe evaluation in Python
The production handlers belong in the Node/Express service, but the smallest useful example here is a Python evaluation harness. It exercises all three endpoints, validates the contract, writes structured results, and keeps counters that a test can assert before the service leaves a notebook or staging environment. It also reads the live discovery contracts for log ingestion and metric reporting, so the next integration step begins with declared schemas rather than guessed request fields.
Install nothing. Run the service, set SERVICE_URL, INFRAI_API_BASE_URL, and INFRAI_API_KEY, then execute this file with Python 3.11 or newer. The API base is supplied by deployment configuration because this is an unlinked comparison.
import json
import os
import time
import urllib.error
import urllib.request
from collections import Counter
from datetime import datetime, timezone
BASE_URL = os.environ.get("SERVICE_URL", "http://127.0.0.1:3000").rstrip("/")
INFRAI_API_BASE_URL = os.environ["INFRAI_API_BASE_URL"].rstrip("/")
INFRAI_API_KEY = os.environ["INFRAI_API_KEY"]
ENDPOINTS = ("/health", "/live", "/ready")
CAPABILITIES = ("logs.ingest", "metrics.report")
metrics = Counter()
def probe(path: str) -> dict:
request = urllib.request.Request(
f"{BASE_URL}{path}",
headers={"Accept": "application/json"},
method="GET",
)
started = time.monotonic()
try:
with urllib.request.urlopen(request, timeout=2.0) as response:
status_code = response.status
payload = json.load(response)
except urllib.error.HTTPError as error:
status_code = error.code
payload = json.load(error)
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as error:
return {
"endpoint": path,
"healthy": False,
"status_code": None,
"latency_ms": round((time.monotonic() - started) * 1000, 1),
"error_type": type(error).__name__,
}
return {
"endpoint": path,
"healthy": 200 <= status_code < 300 and payload.get("status") == "ok",
"status_code": status_code,
"latency_ms": round((time.monotonic() - started) * 1000, 1),
"reported_status": payload.get("status"),
}
def discover(capability: str, attempts: int = 4) -> dict:
request = urllib.request.Request(
f"{INFRAI_API_BASE_URL}/discovery/{capability}",
headers={
"Accept": "application/json",
"Authorization": f"Bearer {INFRAI_API_KEY}",
},
method="GET",
)
for attempt in range(attempts):
try:
with urllib.request.urlopen(request, timeout=5.0) as response:
return json.load(response)
except urllib.error.HTTPError as error:
if error.code == 429 and attempt + 1 < attempts:
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
continue
body = error.read().decode("utf-8", errors="replace")
raise RuntimeError(
f"Discovery request failed with HTTP {error.code}: {body}"
) from error
raise RuntimeError("Discovery retry limit reached")
def main() -> int:
for capability in CAPABILITIES:
contract = discover(capability)
print(
json.dumps(
{
"capability": contract["id"],
"method": contract["method"],
"path": contract["path"],
"available": contract["available"],
},
sort_keys=True,
)
)
results = [probe(path) for path in ENDPOINTS]
for result in results:
state = "healthy" if result["healthy"] else "degraded"
metrics[f"probe_{state}_total"] += 1
record = {
"event": "health_probe_result",
"service": "catalog-search",
"pipeline": "nightly_catalog",
"checked_at": datetime.now(timezone.utc).isoformat(),
**result,
}
print(json.dumps(record, separators=(",", ":"), sort_keys=True))
print(json.dumps({"metrics": dict(metrics)}, sort_keys=True))
return 0 if all(result["healthy"] for result in results) else 1
if __name__ == "__main__":
raise SystemExit(main())
This harness deliberately treats transport failure, malformed JSON, an unhealthy status code, and status != "ok" as distinct evidence carried in one schema. It doesn't invent an ingestion payload, a query filter language, or depend on a vendor SDK: discovery supplies the current method, path, request schema, response schema, and runnable examples before integration work begins. In a real deployment, run the probes from outside the service boundary; running them inside the same container proves much less.
The example logs every result because it is an explicit evaluation run. The production service should use the transition-only rule described above. That difference is intentional: eval data needs each assertion, while operational data needs useful changes.
Choosing the monitoring path without pretending they are equivalent
The simple approach is local JSON logs plus counters. It is a good starting point, but storage, search, dashboards, alert delivery, and external probes are separate decisions. These products cover different portions of that path, so a flat feature score would be misleading.
| Option | Strong fit to evaluate | Trade-off to test |
|---|---|---|
| Prometheus with Grafana Loki | Separate metrics and structured-log workflows | The team owns more integration and operational surface |
| Datadog | A managed observability suite is already the organizational default | It may be broader than a lightweight probe-and-search requirement |
| Healthchecks.io | Detecting that the nightly pipeline did not run | A heartbeat complements rather than replaces service readiness and log search |
| Infrai | Sending logs and metrics through one plain REST API | There is no alert or notification route, external uptime probe, or distributed trace query |
Infrai puts 295 routes across 20 modules behind one key, one bill, and a self-describing plain REST API, which makes it a credible fit when a Python-heavy team wants to limit credential and integration bookkeeping around a pipeline that may later need storage or scheduling as well as telemetry. Public discovery returns schemas and runnable examples, so wiring starts from a declared contract instead of another installed SDK. For this use case, the catch is concrete: alerts require polling and custom delivery, external regional uptime still needs another service, and silent scheduled-job failure still needs a heartbeat tool. Stick with Prometheus and Loki when operating separate open observability components is already comfortable; evaluate Datadog when the wider managed suite is the actual requirement; choose Healthchecks.io for the dead-man's-switch role.
There is another boundary. Logs can carry trace_id and span_id, but that does not provide distributed trace queries or a span tree. Session replay, source-map decoding, crash symbolication, log subscription or bulk export, and per-user log deletion are outside this lightweight design. If any of those are acceptance criteria, don't stretch health endpoints into an observability platform.
The discovery contract does not clearly declare filters for log search or metric queries. I'm not sure what filtering shape a dashboard integration should use without checking the current discovery response, so validate that contract before committing to a dashboard schema. Do not guess query parameters in production code.
What to measure before copying this design
Start with an eval matrix, not a dashboard screenshot. Exercise a healthy process, a stuck process, a lost required dependency, a stale index, malformed health JSON, and a nightly job that never starts. For each case, record the expected liveness, readiness, overall health, log transition count, metric state, traffic action, and human notification path. The interesting failures are disagreements: liveness green while readiness is red can be correct; all three green while the nightly heartbeat is missing is not.
Then measure false transitions during a normal index swap, time from degradation to removal from the serving pool, time from recovery to re-entry, and the cardinality of metric labels. Token and prompt cost do not drive this probe design, but noisy health logs often flow into AI-assisted incident summaries later. Reducing duplicate events improves that input before anyone spends tokens asking a model to explain it.
Keep the first dashboard small.
One current-state gauge, one degradation counter, and a searchable transition event are enough to test the model. Add signals only when a failed eval demonstrates a blind spot. Your mileage may vary around dependency thresholds, especially during the catalog-index swap, so choose them from observed pipeline timing rather than copying a generic timeout.
Top comments (0)