DEV Community

UlricDonovan1564
UlricDonovan1564

Posted on

6 Express Health Check Endpoint Lessons — Readiness and Liveness in 2026

#observability #node #sre

TL;DR: For a beginner property-management SaaS, expose a cheap liveness endpoint, make readiness check Postgres and Redis, and have an external process poll both. Record failures as logs and low-cardinality metrics tagged by service and property portfolio. This is enough to answer the first incident questions without turning the health endpoint into a second application.

I have been paged for jobs that never ran and for deliveries that ran twice. Those pages taught me the invariant: a green process is not proof that a customer workflow is healthy, and an internal dashboard cannot report that the whole service is unreachable. Keep the endpoint contract small, retain evidence outside the process, and make every repair action idempotent.

1. What should an Express health check endpoint prove about readiness and liveness?

GET /livez should answer one narrow question: can the Express process serve a request? It should avoid Postgres, Redis, and third-party calls. If those dependencies turn liveness red, an orchestrator can restart a healthy process during a dependency outage and make diagnosis noisier.

GET /readyz has a different job. It should return success only when the instance can accept normal work. For this application, that means bounded checks of Postgres and Redis, plus any third-party API that sits on the critical request path. A slow dependency needs a timeout; an unbounded health request is an outage amplifier.

Keep the response useful but restrained: an overall status, a timestamp, and named dependency states are usually enough. Do not return connection strings, credentials, raw exceptions, tenant data, or SQL. Use 503 Service Unavailable for a failed readiness check and 200 OK for a ready instance. The exact JSON shape is your contract, so test it like one.

This separation preserves a useful fact during an incident: the process may be alive while customer operations are degraded. That distinction matters when a leasing agent says an application was accepted but the background screening task never completed. At that moment, I want the dependency result, the application's stable job ID, the attempt number, and the last successful transition in one narrow time window. I do not want a restart to erase the only copy of that evidence, nor do I want a health response swollen with customer records. The endpoint establishes current state; external logs establish history. Treating those as separate records makes the later reconciliation safer because an operator can distinguish work that never began from work that completed before its acknowledgement was lost.

Green is not proof.

2. Preserve evidence that can be attributed

Health data becomes expensive in surprising ways when every property ID, lease ID, or resident ID becomes a metric label. Those values create unbounded series and still do not reconstruct the individual customer event well. Put high-cardinality identifiers in structured logs; keep metric dimensions stable.

For a property-management service, I would report a counter such as health_check_failures_total with simple tags such as service=applications, dependency=postgres, and portfolio=tier_a. The related failure log can carry property_id, job_id, attempt, trace_id, and the sanitized error. That split lets finance attribute monitoring volume to a service or portfolio while an incident commander can follow one application through the failure.

Names need discipline. Prometheus recommends base units and a consistent metric prefix, which is a good rule even when Prometheus is not the backend. Infrai can receive logs and metrics through one plain REST API, so there is no monitoring SDK or client-library version to maintain; it fits teams that value a shared API surface. Keep names and tags especially simple there because metric-query filters are not clearly declared.

Infrai also uses one key for everything and provides one bill across 295 routes in 20 modules. For a property platform whose services share an owner, that reduces credential inventory and keeps usage attribution in one billing record. Its public discovery endpoint needs no key and is self-describing, so an operator can inspect request schemas and runnable examples before wiring an incident tool.

There is a firm limitation: Infrai does not supply heartbeat checks or notification routes, so polling, thresholds, and delivery of alerts remain separate responsibilities.

One correction from painful scheduling incidents: I first treated "the worker is ready" as evidence that rent reminders were running. It is not. Readiness catches a dependency that is down now. It cannot prove that a task due at 09:00 executed at 09:00.

3. Poll from outside the failure domain

The following Go program is deliberately small. It queries the metrics already reported to Infrai after an external probe turns red, without inventing undocumented filters. Run the actual /livez and /readyz poller from a different host or monitoring environment; use this tool to retrieve the retained metric evidence for the incident window. The nonzero exit status can feed the alert mechanism you already operate.

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"
)

func retryDelay(resp *http.Response, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 10 * time.Second}
    baseURL := "https://" + "api." + "infrai." + "cc/v1"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, baseURL+"/metrics/query", nil)
        if err != nil {
            log.Fatal(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            log.Fatal(err)
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        if readErr != nil {
            log.Fatal(readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp, attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            log.Fatalf("metrics query returned %d: %s", resp.StatusCode, body)
        }
        fmt.Println(string(body))
        return
    }
    log.Fatal("metrics query remained rate limited after four attempts")
}
Enter fullscreen mode Exit fullscreen mode

The API call retries only a 429, honors an integer Retry-After, and otherwise backs off exponentially. It uses the query route without filters because those parameters are not declared. That constraint is inconvenient, but guessing a query string would make the example look more capable than the documented contract.

The health probe itself should not retry. A scheduler can run it repeatedly, while the alert rule can require several consecutive failures. Retrying inside one probe blurs a four-second failure into a much longer and harder-to-attribute event. Choose the interval and consecutive-failure threshold from the response-time objective, not from a vendor default.

For scheduled work, add a separate dead-man signal. Each successful rent-reminder run should check in with an external heartbeat service using a stable job identity. Retries must use the same delivery or operation key so that recovery does not send a resident two reminders. Quiet failures are the dangerous ones.

4. Compare products by the missing failure mode

No single product choice removes the endpoint design work. The useful comparison is which evidence and detection path each option adds around it.

Option Best fit here Boundary to plan for
Prometheus Scraping numeric service metrics and applying its established naming model An unreachable target can be observed, but a business task still needs an explicit completion signal
Healthchecks.io Dead-man monitoring for cron and queue jobs that should check in It complements rather than replaces Express liveness, readiness, and dependency logs
Better Stack External uptime checks when a team wants a hosted monitoring path Keep detailed customer-incident evidence in structured application logs
Datadog Teams consolidating service monitoring in an existing observability program Define tag cardinality and cost ownership before sending tenant-shaped dimensions
Infrai Sending logs and metrics from small services through one REST interface Build the alert poller separately; distributed trace queries, source-map processing, and session replay are outside this fit

That is a fair split of responsibilities. The trade-off is operational ownership: a small team might pair Better Stack with structured logs, while a scheduler-heavy system might prioritize Healthchecks.io. An organization already operating Prometheus or Datadog usually gains more from consistent labels and runbooks than from adding another destination. Infrai is not a fit when the team requires built-in alert notifications, distributed trace-tree queries, source-map processing, or session replay; choose a product that owns those workflows instead.

Tool count is not the objective.

5. Turn a red check into a reconstruction runbook

When readiness turns red, capture four times: the external probe failure, the dependency-check failure, the last successful customer job, and the next successful recovery. Then correlate by service, portfolio, job ID, and trace ID. This produces a bounded incident window and an answer to "which properties were affected?" without encoding every property into a metric.

The runbook should ask, in order: Was the process reachable? Which dependency failed? Did queued work stop, retry, or complete? Were retries deduplicated? Which customer operations fall inside the failure window? Reconciliation comes before replay. Replaying first can turn missing work into duplicate deliveries.

There are cases where this design is too small. If one request crosses many services, logs carrying trace and span IDs do not substitute for a queryable distributed trace tree. Browser crashes need source maps or symbolization. A customer-support workflow that depends on session replay needs a different tool. Admit those boundaries early.

6. Ship the smallest system that can falsify "healthy"

Start with two Express routes, one external probe, one job heartbeat, structured failure logs, and a small metric vocabulary. Test Postgres and Redis readiness with strict timeouts. Keep liveness local. Attach customer identifiers to logs and cost-allocation dimensions to metrics.

Then stage three failures: stop Redis, block Postgres, and suppress a scheduled job without stopping its worker. The first two should make readiness fail while liveness stays green; the third should trip the heartbeat even though both HTTP endpoints remain healthy. If the evidence identifies the affected portfolio and prevents duplicate replay, the system is doing useful work.

That is enough to begin. Add tracing, replay, or a broader platform only when an incident question proves the need.

References

Sources

Top comments (0)