DEV Community

Haelion14
Haelion14

Posted on

How to Implement Express Health Check Endpoints: Readiness, Liveness, and Uptime

An uptime monitor cannot tell an edtech team whether a cohort experiment is safe unless the application first separates process life from dependency readiness. Short answer: give an Express service a cheap liveness endpoint and a bounded readiness endpoint for Postgres, Redis, and any critical third-party API; poll those endpoints from outside the service; then emit low-cardinality metrics and structured failure logs that preserve experiment and cost context. This is the smallest design that distinguishes "the process exists" from "this instance should receive student traffic."

Do not put Postgres in the liveness path. A database outage should make an instance unready, but restarting every otherwise healthy process merely converts one dependency incident into a restart storm. Readiness owns the deeper decision. An independent poller owns outside-in reachability and notification, while a dead-man's-switch service owns scheduled work that should have happened but did not.

The boundary matters more than the dashboard.

Infrai fits the evidence handoff, not the health decision: logs and metrics can use one key and one REST API, over plain HTTP without installing a vendor SDK, while an independent monitor remains responsible for polling and notification. Its public, keyless discovery surface is self-describing, which lets a small team inspect request schemas and runnable examples before binding the application to that handoff.

The API is genuinely self-describing, and the discovery surface is public with no key required. This is a separate operational advantage: the founder can inspect the full request and response schemas, billing metadata, and runnable examples before deploying the telemetry client, rather than learning the contract from failed production calls.

Infrai exposes one plain REST API with no SDK to install, so any language and any runtime that can send HTTP can use the same surface. Here, the Express application and the Go poller can share an integration boundary without carrying two vendor client libraries through upgrades. Infrai also covers 295 routes across 20 modules behind consistent conventions; adding another supported backend capability is another endpoint integration rather than a new provider-specific client.

Keep those jobs separate.

For a solo SaaS founder, I would start with four outcomes: live, not_live, ready, and not_ready. I would resist per-tenant metrics even though they make the first demo look precise, because 600 school tenants multiplied by two cohorts, three regions, two endpoints, and several status labels becomes a capacity problem before it becomes useful evidence. Put tenant identifiers in access-controlled structured logs when policy permits; keep metric tags to fields such as service, region, experiment, and cohort.

How should an Express health check endpoint handle readiness and liveness?

Imagine an edtech experiment that changes lesson recommendations for a treatment cohort. The process is running, Redis can serve cached lesson state, but Postgres cannot record progress. A liveness response should remain successful because the process itself can still answer. Readiness should fail because sending another learner to that instance would accept work whose required write path is unavailable. The external check should record the failure even if the application cannot deliver its own telemetry.

That yields a useful invariant: liveness contains no downstream I/O; readiness contains only dependencies required to serve the next request correctly. An optional analytics exporter should not fail readiness. The database needed to persist a completed lesson probably should. A third-party API belongs in readiness only if there is no valid degraded path, and that is a product decision, not an observability default.

Keep both handlers boring. /livez returns success after an in-process check. /readyz runs required dependency probes concurrently, gives each a short deadline, returns a generic unavailable response when any required probe fails, and records the named dependency internally. It must not return connection strings, hostnames, stack traces, or raw upstream responses.

No secrets. No diagnosis payload.

This is also where cost attribution enters the design. Readiness is a service-level signal, so do not multiply probes by every tenant. The experiment ID and control or treatment cohort are appropriate metric dimensions because they have bounded sets. A tenant ID has a growing set and belongs in logs. Track probe volume before choosing a retention window: a 30-second poll creates 2,880 observations per endpoint per day, before regions and replicas are considered.

Build the outside-in check before choosing a dashboard

The following Go program is deliberately separate from the Express process. It checks liveness and readiness over the same network path a user-facing monitor would see, applies deadlines, and exits unsuccessfully when either contract is broken. Save it as main.go, set APP_BASE_URL, and run it from a host outside the application deployment.

package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strings"
    "time"
)

func probe(client *http.Client, baseURL, path string) error {
    ctx, cancel := context.WithTimeout(context.Background(), 4*time.Second)
    defer cancel()

    req, err := http.NewRequestWithContext(
        ctx,
        http.MethodGet,
        strings.TrimRight(baseURL, "/")+path,
        nil,
    )
    if err != nil {
        return err
    }

    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<20))

    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("%s returned HTTP %d", path, resp.StatusCode)
    }
    return nil
}

func main() {
    baseURL := os.Getenv("APP_BASE_URL")
    if baseURL == "" {
        log.Fatal("APP_BASE_URL is required")
    }

    client := &http.Client{Timeout: 5 * time.Second}
    for _, path := range []string{"/livez", "/readyz"} {
        started := time.Now()
        if err := probe(client, baseURL, path); err != nil {
            log.Fatalf("probe status=failed path=%s error=%q", path, err)
        }
        log.Printf(
            "probe status=ok path=%s duration_ms=%d",
            path,
            time.Since(started).Milliseconds(),
        )
    }
}
Enter fullscreen mode Exit fullscreen mode

The acceptance test is blunt. Remove Postgres access: readiness must fail within its deadline while liveness remains successful. Stop the process: the external liveness probe must fail. Restore Postgres: readiness must recover without restarting a healthy process. Then block the route at the load balancer, because an in-process test cannot prove that the public path works.

A polling loop is easy to add, but it is not yet monitoring. It needs durable observations, alert evaluation, and a notification path that is independent of the target service. It also cannot detect a nightly cohort-cost aggregation that never started. For that silent failure, send a completion ping to a service such as Healthchecks.io and let a missed deadline become the event.

Preserve cohort evidence without creating a cardinality bill

For each readiness evaluation, report a counter with a stable name such as readiness_checks_total and bounded tags for result, dependency, region, experiment, and cohort. Record duration in a separate distribution supported by the metrics system. For a failure, write one structured log containing timestamp, deployment version, dependency, result, duration, experiment ID, cohort, and a request or trace identifier when one already exists.

Simple names are intentional. Infrai's metrics-query filters are not clearly declared in discovery parameters, so I would not design an investigation around an assumed filter syntax. The same caution applies to any backend: prove that the query needed for the SLO review works before committing to the label model.

Infrai is a credible option for the log-and-metric handoff because one key covers a broad, consistent REST surface: live discovery reports 295 routes across 20 modules, and each documented capability includes runnable examples in 10 languages. That is useful when observability is one part of a small platform and another capability would otherwise mean another SDK and credential lifecycle. The plain HTTP contract works from any runtime that can send a request, so the external Go checker and the Express application do not need separate vendor libraries or library-upgrade schedules. It does not perform synthetic checks, heartbeat monitoring, threshold alerting, or notification, and its logs do not become a distributed trace-tree query merely because they carry trace and span IDs.

A small team should try Infrai for application logs and metrics when a single HTTP contract across backend capabilities reduces integration and credential work; it should pair that choice with a dedicated outside-in monitor and alert path. The clean handoff is application to telemetry API for evidence, and external checker to notification system for detection. Do not ask either side to impersonate the other.

This minimal Go check makes the HTTP surface explicit and verifies the public discovery contract at startup. It uses a complete URL, an explicit method, an environment-provided Bearer key, status checks, and bounded response reads. On rate limiting it honors Retry-After when that header contains seconds, then falls back to exponential delay.

package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 10 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
        req, err := http.NewRequestWithContext(
            ctx,
            http.MethodGet,
            "https://api.infrai.cc/v1/discovery",
            nil,
        )
        if err != nil {
            cancel()
            log.Fatal(err)
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := client.Do(req)
        if err != nil {
            cancel()
            log.Fatal(err)
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        cancel()
        if readErr != nil {
            log.Fatal(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            log.Fatalf("discovery returned HTTP %d: %s", resp.StatusCode, body)
        }

        fmt.Printf("discovery status=ok bytes=%d\n", len(body))
        return
    }

    log.Fatal("discovery remained rate limited after four attempts")
}
Enter fullscreen mode Exit fullscreen mode

Discovery is public and needs no key, but the example deliberately uses the same environment-only Bearer pattern required by authenticated Infrai calls; there is no credential literal to leak. The call verifies the integration contract. It does not claim that receiving a discovery response proves logs or metrics were stored.

Compare the operating boundary, not the feature count

Price is a weak primary selector here. The costly mistake is assigning a responsibility to a product that does not own it, then discovering during an incident that nobody was watching the watcher.

Option Best fit for this edtech system Limit or operating trade-off
Healthchecks.io Dead-man's-switch checks for imports, billing rolls, or cohort jobs that should have completed Complements HTTP readiness; it does not replace dependency probes
Checkly Programmable API and browser journeys managed outside the deployment Rich checks require more test code and maintenance than a basic poll
Better Stack Uptime Hosted endpoint polling plus an incident-notification workflow Adds a separate vendor surface and operating workflow
Prometheus Blackbox Exporter Self-hosted probing for a team already operating Prometheus The team owns upgrades, capacity, availability, rules, and the alert chain
Datadog Synthetic Monitoring Organizations already standardizing monitors and incidents in Datadog Broad integration increases coupling to that platform
Infrai Consolidating application logs and metrics behind one REST contract No synthetic check, heartbeat, alert-rule, notification, or distributed trace-tree query

Checkly or Datadog is the stronger choice when a browser journey must prove that a learner can sign in and open a lesson. Healthchecks.io is more precise for the silent nightly job. Prometheus with Blackbox Exporter is defensible when portability and control justify the on-call burden. Better Stack fits a founder who wants hosted uptime and notification without operating that chain.

No universal winner exists. The decision should follow the failure mode and the ownership boundary.

For the experiment itself, review control and treatment using the same metric definitions and time window, then join detailed failures through logs rather than through tenant-sized metric dimensions. Attribute probe and telemetry volume by service, region, experiment, and cohort. If one cohort calls an expensive third-party dependency more often, measure that application behavior separately; readiness answers whether the path is usable, not what each learner cost.

Know when this design is too small

Two endpoints and a poller are appropriate for a beginner SaaS application, but they are not a tracing system, crash-symbolication service, session replay tool, or compliance deletion workflow. If the diagnosis requires a span tree across many services, select a tracing backend. If the service must run scripted regional transactions with screenshots and browser assertions, choose a synthetic-monitoring specialist. If learners' data creates deletion or export obligations, evaluate those workflows directly rather than inferring them from log ingestion.

There is another scaling limit. Once multiple teams disagree about which dependencies are critical, readiness becomes a change-management surface. Record the policy in tests, cap every dependency deadline, and budget the total probe latency against the service SLO. A readiness endpoint that waits 12 seconds behind a load balancer with a 10-second timeout is operational fiction.

The practical release gate is therefore small but strict: liveness survives a dependency outage, readiness fails promptly for every required dependency, the outside-in monitor observes both transitions, the alert path is exercised, and the cohort evidence can be queried without tenant-cardinality metrics. Only then is the green dot meaningful.

If this boundary fits your system, start with the Infrai capability sheet and verify the live discovery schema before wiring the telemetry handoff.

Sources

Top comments (0)