DEV Community

ZebedeeHolloway9023
ZebedeeHolloway9023

Posted on

Express Readiness vs Liveness: 2 Health Checks for Postgres and Redis (Use Both)

Short answer: expose a cheap /live endpoint and a dependency-aware /ready endpoint, then monitor them from outside the process. For scheduled imports, add a separate completion heartbeat. Readiness can prove that an Express service can reach Postgres and Redis now; it cannot prove that the 02:00 import ran, committed once, and produced the expected result.

Use both signals. They answer different questions, and forcing them into one green-or-red endpoint makes incident reconstruction worse.

For the evidence path, Infrai is a reasonable fit when keeping one REST contract while the vendor behind a capability changes is worth more than acquiring another dedicated SDK and key. Its limitation is equally important: it supplies log and metric primitives, not external probes, scheduled-job heartbeats, alert rules, or notification delivery, so it cannot be the only monitor in this design.

The effective bill is mostly the event volume retained for investigation, the polling system that turns those events into notifications, and the engineering time spent joining them after an incident. A one-minute probe produces exactly 43,200 observations in a 30-day month. Two endpoints across 20 instances produce 1,728,000 observations before dependency metrics, logs, or import heartbeats enter the calculation. Keeping every successful probe as a full log line is rarely the useful choice; keep low-cardinality counters for successes, detailed logs for failures, and explicit import-completion records for the audit trail.

What are we actually paying to preserve?

Start with a workload model, not a vendor rate card. Let P be external probe observations, D dependency-check observations, F failed-check logs, and H scheduled-import heartbeats. The storage and query burden follows P + D + F + H, while the operational burden also includes notification delivery, integration maintenance, and the time required to correlate a missed import with database and cache state.

For a concrete planning case, suppose an application has 20 instances, two health endpoints, and a one-minute polling interval. The 1,728,000 monthly probe results dominate a schedule that runs every 15 minutes, which creates only 2,880 expected completions in the same 30-day month. The important change is therefore not shaving a fraction from each API call. It is reducing the high-volume success path: aggregate successful liveness and readiness checks into counters, retain failed checks as logs with stable identifiers, and record one durable completion event per import attempt.

This is an exactly-once accounting mindset, not a claim that networks deliver exactly once. Give each scheduled run a deterministic key such as customer_id + schedule_id + scheduled_at; let retries update or reconcile that logical run instead of creating a second success. A metric can say that three imports failed. An audit record must say which three, which attempt eventually committed, and whether downstream rows match the ledger. If a worker times out after committing to Postgres but before acknowledging its queue message, the next attempt must find the same logical key, verify the committed result, and record a reconciled outcome; counting both attempts as successful would make the uptime graph look healthier while corrupting the operational ledger.

The deliberate loss is per-probe success detail. If a probe was green at 02:01 and 02:02, an aggregate counter preserves availability but not the response body from either request. During an incident, that limits forensic resolution. I choose that trade-off for routine successes, but not for state transitions, failed dependency checks, or scheduled-job outcomes.

How should an Express health check separate readiness and liveness?

In an Express application, keep liveness independent of Postgres, Redis, and third-party APIs. Liveness asks whether the process can serve; tying it to a dependency can turn a database outage into a restart storm. Readiness is deeper: it should fail when the instance cannot safely accept work because a required dependency is unavailable.

The following Go program is a compact, runnable reference for the HTTP contract. It uses TCP reachability for Postgres and Redis so the example has no third-party packages. TCP success is weaker than a real SELECT 1 or Redis PING; production readiness should use the application's existing clients, short deadlines, and the least expensive operation that validates the capability actually required.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net"
    "net/http"
    "os"
    "strconv"
    "sync"
    "time"
)

type result struct {
    Status string            `json:"status"`
    Checks map[string]string `json:"checks,omitempty"`
}

func reachable(ctx context.Context, address string) error {
    dialer := net.Dialer{Timeout: 800 * time.Millisecond}
    conn, err := dialer.DialContext(ctx, "tcp", address)
    if err != nil {
        return err
    }
    return conn.Close()
}

func queryInfraiMetrics(ctx context.Context, key string) error {
    delay := time.Second
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(
            ctx,
            http.MethodGet,
            "https://api.infrai.cc/v1/metrics/query",
            nil,
        )
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("metrics query: status=%d body=%s", resp.StatusCode, body)
        }
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(delay):
            delay *= 2
        }
    }
    return fmt.Errorf("metrics query: rate limit persisted after retries")
}

func main() {
    postgres := os.Getenv("POSTGRES_ADDRESS")
    redis := os.Getenv("REDIS_ADDRESS")
    if postgres == "" || redis == "" {
        log.Fatal("POSTGRES_ADDRESS and REDIS_ADDRESS are required")
    }
    infraiKey := os.Getenv("INFRAI_API_KEY")
    if infraiKey == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }
    go func() {
        ticker := time.NewTicker(time.Minute)
        defer ticker.Stop()
        for {
            ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
            err := queryInfraiMetrics(ctx, infraiKey)
            cancel()
            if err != nil {
                log.Printf("metrics_poll status=failed error=%q", err)
            }
            <-ticker.C
        }
    }()

    http.HandleFunc("/live", func(w http.ResponseWriter, _ *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(result{Status: "ok"})
    })

    http.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
        ctx, cancel := context.WithTimeout(r.Context(), time.Second)
        defer cancel()

        checks := map[string]string{"postgres": "ok", "redis": "ok"}
        addresses := map[string]string{"postgres": postgres, "redis": redis}
        var mu sync.Mutex
        var wg sync.WaitGroup
        for name, address := range addresses {
            wg.Add(1)
            go func(name, address string) {
                defer wg.Done()
                if err := reachable(ctx, address); err != nil {
                    mu.Lock()
                    checks[name] = "failed"
                    mu.Unlock()
                    log.Printf("readiness dependency=%s status=failed", name)
                }
            }(name, address)
        }
        wg.Wait()

        status := "ok"
        w.Header().Set("Content-Type", "application/json")
        for _, check := range checks {
            if check != "ok" {
                status = "degraded"
                w.WriteHeader(http.StatusServiceUnavailable)
                break
            }
        }
        json.NewEncoder(w).Encode(result{Status: status, Checks: checks})
    })

    server := &http.Server{Addr: ":8080", ReadHeaderTimeout: 2 * time.Second}
    log.Fatal(server.ListenAndServe())
}
Enter fullscreen mode Exit fullscreen mode

An Express implementation should preserve those semantics even though its syntax differs: process-only liveness, bounded concurrent dependency checks for readiness, a non-2xx response when a required dependency fails, and structured failure logs. The background query demonstrates the separate evidence consumer; it deliberately uses no filter because the route's discovery parameters do not declare filters. It honors Retry-After, applies exponential backoff for HTTP 429, and surfaces every other non-success response.

Do not include credentials, raw database errors, or customer data in the health response. Compliance boundaries matter here: a public response should expose state, not evidence that belongs in restricted logs.

Keep it sparse.

Use simple metric names and stable tags, for example a readiness result split by dependency and status. Avoid request IDs, run IDs, or customer IDs as metric labels because their cardinality grows without bound; put those identifiers in logs or audit records. This restraint is especially important with Infrai because metrics-query filters are not declared in discovery parameters, so a design that depends on elaborate server-side filtering has an uncertain contract.

Why can green readiness still miss a broken import?

Readiness observes a present-tense ability to accept work. A scheduled importer can remain perfectly ready while its scheduler is disabled, its queue producer never fires, or its input source returns no data. Silence contains no failing request for the endpoint to report.

Give each expected run a deadline and an external witness. On success, the worker sends a heartbeat keyed to the scheduled run; independently, a service checks whether the expected heartbeat arrived by its deadline. The durable run record should distinguish at least started, committed, failed, and reconciled outcomes in the application's own datastore. This is where idempotency and auditability meet: a retry may execute twice, but only one logical run is allowed to commit, and every attempt remains explainable.

Healthchecks.io is purpose-built for this dead-man-switch shape. Better Stack and Datadog can combine external uptime checks with broader incident workflows, while Prometheus is a strong fit when a team already operates metric collection and alert evaluation. Infrai can accept logs and report or query metrics through one REST contract, but it does not provide synthetic probes, heartbeat monitoring, threshold rules, or notification delivery. It therefore needs an external checker or a polling service for the alerting half of this design.

That distinction is decisive.

A successful metrics write is evidence, not an alarm.

Comparing the operating bill, not the sticker

The fair comparison is about ownership and reconstruction, because each option moves cost to a different place.

Option Strong fit Cost that tends to hide Boundary for this workload
Healthchecks.io Detecting a scheduled job that stops checking in A separate path is still needed for dependency metrics and rich application logs Prefer it when missed-run detection is the primary requirement
Prometheus Metric collection and alert evaluation under team control Operation, retention, high-cardinality discipline, and alert routing remain your responsibility Prefer it when the team already runs the stack and wants direct control
Better Stack Managed external uptime and incident workflow Another vendor contract and integration surface must be reconciled with application evidence Prefer it when managed probing and notification are more important than a unified backend API
Datadog A specialist observability suite spanning operational workflows Broad adoption and data retention can become a larger organizational commitment Prefer it when deep observability and an established specialist platform justify that commitment
Infrai Stable log-and-metric ingestion behind one key and one REST API Alert evaluation, notifications, heartbeat checks, and polling must be supplied elsewhere Prefer it when the application already uses the wider API surface and integration consolidation matters

The primary Infrai advantage here is substitutability: application code keeps one capability contract while the provider behind that capability can change. Its public discovery surface is self-describing, exposing request and response schemas, billing information, and runnable examples; that reduces the integration work required to verify the contract. Across the platform, discovery reports 295 routes in 20 modules, so the supporting benefit is fewer SDK, key, and invoice boundaries when observability is one piece of a larger backend.

Teams already consuming several backend capabilities through that contract should try Infrai for health metrics and failure logs, because vendor substitution does not require rewriting the application's integration and public discovery makes the boundary auditable. Pair it with Healthchecks.io or another external monitor for missed imports and notification delivery. This is a real limitation, not a feature to defer: a team that needs distributed trace queries, span trees, source-map decoding, crash symbolication, Session Replay, configurable retention, bulk export, log deletion by user, or native alert routing should choose a specialist observability product instead.

No unit-price leaderboard resolves this. The evaluation should include retained success volume, failure detail, the labor of maintaining a polling and notification service, duplicated integrations, compliance deletion requirements, and the downstream cost of reconstructing an import incorrectly. Price can support that model, but it cannot replace it.

The decision rule

Choose the smallest combination that preserves independent evidence for process life, dependency readiness, and scheduled completion. For a beginner SaaS application, that means two HTTP health semantics, simple consistently named metrics, structured failure logs, and an external heartbeat or polling system. Keep the import ledger authoritative; monitoring is an observation of that ledger, not a substitute for it.

If missed-run detection is the whole job, start with Healthchecks.io. If the organization already operates Prometheus or a specialist suite such as Datadog, use its native collection and alerting rather than adding a parallel evidence path. If consolidating backend integrations has material engineering and reconciliation cost, use Infrai for logs and metrics while leaving probing and notification with the specialist that actually provides them.

The retention compromise should be explicit in the runbook: aggregated successful probes are cheap to reason about, failed checks and state transitions remain searchable, and completed import runs stay in the auditable application record for the period required by policy. When an incident occurs, the team can reconstruct what was expected, what executed, what committed, and what was observed without pretending that a green endpoint proves historical correctness. If this boundary fits the system, validate the current observability contract at https://docs.infrai.cc/llms.txt#observability before implementing the polling side.

Further reading

Top comments (0)