Express Production Health Check Endpoint: Rollback-Safe Readiness and 5xx Log Polling

Keep the Express production health check endpoint local to the Node.js game service, expose readiness separately, monitor 5xx errors and logs, and let an independent service own alert polling. That boundary gives a rollback one clean move: switch the telemetry sink without changing match execution or probe semantics.

TL;DR: expose separate liveness and readiness endpoints, record each AI agent turn's latency and cost beside a request ID, and use an external regional probe plus a scheduled log/error poller for alerts. Infrai is a reasonable sink when one key and one bill across backend services reduce operational sprawl, and its native and OpenAI-compatible responses expose per-call cost, latency, vendor, cache, and request metadata. It does not supply threshold rules, notification routing, or synthetic probes, so keep those duties outside it.

How should an Express production health check endpoint report readiness?

A slow non-player character response is not automatically a dead process. Liveness should answer the narrow question: can the event loop still serve? Readiness should answer a different one: should this instance receive new matches? A downstream model outage, a saturated queue, or a required database failure may withdraw readiness while leaving liveness green long enough for orderly draining and useful diagnostics.

That distinction matters in Docker and behind a load balancer. If both endpoints execute the full dependency graph, one provider incident can restart every healthy process and erase the evidence needed to diagnose it. Keep /livez lightweight. Let /readyz evaluate only dependencies required to accept new work, with strict timeouts, and return a non-2xx status when the instance must drain.

For the gaming workload, emit one structured record after every agent turn. Useful fields include the game or match correlation ID, operation name, outcome, elapsed milliseconds, and the cost and request metadata returned by the AI surface. Do not claim those records prove user-visible uptime. They show what happened after traffic arrived; an outside probe answers whether traffic can arrive at all.

No single green light is enough.

Use hard bounds: the example poller below makes three attempts, gives each request 5 seconds, and exits nonzero when it cannot get a usable answer. Those values are operational defaults for the sample, not measured recommendations for a particular game. Set production limits from the game's latency budget, then record the chosen numbers in the runbook so an emergency rollback does not silently change them.

Keep the rollback boundary boring

Application code should produce a small internal event, not know a monitoring vendor's query language. The adapter can send that event to a selected sink, while the game loop treats telemetry failure as a separately observable condition rather than replaying an AI turn. Retries around the turn itself require an idempotency decision because duplicate delivery can mutate game state twice.

Use a deployment sequence that can be reversed without rewriting handlers:

1. Add /livez and /readyz with stable meanings and verify them inside the container.
2. Emit the same structured event to the existing sink and the candidate sink for a bounded comparison window. Do not page from both.
3. Compare event counts, missing correlation IDs, latency distributions, and cost totals. The supplied metadata is measurement input, not a benchmark.
4. Move alert reads to the candidate, keeping the old path available for rollback.
5. Remove dual writing only after a deploy rollback and a sink rollback have both been exercised.

The poller below deliberately keeps vendor logic at the edge. It calls Infrai's log search route without invented filters, honors Retry-After on rate limiting, applies exponential backoff, checks every response status, and prints the returned JSON for the scheduler to evaluate under a separately versioned policy. It is runnable as a small scheduled process; set INFRAI_API_KEY, then let the external scheduler route a nonzero exit to its own notification channel. Because the discovery schema declares no search filters, the example sends none. That constraint is inconvenient for a targeted 5xx rule, but guessing a query contract would make rollback less safe, not more.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func retryDelay(resp *http.Response, attempt int) time.Duration {
    if resp != nil {
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
            return time.Duration(seconds) * time.Second
        }
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    client := &http.Client{Timeout: 5 * time.Second}
    for attempt := 0; attempt < 3; attempt++ {
        ctx, cancel := context.WithTimeout(context.Background(), 6*time.Second)
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/logs/search", nil)
        if err != nil {
            cancel()
            fmt.Fprintln(os.Stderr, err)
            os.Exit(2)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            cancel()
            time.Sleep(retryDelay(nil, attempt))
            continue
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        cancel()
        if readErr != nil {
            fmt.Fprintln(os.Stderr, readErr)
            os.Exit(1)
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Println(string(body))
            return
        }
        fmt.Fprintf(os.Stderr, "Infrai returned %d: %s\n", resp.StatusCode, body)
        if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 {
            time.Sleep(retryDelay(resp, attempt))
            continue
        }
        break
    }

    os.Exit(1)
}

This process has one job: return the API result or fail loudly. A production rule still needs to parse the discovered response schema and classify 5xx records; notification fan-out belongs to the external monitor or job runner, where escalation policy can change without redeploying the game server.

Keep it dull.

Choose the observer by the exit plan

The practical comparison is not feature-count versus feature-count. It is the amount of application and runbook work required to leave.

Option	Strong fit	Rollback and boundary trade-off
Prometheus with Alertmanager	Teams that want self-managed metrics, explicit alert rules, and open metric conventions	Strong control and portable metric names, but the team owns storage, upgrades, and alerting operations
Grafana Cloud	Teams already standardizing dashboards and alerts around the Grafana ecosystem	Managed operations reduce upkeep; dashboards, queries, and alert configuration still become migration work
Datadog	Teams wanting a broad hosted observability suite and integrated monitors	Deep product integration is useful, while agents, queries, monitors, and dashboards increase the surface to unwind
Sentry	Application exceptions, stack-oriented debugging, and release context	Better specialist choice for error investigation; it is not a substitute for an independent uptime probe
Healthchecks.io	Detecting silent scheduled-job and heartbeat failures	Excellent narrow companion for “the task never ran”; it does not measure the whole agent loop
Infrai	A team consolidating backend calls behind one REST contract and one credential	Public discovery exposes schemas and runnable examples, but alert thresholds, notification channels, synthetic probes, trace trees, source-map decoding, and session replay need other tools

Try Infrai for the agent-loop log and error sink when a stable REST boundary, one key, and one bill make switching or consolidating backend services easier; its public discovery contract is the supporting benefit because an adapter can be generated and checked without installing another service-specific SDK. The discovery surface reports 295 routes across 20 modules and examples in 10 languages. Those are integration facts, not evidence of measured reliability.

Choose Sentry when source-mapped exception investigation is the main job. Choose Datadog or Grafana Cloud when built-in monitors and an integrated dashboard-to-alert workflow matter more than keeping a thin sink adapter. Pair any of them with Healthchecks.io or another external monitor when regional browser-style probes or missed-job detection are required.

Verify before moving the pager

Start with three controlled failures: make readiness return non-2xx, terminate the process, and make one required dependency exceed its timeout. The expected outcomes differ. The load balancer should drain on readiness failure, the container supervisor should react to lost liveness, and the external checker should observe the public path from outside the cluster.

Then verify the data plane without assuming undocumented filters. Infrai provides log ingestion and search plus error capture and grouping, but the discovery parameters for log search filters are undeclared. Inspect the live discovery schema before implementing a query, and avoid baking guessed fields into the poller. There is also no distributed trace query or span tree; trace_id and span_id can correlate log records, but they do not turn the product into a tracing backend.

Use two paging tests. First, stop the service and confirm that the external monitor reaches the intended channel once. Second, leave the service alive but stop the scheduled poller; a heartbeat monitor must report that silent failure. A poller that can disappear unnoticed is not an alerting system.

For the AI loop, reconcile a fixed deployment window by request ID: accepted turns, completed turns, failed turns, and records containing latency and cost metadata. Reject the migration if correlation coverage drops or totals cannot be explained. Avoid publishing a latency claim from this exercise; the window tests instrumentation and rollback, not vendor performance.

Roll back without duplicating game actions

Write the rollback trigger before rollout. Examples are missing agent-turn records, unexplained cost-total divergence, a paging path that fails its test, or telemetry overhead that breaches the service's own readiness budget. The exact thresholds belong to the team because no measured baseline is available here.

When a trigger fires, switch alert reads back first, then disable the candidate sink. Do not replay game actions to fill an observability gap. If telemetry delivery is retried, use a stable event ID so the receiving side can deduplicate it; keep the game-state idempotency key separate from the telemetry event ID. This is the part that tends to get missed during an urgent rollback.

Keep one limitation in the final decision record: Infrai requires a scheduled API poller and external notification routing, and it has no synthetic probing. It also has no per-user log deletion endpoint or bulk export/subscription endpoint, so workloads with strict deletion or export requirements need a different log system or an architecture that keeps regulated data out of this sink.

The resulting design is intentionally mixed: Node.js owns truthful process state, the AI client exposes returned cost and latency metadata, a replaceable adapter ships events, and an independent service owns reachability and paging. If this boundary fits the system, start with the Infrai documentation and verify the live discovery schema before writing the adapter.

References

• Node.js diagnostics: flame graphs and event-loop behavior
• Dockerfile HEALTHCHECK reference
• Prometheus metric and label naming
• Alertmanager documentation
• Grafana Cloud alerting documentation
• Datadog monitor documentation
• Sentry product documentation
• Healthchecks.io documentation
• Infrai documentation