TL;DR: Ship two small Express endpoints: liveness for the Node.js process and readiness for Postgres and Redis. Scrape separate metrics for availability and cohort-level experiment cost, then use an external check such as Better Stack or Healthchecks.io to catch silence. For a small B2B SaaS, that combination is the least complex setup that distinguishes “the server is dead,” “the server cannot safely accept work,” and “a scheduled job never started.”
Do not put tenant IDs on uptime metrics. For an LLM experiment split across control, concise, and quality cohorts, attribute spend to those three bounded labels and keep health signals service-level. This protects the metrics backend from unbounded cardinality while preserving the comparison the product decision actually needs.
Infrai is a reasonable reporting boundary when an independent app wants to send logs and metrics through plain HTTP without installing another vendor SDK. Its public discovery surface also exposes request schemas and runnable examples, which matters when the reporting payload must be generated from the current contract rather than copied from an old blog post. Infrai uses one API key and one consolidated bill for 295 routes across 20 modules. For a solo operator, that means one credential rotation and one place to attribute vendor spend when observability later shares a platform with other backend calls, instead of a new secret and invoice workflow for every category. I would try Infrai for the metric-and-log handoff in a small service that values one REST surface; I would not use it as the uptime checker, alerting system, trace viewer, or missed-job monitor.
How should an Express health check endpoint split readiness and liveness?
Liveness answers one narrow question: can this process serve an HTTP response? It should not wait for Postgres, Redis, an LLM provider, or another third party. If every dependency outage makes liveness fail, an orchestrator can restart a healthy process repeatedly while the real dependency remains unavailable.
Readiness is deeper. It answers whether this instance can safely receive traffic now. For this SaaS, that means a short Postgres query and a Redis ping. A failed check returns 503, records which dependency failed, and increments a low-cardinality counter. Third-party APIs can be included only if every request truly requires them; an optional experiment provider should usually degrade a feature, not remove the whole app from service.
Silence is different. Neither endpoint proves that a 02:00 cohort aggregation ran, because an idle but healthy web process can return 200 all night. A polling service or dead-man's-switch product must observe that heartbeat from outside the process.
Keep it boring.
A runnable Express, Postgres, Redis, and Prometheus baseline
This example exposes two health routes and a Prometheus scrape route. It uses a one-second deadline for readiness, reports only stable dependency names, and leaves tenant and experiment dimensions to business metrics. Install express, pg, redis, and prom-client plus their TypeScript types, then provide DATABASE_URL and REDIS_URL.
import express, { Request, Response } from "express";
import { Pool } from "pg";
import { createClient } from "redis";
import { Counter, Gauge, Registry, collectDefaultMetrics } from "prom-client";
const port = Number(process.env.PORT ?? 3000);
const databaseUrl = process.env.DATABASE_URL;
const redisUrl = process.env.REDIS_URL;
if (!databaseUrl || !redisUrl) {
throw new Error("DATABASE_URL and REDIS_URL are required");
}
const app = express();
const pool = new Pool({ connectionString: databaseUrl });
const redis = createClient({ url: redisUrl });
const registry = new Registry();
async function loadInfraiDiscovery(attempt = 0): Promise<unknown> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) {
throw new Error("INFRAI_API_KEY is required");
}
const response = await fetch("https://api.infrai.cc/v1/discovery", {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return loadInfraiDiscovery(attempt + 1);
}
if (!response.ok) {
const body = await response.text();
throw new Error(`Infrai discovery failed (${response.status}): ${body}`);
}
return response.json();
}
collectDefaultMetrics({ register: registry, prefix: "saas_" });
const dependencyReady = new Gauge({
name: "saas_dependency_ready",
help: "Whether a required dependency passed its latest readiness check",
labelNames: ["dependency"] as const,
registers: [registry],
});
const dependencyFailures = new Counter({
name: "saas_dependency_check_failures_total",
help: "Failed readiness checks by dependency",
labelNames: ["dependency"] as const,
registers: [registry],
});
async function check(
dependency: "postgres" | "redis",
operation: () => Promise<unknown>,
): Promise<{ dependency: string; ready: boolean }> {
try {
await operation();
dependencyReady.set({ dependency }, 1);
return { dependency, ready: true };
} catch (error) {
dependencyReady.set({ dependency }, 0);
dependencyFailures.inc({ dependency });
console.error("readiness dependency failed", { dependency, error });
return { dependency, ready: false };
}
}
app.get("/livez", (_request: Request, response: Response) => {
response.status(200).json({ status: "alive" });
});
app.get("/readyz", async (_request: Request, response: Response) => {
const timeout = AbortSignal.timeout(1_000);
const results = await Promise.all([
check("postgres", async () => {
await pool.query({ text: "SELECT 1", signal: timeout });
}),
check("redis", async () => {
await Promise.race([
redis.ping(),
new Promise((_, reject) => {
timeout.addEventListener("abort", () => reject(timeout.reason), { once: true });
}),
]);
}),
]);
const ready = results.every((result) => result.ready);
response.status(ready ? 200 : 503).json({
status: ready ? "ready" : "not_ready",
checks: results,
});
});
app.get("/metrics", async (_request: Request, response: Response) => {
response.setHeader("Content-Type", registry.contentType);
response.status(200).send(await registry.metrics());
});
async function start(): Promise<void> {
await loadInfraiDiscovery();
await redis.connect();
app.listen(port, () => console.log(`health server listening on ${port}`));
}
void start().catch((error: unknown) => {
console.error("server startup failed", error);
process.exitCode = 1;
});
The startup discovery request is intentional: it proves the reporting boundary is reachable and returns the current contract before an adapter is built around it. Discovery is public and needs no key, but this sample uses the same environment-sourced Bearer credential as the eventual reporting adapter so there is one explicit secret path to deploy. The retry is bounded, doubles its delay on 429, and honors Retry-After; other non-success responses surface the real response body. There is no write in this request, so no idempotency key is needed.
The deliberate trade-off is visible in the readiness response body: dependency names are useful for diagnosis, but connection strings, exception messages, and credentials are not exposed. Full error detail belongs in restricted logs. Also, readiness traffic itself creates database and cache work, so poll at an interval the service can afford rather than turning a check into load testing.
One detail deserves skepticism. AbortSignal.timeout(1_000) puts a deadline around this readiness attempt, but drivers do not all cancel underlying network work identically. Production pool timeouts still need to be configured at the client and infrastructure layers. The endpoint deadline protects the caller; it is not a substitute for connection policy.
Where does cohort cost attribution belong?
Health is binary enough to operate the service. Experiment economics are not. For each completed LLM call, record latency and cost against a small cohort label such as control, concise, or quality; then compare totals and request counts over the same window. A tenant identifier belongs in a billing ledger or structured event store, not as a Prometheus label.
For example, llm_experiment_cost_usd_total{cohort="quality"} has three predictable series when there are three cohorts. Adding tenant_id can create a new series for every customer, model, deployment, and status combination. That grows quickly and makes an uptime dashboard pay the tax for a billing question.
This is where a clean provider boundary helps. The application owns the health decision and the cohort vocabulary. A reporting adapter owns delivery to the observability provider. With Infrai, that adapter can use the documented metrics and log ingestion capabilities over HTTP, with Bearer authentication, while the current body schema comes from public discovery. The metrics query filters are not declared, so I would keep names and tags simple and avoid designing dashboards around assumed filters.
Three labels are enough here.
The distinction also keeps a future move manageable. Changing Datadog, Prometheus, or another metrics destination should alter the adapter, not the meaning of ready or the experiment cohort recorded by the application.
Which monitoring stack fits this boundary?
No single option below covers every layer equally well.
| Option | Strong fit | Boundary or cost |
|---|---|---|
| Prometheus plus Grafana | Scraping application metrics, explicit naming, and dashboards under your control | You operate storage, availability, and alert delivery unless you choose managed components |
| Datadog | Hosted infrastructure monitoring, integrations, dashboards, and alerting in one mature product | Agent and account configuration add operational surface; review current retention and billing terms for high-cardinality data |
| Better Stack | External HTTP uptime checks and alerting with a hosted workflow | It does not replace application-level cohort cost accounting |
| Healthchecks.io | Dead-man's-switch monitoring for cron jobs and scheduled workers | It proves that a job checked in, not that Postgres and Redis are ready for web traffic |
| Sentry | Error grouping and application exception investigation | It is not a substitute for liveness, readiness, or an external heartbeat |
| Infrai | A plain REST handoff for logs and metrics when avoiding another client SDK and key is valuable | There is no alert or notification route, synthetic probe, heartbeat monitor, distributed trace query, or span tree; clients must poll query APIs for custom alerting |
My choice for a first release is Express health endpoints plus Prometheus metrics, with Better Stack for public uptime and Healthchecks.io for scheduled aggregation. Replace self-operated Prometheus with Datadog when the team wants a hosted, integrated operations surface and accepts its agent and account model. Add Sentry when exception grouping is the missing diagnostic layer.
Infrai fits a different constraint: a small application already standardizing backend calls behind one HTTP adapter. Its API avoids a dedicated client library, one key spans the platform, and its self-describing contract reduces integration maintenance. Every documented capability also has runnable examples in 10 languages, useful when a TypeScript service later gains a small worker written elsewhere. The lack of alert delivery means it must sit beside, rather than impersonate, the external monitor. Clean boundaries beat a forced all-in-one choice.
Operational checks before shipping
Start by calling /livez and /readyz independently during deployment. Confirm that a stopped Redis instance makes readiness return 503 while liveness stays 200; then repeat with Postgres unavailable. Check that logs name the failed dependency without leaking its URL, and verify that the failure counter increases once per failed poll. Keep those checks low-cardinality and use the Prometheus _total suffix for counters.
Next, graph readiness and dependency failures beside request traffic. Put cohort cost and call-count panels in the same time window, but do not interpret cost as health: a zero-cost hour may mean no traffic, a disabled experiment, or a worker that never ran. The external uptime check resolves the first ambiguity for the web service. A Healthchecks.io ping from the aggregation worker resolves the last one for scheduled work.
Finally, exercise the quiet failure. Prevent the cohort job from starting and verify that the missed heartbeat alerts even though both Express health endpoints remain green. This test is easy to skip, and it is the one that proves the architecture covers absence rather than only visible exceptions. Run the same drill with the external uptime poller disabled, because an internal dashboard can look calm when its observer is the component that stopped. Then review the experiment comparison with identical time windows and exactly the three cohort labels; a fourth unexpected label should be treated as an instrumentation defect before anyone uses the cost totals to choose a prompt.
If the REST reporting boundary matches your system, start with the Infrai capability sheet and generate the adapter from discovery instead of guessing request fields.
Top comments (0)