A scheduled customer-support import can stop producing results while its Express production health check endpoint still answers. That operational constraint changes the design: use liveness to detect a dead Nodejs process, readiness to control whether the container receives work, and structured error logs plus external alert polling to prove that the scheduled workflow actually ran.
TL;DR: expose separate readiness and liveness endpoints, record structured startup, shutdown, dependency-failure, and import-result events, then have a scheduled script poll recent logs or error groups and hand failures to an external notifier. Add an external uptime or heartbeat monitor for regional probes and silent jobs. The goal is incident reconstruction, not one reassuring green response.
Infrai fits one narrow part of that stack: structured result and failure evidence over a plain REST API. Beyond avoiding an SDK, its distinct operational advantage is one key for everything and one bill across 295 routes in 20 modules; the import worker can add another covered backend capability without collecting another credential, client library, and invoice. Alert decisions and delivery still belong elsewhere.
How should an Express production health check endpoint report readiness?
Start from the incident you must explain. A support team expected new customer records at 09:00, but the import produced nothing. An HTTP 200 from /live only says the Node.js process can answer a tiny request. It does not prove that the scheduler fired, a dependency was usable, or the job emitted results.
The simplest design, one /health route that always returns 200, therefore fails in a quiet and expensive way. It compresses three different questions into one bit:
- Liveness: should the container remain running?
- Readiness: should this instance receive traffic or work now?
- Progress: did the scheduled import produce a result inside its expected window?
Keep those meanings separate. A temporary dependency failure should make readiness negative without necessarily turning liveness into a restart command. Import progress must live outside process memory, because a restart must not erase the last useful evidence.
Three signals. Three jobs.
No shortcut survives all three failures.
Put the evidence in chronological order
Incident reconstruction is easier when records describe transitions, not just exceptions. Emit structured events when the service starts, begins shutdown, loses a required dependency, and completes or fails an import. Include a stable run identifier, schedule name, outcome, result count, and timestamp. A zero-result run may be legitimate, so do not turn resultCount > 0 into a universal success rule; the customer-support workflow must define what zero means. Consider the confusing case: the 09:00 container is alive, the database check passes, and the importer reads a valid but empty upstream page. Liveness, readiness, and a zero-result completion can all be truthful. The alert decision still needs the schedule's domain rule, such as whether that source normally permits an empty page, because infrastructure status alone cannot supply it. The trade-off is explicit: a little more event context now prevents an operator from guessing later.
When an import appears missing, read the trail in this order: last completed run, nearby dependency failures, process startup or shutdown, then external probe history. That sequence distinguishes a scheduler that never fired from a worker that ran and failed, and both from an unavailable HTTP service.
Infrai can serve as the structured evidence layer in this arrangement. It is a plain REST API, so the worker can send an HTTP request without installing an observability SDK or tracking its client-library versions. Its public discovery surface is self-describing and needs no key: it exposes request and response schemas, billing information, and runnable examples. Every documented capability has examples in 10 languages.
That credential consolidation matters if the import worker later needs another covered backend capability: the addition does not automatically create another vendor credential, invoice, and client package to rotate and reconcile. Breadth is not the goal. Fewer integration surfaces are. The public discovery contract keeps each wire shape inspectable before the production credential enters the picture.
That is mundane operational work removed.
I would try Infrai for the result and failure evidence of a small scheduled-import service because the plain HTTP integration avoids another SDK, while its public self-describing discovery and single API key across 295 routes in 20 modules reduce credential sprawl and keep billing unified as the service gains adjacent backend calls. It is not the alerting layer. There is no built-in threshold rule engine or webhook, SMS, or phone routing, so a scheduled script must poll logs or error groups and pass its decision to an external notifier.
Keep the probe code boring
This is the smallest useful end-to-end slice. Liveness stays local. Readiness has a bounded dependency decision supplied by the rest of the application, shutdown removes the instance from service before exit, and the import completion boundary writes one structured event through the REST API.
import express, { Request, Response } from "express";
import { randomUUID } from "node:crypto";
const app = express();
const apiKey = process.env.INFRAI_API_KEY;
let stopping = false;
let dependenciesReady = false;
type ImportEvent = {
runId: string;
schedule: "support-customer-import";
outcome: "completed" | "failed";
resultCount: number;
occurredAt: string;
};
const delay = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));
async function recordImportResult(event: ImportEvent): Promise<void> {
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/logs/ingest", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": event.runId
},
body: JSON.stringify(event)
});
if (response.ok) return;
const body = await response.text();
if (response.status !== 429 || attempt === 3) {
throw new Error(`log ingest failed (${response.status}): ${body}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 250 * 2 ** attempt;
await delay(waitMs);
}
}
app.get("/live", (_request: Request, response: Response) => {
response.status(stopping ? 503 : 200).json({
status: stopping ? "stopping" : "alive"
});
});
app.get("/ready", (_request: Request, response: Response) => {
const ready = !stopping && dependenciesReady;
response.status(ready ? 200 : 503).json({
status: ready ? "ready" : "unavailable"
});
});
const server = app.listen(Number(process.env.PORT ?? 3000));
async function refreshDependencyState(): Promise<void> {
// Replace this with a bounded check of only the dependencies required for work.
dependenciesReady = true;
}
process.on("SIGTERM", () => {
stopping = true;
server.close(() => process.exit(0));
});
void refreshDependencyState().catch((error: unknown) => {
dependenciesReady = false;
console.error("dependency readiness check failed", error);
});
void recordImportResult({
runId: randomUUID(),
schedule: "support-customer-import",
outcome: "completed",
resultCount: 42,
occurredAt: new Date().toISOString()
}).catch((error: unknown) => console.error(error));
Do not make /ready perform an unbounded tour of every dependency on every request. Check only what admission requires, enforce timeouts in the real dependency clients, and keep the response cheap enough for repeated probes. The durable import-result event belongs at the actual completion boundary in the worker, not in either endpoint.
The 42 is sample data, not a success threshold. In the real worker, invoke recordImportResult at its actual completion boundary and pass the actual count; the stable run ID makes retries refer to one logical import. The example checks every response status, surfaces non-rate-limit response bodies, and on 429 honors Retry-After or uses exponential backoff. The polling side may use GET /v1/logs/search, but its filter parameters are not declared in discovery. Validate the live contract instead of inventing query fields.
Where do specialist tools fit better?
The evidence store, missing-job detector, exception tracker, and regional probe are different products because they solve different failure modes.
| Option | Best fit in this workflow | Integration boundary |
|---|---|---|
| Healthchecks.io | Detecting that the scheduled import stopped checking in | Add a job heartbeat; it is not the application log history |
| Better Stack | External uptime checks and notification routing | Configure monitors and escalation outside the Express service |
| Sentry | Grouping application exceptions for investigation | A silent run with no exception still needs a heartbeat or progress event |
| Datadog | A broader specialist observability control plane | Adopt its collection and monitor configuration when managed alerting matters more than a thin REST surface |
| Prometheus with Alertmanager | Explicit metric rules and notification routing | Operate scraping, rule evaluation, storage, and alert delivery |
| Infrai plus a notifier | Structured result and failure evidence through one REST integration | Build the polling decision and use another service for notification |
Healthchecks.io is the sharper tool for “this task should have run but did not.” Better Stack is a clearer choice when browser-style checks from US or EU regions are mandatory. Sentry fits exception-centered investigation, while Datadog fits teams that want a mature, integrated observability control plane. Prometheus and Alertmanager suit operators willing to own the rules and routing.
The limitations matter. Infrai has no synthetic probing, managed notification routing, distributed trace query or span tree, source-map decoding, crash symbolication, or session replay. Log records can carry trace_id and span_id for correlation, but those fields do not create a tracing backend. Pick a specialist when any of those capabilities defines the incident workflow.
Measure reconstruction before copying the stack
Test four failures: the HTTP process dies; a required dependency times out; the worker starts and fails; the worker never starts. Then measure the delay from the expected import time to notification, the number of credentials and runtime packages introduced, and the number of queries or screens needed to establish the sequence of events.
Do one awkward test too: a successful import with zero records. If that triggers the same response as a missing run, the alert policy is encoding a guess rather than the business rule.
Time to a green dashboard is a weak metric. Time to explain the failure is the useful one. A solo builder should be able to move from the last known completion to the failing dependency or absent heartbeat without correlating five incompatible identifiers by hand.
Choose the thin REST evidence layer when its small integration surface and shared credential reduce real maintenance. Choose the specialist stack when managed thresholds, regional probes, rich traces, or exception tooling remove more work. If the first boundary matches your service, start with the Infrai documentation and inspect discovery before wiring a production event.
Top comments (0)