| Choice | Detects a dead API | Detects a missed scheduled run | Reconstructs one agent loop | Operational burden |
|---|---|---|---|---|
| Status check only | Yes | No | No | Low |
| Status check plus heartbeat | Yes | Yes | Partly | Low |
| Correlated logs, metrics, and traces plus both checks | Yes | Yes | Yes | Medium |
TL;DR: use the third design for a fintech agent that makes consequential decisions. Keep the status endpoint and cron heartbeat deliberately boring, then correlate them with one execution ID carried through traces, structured logs, and metrics. This is the least complex option that can answer both operational questions: is work happening? and what happened during this specific loop?
The recommendation is not to collect everything. Record enough stable evidence to replay the sequence of an incident: trigger, dependency calls, model attempt, tool calls, outcome, duration, and cost inputs. Sampling and retention can control volume after that boundary is clear. A green endpoint alone cannot establish that a scheduled agent ran, and a heartbeat alone cannot explain a slow or expensive run.
For a solo SaaS, this is a revenue-per-hour decision. A signal earns its place when it shortens a real investigation or prevents a consequential silent miss. Ship the first correlated slice in a week. Outsource storage and paging if they are undifferentiated, but keep signal names and correlation fields portable.
How Should Node.js API Uptime Health Monitoring Report Status?
The first criterion is incident reconstruction. Start with the question an operator will ask under pressure: "Why did transfer-review run exec_7f3a finish late, and which external step consumed the time?" Every signal should help order that execution's events or rule out a branch. An execution_id shared by the scheduler event, spans, logs, and final outcome gives the investigation a spine. A request ID is insufficient when one incoming request can start several background steps. In Node.js, the status handler should stay separate from the scheduled job so an API uptime check does not masquerade as cron health.
The second criterion is failure independence. The probe, heartbeat receiver, and workload should not all depend on the same process. If the process that performs the job is also the only observer declaring it healthy, one crash can remove both the work and the evidence. An external status probe tests reachability. A heartbeat records completion of a particular schedule window. Neither substitutes for the other.
This separation also changes alert meaning. Page on a symptom tied to user or financial risk: repeated failed probes, a missing required execution, or a sustained latency breach. Keep high-cardinality details such as execution_id in logs and traces rather than metric labels. Prometheus naming guidance recommends a consistent application prefix, base units, and labels whose dimensions remain meaningful across the metric. That points toward a counter such as agent_executions_total and a duration histogram in seconds, split by bounded labels like outcome and workflow, not by customer or execution.
One rule is blunt.
If an alert cannot name the missing or degraded obligation, it is probably too vague to page a one-person team.
A small implementation that leaves an evidence trail
The endpoint should report process readiness, not perform a deep dependency tour on every poll. Deep healthchecks can amplify a dependency slowdown and make the monitor part of the incident. Return a compact state and the build identifier; expose no secrets, account details, prompts, or financial records. A shallow status endpoint is an alternative to probing every dependency, not an alternative to dependency telemetry.
import express from "express";
const app = express();
let ready = false;
app.get("/status", (_request, response) => {
response.status(ready ? 200 : 503).json({
status: ready ? "ready" : "not_ready",
build: process.env.BUILD_SHA ?? "unknown",
});
});
async function start(): Promise<void> {
// Initialize required local resources before accepting work.
ready = true;
app.listen(3000);
}
void start();
The agent loop needs a separate record. The following generic interfaces keep transport and telemetry backends outside the business logic. The example uses a proposed 90-second service objective and a per-run budget chosen by the application owner; those are policy values, not universal thresholds. Cost is calculated from the pricing snapshot supplied to the function, so a later investigation can use the same inputs rather than today's price.
type Outcome = "approved" | "review" | "failed";
type RunEvidence = {
executionId: string;
workflow: "transfer_review";
startedAt: string;
finishedAt: string;
durationMs: number;
inputTokens: number;
outputTokens: number;
costMicros: number;
outcome: Outcome;
model: string;
pricingVersion: string;
};
interface EvidenceSink {
write(record: RunEvidence): Promise<void>;
}
interface HeartbeatSink {
complete(input: { executionId: string; completedAt: string }): Promise<void>;
}
async function recordCompletedRun(
evidence: RunEvidence,
evidenceSink: EvidenceSink,
heartbeatSink: HeartbeatSink,
): Promise<void> {
await evidenceSink.write(evidence);
await heartbeatSink.complete({
executionId: evidence.executionId,
completedAt: evidence.finishedAt,
});
}
Order matters here. Writing completion evidence before emitting the heartbeat prevents a successful heartbeat from getting ahead of the durable incident record. The receiver must treat repeated completion messages for the same execution as idempotent because retries happen. It should also reject an execution ID that belongs to the wrong workflow rather than silently merging unrelated runs.
Do not log raw prompts or tool payloads by default in a financial workflow. Store bounded metadata and references to access-controlled records. Redaction must occur before export; deleting sensitive fields downstream leaves extra copies in queues and collectors. This costs a little implementation time now and avoids turning observability into another sensitive-data system.
How should sampling preserve the expensive failure?
OpenTelemetry distinguishes head sampling, decided when a trace begins, from tail sampling, decided after all or part of the trace is available. That distinction matters for agent loops. A head sampler cannot yet know that the last tool call will fail or that total latency will cross the service objective. Tail sampling can use the completed trace's outcome and latency, but it requires a collector path capable of buffering and making that later decision.
For a small service, start with a written retention rule rather than a clever probability. Keep every failed run and every run that breaches the chosen latency or cost policy. Sample ordinary successful runs. Preserve the structured completion record for every financially relevant execution even when its detailed trace is not retained.
This creates two layers of evidence. The completion record supports counts, reconciliation, and cost review. The sampled trace explains the internal sequence. Logs fill in discrete decisions, but only with stable event names and the same execution ID.
Nothing else can infer the missing timeline.
The cost metric also needs care. Export aggregated cost in a stable base unit selected for the system, and document it in the metric help text. Keep mutable pricing tables versioned outside metric labels. Model names, pricing versions, and execution IDs can create unbounded label sets; those belong in the completion record or trace attributes. The metric should answer, "Is cost moving outside policy for this workflow?" The evidence record should answer, "Which run and pricing inputs produced it?"
When is the simpler runner-up enough?
Status checks plus completion heartbeats are the runner-up. They are enough for a low-consequence job when the only required answers are whether the API is reachable and whether the last scheduled run completed within its window. This option has fewer moving parts, makes a good first weekly shipment, and can be upgraded later if the heartbeat payload already carries a stable execution ID. Its limitation is equally clear: it cannot reconstruct steps inside a run.
Choose it when a missed run is recoverable, dependencies are few, and replaying the job is cheaper than diagnosing its internal path. Do not pretend it provides causality. It can identify absence and lateness, not which model attempt, tool call, or queue delay caused them.
A status-only design is narrower still. It fits a stateless API whose useful work occurs entirely within probed requests and whose dependencies are covered elsewhere. It is not suitable for scheduled agent loops because the process can stay reachable while no scheduled execution completes. The full correlated design has its own trade-off: tail sampling adds collector state and operating work, so it is not suitable when replay is cheaper than diagnosis.
The upgrade trigger should be observable: an incident requires correlating records by hand, cost cannot be attributed to a workflow, or a latency alert cannot distinguish queue time from dependency time. At that point, the medium operational burden in the matrix is buying investigation speed rather than dashboard decoration.
The weekly operating loop
Instrumentation rots when nobody uses it. Run one reconstruction exercise after each meaningful deployment: select an execution, follow its ID from trigger to outcome, confirm duration and cost inputs, and verify that sensitive payloads are absent. This is a compact release check, not a ceremony.
Then test the negative paths. Stop the probe target and confirm reachability fails. Withhold a scheduled completion and confirm the heartbeat becomes overdue. Force an application-level failure in a non-production run and confirm the trace retention rule keeps it. Each test proves a different observer.
Review metric labels before adding them, because cardinality is much easier to prevent than unwind. Review tail-sampling capacity when traffic or trace size changes. Finally, keep the decision note beside the code: what gets paged, what gets retained, which policy values are configurable, and who can access detailed evidence.
That is enough process for a solo operator. The system stays legible, the weekly ship cadence survives, and an incident produces a timeline instead of a pile of green checks.
Top comments (0)