JaggerBlack5781

Posted on Sep 20

Custom Metrics API and Healthchecks for Cron Monitoring Incident Reconstruction

#observability #node #saas

For a nightly logistics pipeline, the least complex reliable choice is a hosted heartbeat monitor. Add structured logs and metrics for reconstruction, but do not ask them to detect silence. A missing run emits no event, so an event store cannot know by itself that the job should have arrived.

TL;DR: send a heartbeat to Healthchecks, Cronitor, or Better Uptime for the missed-run alert. Send run records to a searchable store for the second question: which tenant, region, carrier batch, and stage failed? Infrai can fill that secondary role when keeping the same REST contract while changing the provider behind a capability matters. It does not supply the dead-man switch or notification pipeline.

Choice	Detects no start	Reconstructs a run	Operational burden	Best fit
Healthchecks	Yes	Limited event history	Low	A focused dead-man switch
Cronitor	Yes	Run telemetry aimed at jobs	Low	Cron-focused monitoring with more job context
Better Uptime	Yes	Broader incident context	Low to medium	Teams combining heartbeats with on-call workflows
Sentry	No heartbeat assumption here	Strong error grouping and application context	Medium	Diagnosing thrown errors alongside an existing scheduler monitor
Datadog	Yes, with its cron monitoring	Logs, metrics, and traces	Medium	An integrated observability stack
Grafana Cloud	Yes, with synthetic monitoring	Logs, metrics, and dashboards	Medium	Teams already using the Grafana ecosystem
Portable REST logs and metrics	No	Structured per-run evidence	Medium	A stable API contract across backend providers

My recommendation is intentionally two-part: buy the silence detector, then keep the evidence layer small. For a one-person SaaS, an alerting system is undifferentiated infrastructure. The revenue-per-hour move is to outsource it and get back to shipping this week's feature. Datadog is appealing when one integrated observability stack is worth its larger surface area; Grafana Cloud fits a team that already wants Grafana dashboards and managed telemetry. Neither changes the core rule that a component must own the expected schedule.

Should Cron Monitoring Use Healthchecks or a Custom Metrics API?

A custom metric proves that code reached the reporting call. It cannot prove that the scheduler started the process, that a queue delivered the message, or that the host was alive at 02:00. This is the observability version of waiting for a witness who never entered the building.

The distinction matters in a nightly logistics import. Suppose the EU job normally starts at 01:00 UTC and the US job at 06:00 UTC. A successful run can report duration, a success count, and a failure count. A failed parser can log the carrier file and pipeline stage. But if the cron trigger never fires, all of those values are absent. Absence in an event database is ambiguous unless a separate component knows the expected schedule and deadline.

That component is the dead-man switch. Healthchecks documents ping URLs and signals a failure when an expected ping does not arrive. Cronitor describes cron monitoring around expected schedules and missed executions. Better Uptime documents heartbeat monitors for recurring jobs. These products own the clock and the notification path; that is the crucial difference from a general metrics or log ingestion API.

Use two independent signals. Ping the heartbeat monitor at the job boundary, then write structured evidence as the work proceeds. If the monitor says "late" and the log store has no run_started record, investigate scheduling or delivery. If run_started exists but run_finished does not, inspect the last completed stage. If both exist and a business count is wrong, inspect the batch itself.

One bit wakes you up.

The record explains why.

Incident reconstruction is the real data-model decision

The useful question at 07:10 is rarely "did something fail?" The alert already answered that. The useful question is "what can I replay without duplicating shipments or losing a tenant's updates?"

Give every scheduled attempt a client-generated run_id. Include pipeline, region, stage, status, and timestamps in each structured record. Add business identifiers only when they are safe and necessary. In this scenario, a carrier batch ID is more useful than an unbounded message; it lets an operator connect validation, normalization, and import stages without treating a trace system as a prerequisite.

Correlation IDs can connect records without pretending that logs are traces. If distributed trace queries, a span-tree view, source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay are central to the incident, Sentry or a tracing product belongs in the design. Sentry's grouping and fingerprint controls are particularly relevant when the repeated unit is an exception rather than a scheduled run.

There is another boundary with practical consequences for logistics data: the log service has no per-user deletion API, bulk export API, or subscription API. Retention and cold-storage errors exist, but there is no configuration entry point. Do not place personal delivery details in this evidence stream and assume they can later be selectively erased. Keep the payload operational: opaque tenant ID, batch ID, counts, stages, and timing.

This is where the provider contract becomes valuable. Infrai exposes logs and metrics through one REST API under one key, alongside a public self-describing discovery surface; its manifest covers 295 routes across 20 modules, and capability documents include request schema, response schema, billing, and runnable examples in 10 languages. Code can stay behind one contract while the vendor serving a capability changes, and one credential reduces secret rotation work across the broader backend. The trade-off is firm: it has no heartbeat, dead-man switch, or alert notification pipeline. Use Healthchecks, Cronitor, Better Uptime, Datadog, or Grafana instead when a single product must own missed-run detection.

A small TypeScript monitoring boundary

The following runnable TypeScript keeps the two responsibilities visible. The heartbeat URL and evidence API URL come from the chosen services, so the job does not bake either vendor into its control flow. It uses a deterministic idempotency key, checks every response, and honors Retry-After on HTTP 429.

import { randomUUID } from "node:crypto";

const infraiApiBaseUrl = process.env.INFRAI_API_BASE_URL;
const infraiApiKey = process.env.INFRAI_API_KEY;
const heartbeatPingUrl = process.env.HEARTBEAT_PING_URL;
if (!infraiApiBaseUrl || !infraiApiKey || !heartbeatPingUrl) {
  throw new Error("INFRAI_API_BASE_URL, INFRAI_API_KEY, and HEARTBEAT_PING_URL are required");
}

const runId = process.env.RUN_ID ?? randomUUID();
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

function retryDelay(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter && /^\d+$/.test(retryAfter)) return Number(retryAfter) * 1_000;
  return 500 * 2 ** attempt;
}

async function requestWithRetry(
  url: string,
  init: RequestInit,
): Promise<unknown> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(url, init);

    if (response.status === 429 && attempt < 3) {
      await sleep(retryDelay(response, attempt));
      continue;
    }

    const body = await response.text();
    if (!response.ok) {
      throw new Error(`Monitoring request failed (${response.status}): ${body}`);
    }
    return body ? JSON.parse(body) : null;
  }
  throw new Error("Log ingestion exhausted all retries");
}

const event = {
  run_id: runId,
  pipeline: "carrier-manifest-import",
  region: "eu",
  stage: "download",
  status: "started",
  occurred_at: new Date().toISOString(),
};

const logIngestUrl = new URL("/v1/logs/ingest", infraiApiBaseUrl).toString();
await requestWithRetry(logIngestUrl, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${infraiApiKey}`,
    "Content-Type": "application/json",
    "Idempotency-Key": `nightly-import:${runId}:started`,
  },
  body: JSON.stringify(event),
});

await requestWithRetry(heartbeatPingUrl, { method: "POST" });

Set INFRAI_API_BASE_URL to the service's documented v1 API base before running the script, then validate event against the public discovery schema for logs.ingest. Reusing the same key for this logical write prevents a rate-limit retry from creating a second event; the platform convention specifies a 24-hour default deduplication window.

The heartbeat call should remain separate. Put it at the boundary agreed with the monitoring service: usually a success ping after the complete import, with the service configured around the actual schedule and acceptable grace period. If a run can partially succeed, decide whether "success" means file downloaded, rows validated, or all tenant changes committed. Pick one. An optimistic ping halfway through the pipeline creates a green monitor and a broken morning.

Which runner-up is better?

Pick Healthchecks when the requirement is narrow: a recurring task must check in, and silence must turn into a notification. Its focused model is easy to reason about. That clarity is useful when there is one person on call and every extra dashboard competes with product work.

Cronitor is the better fit when cron jobs are themselves the main operational surface and you want monitoring organized around their schedules and executions. Better Uptime makes more sense when heartbeat failures should join a broader incident-management and on-call workflow. Neither choice removes the need to decide what evidence the pipeline records.

Choose Sentry as the secondary tool when exception diagnosis dominates reconstruction. Its event grouping and fingerprint mechanics help combine related errors or split errors that need separate treatment. It still should not be treated as proof that a scheduled job ran unless a documented monitor owns that expectation.

Choose a general logs-and-metrics API when portability at the integration boundary matters more than an all-in-one incident UI. This route has a clear limitation: ingestion alone provides no threshold rules, phone, SMS, or webhook notifications, so alerting requires an external heartbeat service or a polling system you build yourself. Building that poller is rarely a good first-week task for a solo SaaS.

The operating rule I would ship

Keep the policy short enough to use under pressure: heartbeat for liveness, structured records for reconstruction, error tooling for exceptions. Test the missed-ping path before trusting it. Then test a run that starts and stalls after one stage, because that failure needs different evidence from a run that never starts.

Review the data fields before rollout in both EU and US processing. Opaque identifiers and bounded operational attributes make incident search useful without turning logs into a second customer database. Record duration, success count, and failure count after completion; record stage transitions when replay boundaries matter. Avoid inventing query filters around either logs or metrics, because their query filter parameters are not declared in discovery.

Ship weekly.

Spend the setup budget on one end-to-end drill: disable a test schedule, wait past its grace period, verify the external notification, and then reconstruct a separate failed run from its run_id. Next, force a parser failure after the download stage and confirm that the same run_id reveals exactly where replay should begin. This is a deliberate trade-off: two narrow signals create one extra correlation field, but they avoid coupling wake-up delivery to the database being investigated. The 15 minutes spent on that drill are more useful than polishing another dashboard because the exercise proves the two halves independently. A dashboard screenshot does not.

DEV Community