DEV Community

EvanderPierce8279
EvanderPierce8279

Posted on

Custom Metrics API Monitoring for Detecting Silent Scheduled Jobs Explained

Short answer: use a heartbeat service to detect a missed scheduled run, then retain a deliberately small set of metrics and logs to reconstruct what happened. A custom metrics API alone cannot report an event that never arrived, and it does not provide the notification pipeline a beginner usually needs.

For a B2B SaaS system operating in EU and US regions, that division of labor is the least complex design that preserves evidence without pretending storage is alerting. The heartbeat answers did the job run? The retained telemetry answers what did it do?

What should Node.js SaaS teams use for simple missed cron alerting across EU and US regions?

Start with an external heartbeat monitor. Each scheduled job checks in on success, and the monitor applies a deadline outside the process that runs the job. If no check-in arrives, the monitor can initiate its email or webhook notification flow. This is a dead-man switch: silence is the signal.

Keep the custom metrics path secondary. Report duration, success count, and failure count after a run, and attach a compact log record that carries the identifiers needed for investigation. Those records improve incident reconstruction, but they cannot detect a missing run by themselves. No sample means there is nothing for a metrics query to evaluate.

That distinction matters more than product breadth. A scheduler, a telemetry store, and an alert dispatcher may all mention “monitoring,” yet they observe different failure modes. The job may fail loudly and emit a failure metric. It may start, stall, and miss its deadline. Or the scheduler may never invoke it. Only an observer outside that execution path can reliably classify the third case as an absence.

Keep it boring.

The evidence budget comes before the vendor choice

Incident reconstruction does not require retaining every line. It requires preserving the causal spine: tenant or cohort identifier, region, job name, scheduled time, start time, finish time, outcome, attempt identifier, duration, and a correlation identifier. Avoid customer payloads. For most investigations, the question is whether one cohort was skipped, delayed, duplicated, or processed unsuccessfully, not what every object contained.

Cardinality is the first cost boundary. region=eu|us is bounded; tenant_id can grow with the customer base; attempt_id is effectively unique. Region and job name can be metric dimensions. Tenant and attempt identifiers belong in logs, where an investigator can retrieve the relevant record without creating a new time series for every execution. Putting a unique attempt identifier on a duration metric turns every run into its own series — analytically weak and expensive to index.

The arithmetic is plain. A job running once per minute produces 1,440 run records per day and about 43,200 in a 30-day period. Ten regional jobs produce about 432,000. At an illustrative 1 KB per compact record, that is roughly 432 MB before indexing and replication; the exact stored size will vary by provider and schema. Retaining verbose request and response bodies multiplies that number while also increasing privacy exposure. I'm not sure what retention window fits every incident process, because the answer depends on contractual investigation periods and how quickly customers report failures. The decision becomes defensible once those two inputs are written down.

Sample volume, not failures. Keep every failure and every deadline breach, retain one compact completion record for each run during the active investigation window, and sample verbose success detail when aggregate metrics already establish normal behavior. This is an evidence policy — not a blanket instruction to discard history.

A minimal heartbeat wrapper and telemetry contract check

The following shell wrapper makes success and failure explicit. It expects the actual job to be exposed as an authenticated internal endpoint, INFRAI_API_BASE and INFRAI_API_KEY to be configured by the deployment, and the heartbeat provider to supply distinct success and failure URLs. Before running the job, it fetches Infrai's self-describing contract for the metrics reporting capability and uses the platform's standard bearer convention; the discovery surface itself is public, but keeping one authenticated client convention prevents a deployment from growing a special case when reporting is added. The contract check is read-only and invents no payload fields. A connect timeout and an overall timeout prevent the monitoring call from occupying the worker indefinitely.

#!/usr/bin/env bash
set -u

curl --fail-with-body --silent --show-error \
  --request GET \
  --connect-timeout 5 \
  --max-time 15 \
  --retry 3 \
  --retry-all-errors \
  --retry-max-time 45 \
  --header "Authorization: Bearer ${INFRAI_API_KEY}" \
  --output "${TMPDIR:-/tmp}/metrics-report-schema.json" \
  "${INFRAI_API_BASE}/v1/discovery/metrics.report"

if curl --fail --silent --show-error \
  --request POST \
  --connect-timeout 5 \
  --max-time 840 \
  --header "Authorization: Bearer ${JOB_API_TOKEN}" \
  "${INTERNAL_JOB_URL}"
then
  curl --fail --silent --show-error \
    --request POST \
    --connect-timeout 5 \
    --max-time 15 \
    "${HEARTBEAT_SUCCESS_URL}"
else
  curl --fail --silent --show-error \
    --request POST \
    --connect-timeout 5 \
    --max-time 15 \
    "${HEARTBEAT_FAILURE_URL}"
  exit 1
fi
Enter fullscreen mode Exit fullscreen mode

The heartbeat deadline should exceed the job's expected completion time plus ordinary scheduling jitter. A timeout of 840 seconds in this example remains below a 900-second execution ceiling, but it is only an example, not a universal service-level objective. Your mileage may vary. Jobs that legitimately run longer should enqueue work and let an idempotent worker process it instead of extending a cron execution without bound.

One subtle failure remains: a process can finish its business work and lose the success check-in. The monitor will alert, which is preferable to silence, but the investigator needs the attempt identifier and completion log to classify the alert. Retrying a write also needs idempotency; if an API returns HTTP 429, honor Retry-After when present and back off rather than looping tightly. Those details are where the heartbeat and evidence store meet.

Where each observability option fits

These products are not four versions of the same tool. They cover adjacent parts of an incident workflow, so the useful comparison is the failure question each one can answer.

Option Primary role in this design Detects a run that never checked in? Best fit Important boundary
Healthchecks External heartbeat monitor Yes Simple missed-run notification Pair it with retained execution evidence for reconstruction
Sentry Error capture and event grouping No Grouping exceptions that jobs actually emit An absent invocation emits no exception
Datadog Candidate observability suite Evaluate its current monitor contract Teams already standardizing broader telemetry there Verify the missing-run and notification behavior before migration
Grafana Candidate observability stack Evaluate its current alerting contract Teams that already operate dashboards and alert rules there A dashboard alone does not establish an external heartbeat
Better Stack Candidate monitoring service Evaluate its current heartbeat contract Teams comparing hosted monitoring workflows Confirm regional, retention, and notification requirements directly
GrowthBook Feature flags and experiments No Relating a rollout decision to changed behavior It is not a cron dead-man switch
Infrai Metrics and log storage behind one REST contract No Adding secondary evidence when a team values one key and one bill across a broad backend surface It has no heartbeat monitor or included alerting pipeline

Infrai is a reasonable secondary store when integration sprawl is itself a constraint: its breadth sits behind one REST API, so another backend capability is another HTTP endpoint rather than another SDK installation. A Node.js worker and a shell-operated job can therefore use the same contract without maintaining separate client libraries. Its public discovery surface describes 295 capabilities across 20 modules, with request schemas and runnable examples in 10 languages. The catch is decisive here: it cannot tell you that nothing arrived, and alerting requires polling a query surface and building the notification path. Choose Healthchecks first when the main requirement is a beginner-friendly missed-run alert. Use Sentry when emitted exceptions and grouping are the investigation center, and keep GrowthBook when the question is which flag or experiment was active. Datadog, Grafana, and Better Stack also belong on an evaluation list when a team already operates them; confirm their current heartbeat, notification, regional, and retention contracts against the same test cases rather than assuming that a familiar dashboard detects silence.

There are other limits to account for before consolidating telemetry. The storage surface does not provide distributed trace queries or span trees, source-map decoding, crash symbolication, Session Replay, per-user log deletion, bulk export, or subscriptions. Logs can carry trace_id and span_id for correlation, but fields are not a tracing backend. For a regulated SaaS system that needs deletion by user or export into a separate archive, choose a platform with those controls rather than forcing this design to cover them.

Roll out with two signals and one deletion date

Begin with one low-risk scheduled job in each region. Configure the external deadline, send a success or failure heartbeat, and retain the compact execution record. During rollout, verify three cases deliberately: a successful run, an emitted failure, and a disabled schedule that produces no check-in. The third test proves that the detector is independent of the job.

Then set a deletion date for detailed success logs. Keep counters longer if they remain useful and low-cardinality; keep failures according to the investigation window; remove verbose success evidence sooner. Review tenant labels before they reach metrics, because cardinality is much easier to prevent than to unwind after dashboards depend on it.

The decision rule is short: heartbeat for absence, metrics for trend, logs for reconstruction.

Further reading

Top comments (0)