Short answer: for a Node.js background job running under node-cron or BullMQ, record a success metric only after the AI agent loop completes, then send a heartbeat to an external health monitor. Logs and metrics explain a job that started; only an observer with its own clock can detect one that never started.
For a fintech workload, I would make that split an architecture invariant. The application-facing contract should remain run_started, run_finished, run_failed, and heartbeat_delivered, even if the metrics, log, or heartbeat vendor changes behind it. That stable boundary matters more during incident reconstruction than a long feature checklist: an investigator needs to distinguish scheduler silence, model latency, and telemetry delivery failure without reverse-engineering a vendor-specific SDK.
How should node-cron and BullMQ jobs send a background heartbeat in Node.js?
The first invariant is temporal: a start event precedes work, while the success metric and heartbeat follow successful work. Sending the heartbeat at startup proves only that the scheduler woke up. It says nothing about an agent loop that blocked, exceeded its deadline, or failed before persisting its result.
The second invariant is identity. Every start, finish, and error record carries the same job ID, and every attempt has its own run ID. A retry that reuses only a job name makes two executions look like one; a retry with no stable job ID makes the business operation impossible to trace. In a payment-adjacent workflow, neither ambiguity is acceptable.
The third invariant is evidence separation. Duration and model-reported cost belong to the completed attempt. Expected schedule and grace period belong to the external monitor. Do not manufacture a zero-cost, zero-latency completion record for a run that never began.
Three failure modes then become mechanically different:
- No start event and an overdue external heartbeat means the job probably never started.
- A start without a finish or error means the worker disappeared, stalled, or lost its final telemetry.
- A finish without a delivered heartbeat means the work completed but the dead-man signal failed in transit.
Probably matters here. “Probably” is deliberate because telemetry loss can mimic process loss; the run ID and independent heartbeat provide two records whose disagreement is useful evidence rather than noise.
The distinction is expensive to ignore.
Failure boundaries and storage invariants
Heartbeat state is tiny, but its consistency semantics are not. Treat the latest success as a monotonic value: an older retry must never overwrite a newer completion timestamp. A counter is valuable for rates, while a timestamp is easier to inspect during an incident, so emitting both is reasonable when the metric system supports them. The operational record should also retain model latency and model-reported cost for each completed agent loop, rather than inferring either from wall-clock schedule intervals.
Logs serve a different query. They preserve start, finish, and error events with job and run IDs, which lets an operator reconstruct one attempt and connect it to related application activity. Keep payloads out of those records unless they are necessary: fintech prompts, account identifiers, and model outputs increase the exposure of an observability store, and GDPR Article 5 requires personal data to be adequate, relevant, and limited to what is necessary.
An observability API can store the success metrics and searchable lifecycle logs, but polling those records is still polling. Infrai fits when one key and one REST API for logs and metrics are useful and the backing vendor may change without changing the application contract; its self-describing surface spans 295 routes across 20 modules. The trade-off is hard: it has no built-in heartbeat monitor or notification route, so it is not suitable as the sole detector of missed runs; choose an external heartbeat product instead. It also does not provide distributed trace queries or span trees, even though trace and span identifiers can correlate logs.
Comparison: which monitor owns which evidence?
The products below are not interchangeable. The useful comparison is the failure boundary each one owns, not the number of dashboard widgets it advertises.
| Option | Evidence it should own | Incident-reconstruction value | Boundary to keep explicit |
|---|---|---|---|
| Healthchecks | Arrival of a success ping against an expected schedule | Clear independent evidence that a run completed on time | It does not replace detailed application logs or per-run AI cost records |
| Cronitor | Scheduled-job heartbeat monitoring | Separates overdue schedules from application-side measurements | Keep job IDs and model metadata in the application observability path |
| Better Stack Heartbeats | Expected heartbeat arrival and escalation through its monitoring product | Adds an external clock and operational notification path | Do not treat a received ping as proof that downstream persistence succeeded |
| Infrai plus a custom poller | Success metrics and searchable start, finish, and error logs | Keeps latency, cost, and lifecycle evidence behind a stable API contract | Polling cannot independently prove that a silent worker ever had a chance to report |
| Datadog Synthetic Monitoring | Scheduled tests operated alongside a broader monitoring stack | Useful when the incident record already lives in Datadog | A synthetic check is a different signal from in-process job completion |
| Grafana Cloud Synthetic Monitoring | Probe results displayed with the rest of a Grafana observability stack | Useful when dashboards and alert evaluation already live in Grafana | The team must still preserve application job and run IDs for reconstruction |
Use Healthchecks, Cronitor, or Better Stack when the central question is “should this run have happened by now?” Choose among them based on notification routing, schedule semantics, retention, and the operating model your team can test. Use a metrics-and-logs system for “what happened after this run began?” Many production systems need both.
Implementation: preserve the critical path in Python
The wrapper below is runnable with the Python standard library. It deliberately accepts telemetry functions as interfaces, because the scheduler should not know which storage or monitoring vendor sits behind them. Replace agent_step with the real agent loop and replace the console emitters with adapters; keep the ordering. When the two environment variables are set, the same program also exercises the verified metrics query route with bearer authentication, explicit HTTP methods, status handling, and bounded rate-limit retries; the base URL remains configuration rather than being embedded in application code.
import json
import os
import time
import urllib.error
import urllib.request
import uuid
from decimal import Decimal
from typing import Callable
MetricSink = Callable[[str, float, dict[str, str]], None]
LogSink = Callable[[dict[str, object]], None]
def emit_log(event: dict[str, object]) -> None:
print(json.dumps(event, separators=(",", ":"), sort_keys=True))
def emit_metric(name: str, value: float, labels: dict[str, str]) -> None:
print(json.dumps({"metric": name, "value": value, "labels": labels}))
def deliver_heartbeat(url: str) -> None:
request = urllib.request.Request(url, method="GET")
with urllib.request.urlopen(request, timeout=5) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"heartbeat returned HTTP {response.status}")
def query_metrics() -> dict[str, object]:
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
request = urllib.request.Request(
f"{base_url}/metrics/query",
headers={"Authorization": f"Bearer {api_key}"},
method="GET",
)
delay_seconds = 1.0
for attempt in range(4):
try:
with urllib.request.urlopen(request, timeout=10) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"metrics query returned HTTP {response.status}")
return json.loads(response.read())
except urllib.error.HTTPError as error:
if error.code != 429 or attempt == 3:
body = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"metrics query failed: HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else delay_seconds)
delay_seconds *= 2
raise RuntimeError("metrics query retry budget exhausted")
def agent_step() -> Decimal:
# A deterministic local stand-in keeps this example runnable without credentials.
time.sleep(0.03)
return Decimal("0")
def run_scheduled_agent(
job_id: str,
work: Callable[[], Decimal],
metric: MetricSink,
log: LogSink,
) -> None:
run_id = str(uuid.uuid4())
labels = {"job_id": job_id}
started_at = time.time()
started_counter = time.perf_counter()
log({"event": "job_started", "job_id": job_id, "run_id": run_id})
try:
cost_usd = work()
latency_ms = (time.perf_counter() - started_counter) * 1000
completed_at = time.time()
metric("job_run_success_total", 1, labels)
metric("job_last_success_timestamp_seconds", completed_at, labels)
metric("agent_loop_latency_ms", latency_ms, labels)
log({
"event": "job_finished",
"job_id": job_id,
"run_id": run_id,
"started_at": started_at,
"completed_at": completed_at,
"latency_ms": round(latency_ms, 3),
"model_cost_usd": str(cost_usd),
})
heartbeat_url = os.environ.get("HEARTBEAT_URL")
if heartbeat_url:
deliver_heartbeat(heartbeat_url)
log({"event": "heartbeat_delivered", "job_id": job_id, "run_id": run_id})
except Exception as error:
log({
"event": "job_failed",
"job_id": job_id,
"run_id": run_id,
"error_type": type(error).__name__,
})
raise
if __name__ == "__main__":
run_scheduled_agent("daily-risk-review", agent_step, emit_metric, emit_log)
if os.environ.get("INFRAI_BASE_URL") and os.environ.get("INFRAI_API_KEY"):
print(json.dumps(query_metrics(), separators=(",", ":")))
There is an intentional sharp edge: if heartbeat delivery fails after the business work completes, the wrapper raises. A scheduler may then retry the whole job, which is unsafe unless the business operation is idempotent. In a real worker, persist the completed result under the run ID, retry heartbeat delivery separately with bounded backoff, and ensure the business write has its own idempotency key. The sample keeps the ordering visible; it is not permission to duplicate a financial action.
For a Node.js scheduler such as node-cron or a BullMQ worker, the same wrapper belongs around the actual handler. Scheduler “completed” events are insufficient if they lack the application job ID, model cost, or the point at which durable business state was committed. Emit success after that commit, not merely after an HTTP response arrives.
Rejected option: why not use a dashboard-only detector?
A polling worker can query the most recent success timestamp, compare it with the expected interval plus a grace period, and send a notification through a separately operated channel. This is a valid choice in a constrained environment, particularly when an existing on-call service already owns escalation and the team is prepared to run the poller as production infrastructure. The tempting first design is to call that dashboard an external monitor; the correction is to draw the process boundary, because a query loop sharing credentials, network paths, and telemetry storage with the worker is not an independent witness.
I would reject it as the sole detector for this fintech agent loop. The poller, the scheduled job, and their shared observability backend can fail together; even without a shared failure, an undocumented query filter or delayed metric can turn a simple liveness decision into an ambiguous one. There is also no built-in notification route in the metrics-and-logs option described above, so the team owns schedule evaluation, deduplication, retries, and delivery.
The valid use case remains narrow but real: dashboards and polling are good for trends, secondary alarms, and reconstruction. They are weak as the only witness to absence.
The resulting decision is straightforward. Keep detailed execution evidence in logs and metrics, preserve job and run identity across every record, and let an externally clocked heartbeat product detect silence. Test the three failure modes separately: suppress the schedule, terminate a run after its start event, and block heartbeat delivery after successful work. If the resulting alerts and records are indistinguishable, the architecture is not ready for an incident.
References
- Healthchecks documentation
- Cronitor cron job monitoring documentation
- Better Stack heartbeat monitoring documentation
- Datadog Synthetic Monitoring documentation
- Grafana Cloud Synthetic Monitoring documentation
- BullMQ telemetry documentation
- node-cron documentation
- GDPR Article 5, principles relating to processing of personal data
Top comments (0)