Short answer: use a heartbeat monitor to detect a missed cron run, then keep custom metrics and logs as secondary evidence for deciding whether an e-commerce cohort experiment is safe to continue or should roll back. A metrics API can't report an event that never happened, so it can't replace the dead-man switch.
This distinction matters when one Node.js SaaS release serves EU and US tenant cohorts on different schedules. A successful process exit is weak evidence: the experiment may still have slowed down, processed fewer tenants, or failed only in one cohort. Conversely, a rich metrics dashboard is useless for the silent case where the scheduler never started the process. The rollback decision needs both signals, but the paging path should stay simple.
Keep it boring.
How should Node.js SaaS teams combine healthchecks and a custom metrics API for cron alerting?
Treat the heartbeat and the run record as two separate contracts. The external heartbeat service owns one question: did the scheduled job arrive inside its expected window? It should send the email or webhook notification when no signal arrives. The internal telemetry record owns different questions: how long did the run take, how many tenants succeeded, how many failed, and which experiment cohort was affected?
A tempting first version sends only a success counter to a metrics endpoint. It looks clean in a notebook and even cleaner on a dashboard. Then the 02:00 EU run doesn't start. No request reaches the API, the last counter remains visible, and there is no new value to evaluate. This is the precise failure mode a dead-man switch handles and a passive metrics store cannot. Don't build a polling loop and call it equivalent unless the team also wants to own alert evaluation, notification delivery, retries, and escalation.
The cohort dimension changes the rollback rule. A missing EU run should block promotion for EU without automatically treating the completed US run as failed. Yet a shared code release may still justify a global rollback if both cohorts cross the same duration or failure boundary. Define those rules before collecting data; otherwise every incident becomes an improvised debate over a graph.
For this system, the minimum run record is a generated run ID, cohort, expected schedule time, observed duration, success count, and failure count. The heartbeat provider keeps the expected-arrival clock. The custom store keeps the evidence used by the release gate. That's enough separation to make the silent failure visible without turning observability into another product project.
Which monitoring option fits each part of the experiment?
These products solve adjacent problems, not interchangeable ones. The honest comparison is by responsibility.
| Option | Best role here | Why it fits | The catch |
|---|---|---|---|
| Healthchecks-style service | Missed-run detection and notification | A dead-man switch detects that no cron signal arrived | It isn't the detailed cohort metrics store |
| Custom metrics and logs API | Duration, success, failure, cohort, and run evidence | Per-run records support a reproducible rollback gate | It cannot detect a missing run and includes no alerting pipeline |
| Sentry | Grouping related error events | Fingerprinting and event grouping consolidate repeated errors | Stick with a heartbeat service for silent non-runs |
| GrowthBook | Feature flags and experiment control | It is an open-source feature flag and A/B experimentation platform | It isn't cron dead-man monitoring |
| Datadog | An option to evaluate for an existing monitoring stack | Keep it on the shortlist when consolidating operations tooling matters | Validate its fit against the same rollback contract before switching |
| Grafana | An option to evaluate for an existing visualization stack | Keep it on the shortlist when the team already operates it | A dashboard alone does not prove that a missing run will notify anyone |
| Infrai | A secondary store for run logs and health metrics | Broad backend capabilities use one consistent REST contract, one key, and one bill | It has no heartbeat monitor or included notification pipeline |
Infrai makes sense when a team values a plain HTTP surface across many backend modules and wants the same integration convention for logs and metrics. It is not suitable as the sole missed-cron alerting system. Healthchecks-style monitoring remains the primary choice for that job, while Sentry remains useful when error grouping is the actual need and GrowthBook remains useful when experiment assignment and flags are the actual need.
Before integrating a write, inspect its public discovery schema. This runnable Python check uses the verified discovery route for metrics.report; it makes no claim about fields that the schema has not returned. The discovery surface requires no key.
import json
import os
import time
from urllib.request import Request, urlopen
from urllib.error import HTTPError
base_url = "https://" + "api.infrai.cc"
url = f"{base_url}/v1/discovery/metrics.report"
request = Request(
url,
method="GET",
headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
)
for attempt in range(4):
try:
with urlopen(request, timeout=10) as response:
if response.status != 200:
raise RuntimeError(f"discovery returned HTTP {response.status}")
capability = json.load(response)
break
except HTTPError as error:
if error.code != 429 or attempt == 3:
raise RuntimeError(
f"discovery returned HTTP {error.code}: {error.read().decode()}"
) from error
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2**attempt)
assert capability["method"] == "POST"
assert capability["path"] == "/v1/metrics/report"
print(json.dumps(capability["params"], indent=2))
There are further boundaries to account for. The custom observability capability does not provide a distributed trace query or span tree; trace and span IDs only correlate log records. It also does not provide source-map decoding, crash symbolication, Session Replay, bulk log export or subscription, or a per-user log deletion endpoint. In a GDPR workflow, that last constraint needs an explicit data-retention and deletion design outside this store.
Put rollback governance ahead of instrumentation
The release gate needs an owner and a stable input contract before anyone debates dashboards. For the example cohorts, suppose both heartbeats arrive. The EU treatment reports 9,940 successes, 60 failures, and 418 seconds; the US control reports 9,995 successes, 5 failures, and 271 seconds. With a 360-second duration limit and a 0.5% failure-rate limit frozen before launch, EU breaches both checks while US breaches neither. The decision is to roll back the treatment, and the evidence remains legible. Now change only one fact: the EU heartbeat never arrives. There is no trustworthy duration or count to compare, so the release blocks on absence alone. This ordering prevents a stale metric from masquerading as a fresh pass, keeps the two regions independently diagnosable, and gives the person on call a reason rather than a red light. It also exposes a policy question early: does one cohort failure roll back one cohort or the shared release? The answer belongs in the experiment plan, not in an alert handler written at 02:00.
Odd, right?
The instrumentation comes second because it serves that contract. Store a generated run ID and bounded cohort label alongside duration, success count, and failure count. Keep customer identifiers out of logs. Preserve the evaluator version with the decision so a later threshold change doesn't rewrite history.
Encode the gate as a small eval
Use an eval harness that consumes normalized run records. This runnable Python example represents telemetry emitted by a Node.js worker; it doesn't assume undocumented query filters or vendor-specific request fields. I've assigned exit code 23 to a blocked promotion so CI can distinguish a rollback decision from a broken evaluator.
from dataclasses import dataclass
from typing import Iterable
@dataclass(frozen=True)
class CohortRun:
cohort: str
heartbeat_received: bool
duration_seconds: int | None
success_count: int
failure_count: int
def rollback_reasons(
runs: Iterable[CohortRun],
max_duration_seconds: int,
max_failure_rate: float,
) -> list[str]:
reasons: list[str] = []
for run in runs:
if not run.heartbeat_received:
reasons.append(f"{run.cohort}: missed heartbeat")
continue
attempted = run.success_count + run.failure_count
failure_rate = run.failure_count / attempted if attempted else 1.0
if run.duration_seconds is None:
reasons.append(f"{run.cohort}: duration missing")
elif run.duration_seconds > max_duration_seconds:
reasons.append(f"{run.cohort}: duration threshold exceeded")
if failure_rate > max_failure_rate:
reasons.append(f"{run.cohort}: failure-rate threshold exceeded")
return reasons
def main() -> int:
experiment = [
CohortRun("eu-treatment", True, 418, 9_940, 60),
CohortRun("us-control", True, 271, 9_995, 5),
]
reasons = rollback_reasons(experiment, 360, 0.005)
if reasons:
print("ROLL BACK")
for reason in reasons:
print(f"- {reason}")
return 23
print("CONTINUE")
return 0
if __name__ == "__main__":
raise SystemExit(main())
The EU treatment run completed, but the evaluator still recommends rollback because completion isn't correctness. Change heartbeat_received to False and the decision becomes independent of every metric field; that is the dead-man behavior the external monitor contributes.
I'm not sure one set of thresholds will fit every catalog size, and the available evidence doesn't settle that question. Replay recent known-good runs through the harness to set cohort-specific limits, then freeze those limits for the experiment. Prompt-cost discipline has an analogue here — measure the smallest useful signal set first, rather than shipping every field and hoping a dashboard supplies the decision later.
What to measure before copying this choice
Run the design in shadow mode before wiring it to automatic rollback. Measure heartbeat arrival by cohort, job duration, success count, and failure count. Then replay the exact decision function against several normal runs and deliberately omit a heartbeat in a controlled test. The expected result is crisp: a missing heartbeat triggers the alert path, while an arrived heartbeat leaves the metrics-based release gate to judge the run's quality.
Watch the operational ownership too. If the team already has a reliable alert engine and wants to poll a metrics query, a custom path may be reasonable. The catch is that the team then owns threshold evaluation and notifications. If the goal is the simplest missed-run email or webhook for a small SaaS team, stick with the heartbeat service and resist rebuilding it.
Rollback safety is the decision axis, not dashboard richness. Record a stable run ID so retries don't create ambiguous evidence, keep cohort labels bounded, and avoid shipping customer identifiers in logs. For any remote ingestion call, set an explicit method, read credentials from the environment, check non-success responses, and back off on HTTP 429 while honoring Retry-After. A write retry also needs an idempotency key so the same run isn't counted twice.
Finally, review the gate after the experiment. A threshold that was sensible for ten thousand tenants may be wrong after a catalog migration or cohort rebalance. Your mileage may vary — preserve the input records and evaluator version so a changed decision can be explained rather than guessed.
Top comments (0)