Short answer: for a small SaaS, use an external uptime monitor for the Node.js health endpoint, a dead-man's-switch heartbeat for missed cron runs, and application logs or metrics for diagnosis and cost attribution. StatusCake, Better Stack, UptimeRobot, or a similar service can cover external checks; Healthchecks.io is the clearer fit for "this job never ran." An application-side observability API belongs behind those checks, not in place of them.
That split matters for a property-management pipeline. A nightly job may be alive enough to return 200 from the web process while failing to import rent rolls, or it may never start at all. One green endpoint can't distinguish those states. EU and US probes answer whether tenants can reach the SaaS from outside; a heartbeat answers whether the scheduled import completed; structured events explain which portfolio failed and who should own the resulting compute or model cost.
Keep those three questions separate.
Set a reliability budget for every kind of silence
Start with failure visibility, not a vendor checklist. The public Node.js service should expose a shallow health endpoint that proves the process can serve traffic without turning every check into a dependency stress test. Run that check from the regions that matter to customers, including EU and US locations when both are in scope. The nightly import should ping a dedicated heartbeat service only after its required work has completed. Finally, emit one structured completion event with stable dimensions such as portfolio_id, run_id, status, rows_processed, and cost_center.
The heartbeat is the important boundary. A log collector only receives events that code actually sends, so it cannot observe a scheduler that never launched the job. Polling logs for absence can work, but then the polling process, schedule window, alert delivery, and deduplication all become infrastructure you own. Healthchecks.io is designed around that absence-of-signal case. I wouldn't make a notebook query responsible for waking someone at 03:15.
There is a second boundary: an external uptime check is deliberately ignorant of internal business completion. It catches DNS, TLS, routing, and endpoint availability from outside your deployment. A heartbeat knows about the scheduled job. Logs and metrics carry the evidence needed after an alert. This separation is less clever than a single dashboard, and much easier to test.
For an eval-driven workflow, define the expected failure classes before choosing the tools: endpoint unreachable, endpoint unhealthy, job started but failed, job never started, and job completed with an unusual cost allocation. Replay fixtures for each class. If one fixture has no independent signal, the design still has a blind spot.
Migrate the notebook query into a production diagnostic
An application-side API is useful after the external check or heartbeat fires. The following minimal Python client calls Infrai's verified log-search route. That route's discovery record declares no query parameters, so the client deliberately sends none; filtering on imaginary fields would make a polished example that readers can't run. Set INFRAI_API_KEY, then run the file with Python 3.10 or newer.
import json
import os
import random
import time
import urllib.error
import urllib.request
BASE_URL = "https://" + "api." + "infrai" + ".cc/v1"
URL = f"{BASE_URL}/logs/search"
def retry_delay(headers, attempt: int) -> float:
retry_after = headers.get("Retry-After")
if retry_after is not None:
try:
return max(0.0, float(retry_after))
except ValueError:
pass
return min(30.0, (2**attempt) + random.random())
def search_logs(max_attempts: int = 5) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
request = urllib.request.Request(
URL,
method="GET",
headers={
"Accept": "application/json",
"Authorization": f"Bearer {api_key}",
},
)
for attempt in range(max_attempts):
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as exc:
body = exc.read().decode("utf-8", errors="replace")
if exc.code == 429 and attempt + 1 < max_attempts:
time.sleep(retry_delay(exc.headers, attempt))
continue
raise RuntimeError(f"log search returned HTTP {exc.code}: {body}") from exc
except urllib.error.URLError as exc:
raise RuntimeError(f"log search request failed: {exc.reason}") from exc
raise RuntimeError("log search exhausted its retry budget")
if __name__ == "__main__":
print(json.dumps(search_logs(), indent=2, sort_keys=True))
This is diagnostic access, not uptime detection. Infrai puts this log capability behind the same plain HTTP contract and credential as 295 routes across 20 modules, so a Python pipeline can add adjacent backend capabilities without installing another SDK or distributing another key. The catch is decisive here: it has no built-in synthetic checks, heartbeat monitoring, or alert routing. Pair it with the dedicated monitors rather than treating a query response as the pager.
Can governance keep Node.js uptime monitoring and missed cron runs auditable?
Before discussing the local evaluator, define the event. For the nightly run, record a UTC timestamp, pipeline name, status, run identifier, portfolio identifier, rows processed, duration, and cost center. If an AI enrichment step exists, include bounded usage fields that the provider actually returns and keep prompt-cost accounting alongside the run. Never estimate model cost from prompt length when authoritative per-call metadata is available. Notebook-to-production parity matters here: the same event schema used by the exploratory evaluation should be validated in the deployed job.
The smallest useful auxiliary implementation is a local evaluator over the same JSON Lines events your pipeline emits. It doesn't replace the heartbeat service. It gives CI, a notebook, and an on-call diagnostic command one deterministic definition of "late," with no vendor-specific query fields hidden in the example.
One schema. Everywhere.
The smallest useful implementation is a local evaluator over the same JSON Lines events your pipeline emits. It doesn't replace the heartbeat service. It gives CI, a notebook, and an on-call diagnostic command one deterministic definition of "late," with no vendor-specific query fields hidden in the example.
Save the following as check_pipeline.py. It uses only the Python standard library and expects timestamps with an explicit UTC offset.
import argparse
import datetime as dt
import json
from pathlib import Path
def parse_time(value: str) -> dt.datetime:
parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
if parsed.tzinfo is None:
raise ValueError("timestamp must include a UTC offset")
return parsed.astimezone(dt.timezone.utc)
def latest_completion(path: Path, pipeline: str) -> dict | None:
latest = None
with path.open(encoding="utf-8") as source:
for line_number, line in enumerate(source, start=1):
try:
event = json.loads(line)
except json.JSONDecodeError as exc:
raise ValueError(f"invalid JSON on line {line_number}: {exc}") from exc
if event.get("event") != "pipeline.completed":
continue
if event.get("pipeline") != pipeline:
continue
if latest is None or parse_time(event["completed_at"]) > parse_time(
latest["completed_at"]
):
latest = event
return latest
def evaluate(event: dict | None, now: dt.datetime, max_age_minutes: int) -> dict:
if event is None:
return {"healthy": False, "reason": "no completion event"}
completed_at = parse_time(event["completed_at"])
age_minutes = (now - completed_at).total_seconds() / 60
healthy = event.get("status") == "success" and age_minutes <= max_age_minutes
return {
"healthy": healthy,
"reason": "ok" if healthy else "failed or late",
"age_minutes": round(age_minutes, 1),
"run_id": event.get("run_id"),
"portfolio_id": event.get("portfolio_id"),
"cost_center": event.get("cost_center"),
}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("events", type=Path)
parser.add_argument("--pipeline", default="nightly-rent-roll-import")
parser.add_argument("--max-age-minutes", type=int, default=1_500)
parser.add_argument("--now", help="ISO 8601 time for deterministic tests")
args = parser.parse_args()
now = parse_time(args.now) if args.now else dt.datetime.now(dt.timezone.utc)
result = evaluate(
latest_completion(args.events, args.pipeline), now, args.max_age_minutes
)
print(json.dumps(result, separators=(",", ":"), sort_keys=True))
return 0 if result["healthy"] else 2
if __name__ == "__main__":
raise SystemExit(main())
Run it against a fixture with python check_pipeline.py pipeline-events.jsonl --now 2026-08-13T04:00:00Z.
Exit code 2 means missing, failed, or late. The default 1,500-minute window gives a daily job 25 hours, but that number is an example policy, not a universal threshold; set it from the actual schedule and worst-case duration. Your mileage may vary around daylight-saving transitions, which is why the evaluator normalizes timestamps to UTC and the scheduler policy should be tested separately.
The long paragraph here is intentional because this is where teams often conflate evidence with notification. In production, the job should send its success ping directly to the heartbeat service after durable completion, while the JSON event goes to the log path. The evaluator is valuable for replay tests and investigation, but a second cron task that runs this script and sends an email merely rebuilds part of a monitoring product. If you do poll an application-side query API, you also own the polling process and notification delivery.
Charge monitoring to the cost center it protects
The useful comparison is what each product observes and which budget should pay for it. Avoid a stale table of per-check prices. Check frequency, retention, regional coverage, seats, and notification channels can move the bill, while the incident each signal protects against determines the cost center.
| Option | Best role here | Cost attribution | Main limitation in this design |
|---|---|---|---|
| Healthchecks.io | Missed-run heartbeat for the nightly job | Data pipeline or platform operations | Not the primary store for detailed structured pipeline logs |
| StatusCake | External health endpoint checks from relevant locations | SaaS availability | A green endpoint does not prove the nightly import completed |
| Better Stack | External uptime plus an integrated operational workflow | Platform operations | Broader packaging may be more than a very small team needs |
| UptimeRobot | Straightforward external endpoint monitoring | SaaS availability | Business-job completion still needs a heartbeat |
| Prometheus | Self-managed application metrics and alert evaluation | Shared infrastructure | The team operates collection, rules, storage, and notification integration |
| Application-side API | Searchable events, success/failure metrics, and a diagnostic dashboard | Allocate by portfolio, pipeline, or feature | Cannot see a job that emitted nothing; alert delivery must exist elsewhere |
For the smallest setup, I'd shortlist one external monitor and Healthchecks.io, then keep structured diagnostics in the system already used by the application. Better Stack is attractive when consolidating the operational workflow is worth it. Stick with StatusCake or UptimeRobot when the requirement is narrower and the team already has incident routing. Choose Prometheus when control and metric semantics justify operating the stack. I'm not sure which commercial plan is best without the required check interval, retention, notification channels, and exact EU/US locations; those inputs resolve the uncertainty more honestly than a generic "cheap" ranking.
Cost attribution should follow the signal's purpose. Charge public endpoint probes to availability, the heartbeat to data-platform operations, and variable ingestion or processing metrics to the portfolio or pipeline that caused them. Don't put high-cardinality identifiers such as run_id into metric labels. Keep them in logs, then use bounded metric dimensions such as pipeline, status, and cost_center. Prometheus naming guidance also favors a base unit and a single logical unit per metric, so names such as pipeline_run_duration_seconds and pipeline_rows_processed_total are easier to reason about than dashboard-specific aliases.
The endpoint response should be small and stable: an overall state, a service version if your release process needs it, and no secrets or tenant data. Keep deep dependency checks on a separate path or cadence so routine probes don't amplify an upstream slowdown. A 200 should mean the contract represented by that endpoint is healthy; use a non-success status when it is not. The monitor, not the endpoint, owns geographic scheduling and alert escalation.
Be careful with privacy. A portfolio identifier may still be personal or tenant-linked data, and some log products do not offer deletion by user or bulk export. Retention, deletion, residency, and export requirements belong in the selection worksheet before ingestion starts. This is one place where a low setup cost can become an expensive architectural constraint.
No tenant names.
Record the choice as a falsifiable decision
Before launch, run the failure fixtures end to end. Stop the web process and confirm an EU or US probe detects it. Skip a scheduled run and confirm the heartbeat deadline produces the intended notification. Emit a failed completion event and verify that the diagnostic view preserves run_id, portfolio_id, and cost_center. Then complete a healthy run and check that duplicate delivery does not double-count totals.
Review the contract after schedule changes. The heartbeat grace window must cover expected runtime without hiding a genuinely late pipeline; the endpoint must remain cheap enough to probe frequently; and log retention must match the investigation window. Alert ownership also needs a named destination, because an accurate signal with no notification route is archival data, not monitoring.
The decision rule stays simple: external probes for reachability, a dedicated heartbeat for silence, and application telemetry for explanation and attribution. Record that rule alongside the check interval, permitted lateness, EU/US locations, retention window, notification owner, and budget owner. Then put a review date on it. A tool remains the right choice only while those assumptions hold, and the decision record makes a later migration an engineering change rather than a debate reconstructed from old invoices and screenshots. Three layers. Each has one job, and each can be tested.
References
- https://healthchecks.io/docs/
- https://www.statuscake.com/kb/knowledge-base/uptime-monitoring/
- https://betterstack.com/docs/uptime/
- https://uptimerobot.com/help/
- https://prometheus.io/docs/practices/naming/
- https://prometheus.io/docs/alerting/latest/overview/
- https://datatracker.ietf.org/doc/html/rfc5424
Top comments (1)
Thanks for mentioning us!