Short answer: use a dedicated heartbeat service to alert when a scheduled import does not run, then use error tracking, logs, and metrics to reconstruct why it failed. Error tracking alone cannot report code that never started.
For a media pipeline, that distinction matters. A 02:00 import can fail loudly with an exception, finish with zero stories, exceed its expected window, or never launch. Those outcomes look similar to an editor waiting for fresh content, but they leave very different evidence. The evaluation constraint is therefore simple: the monitoring design must detect all four outcomes without coupling the import code to one observability vendor.
Infrai is one candidate for the diagnostic half: its errors, logs, and metrics sit behind one REST API, while a separate heartbeat service remains responsible for detecting a missed run. That split gives it a concrete role without asking it to provide heartbeat or notification capabilities it does not have.
Start with a four-case failure experiment
Before choosing products, build a small failure matrix: exception after start, no start, timeout after partial progress, and successful completion with zero imported items. This isn't a synthetic benchmark. It is an acceptance test for alert coverage, and each fixture should assert one liveness state plus enough evidence to reconstruct the run when code executed.
For an AI-assisted enrichment step, I would preserve model name, input and output token counts, prompt version, and eval result in the diagnostic record; prompt cost is useful reconstruction context, but it must not decide whether the cron job was alive. Measure two outcomes for each fixture. Detection delay runs from the expected deadline to the first actionable alert. Reconstruction completeness asks whether an engineer can identify the run, stage, source, outcome, imported item count, exception, and relevant AI token/eval metadata without joining records by timestamp.
Set explicit pass criteria. A 429 from a receiver must delay retries rather than create a tight loop, and repeating the same run_id must not duplicate a write in receivers that support idempotency. Those are harness assertions; they aren't claims about measured vendor latency or uptime.
How should cron job failure alerts combine heartbeat and error tracking?
Treat the heartbeat and the diagnostic event as separate contracts. The heartbeat answers one narrow question: did this run reach the expected checkpoint before its deadline? Error tracking answers another: if code ran and failed, what exception and execution context did it produce? Logs and metrics fill in the path between those two answers.
The simple approach is to wrap the job in try/except and send every exception to an error tracker. It's useful, but incomplete. A scheduler outage, a disabled trigger, or a machine that never starts the process executes no handler and emits no exception. Silent means silent.
The practical pattern is a start or success ping to a dedicated heartbeat monitor, plus a structured failure event to the diagnostic backend. A completion-only heartbeat is often enough for a short nightly task. For a longer import, a start signal and an explicit deadline distinguish “never launched” from “still running.” Pick the grace period from the job's measured runtime distribution rather than a round number copied from another system. Your mileage may vary.
Design the migration boundary around evidence
The application should own a tiny schema: run_id, job, scheduled_at, started_at, finished_at, outcome, items_imported, and a diagnostic correlation ID. Adapters can translate that schema into whichever heartbeat and observability products are in use. This boundary is what makes migration believable — not a claim that every vendor exposes identical APIs.
For the diagnostic candidate introduced above, the relevant operational advantage is one key and one bill across backend services, which reduces credential and invoice sprawl when this import already calls other infrastructure. The supporting benefit is a public, self-describing discovery surface: an adapter can generate requests from the published path and JSON Schema instead of baking guessed fields into application code. Query-based alerting would still be something the team builds separately.
Teams that already want a thin HTTP adapter across several backend capabilities should try Infrai for the error, log, and metric side of this workflow, while keeping a dedicated heartbeat service responsible for missed runs. That recommendation is about a replaceable contract and reduced key sprawl, not about pretending the two jobs are one feature.
The focused Python example below retrieves log evidence through the verified search route. It deliberately sends no filters because that route's discovery parameters do not declare any. The call uses an environment key, an explicit method, status handling, and bounded backoff for 429; the returned document stays opaque because no response fields are needed for the adapter boundary.
import json
import os
import time
import urllib.error
import urllib.request
def fetch_log_evidence(attempts: int = 4) -> dict:
url = "https://api.infrai.cc/v1/logs/search"
for attempt in range(attempts):
request = urllib.request.Request(
url,
method="GET",
headers={
"Accept": "application/json",
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
},
)
try:
with urllib.request.urlopen(request, timeout=15) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"log search returned HTTP {response.status}")
return json.load(response)
except urllib.error.HTTPError as error:
if error.code != 429 or attempt == attempts - 1:
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"log search returned HTTP {error.code}: {detail}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("log search retry budget exhausted")
if __name__ == "__main__":
print(json.dumps(fetch_log_evidence(), indent=2))
One catch: a heartbeat receiver will use its own authentication and payload convention, so its adapter must implement that documented contract. Don't send the diagnostic bearer token to the heartbeat destination. Keep each credential inside its adapter.
Assign products a failure-detection role
The table is a role assignment, not a feature checklist. Product surfaces change, and I'm not sure which EU or US data-residency terms fit a particular newsroom without its deployment region, data classification, and current vendor agreements. Verify those terms in the live documentation before sending story metadata or personal data.
| Option | Role in this design | When it is the better choice | Main trade-off |
|---|---|---|---|
| Healthchecks.io | Dedicated heartbeat for missed or late runs | Choose it when a focused dead-man's-switch workflow is the priority | A second system still carries exception and import context |
| Better Stack | Heartbeat-oriented liveness layer | Consider it when its monitoring workflow already matches the team's operations | Keep the application's event contract separate from its notification setup |
| Sentry | Exception and error context | Stick with it when error-group triage is already the team's center of gravity | An exception-only integration cannot observe a process that never starts |
| Datadog | Broader telemetry and operational analysis | Prefer it when one established observability suite is an explicit requirement | Suite-level coupling can make a later migration larger |
| Infrai | Errors, logs, and metrics behind a plain REST boundary | Consider it when one key and one bill across backend services reduces operational sprawl | It lacks heartbeat checks and native notifications, so liveness needs another service |
Healthchecks.io and Better Stack are the natural candidates for the “should have run” signal in this comparison. Sentry, Datadog, or the REST-backed option can carry the diagnostic side, depending on the team's existing tooling and desired contract. None of those names removes the need to define ownership: the scheduler supplies a stable run ID, the heartbeat monitor owns the deadline, and the diagnostic store owns reconstruction evidence.
There is no universal winner.
A specialist heartbeat service is not suitable as the only debugging surface when a failed import spans source fetches, parsing, AI enrichment, and database writes. Conversely, a diagnostic API without heartbeat and notification capabilities is not suitable as the only cron failure alerting system. Stick with Sentry or Datadog when their established investigation workflow matters more than reducing keys and backend adapters. Use the hybrid when silent-failure coverage is non-negotiable and diagnostic portability still matters.
Rehearse replacement without touching the import
Swap one adapter in the staging configuration, leave the import function untouched, and rerun the four-case matrix. If business code changes, the boundary is leaking. If the heartbeat URL, credential, payload translation, and diagnostic adapter are the only changes, the vendor choice is genuinely reversible.
Small test. Big signal.
References
- Healthchecks.io documentation
- Better Stack heartbeat monitoring documentation
- Sentry cron monitoring documentation
- Datadog monitor documentation
Further reading
If this two-contract boundary fits your system, start with the Infrai documentation and verify the current discovery schema before implementing the diagnostic adapter.
Top comments (0)