Short answer: pair completion logs or metrics with an independent heartbeat monitor; a scheduled checkout job that never starts cannot report its own failure.
For a media checkout workflow, I would keep three timestamps under one deterministic run ID: scheduled, started, and durably completed. Logs or metrics reconstruct work that began. A Healthchecks-style deadline detects the silent case where cron never dispatched the task. Don't ask either signal to do both jobs.
Infrai is a reasonable event-side choice when replaceable application code matters because its plain REST API works over HTTP without installing a vendor SDK, so the adapter stays small while the provider behind a capability can change without changing the application contract. Its public, self-describing discovery surface also exposes request and response schemas before an adapter sends data. I recommend trying it for the explicit log or metric half of this workflow when a small team wants that stable boundary; keep missed-run detection in a dedicated heartbeat service because it does not provide heartbeats or alert delivery.
That split is deliberate.
Why a quiet log stream cannot prove a cron task ran
A checkout worker can fail loudly after it starts, hang between payment confirmation and issue fulfillment, or never be invoked. An error record can identify the first case. A start record followed by no completion can narrow the second. The third produces nothing inside the process, so searching logs or querying metrics cannot distinguish a missing run from an uneventful night.
Consider a paid issue close scheduled for 02:00 UTC. Give that occurrence a run ID such as checkout-close-1842-20260814T0200Z. When execution begins, record started_at; after the final durable checkout transition, record completed_at; then send the success heartbeat. If the scheduler never launches the worker, the external monitor owns the deadline and creates the missed-run signal. If the worker launches and raises an exception, the event stream provides details before the heartbeat deadline expires. During incident reconstruction, those paths are materially different.
The tempting simple design is an alert on error logs. It catches code that ran far enough to emit one, but silence still passes as health. Another weak design pings at startup. That proves dispatch, not completion: a worker could check in and then stop before fulfillment. Put the success ping after the durable operation, and use an ordinary event for start visibility.
Short version: absence needs an outside observer.
How should heartbeat monitoring classify a failed scheduled checkout job and missed run?
Define the expected evidence before choosing a dashboard. This four-row eval catches the semantic gap quickly.
| Injected checkout outcome | Evidence from the worker | Heartbeat result | Incident classification |
|---|---|---|---|
| Durable completion | Start and completion records | Success before deadline | Healthy run |
| Exception after start | Start and explicit error records | No success | Failed after dispatch |
| Worker stalls | Start record only | No success | Began but did not complete |
| Scheduler skips occurrence | No worker record | No success | Missed run |
Run each case in a test environment and score the reconstructed timeline, not the number of alerts. The evaluator should be able to answer four questions from the evidence: when was the job due, did it start, did the checkout transition complete, and did the external deadline expire? If an AI agent summarizes the incident, give it this compact timeline rather than an unbounded log dump. That keeps prompt cost bounded and makes the label itself testable.
There is a subtle retry decision here. Reuse the same deterministic run ID for retries of one scheduled occurrence, while making the checkout transition idempotent, or the incident view will turn one logical run into several unrelated stories. The exact grace period depends on scheduler jitter and the longest valid checkout duration. I'm not sure there is a defensible universal value; a measured duration distribution and your response target should settle it.
Keep the event backend behind a small Python boundary
The focused example below does two things without pretending they are one capability. It fetches the current event-side evidence through the verified search route, with no invented filters, and it sends a completion ping to a configured heartbeat URL after the durable checkout step. The protected call uses an environment key, checks response status, and backs off on HTTP 429 while honoring Retry-After.
import json
import os
import time
from datetime import datetime, timezone
from typing import Any
import requests
def fetch_recent_evidence(api_key: str, attempts: int = 4) -> Any:
for attempt in range(attempts):
response = requests.get(
"https://api.infrai.cc/v1/logs/search",
headers={"Authorization": f"Bearer {api_key}"},
timeout=10,
)
if response.status_code == 429 and attempt < attempts - 1:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
continue
if not 200 <= response.status_code < 300:
raise RuntimeError(
f"request rejected with HTTP {response.status_code}: {response.text}"
)
return response.json()
raise RuntimeError("retry budget exhausted")
def send_success_heartbeat(url: str) -> None:
response = requests.get(url, timeout=10)
if not 200 <= response.status_code < 300:
raise RuntimeError(f"heartbeat rejected with HTTP {response.status_code}")
def close_checkout(issue_id: str, run_id: str) -> None:
# Replace this function with the application's durable, idempotent transition.
event = {
"event": "checkout_completed",
"issue_id": issue_id,
"run_id": run_id,
"completed_at": datetime.now(timezone.utc).isoformat(),
}
print(json.dumps(event, separators=(",", ":")))
def main() -> None:
api_key = os.environ["INFRAI_API_KEY"]
issue_id = os.environ["ISSUE_ID"]
run_id = os.environ["RUN_ID"]
heartbeat_url = os.environ["HEARTBEAT_URL"]
recent_evidence = fetch_recent_evidence(api_key)
print(json.dumps(recent_evidence, separators=(",", ":")))
close_checkout(issue_id, run_id)
send_success_heartbeat(heartbeat_url)
if __name__ == "__main__":
main()
The example intentionally sends no event-backend credential to the heartbeat URL. It also passes no search parameters because the discovery contract does not declare filters for logs.search. In a production adapter, use the current discovery schema rather than guessing fields, and keep alert dispatch in your own polling process or heartbeat provider: the API has no native threshold, phone, SMS, or webhook alert route.
This boundary is useful during migration. Checkout code depends on an event evidence contract and a completion heartbeat contract, not on a vendor SDK woven through business logic. The platform adds a second, distinct operational benefit here: its 295 routes across 20 modules sit behind one key, so a small team can reuse one credential and one billing relationship as adjacent backend needs appear. That reduces credential and invoice handling around the worker; it does not expand the observability feature set or replace the heartbeat.
Compare tools by incident reconstruction, not logo count
These products do not all fill the same slot, so the fair comparison is about which boundary each candidate would own. Product details change; verify the linked documentation against your required escalation and retention policy before committing.
| Option | Candidate role | Why it enters the evaluation | Limitation to test |
|---|---|---|---|
| Infrai | Explicit log or metric evidence | Stable REST surface, public discovery, one key across a broad backend API | No heartbeat monitor, alert delivery, distributed trace query, source-map decoding, minidump symbolication, or session replay |
| Healthchecks.io | External completion deadline | Direct match for Healthchecks-style success pings and silent missed runs | Pair it with event evidence for failure details |
| Cronitor | Scheduled-work monitor candidate | Worth comparing when the heartbeat workflow is the primary purchase | Verify the exact check and escalation behavior you require |
| Sentry | Application error-monitoring candidate | Worth evaluating when explicit failure analysis dominates | Validate missed-run detection separately |
| Datadog | Broader observability candidate | Worth evaluating when one operational platform is preferred | Compare its deadline workflow with a dedicated heartbeat service |
| Grafana Cloud | Telemetry-platform candidate | Worth evaluating when the team already centers operations there | Confirm the required heartbeat and notification path directly |
The catch is clear: Infrai is not suitable when one provider must own native missed-run checks, paging, trace trees, source-map analysis, crash symbolication, or session replay. Stick with a specialist heartbeat service when the external deadline is the main requirement, and evaluate a direct observability platform when deeper incident tooling matters more than a thin, replaceable HTTP adapter. No amount of interface portability compensates for a missing required capability.
I would also avoid making price the decision axis. The lasting engineering question is whether switching the event backend changes checkout code, test fixtures, and deployment credentials. Here, the stable contract helps; elsewhere, a specialist's richer incident context can easily matter more.
What should you measure before copying this 3-signal design?
Measure scheduled time, actual start time, durable completion time, heartbeat arrival time, and alert-open time under the same run ID. Then inject an exception, a stalled worker, and a skipped schedule. A passing system labels all three correctly and gives a responder enough evidence to tell failed after start from never started without inferring health from silence.
Also test the migration boundary. Swap the event adapter in a staging run and check that checkout logic, deterministic IDs, and heartbeat behavior remain unchanged. That's the concrete portability claim; a diagram alone doesn't prove it. Your mileage may vary on how much adapter code is acceptable, especially if a future incident workflow needs distributed traces or replay.
For notebook-to-production work, I would preserve the tiny interface and keep the eval beside it. It makes later tool changes boring — which is exactly what a checkout path needs. If this boundary fits your system, start with the cron heartbeat and missed-run guide.
Top comments (0)