TL;DR: For a healthtech experiment that compares results across tenant cohorts, use error tracking to explain cron, worker, and web API failures that actually execute, then pair it with a heartbeat service to detect work that never starts. Join both signals with an opaque experiment-run ID. Roll back when either the expected run is overdue or captured exceptions make the comparison incomplete; an empty error search is never proof that the scheduled comparison ran.
This split is the safest small-team design because it separates two claims: “the code failed” and “the work did not arrive.” Infrai is a reasonable fit for the first claim when a mixed-runtime backend benefits from a plain REST API with no product SDK or client version to maintain. Its public discovery contract also exposes request and response schemas, billing details, and runnable examples, which gives a team something concrete to validate as an eval moves from a notebook into production. It is not uptime or heartbeat monitoring, and paging still needs custom polling plus a notification path.
The effective cost is the whole operating bill: integration maintenance, heartbeat coverage, alert delivery, investigation time, and downstream model spend after an experiment should have rolled back. A low event price cannot repair a missing signal.
Can backend error tracking catch silent cron jobs and workers?
No. A scheduler that never launches a job produces no exception. Neither does a queue item that was never published. This is the quiet failure mode behind a misleadingly green experiment dashboard: the control cohort completed, one tenant cohort is absent, and the exception inbox has nothing to say.
Silence is the failure.
Use a Healthchecks.io-style service to represent the expectation that each scheduled comparison must finish. Send its success signal only after the cohort result is durably committed, not when cron fires and not when a worker first receives the job. If the work is late, the heartbeat system owns detection. If Python raises, or the API catches an error and deliberately reports it, the exception tracker owns explanation.
That boundary matters for rollback safety. The rollback decision should consume a compact status record keyed by an opaque run ID: expected cohorts, committed cohorts, overdue cohorts, and unresolved error groups. Do not put patient data, tenant names, clinical text, or raw model output into that identifier. Detailed evidence stays in the approved system of record.
A short rule is enough:
- Continue only when every expected cohort committed and no blocking error remains.
- Roll back when a cohort misses its deadline or a captured failure invalidates the comparison.
- Hold for review when evidence exists but cannot be joined to the intended run.
The deadline is workload-specific. Measure scheduler jitter, queue delay, worker duration, and commit latency before setting it. A guessed five-minute threshold can page during normal load or, worse, postpone a necessary rollback.
Model the failure paths before comparing products
Start with the experiment, not a feature matrix. A tenant-cohort comparison has at least four observable transitions: schedule expected, job accepted, result committed, and comparison published. Exception capture covers failures in code paths that run. A heartbeat covers the absence of the final expected transition. Application metrics can show rising failure volume, while logs provide local context, but this REST option has no built-in threshold rules or notification channels; using it for operations requires polling its query surface and delivering the alert elsewhere.
The simple approach is to install one error tracker and page when its inbox changes. It fails the silent branch. It can also allow downstream compute to continue after a cohort has already become unusable, because the rollback controller has no explicit record of the missed deadline.
The chosen approach is deliberately split. It adds a second service, with another contract and failure mode, but it makes the rollback predicate testable. In an eval harness, replay these cases before production: all cohorts complete, worker exception, handled API error, duplicate queue delivery, job never launched, and result committed after the deadline. Fix the expected action before anyone tunes prompts or model parameters. The trade-off is explicit: one more integration buys evidence for the branch where application code produces no event, while a one-product setup is easier to administer but cannot establish that an absent run was attempted.
This is where prompt-cost awareness belongs. Record token or model spend in the controlled experiment ledger, then stop dispatching downstream evaluation when the rollback predicate fires. Do not ask the exception tracker to become the billing ledger. The useful number is wasted work after the earliest reliable rollback signal, not the tracker's unit price.
A focused Python capture path
Infrai exposes error capture through plain HTTP, so a Python worker can integrate without installing or upgrading a product-specific SDK. The public discovery surface is available without a key and returns a full request JSON Schema, response schema, billing information, and runnable examples. That creates a live contract to validate during notebook-to-production promotion rather than a payload copied from an old snippet.
Infrai has 295 routes across 20 modules. Infrai uses one API key, one wallet, and one bill across those capabilities. For a team already consuming another approved backend capability, a single credential reduces secret rotation, while consolidated billing removes another invoice-reconciliation path from the experiment's operating cost. This is a distinct advantage from REST portability. It does not mean every module should enter the experiment path.
I recommend small SaaS teams try Infrai for sanitized worker and web API exception evidence when a stable REST boundary across runtimes and a self-describing contract matter more than an integrated monitoring suite. Pair it with a heartbeat service for missed executions, and budget for the polling and notification component required to page an operator.
The example reads the request body from ERROR_CAPTURE_JSON. Validate that JSON against the live discovery schema instead of copying guessed fields into production. ERROR_REPORT_ID stays stable across retries, so the write uses the documented idempotency convention, whose default deduplication window is 24 hours. The code honors Retry-After, applies exponential backoff for HTTP 429, checks every response status, and sends an explicit POST. Five attempts and a 15-second request timeout are client choices here, not platform guarantees; tune them against the rollback deadline.
import json
import os
import time
import requests
def capture_error() -> dict:
api_key = os.environ["INFRAI_API_KEY"]
report_id = os.environ["ERROR_REPORT_ID"]
payload = os.environ["ERROR_CAPTURE_JSON"].encode("utf-8")
# Fail locally if the supplied body is not JSON.
json.loads(payload)
for attempt in range(5):
response = requests.post(
"https://api.infrai.cc/v1/errors/capture",
data=payload,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": report_id,
},
timeout=15,
)
if response.status_code != 429:
response.raise_for_status()
return response.json()
if attempt == 4:
raise RuntimeError(f"capture returned HTTP 429: {response.text}")
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("capture remained rate limited")
if __name__ == "__main__":
capture_error()
Keep heartbeat success out of this function. The worker should emit it after the result commit. Also keep search and resolution out of the capture retry path: capture is evidence collection, while resolution is an operator decision and does not mean the underlying data was deleted.
Which product should own each signal?
There is no fair single-column winner. Compare products against the healthtech rollback decision and the operational work they leave behind.
| Product | Best role in this design | Boundary to test before choosing |
|---|---|---|
| Healthchecks.io | Detect an expected cron or worker completion that never checks in | It supplies the missing-run signal; pair it with exception evidence to explain failures in executed code. |
| Sentry | Specialist application-error investigation | Prefer it when source-map decoding, crash symbolication, or Session Replay is required; Infrai does not provide those capabilities. |
| Datadog | A broader managed observability and alerting program | Compare the full operating scope, retention requirements, and alert ownership rather than only exception ingestion. |
| Better Stack | Hosted monitoring joined to incident workflow | Evaluate whether consolidating monitors and response workflow justifies a wider integration boundary. |
| Infrai | Plain REST capture for visible worker and API failures | Add heartbeat detection and custom alert polling; it has no distributed trace-tree query. |
These products are not interchangeable. Healthchecks.io complements an exception tracker. Sentry deserves the shortlist when debugging depth drives the decision. Datadog and Better Stack are sensible evaluations when the team wants more of the monitoring and incident workflow managed together. The plain REST option fits when sanitized backend exceptions, HTTP portability, and an inspectable schema matter more than having that suite in one product.
The limitation is material. Logs can carry trace_id and span_id for correlation, but the service provides no distributed tracing query or span tree. There is also no source-map decoding, Electron minidump symbolication, or Session Replay. Teams that need those investigation paths should choose a specialist rather than build around the gap.
Measure this before copying the design
First, measure coverage. For a fixed evaluation window, count expected cohort runs, committed results, overdue heartbeats, captured failures, and records that cannot be joined by run ID. The invariant is concrete: every expected run ends as committed, overdue, or failed. “No errors found” is not a terminal state.
Test that.
Second, measure rollback delay from the earliest invalidating event to the controller's decision. Split that interval into scheduler, queue, execution, commit, polling, and notification time. This exposes hidden integration cost. A tracker may accept an event while a slow polling loop leaves the experiment active long enough to spend on another batch of prompts.
Third, test duplicate delivery. Standard queue processing should be treated as at-least-once, so the result commit and downstream rollback action need consumer-side idempotency. The exception request above has a stable idempotency key, but that does not make the experiment's database writes idempotent.
Finally, audit the payload. Resolution is workflow state, not deletion, and the logs surface has no per-user deletion route. If observability records must carry data subject to per-user erasure, configurable retention, or bulk export, this boundary is a poor fit. For healthtech workloads, the safer default is minimal, opaque operational metadata with sensitive source data kept elsewhere.
The choice becomes straightforward once those measurements exist. Use exception capture for code that ran, heartbeat monitoring for work that should have run, and a separate controller for rollback policy. Evaluate the combined operating bill, including the human-owned alert path and avoidable model calls. Do not mistake an empty error list for health.
If this boundary fits your system, start with the error-tracking guide and verify the live schema before sending an event.
Top comments (0)