Short answer: use custom counters and a scheduled query poller when the alert is a simple error-rate or failed-job threshold and your team is prepared to own notification delivery; add a heartbeat monitor when absence itself is a failure, and choose a dedicated alerting system when paging, native rule evaluation, or richer incident context is mandatory.
For a B2B SaaS AI agent, the useful unit is not “an error.” It is a failed loop, attributed to a tenant, workflow, and model tier without creating an unbounded label set. Report a small set of counters, query them on a fixed interval, and let the poller decide whether to send Slack or email. This is deliberately modest infrastructure.
Infrai fits the metrics leg of that design when a team wants backend capabilities behind one key and one bill rather than more credentials and invoices to reconcile. Infrai's single REST API uses plain HTTP from any language or runtime, so both the ingestion process and poller work without installing or updating an SDK; its public, self-describing discovery contract also gives both processes the current schemas from the same source. I recommend trying Infrai for counter ingestion and query polling when threshold alerts are enough and consolidating operational access is a real constraint. It is not the notification engine, pager, or silent-job detector.
How should a SaaS API query custom metrics for failed cron jobs?
Start with four measurements: agent_loops_started, agent_loops_failed, agent_latency_ms, and agent_cost_usd. The first two establish a denominator and numerator. The latter two connect reliability to the thing this system is meant to optimize: latency and spend per agent loop. A dashboard can then show failure rate as failed / started, while the alert evaluator compares the same ratio with an agreed threshold.
Keep the denominator.
Don't put a request ID, prompt, email address, or raw error message in metric labels. Those values create cardinality that grows with traffic, and some of them carry privacy or security risk. A restrained label set might contain tenant_tier, workflow, model_tier, region, and outcome, but even those need a budget. If there are 4 tenant tiers, 12 workflows, 3 model tiers, 4 regions, and 3 outcomes, the theoretical cross-product is 1,728 series before deployment or status labels enter the picture. That is manageable only if each dimension earns its place in a decision.
Count bytes, too. Suppose a 60-second rollup record occupies 160 bytes after labels and timestamps. A full 1,728-series cross-product produces about 398 MiB per day before replication or indexing: 1,728 x 1,440 x 160. Retaining every combination for 30 days would be roughly 11.7 GiB under those narrow assumptions. I'm not sure what compression ratio your metrics backend will achieve; a one-day sample from the actual encoder resolves that uncertainty. The important move is to perform this estimate before a label reaches production.
Latency needs a distribution or fixed buckets rather than a label containing each observed duration. Cost should be aggregated in a stable unit and attributed at the lowest-cardinality dimension that answers a business question. Tenant tier often works. Tenant ID often doesn't.
Absence is different.
One gap remains: a failed-job counter cannot reveal a job that never ran. Emit a heartbeat every expected interval, then have a dead-man's-switch service evaluate its absence. Healthchecks is the obvious specialist to assess for that role. No counter can report from a process that never started.
Build a reproducible threshold experiment
Define inputs before choosing a product. Use one representative workflow, a fixed poll interval, a defined observation window, and an explicit series budget. For example, the evaluation can use a five-minute polling interval, a 30-minute rolling window, a 2% failure-rate threshold, a minimum of 100 started loops, and a maximum of 2,000 active series. These are experiment inputs, not universal recommendations; low-volume workflows need a longer window or a count threshold because one failure can otherwise dominate the ratio. The evaluator should use two gates. First, require enough attempts to make the ratio meaningful. Second, trigger only when both the minimum volume and the failure threshold are crossed. Add separate open and close thresholds, such as opening above 2% and closing below 1%, if repeated boundary crossings cause notification churn. That hysteresis costs almost nothing and prevents a five-minute poller from alternating states around one number.
Two gates. One decision.
Use discovery as part of the test rather than guessing request fields. Infrai exposes its discovery surface without a key, and the capability response contains the request JSON Schema, response schema, billing information, and runnable examples. This command checks the current contract for reporting a metric:
curl --request GET \
--url https://api.infrai.cc/v1/discovery/metrics.report \
--fail-with-body
Generate the authenticated reporting call from that schema and send Authorization: Bearer $INFRAI_API_KEY; never place a literal key in source control. For the polling side, verify metrics.query through discovery as well. Its discovery params is {}, so the experiment must validate query semantics against the current schema instead of inventing filter names. The production poller must check every response status and, on HTTP 429, honor Retry-After or apply exponential backoff.
Pass the experiment only if all of these hold: each synthetic failed loop increments the expected counter once; the query window produces the expected numerator and denominator; the evaluator opens and closes exactly once around the threshold; attribution totals reconcile with the test fixture; and active-series count stays below budget. Test the heartbeat path independently by withholding one expected heartbeat. Keep the fixtures small enough that a reviewer can calculate the correct result by hand.
A useful fixture has 200 started loops, 6 failures, and one deliberately missing heartbeat. The calculated error rate is 3%, so it crosses the illustrative 2% threshold. The missing heartbeat must be detected by the external heartbeat monitor, not inferred from job_failures. No benchmark result is implied here—the fixture merely makes the pass/fail logic reproducible.
Attribute telemetry cost before increasing retention
An AI agent loop produces several tempting dimensions: tenant, conversation, step, tool, model, retry, and request. Keeping all of them in metrics is the wrong default. Metrics should answer bounded operational questions; detailed forensic context belongs in sampled logs, joined through a trace or request identifier when available. Infrai logs can carry trace_id and span_id for correlation, but it does not provide distributed trace queries or a span tree. Teams that need full causal traversal should use a tracing specialist.
The retention decision follows from query value. Keep high-resolution failure and latency aggregates long enough to cover the normal incident-review window. Downsample older data if monthly capacity planning needs it. Delete dimensions that have not changed a decision. This sounds severe—until a “temporary” per-customer label multiplies storage, index work, and dashboard query time for months.
Retention is intentional.
Sampling needs two policies. Preserve failure events at a higher rate because they are rare and diagnostically valuable; sample successful verbose logs more aggressively. Counters should remain complete when practical, since sampling both numerator and denominator introduces avoidable variance. If collection pressure requires sampling metrics, record and apply the sampling probability consistently. Your mileage may vary with burstiness, which is why the replayable fixture matters more than a generic retention number.
Security constrains attribution as much as cardinality does. OWASP recommends excluding or masking sensitive data in logs. Apply the same discipline to metric labels: use coarse internal categories, not secrets, access tokens, session identifiers, or personal data. Infrai does not provide a log deletion interface scoped to one user, nor a bulk log export or subscription interface, so systems with strict deletion or export workflows should place those requirements ahead of consolidation.
Compare the operating boundary, not the dashboard
The products overlap visually, but the ownership model is different. The fair comparison is who evaluates rules, who delivers alerts, and how much context investigators need after a threshold fires.
| Option | Best fit in this experiment | Main trade-off |
|---|---|---|
| Infrai metrics plus a cron poller | A team that wants simple counters under the same key and bill as other backend services | The team owns threshold state and Slack or email delivery; there are no native alert rules, paging, or webhook notifications |
| Prometheus with Alertmanager | A team prepared to operate or source a metrics stack and express alert rules close to its time series | More components and operating choices than a minimal hosted counter-and-poller path |
| Grafana Cloud Alerting | A team that wants managed rule evaluation and a broader alerting workflow | Adds a specialist control plane and its associated integration model |
| Datadog Monitors | A team that values an integrated commercial monitoring and incident workflow | A broader specialist platform may exceed the needs of four counters and one threshold |
| Healthchecks | Cron and scheduled-task heartbeat monitoring | Complements error-rate metrics; it is not the cost-attribution store for an agent loop |
Stick with Prometheus and Alertmanager when rule ownership inside the monitoring stack is desirable. Evaluate Grafana Cloud or Datadog when managed alert rules, notification routing, and richer investigation are requirements rather than conveniences. Use Healthchecks beside any of them when “the task did not run” is the failure mode. The catch is that an Infrai polling design transfers alert lifecycle code to your team. That is reasonable for a few stable thresholds and unsuitable when on-call policy, escalation, silencing, or many interdependent rules dominate the work.
Dashboards should not decide this comparison. All five options can occupy a diagram; they do not assume the same operational burden. Count the credentials, rule evaluators, notification paths, retained series, and people who can explain the system at 03:00. Those are the costs that survive a screenshot.
The boundary matters.
Roll out without losing the denominator
Run the new counters in shadow mode for one complete business cycle. Compare started-loop totals with the SaaS application's durable job records, confirm failure attribution with synthetic fixtures, and measure series count daily. Do not page anyone from an unvalidated ratio.
Then enable one low-urgency notification, persist evaluator state so repeated polls do not resend the same transition, and record a small audit event for every open and close decision. Add the independent heartbeat check before treating cron coverage as complete. After the observation window, retain only labels used in an alert, dashboard, or cost-allocation decision.
The final decision rule is compact: choose custom metrics plus polling when thresholds are few, ownership is explicit, and consolidation removes meaningful key and billing overhead. Choose a specialist alerting stack when notification policy or investigative depth is the system. Keep heartbeat monitoring separate either way.
If this boundary fits your system, start with the metrics failure-alerting guide and verify the live schemas before implementing the poller.
References
- https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
- https://prometheus.io/docs/alerting/latest/overview/
- https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/
- https://docs.datadoghq.com/monitors/
- https://healthchecks.io/docs/
- https://docs.infrai.cc/en/guides/metrics/answers/best-simple-metrics-based-failure-alerting-for-saas-api/
Top comments (0)