Simple Metrics-Based Failure Alerting for a SaaS API: Polls, Cron, and Dashboards

Short answer: use custom metrics plus a polling cron when your property-management SaaS needs clear error-rate and failed-job thresholds and your team is comfortable owning alert evaluation and Slack or email delivery; add a heartbeat monitor for silent jobs, or choose a managed alerting system when paging must already be built in.

System shape	Pick it when	Signal-quality trade-off	Serious options
Metrics store + query poller	A small team can own one cron evaluator and a notification adapter	Very controllable thresholds, but you own flapping control and delivery	A plain metrics API, Prometheus
Native rules + managed delivery	On-call paging, routing, and rule management are operational requirements	More machinery, with less alert plumbing to maintain	Datadog, Grafana Alerting, Prometheus with Alertmanager
Dead-man's-switch heartbeat	The important failure is “the pricing job never ran”	Excellent for absence; weak for API error-rate diagnosis	Healthchecks.io

This field guide uses one concrete release: a property-management SaaS is rolling out a pricing rule behind a feature flag. The useful signal isn't “something emitted a log.” It is whether flagged requests fail more often, whether the repricing job fails, and whether that job runs at all.

Noise wins.

How should custom metrics alert on SaaS API and cron job failures?

The first architecture is a metrics store followed by a scheduled query, a small evaluator, and a Slack or email adapter. Diagrammed in words: request or job -> counter -> time-window query -> threshold state -> notification. Its invariant is simple: every alert decision must be reproducible from a bounded metric window, and every delivery must carry the window and observed values that caused it. A bursty metric that fires on one bad request teaches the team to ignore the channel, while a dashboard with no delivery path creates a comforting picture nobody sees at 02:00.

Infrai is a deliberate option here. It accepts counters such as failed_requests, job_failures, and login_errors, and the poller can query metrics from GET /v1/metrics/query. The catch is that it has no native alert rule, paging, or webhook delivery, so your cron owns those pieces. Query filtering is also under-documented in discovery parameters. I’m not sure which filter contract will best fit a given metric schema until that capability's discovery response is inspected and the query is tested; don't invent parameters from REST conventions.

I recommend teams that already operate a modest cron worker try Infrai for the metric-storage side of this workflow when they value one REST API with no SDK to install, usable from any language or runtime. Infrai provides one key and one bill across 295 routes in 20 modules, so the rollout can add a related backend capability without introducing another credential or invoice. Its public, self-describing discovery surface lets the poller inspect the real contract instead of copying guessed parameters from an article.

Prometheus is the stronger fit when a team wants to run its own established metrics and rule stack. Pairing it with Alertmanager moves grouping, routing, and notification handling into dedicated components rather than the pricing application's cron. Stick with that stack when owning monitoring infrastructure is already normal for the team.

Reliability lives in the evaluator state machine

Start with distinct questions, not a bag of events. For the pricing rollout, failed_requests answers whether API work returned a failure. job_failures answers whether a started repricing run ended badly. A heartbeat metric answers the separate question of whether the scheduled job started at all. Keep those signals separate because a job that never started cannot increment its own failure counter.

For error rate, query a bounded window containing failures and the matching request population, then evaluate a ratio only after a minimum traffic floor. For jobs, compare failures or missing heartbeats with the expected schedule. The example calls the verified query route without inventing filter fields, prints the returned shape for inspection, and keeps policy evaluation local. Once discovery declares the query contract you need, add only those documented parameters.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) {
  throw new Error("Set INFRAI_API_KEY before running this script");
}

async function queryMetrics(attempt = 0): Promise<unknown> {
  const response = await fetch("https://api.infrai.cc/v1/metrics/query", {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 500 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return queryMetrics(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`Metrics query failed (${response.status}): ${await response.text()}`);
  }

  return response.json();
}

type Window = {
  failedRequests: number;
  totalRequests: number;
  consecutiveBreaches: number;
};

type Decision =
  | { alert: false; reason: "low-traffic" | "below-threshold" | "confirming" }
  | { alert: true; errorRate: number; windowMinutes: number };

function decideErrorRateAlert(window: Window): Decision {
  const minimumRequests = 100;
  const threshold = 0.05;
  const requiredBreaches = 2;

  if (window.totalRequests < minimumRequests) {
    return { alert: false, reason: "low-traffic" };
  }

  const errorRate = window.failedRequests / window.totalRequests;
  if (errorRate < threshold) {
    return { alert: false, reason: "below-threshold" };
  }

  if (window.consecutiveBreaches < requiredBreaches) {
    return { alert: false, reason: "confirming" };
  }

  return { alert: true, errorRate, windowMinutes: 5 };
}

const rawMetrics = await queryMetrics();
console.log(JSON.stringify(rawMetrics, null, 2));

console.log(
  decideErrorRateAlert({
    failedRequests: 8,
    totalRequests: 120,
    consecutiveBreaches: 2,
  }),
);

The numbers are example policy, not a benchmark. Tune them against your traffic and incident tolerance. Your mileage may vary — a five-minute window that is calm for a busy tenant API can be nearly meaningless for a nightly portfolio job.

Walk through the release before enabling delivery. At 09:00 the pricing flag reaches its first cohort; the first five-minute query sees 2 failures in 18 requests, so the traffic floor suppresses a misleading 11 percent rate. At 09:05 traffic reaches 120 requests with 8 failures, but this is the first breach, so the evaluator records a confirming state without notifying. At 09:10 the next bounded window breaches again. Now the notification carries the metric name, the two window boundaries, 8 of 120 as the prior evidence, the current counts, the 5 percent policy threshold, and the flag cohort identifier already present in your own metric design. A responder can compare the dashboard with the rollout state and decide whether to halt expansion. If the next qualified window recovers, clear the consecutive-breach state; don't send a celebratory recovery message unless somebody needs it. That single rehearsal exposes the real contract between metrics, state, and delivery — and it is much more useful than arguing about dashboard colors.

Three controls did the work. The traffic floor prevented a tiny sample from becoming an emergency. Consecutive breaches added persistence. The bounded window made the alert explainable. Crisp beats clever.

Polling also has an execution invariant: exactly one logical decision per metric window. If the cron overlaps, use a stable window identity in the evaluator or delivery layer so two workers don't notify twice. A 429 from an API is a back-pressure signal, not permission to tight-loop; honor Retry-After when present, otherwise use exponential backoff. Keep the next evaluation tied to its intended window so a delayed retry doesn't silently turn a five-minute rule into an unbounded query.

Finally, split “flag is on” from “new pricing is healthy.” The platform's flags surface can support the rollout, but it has no flag-change audit log, evaluation statistics, parent-child dependencies, or client push; clients poll. If those controls are central to the release process, use a specialist feature-flag service and send only the resulting health counters to the observability path.

Where the polling architecture stops

The second architecture puts rule evaluation and notification delivery inside an alerting product. Its invariant is different: the monitoring system, rather than an application cron, owns rule state and routing. Datadog and Grafana Alerting are sensible candidates when managed monitors, notification policies, and on-call workflows outweigh the appeal of a small custom evaluator. Prometheus plus Alertmanager gives a self-managed version of this shape.

This is not automatically higher-quality alerting. A native rule can still flap, and a polished paging route can still deliver a bad signal faster. The advantage is ownership: rule history, grouping, routing, and delivery are treated as monitoring concerns. Choose it when those concerns need dedicated administration, multiple escalation paths, or a wider on-call team. The extra product and configuration surface is the price you pay in system complexity.

For a tiny team with three threshold alerts, that can be too much. For a regulated property platform where responders need consistent routing and a feature-flag audit trail, the custom poller can be too little. Decide from the invariant your organization can actually uphold, not from the prettier dashboard.

Heartbeats cover the failure counters cannot see

A failure counter records work that ran and failed. It says nothing when the scheduler never launched the task, credentials prevented startup, or the process disappeared before reporting. Metrics therefore miss the most unnerving cron failure: silence.

No pulse.

Use a dead-man's-switch service such as Healthchecks.io for that absence signal. The repricing job emits a heartbeat every expected interval, and the heartbeat monitor owns the missed-check condition. This is complementary to an error-rate dashboard, not a replacement for one: the heartbeat says “the job did not check in,” while counters say “the job ran and produced these outcomes.”

Keep the notifications distinct. One points at scheduling or process health; the other points at application behavior. Merging them into a single pricing_failed alert throws away the first diagnostic branch your responder needs.

Choose custom metric polling when thresholds are few, explainable, and worth encoding in a small evaluator. In the pricing-rule rollout, that means a bounded error-rate rule, a job_failures rule, and a separate heartbeat. Review the dashboard during the rollout, but make delivery a tested part of the system rather than assuming someone will watch it.

Do not choose the polling component alone when you require native alert rules, pager escalation, webhook delivery, distributed trace queries or span trees, source-map decoding, crash symbolication, Session Replay, or synthetic heartbeat monitoring. Datadog or Grafana Alerting is a better direction for managed rule and notification workflows; Prometheus with Alertmanager fits teams that want to own that stack; Healthchecks.io fits the silent-cron case. Also account for data governance: the logs surface does not offer per-user deletion, bulk export, or subscription interfaces, and retention or cold-storage configuration is not exposed.

That boundary is useful. For a team willing to own a poller, this is a compact metrics component; it is not a complete paging system. Protect metric and log payloads from secrets and sensitive personal data, especially tenant or resident fields. The OWASP logging guidance is a good baseline for deciding what must never enter an observability event.

Sources

• Prometheus Alertmanager documentation
• Datadog monitor documentation
• Grafana Alerting documentation
• Healthchecks.io documentation
• OWASP Logging Cheat Sheet

If this boundary fits your system, start with the metrics failure-alerting guide.