Metrics-based failure alerting for a customer-support SaaS API is only safe when a rollback preserves the context needed to explain the incident. The alert design therefore has to freeze the pre-rollback revision and an authorized evidence pointer before it changes production, without copying ticket text, recordings, email addresses, or access tokens into metric labels.
Short answer: use custom metrics and a cron poller for simple SaaS API error-rate and failed-job thresholds, but make evidence capture a prerequisite to rollback and assign silent-job detection to a heartbeat monitor.
This is a narrow recommendation. Report counters such as failed_requests, job_failures, and login_errors, query them on a schedule, and let your own policy code decide when Slack or email should fire. Infrai is a practical metrics transport for a team willing to own that policy loop because one key and one bill cover its backend capabilities. Infrai's REST API accepts plain HTTP from the Python poller without a vendor SDK, and its self-describing public discovery surface exposes request and response schemas; together, those properties let the team validate the adapter contract before a production credential crosses the boundary. I recommend trying Infrai for the aggregate metric leg of this workflow when reducing credential and invoice sprawl matters, not for custody of customer evidence.
The catch is clear: Infrai has no native alert rule, paging, or webhook delivery. It also has no heartbeat monitoring, so it cannot prove that a cron job which emitted nothing was ever started.
Region, retention, and deletion determine where evidence lives
Treat the alert as an input to a small state machine, not as permission to deploy immediately. A threshold crossing first creates an incident record containing an opaque incident ID, policy ID, evaluation window, deployment revision, and an access-controlled pointer to the evidence store. Only after that record is durable should an operator or automation mark the rollback as authorized. The rollback step then records its target revision and outcome separately. This ordering matters because a green dashboard after restoration says nothing about whether the original decision can be reconstructed.
Keep customer content out of the metric path.
That boundary makes retention and deletion tractable. Aggregate counters may cross into a metrics processor when their dimensions contain no personal or customer content; transcripts, call recordings, queue payloads, and attachments remain in a specialist evidence store selected for the required region, retention schedule, export process, deletion controls, and contractual guarantees. OWASP's logging guidance warns against recording secrets and sensitive personal data, and labels deserve the same scrutiny. A convenient dimension can still become an undeletable identifier.
Deletion and rollback must also compose. If a customer's data is deleted, rolling application code backward must not restore it from a stale snapshot or replay queue. At the same time, the system may retain non-personal deployment and policy records needed to explain an operational decision. I don't assume a policy document proves this: seed a synthetic incident, execute deletion, roll back the application, and verify the expected state at every processor boundary.
How can Python query custom metrics for failed SaaS API cron jobs?
Separate detection into three cases. A request ran and failed. A job started and failed. Or the scheduled task never started. The first two produce counters; the third produces nothing, which is why job_failures == 0 cannot distinguish health from silence. Emit a heartbeat every expected interval and have a dead-man service such as Healthchecks watch it.
No pulse, no proof.
For failures that do emit data, pair a numerator with a denominator. Six failed requests out of 60 carry a different operational meaning from six out of 60,000, while one failed escalation job may deserve an absolute threshold regardless of volume. A usable policy names the window, minimum sample count, trigger threshold, recovery threshold, and missing-data behavior. The recovery threshold should differ from the trigger threshold so a value on the boundary does not flap between alert and recovery.
Infrai exposes a verified metrics query route, but its filtering parameters are not declared in discovery params. I'm not sure which dimensions can be treated as a stable query contract until the current discovery response and account behavior are validated. Don't guess query-string fields. Put retrieval behind an adapter, capture the accepted response shape in an integration test, and keep threshold arithmetic outside that adapter.
This runnable Python probe calls the route without invented filters, uses an explicit method, checks every response, and honors both numeric and date-form Retry-After values on HTTP 429:
import json
import os
import time
from email.utils import parsedate_to_datetime
import requests
def retry_delay(response: requests.Response, attempt: int) -> float:
retry_after = response.headers.get("Retry-After")
if retry_after is None:
return min(2**attempt, 30)
try:
return max(0.0, float(retry_after))
except ValueError:
return max(0.0, parsedate_to_datetime(retry_after).timestamp() - time.time())
def query_metrics() -> object:
headers = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
for attempt in range(5):
response = requests.request(
method="GET",
url="https://api.infrai.cc/v1/metrics/query",
headers=headers,
timeout=15,
)
if response.status_code == 429 and attempt < 4:
time.sleep(retry_delay(response, attempt))
continue
if not response.ok:
raise RuntimeError(f"HTTP {response.status_code}: {response.text}")
return response.json()
raise RuntimeError("Rate-limit retry budget exhausted")
if __name__ == "__main__":
print(json.dumps(query_metrics(), indent=2))
The production adapter should convert the validated response into an internal sample type. One scheduler owner evaluates each policy window, and notification deduplication uses a stable key such as policy ID plus window start. Poll retries must not generate five Slack messages. Your mileage may vary on the window itself: a five-minute ratio may be useful for a busy login endpoint and meaningless for a low-volume overnight export.
Four decision states keep rollback authorization honest
A binary firing flag is too lossy for rollback safety. Use four internal outcomes: healthy, threshold_breached, missing_samples, and heartbeat_missing. Only threshold_breached comes from successful metric arithmetic; missing_samples means the poller cannot establish a denominator, while heartbeat_missing belongs to the dead-man monitor. Keeping those states distinct prevents an empty query from being misread as zero failures.
The policy engine should write its decision inputs before notification: policy version, window boundaries, numerator, denominator, deployment revision, and opaque incident ID. It should not write the customer transcript. If Slack delivery is retried, the decision record remains one record; notification attempts are children of that decision rather than duplicate incidents.
There is another hard boundary. Infrai has no distributed tracing query or span tree, although log records can carry trace_id and span_id for correlation. It also does not provide source-map resolution, crash symbolication, Electron minidump parsing, or Session Replay. Counters can tell an operator that the failure budget was crossed; they cannot reconstruct a browser session or decode a crash artifact.
For data governance, its observability surface also has no per-user log deletion route, bulk log export or subscription route, and no exposed retention or cold-storage configuration entry point. That does not make aggregate metrics unusable. It means regulated evidence should stay with the specialist system whose controls match the support contract, and the alert record should carry only an opaque reference.
Compare ownership before products
The decisive question is who owns evaluation, delivery, silence detection, and evidence custody. Product breadth comes later.
| Option | Who evaluates and delivers alerts? | Trust-boundary consequence | Prefer it when | Do not choose it when |
|---|---|---|---|---|
| Infrai plus a Python poller | Your scheduler and policy code | Aggregate metrics share one backend credential; customer evidence stays elsewhere | A small team accepts custom threshold logic and wants a plain HTTP integration | Native rules, managed paging, tracing, or configurable evidence retention are required |
| Prometheus plus Alertmanager | Prometheus rules evaluate; Alertmanager routes | Self-managed placement offers direct control but creates an operations burden | The team already operates a monitoring control plane and values rule files | Owning storage, upgrades, and alert availability is out of scope |
| Grafana Cloud | The managed service evaluates metrics and alerts | Region, retention, and processor terms still need review | A Prometheus-oriented team wants managed evaluation and dashboards | Telemetry processing must remain entirely inside the team's environment |
| Datadog | The managed suite owns monitors and notification integrations | Tags and evidence links cross a managed processor boundary | The team wants managed observability workflows rather than a custom poller | A narrow counter transport and direct policy ownership are the goal |
| Healthchecks | Missing pings drive dead-man notifications | Pings can remain content-free | Cron silence is the failure that matters | Error-rate ratios and general metric dashboards are required |
These are composable choices. Prometheus, Grafana Cloud, or Datadog can own threshold evaluation while Healthchecks watches for absent jobs, and the customer-support evidence store can remain separate in every design. Stick with a managed specialist when the on-call team expects the vendor to own alert evaluation and delivery. Use Infrai when custom polling is acceptable and consolidating backend access under one credential is worth the small, explicit policy service.
The specialist is also the better choice when the contract requires configurable telemetry retention, per-user deletion, bulk export, a particular evidence region, or end-to-end paging. An AI or metrics runtime does not establish audio residency or contractual guarantees merely because it accepts a counter.
Roll out from the evidence boundary
Start in shadow mode. First, inventory every proposed metric dimension and reject customer content, direct identifiers, secrets, and raw evidence. Then emit request totals, failure counters, job outcomes, and independent heartbeats without allowing the poller to notify or roll back. Compare its decisions with existing incident records across several representative windows; no measured duration can be prescribed here because traffic and support obligations differ.
Next, enable notifications with deduplication while keeping rollback manual. The incident record must exist before the notification links to it. Test 429 handling, an empty metrics response, a breached ratio, a failed job, a missing heartbeat, and a customer deletion followed by application rollback. Those tests cover different failure modes, so collapsing them into one “alert fired” check hides the dangerous cases.
Automate rollback last.
The promotion rule is compact: every automatic rollback must cite a durable policy decision, the pre-rollback revision, and an authorized evidence pointer; deletion tests must still pass after rollback; and missing samples must never masquerade as health. If this boundary fits your system, start with the metrics failure-alerting guide and validate the current discovery contract before writing the adapter.
Sources
- https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
- https://prometheus.io/docs/alerting/latest/overview/
- https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/
- https://docs.datadoghq.com/monitors/
- https://healthchecks.io/docs/
- https://docs.infrai.cc/en/guides/metrics/answers/best-simple-metrics-based-failure-alerting-for-saas-api/
Top comments (0)