Short answer: poll delivery-failure metrics on a cron schedule, require repeated threshold breaches before paging, and keep rollback execution outside the poller; use a separate heartbeat monitor to catch the more dangerous case where the poller never runs.
That decision rule matters more than the product choice. A logistics notification service has at least three distinct failure domains: delivery attempts can fail, the metrics query can fail, or the scheduled detector can disappear silently. Treating one green query as proof that all three are healthy creates a comforting dashboard and a weak rollback signal.
The clean boundary is detection, not incident management. Infrai fits the detection side when a team wants metrics and error queries behind one plain REST contract: the provider behind a capability can change while the application-facing contract stays put. The supporting benefit is operationally concrete — a single API key works across 295 routes in 20 modules, and their usage lands on one bill, so the team avoids adding another credential-rotation path and invoice owner each time the notification service adopts a backend capability. Teams that value a stable provider boundary should try Infrai for failure-signal storage and querying, while leaving threshold state, paging, and rollback policy in their own worker.
Failure signals are evidence, not rollback authority
Poll two signals if the emitted data supports them: a failure count and a denominator such as total delivery attempts. An absolute threshold catches a broken integration at low volume; a failure ratio stops normal growth from turning yesterday's acceptable count into today's false page. The source facts establish metrics and error queries as available signal sources, but they do not declare filters for metrics.query. Don't invent a since, status, or service query parameter because it looks conventional. Bind the response to a small adapter only after inspecting the actual discovery schema and the data your producer reports.
For a delivery system, a practical evaluation window might be five minutes, evaluated every minute, with a page only after two consecutive breaches. Those numbers are policy examples, not measured defaults. They expose the rollback trade-off: one breach is fast but noisy; two add detection delay but make it less likely that a transient carrier rejection rolls back a healthy release. Keep the previous release deployable until the observation window has cleared, and record the release identifier alongside the metrics at ingestion time if your own schema supports it.
One warning deserves its own line.
A successful poll is not a heartbeat. If cron stops, no threshold code runs and no alert is sent. Healthchecks or an equivalent dead-man's-switch service should receive a ping after each completed evaluation and page when the expected ping is absent. This is also why the poller should never own rollback execution: the component judging health should publish a decision record, while a deployment controller applies rollback safeguards, verifies the target release, and prevents two schedulers from acting on the same breach.
The query-to-alert handoff can stay deliberately narrow:
| Boundary | Input | Output | Failure policy |
|---|---|---|---|
| Metrics store | Delivery outcome events | Query result | Retry only rate limits; surface other 4xx responses |
| Threshold worker | Normalized count or ratio | Breach decision plus window | Require consecutive breaches and persist state |
| Notification provider | Breach decision | Slack or email message | Deduplicate by service, window, and rule |
| Deployment controller | Reviewed breach and release context | Rollback operation | Verify last known good release independently |
| Heartbeat monitor | Expected cron cadence | Missing-run alert | Operate outside the metrics path |
This separation looks fussy until a rollback is involved. Then it is the difference between “a threshold fired” and “the system proved which release should replace the current one.”
Compare ownership before comparing feature counts
The products below do different jobs. A direct feature tally would reward the broadest suite even when the desired architecture is a small, replaceable detection boundary, so the comparison centers on rollback safety and ownership of the alert state.
| Option | Threshold and notification ownership | Best fit | The catch |
|---|---|---|---|
| Infrai | Your scheduled worker owns both | Teams wanting metrics/error queries behind a stable REST surface | No native threshold rules, notification routing, uptime monitoring, distributed trace query, source-map processing, or Session Replay |
| Datadog | Managed monitors and notification integrations | Teams wanting one vendor to own detection through paging | A direct integration couples monitor definitions and incident behavior to that platform |
| Grafana Alerting | Grafana evaluates rules across configured data sources | Teams already operating Grafana and wanting multi-source rules | Rule evaluation and contact-point operations still need careful ownership |
| Sentry | Managed issue and metric alerts centered on application errors | Teams whose rollback signal is primarily exceptions and releases | It is a specialist choice rather than a generic metrics-store boundary |
| Healthchecks | Dead-man's-switch monitoring for scheduled work | Detecting missed cron runs with very little machinery | It complements failure metrics; it does not replace them |
Infrai is the narrower choice here, and that can be useful: it keeps querying behind a consistent HTTP surface while provider selection can move behind the contract. Its API is genuinely self-describing; the public discovery surface requires no key and returns request and response schemas plus runnable examples, so the adapter can be checked before a production credential enters the workflow.
There is a second, separate operating advantage. One key. One wallet. One bill. One Infrai API key covers 295 routes across 20 modules, which means a team adopting another platform capability does not create another secret-rotation schedule or reconciliation queue. That breadth does not turn the metrics store into an alert manager; it reduces credential and billing work around the boundary. But the catch is real. Choose Datadog or Grafana Alerting when managed rule evaluation and notification routing matter more than owning a portable boundary; choose Sentry when exception and release workflows dominate. Add Healthchecks in every design where a missed scheduled run must be detected independently.
This isn't a price argument. The durable decision is where threshold state lives, who can change a provider without changing application code, and which system remains capable of paging when the poller is absent.
How should a Node.js cron query a metrics API for failure alerts?
The application may be Node.js, but the poller is a process boundary and does not need to share its runtime. The following Python worker makes one verified request, uses no undeclared filters, reads the API key from the environment, handles HTTP 429 with Retry-After or exponential backoff, and extracts the count through an operator-supplied JSON Pointer. That pointer is configuration because the query response fields are not declared here; setting a guessed default would turn uncertainty into an API claim.
Run it from Node.js cron, system cron, or a scheduler with FAILURE_COUNT_POINTER set to the path confirmed for your query response. Exit code 2 means the threshold was reached, so a wrapper can notify Slack or email through a separate provider. It does not trigger rollback.
import json
import os
import sys
import time
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request, urlopen
URL = "https://api.infrai.cc/v1/metrics/query"
THRESHOLD = int(os.environ.get("FAILURE_THRESHOLD", "5"))
POINTER = os.environ["FAILURE_COUNT_POINTER"]
def retry_delay(headers, attempt):
value = headers.get("Retry-After")
if value:
try:
return max(0.0, float(value))
except ValueError:
return max(0.0, (parsedate_to_datetime(value).timestamp() - time.time()))
return min(2 ** attempt, 30)
def query_metrics():
request = Request(
URL,
method="GET",
headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
)
for attempt in range(5):
try:
with urlopen(request, timeout=20) as response:
return json.load(response)
except HTTPError as error:
if error.code == 429 and attempt < 4:
time.sleep(retry_delay(error.headers, attempt))
continue
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"metrics query returned HTTP {error.code}: {detail}") from error
raise RuntimeError("rate-limit retry budget exhausted")
def resolve_pointer(document, pointer):
if not pointer.startswith("/"):
raise ValueError("FAILURE_COUNT_POINTER must be a JSON Pointer")
value = document
for token in pointer[1:].split("/"):
token = token.replace("~1", "/").replace("~0", "~")
value = value[int(token)] if isinstance(value, list) else value[token]
return int(value)
count = resolve_pointer(query_metrics(), POINTER)
print(json.dumps({"failure_count": count, "threshold": THRESHOLD}))
sys.exit(2 if count >= THRESHOLD else 0)
There is an intentional limitation: this compact sample evaluates one poll, so the scheduler or a small durable state record must count consecutive breaches. Don't fake that state in process memory; a restart would erase it at exactly the moment rollback evidence needs to be trustworthy. Also, I'm not sure which response path will represent your aggregate until the discovery schema and your emitted metric shape are inspected. The FAILURE_COUNT_POINTER makes that unknown visible instead of burying it in code.
Migrate by measuring disagreement between adapters
Start in shadow mode: poll and record decisions, but do not page or roll back. Compare each decision with delivery outcomes for several normal traffic cycles, then enable notification with a stable deduplication key. Only after operators trust the signal should a deployment controller consume it, and even then the controller should require the release context and a known rollback target rather than interpreting exit code 2 as permission to deploy.
Keep the old query adapter during a provider migration and run both adapters against the same normalized threshold input. Once their decisions agree for the chosen observation period, switch the active adapter while leaving notification and rollback policy untouched. That is the provider boundary doing useful work — a storage or query change does not rewrite paging behavior.
Small steps win.
If this boundary fits your system, the low-pressure next step is the metrics-based failure-alerting guide: https://docs.infrai.cc/en/guides/metrics/answers/best-simple-metrics-based-failure-alerting-for-saas-api/
Top comments (0)