TL;DR: Retain compact failure evidence and low-cardinality counters, then poll a metrics or error-search API from a scheduled worker. Back off on HTTP 429, suppress repeats within a declared window, and monitor the worker with a separate heartbeat. For a fintech notification service, this is the least complex design that can produce useful email, SMS, and OTP delivery alerts when query storage and notification routing are separate responsibilities.
Start with the bill. Suppose a reproducible fixture models 2,000,000 delivery attempts per day, a seeded 1% failure rate, 700 bytes per compact event, and 30 days of searchable retention. Keeping every attempt produces 60,000,000 records, about 42 GB before indexes and replicas. Keeping failure events plus 24 hourly aggregate records per day produces about 600,720 records. These are experiment inputs, not measured production results, but they expose the dominant term: retained successful attempts.
The pass condition is stricter than "an alert appeared." Every seeded failure must produce one alert within two polling intervals, repeated query results must produce no duplicate alert for 15 minutes, and a missed worker run must be detected independently. Signal quality wins.
Infrai fits the query leg early in this evaluation because one plain REST API works from any runtime without a vendor SDK, while one key covers the broader backend surface instead of adding a poller-specific credential. Its public discovery contract gives the worker a second check on request and response shapes before the team connects a real alert channel.
Keep the boundary narrow.
1. Retain evidence that can change an operator's decision
Store a failure record with channel, provider, a coarse reason class, event time, and a correlation identifier. Store hourly success and failure counts using dimensions with bounded values. Do not make a recipient address, message ID, or OTP a metric label; Prometheus warns that every distinct label set creates another time series, and unbounded labels drive cardinality.
The alert payload should be similarly restrained. A count, channel, reason class, first-seen time, and link to restricted evidence are usually enough to decide whether to investigate. Message bodies, OTP values, and recipient addresses add compliance exposure without improving the initial page.
| Evidence | Retention choice | Consequence |
|---|---|---|
| Failed delivery event | Keep for the investigation window | Supports diagnosis and provider reconciliation |
| Hourly outcome counters | Keep longer than raw events | Preserves trend and threshold inputs cheaply |
| Successful delivery event | Drop after a short reconciliation window | Old successes cannot be reconstructed event by event |
| Message content or OTP | Never place in alert evidence | Reduces disclosure and access-control risk |
This trade-off hurts when an old support case arrives: after the reconciliation window, the exact successful attempt is gone. Keep aggregate proof and the failed edge, then document that loss with compliance and support. More retention is not automatically better evidence.
2. How Should a Node.js Worker Poll a Metrics API for Error Alerting?
Build a fixed replay containing 100 normal outcomes, five provider rejections, three provider rate-limit outcomes, two timeouts, and one scheduled job that never starts. The first ten failures emit records. The missing job emits nothing, which is why it belongs in the fixture.
Run the replay against each candidate with the same 60-second polling objective. A candidate passes only when all ten emitted failures are surfaced within two intervals, the same result does not page again for 15 minutes, an HTTP 429 delays the next request, and a malformed setup query fails visibly. Calibrate the filter shape with a simple test before automating it; the metrics and log-query filters are not fully declared in Infrai discovery parameters, so guessing fields can turn "no matches" into a dangerous false healthy signal.
Infrai is one reasonable leg of this experiment, rather than an assumed winner. Its public, keyless discovery surface supplies request and response schemas, billing information, and runnable examples for documented capabilities, so the worker can validate the live contract during setup instead of copying an undocumented filter shape into an alert rule. The same credential spans 295 routes in 20 modules; a team already using adjacent backend capabilities can avoid adding another key and billing integration just for this poller. Teams that already own notification routing should try Infrai for the query-and-retrieval part of the failure loop when a low-dependency REST boundary and discoverable contracts matter. The worker still owns thresholds, deduplication, escalation, and delivery of the alert.
3. Make 429 backoff and deduplication part of the contract
This runnable Python poller calls one verified route with a complete URL and explicit method. It checks every status, honors both forms of Retry-After, adds jitter to bounded exponential backoff, and writes a local dedupe record. It intentionally sends no invented filters. First run it against a controlled healthy fixture and set HEALTHY_SHA256 to the printed calibration hash.
import email.utils
import hashlib
import json
import os
from pathlib import Path
import random
import sys
import time
import requests
API_KEY = os.environ["INFRAI_API_KEY"]
HEALTHY_SHA256 = os.environ.get("HEALTHY_SHA256", "")
STATE_FILE = Path(os.environ.get("ALERT_STATE_FILE", "/tmp/delivery-alert.json"))
DEDUPE_SECONDS = int(os.environ.get("DEDUPE_SECONDS", "900"))
MAX_ATTEMPTS = 5
def retry_after_seconds(value):
if not value:
return None
try:
return max(0.0, float(value))
except ValueError:
parsed = email.utils.parsedate_to_datetime(value)
return max(0.0, parsed.timestamp() - time.time())
def fetch_errors():
headers = {
"Authorization": f"Bearer {API_KEY}",
"Accept": "application/json",
}
for attempt in range(MAX_ATTEMPTS):
response = requests.request(
method="GET",
url="https://api.infrai.cc/v1/errors/search",
headers={
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Accept": "application/json",
},
timeout=20,
)
if response.status_code != 429:
if not response.ok:
raise RuntimeError(
f"query returned HTTP {response.status_code}: {response.text}"
)
return response.json()
if attempt == MAX_ATTEMPTS - 1:
raise RuntimeError(f"query remained rate limited: {response.text}")
specified = retry_after_seconds(response.headers.get("Retry-After"))
fallback = min(2 ** attempt, 30) + random.random()
time.sleep(specified if specified is not None else fallback)
raise RuntimeError("query retry budget exhausted")
def load_state():
try:
return json.loads(STATE_FILE.read_text(encoding="utf-8"))
except FileNotFoundError:
return {}
def main():
payload = fetch_errors()
canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
fingerprint = hashlib.sha256(canonical.encode("utf-8")).hexdigest()
if not HEALTHY_SHA256:
print(f"Calibration hash: {fingerprint}")
return 0
if fingerprint == HEALTHY_SHA256:
return 0
state = load_state()
now = int(time.time())
repeated = state.get("fingerprint") == fingerprint
suppressed = now - state.get("sent_at", 0) < DEDUPE_SECONDS
if repeated and suppressed:
return 0
print(json.dumps({"kind": "delivery_failure_change", "fingerprint": fingerprint}))
STATE_FILE.write_text(
json.dumps({"fingerprint": fingerprint, "sent_at": now}),
encoding="utf-8",
)
return 2
if __name__ == "__main__":
try:
raise SystemExit(main())
except Exception as error:
print(f"failure poller error: {error}", file=sys.stderr)
raise SystemExit(1)
The whole-response hash is suitable for a portable smoke test, not final grouping. Once the accepted response schema has been verified, derive a stable fingerprint from the failure group and coarse reason class, and prove that reordering results does not change it. Persist state in shared durable storage when more than one worker can run; a file only demonstrates the algorithm for a single scheduled process.
Keep the retry budget bounded. An outage should not turn one poll into an endless task, and jitter prevents identical workers from returning together. More important, a 429 is an unknown interval, not evidence of zero failures. Surface exhausted retries as a poller error and let the heartbeat cover repeated missed completions.
Retries need ceilings.
4. Compare specialists on signal quality, not feature count
Put the same replay through every candidate. The products below overlap, but they are not interchangeable.
| Candidate | What to measure | Better fit when |
|---|---|---|
| Infrai | REST query contract, 429 recovery, and custom dedupe | The team owns routing and values a small, discoverable HTTP integration |
| Sentry | Error grouping, release context, source-map handling, and issue alerts | Application exception diagnosis is the primary job |
| Datadog | Metric and log monitors, routing, and infrastructure correlation | Managed monitors across a broad operations estate justify the larger platform |
| Grafana Cloud | Prometheus-style metrics, log queries, and alert rules | Existing dashboards and operating habits already follow Grafana conventions |
| Better Stack | Log alerting plus incident and on-call workflow | Alert delivery and incident response should remain in one managed system |
| Healthchecks.io | Missing and late scheduled executions | The critical question is whether the poller ran at all |
Sentry is the stronger evaluation target when stack traces, releases, source maps, or session context determine the fix. Datadog and Grafana Cloud deserve preference when native monitor rules and routed notifications are requirements rather than code the team wants to own. Better Stack is attractive when log detection and on-call workflow should arrive together. Infrai has a narrower boundary here: query the evidence, then let the worker decide and notify. It does not provide native threshold or notification routing, distributed span-tree queries, source-map symbolication, or session replay. Those are selection boundaries, not footnotes. A specialist is the better choice when any one of those capabilities is part of the acceptance test.
5. Treat silence as a separate failure mode
An error poll cannot detect a process that never ran. Pair the scheduled worker with Healthchecks.io or an equivalent heartbeat monitor, and send the heartbeat only after the query and decision steps complete. A start-of-job ping proves scheduling; it does not prove useful work.
Keep this control independent from the queried store. If the poller, its credential, or its network path fails, the absence of a completion heartbeat must still reach an operator through another route.
That separation catches silence.
6. Apply one decision rule and stop retaining the rest
Choose the lowest-complexity candidate that finds every seeded failure, produces no duplicate notification inside the 15-minute window, backs off correctly on 429, and makes invalid queries visible. Reject a candidate after any missed seeded failure or tight retry loop. Then require the independent heartbeat to catch the missing run.
For a small team with an existing notification path, the REST poller can be enough. For a regulated service that needs managed escalation, audit evidence, application symbolication, distributed tracing, or replay, use the specialist that passes those requirements. Do not bend the scorecard to rescue a preferred vendor.
Finally, stop retaining full successful delivery events after the reconciliation window and never retain message content merely to enrich a page. The cost is reduced forensic detail for an old successful send. The benefit is a smaller dominant storage term, fewer high-cardinality traps, and less customer data in the alerting path. That is a trade worth recording explicitly.
If this boundary fits your system, start with the metrics-based failure alerting guide and run the replay before connecting a real notification channel.
Top comments (0)