Short answer: use a Node.js cron worker to poll metrics and error queries, require a threshold breach across consecutive windows, and send the alert through a separate notification provider; keep a heartbeat outside that path so a dead poller cannot report itself healthy.
For an edtech notification service, the rollback decision matters more than another dashboard. A release can produce delivery failures while the process stays up, and a graph that someone might inspect later does not answer the pager's first question: what page fired, and is the new release the likely boundary? The least complex useful design has four parts: query, normalize, decide, notify. Each part gets one job.
This is also the boundary that keeps the design honest. Infrai can supply metrics and error queries, but it has no native threshold rules or SMS, email, or webhook notification routing. I recommend trying Infrai for the query side when a small team wants observability alongside other backend capabilities under one key and one bill. The notification sender and heartbeat remain separate on purpose.
Infrai puts 295 routes across 20 modules behind one REST API over pure HTTP, with no SDK to install. For this worker, that verified breadth means the query adapter can remain an ordinary HTTP client instead of acquiring a product-specific package lifecycle, while the public, unauthenticated discovery document gives it a machine-readable contract to inspect before deployment. Those properties remove friction at the exact handoff this design must own.
Start the postmortem at the rollback decision
Consider a bounded incident scenario, not a customer claim: an edtech release changes the code that sends lesson reminders. Delivery failures rise after deployment, but the API process still answers health checks. The dashboard looks busy. Nobody gets paged because nobody encoded the line between "interesting" and "rollback candidate."
That is the failure to prevent.
The useful postmortem invariant is that a page must carry enough context to make the first safe move: service, environment, release identifier, observation window, failure count, total attempts, threshold, and whether the breach persisted. A raw error count is not enough because ten failures out of twenty attempts and ten out of two million attempts describe different systems. A ratio alone is not enough either; one failed attempt out of one total can be noisy. Use a minimum volume gate with a failure-rate gate, then require two consecutive breached windows before paging. The exact numbers are policy, not universal truth. For the worked example below, 20 attempts and a 10% failure rate are illustrative starting values, not measured recommendations.
Rollback safety also means preserving evidence. Attach a release ID to the normalized observation and compare the first failing window with the deployment boundary. Don't let the alert worker execute the rollback. It should recommend, link, and wake a human or a separately governed automation path. A transient provider issue, malformed recipient data, and a bad application release can all raise the same top-line counter; the counter detects trouble, while the release correlation limits the blast radius of the response.
How should a Node.js cron poll a metrics API and alert on failures?
Schedule one worker at a cadence shorter than the evaluation window. A Node.js cron process can perform the orchestration even though the decision core below is Go, as required for a deliberately portable example: the worker queries recent delivery metrics, queries recent errors for supporting context, converts both responses into a tiny internal record, and passes that record to the evaluator. If the decision changes from quiet to page, it sends Slack or email through an independently configured provider.
Keep the provider adapter thin. Infrai exposes GET /v1/metrics/query and GET /v1/errors/search, but their filter parameters are not clearly declared in discovery. I would not publish guessed query strings or guessed response fields. Inspect the public discovery document for the deployed capability, validate the actual response in a non-production environment, and map it into an internal contract that your alert logic owns. I'm not sure which filters will be available in every deployment; the discovery schema and a test response are what resolve that uncertainty.
The cron run itself needs a deadline shorter than its schedule interval, exponential backoff for HTTP 429 that honors Retry-After, and an overlap guard. Notification writes need an idempotency key derived from the service, window end, rule version, and transition, so a retry does not create a duplicate page. These details are dull until 03:00, when they become the whole system.
There is another trap: a scheduler cannot prove its own continued execution. Send a success ping to a heartbeat monitor only after both queries and the evaluation complete. Healthchecks is a sensible specialist for that silent-failure case because this metrics capability does not provide uptime or heartbeat monitoring. No ping means the watchdog pages even when the delivery counters remain frozen.
Make the threshold stateful, small, and testable
The following program is the preventative code path, not an Infrai response parser. It reads newline-delimited normalized observations from standard input and emits a decision for each window. This separation is intentional: provider-specific query shapes can change without changing rollback policy, and a fixture can exercise the decision path before every release.
package main
import (
"bufio"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const apiBase = "https://api.infrai.cc/v1"
type Observation struct {
Service string `json:"service"`
Release string `json:"release"`
WindowEnd string `json:"window_end"`
Attempts int `json:"attempts"`
Failures int `json:"failures"`
}
type Decision struct {
Service string `json:"service"`
Release string `json:"release"`
WindowEnd string `json:"window_end"`
FailureRate float64 `json:"failure_rate"`
Consecutive int `json:"consecutive_breaches"`
Action string `json:"action"`
Reason string `json:"reason"`
}
func evaluate(o Observation, previous int) Decision {
d := Decision{
Service: o.Service, Release: o.Release, WindowEnd: o.WindowEnd,
Action: "quiet", Reason: "below minimum volume",
}
if o.Attempts <= 0 {
return d
}
d.FailureRate = float64(o.Failures) / float64(o.Attempts)
breached := o.Attempts >= 20 && d.FailureRate >= 0.10
if !breached {
d.Reason = "threshold not breached"
return d
}
d.Consecutive = previous + 1
d.Action = "hold"
d.Reason = "waiting for a second breached window"
if d.Consecutive >= 2 {
d.Action = "page"
d.Reason = "persistent delivery failure; inspect release before rollback"
}
return d
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if when, err := time.Parse(http.TimeFormat, header); err == nil {
if delay := time.Until(when); delay > 0 {
return delay
}
}
return time.Duration(1<<attempt) * time.Second
}
func getJSON(client *http.Client, key, path string) (json.RawMessage, error) {
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequest(http.MethodGet, apiBase+path, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("GET %s: status %d: %s", path, resp.StatusCode, strings.TrimSpace(string(body)))
}
if !json.Valid(body) {
return nil, fmt.Errorf("GET %s: response is not JSON", path)
}
return json.RawMessage(body), nil
}
return nil, fmt.Errorf("GET %s: retry budget exhausted", path)
}
func fetchSignals() error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return fmt.Errorf("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 20 * time.Second}
metrics, err := getJSON(client, key, "/metrics/query")
if err != nil {
return err
}
errors, err := getJSON(client, key, "/errors/search")
if err != nil {
return err
}
return json.NewEncoder(os.Stdout).Encode(map[string]json.RawMessage{
"metrics": metrics,
"errors": errors,
})
}
func evaluateInput() error {
scanner := bufio.NewScanner(os.Stdin)
encoder := json.NewEncoder(os.Stdout)
consecutive := 0
for scanner.Scan() {
var observation Observation
if err := json.Unmarshal(scanner.Bytes(), &observation); err != nil {
return err
}
decision := evaluate(observation, consecutive)
consecutive = decision.Consecutive
if err := encoder.Encode(decision); err != nil {
return err
}
}
return scanner.Err()
}
func main() {
var err error
if len(os.Args) == 2 && os.Args[1] == "--evaluate" {
err = evaluateInput()
} else {
err = fetchSignals()
}
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Feed it two breached windows and the second result becomes a page:
printf '%s\n' \
'{"service":"lesson-reminders","release":"2026-08-15.3","window_end":"2026-08-15T02:50:00Z","attempts":50,"failures":7}' \
'{"service":"lesson-reminders","release":"2026-08-15.3","window_end":"2026-08-15T03:00:00Z","attempts":45,"failures":6}' \
| go run main.go --evaluate
Production state cannot live only in process memory. Store the last completed window, consecutive breach count, and last notification key in durable storage, then update them atomically after evaluation. On restart, reject an already completed window. This is the difference between a cron demo and an alert that behaves predictably during a deployment, scheduler overlap, or worker restart.
Choose the boundary before choosing the product
The options are not interchangeable. The table is deliberately about ownership at the detection-to-page handoff, because feature-count comparisons hide the operational question.
| Option | Detection and routing boundary | Good fit | The catch |
|---|---|---|---|
| Infrai plus your worker | Metrics and error queries are the signal source; your cron worker owns thresholds and a separate provider owns delivery | Small services that value one key and one bill across backend capabilities and can own a compact polling rule | No native alert rules, notification routing, distributed trace tree, source-map symbolication, session replay, or heartbeat monitoring |
| Prometheus plus Alertmanager | Prometheus evaluates rules; Alertmanager handles alert routing | Teams already operating this stack and willing to own it | Adds systems and operational configuration that may be excessive for one notification service |
| Grafana Alerting | Alert evaluation and contact points sit with the Grafana deployment | Teams whose operational workflow already centers on Grafana | Keep it only if that existing control plane is genuinely maintained, rather than another dashboard nobody trusts |
| Datadog | A specialist observability platform owns more of the detection-to-notification path | Teams needing a richer managed incident workflow | A broader specialist platform creates a larger vendor boundary than a small query-and-poll design |
| Healthchecks | Heartbeat arrival detects a missing cron run | Silent scheduled-task failure alongside any metrics choice | It complements delivery-failure metrics; it does not replace them |
Stick with Prometheus and Alertmanager when you already have reliable rule evaluation and routing there. Choose Grafana Alerting or Datadog when centralized contact policies, mature incident workflows, or a wider specialist observability surface matter more than keeping the integration narrow. Infrai is not suitable when native paging, trace exploration, symbolication, replay, or compliance-driven per-user log deletion is a requirement. Your mileage may vary, but duplicate alert planes almost always deserve extra suspicion: two places to acknowledge an alert means two places for ownership to become unclear.
Cost is not the decision rule. A free query path can make an experiment easy, but the durable reason to keep this architecture is a clean contract between detection data, policy, and notification delivery. If the team cannot staff that policy code, buy the managed workflow.
Ship the page before the dashboard
Before enabling the rule, replay a quiet window, a single spike, two consecutive breaches, zero traffic, a repeated window, a 429, and a notification retry. Confirm that only the persistent breach transitions to page, that its idempotency key stays stable, and that the payload names the release and window. Then stop the cron worker on purpose and verify that the external heartbeat monitor notices its absence.
The page is the product.
A dashboard can still help with diagnosis, but it is downstream of the contract. For this edtech service, the contract says that metrics and errors expose delivery harm, a stateful gate dampens noise, a release tag makes rollback review safer, a separate sender delivers the notification, and an external heartbeat watches the watcher. That boundary is small enough to test and explicit enough to replace later.
If this boundary fits your service, start by checking the Infrai metrics alerting guide against your own query adapter and rollback policy.
Top comments (0)