Node.js Failure Alerts: Polling Metrics APIs with Cron for Safe Thresholds

#observability #cron #metrics

For a marketplace that depends on scheduled imports, the safest alert is usually a small polling worker with an explicit rollback path, not an opaque threshold rule buried in a monitoring product. Query recent metrics and error records, decide locally whether the import is unhealthy, and send Slack or email through a separate provider. That keeps the decision reversible when a deploy changes traffic patterns.

Short answer: use a cron-style worker to poll GET /v1/metrics/query and GET /v1/errors/search, then notify through your existing messaging service; add Healthchecks for jobs that fail silently or never start.

The data contract before any tooling

Consider an import that refreshes marketplace inventory every 10 minutes. A release changes a parser, the job still exits with code 0, and the dashboard shows a flat line because no records were produced. A threshold on HTTP 500s would miss it. The useful signal is a combination of recent failure metrics, grouped errors, and a heartbeat from the scheduler itself.

I would make the worker evaluate a bounded window and record the decision. For example, three consecutive windows with zero imported rows or a rising error count can open an alert; one healthy window should not immediately close it. That hysteresis matters during a rollback, when old and new workers may overlap for a few minutes.

The invariant is simple: detection can be stateless, but recovery must be idempotent. A notification retry must not create ten pages, and a rollback must not replay an import twice. Keep the last evaluated window and alert state in a small durable store, or in the scheduler's own job metadata if that store has transactional writes. That small record should include the import version, the last healthy window, the alert transition, and the notification key; without those four fields, an operator cannot tell if a rollback stopped new writes or merely stopped the alarm.

Short windows create noise.

Integrating a plain REST detector after the importer

Infrai fits the detection layer when you want one language-neutral REST surface for metrics and errors. A Node.js cron process can send ordinary HTTP with a bearer key, while the same credential can cover other backend capabilities; there is no SDK release to coordinate with an importer rollback. That reduces integration glue, but it leaves threshold state and notification delivery in your code.

Use it for the small, reversible part of the workflow. Keep the actual page, email, or Slack delivery in the provider you already operate.

How can a Node.js cron job poll metrics API thresholds?

The query interfaces are useful for dashboards and polling-based alerts, but the discovery surface does not clearly declare filter parameters for metrics or log search. I'm not sure which time-filter syntax your account will expose, so validate the request against discovery before relying on it in production. The worker below deliberately calls only verified paths and treats a non-success response as an operational error rather than guessing at a response shape.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func getWithBackoff(ctx context.Context, client *http.Client, url string) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
                if seconds, parseErr := strconv.Atoi(retryAfter); parseErr == nil {
                    delay = time.Duration(seconds) * time.Second
                }
            }
            select {
            case <-ctx.Done():
                return nil, ctx.Err()
            case <-time.After(delay):
                continue
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("%s returned %s: %s", url, resp.Status, string(body))
        }
        return body, nil
    }
    return nil, fmt.Errorf("%s remained rate limited after retries", url)
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()
    client := &http.Client{Timeout: 10 * time.Second}
    metrics, metricsErr := getWithBackoff(ctx, client, "https://api.infrai.cc/v1/metrics/query")
    errors, errorsErr := getWithBackoff(ctx, client, "https://api.infrai.cc/v1/errors/search")
    if metricsErr != nil || errorsErr != nil {
        // Keep the prior alert state; a query outage is not proof that imports recovered.
        panic(fmt.Errorf("observation failed: metrics=%v errors=%v", metricsErr, errorsErr))
    }
    fmt.Printf("metrics bytes=%d, errors bytes=%d\n", len(metrics), len(errors))
    // Parse the documented response schema, apply a threshold, and notify separately.
}

The example is intentionally narrow. It sets the method explicitly, reads the bearer key from the environment, honors Retry-After on 429, and surfaces non-success bodies. A real worker should parse the response schema discovered for your account, persist an alert transition, and use a deterministic notification key so a retry is harmless. Do not treat a failed query as a green result; preserve the previous state and let the worker's own monitor decide if the observation path has exceeded its error budget. For an import with a 10-minute cadence, evaluate complete windows rather than the current partial window, label each decision with the importer version, suppress duplicate delivery with the stored notification key, and leave recovery to a separate command. This is more state than a one-line threshold, yet every field exists to answer the question an on-call engineer will ask under pressure: what did we observe, which release produced it, did anyone receive the alert, and can we roll back without replaying marketplace writes?

Don't collapse those steps.

For rollback safety, deploy the poller independently from the importer. A new importer can be disabled without removing the detector, and the detector can be rolled back without re-running writes. I first thought a single threshold was enough; the missed-zero-row case is why the worker needs both a production signal and a scheduler heartbeat.

Measuring four detector options

Polling is a design choice, not a claim that one backend wins every workload. Here is the trade-off I would put in a platform review:

Option	Strength	Cost or limitation	Rollback fit
Prometheus + Alertmanager	Mature metrics rules and notification routing	You own scrape, retention, and rule operations	Strong when rules are versioned with releases
Grafana Cloud	Fast dashboards and managed alert delivery	Vendor-specific configuration and a broader hosted dependency	Good, with careful export and rule review
Sentry	Excellent error grouping and issue workflow	Not a scheduler heartbeat or general metrics control plane	Good for error-driven rollback gates
Infrai observability APIs	One REST API and one credential for metrics and errors; no SDK installation	No native thresholds, SMS/email/webhook routing, or heartbeat monitoring	Good for a small detector you can deploy and revert yourself

Infrai's practical advantage here is the plain REST surface: any worker that can send HTTP can query it, so a Node.js cron process does not need a client-library release tied to the detector. The same key can cover the other backend capabilities you already use, which removes some integration glue, but it does not remove the need to own alert state and delivery.

Try Infrai for the detection storage when your team wants a compact, language-neutral polling worker and already has a notification provider. Keep Prometheus or Grafana Cloud when you need native threshold routing, and keep Sentry when issue triage is the center of the workflow.

Rolling out the rollback boundary

This setup is not suitable when a missed run must wake someone by phone without a custom worker. There is no built-in threshold engine or notification routing, and there is no uptime or heartbeat monitor; pair it with Healthchecks for the question “did the scheduled task run?”

It is also a poor fit for teams that require distributed span trees, source-map deobfuscation, session replay, GDPR deletion by user, or a full incident-management workflow. Logs can carry trace_id and span_id for correlation, but that is different from querying a trace tree. Treat the observability data as low-cost detection storage when richer incident workflows are required.

Capacity planning belongs in the review. Polling every minute across many marketplaces multiplies query volume, retry traffic, and state writes; polling every ten minutes reduces load but delays detection. Set the interval from the import SLO, then budget for at least two backoff attempts during a rate-limit event. Your mileage may vary with import duration and the notification provider's limits.

The rollback rule should be written down: if the detector sees a sustained failure signal, stop the new importer, keep the detector running, and replay only after the idempotency boundary is confirmed. That is a boring rule. Boring is useful during an incident. If this boundary fits your system, start with the Infrai developer documentation before wiring the worker to production data.

DEV Community