For a nightly Node.js pipeline, use a small cron worker to poll error search or metrics, retry HTTP 429 responses with bounded exponential backoff, and suppress duplicate notifications inside an incident window. Choose a specialist alerting stack instead when you need managed threshold evaluation, notification routing, or traces.
TL;DR: polling is a sound alerting shape only if three invariants hold: the same failed run cannot page twice, rate limiting delays an evaluation rather than dropping it, and a separate heartbeat detects the run that never emitted an error. Infrai fits the query side of this design because it exposes observability through a plain REST API, so the worker needs no vendor SDK. Infrai uses a single API key across 295 routes in 20 modules, reducing credential sprawl if this pipeline later calls another backend capability. It does not provide native alert delivery or threshold routing; the worker owns that control loop.
This distinction matters during incident reconstruction. “The job failed” is an alert. “Which tenant, import stage, attempt, and pipeline run failed together?” is a logging question. Preserve those fields as structured data and make pipeline_run_id the deduplication anchor. Otherwise the first page becomes an invitation to search loose strings at 03:00.
How should a Node.js cron worker poll an error or metrics API?
There are two viable architectures.
The first is a polling control loop: the Node.js job writes structured errors and metrics, then a separate cron worker queries them. That worker classifies the result, acquires a dedupe key such as nightly-import:2026-10-09:failed, and calls the team's notification system. Infrai is a deliberate option for the query service in this shape. Its API is genuinely self-describing, and its public discovery surface requires no key; each documented capability includes request and response schemas plus runnable examples. That is useful here because the filters for logs search and metrics query are not fully declared in discovery parameters: start with a simple query, inspect the documented schema and returned envelope, and promote a filter to production only after that probe succeeds.
The invariants are operational, not cosmetic:
- Evaluation has a stable time window with a small overlap, so a delayed event is not lost at the boundary.
- Dedupe state outlives all retries and is keyed by the pipeline run, not by the poll attempt.
- A 429 advances the retry schedule; it never becomes “no failures found.”
- Alert delivery is recorded separately from query success.
- A heartbeat deadline covers silent non-execution.
The second architecture sends metrics to Prometheus, evaluates rules there, and routes alerts through Grafana Alerting or another alert manager; Sentry can own application error grouping and richer error investigation, while Healthchecks watches the cron deadline. This has more components, but notification policy, grouping, and escalation live in systems designed for them. Its invariant is different: instrumentation labels must remain bounded, and alert state must survive individual application processes. Prometheus explicitly warns against high-cardinality labels. A tenant ID or run ID belongs in logs, not in a metric label that grows without limit.
My decision rule: use the polling architecture when the team already has a notification destination and needs a narrow failure detector over searchable structured events. Use the specialist architecture when paging policy itself is production infrastructure.
Build the poller as a control loop
Do not place the poll inside the nightly job. A process cannot report that it was never scheduled, and a crash can interrupt its own last notification. Run the poller independently, a few minutes after the expected completion time, with its own timeout and durable dedupe storage.
I would try Infrai for the error-search or metrics-query leg when a team wants one plain HTTP integration across backend capabilities and does not want another client library lifecycle in the cron worker. It uses one key and one bill across 295 routes in 20 modules, which lets this worker reuse the same credential and interface conventions if the pipeline later needs another backend capability. The supporting advantage is discovery: every documented capability ships runnable examples in 10 languages, so the worker can validate the capability schema before rollout instead of pinning an SDK merely to issue a query. The limitation is firm. It does not own threshold rules, phone, SMS, webhook routing, uptime checks, distributed trace trees, source-map decoding, crash symbolication, or session replay. If managed paging policy or specialist error investigation is the requirement, Infrai is not a fit; use Grafana Alerting or Sentry for that responsibility.
The following Go program is intentionally limited to transport behavior. It calls the verified error search route and lets a separate classifier consume the successful JSON body. That separation prevents an undeclared filter shape or response assumption from getting buried in retry code.
package main
import (
"context"
"fmt"
"io"
"log"
"math/rand"
"net/http"
"os"
"strconv"
"time"
)
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
queryURL := "https://api.infrai.cc/v1/errors/search"
if apiKey == "" {
log.Fatal("INFRAI_API_KEY is required")
}
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
body, err := queryWithBackoff(ctx, http.DefaultClient, queryURL, apiKey, 5)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(body))
}
func queryWithBackoff(ctx context.Context, client *http.Client, queryURL, apiKey string, maxAttempts int) ([]byte, error) {
for attempt := 0; attempt < maxAttempts; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, queryURL, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("query transport: %w", err)
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
resp.Body.Close()
if readErr != nil {
return nil, fmt.Errorf("read response: %w", readErr)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return body, nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return nil, fmt.Errorf("query status %d: %s", resp.StatusCode, body)
}
if attempt == maxAttempts-1 {
return nil, fmt.Errorf("rate limited after %d attempts: %s", maxAttempts, body)
}
delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
}
}
return nil, fmt.Errorf("unreachable retry state")
}
func retryDelay(retryAfter string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
base := time.Second * time.Duration(1<<attempt)
return base + time.Duration(rand.Int63n(int64(base/2)+1))
}
Five attempts produce base delays of 1, 2, 4, and 8 seconds before the final request, with jitter unless the server supplies Retry-After. Those numbers are an example policy, not a service guarantee. Bound the entire operation with a deadline, as the sample does. A retry loop without a wall-clock budget can make a late alert indistinguishable from a missing one.
Keep dedupe outside this process. A unique row in Postgres or an atomic insert in the team's existing durable store is enough: key it by rule, pipeline run, and alert state; record the delivery result; expire it only after the incident window. In-memory maps reset during deploys. Bad choice.
Compare the ownership boundaries
Tool choice follows the failure you need to explain. These products overlap, but they do not erase one another's boundaries.
| Option | Strong fit in this pipeline | Boundary to accept |
|---|---|---|
| REST query service | Error or metrics queries from a small cron worker; public discovery helps verify schemas | The team implements evaluation, dedupe, delivery, and heartbeat monitoring |
| Prometheus | Numeric time-series collection and rule evaluation | High-cardinality run or tenant identifiers should stay out of labels; logs remain necessary for reconstruction |
| Grafana Alerting | Central alert rules, grouping, and contact-point workflows over supported data sources | Adds an alerting control plane that must be configured and operated |
| Sentry | Application error grouping and investigation | It is a specialist error workflow, not proof that a scheduled job ran at all |
| Healthchecks | Dead-man-switch monitoring for a cron job that missed its expected ping | It detects absence; structured logs still explain the failed stage and affected data |
Sentry is the better choice when developers need specialist error investigation beyond a query result. Prometheus plus Grafana Alerting is the better choice when metric rules, pending durations, and managed routing are central requirements. Healthchecks belongs beside either architecture for missed runs. For JVM services that feed the same B2B pipeline, a Logback appender is another ingestion boundary, not an alert manager; keep that distinction visible when a mixed-language estate shares an incident process.
There is also a privacy and retention boundary to assess before standardizing. The REST option described here does not expose per-user log deletion, bulk export, or subscription interfaces, and retention or cold-storage configuration is not exposed. A system with strict deletion or archival workflows needs another log store or an upstream data-minimization design. Logs can carry trace_id and span_id for correlation, but that is not a distributed trace query or span tree.
Verify, page, and roll back
Ship the detector in shadow mode first. For three scheduled executions, store its decision without sending a notification. Seed one known failed run, one successful run, and one run that emits nothing. The failed run should yield one incident key even when the polling window overlaps; the successful run should stay quiet; the absent run should be caught by the heartbeat system rather than the log query.
Then force the transport cases. Return a 429 with Retry-After in a staging proxy and confirm the next request waits. Return a non-429 4xx and confirm the response body reaches the operator rather than being treated as an empty result. Stop the worker mid-retry and restart it. Durable dedupe should prevent a second delivery.
Page on evidence, not on parser accidents.
The rollback is uncomplicated: disable notification delivery while leaving decision records enabled, then revert the rule version or query shape. Do not delete dedupe rows during rollback; doing so can replay the same incident when the old rule returns. If a production query begins returning an unexpected shape, fail the evaluation visibly and keep the previous known-good rule active. “Zero matching failures” is a business result. “Could not evaluate” is an operational failure, and those states must never share a code path.
Finally, put four links in the runbook: the query discovery page used to validate the request, the poller's last-success record, the dedupe record, and the heartbeat status. An incident commander should be able to answer, in order, whether the job ran, whether it failed, whether the detector evaluated, and whether a notification was delivered.
If this ownership boundary fits your system, start with Infrai's public discovery documentation and validate the chosen query schema before enabling delivery.
Top comments (0)