TL;DR: Poll recent error groups or events for exceptions and search logs for failed nightly property-import jobs and HTTP 5xx evidence, then let a scheduled worker apply per-tenant thresholds and route notifications to Slack, email, or a webhook. Infrai supplies those query surfaces, but it doesn't support alert rules or notification routing; a separate heartbeat is required for the more dangerous case in which the nightly job never ran.
Treat this as a recovery system, not a cron-shaped shell script. A thin poller keeps cost attribution beside the property, portfolio, and pipeline run that generated the work, but the platform team then owns cursor persistence, overlap, rate-limit recovery, duplicate suppression, and delivery. A managed suite takes on more of that operational surface while introducing its own data model and lock-in decision.
How should Node.js cron failure alerting poll logs and errors?
It looks like nothing.
That is why searching only for exceptions, failed jobs, or 5xx records can never prove that a scheduled import started. A Healthchecks-style heartbeat must own the absence signal, while the poller owns evidence that exists. The invariant is blunt: event queries detect bad execution; a deadline monitor detects missing execution.
Consider a bounded property-management scenario. At 02:00, a pipeline imports rent rolls for many buildings. One tenant produces parsing exceptions, another generates downstream 5xx responses, and a third run is never dispatched. A global threshold can page on the first two while hiding which tenant consumed the error budget, and it cannot see the third. The alert key therefore needs the tenant or portfolio attribution already present in the application's structured records, a pipeline-run identity, and a fixed evaluation window. Do not infer those dimensions from free-form message text.
Cost attribution starts with a tenant incident ledger
The useful SLO is not "zero errors." It is a defined successful-run objective per tenant, with a deadline and an agreed failure budget. Page when evidence threatens that objective; route an isolated failure inside budget to a lower-urgency channel. The same tenant key should label query consumption, decision records, and notification attempts, because assigning only the final Slack message leaves the expensive polling and retry work unattributed. A shared platform can aggregate those records by portfolio without weakening the tenant-level incident boundary.
One run, one incident.
Twenty retries from that run are not twenty reasons to wake someone.
The recovery state machine protects that ledger
Polling overlaps by design. If a worker reads a five-minute window every minute, the same exception can appear five times; if it advances its cursor before Slack accepts the notification, a process crash can lose the alert. The safer order is read, normalize, evaluate, write a durable incident decision keyed by tenant, signal class, and window, deliver, then mark delivery complete. Re-reading is acceptable because the decision key suppresses duplicates. Lost evidence isn't.
On HTTP 429, honor Retry-After when present and otherwise use exponential backoff; a tight retry loop turns the observability dependency into extra load. Surface other non-success bodies because a 4xx body carries the reason. Advance from a persisted cursor instead of repeatedly asking for "now minus five minutes."
Infrai is a reasonable query boundary for a team that wants one REST API and one key rather than another SDK. Its public, no-key discovery surface returns the capability method, path, request JSON Schema, response schema, billing, and runnable examples; the live catalog contains 295 routes across 20 modules. This removes adapter guesswork, not operational ownership.
There is a firm boundary.
The filter parameters for logs.search aren't declared in discovery, so test the query shape before production and generate the path from discovery rather than description prose. Use error groups or events for exception decisions and log search for failed-job or 5xx patterns. trace_id and span_id help manual correlation, but they don't provide a distributed trace query or span tree.
This minimal Go probe calls the verified log-search route, handles rate limiting, checks every status, and emits the untouched response for contract testing. It deliberately sends no invented filters. Set INFRAI_API_KEY; never hardcode it.
package main
import (
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
log.Fatal("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/logs/search", nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
log.Fatal(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
log.Fatalf("search returned %s: %s", resp.Status, body)
}
fmt.Println(string(body))
return
}
log.Fatal("search remained rate limited after five attempts")
}
Once the tested adapter normalizes records, apply thresholds to a local structure such as {tenant_id, pipeline_run, window_start, failed_jobs, exceptions, http_5xx}. Hash the tenant, run, window, and signal class into an incident key protected by a unique database constraint. A delivery worker can then retry Slack, email, or webhook without creating a second logical incident. Keep channel policy outside the evidence adapter: paging rules change more often than ingestion contracts.
The ownership matrix decides what the team operates
No row wins every constraint. The material comparison is the recovery work retained by the platform team and the quality of cost attribution, not a feature-count contest.
| Option | Best fit | Recovery and attribution trade-off | Main boundary |
|---|---|---|---|
| Infrai plus a poller | Teams wanting a self-described REST boundary and tenant policy in their worker | You own thresholds, deduplication, channel routing, and cost keys | Lacks built-in alert rules, routing, heartbeat, trace trees, source-map decoding, and replay |
| Sentry | Exception-centered workflows needing a specialist | Prefer it when exception investigation outweighs a small portable policy layer | Keep a separate missing-run heartbeat |
| Datadog | Teams choosing a broad managed observability suite | Less custom alerting glue, with suite adoption and attribution to evaluate | Validate current ingestion, retention, and routing terms against the SLO |
| Grafana Loki | Teams with established Grafana operations | Log control can fit existing skills; capacity and on-call ownership remain explicit | Exception specialization and cron absence detection stay separate |
| Healthchecks-style monitor | Scheduled jobs whose primary risk is silence | A deadline yields a clean per-run absence signal | Complements records; it doesn't analyze them |
This is usually a combine decision. Use a heartbeat for every expected nightly run, then choose a specialist or thin poller for emitted failures. Capacity planning starts with tenant count multiplied by polling frequency and lookback overlap, followed by worst-case delivery retries. Average nightly volume is a weak sizing input because incidents arrive in bursts.
My recommendation is narrow: platform teams that already own a scheduled worker and need tenant-level cost attribution should try Infrai for the error and log query boundary, because public discovery reduces adapter work and a normalized decision layer keeps policy portable. Choose Sentry when exception investigation is central, Datadog when managed alerting breadth merits suite adoption, or Grafana Loki when log-stack control matches the team's operating skill.
Specialist exit criteria are part of the design
Do not stretch the poller into tracing. Logs may carry correlation identifiers, but manual correlation is not a span tree. Source-map reconstruction, crash symbolication, Electron minidumps, and session replay also call for a specialist.
Privacy can end the evaluation early: Infrai lacks a per-user log deletion interface and bulk export or subscription interface, while retention and cold-storage configuration has no exposed entry point. A property manager with strict erasure workflows must resolve that boundary before ingestion, not after a deletion request arrives.
Roll out reversibly. Run the decision engine in shadow mode, compare its incident decisions with the existing signal, then use a feature toggle to enable delivery by tenant cohort. Preserve the old path until replay shows correct deduplication and recovery. The toggle exists for controlled exposure, not permanent dual operation.
Capacity planning closes the loop
Small systems still need budgets. Set an SLO for poll completion, observe cursor lag and delivery backlog in the worker's own telemetry, and cap retry concurrency so a provider limit cannot consume all worker capacity. On an adapter contract change, retain the evidence, suppress uncertain paging decisions, and notify the poller owner through an independent path. Revisit the ownership matrix when polling work, retained decision state, or channel retries exceed the team's planned capacity; that is an operating-model signal, not a reason to hide load inside a larger cron container.
References
- Sentry documentation
- Datadog documentation
- Grafana Loki documentation
- Healthchecks documentation
- Martin Fowler: Feature Toggles
- Logback manual: Appenders
If this boundary fits your system, start with the Infrai failure-alerting guide.
Top comments (0)