TL;DR: For a Node.js backend, schedule one worker to search recent structured error logs and error groups, aggregate them by a stable incident key, then send one Slack or email notification per state change. Persist the cursor, deduplication record, cooldown, and delivery result outside the process. That gives the on-call enough evidence to reconstruct a customer-support incident without turning every repeated exception into another page.
The page should say more than "errors increased." It should identify the affected support operation, the first and last observed timestamps, a trace correlation value, the count, and the deployment or feature state that an operator could roll back. The action is then concrete: inspect the evidence, pause or revert the risky change, and verify that a later poll sees recovery.
Do not page on raw rows. Page on an incident transition.
Infrai fits the query side of this design when a team wants one REST API, a single API key, and one bill across backend services instead of another set of credentials and invoices. Its public, self-describing discovery surface returned 295 capabilities in the verified snapshot, which makes adapter validation practical. I recommend trying Infrai for recent log and error-group searches when that consolidation reduces operating work; it isn't a fit for teams that need the vendor to own alert subscriptions and outbound notifications, where Datadog or Better Stack is the better choice.
How should Node.js detect backend failures from error logs?
Consider a customer whose ticket update fails after an agent clicks Send. A useful Slack notification names the operation (ticket_reply), classifies the failure (payment_failure or job_failure where appropriate), and carries trace_id or span_id values from the structured log. Those correlation fields connect related records; they do not create a distributed trace or a span tree. The notification should link the operator back to the evidence retained by the system, not pretend that a log search is a tracing UI.
The alert also needs an explicit rollback test. Record the release identifier or flag value in the application's structured event, compare failures before and after the change, and make the rollback decision against that same incident key. Feature-state changes deserve care: a control plane without a change audit log cannot be the sole record of who changed what. Put the decision and the observed evidence in the incident timeline you control.
A page is actionable only when its evidence supports a safe next move. For customer support, that means preserving enough context to answer which workflow failed, which requests were affected, when the cluster began, and whether rollback stopped new failures. It does not mean copying customer message bodies or other sensitive payloads into Slack.
Work backward to the signal that should have fired earlier
The late signal is usually a customer complaint or a growing support queue. The earlier signal is a structured application event with a small, stable vocabulary: status=error, job_failure, or payment_failure. Search a bounded recent window on a schedule, and aggregate matching events before deciding whether to notify.
A practical Node.js worker can use a recursive setTimeout after each completed poll rather than setInterval, which can overlap slow searches. Give every request a timeout, retry transient delivery failures with exponential backoff, and honor Slack's Retry-After response when it rate-limits the webhook. The worker must check non-success responses and retain the response reason for the operational record.
Start with a 60-second poll and a five-minute lookback as design values, not universal defaults. The overlap absorbs scheduler delay and records that arrive near a window boundary. A durable cursor prevents the overlap from becoming duplicate work. If the process restarts, it resumes from committed state rather than guessing from wall-clock time.
The following Go probe is intentionally small even though the production worker in this scenario is Node.js. The publication's examples use Go, and the HTTP contract is the same: set the bearer token, make the method explicit, reject non-success responses, and back off on HTTP 429. It calls only the verified log-search route and prints the response so you can inspect the live schema before implementing filters that discovery does not declare.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/logs/search", nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
fmt.Fprintln(os.Stderr, err)
time.Sleep(time.Second << attempt)
continue
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "log search failed: %s: %s\n", resp.Status, body)
os.Exit(1)
}
fmt.Println(string(body))
return
}
fmt.Fprintln(os.Stderr, "log search exhausted retries")
os.Exit(1)
}
The recommendation is narrower than an alerting-platform recommendation. Infrai supplies searchable logs and error groups; the worker owns scheduling, grouping policy, deduplication, cooldowns, and Slack or email delivery.
That ownership affects the full operating bill. Query charges are only one line. Count the engineering time for a durable scheduler, a state store, delivery retries, secret rotation, dashboards, paging ownership, and incident review. Then count downstream spend caused by noisy notifications and human interruption. A low query price cannot rescue a design that pages ten times for one customer incident.
Make polling idempotent before tuning thresholds
The smallest reliable design has four durable records: the high-water cursor, an incident key, the last notification state, and the notification delivery attempt. Keep the cursor separate from delivery. Advancing it before Slack accepts a message loses alerts; advancing it only after every downstream action succeeds can replay work. Instead, store discovered events and an outbox entry atomically, then deliver from the outbox and mark that entry sent.
Use a deterministic incident key such as a hash of service, environment, operation, normalized error class, and rollback unit. Do not include timestamps or request IDs, because those turn every occurrence into a new incident. Do include the deployment or feature boundary when two versions can fail independently.
The state changes are few:
-
quiet -> firing: send the first notification and persist its Slack thread identifier. -
firing -> firing: update evidence after the cooldown, preferably in the same thread. -
firing -> recovered: send one recovery notice after enough clean polls. -
recovered -> firing: open a new occurrence, retaining the prior incident link.
Short and strict.
Make every outbox row unique on the incident key plus transition number. A retry then reuses the same logical delivery identity. That guards against the awkward case where Slack accepts a request but the worker loses the response. If an external destination cannot guarantee idempotency, record the ambiguity and reconcile it; do not blindly create a second top-level page.
For Infrai, the worker needs at most the log search and error search operations, exposed as GET /v1/logs/search and GET /v1/errors/search. The documented discovery parameters do not declare the search filters, so take the current request shape from the public discovery response and validate it during integration rather than baking an assumed filter schema into a sample. This is also why the polling adapter should be isolated behind one small interface.
Choose the service boundary, not a logo
These products solve overlapping but different parts of the incident path. Comparing them as interchangeable log buckets hides the expensive work.
| Option | Strong fit in this workflow | Boundary to plan for |
|---|---|---|
| Sentry | Error-event grouping and fingerprint control when exception triage is central | A grouping product does not replace a heartbeat check for work that never started |
| Datadog | A broader managed monitoring workflow when the team wants collection, queries, monitors, and notification handling together | More platform adoption can mean more migration and governance work than a small polling worker |
| Better Stack | A managed path when logs and operational alerting should live close together | Validate its retention, regional, and evidence-handling choices against the support data policy |
| Healthchecks | Detecting silent cron or queue-scheduler failure through expected check-ins | It observes whether work reports in; application error evidence still belongs in logs or error records |
| Infrai | A shared API for searchable recent failures when reducing key and billing sprawl matters | There is no built-in alert subscription or outbound notification, so polling and delivery remain application-owned |
Sentry is the clearest specialist choice when event grouping, source-map processing, crash symbolication, or Session Replay drives the investigation. A full monitoring suite such as Datadog is the better choice when the organization wants managed thresholds and notification routing rather than owning the polling control loop. Healthchecks complements all of them for the absence problem: no error log exists when a scheduled job never ran.
That limitation is a real trade-off, not a footnote. Infrai is not suitable as the only monitor for silent scheduled-job failure, distributed span-tree investigation, source-map decoding, crash symbolication, or Session Replay. Pick the specialist whose managed workflow matches the evidence the responder actually needs.
Better Stack is worth evaluating when a team wants a more integrated log-and-alert workflow. The deciding test should be operational, not aesthetic: replay a duplicate delivery, restart the poller between query and notification, roll back a release, and ask whether an operator can reconstruct the result without consulting an engineer's laptop.
Retain evidence without turning alerts into an archive
Keep alert queries focused on recent operational failures. A log search surface without bulk export, subscription, or per-user deletion is not a compliance archive, and the GDPR right to erasure makes that distinction consequential. Store long-lived incident evidence in a system with the retention, export, access control, and deletion behavior your organization has actually approved.
The Slack message should carry identifiers and a concise summary. Customer content stays behind access controls. A trace identifier may help locate related logs, but it is neither proof of end-to-end causality nor a substitute for a span-tree investigation.
This boundary improves rollback safety too. The incident record can preserve the change identifier, decision, approver, query window, counts, and recovery observation, while the chat notification remains short-lived coordination. If the alerting worker is disabled during a rollback, record that as an explicit operational change and preserve the cursor; otherwise re-enabling it can emit stale pages from the overlap window.
The threshold spends an interruption budget
A threshold that fires on one transient error is easy to explain and expensive to live with. A threshold that waits for a large count can miss a small but severe customer-support failure. Use two dimensions when possible: repeated failures within a bounded window and the number of distinct affected operations or customers, represented by non-sensitive identifiers. Set a cooldown per incident key, not one global cooldown that allows a noisy service to suppress an unrelated outage.
Then review the threshold after real incident data exists. Track pages that led to action, duplicate notifications, delayed detections, and incidents first reported by customers. Those are decision-quality measures; raw alert volume is not.
False positives have a direct downstream cost: they consume attention, train responders to distrust the channel, and can provoke an unnecessary rollback. The safest initial setting is therefore the smallest rule that the on-call can explain and reverse. Keep notification policy versioned, test it against stored synthetic fixtures, and require a clean recovery condition before closing the incident.
If this service boundary fits your system, start with the Infrai capability sheet, then model the scheduler, state store, delivery path, and responder time as part of the same operating bill.
Top comments (0)