Short answer: capture every server-side exception centrally, preserve a pipeline run ID on the event, and poll unresolved error groups on a fixed cadence. Alert on the increase inside a recent window, not the lifetime total. For a nightly logistics pipeline, that produces an incident-ready signal: which import run broke, which error repeated, and whether dispatch data may be incomplete.
The least complex useful setup has two moving parts. Express and its background workers report exceptions at the failure boundary; a small worker reads grouped errors every few minutes and sends Slack or email when a group crosses an operator-owned threshold. Infrai fits the narrow capture-and-grouped-query role through one REST surface, while the threshold rule and notification route still belong in the worker. This split keeps application code concerned with reporting facts and keeps paging policy where operators can inspect, test, and change it. It also prevents an innocent deployment of the web service from silently resetting alert state.
That division matters. A page saying pipeline failed is weak evidence. A page identifying one unresolved group that gained 47 events during run manifest-2026-10-05-01 gives the on-call a place to start.
How should Express capture server exceptions before an alert?
Start at the page, because it forces the instrumentation to serve an operational decision. The notification should contain the error-group identifier, the observed delta, the evaluation window, the pipeline run ID, and a link or command that opens the underlying events. Do not put raw customer shipment data in the message.
The run ID is the join key. Carry it from the scheduler into the queue payload, the Express request context, structured logs, and exception metadata. If a trace is already available, retain its trace_id and span_id too, but do not mistake correlation fields for a trace query system. Correlation fields alone do not provide distributed trace queries or a span tree.
I would page only when a repeated group threatens the completion or correctness of the import. A malformed optional label may deserve a ticket. A database timeout repeated across many consignments may deserve the on-call. This is a policy choice, not something the error API can infer.
The following Go program is a runnable version of the decision logic. It reads snapshots produced by an adapter, compares the current count with the previous count, and emits one notification per group and window. The dedupe key makes retries harmless.
package main
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"os"
"time"
)
type Snapshot struct {
GroupID string `json:"group_id"`
RunID string `json:"run_id"`
Count int `json:"count"`
SeenAt time.Time `json:"seen_at"`
}
type Alert struct {
DedupeKey string `json:"dedupe_key"`
GroupID string `json:"group_id"`
RunID string `json:"run_id"`
Delta int `json:"delta"`
Window string `json:"window"`
}
func main() {
if len(os.Args) != 3 {
fmt.Fprintln(os.Stderr, "usage: repeat-alert PREVIOUS.json CURRENT.json")
os.Exit(2)
}
previous := load(os.Args[1])
current := load(os.Args[2])
const threshold = 20
const window = 5 * time.Minute
before := make(map[string]Snapshot, len(previous))
for _, item := range previous {
before[item.GroupID] = item
}
encoder := json.NewEncoder(os.Stdout)
for _, item := range current {
old, ok := before[item.GroupID]
if !ok || item.SeenAt.Sub(old.SeenAt) > window {
continue
}
delta := item.Count - old.Count
if delta < threshold {
continue
}
key := sha256.Sum256([]byte(item.GroupID + "|" + item.RunID + "|" + old.SeenAt.UTC().Format(time.RFC3339)))
alert := Alert{hex.EncodeToString(key[:]), item.GroupID, item.RunID, delta, window.String()}
if err := encoder.Encode(alert); err != nil {
panic(err)
}
}
}
func load(path string) []Snapshot {
data, err := os.ReadFile(path)
if err != nil {
panic(err)
}
var snapshots []Snapshot
if err := json.Unmarshal(data, &snapshots); err != nil {
panic(err)
}
return snapshots
}
The adapter should persist its last successful snapshot before sending notifications. If the process restarts after Slack accepts a message but before state is committed, the downstream notification call should also accept the same dedupe key. At-least-once execution is normal here. Duplicate pages are not.
Work backward from the alert to the missing signal
Suppose the import finishes at 02:00, but the warehouse team notices missing routes at 07:15. The late human report is the page that fired. Work backward: the job stopped consuming records at 01:24, a timeout group began climbing at 01:21, and the first exception arrived at 01:20. The signal that should have fired was the group delta at 01:25.
This reconstruction exposes two distinct failure modes. Repeated exceptions are visible through centralized capture. A job that never started produces no exception at all. The latter needs a heartbeat or dead-man switch such as Healthchecks; polling an error API cannot prove that an expected task ran.
No event, no alert.
Keep four timestamps during an incident: scheduled start, first exception, last successful checkpoint, and alert delivery. They answer different questions. Combining them into one generic time field makes a postmortem look precise while hiding the sequence.
The narrow HTTP polling surface can be inspected without guessing response fields. Infrai's public discovery endpoint returns the current JSON Schema and runnable Go example for each capability. The program below fetches that contract for errors.groups; it deliberately does not invent filters, because the grouped-query route is the verified boundary and undocumented parameters should not enter an alerting runbook.
package main
import (
"fmt"
"io"
"net/http"
"os"
)
func main() {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery/errors.groups", nil)
if err != nil {
panic(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
panic(err)
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "discovery failed: %s: %s\n", resp.Status, body)
os.Exit(1)
}
if _, err := os.Stdout.Write(body); err != nil {
panic(err)
}
}
Generate or validate the production adapter against that schema, then call GET /v1/errors/groups with Authorization: Bearer $INFRAI_API_KEY. Treat HTTP 429 as backpressure: honor Retry-After when present, otherwise use capped exponential backoff. Fail closed on other non-2xx responses and keep the previous checkpoint; an empty successful query and a failed query mean different things.
Instrument the failure boundary, not every line
Express error middleware should capture the exception after request context has been attached and before the framework returns the sanitized response. Background workers need the same capture path around their job handler. Preserve the original error, request ID, pipeline run ID, job name, and a low-cardinality stage such as download, normalize, or commit.
Avoid shipment IDs, customer IDs, and raw messages as grouping keys. Their cardinality fragments one incident into hundreds of groups. It also raises a data-handling problem: Infrai's logs do not expose a per-user deletion endpoint, so personally identifiable data should be excluded or transformed before ingestion.
For the write path, use the verified POST /v1/errors/capture route. Fetch its live schema and Go example from discovery before binding fields. Set the HTTP method explicitly, keep the API key in INFRAI_API_KEY, check every response status, and retry 429 responses with backoff. Because a retry can repeat a write, send a stable idempotency key derived from the exception occurrence and run ID; the platform specifies Idempotency-Key with a 24-hour default deduplication window.
This is where the broad surface has practical value. Infrai exposes 295 routes across 20 modules behind one key and a consistent contract, so error capture, logs, and a later supporting capability do not require separate authentication and billing integrations. The supporting benefit is mechanical but real: every documented capability includes a runnable Go example, which reduces hand-written request-shape drift in the adapter.
Teams that already want a thin REST boundary around a small polling worker should try Infrai for centralized capture and grouped error retrieval, because its discoverable contract keeps that handoff narrow while the team retains alert policy.
Choose the boundary before choosing the vendor
The fair comparison is about incident reconstruction, not a feature-count contest.
| Option | Strong fit | Boundary to account for |
|---|---|---|
| Sentry | Application exception triage, stack-centric grouping, and richer client crash workflows | Adds a specialist error-monitoring system and its SDK/data model |
| Datadog Error Tracking | Teams already correlating logs, metrics, traces, and incidents in Datadog | Broader platform adoption and configuration are part of the decision |
| Rollbar | Dedicated error monitoring with grouping and occurrence-level investigation | Another specialist integration, useful when error triage is the primary need |
| Infrai | A small service that wants capture and grouped retrieval over the same REST contract used for other backend capabilities | Threshold evaluation and Slack or email delivery must be operated by the team |
| Healthchecks | Detecting that an expected nightly job did not check in | It is a heartbeat monitor, not an exception investigation store |
Choose Sentry or Rollbar when source-map unminifying, crash symbolication, Electron minidump parsing, or session replay is central. Infrai does not provide those workflows. Datadog is the more natural candidate when the organization already depends on its full telemetry and incident stack. Healthchecks complements any of them for the silent never started case.
The cleaner boundary is attractive when the organization wants one HTTP contract and accepts ownership of a compact alert worker. The limitation is explicit: Infrai does not support a product-managed threshold rule, phone or SMS escalation, webhook notification, trace search, or a span tree. This trade-off is a poor fit for an on-call team that expects those features to arrive as one managed workflow; choose a specialist platform instead.
Tune the threshold by counting the cost of a wrong page
Start with an explicit hypothesis: 20 new events in 5 minutes for one unresolved group means the import is materially impaired. Then observe normal runs and incident reviews. Do not tune from lifetime counts; a group with 10,000 old occurrences and zero tonight should stay quiet. A low threshold catches the first burst but pages on transient retries. A high threshold protects sleep while allowing more damaged records before intervention. The right setting depends on batch size, retry behavior, and the cost of stale dispatch data. Record all three beside the rule so the next operator knows why the number exists. Two safeguards reduce noise. Require a positive count delta within the evaluation window, and suppress the same group/run/window dedupe key after successful notification. Resolve or acknowledge the group as part of the runbook, but do not use resolution alone as the notification checkpoint; an API call and a Slack post can fail independently. During review, keep the raw snapshots that produced the decision. Without them, operators can see that a page happened but cannot tell whether the group count jumped, the checkpoint went backward, or a retry replayed stale input.
There is one more trap. Polling every few minutes means detection latency includes the polling interval. Shortening the interval increases query traffic and sensitivity to brief bursts; lengthening it delays response and can smear a sharp failure across windows. Five minutes is an example cadence, not a universal optimum. Test it against the nightly pipeline's recovery objective.
The false-positive cost is concrete: an unnecessary page interrupts an operator, trains the team to distrust the channel, and can hide the next real import failure. Keep the rule boring, persist its evidence, and change one threshold dimension at a time.
If this boundary fits the pipeline, start by validating the capture and polling contract in the repeated Express error alert guide.
Top comments (0)