A scheduled import can return no results without throwing an error, so error-only alerting is the wrong control. Short answer: record an immutable result for every Node.js cron run, poll a small unresolved-groups API from an independent process, and notify email, Slack, or a webhook only when a durable state transition occurs. That design makes incident reconstruction possible and prevents every poll from becoming another alert.
The operational constraint is independence. If the importer and its monitor share a scheduler, deploy, database connection pool, and process, one failure can silence both. I want the monitor to answer a harder question than "did an exception happen?": which run was expected, what evidence arrived, when did the state become actionable, and which notification attempt owns the alert?
I've been paged by missed jobs and duplicate deliveries. Those two pages look unrelated until the incident timeline is rebuilt: in both cases, an implicit state transition was mistaken for a one-off message. The invariant I carry into the runbook is blunt — persist state first, derive alerts second.
Data retained for a silent cron run
Consider a Node.js import scheduled for 02:00. Its process starts, reads an upstream feed, writes zero records, and exits cleanly. There is no stack trace for conventional error alerting to collect. A separate failure mode is even quieter: the scheduler never starts the process. Both outcomes are operational failures if the contract says that a result must exist by a deadline, but neither can be reconstructed from exception text alone. The useful unit is therefore a run record, not a log line. Give each expected execution a stable key derived from the schedule and logical time window. Store expected_at, started_at, finished_at, outcome, result count, and a correlation ID. Keep the observed events append-only if practical, then derive the current status. Logs still matter: OpenTelemetry's logs model provides a common way to associate log records with resource and trace context. The run record adds the business invariant that a log pipeline cannot infer by itself: an import result was due.
Use three terminal outcomes in the first version: succeeded, failed, and empty. Keep missing as a derived state when no start evidence exists after the grace period. This distinction matters during a postmortem. A failed run began and reported a terminal error; an empty run completed but violated a result expectation; a missing run produced no trustworthy execution evidence. Folding all three into "cron error" saves a column and destroys the timeline.
Silence is data.
The bounded incident sequence should read like this: an expected-run row is created before the deadline; the importer claims that row using its stable key; completion appends an outcome; the monitor later resolves the row or opens an alert group. If the importer retries, the same key is reused, so the retry updates evidence around one logical execution instead of inventing a second job. If notification delivery retries, it uses a separate alert key. This split is the idempotency reflex that keeps a harmless retry from looking like two incidents.
How should Node.js poll an error API for unresolved cron groups?
Expose a read-only API that returns unresolved groups, but define the response around incident state rather than raw exceptions. A group needs a stable ID, the logical job name, the first and latest observed timestamps, the current reason, and a monotonically increasing version. The poller stores the last version it successfully handled. It doesn't infer resolution from an empty page unless the API contract explicitly says absence means resolution; explicit resolved_at evidence is much easier to audit.
Polling should overlap windows and deduplicate locally. That sounds wasteful, but it closes the gap between "response received" and "checkpoint committed." Suppose poll A returns group import:catalog:2026-08-17T02:00Z, the notifier sends Slack, and the process exits before saving its cursor. Poll B will see the group again. With an idempotency key built from group ID, transition, and version, the second delivery is suppressed or safely coalesced. Without that key, reducing the polling interval only creates duplicate notifications faster.
Retries will happen.
The monitor below is deliberately separate from the Node.js worker. The endpoint is pseudonymous, and the transport interface can represent email, Slack, or a generic webhook without baking any provider into the state machine.
package monitor
import (
"context"
"encoding/json"
"fmt"
"net/http"
"time"
)
type Group struct {
ID string `json:"id"`
Job string `json:"job"`
Reason string `json:"reason"`
Version int64 `json:"version"`
FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time `json:"last_seen"`
ResolvedAt *time.Time `json:"resolved_at"`
}
type Sender interface {
Send(ctx context.Context, idempotencyKey string, group Group) error
}
type ReceiptStore interface {
WasSent(ctx context.Context, idempotencyKey string) (bool, error)
MarkSent(ctx context.Context, idempotencyKey string, at time.Time) error
}
func Poll(ctx context.Context, client *http.Client, endpoint string, sender Sender, receipts ReceiptStore) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return fmt.Errorf("build unresolved-groups request: %w", err)
}
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("poll unresolved groups: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("poll unresolved groups: unexpected status %d", resp.StatusCode)
}
var groups []Group
if err := json.NewDecoder(resp.Body).Decode(&groups); err != nil {
return fmt.Errorf("decode unresolved groups: %w", err)
}
for _, group := range groups {
transition := "opened"
if group.ResolvedAt != nil {
transition = "resolved"
}
key := fmt.Sprintf("%s:%s:%d", group.ID, transition, group.Version)
sent, err := receipts.WasSent(ctx, key)
if err != nil {
return fmt.Errorf("read receipt %s: %w", key, err)
}
if sent {
continue
}
if err := sender.Send(ctx, key, group); err != nil {
return fmt.Errorf("send alert %s: %w", key, err)
}
if err := receipts.MarkSent(ctx, key, time.Now().UTC()); err != nil {
return fmt.Errorf("store receipt %s: %w", key, err)
}
}
return nil
}
Set a finite HTTP timeout in the caller and ensure only one poll iteration owns a lease at a time. A short overlap between pages is fine; concurrent uncoordinated pollers are not. The API should sort by a stable tuple such as (last_seen, id) and return an opaque continuation token. Offset pagination can skip or repeat groups as new failures arrive. Repetition is acceptable because the consumer is idempotent. Skipping is not.
There is one uncomfortable edge in the sample: a notification can be delivered and its receipt write can fail. Exactly-once delivery across an arbitrary external transport and a local receipt store isn't something this interface promises. The operational answer is an idempotency key passed to a transport that can honor it, or a transactional outbox feeding the sender. If neither is available, accept at-least-once delivery, include the stable group ID in the message, and make duplicate acknowledgment cheap.
Test the grouping model before it can page anyone
Grouping determines the page volume and the evidence an operator sees. Sentry documents event grouping through fingerprints, which is a useful model: multiple events can map to one issue based on a stable grouping decision. For scheduled imports, don't group on volatile exception messages, timestamps, result counts, or hostnames. Start with job identity plus failure class. Add tenant or source only when those dimensions have separate ownership or remediation.
A good group changes version only when an operator-relevant fact changes: the incident opens, severity changes, recovery occurs, or the failure class changes. Repeated observations update counters and last_seen; they don't create fresh pages. The notification message should carry the group ID, expected run time, latest evidence time, reason, and a link into the internal incident view. Email, Slack, and webhooks are output adapters. None should own grouping logic.
This is where I first reach for an explicit state table rather than clever debounce timers. A ten-minute debounce may hide a burst, but after a restart it cannot explain which event won, why the alert fired, or whether recovery was delivered. Durable state can. During incident review, the difference is the ability to produce a sequence instead of a theory.
Test the gap.
| Evidence | Derived state | Notification action |
|---|---|---|
| No start after deadline and grace period | Missing | Open or update one job/run group |
| Terminal failure recorded | Failed | Open or update the group immediately |
| Success with zero results where results are required | Empty | Open or update with count evidence |
| Later successful run satisfies recovery rule | Resolved | Send one recovery transition |
Keep the recovery rule explicit. One success may be enough for an hourly import; a noisy upstream might require two consecutive successful windows. Your mileage may vary because the right rule depends on the data contract, not the notification transport. Record the chosen rule beside the group so an operator can distinguish automatic recovery from a manual acknowledgment.
Email, Slack, and webhook integration boundaries
Choose the destination by response workflow. Email fits low-urgency, asynchronous ownership and produces a searchable trail, but inbox handling is a weak acknowledgment protocol. Slack is useful when a team already coordinates there, yet a reaction or thread should not become the source of truth for resolution. A webhook is the cleanest integration boundary when another system owns routing, escalation, or on-call acknowledgment; it also means your team owns authentication, retries, signing, and receiver compatibility.
The cheapest setup is not a universal product or the smallest monthly line item. I'm not sure which transport will be cheapest for a given team without event volume, retention, compliance, and on-call labor data. A small setup can use one poller, a durable relational table, and one existing team destination. Price the system by stored event volume, polling traffic, retention, notification attempts, and operator time spent removing duplicates. Then test those assumptions with a week of shadow notifications before paging anyone.
Alternatives have real boundaries. Direct log alerts are quick when every relevant failure emits a reliable record, but they cannot prove that an expected run never started. Metrics with a "last successful run" gauge make absence visible and work well for aggregate dashboards, but labels must stay bounded and the metric alone may not retain the evidence needed to reconstruct one run. Trace-based alerting connects failures to downstream calls when trace context exists, while a scheduler that never launched creates no trace. The run ledger can reference logs, metrics, and traces; it should not be replaced by any one of them.
Don't send production pages on the first deployment. Replay recorded run transitions through the grouping function, verify that repeated poll responses create one receipt, inject a timeout between send and receipt storage, and confirm that recovery is a distinct version. Deploy the monitor in shadow mode, compare its unresolved set with the run ledger, then enable a low-urgency destination before on-call escalation. The test oracle is a transition ledger, not the number of messages in a channel.
Rollout and rejection without a paging surprise
The catch is that a polling control plane adds a database, a consistency contract, and another process to operate. It is not suitable when the job already runs inside a scheduler that durably records expected executions, outcomes, retries, and alert transitions with the incident history your team needs. Stick with that scheduler's native state and export standards-based telemetry rather than building a parallel ledger.
Don't duplicate authority.
Polling is also the wrong trigger when latency must be lower than a safe polling interval or unresolved volume is too large to scan efficiently. In that case, publish state changes through a durable queue and retain the read API for reconciliation. Keep the same group IDs and idempotency keys. Event delivery improves latency; periodic reconciliation still detects anything the event path missed.
For a small, noncritical import with a human checking results each morning, the operational machinery may cost more than the missed-data risk. Write the service-level expectation first. If there is no response deadline, owner, or recovery action, an alert is only noise with a webhook attached.
The decision rule is simple: use a run ledger and independent unresolved-group poller when absence is itself a failure and incident reconstruction matters. Use existing scheduler state when it already preserves the same evidence. Use event-driven delivery when latency requires it, but keep reconciliation and idempotency. Whatever transport carries the message, page from durable transitions rather than transient errors.
References
- OpenTelemetry, "Logs signal concepts": https://opentelemetry.io/docs/concepts/signals/logs/
- Sentry, "Event grouping and fingerprint mechanics": https://docs.sentry.io/concepts/data-management/event-grouping/
Top comments (0)