Short answer: for Node.js failure alerting, use a rollback-safe cron job that polls logs and an errors API for notification failures, then sends one thresholded alert while a separate heartbeat watches the poller itself.
For an edtech notification service, a bad alert rule can be as disruptive as a bad delivery change. A threshold that is too low pages someone during a normal retry burst. A threshold that is too high hides a broken enrollment reminder. A rollback must restore both the delivery code and the interpretation of its failures.
The data flow is plain. A scheduler starts the poller, the poller reads a bounded time window from a logs or errors API, groups records by a stable incident key, compares each group with a threshold, and sends a Slack, email, or generic webhook notification. A small durable state record suppresses duplicates. A separate heartbeat answers the question the error stream cannot: did the scheduled check run at all?
What should a Node.js failure alerting job measure before a rollback?
Start with the signals that can change during a deployment. For notification delivery, that usually means failed jobs, exceptions, and HTTP 5xx responses, with the notification type and release identifier attached when the application can provide them. The release identifier matters because a rollback decision needs to distinguish a new regression from an old, known failure.
Count incidents, not raw lines. Five retries for one learner should not look like five unrelated outages. Conversely, five 5xx responses for five different notification batches may deserve attention even if each individual exception count is small. The grouping key should be stable across polling runs and specific enough to route the alert to the owner.
Group first.
Keep the window and threshold in configuration, then record the rule version with every notification. During a rollback, the operator can see which policy produced the page and can avoid silently mixing results from two releases. This is a small detail. It prevents a large argument at 2 a.m.
A poller also needs a failure budget of its own. Bound the request duration in the scheduler, retry only transient responses with a cap, and treat an unavailable data source as monitor health rather than as zero failed jobs. Zero is a measurement; no response is missing data.
For the edtech case, a useful test dataset has three notification types, two releases, one repeated exception, and several 5xx responses. Run the old and new rule against that same dataset before switching traffic: the repeated exception should form one incident group, a release-specific regression should remain attributable to its release, and an empty result should remain quiet only when the API request completed successfully. This test is more informative than a count copied from a dashboard because it exercises the exact decision that a rollback depends on: which failures are actionable, which are duplicates, and which signal that the monitor itself has lost visibility.
A TypeScript example for polling failed jobs and 5xx exceptions
The example below keeps the provider contract at the boundary. FAILURE_API_URL is the verified URL for the logs or errors API in the deployment, and the response adapter is deliberately explicit. That makes a route or payload change visible in review instead of hidden in a vendor-specific client library.
import { readFile, writeFile } from "node:fs/promises";
import { setTimeout as sleep } from "node:timers/promises";
const apiUrl = process.env.FAILURE_API_URL;
const apiKey = process.env.FAILURE_API_KEY;
const webhookUrl = process.env.ALERT_WEBHOOK_URL;
const threshold = Number(process.env.ALERT_THRESHOLD ?? "5");
const ruleVersion = process.env.ALERT_RULE_VERSION ?? "delivery-v1";
const statePath = process.env.ALERT_STATE_PATH ?? ".delivery-alert-state.json";
const windowMs = 5 * 60_000;
if (!apiUrl || !apiKey || !webhookUrl || !Number.isInteger(threshold) || threshold < 1) {
throw new Error("FAILURE_API_URL, FAILURE_API_KEY, ALERT_WEBHOOK_URL, and a positive ALERT_THRESHOLD are required");
}
type Failure = {
id: string;
kind: "failed-job" | "exception" | "5xx";
occurredAt: string;
release?: string;
notificationType?: string;
};
type State = Record<string, number>;
async function loadState(): Promise<State> {
try {
return JSON.parse(await readFile(statePath, "utf8")) as State;
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return {};
throw error;
}
}
async function readFailures(attempt = 0): Promise<Failure[]> {
const response = await fetch(apiUrl, {
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter) && retryAfter > 0
? retryAfter * 1000
: 500 * 2 ** attempt;
await sleep(delayMs);
return readFailures(attempt + 1);
}
if (!response.ok) {
throw new Error(`Failure data request was rejected (${response.status})`);
}
const payload: unknown = await response.json();
if (!Array.isArray(payload)) throw new Error("Expected an array of failure records");
return payload as Failure[];
}
function recentFailures(failures: Failure[]): Failure[] {
const cutoff = Date.now() - windowMs;
return failures.filter((failure) => Date.parse(failure.occurredAt) >= cutoff);
}
function incidentKey(failure: Failure): string {
return [failure.kind, failure.release ?? "unknown-release", failure.notificationType ?? "all"].join(":");
}
async function notify(key: string, count: number): Promise<void> {
const response = await fetch(webhookUrl, {
method: "POST",
headers: { "content-type": "application/json", "idempotency-key": `${ruleVersion}:${key}` },
body: JSON.stringify({
text: `${key} reached ${count} failures in ${windowMs / 60_000} minutes (rule ${ruleVersion})`,
}),
});
if (!response.ok) throw new Error(`Alert destination rejected the notification (${response.status})`);
}
async function main(): Promise<void> {
const failures = recentFailures(await readFailures());
const counts = new Map<string, number>();
for (const failure of failures) {
const key = incidentKey(failure);
counts.set(key, (counts.get(key) ?? 0) + 1);
}
const state = await loadState();
for (const [key, count] of counts) {
if (count < threshold || state[key]) continue;
await notify(key, count);
state[key] = Date.now();
}
await writeFile(statePath, JSON.stringify(state), "utf8");
}
main().catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
The adapter is the part to test against the selected API's documented response. The sample validates the top-level shape, but production code should also validate required fields before counting them. I'm not sure every scheduler supplies a stable release identifier, so the fallback is explicit; your mileage may vary, and a deployment manifest is the better source when that identifier is available.
The notification request should be treated as a side effect. Local suppression prevents repeated pages after a restart, while the idempotency key gives a receiving service enough information to collapse duplicates when it supports that behavior. Write state only after delivery succeeds. Otherwise a temporary webhook failure can erase the next chance to notify.
| Signal | What it can establish | What it cannot establish |
|---|---|---|
| Failed job record | A scheduled delivery reported failure | That every scheduled job ran |
| Exception record | An application path emitted an exception | That the exception affected every learner |
| HTTP 5xx record | A request returned a server-side failure | That the downstream provider is the only cause |
| Heartbeat deadline | The scheduled worker missed its expected completion | Why the worker missed it |
How do polling logs, errors, cron jobs, thresholds, Slack, and email fit together?
Polling is useful when the application already records the evidence and the team wants a narrow, inspectable rule. It is a poor substitute for every observability capability. A polling reader does not automatically provide distributed trace queries or span trees, source-map reversal, crash symbolication, Electron minidump parsing, synthetic checks, or Session Replay. It also does not provide log export or subscriptions, per-user log deletion, or a user-facing retention and cold-storage configuration surface.
Those are capability boundaries, not defects. Choose a system that directly supports one of them when it is a requirement. A generic webhook can route to Slack or an email service, but the destination still owns delivery policy, access control, and escalation. The poller should report the incident; it should not become an accidental paging platform.
The other important boundary is silence. If a cron job never starts, there may be no failed job, exception, or 5xx record to count. Add a heartbeat after successful completion and alert when the deadline passes. Do not infer health from an empty error window.
Silence needs its own signal.
Rollback safety needs a testable operating rule
Use report-only mode first. Observe the normal retry noise, then send one controlled failure through the same ingestion path and verify the complete chain: scheduled read, time filtering, grouping, threshold comparison, suppression, and destination delivery. Test one event below the threshold and exactly enough events to cross it. The expected result is silence in the first case and one notification in the second.
During deployment, keep the prior alert rule available until the new rule has passed that check. Store rule version and release in the payload, and make rollback restore the previous pair together. A code rollback with a new threshold can leave the team watching the wrong signal.
The deployment record should answer more than “did the page fire?” It should show which release produced each event, when the poller read it, which grouping key was selected, what threshold was active, and whether the destination acknowledged the request; retaining those fields for the same period as the incident state lets an operator reconstruct a false positive without replaying production traffic, compare the old rule with the new one, and decide whether a rollback fixed the delivery path or merely changed the amount of noise. That audit trail is especially useful for student-facing messages, where a duplicate reminder and a missing reminder have different consequences even when both begin as a generic request failure.
The catch is operational ownership. This design is not suitable when the team needs managed schedules, acknowledgements, maintenance windows, escalation chains, or a large fleet of alert policies. In that case, use an observability or incident platform that provides those controls directly. Keep the poller for a small, custom rule only when its narrow scope is genuinely easier to operate.
The final checklist belongs in the runbook: define the incident key, choose a bounded window, document what counts as failed, persist suppression state, cap retries, monitor the monitor with a heartbeat, test Slack and email destinations, and record the rollback rule version. Then review false positives after the first release. Alerting is a control loop, not a one-time cron snippet.
References
- Martin Fowler, “Feature Toggles”: https://martinfowler.com/articles/feature-toggles.html
- Logback manual, “Appenders”: https://logback.qos.ch/manual/appenders.html
Top comments (0)