A low-noise watchdog should alert on missing progress, then attach error evidence. Counting every exception or 5xx response sounds simpler, but it answers the wrong question for a scheduled edtech import: did something complain, or did the expected student results actually arrive? TL;DR: poll a durable progress record, require two consecutive missed deadlines, and use logs and errors to explain the alert rather than trigger it.
| Trigger | Signal quality | Noise risk | Best fit |
|---|---|---|---|
| Missing progress past a deadline | High for scheduled imports | Low if lateness is modeled | Primary choice |
| Failed-job ratio | Good with enough volume | High for small batches | Busy queues |
| Raw 5xx or exception count | Fast but indirect | High during retries | Supporting evidence |
The recommendation is the first row. A watchdog should compare an expected completion window with the last successful result, deduplicate by import and window, and send one generic webhook notification. Keep error polling in the same run, but put those records in the payload as context. This choice adds one small progress contract. It removes a surprising amount of alert glue.
How should Node.js failure alerting poll logs and errors?
A scheduled import can stop producing results without producing a fresh error. The scheduler may never start it. An upstream file may not arrive. A worker may accept a job and then cease making progress. None of those states guarantees a 5xx response, and an exception stream can stay perfectly quiet. Quiet is the bug signal here.
Silence has no stack trace.
The reverse is also common by design: an import can emit errors and still complete. A malformed optional row, a transient HTTP failure followed by a retry, or one rejected enrollment among thousands may be worth recording without paging anyone. A threshold such as errors >= 5 throws away the denominator and the deadline. Five failures in a six-row tutoring cohort mean something different from five failures in a district-wide batch.
This is the first decision criterion: measure the job's promised outcome, not its loudest implementation detail. For this system, the outcome is a fresh result checkpoint. Give every scheduled import a stable ID, its expected completion time, and a last-success timestamp. If partial progress matters, store the last committed cursor too. Do not scrape free-form log text to reconstruct those fields after the fact.
Logs still matter. They answer the next question an engineer asks: what happened? A custom log appender is one established way to route events to another destination, and the Logback manual documents the appender contract. The useful architectural point is broader than one logging library. Emission and alert policy are separate jobs. Keep them separate.
Should one late poll trigger a notification?
The second criterion is repeated evidence. One late poll should not become one alert. Schedulers drift, completion times have a tail, and the poller itself can briefly lose access to its data source. Require two consecutive observations beyond the deadline before opening an incident. Clear it only after the watchdog sees a newer successful checkpoint.
Two misses are a trade-off, not a universal constant.
That rule has four inputs: expected completion time, grace period, consecutive miss count, and recovery checkpoint. Four is enough. Resist a configuration object with separate thresholds for each exception class, status code family, notification channel, weekday, and tenant. Config bloat moves the policy out of sight while making it harder to test.
Signal quality also depends on identity. Use a deduplication key such as importId:scheduledWindow. The first confirmed miss opens the alert. Later polls update evidence without sending another notification. A successful result closes it. This state machine is small enough to exhaustively test, which is a better DX property than a dozen mutually interacting switches.
Feature toggles can help deploy this policy without coupling rollout to a release. Martin Fowler's treatment of feature toggles also makes the cost clear: toggles introduce configuration and need active management. Use one temporary toggle for notification delivery if rollout needs it; do not create permanent toggles for every threshold. Test both states, then remove the rollout toggle after the policy settles.
Should the TypeScript watchdog own cron scheduling?
The core does not need to know about email, chat, or any commercial alerting API. It needs a progress reader, an evidence reader, a tiny state store, and a notifier interface. Every timestamp is supplied or returned as data, so tests do not have to fake global time.
type Checkpoint = {
importId: string;
scheduledFor: string;
expectedBy: string;
lastSuccessAt: string | null;
};
type Evidence = { failedJobs: number; serverErrors: number; exceptionKinds: string[] };
type AlertState = { consecutiveMisses: number; open: boolean };
type Dependencies = {
readCheckpoint(importId: string): Promise<Checkpoint>;
readEvidence(importId: string, since: string): Promise<Evidence>;
readState(key: string): Promise<AlertState | null>;
writeState(key: string, state: AlertState): Promise<void>;
notify(payload: Record<string, unknown>): Promise<void>;
};
export async function checkImport(
importId: string,
observedAt: string,
deps: Dependencies,
): Promise<"healthy" | "waiting" | "alerted"> {
const checkpoint = await deps.readCheckpoint(importId);
const key = `${checkpoint.importId}:${checkpoint.scheduledFor}`;
const previous = (await deps.readState(key)) ?? { consecutiveMisses: 0, open: false };
const completed = checkpoint.lastSuccessAt !== null &&
Date.parse(checkpoint.lastSuccessAt) >= Date.parse(checkpoint.scheduledFor);
if (completed) {
await deps.writeState(key, { consecutiveMisses: 0, open: false });
return "healthy";
}
if (Date.parse(observedAt) <= Date.parse(checkpoint.expectedBy)) return "waiting";
const consecutiveMisses = previous.consecutiveMisses + 1;
const shouldNotify = consecutiveMisses >= 2 && !previous.open;
await deps.writeState(key, { consecutiveMisses, open: previous.open || shouldNotify });
if (!shouldNotify) return "waiting";
const evidence = await deps.readEvidence(importId, checkpoint.scheduledFor);
await deps.notify({
event: "scheduled_import_stalled",
deduplicationKey: key,
observedAt,
checkpoint,
evidence,
});
return "alerted";
}
There are no cron-library calls in that function. Good. Let a scheduler invoke it at a fixed cadence, but keep scheduling outside the policy. Let the notifier translate the generic payload into an email or webhook request, with an explicit timeout and retry budget. Persist state before notifying so a slow notification endpoint cannot produce a fresh alert on every poll. Delivery can be retried with the same deduplication key.
Keep the adapter boring.
The timestamp comparison assumes valid ISO 8601 strings with an explicit offset. Validate them at the storage boundary. Also define who writes lastSuccessAt: it should move only after result data is durably committed, not when the worker starts or downloads its input. Otherwise the monitor reports activity while students still have no imported results.
Test the state transitions, not the cron expression. At minimum, cover a completion before the deadline, one late observation, two late observations, repeated polls after an alert opens, and a later successful checkpoint. Add a case where evidence retrieval fails: the primary stall alert should still be deliverable with an evidenceUnavailable marker because diagnosis is secondary to detection.
When is the runner-up better?
A failed-job ratio is the runner-up, and it wins when work is continuous rather than deadline-based. If jobs arrive all day and there is no honest answer to "expected by when?", progress lateness becomes arbitrary. In that system, compare failed jobs with total completed attempts over a defined window and require a minimum sample size. The denominator matters more than another notification integration.
Raw exception or 5xx thresholds have a narrower role. They are useful for fast detection when one class of failure is always actionable and cannot be retried safely. Keep the rule scoped to that invariant. A generic 5xx > 0 alert tends to encode transport noise as business impact, while a high fixed threshold misses a small but important import.
This progress-checkpoint design has a real limitation: it is not suitable when the producer cannot write a trustworthy completion record. Adding a synthetic checkpoint beside an opaque upstream system creates a second source of truth, and stale checkpoint writes can be worse than a noisy error counter because they claim success. In that case, choose the failed-job ratio when attempt totals are available, or choose a narrowly scoped exception rule when a specific exception always means lost results. There is another cost. Two consecutive observations increase detection delay by one polling interval. A five-minute poll therefore adds up to five minutes after the first confirmed late observation before notification. Do not hide that latency in configuration. Put it in the operating agreement, then shorten the interval only if the data store and notification path can handle the extra reads without making the watchdog part of the incident.
Boundaries matter.
Benchmark the policy with replay, not vibes. Feed historical checkpoint and evidence records through candidate rules, then count alerts, duplicate notifications, detected missed imports, and detection delay. No invented target percentage survives contact with a school's schedule and batch sizes. Pick the rule that catches known missing-result windows with the fewest unactionable notifications, document that decision, and rerun the replay when schedules or import volume change.
The final shape stays compact: a durable progress checkpoint, a two-observation gate, a deduplication key, and error evidence attached after the decision. Page on absent outcomes; inspect errors to find the cause.
Preserve the alert contract across notification channels
Email and webhook adapters should receive the same event shape. Preserve the import ID, scheduled window, observation time, last successful checkpoint, evidence summary, and deduplication key; presentation belongs in the adapter. A chat message can be brief while an email carries more context, but both must identify the same open incident. The recovery event should reuse that identity and carry the newer success timestamp. Otherwise a channel migration quietly changes incident semantics, dashboards count one stall twice, and responders cannot pair the alert with its recovery.
Do not put raw student rows, access tokens, or unbounded log bodies in the notification payload. Send counts and exception categories, then link the incident workflow to access-controlled evidence by stable identifiers. This is an operational boundary rather than a vendor feature: the detector decides impact, the adapter handles delivery, and the evidence store retains diagnostic detail.
Top comments (0)