Error Tracking Alerts: Polling Recent Unresolved Failures from a Cron Job

#errors #tracking #alerts

TL;DR: For a small media SaaS without native alerting, poll the error source on a fixed schedule, write each observation to an append-only evidence ledger, and notify chat and email only after the ledger accepts a new deduplication key. Keep the polling cursor separate from notification state. That is the least complex design I would ship because a rollback can replay work without erasing the record needed to reconstruct a customer incident.

Choice	Incident evidence	Rollback behavior	Operating load
Poll, persist, then notify	Observation plus delivery attempts	Replays use stable deduplication keys	One worker and one small store
Poll and notify directly	Destination history	A replay can duplicate or skip alerts	Lowest until reconstruction is needed
Stream every event	Rich event trail	Depends on managed offsets and consumers	More moving parts

Recommendation: use the first option for a one-person media product. It preserves evidence around failed publishing, transcoding, or feed-generation jobs without turning observability into a second product to operate. Ship the narrow loop first. Add streaming only when polling latency is demonstrably too slow for the customer impact.

How should error tracking polling send email alerts?

An alert is not the evidence. It is a delivery attempt derived from evidence. Chat messages can be edited, email can be filtered, and either destination can be temporarily unavailable. The durable record should answer four questions: which customer-facing job failed, when the source first and last observed it, which release was running, and what notification attempts followed.

For a media workflow, use a business identifier such as publicationId or assetId alongside the source error identifier. A stack trace can identify a code path while leaving the operator unable to tell which scheduled edition, video, or feed a customer missed. Capture the release identifier too. During rollback, that field separates failures created by the outgoing release from observations made by the restored release.

The ledger entry should be immutable. Resolution is another entry or a separately updated projection, not deletion of the original failure. This is a deliberate trade-off: a little duplicate metadata buys a coherent timeline when the only engineer available is also handling support and shipping the next weekly release.

Preserve two clocks. observedAt says when this poll saw the item. sourceOccurredAt says when the source says it happened. They differ whenever the scheduler runs late or the source returns an older unresolved error.

Tiny distinction. Big payoff.

Criterion one: evidence before delivery

Persist the observation first, reserve a deduplication key second, then attempt each notification. Record every attempt and result. If the worker stops after persistence but before delivery, the next run can retry. If it stops after chat delivery but before recording success, a retry may duplicate at the destination, so put the same incident key in the message and expose that rare ambiguity in the ledger. Derive the key from stable source data, not the current time: combine the source error ID, occurrence timestamp, and notification policy version. The policy version allows a changed audience or threshold to create a fresh alert. Do not add the deployed application version; rolling back would manufacture a new identity for the same failure. Perfect exactly-once delivery across an error source, a database, chat, and email would require coordination those systems do not share. An idempotent local reservation plus an auditable delivery record is the practical revenue-per-hour boundary: duplicates are constrained, missing attempts are recoverable, and the mechanism stays small enough to inspect during an incident.

My starting policy would be a five-minute poll, one retry queue, and two independent destinations. Five minutes is a decision, not a universal threshold: shorten it only when customer impact makes that delay unacceptable, because every shorter interval increases source queries and overlap pressure. I would rather explain that trade-off in one line of configuration than bury it in scheduler code.

The Google SRE monitoring chapter frames monitoring around latency, traffic, errors, and saturation. This worker handles the error signal and notification evidence. It does not replace monitoring for the other signals.

Criterion two: cursors cannot be acknowledgements

A polling cursor answers, "How far has collection progressed?" An alert acknowledgement answers, "Which observation entered the notification workflow?" Combining them creates a rollback hazard. An older release can load a cursor written by newer code, fail to understand it, and restart too far back or jump ahead. Neither outcome should decide whether evidence exists.

Use an opaque cursor only with the adapter that obtained it, and version the saved envelope. Commit the next cursor after every returned item is written to the ledger, even if delivery fails. Notification retries should read pending attempts from the ledger instead of fetching the source page again. Collection owns source progress. Delivery owns destination progress.

Retention needs the same discipline. Keep raw payloads only as long as useful and permitted, but retain a compact incident timeline for support and review. Redact tokens, message bodies, and customer content before persistence. Media jobs can carry titles, private asset locations, or unpublished copy in arguments; those values do not belong in a chat alert merely because they appeared in an exception.

Log volume has an operating cost in hosted systems; the CloudWatch pricing page documents ingestion charges by data volume. Store compact, structured evidence instead of copying the entire payload into every retry record. Still retain the identifiers and timestamps required for reconstruction.

A small TypeScript polling loop

Keep the core vendor-neutral. Put the error source, evidence store, and destinations behind interfaces, with no hard-coded route or assumed commercial API shape.

type Failure = {
  sourceId: string;
  sourceOccurredAt: string;
  publicationId: string;
  releaseId: string;
  summary: string;
};

type Page = { items: Failure[]; nextCursor?: string };

interface FailureSource {
  recentUnresolved(cursor?: string): Promise<Page>;
}

interface EvidenceStore {
  readCursor(version: number): Promise<string | undefined>;
  appendObservation(item: Failure, observedAt: string): Promise<void>;
  reserveNotification(key: string): Promise<boolean>;
  appendAttempt(key: string, channel: string, outcome: string): Promise<void>;
  writeCursor(version: number, cursor: string): Promise<void>;
}

interface Destination {
  channel: "chat" | "email";
  send(message: string, key: string): Promise<void>;
}

const CURSOR_VERSION = 1;
const POLICY_VERSION = "media-unresolved-v1";

function incidentKey(item: Failure): string {
  return [item.sourceId, item.sourceOccurredAt, POLICY_VERSION].join(":");
}

async function pollOnce(
  source: FailureSource,
  store: EvidenceStore,
  destinations: Destination[],
): Promise<void> {
  const cursor = await store.readCursor(CURSOR_VERSION);
  const page = await source.recentUnresolved(cursor);

  for (const item of page.items) {
    await store.appendObservation(item, new Date().toISOString());
    const key = incidentKey(item);
    if (!(await store.reserveNotification(key))) continue;

    const message = `Publishing failure ${key}: ${item.summary}`;
    for (const destination of destinations) {
      try {
        await destination.send(message, key);
        await store.appendAttempt(key, destination.channel, "sent");
      } catch {
        await store.appendAttempt(key, destination.channel, "failed");
      }
    }
  }

  if (page.nextCursor) {
    await store.writeCursor(CURSOR_VERSION, page.nextCursor);
  }
}

Scheduling stays outside the function, so a platform scheduler, operating-system cron, or queue timer can invoke it. Prevent overlap in the store with a short lease or database lock, and enforce a unique constraint on the incident key. An application-level check is insufficient when two invocations read before either writes.

Test the failure boundaries. Stop after observation, reservation, and the first destination send, then invoke the worker again with the same page. Expect one reservation, preserved observations, independent attempt records, and no cursor advancement before every item is stored. Also give the old worker a newer cursor envelope in a deployment test. It should refuse an unknown version while leaving evidence untouched.

When is the runner-up better?

Direct poll-and-notify is reasonable when the source already provides the durable incident record, destination history is sufficient for audit, duplicate messages are harmless, and rollback reconstruction is not required. That can fit an internal tool. It does not fit this media scenario, where the job is to retain enough evidence to reconstruct a customer incident.

A streaming pipeline fits when tolerated detection delay is shorter than a responsible polling interval, error volume makes repeated queries inefficient, or several independent consumers need the same event stream. It adds offset management, backpressure, retention, and consumer deployments. For a solo SaaS, I would wait for measured pressure before owning them; outsourcing undifferentiated scheduling and storage leaves more of the week for customer-facing work.

Whichever transport wins, rehearse rollback with recorded fixtures before release. Deploy a new policy, ingest a failed publication, roll back the worker, and run it again. The resulting timeline must contain the occurrence, both observations, release identifiers, and delivery attempts without creating a fresh incident identity. Otherwise the path is loud but not reconstructable.

DEV Community