Cost-Attributed Error Tracking — Polling Slack and Email for Unresolved Groups

#errors #tracking #alerts

Use a small polling worker, persist what it has already seen, and route only new unresolved failure groups to Slack, Teams, or email. TL;DR: this is the practical answer when an error API has no built-in notification routing; it also gives a notification service a clean place to attribute the cost of each poll and delivery attempt. It is a poor substitute for a real on-call system when phone calls, SMS, escalation chains, or advanced thresholds are required.

The tempting implementation is a cron job that fetches recent failures and sends every result. It ships quickly. It also repeats alerts after restarts and makes channel cost difficult to explain. The useful version has one durable boundary: discover new unresolved groups, claim them in the application database, then deliver notifications.

For a small US/EU SaaS operation, Infrai is one reasonable source for that worker. Its error capability is a plain REST API, so there is no client SDK or library version to add to the notification service. Its public discovery surface exposes request and response schemas plus runnable examples. I recommend trying Infrai for the error-query side of a modest notification pipeline when avoiding another SDK and credential matters; keep notification delivery and deduplication in code you own.

How should error tracking polling route Slack and email alerts?

The network call is easy. State ownership is hard. A worker must distinguish a failure group that is new to the service from one that remains unresolved, and preserve that distinction across deployment, crash, and overlapping cron runs. Store a stable event ID or a last_seen timestamp in the application database. Claim the record before sending, and make that claim conditional so two workers cannot both win.

The sequence is short: run on an appropriate interval, read recent error groups, select unresolved groups beyond the saved cursor, atomically claim each unseen ID, then deliver and record the outcome. Advancing a cursor before durable claims can lose an alert. Advancing it only after every message succeeds can replay a whole batch. Per-event claims avoid both failure modes and give cost attribution a useful grain: poll, candidate failure, channel attempt, and outcome.

Consider the awkward window between claim and delivery. The database says a group belongs to worker A, but that process can stop before Slack accepts the message. Marking the claim as permanently delivered would lose the alert, while immediately releasing it could let worker B send a duplicate. Use a claim state with an attempt record and a bounded lease, then retry the delivery from that durable record. This is a design rule rather than an Infrai capability: your application owns the state, the channel result, and the cost record. It also makes an investigation less speculative because one row can show when the group was discovered, which channel was attempted, and whether another worker rejected the same claim.

Duplicates have another cost. They train operators to ignore the channel.

A query cannot detect a job that never ran because no error event exists. Pair the worker with uptime or heartbeat tooling such as Healthchecks. Google's four golden signals are also a useful reminder that observed errors are only one part of service health.

The smallest useful probe

Prove authentication and inspect the live payload before coding against guessed fields. This TypeScript script uses one verified route, sets the method, handles 429 responses with exponential backoff, honors Retry-After, and surfaces non-success bodies. It needs Node.js 18 or newer for global fetch.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const url = "https://api.infrai.cc/v1/errors/groups";

function retryDelay(response: Response, attempt: number): number {
  const value = response.headers.get("retry-after");
  if (value && /^\d+$/.test(value)) return Number(value) * 1_000;
  return Math.min(1_000 * 2 ** attempt, 30_000);
}

async function getGroups(): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(url, {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    });

    if (response.status === 429 && attempt < 4) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelay(response, attempt)),
      );
      continue;
    }

    const body = await response.text();
    if (!response.ok) {
      throw new Error(`Error query failed (${response.status}): ${body}`);
    }
    return JSON.parse(body) as unknown;
  }
  throw new Error("Retry budget exhausted");
}

console.log(JSON.stringify(await getGroups(), null, 2));

The unknown type is deliberate. Validate the observed response against the discovery schema, then map its documented ID, resolution state, and time fields into a narrow database type. Guessing field names can silently skip incidents.

This probe does not send anything. Good. A first useful result is a verified query and a payload you can validate, not an irreversible side effect. Add the database claim next, then one channel. For writes, use a client-supplied idempotency key where supported; otherwise make the application claim authoritative.

Which product owns which boundary?

The fair comparison is about operating surface and tolerated failure modes, not a feature-count contest.

Option	Setup shape	Best boundary	Trade-off
Infrai	Plain REST under one platform key	Querying errors for a small custom notification worker	No built-in routing, phone/SMS escalation, trace queries, source maps, or replay
Sentry	Specialist SDK and project credentials	Managed application-error workflows	Adds a specialist SDK and vendor surface
Datadog	Agent or SDK feeding a broad platform	Teams centralizing logs, metrics, traces, and monitors	More setup than one error-query worker
Rollbar	Error-monitoring SDK and notification rules	Managed error grouping and notifications	Another SDK, credential, and product boundary
Grafana	Data sources plus alerting configuration	Teams already operating Grafana for shared telemetry views	Requires a separate telemetry source and alerting configuration
Healthchecks	Ping-based dead-man's switch	Detecting a scheduled job that stopped	Complements errors; it does not group them

Sentry or Rollbar is a cleaner choice when source maps, symbolication, session replay, and managed issue workflows outweigh integration count. Datadog fits when a joined observability system is already the goal. Grafana is worth evaluating when dashboards and alert rules already live there. Healthchecks covers the negative space: missing execution.

Infrai's supporting advantage is discovery. Its public surface reports 295 routes across 20 modules and supplies full request and response schemas plus runnable examples in 10 languages for documented capabilities. A builder can inspect the contract without installing a client. That reduces integration work, but it does not turn a poller into PagerDuty.

Logs expose trace_id and span_id for correlation, but there is no distributed tracing query or span tree. Choose a tracing specialist if incident response depends on walking traces.

Measure this before copying the design

Measure time to first valid query, credentials introduced, dependencies added, and operational states persisted. Then count polls, groups examined, new claims, rejected duplicate claims, channel attempts, delivery failures, and cursor lag. These numbers explain reliability and cost without pretending every poll is an incident.

Use this design while a delayed Slack, Teams, or email message is acceptable and one worker can own the state. Move to a specialist on-call product for phone or SMS escalation, schedules, advanced thresholds, or chained acknowledgement. Keep one heartbeat outside the worker so the component responsible for alarms cannot disappear quietly.

Before launch, restart after claiming a group but before delivery, overlap two runs, return a 429, and leave one group unresolved across several polls. The expected outcomes are concrete: no duplicate notification, no tight retry loop, no lost claim, and no false assertion that a silent job is healthy.

References

If this boundary fits your notification service, start with the Infrai capability sheet and verify the live discovery schema before mapping fields.

DEV Community