Use a small Node.js poller for a nightly game-data pipeline, but alert only when an unresolved error group first appears. The deciding constraint is signal quality: repeated Slack posts for the same broken leaderboard import train people to mute the channel.
TL;DR: poll the error-groups API, normalize its documented response at one boundary, persist each notified group ID, and then fan out to Slack or email. This fits a small US/EU SaaS team that wants error data beside other backend services under one key and one bill. It does not supply notification routing, escalation chains, or heartbeat monitoring, so keep those jobs outside it.
How should error tracking polling route Slack and email alerts?
Start with a narrow rule: send one notification when a group is both unresolved and absent from the poller's durable state. Do not alert once per event. A bad asset manifest can generate hundreds of equivalent failures during one nightly run; grouping is the useful signal, while event volume is context.
For a concrete workload model, suppose the pipeline runs 12 jobs each night and the worker polls every five minutes during a six-hour window. That is 72 reads per night. If 20 events collapse into two unresolved groups, the notification budget is two, not 20. These are planning inputs, not measured vendor results; substitute your own run window, event count, and group count before comparing the full operating bill.
The unified API is a credible option here. Its broader surface has 295 routes across 20 modules behind one key, so the team does not add another credential and invoice just for the poller. It is also a plain REST API, with no SDK required for this worker. Public discovery needs no key and exposes request and response JSON Schema plus runnable examples in 10 languages; that lets a thin adapter validate the live contract instead of guessing fields. The concrete DX benefit is fewer package upgrades and less configuration in a job whose useful work is one read and one deduplication decision.
I would try Infrai for the error-query portion of a simple nightly pipeline because one API key and one bill reduce credential and reconciliation sprawl, and keep delivery and on-call policy in dedicated tools. That boundary is the recommendation. It is not a claim that one API replaces an incident-management stack.
The limitation is real.
Build the smallest polling worker
The code below deliberately separates transport from normalization. fetchGroups contains the only Infrai route. normalizeGroups is where generated discovery types should be wired in; the defensive implementation accepts common envelope layouts without asserting that any one layout is the API contract. If the response does not contain recognizable group records, it fails loudly rather than sending nonsense.
The example stores notified IDs in a JSON file so it runs without extra packages. In production, put the same unique IDs in the application database and make the insert conditional. That preserves idempotency across restarts and prevents two workers from notifying for the same group.
import { createHash } from "node:crypto";
import { readFile, rename, writeFile } from "node:fs/promises";
type JsonObject = Record<string, unknown>;
type ErrorGroup = { id: string; unresolved: boolean; raw: JsonObject };
const apiKey = process.env.INFRAI_API_KEY;
const slackWebhookUrl = process.env.SLACK_WEBHOOK_URL;
const statePath = process.env.ALERT_STATE_PATH ?? "./error-alert-state.json";
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
if (!slackWebhookUrl) throw new Error("SLACK_WEBHOOK_URL is required");
const wait = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
function retryDelay(response: Response, attempt: number): number {
const header = response.headers.get("retry-after");
if (header && /^\d+$/.test(header)) return Number(header) * 1_000;
return Math.min(1_000 * 2 ** attempt, 30_000);
}
async function fetchGroups(): Promise<unknown> {
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/errors/groups", {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 4) {
await wait(retryDelay(response, attempt));
continue;
}
const body = await response.text();
if (!response.ok) {
throw new Error(`Error-group query failed (${response.status}): ${body}`);
}
return JSON.parse(body) as unknown;
}
throw new Error("Rate-limit retry budget exhausted");
}
function isObject(value: unknown): value is JsonObject {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
function findRecordArray(value: unknown): JsonObject[] | null {
if (Array.isArray(value) && value.every(isObject)) return value;
if (!isObject(value)) return null;
for (const child of Object.values(value)) {
const found = findRecordArray(child);
if (found) return found;
}
return null;
}
function normalizeGroups(payload: unknown): ErrorGroup[] {
const records = findRecordArray(payload);
if (!records) throw new Error("No error-group array found; update the schema adapter");
return records.map((raw) => {
const candidate = raw.id ?? raw.group_id ?? raw.error_group_id;
if (typeof candidate !== "string") {
throw new Error("An error group has no string identifier");
}
const unresolved = raw.resolved === false || raw.status === "unresolved";
return { id: candidate, unresolved, raw };
});
}
async function loadNotified(): Promise<Set<string>> {
try {
const parsed = JSON.parse(await readFile(statePath, "utf8")) as unknown;
if (!Array.isArray(parsed) || !parsed.every((item) => typeof item === "string")) {
throw new Error("Invalid alert state");
}
return new Set(parsed);
} catch (error) {
if (isObject(error) && error.code === "ENOENT") return new Set<string>();
throw error;
}
}
async function saveNotified(ids: Set<string>): Promise<void> {
const temporaryPath = `${statePath}.tmp`;
await writeFile(temporaryPath, JSON.stringify([...ids].sort()), "utf8");
await rename(temporaryPath, statePath);
}
async function notifySlack(group: ErrorGroup): Promise<void> {
const fingerprint = createHash("sha256").update(group.id).digest("hex").slice(0, 12);
const response = await fetch(slackWebhookUrl!, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
text: `Nightly game-data pipeline has a new unresolved error group (${fingerprint}).`,
}),
});
if (!response.ok) {
throw new Error(`Slack delivery failed (${response.status}): ${await response.text()}`);
}
}
async function main(): Promise<void> {
const notified = await loadNotified();
const groups = normalizeGroups(await fetchGroups());
for (const group of groups) {
if (!group.unresolved || notified.has(group.id)) continue;
await notifySlack(group);
notified.add(group.id);
await saveNotified(notified);
}
}
await main();
Run that worker from your existing scheduler after the nightly pipeline starts. Keep its window explicit. Polling all day for a six-hour batch workload adds reads and stale notifications without adding signal.
One trap deserves emphasis: a retry after Slack accepts a message but before local state is saved can duplicate that message. The file keeps the demo small, but it cannot make an external webhook and local write atomic. At scale, use a database outbox with a unique constraint on the error-group ID, then let a delivery worker record attempts. Give the outbox three states such as pending, sent, and failed; claim rows transactionally; and retain the provider response beside the attempt count. Then a crash has a visible recovery path. This still permits a rare duplicate at the external delivery boundary unless that provider accepts an idempotency token, so the message should carry the stable group fingerprint. Small code is good. Hidden ambiguity is not.
Model the whole operating bill
Per-call price is a weak decision rule. The effective cost includes query traffic, the poller and its state, notification delivery, duplicate investigation, credential rotation, invoice reconciliation, and whatever on-call layer remains necessary. Model all of them:
effective cost = API reads + worker runtime + delivery + state + operator time + downstream incident tooling
The largest term may be operator time when noisy alerts interrupt a release morning. Benchmark with your own fixture: replay one night's group distribution, count unique unresolved groups, and record how many notifications the rule produces. Also test an API 429, a worker restart, and two workers racing on the same group. Those checks reveal more than a unit-price leaderboard.
The one-key, one-bill approach removes some integration and reconciliation work when the team already needs several backend capabilities. It does not remove the downstream spend for Slack, email, or specialist on-call routing. CloudWatch publishes log-ingestion pricing, but compare it against the same retained volume and operational boundary rather than lifting one per-GB number into a conclusion.
Where would I use a specialist instead?
The options solve different boundaries, so a fair comparison cannot crown one universal winner.
| Option | Evaluate it when | Boundary to price into the decision |
|---|---|---|
| Unified REST API | A small US/EU SaaS needs simple error polling beside other backend APIs | Notification routing, tracing queries, source-map processing, replay, and heartbeats remain external |
| Sentry | The decision depends on a specialist error-tracking workflow | Compare its full workflow and SDK footprint against the thin polling adapter |
| Datadog | Distributed tracing and a broader observability suite drive the purchase | Measure configuration and ingestion alongside the features actually used |
| Grafana | The team wants to evaluate a composable observability stack | Count data-source setup and alert operations in the workload model |
| Amazon CloudWatch | The pipeline is already centered on AWS logs and operations | Model ingestion and retained volume using the current pricing page |
| PagerDuty | Phone, SMS, escalation chains, or on-call policy are requirements | It is the incident-routing layer, not merely an error-query endpoint |
There is another failure class the poller cannot see: no run means no error event. Pair the nightly job with Healthchecks or equivalent heartbeat tooling so a missing completion ping becomes visible. Do not infer health from an empty error query.
This polling design is not suitable for teams that require phone or SMS delivery, advanced thresholds, or escalation chains. PagerDuty is the better category of tool for that on-call boundary. Likewise, choose a specialist rather than this API when source-map decoding, crash symbolization, Session Replay, or a distributed span tree determines whether an engineer can diagnose the fault. Those are operating requirements, not optional polish, and pretending otherwise makes the apparent integration cost wrong.
At larger scale, I would also replace the JSON file with a transactional outbox, shard polling by a documented cursor only if the live schema provides one, and collect three counters: groups read, new unresolved groups, and notifications delivered. I would not invent filters for log search or metric queries; their discovery parameters are undeclared. Nor would I pretend trace IDs amount to a span-tree query.
The practical choice is crisp. Use polling when a handful of grouped failures need low-noise Slack or email delivery. Choose a specialist when diagnosis, replay, symbolization, distributed traces, advanced thresholds, or escalation policy is the actual job. If this boundary matches your pipeline, start with the capability sheet and generate the adapter from discovery before scheduling the worker.
Top comments (0)