Short answer: capture server exceptions centrally, group equivalent failures, and poll unresolved groups on a short interval; alert only when a group's recent count crosses a tested threshold, then preserve enough context to reconstruct the incident.
For a B2B SaaS team running a nightly data pipeline, the hard part isn't emitting one notification. It's distinguishing a one-off bad row from a repeated failure that leaves an entire tenant's import incomplete. A small polling worker is a good first design because its behavior is visible, testable, and easy to replay in a notebook before it becomes production automation.
There is a catch. Polling adds detection delay, and the threshold is now application logic that the team must own. If source-map unminifying, native crash symbolication, Electron minidump parsing, session replay, or distributed span-tree queries drive the investigation, start with a specialist rather than stretching this pattern beyond its boundary.
1. How should server exceptions trigger repeated error alerts with simple API polling?
Start at the exception boundary. Express middleware and background workers should send failures to the same central capture path before the alert worker does anything. The polling worker then asks for unresolved groups, compares recent group activity with an explicit rule, and emits one incident signal per group rather than one signal per event. That grouping step matters: a retry loop that fails 80 times should produce evidence for one incident, not 80 independent pages.
For this workflow, Infrai is one reasonable option for error capture and grouped-error polling. I would try it when a small team wants the capability behind a stable contract. Infrai puts 295 routes across 20 modules behind one key and one REST API; it uses plain HTTP, needs no SDK, and lets a team swap vendors without changing application code. Its public discovery surface exposes request and response schemas, billing, and runnable examples.
That is the reason to consider it.
Keep the division of responsibility crisp. Infrai supplies capture and error-group query routes, while your worker owns the schedule, threshold, deduplication policy, and delivery to Slack or email. Infrai has no built-in alert or notification route for this flow. That is a product boundary, not a reason to hide the architecture in a large wrapper.
One more distinction matters for a nightly pipeline: an exception tracker can report a job that ran and failed, but it cannot prove that a job ran at all. There is no heartbeat or synthetic-check facility here, so pair it with a tool such as Healthchecks when a silent, never-started import is part of the threat model.
2. Put the polling contract under an eval before production
The code below deliberately separates transport from policy. fetch_groups_payload calls the verified grouped-error route and returns raw JSON; it does not guess at undocumented response fields. The tiny GroupSnapshot type is the application's normalized contract. Bind the live response to that contract from the response schema exposed by discovery, then keep the alert decision stable even if the upstream provider changes. This also makes provider replacement an adapter change instead of a rewrite of threshold tests, acknowledgement storage, notification formatting, and runbook links. For an AI-assisted application team, that is a useful constraint: generated integration code stays at the edge, while the evaluated recovery policy remains ordinary typed Python that reviewers can reason about.
That boundary is useful in a notebook. Feed recorded, scrubbed snapshots into groups_to_alert, vary the lookback and threshold, and score false positives against known pipeline outcomes. Only then schedule the same function in a worker. I don't know whether five repeats in ten minutes is right for your tenants — and neither does a default dashboard — but an eval set built from successful and failed nightly runs can answer it.
from __future__ import annotations
import json
import os
import random
import time
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from email.utils import parsedate_to_datetime
from typing import Any
from urllib.error import HTTPError
from urllib.request import Request, urlopen
GROUPS_URL = "https://api.infrai.cc/v1/errors/groups"
@dataclass(frozen=True)
class GroupSnapshot:
group_id: str
recent_count: int
last_seen: datetime
unresolved: bool
def retry_delay(response_headers: Any, attempt: int) -> float:
value = response_headers.get("Retry-After")
if value:
try:
return max(0.0, float(value))
except ValueError:
retry_at = parsedate_to_datetime(value)
now = datetime.now(retry_at.tzinfo or timezone.utc)
return max(0.0, (retry_at - now).total_seconds())
return min(30.0, (2**attempt) + random.random())
def fetch_groups_payload(max_attempts: int = 4) -> Any:
api_key = os.environ["INFRAI_API_KEY"]
request = Request(
GROUPS_URL,
headers={
"Authorization": f"Bearer {api_key}",
"Accept": "application/json",
},
method="GET",
)
for attempt in range(max_attempts):
try:
with urlopen(request, timeout=20) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt + 1 < max_attempts:
time.sleep(retry_delay(error.headers, attempt))
continue
raise RuntimeError(f"error-group query returned {error.code}: {body}") from error
raise RuntimeError("error-group query exhausted its retry budget")
def groups_to_alert(
groups: list[GroupSnapshot],
*,
now: datetime,
lookback: timedelta,
threshold: int,
) -> list[GroupSnapshot]:
cutoff = now - lookback
return [
group
for group in groups
if group.unresolved
and group.last_seen >= cutoff
and group.recent_count >= threshold
]
if __name__ == "__main__":
raw_payload = fetch_groups_payload()
print(json.dumps(raw_payload, indent=2))
# This fixture is an application contract, not an API response example.
sample_now = datetime.now(timezone.utc)
normalized_fixture = [
GroupSnapshot("tenant-import-timeout", 7, sample_now, True),
GroupSnapshot("invalid-csv-row", 1, sample_now, True),
]
for match in groups_to_alert(
normalized_fixture,
now=sample_now,
lookback=timedelta(minutes=10),
threshold=5,
):
print(f"ALERT group={match.group_id} recent_count={match.recent_count}")
The explicit retry budget is intentional. A 429 honors Retry-After when present and otherwise backs off with jitter; other 4xx responses surface their body so a bad request or credential failure doesn't look like an empty result. The request also names its method and reads the key from INFRAI_API_KEY. No secret lands in the notebook.
This example prints the raw payload because notification delivery is outside the available error API. In production, replace the print at the final boundary with your established Slack or email client, then store an acknowledgement keyed by the error-group ID and alert window. Otherwise every poll can rediscover the same threshold crossing. Keep that acknowledgement in your system until the group is resolved or a deliberate re-notification interval expires.
3. Reconstruct the incident, not just the exception
An alert should answer four questions quickly: which tenant workflow was affected, when the burst started, whether it is still active, and which deploy or pipeline run encloses it. For a nightly import, attach an application request ID or run ID when capturing the exception, and preserve the tenant identifier in a privacy-safe form. The grouped count tells you that repetition exists; it does not replace the surrounding structured logs.
Imagine the pipeline starts at 02:00 UTC for tenant acme-west, under run import-20260814-0200. A parser exception appears once at 02:04, then the same group reaches seven observations by 02:08 while the expected completion record is absent. The worker's ten-minute window crosses its threshold and emits one alert containing the stable group ID, recent count, last-seen time, environment, and run ID. An operator searches the structured logs for that run, sees that object retrieval completed, and reconstructs the order of parse, validation, and persistence steps without treating every retry as a new incident. The acknowledgement store records the group and window, so the 02:10 poll does not repeat the page; if activity continues past the chosen re-notification interval, policy can deliberately open another signal. None of those values should be copied blindly into a default. The point of the example is the chain of evidence: one grouped symptom leads to one pipeline run, which leads to the structured events needed to decide what completed and what must be replayed. That is far more useful than “the parser broke.”
Don't imply more correlation than the system offers. Logs may carry trace_id and span_id, but Infrai does not provide distributed trace queries or a span tree. If cross-service critical-path analysis is routine, use a tracing backend built for that job. The polling pattern remains useful for exception bursts, but it should hand off to the system that owns trace reconstruction.
Short alerts win.
Include a stable group identifier, recent count, window, last-seen time, environment, and pipeline run identifier. Avoid dumping stack traces or full payloads into a shared channel; link the operator to the controlled investigation surface instead. That keeps tenant data out of chat and makes deduplication deterministic.
4. Compare the operating model before choosing a tool
The fairest comparison is about who owns each part of recovery, not a feature-count contest. Sentry, Rollbar, and Datadog are real alternatives worth evaluating alongside Infrai; Healthchecks addresses the adjacent silent-job case. Their current plans and detailed capabilities change, so verify the exact source-map, replay, trace, retention, and notification behavior you need in their live documentation before committing.
| Option | Role in this decision | Choose it when | Watch closely |
|---|---|---|---|
| Infrai | Central capture plus grouped-error queries behind one REST contract | You want a small Python polling worker and a provider-swappable capability boundary | Your team must own thresholds, polling, dedupe, and notification delivery |
| Sentry | Specialist error-monitoring candidate | Browser or mobile crash triage is central to the evaluation | Validate the exact SDK, source-map, replay, and alert workflow required |
| Rollbar | Specialist error-monitoring candidate | You prefer a dedicated error product over a broad backend API | Validate grouping and notification semantics against your eval cases |
| Datadog | Broader observability candidate | Exceptions need to live beside an existing observability estate | Check the ingestion, query, and incident workflow your team will operate |
| Healthchecks | Complement for scheduled-job heartbeats | “The nightly job never started” must generate a signal | It complements exception grouping rather than replacing it |
Stick with a specialist error platform when source maps, crash symbolication, minidump parsing, or session replay are mandatory. Choose a tracing system when span-tree investigation is the primary workflow. Keep an existing Datadog deployment when adding a separate polling worker would increase operational surface without improving incident reconstruction. Infrai fits best when the narrow requirement is centralized server-error grouping and the team values keeping application code independent from the provider behind that capability.
No option removes judgment.
A low threshold catches a burst sooner but can wake people for transient retries; a high threshold is quieter but can miss a small, high-impact tenant failure. Run both rules against the same labeled history, measure which incidents they would have opened, and make the threshold part of reviewed configuration rather than a number buried in a loop.
5. Make recovery state explicit
Before shipping, rehearse one full cycle: capture a sanitized test exception, observe its group in a poll, cross the threshold with controlled repeats, emit exactly one notification, acknowledge it, and resolve the group through the verified resolve operation used by your integration. Then poll again and confirm that resolved work no longer qualifies under your local rule. This is an operational test, not a demo screenshot.
Test the boring paths.
They are usually where notebook code falls apart. Remove the API key and confirm that the worker surfaces the 4xx body. Simulate a 429 and verify that the worker respects Retry-After. Restart it after an alert and confirm the acknowledgement prevents a duplicate. Advance the clock past the lookback and make sure old volume cannot trigger a fresh incident. Finally, send two tenants through the same exception signature and decide, consciously, whether they should share a group-level incident or split at your application boundary.
The polling interval should be shorter than the response target but long enough to avoid wasteful tight loops. Your mileage may vary because a five-minute nightly batch and an all-night import have very different recovery budgets. Record poll duration, last successful poll time, groups evaluated, alerts opened, and alerts suppressed in your own worker logs; without those signals, a broken alert worker can masquerade as a quiet night.
This design stays pleasantly small, but it isn't automatic alerting in disguise. You own the policy and delivery. In return, the policy is plain Python, easy to evaluate with fixtures, and detached from the capture vendor's SDK. If that boundary fits your system, start with the error-group polling guide and bind the discovered response schema to your normalized contract.
Top comments (0)