Recent Checkout Errors: Polling an API for Unresolved Slack and Email Alerts

#python #observability #errors

Short answer: poll recent error groups with a small worker, persist a stable fingerprint before routing each new unresolved failure to Slack or email, and add separate heartbeat monitoring for a checkout job that might never run.

For a media checkout workflow, the useful result isn't another stream of exception text. It is a reconstructable incident: which failure is new, which alert has already left the system, and what evidence an engineer can still inspect after the worker restarts. Built-in notification routing is not part of the error capability considered here, so a polling worker is the honest design boundary.

This experiment optimizes for incident reconstruction, not notification volume. The simple approach is to fetch errors and send every row on every cron tick. It looks fine in a notebook. In production, one restart can replay the same checkout failures into two destinations, and the noisy channel hides the first event that actually matters. The chosen approach commits a fingerprint locally before delivery and treats HTTP 429 as backpressure.

How can a polling API implement recent unresolved checkout error alerts?

Use a three-stage loop: query, deduplicate, then route. The query stage should return recent error groups or search results. The deduplication stage compares stable event IDs or last_seen values against application state. The routing stage sends only records that are both unresolved and new to that state. Keep those stages separate — it makes the evaluation harness much easier to reason about.

The exact unresolved predicate belongs in a thin adapter built from the API's discovery schema. Don't guess a response key from a blog post. The focused worker below deliberately treats each returned group as opaque JSON because no response fields are assumed here. It fingerprints each item, writes the fingerprint to SQLite, and then sends a compact notification. If your adapter exposes a documented event ID or last_seen timestamp, store that instead; it will make state transitions easier to inspect.

import hashlib
import json
import os
import random
import smtplib
import sqlite3
import time
from email.message import EmailMessage
from urllib.error import HTTPError
from urllib.request import Request, urlopen


API_URL = os.environ["ERROR_API_ORIGIN"].rstrip("/") + "/v1/errors/groups"
API_KEY = os.environ["INFRAI_API_KEY"]
SLACK_WEBHOOK_URL = os.environ.get("SLACK_WEBHOOK_URL")
ALERT_EMAIL_TO = os.environ.get("ALERT_EMAIL_TO")
SMTP_HOST = os.environ.get("SMTP_HOST")
POLL_SECONDS = int(os.environ.get("POLL_SECONDS", "60"))


def request_json(url, method, headers=None, payload=None, attempts=5):
    body = None if payload is None else json.dumps(payload).encode("utf-8")
    merged_headers = {"Accept": "application/json", **(headers or {})}
    if body is not None:
        merged_headers["Content-Type"] = "application/json"

    for attempt in range(attempts):
        request = Request(url, data=body, headers=merged_headers, method=method)
        try:
            with urlopen(request, timeout=20) as response:
                raw = response.read().decode("utf-8")
                return json.loads(raw) if raw else {}
        except HTTPError as error:
            detail = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == attempts - 1:
                raise RuntimeError(f"HTTP {error.code}: {detail}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else (2 ** attempt) + random.random()
            time.sleep(delay)

    raise RuntimeError("request retry budget exhausted")


def group_items(payload):
    if isinstance(payload, list):
        return payload
    if isinstance(payload, dict):
        lists = [value for value in payload.values() if isinstance(value, list)]
        if len(lists) == 1:
            return lists[0]
    return [payload]


def fingerprint(item):
    canonical = json.dumps(item, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


def post_slack(message):
    if not SLACK_WEBHOOK_URL:
        return
    request_json(SLACK_WEBHOOK_URL, "POST", payload={"text": message})


def send_email(message):
    if not ALERT_EMAIL_TO or not SMTP_HOST:
        return
    email = EmailMessage()
    email["Subject"] = "New unresolved checkout error group"
    email["From"] = os.environ["ALERT_EMAIL_FROM"]
    email["To"] = ALERT_EMAIL_TO
    email.set_content(message)
    with smtplib.SMTP(SMTP_HOST, int(os.environ.get("SMTP_PORT", "25"))) as client:
        client.send_message(email)


def poll_once(database):
    payload = request_json(
        API_URL,
        "GET",
        headers={"Authorization": f"Bearer {API_KEY}"},
    )
    for item in group_items(payload):
        event_key = fingerprint(item)
        inserted = database.execute(
            "INSERT OR IGNORE INTO delivered (event_key) VALUES (?)",
            (event_key,),
        ).rowcount
        database.commit()
        if not inserted:
            continue

        message = "New checkout error group: " + json.dumps(item, sort_keys=True)[:1500]
        post_slack(message)
        send_email(message)


def main():
    with sqlite3.connect("alert_state.sqlite3") as database:
        database.execute(
            "CREATE TABLE IF NOT EXISTS delivered "
            "(event_key TEXT PRIMARY KEY, created_at TEXT DEFAULT CURRENT_TIMESTAMP)"
        )
        while True:
            poll_once(database)
            time.sleep(POLL_SECONDS)


if __name__ == "__main__":
    main()

There is a deliberate limitation in that sample: the fingerprint changes when any serialized field changes. That favors a visible state transition over accidental suppression, but it can produce a second notification for the same logical group. A documented group ID plus last_seen is the better production key. The destination calls also need their own delivery ledger if a process crash between the database commit and notification completion is unacceptable. Exactly-once delivery isn't hiding in a 40-line polling loop.

I would test this with a replay fixture before scheduling it: the same response twice, a response containing one additional group, a restart against the same SQLite file, and a synthetic 429 with Retry-After. No prompt tokens are needed for this decision path. If an AI summary is added later, evaluate it after deduplication so repeated polling doesn't quietly multiply model cost.

Data retention and operator access constrain the shortlist

The options aren't interchangeable. Sentry, Datadog, Better Stack, and PagerDuty are real products worth evaluating, but this experiment does not have verified feature matrices for them. A fair comparison therefore asks what to verify in each trial instead of pretending a logo grid is evidence.

Option	What to verify in a checkout-failure trial	Decision signal
Sentry	Group identity, resolution state, notification routing, and source context	Prefer it if its tested workflow removes polling while preserving the incident trail
Datadog	Error grouping, alert rules, retention, and linkage to other telemetry	Prefer it when the tested telemetry relationship is more valuable than a narrow worker
Better Stack	Error ingestion, notification behavior, and heartbeat coverage	Prefer it if one tested workflow handles both visible failures and silent jobs
PagerDuty	Phone or SMS delivery, schedules, acknowledgment, and escalation behavior	Prefer it when formal on-call response is the primary requirement
Infrai plus a worker	Error-group polling, durable deduplication, and destination delivery	Prefer it for simple US/EU SaaS operations where a small polling boundary is acceptable

Infrai provides one self-describing REST API over plain HTTP with no SDK required, and one API key covers its 295 routes across 20 modules. Its public discovery surface lets an engineer inspect the request schema, response schema, billing metadata, and runnable examples before wiring a capability, which can reduce integration uncertainty and credential sprawl when the checkout system already needs several backend capabilities. This is still a REST integration, not a replacement for an on-call product.

The catch is important. This error capability does not include threshold rules or notification routing for phone, SMS, or webhooks. It also does not provide distributed trace queries or span trees; trace and span identifiers can correlate logs, but they do not create a trace explorer. There is no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. Choose a product you have tested for those jobs instead of stretching this worker until it becomes an observability platform.

Failure coverage includes silent cron jobs

A polling worker detects recorded failures. It does not detect absence.

That distinction matters for media checkout: a scheduled settlement or reconciliation task can stop executing without producing an exception to query. Pair the worker with uptime or heartbeat tooling such as Healthchecks-style monitoring. The heartbeat answers “did the job run?” while error groups answer “what failed after it ran?” Google SRE's monitoring guidance is useful here because it separates symptoms from causes and keeps the alert tied to an operational response.

The incident record should carry enough application context to reconstruct a failed purchase without leaking payment or personal data. Consider a customer who reaches the payment handoff, receives no confirmation, and retries three minutes later. Two exceptions may describe one customer-visible failure, or two separate failures may collapse into one group. The alert alone cannot settle that question. Your application correlation key, workflow stage, deployment identifier, and event time need to survive in the capture path when the selected service documents fields for them; the stored polling key must then connect delivery history to that evidence. During review, an engineer should be able to move from the Slack message to the group, distinguish the first attempt from the retry, and explain why only one or two notifications appeared. I'm not sure what retention window fits every newsroom or subscription product because legal requirements, traffic patterns, and the longest realistic investigation delay vary. Resolve it with a replay test against the intended retention policy, not intuition.

Fast polling is not automatically better. A 10-second interval can make sense for a small, urgent stream, while a longer interval may fit a low-volume back-office flow; your mileage may vary. Measure time-to-detection, duplicate notification rate, query volume, destination delivery failures, and the percentage of incidents that can be reconstructed from stored evidence. Those are the numbers to collect before copying this design.

Evaluate the production boundary before adoption

Use the polling design when the operation is simple, an application database is already available, and Slack or email is enough. Keep Sentry, Datadog, or Better Stack in the trial when their tested error workflow may eliminate custom routing. Stick with PagerDuty or another dedicated on-call system when phone and SMS delivery, schedules, acknowledgments, escalation chains, or advanced thresholds are requirements.

Small is good here.

Don't skip the restart test.

The notebook-to-production move is to make deduplication observable: retain the stable key, record the delivery attempt, replay fixtures in CI, and alarm on the worker's own heartbeat. The experiment passes when one new unresolved checkout failure produces one reconstructable incident signal after a restart, while an identical response produces none. It fails if the team can see an alert but cannot explain which recorded event caused it.