TL;DR: Error tracking can drive threshold alarms, notifications, and webhooks, but none of those delivery paths is an incident record. For a nightly logistics pipeline, keep durable run and shipment identifiers in structured events, evaluate thresholds over completed time windows, and store every alarm decision with its evidence. Use a webhook for low-latency delivery and API polling as reconciliation. If a free tier omits one of those features, move the small rule engine into your own scheduler rather than letting the notification channel define what you can detect.
The constraint is forensic: at 06:00, an operator must be able to explain which manifest failed, which shipments were affected, and whether a retry repaired them. A page saying “errors increased” cannot answer that. Fast delivery matters, but reconstructable evidence matters more.
What must survive when the notification does not?
A useful event needs stable correlation fields before it needs a destination. For this pipeline, I would record run_id, manifest_id, carrier, stage, shipment_id, attempt, and an error classification. The exception message remains useful for debugging, but it is a poor grouping key: messages can contain tracking numbers, paths, or other high-cardinality values, and small wording changes split one failure into several groups.
Treat notification delivery like OTP delivery. Acceptance by the next hop is not proof that a person saw it, and a retry can create duplicates. The same discipline used around rate limits and delivery gaps applies here: identify the message, make consumers idempotent, preserve the originating event, and reconcile later.
Three records should therefore have different lifetimes:
- The pipeline run record says what was scheduled, when it started, and whether it completed.
- Structured error events say which unit of work failed and at which stage.
- The alarm ledger says which rule was evaluated, over what window, against which event IDs, and where the resulting notification was sent.
Keep them separate. A chat message is an interface for attention, not a ledger.
Derive the rule from the operator's decision
“Alert after ten errors” sounds precise but hides the denominator. Ten address-validation failures among 40 shipments may stop dispatch; ten among two million may be routine bad input. An incident-reconstruction rule needs both a count and context, such as a failure ratio among attempted shipments for one carrier and one pipeline stage. It also needs a minimum sample size so that one failure out of one shipment does not look like a fleet-wide event.
For nightly work, closed windows are easier to defend than a perpetually moving counter. Evaluate after a manifest stage closes, or after a bounded grace period if completion can be delayed. Save the window boundaries in UTC and retain the pipeline's business date separately. Daylight-saving changes and late files otherwise make “last night” ambiguous.
Here is a small, vendor-independent evaluator. It returns evidence rather than sending anything, which keeps policy testable and delivery replaceable.
from dataclasses import dataclass
from datetime import datetime
@dataclass(frozen=True)
class Event:
event_id: str
run_id: str
stage: str
occurred_at: datetime
failed: bool
def evaluate_failure_ratio(
events: list[Event], *, minimum_attempts: int, ratio_limit: float
) -> dict[str, object]:
attempted = len(events)
failed_ids = [event.event_id for event in events if event.failed]
ratio = len(failed_ids) / attempted if attempted else 0.0
triggered = attempted >= minimum_attempts and ratio >= ratio_limit
return {
"triggered": triggered,
"attempted": attempted,
"failed": len(failed_ids),
"failure_ratio": ratio,
"evidence_event_ids": failed_ids,
}
The exact limits are business policy, not universal facts. Set them from the number of shipments an operator can safely defer, then test boundary cases: zero attempts, exactly the minimum sample, exactly the ratio limit, duplicate events, a late success, and a retry that belongs to the same shipment. A threshold without these semantics creates arguments during an incident instead of resolving them.
Can error tracking alerts use webhooks instead of polling?
These are delivery mechanisms with different failure surfaces. They are not competing definitions of an alarm.
| Mechanism | Best role | Forensic limitation | Required control |
|---|---|---|---|
| Built-in notification | Human attention | Formatting may omit correlation fields | Link or attach stable alarm and run IDs |
| Webhook | Low-latency machine handoff | Timeouts and retries can duplicate delivery | Signed requests, idempotency key, bounded retries |
| API polling | Reconciliation and backfill | Detection lags behind the polling interval | Durable cursor, overlap window, rate-limit handling |
A webhook receiver should acknowledge only after durably recording the delivery ID and body. If processing is expensive, enqueue it after that write. Authenticate the sender using the mechanism its contract specifies, compare signatures without timing leaks, reject stale timestamps where timestamps are part of that contract, and never log authentication secrets.
Polling deserves equal care. Store the cursor outside process memory. Query with a small overlap because events can arrive late or share a timestamp, then deduplicate by immutable event ID. Respect 429 Too Many Requests; RFC 6585 defines that status and notes that a response may include Retry-After. Add jitter to backoff so several workers do not wake together.
This yields a useful arrangement: webhook delivery opens the incident quickly, while polling repairs missed deliveries and confirms that the evidence set is complete. One channel can fail without erasing the history.
Build a replayable alarm ledger
The ledger turns an alert from a transient side effect into an explainable decision. Each row should include an alarm_id, rule version, evaluation time, window start and end, group dimensions, observed numerator and denominator, evidence IDs, state, and delivery attempts. Store secrets nowhere in that row. Retention and access controls should reflect that shipment identifiers can be sensitive operational data.
Rule versioning is crucial. If the ratio changes next month, an engineer must still be able to replay last night's evaluation under last night's rule. Mutable configuration without a captured version makes that impossible.
The delivery worker can then claim pending ledger rows and use alarm_id as the idempotency key. A receiver that has already accepted that ID returns success without repeating downstream work. This is ordinary distributed-systems hygiene, but it prevents a particularly nasty operational failure: a retry storm that pages several people for one manifest while the original pipeline error remains unexplained.
Quiet periods need detection too. A pipeline producing no error events may be healthy, or it may never have started. Emit a run-start record and a completion record, and alarm on a missing completion after the agreed deadline. Silence is data.
Practical limits of a free tier
The word “free” does not establish a technical contract. Before depending on any hosted error tracker, inspect its current documentation and test the exact account: Are rule-based alarms included? Can a rule filter on the structured fields you need? Are outbound webhooks supported? Is API access available, and what pagination, retention, and rate-limit behavior applies? Can delivery attempts be audited and replayed?
Do not infer those answers from a pricing label. Entitlements and quotas can change, while incident obligations remain. Capture the evaluated capability in a deployment check and keep the rule logic portable. If the service supplies events but not outgoing notifications, poll and evaluate locally. If it supplies webhooks but weak history, persist an alarm ledger on receipt. If it cannot export stable event IDs and timestamps, it cannot be the sole forensic source for this job.
There is also a compliance boundary. Structured fields should carry opaque internal identifiers, not addresses, phone numbers, or full shipment payloads. Redact exception context before export, document retention, and give responders the least access needed to investigate. The DO_NOT_TRACK convention concerns telemetry emitted by command-line tools; it is useful background for respecting operator choice, but it does not replace an application's data-governance policy.
Roll out without losing the old trail
Start in shadow mode for several nightly runs: write ledger decisions, send no pages, and compare each decision with pipeline completion records. Then enable one destination while continuing reconciliation polling. Track notification outcomes separately from pipeline outcomes, because “message delivered” and “manifest repaired” are different states.
Before retiring an older alarm, replay a fixed set of captured, redacted events through both rules and explain every difference. Keep a rollback path to the prior rule version. The finished design is modest: durable structured events, explicit window semantics, a versioned decision ledger, idempotent delivery, and reconciliation. Those pieces make the next morning's reconstruction possible even when the first notification never arrives.
Top comments (0)