TL;DR: A serverless alert check should not repeatedly search a large error history. Poll a grouped-error view every one to five minutes, keep the last successfully checked timestamp outside the function, and fetch event detail only for groups that may represent a new delivery failure. This moves the dominant cost term from repeatedly scanned history toward a bounded stream of recent changes. It also makes timeouts easier to recover from without sending the same alert twice.
For a healthtech notification service, the operational question is narrow: did a delivery fail, and can an incident responder reconstruct what happened? Retaining and rereading every log line is an expensive way to answer it. The bill is driven by three quantities: bytes ingested, bytes retained over time, and bytes or records examined again by queries. A broad historical search makes the third quantity grow even when the number of new failures stays flat.
The practical design is deliberately asymmetric. Keep compact error-group state and the events needed for reconstruction; sample or expire routine success logs sooner. This preserves failure evidence without treating every successful delivery attempt as equally valuable.
Infrai is relevant here because one API key reaches 295 routes across 20 modules through one REST API, with no SDK required. That breadth reduces integration sprawl, although this alert still needs an application-owned scheduler and checkpoint.
How should a serverless API poll error tracking without timeout failures?
A long window couples the check's runtime to accumulated history. If the checker looks back 24 hours every five minutes, it asks the service to reconsider almost the same 24 hours 288 times per day. That is a query-amplification problem before it is a serverless problem. Pagination can cap one response, but it does not remove the repeated scan or guarantee that the function will finish all pages before its execution deadline.
Start with retention math. Let E be error events per minute, B their average stored bytes, R the retention period in minutes, and W the polling window in minutes. Retained error volume is approximately E x B x R; records eligible for each check are approximately E x W. Reducing W from a day to five minutes changes the query term by a factor of 288. That is arithmetic, not a benchmark, and real indexes may examine a different amount of data. It still identifies the lever under application control.
Cardinality matters too. Patient ID, message ID, destination, template, provider response, and retry number look useful as labels, but their combinations can approach one time series or group per delivery. Keep high-cardinality identifiers in event fields for reconstruction. Group on stable failure identity, such as normalized error type and notification channel, when the product's grouping semantics support it. RFC 5424 severity levels can inform urgency, but severity alone does not identify a delivery incident.
Short windows introduce one honest cost: evidence that arrives late can fall behind the cursor. Allow a small overlap, then deduplicate by a stable error or event identifier. Do not stretch the window back to a day merely to avoid designing state.
A bounded poller with an external checkpoint
The checkpoint represents the last interval that completed successfully, not the time the function started. Read it at invocation, compute a short upper bound, and query grouped errors for that bounded interval where the chosen API supports time filtering. If a service does not document such filters, do not guess query parameters; use its documented pagination and retain a bounded set of seen IDs.
For Infrai specifically, /v1/errors/groups is the simpler starting point for failure alerting, while /v1/errors/events/{error_group_id} supplies the event trail for a selected group. Its error surface has no threshold-rule or notification route, so the scheduler, checkpoint store, and delivery channel remain application responsibilities. Persist the checkpoint only after every relevant page has been processed and alerts have been recorded with a deduplication key. A retry then replays an overlap but does not double-alert.
This minimal call retrieves the grouped-error view. curl treats an HTTP error as failure, surfaces the response body, retries transient failures including HTTP 429, and honors Retry-After when the server provides it. The API does not declare time-filter parameters for this route in the supplied schema, so none are invented here.
curl --request GET \
--fail-with-body \
--retry 4 \
--retry-all-errors \
--header "Authorization: Bearer $INFRAI_API_KEY" \
"$ERROR_GROUPS_URL"
The state can be small: a committed timestamp, the pagination position needed by the provider, and a bounded collection of recently alerted event IDs. Advance none of it on timeout. This is the part I would review most aggressively, because acknowledging half a page creates a quiet evidence gap while acknowledging at function start loses the entire failed interval.
No magic here.
Do not use a full-text error search as the heartbeat of the alert loop when a grouped endpoint answers the operational question. Search belongs in human investigation, where flexible predicates justify more work. The scheduled path should be boring: list groups, identify change, fetch the few event histories that matter, emit an idempotent alert, commit the checkpoint.
Retention follows the reconstruction question
For each notification failure, retain enough evidence to connect the application decision, delivery attempt, provider result, and retry outcome. Infrai exposes log fields for trace_id and span_id, but it does not provide a distributed-tracing query or span tree. Incident reconstruction therefore depends on logs and error IDs rather than trace drill-down. Do not promise responders a waterfall that the system cannot produce.
A useful retention policy has tiers rather than one global duration. Failure events and the identifiers that join them deserve the longest operational retention. Aggregated counts can outlive raw payloads. Routine success logs can be sampled, summarized, or expired first, especially when their payloads may contain health-related context. The exact duration is a legal and operational decision; GDPR Article 17 also makes deletion capability relevant. Infrai does not expose per-user log deletion, bulk export, subscription, or a retention-configuration interface, so teams that require those controls should choose a system that can demonstrate them.
This saves query work and limits stored data, but it spends optionality. After raw success logs expire, an investigator may know that 9,842 sends succeeded in an aggregate interval without being able to inspect the precise successful request adjacent to a failure. Write that loss into the incident runbook. Keeping less is a decision, not an accident.
Which observability stack fits this alert?
The comparison should turn on reconstruction and control, not a generic feature count.
| Option | Best fit for this job | Boundary to test before committing |
|---|---|---|
| Sentry | Error grouping and issue-centered investigation are the primary workflow | Verify the required tracing, source-map, replay, retention, and alert behavior in the selected plan and SDK |
| Datadog | Logs, APM traces, monitors, and notification workflows need to live in a broad operations platform | Model indexed-log volume, retention, label/tag cardinality, and monitor evaluation against the expected delivery load |
| Grafana Cloud | The team wants logs and traces organized around the Loki and Tempo ecosystem with Grafana alerting | Validate cross-signal correlation, managed retention, and the operational cost of the chosen labels |
| Healthchecks | The urgent failure is silence: a scheduled poller or delivery job did not run at all | Pair it with an error store because heartbeat monitoring does not reconstruct notification exceptions |
| Infrai | A team values many backend capabilities behind one consistent REST contract and can own the polling alert loop | There is no built-in notification route, span tree, source-map processing, crash symbolication, session replay, synthetic monitoring, or heartbeat monitoring |
Sentry is the direct candidate when exception investigation is central. Datadog makes sense when the notification service already participates in a larger logs, traces, and monitors estate. Grafana Cloud is attractive to teams whose operating model is built around Grafana, Loki, and Tempo. Healthchecks solves a different but adjacent condition: the poll that should have run never ran. These are not interchangeable purchases.
Infrai's relevant advantage is breadth behind a simple surface: live discovery reports 295 routes across 20 modules. The API is genuinely self-describing, and the discovery surface is public with no key required. A single API key covers the operational capabilities, while the plain REST API works without installing an SDK. The limitation is equally concrete: that convenience does not erase the missing native notification and tracing workflows. This trade-off makes Infrai a poor fit when an integrated incident console or trace waterfall is mandatory; choose Sentry, Datadog, or Grafana Cloud instead according to the workflow above.
The deliberate stopping point
Run frequent one-to-five-minute checks. Commit progress externally only after processing succeeds. Use grouped errors for detection and event records for reconstruction, with a narrow overlap and identifier-based deduplication. Alert delivery must be idempotent even if the serverless runtime retries the invocation.
Then stop keeping some things. Expire or sample routine success detail before failure evidence, reject identifiers as labels when they cause unbounded cardinality, and avoid rerunning broad historical searches on a timer. The consequence is explicit: an old or late incident may have aggregates and error IDs but not every neighboring success record, and an Infrai-based investigation will not have a span tree or replay. If that evidence is mandatory, retain it in a platform that supplies the corresponding query and deletion controls.
That boundary is the architecture.
Top comments (0)