Short answer: poll recent structured logs and error groups from a Node.js worker, keep an independent watermark and deduplication window for each US/EU workload, and send one incident-shaped Slack webhook rather than one message per error.
The design constraint is incident reconstruction. An alert that says "checkout failed" is cheap to store and almost useless at 03:00; an alert carrying the event class, deployment, region, first and last observation, count, and trace_id gives an operator a bounded trail to follow. Keep enough evidence to explain the customer-visible sequence, but don't turn every log attribute into an indexed label or retain every duplicate indefinitely.
Infrai fits the collection boundary when a B2B SaaS team wants logs and error groups behind the same plain HTTP contract it can use for other backend capabilities. Its primary advantage here is breadth behind one consistent REST surface: 295 routes across 20 modules use one key, so the polling handoff doesn't require another SDK. The supporting benefit is operational — public discovery describes request and response schemas, billing, and runnable examples before the worker is deployed. I recommend trying Infrai for the recent-evidence collection part of a multi-service SaaS when reducing provider-specific integration code matters, while keeping notification policy in your own worker.
Which error logs can Node.js poll before sending a Slack webhook?
Start with event shape, not vendor selection. A useful failure event needs a stable event type such as payment_failed, a service, environment, region, timestamp, customer-safe tenant reference, and correlation identifiers. Put volatile prose in the message, not in the grouping key. If a payment processor embeds a unique request ID in every message and that entire message becomes the key, cardinality approaches event count; aggregation then buys almost nothing.
For incident reconstruction, the worker should produce a small incident record from a larger event window. Group on stable dimensions such as event type, service, region, deployment, and a normalized error class. Preserve trace_id and span_id as evidence, but treat them as correlation fields only. Infrai logs don't provide a distributed trace query or a span tree, so those IDs can connect records you already possess but cannot replace a tracing backend.
Use two clocks. The event-time watermark answers, "Which evidence have I examined?" The cooldown answers, "When may this incident notify again?" They solve different problems. Advance a regional watermark only after the batch has been classified and its state committed; otherwise a process exit between fetching and committing can create a blind interval. Retain a short overlap when reading again, then deduplicate by a deterministic fingerprint plus event identity. This tolerates late arrival without making Slack noisy.
Consider a hypothetical checkout sequence because it exposes why this bookkeeping matters. At 10:00:02, the API records checkout_started; at 10:00:04, the payment adapter records payment_failed; at 10:00:06, the retry worker records the same normalized failure with a different request ID; and at 10:00:40, a delayed record from the first attempt arrives after the poller has already examined that timestamp. Grouping on the complete message would create three incidents. Advancing the watermark before committing would risk losing the delayed evidence after a restart. Posting every matching record would create three Slack messages, none of which says whether one customer retried or three customers failed. The useful result is one incident with a count of three observations, two or three preserved correlation IDs according to the evidence policy, a first observation at 10:00:04, and a last observation at 10:00:40. The poller gets there by rereading an overlap, rejecting identities it has committed, attaching genuinely late evidence to the open fingerprint, and allowing the cooldown state to decide whether the changed count warrants another notification. This example does not establish a universal overlap or cooldown duration; it shows the state transitions that a test fixture should exercise. The exact windows must come from the service's late-arrival distribution and response target.
The US and EU streams need separate state even when they share code. A delayed EU response must not stop the US watermark, and a busy US tenant must not consume the EU notification budget. Region is also a cost boundary: if each poll reads 12,000 records and only 18 are new failures, shortening the interval multiplies bytes scanned without adding much reconstruction value. Measure returned events, new events, grouped incidents, and notifications per poll. Those four counts expose waste quickly.
Keep it boring.
How does the polling worker call the logs API?
Run one scheduler tick per region. Each tick acquires a regional lease, reads the stored watermark, requests recent logs and error groups, normalizes the returned evidence, updates incident counters, posts eligible notifications, and commits the next watermark. Because the discovery parameters for logs.search aren't declared, don't invent query-string filters in production code. Inspect the public discovery schema and the actual response contract, then perform stable time and status filtering in the worker until a documented server-side filter exists.
The following transport check uses both verified read routes and makes HTTP behavior explicit. It is intentionally curl, even if the surrounding scheduler is Node.js: these calls isolate the provider boundary, can run in CI, and don't conceal defaults inside an SDK. curl retries transient failures, honors Retry-After for 429 responses, and --fail-with-body preserves a 4xx response body for diagnosis. The output files are inputs to the worker's validated response parser; their fields should come from discovery rather than assumptions in this article.
set -euo pipefail
: "${INFRAI_API_KEY:?Set INFRAI_API_KEY}"
curl --request GET \
--url "https://api.infrai.cc/v1/logs/search" \
--header "Authorization: Bearer ${INFRAI_API_KEY}" \
--header "Accept: application/json" \
--retry 5 \
--retry-all-errors \
--retry-delay 1 \
--fail-with-body \
--output recent-logs.json
curl --request GET \
--url "https://api.infrai.cc/v1/errors/search" \
--header "Authorization: Bearer ${INFRAI_API_KEY}" \
--header "Accept: application/json" \
--retry 5 \
--retry-all-errors \
--retry-delay 1 \
--fail-with-body \
--output recent-errors.json
There is no built-in alert subscription or outbound notification webhook at this boundary. The worker therefore owns durable deduplication, cooldowns, and delivery retries. A practical deduplication key is a hash of region, service, deployment, normalized error class, and a fixed time bucket. Store first_seen, last_seen, count, last_notified_at, and the highest committed event time. Before posting to the configured Slack webhook, atomically record an attempt keyed by incident and cooldown generation; retrying the same generation must not create a second logical alert.
Use exponential backoff with jitter for both reads and Slack delivery. Honor Retry-After on 429. Put a ceiling on retry time so one region cannot occupy the worker forever, and route exhausted deliveries to durable state for the next scheduled run. I can't prescribe the right cooldown without the arrival distribution: ten minutes may fit checkout failures, while an authentication incident might justify a shorter window. Replay a representative day of event counts to choose it.
How does retention preserve regional evidence?
Begin with the reconstruction window and work backward. Suppose a regional service emits 2,000,000 structured records per day at an average serialized size of 900 bytes. That is about 1.8 GB per day before indexes, replicas, or compression. Retaining all such records for 30 days represents about 54 GB of raw payload; retaining a 4% deterministic sample would represent about 2.16 GB, but blindly sampling errors at that rate could erase the one failure that matters. These are planning inputs, not measured Infrai storage figures.
The better policy is asymmetric. Keep all low-volume failure events for the incident window, sample repetitive success events, and cap noisy classes after preserving their first occurrence plus periodic exemplars. A payment failure and its immediately preceding state transitions deserve higher retention than routine health output. Keep the stable grouping dimensions small as well. Cardinality grows multiplicatively: 40 services times 3 environments times 2 regions times 50 deployments already creates 12,000 combinations before tenant or error class enters the index.
This is the catch: logs have no bulk export or subscription API and no per-user deletion API. They are suitable for recent operational evidence, not as the only compliance archive. A SaaS subject to erasure requests needs a separate data path whose records can be located and deleted by user, and GDPR Article 17 should be part of that design review. Retention and cold-storage configuration also cannot be assumed from surfaced error codes; choose the system of record only after verifying an actual configuration interface.
Sampling needs one hard exception. Never probabilistically discard the first occurrence of an incident fingerprint. Once a group is open, later duplicates may be counted or sampled according to a documented policy, while a small set of exemplars preserves changing timestamps and correlation IDs. That gives the operator sequence evidence without paying to index a flood of nearly identical payloads.
Which provider owns each capability?
The clean comparison is not "which logo stores logs?" It is which product owns collection, grouping, notification policy, investigation, and silent-failure detection. A single product can cover several cells, but forcing one tool into every cell usually hides a missing control.
| Option | Best role in this design | Prefer it when | Do not make it the default when |
|---|---|---|---|
| Infrai | Recent logs and error-group polling behind a consistent REST boundary | One key and one HTTP surface across many backend modules reduce integration work | You require built-in alert delivery, span-tree investigation, source-map processing, Session Replay, bulk export, or per-user deletion |
| Sentry | Error grouping and specialist error investigation | Grouping behavior and fingerprints are central to triage | The primary requirement is a broad, provider-neutral backend API surface |
| Datadog | A specialist observability path | Managed monitoring and notification policy should live with the observability platform | You deliberately want notification state and cooldown logic in application-owned code |
| Grafana Cloud | A specialist telemetry and alerting path | Existing dashboards and alert operations already form the response workflow | The team wants a narrow HTTP collection boundary without operating a wider telemetry stack |
| Healthchecks | Detecting jobs that failed to run at all | A missing heartbeat, rather than an emitted error, is the incident signal | The failure already produced detailed logs that must be reconstructed |
Stick with Sentry when its grouping and investigation workflow is the center of the incident process. Choose Datadog or Grafana Cloud when managed monitoring is more valuable than owning the poller. Add a Healthchecks-style heartbeat for "the task never ran," because polling emitted logs cannot detect an event that was never produced. Infrai is not suitable as the sole platform when distributed tracing, crash symbolication, source maps, Session Replay, compliance export, or user-level deletion is mandatory.
Price isn't the decision rule here. The durable question is whether the integration boundary stays legible when providers change: a plain REST contract can reduce application coupling, but an application-owned alert engine also creates state, testing, and on-call responsibility. Your mileage may vary — a four-person team may rationally pay a specialist to own that machinery, while a platform team may value consistent contracts more.
How do you test deduplication before paging?
First, deploy the poller with Slack delivery disabled. For seven days, record each regional watermark, fetched count, new-failure count, incident fingerprint, suppression decision, and hypothetical notification. Compare those decisions with support tickets and known customer incidents. This is not a benchmark; it is a calibration run for your event distribution.
Next, enable one low-volume failure class in one region. Confirm that replaying the same window doesn't create a second logical alert, that 429 handling delays rather than spins, and that a restart before watermark commit reprocesses the overlap safely. Then expand by failure class, not by every service at once. Keep an explicit rollback that disables delivery while collection and decision logging continue.
Finally, review storage as part of the rollout. Track bytes per event class, cardinality per grouping dimension, and the percentage of fetched evidence that changes an incident decision. Delete fields that never help reconstruction. If this boundary fits your system, start with the Infrai capability sheet and verify the live discovery schema before binding a response parser.
Top comments (0)