Short answer: poll an append-only error-log feed with a durable per-region cursor, turn each delivery failure into a stable incident key, and send a webhook only when that key changes state. The alert is the last step. The real requirement is preserving enough evidence to answer which gaming notification failed, where it failed, and what page fired without trusting a dashboard's current view.
For a notification service spanning the US and EU, one global last_seen_at value is a trap. Clocks tie, ingestion runs late, workers restart, and a recovered process can reread the same window. A page built directly from that window will either repeat or quietly skip evidence. Neither outcome is acceptable at 3 a.m.
Why incident reconstruction changes the polling design
Start the postmortem before writing the poller. Imagine a synthetic delivery event with event_id=evt_01JQ8K, region=eu, game_id=arena-17, channel=push, error_code=PUSH_TOKEN_REJECTED, and an occurrence timestamp. The useful incident question isn't "did errors increase?" It is "which delivery attempts shared a cause, when did that cause begin, and did the notification path recover?" A dashboard can answer the first question while erasing the sequence needed for the other three.
Keep two identities separate. The event ID identifies one immutable observation and makes replay harmless. The incident key groups related observations for paging; a reasonable starting shape is service, region, channel, and normalized error code. Don't include request IDs or timestamps in that key, because they manufacture a new incident for every attempt. Don't omit region either, because an EU-only delivery break and a simultaneous US break can have different owners, data boundaries, and rollback decisions.
Grouping is policy, not truth. Sentry documents how grouping algorithms and explicit fingerprints affect which events become one issue; the same warning applies to a homegrown watcher. A broad fingerprint hides distinct causes. A narrow one creates alert confetti. Store the raw fields alongside the computed key so an incident review can challenge the grouping later instead of accepting it as historical fact.
One detail matters more than it looks: the polling cursor is evidence, too. Persist the provider-issued continuation token when the logs API offers one. If the API only supports time ranges, use an overlapping window and deduplicate by immutable event ID, then advance the watermark only after both event storage and incident-state updates succeed. This is an at-least-once pipeline by design — duplicates are cheaper than missing the first failure that should have fired the page.
No cursor, no confidence.
How should a Node.js SaaS poll an error logs API for US and EU failures?
Keep failure detection outside the Node.js request path. The application should emit structured delivery outcomes, while a small watcher polls each regional logs API independently. That separation prevents a slow webhook destination from adding latency to game traffic, and it lets the watcher replay old observations without asking the notification service to fail again.
The watcher below is written in Go, but the contract is language-neutral: the logs endpoint and webhook are configuration, the cursor is opaque, and an event ID is required. The sample intentionally does not assume a public vendor route. Its state store is an interface because the safe implementation needs a transactional database or equivalent durable mechanism; an in-memory map would make a restart look like a brand-new incident.
package watcher
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
"time"
)
type Event struct {
ID string `json:"event_id"`
Occurred time.Time `json:"occurred_at"`
Service string `json:"service"`
Region string `json:"region"`
Channel string `json:"channel"`
ErrorCode string `json:"error_code"`
}
type Page struct {
Events []Event `json:"events"`
NextCursor string `json:"next_cursor"`
}
type Store interface {
Cursor(ctx context.Context, region string) (string, error)
ApplyPage(ctx context.Context, region, nextCursor string, events []Event) ([]Event, error)
}
type Poller struct {
Client *http.Client
LogsURL string
WebhookURL string
Store Store
}
func (p *Poller) Poll(ctx context.Context, region string) error {
cursor, err := p.Store.Cursor(ctx, region)
if err != nil {
return fmt.Errorf("read %s cursor: %w", region, err)
}
u, err := url.Parse(p.LogsURL)
if err != nil {
return fmt.Errorf("parse logs URL: %w", err)
}
q := u.Query()
q.Set("region", region)
if cursor != "" {
q.Set("cursor", cursor)
}
u.RawQuery = q.Encode()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
if err != nil {
return err
}
resp, err := p.Client.Do(req)
if err != nil {
return fmt.Errorf("poll logs: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("poll logs: status %d", resp.StatusCode)
}
var page Page
if err := json.NewDecoder(resp.Body).Decode(&page); err != nil {
return fmt.Errorf("decode log page: %w", err)
}
if page.NextCursor == "" {
return errors.New("logs response omitted next_cursor")
}
// ApplyPage atomically deduplicates IDs, updates incidents, and commits the cursor.
opened, err := p.Store.ApplyPage(ctx, region, page.NextCursor, page.Events)
if err != nil {
return fmt.Errorf("apply page: %w", err)
}
for _, event := range opened {
if err := p.notify(ctx, event); err != nil {
return err
}
}
return nil
}
func (p *Poller) notify(ctx context.Context, event Event) error {
payload := struct {
Text string `json:"text"`
}{Text: fmt.Sprintf(
"notification incident opened: service=%s region=%s channel=%s code=%s event=%s",
event.Service, event.Region, event.Channel, event.ErrorCode, event.ID,
)}
body, err := json.Marshal(payload)
if err != nil {
return err
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, p.WebhookURL, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
resp, err := p.Client.Do(req)
if err != nil {
return fmt.Errorf("send webhook: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("send webhook: status %d", resp.StatusCode)
}
return nil
}
This code still leaves one deliberate boundary: ApplyPage decides which state transition opens an incident. That decision belongs beside durable state, where event insertion, deduplication, incident mutation, and cursor advancement can commit together. If notification delivery fails after that commit, enqueue an outbox record in the same transaction and retry it separately. Otherwise a process crash between "incident opened" and "webhook sent" forces an ugly choice between losing the alert and reopening the incident.
Use timeouts on both clients, cap each response page, and serialize polling per region. Don't run two pollers against the same cursor unless the store provides a lease or compare-and-swap rule. More workers don't create more evidence; without coordination, they create races.
Make the page carry reconstruction evidence
A Slack webhook should carry a compact incident record, not a pasted stack trace. Put the incident key, region, first and latest occurrence, event count, representative error code, deployment identifier when available, and a link to the retained evidence in the message. Leave player tokens, message bodies, and other personal data out. The person carrying the pager needs a route to evidence, not the entire evidence set copied into another retention system.
The catch is notification delivery and incident evidence have different guarantees. A webhook is convenient for human attention, but it isn't suitable as the system of record: messages can be retried, channels can change, and chat retention may not match the incident policy. Keep canonical incident state in the watcher store, attach a unique notification ID to every outbox entry, and record the response class and attempt time. Slack is then a projection of incident state rather than the only place the incident exists.
For EU events, retention needs an explicit owner and deletion path. GDPR Article 17 defines a right to erasure with stated exceptions; it does not translate into "delete every log immediately" or "keep every log forever." I'm not sure one retention period can serve every gaming notification system, because the correct period depends on the data in each field, the lawful basis, and applicable obligations. Resolve that uncertainty with legal and privacy owners, then test deletion against raw events, incident aggregates, backups, and exported chat content. Pseudonymous event IDs help correlation, but they don't by themselves settle whether the record is personal data.
Keep US and EU cursors, raw-event partitions, and incident keys region-scoped when residency or access policy requires it. The central alert may contain only a regional incident reference. This makes cross-region correlation less convenient — a real limitation — but it prevents the alerting path from becoming an accidental data-export path.
Verify the detector before trusting the dashboard
Verification begins with a synthetic event whose outcome is known. Insert one failed notification in a non-production game tenant, confirm it appears once in the raw-event store, confirm its regional cursor advances, confirm exactly one incident transition is recorded, and confirm exactly one outbox item reaches the webhook. Replay the same page. Nothing new should fire.
Replays must be boring.
Then test the less comfortable edges:
- Return two events with the same occurrence time and different IDs; both must survive.
- Stop the watcher after storing a page but before webhook delivery; the outbox must deliver after restart without reopening the incident.
- Return the previous page again; event counts and notifications must remain stable.
- Open the same error code in US and EU data; the records must stay separable.
- Resolve an incident, then emit a later matching failure; the documented reopen policy must decide whether a fresh page fires.
Ask what page fired.
If the answer is merely "error count above zero," the detector isn't ready. A single rejected player token may be expected noise, while a deployment-correlated burst across one channel may deserve immediate attention. Thresholds need a minimum sample, a time window, and a reset rule. Those are operational policies, not universal constants, so tune them from reviewed incident data and record every change. Your mileage may vary, especially for low-volume games where one failure is a large percentage but a weak signal.
Also monitor the monitor: cursor age, poll duration, pages fetched, deduplicated events, open incidents, outbox age, and webhook attempts. The most dangerous state is a green delivery dashboard paired with a stalled cursor. Silence isn't health.
Roll back without deleting the trail
Treat detector rules and fingerprint versions as deployable configuration. A rollback should stop new pages, restore the previous grouping or threshold policy, and preserve the raw observations plus both policy versions. Never rewrite old incident keys in place during an active response; build a new projection and compare it first. Otherwise the rollback changes the evidence while responders are trying to explain it.
This design is not suitable when the logs API cannot provide stable event IDs, a continuation token, or a bounded time query. In that case, stick with direct event streaming or a durable queue from the notification service, because polling cannot manufacture ordering and identity that the source never exposed. Likewise, use platform-native alerting when it already provides durable deduplication, regional controls, and auditable state transitions; a custom watcher adds code ownership, schema migration work, credential rotation, on-call tests, and another component whose cursor can stall.
Rollback the paging action before rolling back collection. Disable webhook dispatch or route it to a test destination, keep ingesting and classifying events, and compare the proposed rule with the last known policy. Once the incident stream is stable, replay only unsent outbox entries. Don't delete duplicates to make a chart look tidy; explain why they were produced, correct the idempotency boundary, and retain the audit trail under the approved policy.
The final acceptance test is plain: given any page, a responder can identify the first durable event, the exact grouping policy, every state transition, the regional cursor that admitted it, and the notification attempt that sought attention. If any link exists only on a dashboard, the reconstruction is incomplete.
References
- Sentry, "Event Grouping and Fingerprinting": https://docs.sentry.io/concepts/data-management/event-grouping/
- GDPR, "Article 17: Right to Erasure": https://gdpr-info.eu/art-17-gdpr/
Top comments (0)