Short answer: poll only a bounded window of recently changed unresolved error groups, deduplicate by a stable incident key, and charge Slack and email delivery to the marketplace service that created the evidence. Keep the raw event briefly, retain the grouped incident longer, and measure the poller itself. That preserves enough context to reconstruct a customer incident without turning every retry into another stored log line or alert.
This design is useful when an error tracker exposes a query API but built-in alerting is unavailable or intentionally disabled. The governing constraint is evidence, not notification volume: an on-call engineer must be able to answer which customer action failed, which release handled it, and which notification path ran. Everything else has to justify its bytes.
What evidence does a marketplace incident actually require?
Start with the reconstruction question. For a checkout or seller-payout failure, an incident record needs a stable error-group identifier, first-seen and last-seen timestamps, current resolution state, affected service and release, a trace or request correlation identifier, and a privacy-safe tenant attribution key. It also needs notification state: the destination class, the first delivery time, and the last material version sent. This is a logical schema, not a claim about any error tracker's response fields; an adapter should map the selected provider's documented response into it.
Do not copy the full event into the alert ledger. Store a pointer to the raw evidence plus the small set of fields required for triage and allocation. A customer email address, request body, or payment token doesn't become safer because it moved from an error tracker into a notification table. Redact before ingestion, then apply access controls and deletion policy to both stores.
Keep less, deliberately.
Retention should follow the longest defensible reconstruction window. A practical policy might keep raw events for 7 days and normalized incident groups for 30 days, but those numbers are design inputs, not universal recommendations. Set them from support escalation latency, refund or dispute windows, legal requirements, and the time engineers actually need to reproduce a release. If a marketplace can receive a customer dispute after 45 days, a 30-day incident ledger is insufficient even when it looks economical. If disputes close in 72 hours, keeping verbose stack-local variables for a year is hard to defend.
Cost attribution belongs in the schema before the first poll. Assign each stored byte and each notification attempt to a service, environment, and cost center. Tenant identifiers are useful for incident search, but they are dangerous metric labels: 50,000 merchants multiplied by 12 services, 3 environments, 4 regions, and 20 error classes has an illustrative upper bound of 144 million label combinations. Without the merchant dimension, the same product is 2,880. Put high-cardinality identifiers in logs or traces where they can be queried under retention controls; reserve metric labels for bounded dimensions.
How should a Node.js cron job poll recent unresolved errors for alerts?
Use an overlap window and a durable cursor together. Suppose the job runs every 5 minutes. Query records updated since the previous successful cursor minus a 2-minute overlap, sort by the provider's stable update field and identifier, and advance the cursor only after the page has been normalized and committed. The overlap catches clock skew and updates near a page boundary; the incident key absorbs duplicates. A plain "now minus 5 minutes" query has neither guarantee.
Duplicates are expected.
The API URL should come from configuration because providers use different paths and filtering syntax. The following request deliberately assumes only an HTTPS query URL supplied by the adapter. The example timestamp is fixed so the command is reproducible; the scheduler substitutes its persisted window start.
curl --fail-with-body --silent --show-error \
--connect-timeout 5 --max-time 20 \
--retry 2 --retry-all-errors \
--get "$ERROR_QUERY_URL" \
--header "Authorization: Bearer $ERROR_API_TOKEN" \
--header "Accept: application/json" \
--data-urlencode "status=unresolved" \
--data-urlencode "updated_after=2026-08-15T03:58:00Z" \
--output "$SNAPSHOT_PATH"
A Node.js worker can schedule that adapter, validate the returned document against the provider's published schema, and write the cursor and normalized groups in one database transaction. Run only one active poller per scope, using a database lease or scheduler concurrency policy. The lease is about duplicate work, while the incident key is the final correctness boundary; deployments, process restarts, and retry timing can still cause the same group to be observed twice.
Pagination deserves more attention than the timer. Freeze the query boundary for one run, follow the documented next-page mechanism, cap total pages, and record whether the cap was reached. If the source supports conditional requests, preserve its ETag and send If-None-Match on the next equivalent query; HTTP defines a matching conditional GET response as a way to avoid retransmitting an unchanged representation. Don't invent that behavior for an API that doesn't document it.
Page caps matter.
A poll run should produce four bounded measurements: duration, source groups read, incident groups changed, and notification attempts by channel and outcome. Watch error rate, latency, and saturation around the worker as well as traffic through it; these are the four monitoring signals described in the Google SRE guidance. Avoid attaching incident IDs or tenant IDs to those counters.
Deduplicate before Slack and email delivery
Notification state should be a deterministic function of the incident, not of a poll execution. One usable key is source + project + error_group_id + material_version + channel. Here, material_version changes only when information that an operator would act on changes: the incident reopens, severity crosses a defined threshold, or a new release becomes implicated. A rising occurrence count can update the incident without sending another message every 5 minutes.
The delivery transaction has an awkward boundary — the database and a remote notification endpoint cannot usually commit atomically. An outbox makes that boundary explicit. In the same transaction that upserts the incident, insert one outbox row protected by a unique notification key. A separate sender claims pending rows, records attempts, and marks successful delivery. Retries reuse the row rather than manufacture a new alert. Slack and email are two channel projections of the same incident version, so their attempts can be costed separately without creating two incident histories.
For Slack, send a compact summary and a link to the authorized incident view. The webhook endpoint is secret configuration, not application data.
curl --fail-with-body --silent --show-error \
--connect-timeout 5 --max-time 20 \
--retry 2 --retry-all-errors \
--request POST "$SLACK_WEBHOOK_URL" \
--header "Content-Type: application/json" \
--data-binary "@$SLACK_PAYLOAD_PATH"
Email should use the same outbox contract through the organization's approved mail transport. Don't place a large stack trace in either channel. Besides leaking data into another retention domain, that makes notification bytes grow with evidence bytes. The alert should identify the incident and the reason it became actionable; the evidence store should hold the detail.
The catch is latency. A 5-minute cron interval plus API and queue time cannot satisfy a 30-second paging objective. Polling is also not suitable when the source offers no stable ordering, cursor, or updated timestamp, because a changing result set can create gaps during pagination. Stick with a documented push integration, event stream, or native alert path when the response-time objective is tighter than the poll cycle or when the source can provide stronger delivery semantics.
Attribute storage and alert cost without guessing
Cost starts with a byte model. For an illustrative workload of 100,000 error events per day at an observed average of 1,200 bytes after redaction, raw ingest is 120 MB per day, or 3.6 GB over 30 days using decimal units. This is arithmetic, not a benchmark. Compression, indexing, replicas, query scans, and regional pricing can move the billed figure, so measure serialized bytes at the ingestion boundary and reconcile them with the provider invoice. CloudWatch, for example, documents log charges in terms that include data ingestion and other usage dimensions; the linked pricing page is the current authority for its regional rates.
A useful ledger separates quantities that teams can control.
| Cost object | Allocation key | Quantity | Policy lever |
|---|---|---|---|
| Raw error evidence | service, environment, cost center | ingested bytes | redaction, sampling, retention |
| Normalized incident | owning service | rows and retained bytes | grouping, resolution retention |
| API polling | poller scope | requests and bytes read | interval, conditional reads, page size |
| Slack delivery | service, incident version | attempts and payload bytes | routing, deduplication |
| Email delivery | service, incident version | attempts and payload bytes | severity policy, digesting |
Sampling requires care because the rare event may be the one a support engineer needs. Sample repeated event bodies only after grouping, preserve the first event for every group and release, and retain aggregate counts for the omitted repeats. A fixed 10% sample is easy to explain but can erase a low-volume payment failure while retaining thousands of common validation errors. Adaptive rules are better aligned with reconstruction: keep novel groups and state transitions, then reduce identical repeats. Still, your mileage may vary; replay tests against historical incident shapes are what resolve that uncertainty.
Count notification cost at attempt time, not only on success. Failed attempts consume worker time and may consume provider requests, while a success-only ledger assigns retry-heavy services an artificially low share. At the same time, alert policy needs a noise budget: record the ratio of delivered notifications to acknowledged or acted-on incidents, then review routes that generate repeated unowned alerts. It isn't a universal quality score, but it exposes where storage and delivery are paying for no operational decision.
Roll out the poller with replay evidence
Begin in shadow mode for one full retention cycle: query and normalize, but route the outbox to a test sink. Compare sampled source groups with normalized incidents, force page boundaries, restart between cursor writes, reopen a resolved fixture, and verify that two overlapping runs create one outbox row per channel. Also test secret rotation, timeouts, rate-limit responses, malformed source documents, and an exhausted page cap. The intended response is explicit: preserve the last committed cursor, record the run outcome with bounded labels, and retry according to the source's documented guidance.
Then enable one low-risk marketplace service, first with Slack and later with email. Review raw bytes, normalized bytes, poll requests, notification attempts, duplicate suppression, and reconstruction success before expanding scope. Rollback should disable new outbox creation while preserving cursor and incident state; deleting that state turns a routine rollback into a duplicate-alert event.
Done is boring: a support engineer can move from a customer report to one incident record, the on-call sees one actionable notification per material change, and finance can attribute the evidence and delivery quantities to the service that produced them.
Top comments (0)