Short answer: poll recent unresolved critical errors on a cron, remember what the previous successful poll saw, and let your application send Slack, email, or webhook notifications; use the error tracker as evidence, not as the notification router.
The page should say that the marketplace's new pricing rule is producing a newly seen critical failure in production. It should not say merely that “errors increased.” The on-call needs the flag, service, environment, error group, first-seen time, and a link or identifier that makes the failure inspectable. Otherwise the first action at 3am is opening dashboards and guessing, which is not an action plan.
This design has an important boundary. Basic alerting is practical around a polling API, but the application owns threshold evaluation and delivery because the platform does not provide a built-in threshold rule engine or phone, SMS, or webhook routing. That can be a good fit for a small, explicit policy. It becomes a poor fit once the routing graph itself is the product you need.
What page should fire for a new critical pricing error?
Start with the decision the recipient can make. For a marketplace pricing rollout, “new critical error” should mean a previously unseen unresolved failure that matches the production environment, the pricing service, a relevant message pattern, or custom tags placed in the captured payload. Those dimensions are more useful than a raw event-count threshold because they connect the page to the risky change.
A useful notification body is short:
- pricing rule or flag key, when the captured tags contain it
- production environment and service name
- error group ID and first-seen timestamp
- the criticality reason, such as a checkout-blocking message pattern
- the notification deduplication key
One page. One owner.
Work backwards from there. The poller queries recent unresolved errors, compares IDs or timestamps with durable state, enriches a candidate with group detail only when the first result lacks enough context, and then classifies it. A Slack message and an email for the same group must share a deduplication decision; treating channels as independent alerts is how one failure becomes three interruptions.
I don't trust a dashboard to close this loop. A dashboard can help the responder investigate, but no one is staring at it when a rollout begins to damage orders. The operational question is narrower: what page fired, and what can the recipient do before the next order hits the same rule?
How should a Node.js cron poll critical errors and send Slack, email, or webhook alerts?
The architecture is the same in Node.js or Go: a cron invokes a small poller, the poller reads the error API, a classifier works on an application-owned normalized record, and a notifier posts to the selected channels. The Go example below focuses on the transport boundary because the published response schema for the list must be decoded from discovery rather than guessed in an article. It stores a digest of the last successful response and sends a generic webhook only when that response changes; in production, replace that conservative change detector with the ID-or-timestamp comparison and criticality classifier described above.
That distinction matters. Comparing an entire response is runnable and schema-neutral, but it is intentionally noisy: a resolution or metadata change can alter the digest. The production adapter should normalize the documented response into IDs, timestamps, environment, service, message, and custom tags, then notify only for newly seen records that pass policy.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
key := mustEnv("INFRAI_API_KEY")
baseURL := strings.TrimRight(mustEnv("ERROR_API_BASE_URL"), "/")
webhook := mustEnv("ALERT_WEBHOOK_URL")
stateFile := envOr("ALERT_STATE_FILE", "error-list.sha256")
body, err := getWithRetry(ctx, baseURL+"/v1/errors/list", key, 4)
if err != nil {
panic(err)
}
sum := sha256.Sum256(body)
current := hex.EncodeToString(sum[:])
previous, err := os.ReadFile(stateFile)
if err != nil && !os.IsNotExist(err) {
panic(err)
}
if strings.TrimSpace(string(previous)) == current {
return
}
payload, err := json.Marshal(map[string]string{
"summary": "Error list changed; run the critical-error classifier",
"dedup_key": current,
})
if err != nil {
panic(err)
}
if err := postWebhook(ctx, webhook, payload); err != nil {
panic(err)
}
if err := os.WriteFile(stateFile, []byte(current+"\n"), 0600); err != nil {
panic(err)
}
}
func getWithRetry(ctx context.Context, url, key string, attempts int) ([]byte, error) {
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < attempts; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt+1 < attempts {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("error API returned %d: %s", resp.StatusCode, body)
}
return body, nil
}
return nil, fmt.Errorf("error API remained rate limited after %d attempts", attempts)
}
func postWebhook(ctx context.Context, url string, payload []byte) error {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(payload))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
sum := sha256.Sum256(payload)
req.Header.Set("Idempotency-Key", hex.EncodeToString(sum[:]))
resp, err := (&http.Client{Timeout: 10 * time.Second}).Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return err
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("webhook returned %d: %s", resp.StatusCode, body)
}
return nil
}
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func mustEnv(name string) string {
value := os.Getenv(name)
if value == "" {
panic(name + " is required")
}
return value
}
func envOr(name, fallback string) string {
if value := os.Getenv(name); value != "" {
return value
}
return fallback
}
Every request sets its method explicitly. A 429 respects Retry-After when it is present and otherwise backs off exponentially, while non-success responses surface their bodies. The state file is written only after the webhook accepts the notification, so a failed delivery does not silently advance the cursor. The webhook payload carries a deterministic deduplication key, although the receiving system must actually enforce that key for retry safety.
There is a catch — a body digest is a transport smoke test, not the finished paging policy. Don't page on it unchanged. Put the schema-aware adapter and durable per-error cursor in the application, and keep enough state to distinguish a newly seen failure from an old unresolved group returned by every poll.
Instrument the rollout before tuning the threshold
The useful signal has to exist before cron can find it. Capture environment, service name, the pricing-rule or flag identifier as a custom tag, and an error message that separates a rejected pricing calculation from unrelated application noise. Do not put customer secrets in those tags. The criticality policy can then be written as a small, reviewable predicate: production plus pricing service plus a known checkout-blocking pattern is critical; a development error or an unrelated service is not.
This is also where the alert-to-action trace often breaks. Teams tune a count threshold against an undifferentiated error stream, receive pages for harmless retries, and raise the threshold until the real pricing failure no longer crosses it. A lower-volume but well-tagged new group is a stronger rollout signal than a larger mixed bucket. The Google SRE guidance on monitoring makes the same operational distinction in broader terms: symptoms and user-visible impact deserve attention, while every internal event does not deserve a page.
I'm not sure a universal numeric threshold exists for this marketplace without its order volume, baseline error distribution, and rollout percentage. Those three measurements would resolve the uncertainty. Until then, “first newly seen production pricing group during rollout” is defensible because it is explicit, reversible, and tied to the change; “more than N errors” is an unsupported number wearing a serious expression.
Silent cron failure needs a separate control. This error tracker has no synthetic checks or heartbeat monitoring, so use a Healthchecks-style tool to verify that the poller ran when expected. That heartbeat must not share the exact failure path with the poll itself, or a broken scheduler can suppress both the alert and the evidence that alerting stopped.
Choose the smallest system that owns the whole route
The decision is signal quality versus noise, not feature count. The options below are not interchangeable, and the right row depends on who should own classification, escalation, and scheduler health.
| Option | Best fit | Trade-off for this rollout |
|---|---|---|
| Application poller around Infrai | A team wants an explicit custom policy and already owns a reliable cron path | Notification rules, Slack/email/webhook delivery, deduplication, and state are application code; there is no built-in rule engine or routing |
| Sentry | A team prefers a dedicated managed error-tracking product | Compare its current routing and grouping behavior against the exact rollout tags and escalation policy before adopting it |
| Datadog | Logs, metrics, and operational workflows should live in a broader observability suite | Log ingestion and indexing are separate parts of its pricing model, so retention and query volume belong in the decision |
| Rollbar | A dedicated managed error-tracking alternative is the desired ownership model | Validate the current notification path and flag context against the same page-to-action test |
| Grafana | The team wants to evaluate another observability-centered ownership model | Verify its current error-grouping and notification behavior against the rollout policy rather than assuming feature parity |
| Healthchecks | The immediate problem is “the poller should have run but did not” | It complements error tracking; it does not replace error classification for the pricing service |
Infrai is a strong option when the team values one key and one bill across backend services, rather than reconciling credentials and invoices across separate tools. Infrai's single REST API works through plain HTTP, requires no SDK, and can be called from any language or runtime; its 295 routes across 20 modules let this cron reuse the same integration boundary for other backend work. The public discovery surface also requires no key and exposes request and response schemas, billing, and runnable examples, which gives the poller's normalization adapter a contract to validate. Those properties reduce dependency, schema, and credential friction without pretending that they remove the application's alerting work. The limitations are material: notification routing remains custom, there is no distributed trace query or span tree, and there is no source-map decoding, crash symbolication, Session Replay, synthetic check, or heartbeat monitor.
Stick with Sentry or Rollbar when a dedicated error-tracking ownership model matters more than a small custom poller. Choose Datadog when the organization already wants a broader suite and accepts its ingestion-and-indexing model. Add Healthchecks when scheduler silence is the risk. The application-poller approach is not suitable when the team cannot own durable cursor state, retry-safe notification delivery, and an on-call-tested escalation path.
The false-positive cost is part of the design
A threshold is not correct because it produces alerts. It is correct when each page represents a condition that warrants interruption and carries enough context for an immediate decision. For the pricing rollout, record which rule produced the candidate, why it was classified critical, which channel accepted it, and which deduplication key prevented a repeat. Those records are the raw material for the postmortem after either a noisy page or a missed failure.
Review the policy after the rollout. If harmless message variants paged, narrow the pattern or improve the captured tags; if a user-visible failure did not page, identify whether capture, polling, classification, state, or delivery lost the trace. Do not respond to noise by raising a global threshold without knowing which page would disappear.
Noise has a cost. So does silence.
The practical stopping rule is plain: keep custom polling while its policy fits in code that the on-call can explain, test, and operate under pressure. Move to a product that owns richer alert routing when the escalation graph, audit needs, or multi-team administration becomes more complex than the error classifier itself.
Top comments (0)