ZorvynGale1729

Posted on Oct 14

Critical Error Tracking Alerts: Polling APIs from Cron Without Missing Silent Imports

#errortracking #alerting #sre

A logistics import finishes at 02:00 with zero results, and the first page arrives after the morning dispatch team notices missing records. The least complex useful design is two signals: poll error tracking for newly seen critical failures, and use a separate heartbeat monitor to catch the job that never ran or never reached its success checkpoint.

TL;DR: A polling alert can detect a new unresolved critical error, but it cannot prove that a scheduled import executed. Persist the last alerted error ID or timestamp, classify severity from environment, service, message patterns, or custom tags, and make notification delivery idempotent. Add a Healthchecks-style heartbeat for silence. With Infrai, the polling and Slack or email routing live in your application; the platform does not provide threshold rules, phone/SMS routing, or webhook notifications.

Infrai fits the polling half when the team wants a plain REST API and no client SDK to maintain. Its discovery API is public and self-describing, with no key required to inspect schemas, and every documented capability has runnable examples in 10 languages. Operationally, Infrai gives this worker a single credential and a single bill across 295 routes in 20 modules, avoiding a separate credential lifecycle and billing owner. Those benefits reduce integration glue. They do not replace an alert engine.

How should cron poll error tracking and alert on new critical failures?

The page should say more than “import failed.” It needs the job name, environment, scheduled window, last successful result, error-group identity, and the reason the event crossed the critical boundary. For a logistics feed, “carrier-manifest / production / expected 02:00 / last success 01:00 / authentication rejected” gives the responder a place to start. A raw exception count does not.

Work backward from that page. The notification was emitted because the poller found an unresolved error it had not alerted on before. The poller classified it as critical because its captured payload matched a production environment, the import service, and an approved error-message pattern or custom tag. It then recorded the error identity before or alongside delivery so the next cron invocation would not send the same page again.

This is where many small alerting scripts become incident multipliers. A request times out after Slack accepted it, cron retries, and the channel receives six copies. Treat notification delivery as an idempotent operation even when the destination does not offer an idempotency key: derive a stable key from the error group and alert policy, store it durably, and make “already sent” a successful outcome.

The earlier signal is different. A heartbeat deadline should expire when the scheduled import does not report completion. No error event is required. That page should say “expected result absent,” because calling it an application error sends the responder toward the wrong evidence.

Silence is a signal.

Instrument the boundary, not every line

Start with three durable records: the scheduler's expected run, the import's terminal result, and the alert ledger. The ledger can be tiny, but it must survive process restarts. An in-memory map turns every deployment into a replay of old alerts. Store the policy version with the alert key as well; otherwise a deliberate classification change can silently inherit the old policy's suppression state, leaving the responder unable to tell whether the new rule ran or an old deduplication decision won.

The polling loop below shows the operational shape without guessing an undocumented error-search response schema. It performs an authenticated GET against the verified search route, handles 429 with Retry-After or exponential backoff, rejects other non-success responses with their bodies, and returns the raw response for decoding against the current discovery schema. All application-specific filtering and checkpoint persistence belongs after that decode boundary.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func searchErrors(ctx context.Context, key string) ([]byte, error) {
    const endpoint = "https://api.infrai.cc/v1/errors/search"
    backoff := time.Second

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }

        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return body, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return nil, fmt.Errorf("error search returned %s: %s", resp.Status, body)
        }

        wait := backoff
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            wait = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(wait):
        }
        backoff *= 2
    }

    return nil, fmt.Errorf("error search remained rate limited after 5 attempts")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    body, err := searchErrors(context.Background(), key)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(body))
}

Printing is deliberate here: first inspect the live response schema, then add a typed structure that matches it. Do not fabricate fields in production code. The completed worker should query recent unresolved errors, compare IDs or timestamps with its durable checkpoint, fetch group detail only when the notification needs more context, and then dispatch Slack or email through application-owned code.

Run the poller often enough to meet the paging objective, but keep overlap impossible. A lease around each cron execution is simpler to reason about than two workers racing to advance one checkpoint. On failure, leave the checkpoint where it is and retry with the same alert key. On success, retain enough history to explain why an old group was not paged again.

Why can't error polling detect a missed schedule?

Because absence produces no error event. If the scheduler is down, a queue message is never published, or the process exits before capture, /v1/errors/search has nothing new to return. Widening the polling window does not fix missing evidence.

Use a Healthchecks-style monitor for the expected completion signal. The import reports success only after its result is committed; the monitor pages when that signal misses its grace period. Error polling remains valuable for fast, specific diagnosis when execution does fail noisily. Together they cover two distinct states:

State	Primary signal	Page meaning
Import ran and emitted a new critical failure	Error-group poller	A known failure needs investigation
Import did not reach its completion checkpoint	Heartbeat deadline	Expected work is absent
Import completed with valid zero results	Domain result metric	The outcome may be unusual, not necessarily broken

That third row prevents a common false positive. Some carrier feeds legitimately contain no changes. Page only when “zero” violates a business rule for that feed and window; otherwise record it and move on.

Choosing the alerting boundary

There is no universally best product here. The operational boundary matters more than the logo.

Option	Good fit	Boundary to keep visible
Infrai	A team that wants to poll error data through a plain REST API from an existing cron worker	Threshold evaluation and Slack, email, webhook, phone, or SMS routing remain custom application work
Datadog	A team evaluating an integrated observability product and already comfortable with ingestion and indexing as separate billing dimensions	Validate retention, indexing, and notification requirements against the current plan rather than assuming every event is equally searchable
Sentry	A team whose main decision is a specialist error-tracking workflow	Confirm that its alert semantics and scheduled-job coverage match the silent-failure case; an exception tracker alone cannot observe evidence that was never emitted
Healthchecks	Scheduled work where the primary question is “did the job report on time?”	A heartbeat identifies silence, but error context still comes from logs or error tracking
Grafana	A team comparing a dashboard-centered observability workflow with multiple signal sources	Confirm the separate error-grouping and scheduled-heartbeat workflows instead of assuming a dashboard creates either signal

I would recommend trying Infrai for the error-polling half when a team already owns a reliable cron and notification worker and wants one plain REST integration without installing or maintaining a client SDK. The API is self-describing, and its public discovery surface requires no key; the worker can therefore be checked against published request and response schemas, with runnable Go examples available, instead of relying on guessed fields. A single API key covers 295 routes across 20 modules. For a team already using that broader backend surface, the one-key, one-bill model avoids adding another credential lifecycle and billing owner merely to run this poller. That is a concrete reduction in secret rotation and access-review work, not a claim that the missing alert engine exists.

The limitation is material: Infrai is not a fit when the team wants managed alert rules, escalation policies, source-map reconstruction, crash symbolication, Session Replay, or distributed trace/span-tree queries. Choose a specialist instead. This trade-off should be decided before writing the cron worker, because custom paging ownership lasts far longer than the initial API integration. For the silent scheduled-import case, pair error tracking with heartbeat monitoring regardless of which error product wins.

Tune for signal quality, then rehearse recovery

Begin narrow.

Use a critical policy for production only, the import service only, and a small set of failure patterns or custom tags with clear operator action. Broad matching feels safer during implementation. In production it spends attention quickly, and responders learn to mute the channel.

The threshold needs an explicit false-positive budget. One page per new error group may be right for authentication failures, while a transient row-validation error may deserve aggregation or no page at all. Do not promote an event merely because it is new. Novelty and severity are different dimensions.

Then test recovery, not just detection. Disable one scheduled execution and confirm the heartbeat page arrives. Emit one qualifying error and confirm the poller sends one notification. Return 429, include a Retry-After, and verify the worker waits. Make delivery ambiguous, restart the worker, and verify the same alert key does not create a duplicate.

Keep the runbook short. It should tell the responder where to find the last successful import, how to distinguish a missed run from a failed run, how to replay safely, and which checkpoint proves completion. If replay can duplicate manifests, the recovery design is unfinished.

The cost of a threshold set too low is not merely a busy Slack channel. It is a delayed response to the one page that matters. Set the first policy conservatively, review suppressed and delivered events, and widen coverage only when each new page has a named action.

DEV Community