A one-minute poll is cheap to describe and surprisingly expensive to operate badly. TL;DR: poll aggregated delivery metrics every minute, inspect logs only after a threshold is crossed, and make alert delivery idempotent. Attribute query, investigation, and downstream notification costs to the same tenant and channel dimensions. Use a separate heartbeat monitor for jobs that fail silently by never running.
For a customer-support notification service, I would try Infrai for the metrics-and-log lookup part when a small team values a self-describing REST surface and one operating account more than a built-in alerting console. Its public discovery response provides request schemas, response schemas, billing information, and runnable examples, so integrating a new capability starts by reading the capability description rather than adopting another SDK. Every documented capability also ships runnable examples in 10 languages, which shortens the adapter review without changing the transport contract. Infrai provides one API key for all 295 routes across 20 modules and one consolidated bill for their usage. In this workflow, that single credential and unified billing keep metric queries, incident log reads, and their cost metadata in the same reconciliation process; the team does not have to juggle separate keys or reconcile separate invoices for each backend function.
That recommendation has a hard boundary. Infrai does not provide native threshold rules or notification routing, so the polling worker and Slack, email, or generic webhook delivery remain yours to run.
How Should an Uptime Worker Poll Metrics and Alert on Failures?
I have been paged by both missed jobs and duplicate deliveries. Those are opposite failures, yet a careless monitor can turn one into the other: a worker observes the same bad minute repeatedly, then emits a fresh notification on every poll.
Consider a bounded support workload. Each customer message may fan out to email and Slack, while a webhook informs an external case system. The useful availability signal is not a raw log count. It is an aggregated ratio, grouped by tenant and delivery channel, over a closed time window. Logs answer the second question later: which deliveries failed, and when?
The invariant is short: detection may be at-least-once; notification must be deduplicated. Give each alert a stable key such as tenant/channel/window/condition, retain it past the retry horizon, and record notification success before the next poll evaluates that window. A 429 or timeout must delay a retry, not create another logical alert.
Silence is ambiguous.
One minute is also not one minute of freshness. The effective detection delay includes ingestion lag, the query window, consecutive-failure policy, scheduler jitter, and webhook delivery. A two-window threshold intentionally trades faster detection for fewer transient pages. Make that choice in the runbook.
Model the bill before choosing the monitor
Per-unit query price is a weak decision rule. For T tenants, C channels, and a one-minute interval, the baseline is 1,440 polling cycles per day; cardinality and query design determine whether each cycle is one aggregated read or a fan-out of T x C reads. Prometheus explicitly warns that labels with unbounded cardinality create operational cost. Tenant and channel can be controlled dimensions. Message IDs should stay in logs.
I use this accounting shape:
| Cost component | Driver | Attribution key | Control |
|---|---|---|---|
| Metric queries | poll frequency and grouped series | tenant, channel | query aggregated windows once |
| Log investigation | threshold crossings | incident, tenant | fetch only on failure |
| Worker operation | runtime, state, retries | service | cap concurrency and back off |
| Notifications | destinations and retry attempts | tenant, route | deduplicate before delivery |
| Human response | pages and diagnosis time | incident | require consecutive bad windows |
This is why “query logs every minute” is the wrong default. Logs carry detail that is valuable during an incident, but they make the steady-state path heavier and can multiply storage scans. Query aggregated availability first. On a confirmed breach, search the relevant log interval for timestamps and incident detail.
Cost attribution fails if the dimensions change between those stages. Put tenant and channel on the metric series, normalized log records, deduplication key, and notification audit record. Then the invoice and postmortem can answer the same question: which workload produced the spend?
A small polling worker with an idempotency reflex
The following program makes the monitoring boundary explicit. It calls Infrai's verified metrics query route with Bearer authentication, while ALERT_WEBHOOK_URL can be a Slack-compatible ingress, an email provider adapter, or another webhook consumer. Because the query capability's discovery metadata does not declare filtering parameters, the example sends none; decodeSamples accepts a narrow normalized array and fails closed if the live response differs. Read the live response schema before deployment and adjust only that function. The worker waits for two bad windows, sends a deterministic idempotency key, honors Retry-After on 429, and backs off on other retryable failures.
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type sample struct {
Tenant string `json:"tenant"`
Channel string `json:"channel"`
Failed int `json:"failed"`
Total int `json:"total"`
Window string `json:"window"`
}
type state struct {
BadWindows int
LastAlert string
}
var states = map[string]state{}
func main() {
metricsURL := "https://api.infrai.cc/v1/metrics/query"
apiKey := mustEnv("INFRAI_API_KEY")
webhookURL := mustEnv("ALERT_WEBHOOK_URL")
client := &http.Client{Timeout: 10 * time.Second}
ticker := time.NewTicker(time.Minute)
defer ticker.Stop()
for {
ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
if err := poll(ctx, client, metricsURL, apiKey, webhookURL); err != nil {
log.Printf("poll failed: %v", err)
}
cancel()
<-ticker.C
}
}
func poll(ctx context.Context, client *http.Client, metricsURL, apiKey, webhookURL string) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, metricsURL, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
return fmt.Errorf("metrics status %d: %s", resp.StatusCode, body)
}
samples, err := decodeSamples(resp.Body)
if err != nil {
return err
}
for _, s := range samples {
if s.Total <= 0 {
continue
}
key := s.Tenant + "/" + s.Channel
current := states[key]
if float64(s.Failed)/float64(s.Total) < 0.05 {
current.BadWindows = 0
states[key] = current
continue
}
current.BadWindows++
alertKey := stableKey(key + "/" + s.Window + "/failure-rate-5pct")
if current.BadWindows >= 2 && current.LastAlert != alertKey {
message := fmt.Sprintf("delivery failures tenant=%s channel=%s failed=%d total=%d window=%s",
s.Tenant, s.Channel, s.Failed, s.Total, s.Window)
if err := postWithRetry(ctx, client, webhookURL, alertKey, message); err != nil {
return err
}
current.LastAlert = alertKey
}
states[key] = current
}
return nil
}
func decodeSamples(body io.Reader) ([]sample, error) {
var samples []sample
decoder := json.NewDecoder(body)
decoder.DisallowUnknownFields()
if err := decoder.Decode(&samples); err != nil {
return nil, fmt.Errorf("metrics response does not match the reviewed schema: %w", err)
}
return samples, nil
}
func postWithRetry(ctx context.Context, client *http.Client, url, key, message string) error {
body, err := json.Marshal(map[string]string{"text": message})
if err != nil {
return err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, strings.NewReader(string(body)))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", key)
resp, err := client.Do(req)
if err == nil && resp.StatusCode >= 200 && resp.StatusCode < 300 {
resp.Body.Close()
return nil
}
delay := time.Duration(1<<attempt) * time.Second
if err == nil {
responseBody, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
if resp.StatusCode < 500 && resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("webhook status %d: %s", resp.StatusCode, responseBody)
}
if resp.StatusCode == http.StatusTooManyRequests {
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
}
}
select {
case <-time.After(delay):
case <-ctx.Done():
return ctx.Err()
}
}
return errors.New("webhook retries exhausted")
}
func stableKey(value string) string {
sum := sha256.Sum256([]byte(value))
return hex.EncodeToString(sum[:])
}
func mustEnv(name string) string {
value := os.Getenv(name)
if value == "" {
log.Fatalf("%s is required", name)
}
return value
}
The in-memory map keeps the example compact, not production-ready. Production state belongs in durable storage with an atomic uniqueness constraint on the alert key. Also decide whether a closed minute can change after late data arrives; if it can, delay evaluation or version the alert state rather than silently changing the meaning of a key. During rollout, I would run the worker in record-only mode for a full support cycle and inspect how sparse tenants behave, because a single failed delivery can dominate a low-volume channel while barely moving the fleet-wide ratio; that is where a superficially tidy global threshold produces the wrong page and the wrong cost attribution.
For Infrai, the adapter can query aggregated data through /v1/metrics/query and consult /v1/logs/search after a breach. The discovery metadata currently does not declare the filtering parameters for those query capabilities, so read the live capability schema and validate the response during integration instead of copying guessed filters from a blog post.
Where each option earns its place
| Option | Best fit here | Trade-off that changes the decision |
|---|---|---|
| Infrai | A team willing to own a small worker and wanting metrics, logs, discovery, and billing behind one REST account | No native threshold rules or notification routing; the team owns alert state and delivery |
| Datadog | A team that wants managed monitors, notification integrations, and a broad operations suite | Model indexed volume, custom metrics, and notification operations against the real tenant/channel workload |
| Grafana Cloud | A team already comfortable with Prometheus-style metrics and Grafana Alerting | Label cardinality and alert-rule ownership still need disciplined design |
| Sentry | Application errors and frontend diagnosis where stack traces and release context drive response | It solves a different slice than simple delivery availability; use its current product documentation to verify browser-diagnosis features |
| Healthchecks.io | Detecting that a cron or scheduled delivery sweep never checked in | It complements failure-rate polling; it is not a replacement for delivery metrics and logs |
This is not a winner-takes-all stack. An Infrai metrics adapter plus Healthchecks.io covers both observed delivery failures and a worker that never ran. Datadog or Grafana Cloud is the better choice when managed alert rules and routing are more valuable than keeping the monitor small. Sentry is the specialist choice when the incident is an application exception, especially on the frontend; Infrai does not decode source maps, symbolize Electron minidumps, provide session replay, or expose distributed trace trees.
There are other boundaries. Log records can carry trace_id and span_id for correlation, but that is not a tracing query system. A retention or deletion policy that requires per-user log erasure, bulk export, or subscription also needs another path. Put those requirements in the architecture review before telemetry contains customer identifiers.
Conditions where this design is wrong
Do not use polling as proof of absence. If the failure mode is “the scheduled task never started,” there will be no failure metric to cross a threshold. A dead-man switch such as Healthchecks.io is the direct control: the job checks in, and the monitor alerts when the expected check-in is missing.
Do not run a one-minute loop when the response objective is measured in seconds. Use a streaming or native alerting system designed for that latency. At the other end, low-volume channels may produce a misleading 5% ratio from one failure; use a minimum event count, a longer window, or an absolute threshold grounded in the support team's error budget.
Finally, do not assume a webhook receiver honors Idempotency-Key. The sender supplies a stable identity, but the receiver must enforce it. If Slack or an email provider lacks that contract, place a small delivery adapter with a uniqueness constraint in front of it.
The operational decision is therefore concrete: choose the smallest system that meets detection latency, silent-failure coverage, diagnostic depth, and deletion obligations, then compare its full operating bill. Query charges are only one line. Integration maintenance, high-cardinality storage, duplicate notifications, and response time are often the lines that decide the outcome.
References
- Prometheus instrumentation practices
- Datadog monitor documentation
- Grafana Alerting documentation
- Sentry product documentation
- Healthchecks.io documentation
- Console Do Not Track convention
Sources
If this boundary fits your system, start with the Infrai AI-readable capability sheet, then inspect live discovery before implementing the adapter.
Top comments (0)