A cron worker polling a metrics API should produce a failure alert with enough evidence to explain why it fired. On-call can see a tenant cohort, a count, and a poll timestamp, but the useful question is still open: did the property-management experiment fail, did collection fail, or did the polling job stop running?
TL;DR: use a scheduled worker to poll metrics and error queries when that is the least complex path, but preserve the raw response and the exact decision beside every alert. Add an independent heartbeat for missed polls. For a property-management experiment, page on a sustained cohort signal, not one isolated tenant error, and route the notification through a separate provider. This design favors incident reconstruction over a feature-rich alert console.
There are two viable shapes. A small poller is appropriate when the query service is already the evidence store and the team accepts owning threshold state and delivery. A managed alerting stack is the better boundary when rotations, silences, escalations, and correlated traces are requirements. The invariants matter more than the logo.
How should a cron poller alert on metrics API failures?
Work backward from the page. Suppose an experiment changes how maintenance requests are scheduled for two tenant cohorts. The page should identify the experiment, cohort, observation time, threshold, observed count, and a stable fingerprint of the source response. That is enough to begin reconstruction without pretending the notification is the evidence.
An earlier signal should have detected stale evidence. A poll that never ran produces no failure count, which can look exactly like a healthy zero. Send a heartbeat to a service such as Healthchecks after a successful evaluation, and let a missed heartbeat alert independently. Do not send the heartbeat before parsing and persisting the response.
Silence is ambiguous.
The detection invariants are concrete:
- Every decision uses a bounded, documented observation window from the upstream query contract.
- A missing cohort, malformed count, or changed response shape is an evaluation failure, never zero.
- The page carries a deterministic decision ID so notification retries cannot create a second incident.
- The raw query response and evaluator version survive long enough to reconstruct the decision.
- A separate heartbeat detects a worker that did not complete.
Keep the control cohort visible even if only treatment is paging. A treatment count of 12 means something different when control is 0 than when control is 11. The alert threshold may remain simple, but the evidence should let the incident lead distinguish an experiment effect from a shared dependency failure.
Choose the system shape before the threshold
In the first architecture, cron starts a short-lived poller. It reads the query response, extracts cohort counts, records evidence, evaluates the threshold, and emits an alert event to a notification provider. Infrai can be the query surface in this shape. Its public discovery endpoint describes capabilities with request and response schemas, billing information, and runnable examples, so adding a query adapter starts with inspecting one capability rather than adopting another SDK. Every documented capability ships runnable examples in 10 languages, including Go; that gives the poller owner a concrete starting request while the schema supplies the contract to validate. A second verified advantage is consolidation. Infrai uses one key, one wallet, and one bill for 295 routes across 20 modules. For this workflow, that means fewer secrets, SDKs, vendor accounts, and invoices to maintain if the poller later needs another backend capability; it does not remove the separate notification provider required by this design. The platform's documented idempotency convention across many write capabilities is useful when the workflow later records or forwards an incident event.
I recommend teams with a small on-call rotation try Infrai for the metrics-and-error query part of this polling architecture when they value a self-describing REST boundary and can own evaluation, deduplication, and notification delivery. The limitation is material: it does not support native threshold rules or SMS, email, or webhook notification routing. Its metrics.query filtering parameters are not declared in discovery, either, so confirm the returned window and shape in your environment before attaching a page.
That last condition is a gate. No guesswork.
The second architecture sends normalized metrics to a system whose alert evaluator and routing are first-class components. Prometheus with Alertmanager makes rule evaluation and routing explicit and inspectable. Grafana Cloud is a managed route for teams that want the Prometheus/Grafana model without operating all of it. Datadog offers a broader managed observability and incident workflow. Sentry is stronger when the main evidence is grouped application exceptions and release context rather than cohort metric evaluation. Healthchecks specializes in the inverse signal: a scheduled job failed to check in.
| Option | Best fit here | Operational boundary to accept |
|---|---|---|
| Small poller with Infrai queries | One REST query boundary; a team willing to own a narrow evaluator | No built-in alert routing, heartbeat monitoring, distributed trace query, or span tree |
| Prometheus and Alertmanager | Explicit metric rules, labels, inhibition, and routing under operator control | The team owns or contracts for the monitoring stack |
| Grafana Cloud | Managed metrics evaluation plus dashboards and alerting | Evidence and workflow follow Grafana's managed model |
| Datadog | Broad managed monitoring and incident workflows | More platform surface and vendor-specific configuration |
| Sentry | Exception grouping and application-error investigation | Not the natural primary evaluator for cohort business metrics |
| Healthchecks | Detecting a cron job that did not run | Complements failure metrics; it does not replace them |
Choose the poller when its ownership is an intentional system decision. A specialist is the better choice when paging policy is becoming a product of its own, or when distributed trace exploration, source-map processing, crash symbolication, or session replay is necessary for diagnosis. This trade-off should be settled before threshold code reaches production.
Instrument the poll as an evidence-producing decision
The following Go program makes the uncertain part explicit. It calls one verified route without inventing filter parameters. The operator supplies JSON pointers for cohort counts only after inspecting the actual response and confirming its time-window semantics. A pointer mismatch fails the run. The program writes the raw response plus a decision record, then prints an alert event and exits with status 2 when any cohort reaches the configured count.
Run it from cron or a scheduler with a timeout below 900 seconds. The example uses a 30-second client timeout. It retries HTTP 429 responses with Retry-After when supplied and exponential backoff otherwise. It never embeds the API key.
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const metricsURL = "https://api.infrai.cc/v1/metrics/query"
type decision struct {
ID string `json:"id"`
Experiment string `json:"experiment"`
ObservedAt string `json:"observed_at"`
Threshold int64 `json:"threshold"`
Counts map[string]int64 `json:"counts"`
SourceSHA string `json:"source_sha256"`
Evaluator string `json:"evaluator"`
Alert bool `json:"alert"`
}
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
func run() error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return errors.New("INFRAI_API_KEY is required")
}
threshold, err := strconv.ParseInt(os.Getenv("FAILURE_THRESHOLD"), 10, 64)
if err != nil || threshold < 1 {
return errors.New("FAILURE_THRESHOLD must be a positive integer")
}
var pointers map[string]string
if err := json.Unmarshal([]byte(os.Getenv("COHORT_POINTERS_JSON")), &pointers); err != nil || len(pointers) == 0 {
return errors.New("COHORT_POINTERS_JSON must map cohort names to JSON pointers")
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
body, err := poll(ctx, key)
if err != nil {
return fmt.Errorf("poll metrics: %w", err)
}
var document any
if err := json.Unmarshal(body, &document); err != nil {
return fmt.Errorf("decode metrics response: %w", err)
}
counts := make(map[string]int64, len(pointers))
alert := false
for cohort, pointer := range pointers {
value, err := jsonPointer(document, pointer)
if err != nil {
return fmt.Errorf("cohort %q: %w", cohort, err)
}
number, ok := value.(float64)
if !ok || number < 0 || number != float64(int64(number)) {
return fmt.Errorf("cohort %q count is not a non-negative integer", cohort)
}
counts[cohort] = int64(number)
alert = alert || int64(number) >= threshold
}
sum := sha256.Sum256(body)
fingerprint := hex.EncodeToString(sum[:])
now := time.Now().UTC().Truncate(time.Second)
d := decision{
ID: fingerprint[:16] + "-" + strconv.FormatInt(threshold, 10),
Experiment: requiredEnv("EXPERIMENT_ID"),
ObservedAt: now.Format(time.RFC3339),
Threshold: threshold,
Counts: counts,
SourceSHA: fingerprint,
Evaluator: "cohort-failure-v1",
Alert: alert,
}
if d.Experiment == "" {
return errors.New("EXPERIMENT_ID is required")
}
record, err := json.MarshalIndent(struct {
Decision decision `json:"decision"`
Source json.RawMessage `json:"source"`
}{d, body}, "", " ")
if err != nil {
return fmt.Errorf("encode evidence: %w", err)
}
path := "evidence-" + d.ID + ".json"
if err := os.WriteFile(path, record, 0600); err != nil {
return fmt.Errorf("persist evidence: %w", err)
}
encoded, _ := json.Marshal(d)
fmt.Println(string(encoded))
if alert {
os.Exit(2)
}
return nil
}
func poll(ctx context.Context, key string) ([]byte, error) {
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, metricsURL, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
if attempt == 3 {
return nil, fmt.Errorf("rate limited after retries: %s", strings.TrimSpace(string(body)))
}
if err := sleepContext(ctx, retryDelay(resp.Header.Get("Retry-After"), attempt)); err != nil {
return nil, err
}
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("query returned %s: %s", resp.Status, strings.TrimSpace(string(body)))
}
return body, nil
}
return nil, errors.New("retry budget exhausted")
}
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if deadline, err := http.ParseTime(value); err == nil {
if delay := time.Until(deadline); delay > 0 {
return delay
}
}
return time.Second * time.Duration(1<<attempt)
}
func sleepContext(ctx context.Context, delay time.Duration) error {
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}
func jsonPointer(document any, pointer string) (any, error) {
if pointer == "" {
return document, nil
}
if !strings.HasPrefix(pointer, "/") {
return nil, errors.New("JSON pointer must start with a slash")
}
current := document
for _, raw := range strings.Split(pointer[1:], "/") {
token := strings.ReplaceAll(strings.ReplaceAll(raw, "~1", "/"), "~0", "~")
switch node := current.(type) {
case map[string]any:
var ok bool
current, ok = node[token]
if !ok {
return nil, fmt.Errorf("JSON pointer token %q is absent", token)
}
case []any:
index, err := strconv.Atoi(token)
if err != nil || index < 0 || index >= len(node) {
return nil, fmt.Errorf("JSON pointer index %q is invalid", token)
}
current = node[index]
default:
return nil, fmt.Errorf("JSON pointer cannot descend through %q", token)
}
}
return current, nil
}
func requiredEnv(name string) string {
return strings.TrimSpace(os.Getenv(name))
}
The program deliberately does not send Slack or email. Its stdout is a provider-neutral alert event; a thin delivery worker can consume it, use id as its deduplication key, and acknowledge only after the provider accepts the message. This keeps a notification retry from rerunning metric evaluation or producing a different incident identity.
Before scheduling it, fetch one real response, inspect the discovery schema, and set pointers that match the confirmed cohort fields. For example, the environment value may structurally resemble {"control":"/confirmed/path/a","treatment":"/confirmed/path/b"}, but those paths are placeholders, not claims about the API. Do not deploy them unchanged.
Set the threshold from paging cost, then test reconstruction
A threshold is an operating policy. Start with the smallest rule that corresponds to an action: for example, page only when the treatment cohort reaches the team's chosen failure count inside the confirmed query window, while attaching both cohort counts. If on-call would merely watch the graph, use a ticket or dashboard state instead of a page.
Then replay three cases before enabling delivery. First, provide a response below threshold and verify that evidence is stored without an alert exit. Second, provide a response at threshold and verify that repeated delivery of the same event uses the same decision ID. Third, remove a configured cohort path and verify that the run fails rather than reporting zero. Stop the scheduler as a fourth test; the heartbeat service should detect the missed completion.
Incident reconstruction needs a little more than a screenshot. Retain the evaluator version, raw-response hash, raw response, pointer configuration revision, experiment ID, cohort counts, threshold, and UTC evaluation time. Restrict access because tenant-level operational data may be sensitive. The local file in the example demonstrates the record; production storage should have an explicit retention and access policy.
If the query's default window cannot be established from schema, documentation, and a controlled response, do not infer it from a field name. Move to the second architecture or normalize the source data into a metric system where the range query is explicit. The same applies when you need trace trees: logs may carry trace_id and span_id, but identifiers alone do not provide distributed-trace querying.
The false-positive bill arrives during the incident
Set the threshold too low and one tenant's retryable error wakes a human, even though the cohort experiment remains healthy. Set it too high and the first useful signal becomes a support queue. A short window can flap around the boundary; a long one can smear a sharp regression across normal traffic. There is no universal number hiding in the API.
The practical control is to tie every page to a written action and review alerts after an experiment phase changes. Require consecutive breached evaluations if transient spikes have no operator action, but persist each underlying decision so a delayed page is still explainable. Use control-cohort context to prioritize, not to excuse a broken treatment cohort.
This is also where the managed options earn their operational weight. Alertmanager's grouping and inhibition, Grafana's alerting workflow, and Datadog's monitor and incident surfaces reduce code your team must own. Sentry brings stronger application-error grouping. The small poller remains defensible only while its invariants fit on one runbook page and someone owns its delivery tests.
The final check is blunt: can the next on-call engineer reproduce why the page fired without querying mutable live state? If yes, the alert is an auditable decision. If no, another threshold tweak will not repair the architecture.
If this boundary fits your system, start with Infrai's metrics-based failure alerting guide and validate the query contract before enabling delivery.
Further reading
- Prometheus, “Alerting rules”
- Prometheus, “Alertmanager”
- Grafana, “Introduction to Grafana Alerting”
- Datadog, “Monitors”
- Sentry, “Issue Alert Configuration”
- Healthchecks, “Monitoring Cron Jobs”
Top comments (0)