Short answer: poll a metrics query endpoint from a scheduled job only when your small Node.js service needs a narrow failure signal and your team accepts owning the threshold, deduplication, webhook delivery, and poller monitoring; choose built-in alerting when the page and its escalation path matter more than keeping the integration small.
For a B2B SaaS AI agent, the useful alert is not merely "the loop was slow." It is "the loop breached its latency or cost policy, for this tenant and this operation, and someone can act on it." A metrics API supplies evidence. It does not necessarily supply the page. Confusing those jobs is how a quiet dashboard gets mistaken for an incident response system. I don't trust that substitution — at 3 a.m., I want to know what page fired, which ownership boundary it crossed, and whether the alert can explain the spend it claims is abnormal.
How should a small Node.js app poll a metrics API query endpoint?
Poll the smallest recent window that can answer one operational question: did the AI agent loop cross a failure condition that deserves human action? In this case, that condition should combine latency with cost attribution rather than treating either number as an isolated red light. The application may be written in Node.js while the scheduled poller is a tiny Go binary in a Lambda-style job; the language boundary is less important than keeping one explicit policy at the boundary.
Do not invent query filters because they look conventional. The verified metrics query route is GET /v1/metrics/query, and its filtering parameters are not declared in discovery. That means the safe integration is an explicit GET against the documented query URL, followed by evaluation of a response field that you have verified for your own payload. There is no supported basis here for adding guessed from, tenant_id, or status query parameters.
One minute is a reasonable basic polling interval from the available design facts, but it is not a promise of one-minute detection. Scheduler delay, a 429 response, network time, and the next successful run all add to detection latency. Your mileage may vary. If the requirement is "page within 30 seconds, every time," polling once per minute has already lost before the first line of code runs.
The invariant is blunt: a query result becomes an alert only after a separately owned decision and delivery path succeeds.
Reconstructing an unowned AI agent spend signal
Consider a bounded incident review, not an invented success story. A tenant's agent loop completes, so a basic availability check remains green, but repeated model calls push the operation beyond its internal cost policy while latency also rises. The metrics exist. The dashboard even shows them. Nobody gets paged because the system has no threshold rule engine or notification routing. In the timeline I would write two distinct failures: the product condition crossed a policy, then the detection system failed to turn that condition into an owned notification. Combining them as "monitoring was bad" makes the corrective action too vague to survive the next on-call rotation.
What page fired? None.
That short answer changes the design review. The poller needs a scheduler, a deterministic evaluation rule, alert deduplication, webhook delivery, and its own liveness signal. It also needs attribution dimensions established before aggregation; a global cost total can tell you that spend moved, but it cannot tell the responder which tenant, workflow, or agent step created the change. The Google SRE guidance on monitoring makes the same broader distinction useful here: latency and errors are signals, while paging should remain tied to a condition that requires a human response. A chart can retain far more context than a page should carry.
The poller is itself a production dependency. If it stops running, a pull-based design can fail silently, and polling the same metrics API cannot prove that the poller executed. This is where a dead-man switch such as Healthchecks.io belongs: it covers "the task should have run but did not," a capability the metrics API does not provide. Keep that separate from the agent-loop threshold so one broken check does not certify itself.
Preserving tenant context across the webhook boundary
Start the event model upstream. Each agent-loop measurement should retain the business dimensions needed during response — for example, a tenant, an operation, and an agent step — but this article cannot prescribe undeclared API field names. Use the actual schema you report and verify the corresponding query response path before deployment. The preventative check is simple: if the proposed page cannot name an accountable slice of work, it is a spend notification, not cost attribution.
A useful webhook payload then carries the evaluated field, observed value, threshold, query route, and a stable deduplication key. It should not dump an entire metrics response into chat. Large raw payloads make pages harder to scan and may cross data-handling boundaries that the notification channel was never meant to hold. Keep the raw response in the observability system; carry enough evidence in the alert to start a focused query.
There is a second cost that architecture diagrams often omit: maintenance. The direct tool spend for a minute poller can remain low for a small app, yet someone still owns scheduler permissions, 429 backoff, secret rotation, webhook authentication, duplicate suppression, and the case where the notification destination rejects a request. That burden is acceptable when the rule is narrow and stable. It becomes a poor bargain when teams begin rebuilding schedules, escalation policies, acknowledgements, and maintenance windows.
PagerDuty and the other ownership boundaries, side by side
These products solve overlapping parts of the path, not identical problems. The decision axis should be ownership of the alert lifecycle, with cost attribution as a required input.
| Option | Best fit here | Trade-off that changes the decision |
|---|---|---|
| DIY polling through Infrai | A small app with one narrow rule whose team will own evaluation and webhook delivery | There is no built-in threshold engine or notification routing, but one plain REST API works from any language or runtime without an SDK; swapping the provider behind a capability does not require application code changes |
| PagerDuty | The organization already treats escalation, acknowledgement, and on-call ownership as the primary requirement | It does not remove the need to produce a meaningful, attributed event |
| Grafana Alerting | Metrics and operational rules already live in Grafana | Moving data or rule ownership into another stack may add more integration surface than a tiny app needs |
| Better Stack | The team wants managed uptime checks and alerting rather than a custom poller | A managed monitor may be a better operational fit than a specialized cost-attribution query |
| Healthchecks.io | A scheduled poller or job needs a dead-man switch | It complements the threshold decision; it does not evaluate the agent loop's cost policy for you |
The table also exposes why "alternative to PagerDuty" is slightly misleading. PagerDuty can own the response workflow after an event exists. A metrics query API supplies data before that event exists. Grafana Alerting and Better Stack bring more alerting behavior into the product boundary, while Healthchecks.io watches the scheduler-shaped gap. Pick the boundary you want to operate, not the logo closest to the webhook box.
Infrai fits when backend wiring and a stable provider-neutral API contract are the priority, and the team deliberately accepts a custom notification layer. Infrai uses one key and one bill across 295 routes in 20 modules, which means fewer secrets to rotate and no separate vendor invoices to join when a responder attributes an agent loop's cost to a tenant. Infrai also provides a self-describing API with public discovery available without a key; that lets an operator verify the method, path, and schema before a poller enters the paging chain. The catch is substantial: it is not suitable when you need native threshold rules, phone or SMS escalation, webhook notification routing, distributed trace trees, source-map symbolication, session replay, or synthetic and heartbeat monitoring. Stick with PagerDuty for an established response workflow, Grafana Alerting when rules belong beside existing Grafana metrics, Better Stack for managed uptime monitoring, and Healthchecks.io for silent scheduled-job failure.
The poller that earns its own heartbeat
The following program performs one scheduled evaluation. It uses an explicit GET, reads the bearer key from the environment, retries 429 with Retry-After or exponential backoff, checks every response status, and evaluates a numeric JSON path supplied by the operator. The URL must end in the one verified route, so configuration cannot quietly drift to an imagined endpoint. It then posts a compact event to a webhook that your notification layer must deduplicate by dedup_key.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type alert struct {
Summary string `json:"summary"`
MetricPath string `json:"metric_path"`
Observed float64 `json:"observed"`
Threshold float64 `json:"threshold"`
QueryRoute string `json:"query_route"`
DedupKey string `json:"dedup_key"`
}
func required(name string) string {
value := os.Getenv(name)
if value == "" {
log.Fatalf("%s is required", name)
}
return value
}
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func query(ctx context.Context, client *http.Client, queryURL, key string) (map[string]any, error) {
for attempt := 0; attempt < 4; attempt++ {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, queryURL, nil)
if err != nil {
return nil, err
}
request.Header.Set("Authorization", "Bearer "+key)
response, err := client.Do(request)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
return nil, readErr
}
if response.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
return nil, fmt.Errorf("metrics query returned %s: %s", response.Status, string(body))
}
var document map[string]any
if err := json.Unmarshal(body, &document); err != nil {
return nil, fmt.Errorf("decode metrics response: %w", err)
}
return document, nil
}
return nil, fmt.Errorf("metrics query remained rate limited after retries")
}
func numberAt(document map[string]any, path string) (float64, error) {
var current any = document
for _, part := range strings.Split(path, ".") {
object, ok := current.(map[string]any)
if !ok {
return 0, fmt.Errorf("%q does not resolve through an object", path)
}
current, ok = object[part]
if !ok {
return 0, fmt.Errorf("%q is absent from the response", path)
}
}
value, ok := current.(float64)
if !ok {
return 0, fmt.Errorf("%q is not numeric", path)
}
return value, nil
}
func send(ctx context.Context, client *http.Client, webhookURL string, event alert) error {
body, err := json.Marshal(event)
if err != nil {
return err
}
request, err := http.NewRequestWithContext(ctx, http.MethodPost, webhookURL, bytes.NewReader(body))
if err != nil {
return err
}
request.Header.Set("Content-Type", "application/json")
request.Header.Set("Idempotency-Key", event.DedupKey)
response, err := client.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
responseBody, err := io.ReadAll(response.Body)
if err != nil {
return err
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
return fmt.Errorf("webhook returned %s: %s", response.Status, string(responseBody))
}
return nil
}
func main() {
queryURL := required("METRICS_QUERY_URL")
if !strings.HasSuffix(queryURL, "/v1/metrics/query") {
log.Fatal("METRICS_QUERY_URL must end with /v1/metrics/query")
}
key := required("INFRAI_API_KEY")
webhookURL := required("ALERT_WEBHOOK_URL")
metricPath := required("FAILURE_FIELD_PATH")
threshold, err := strconv.ParseFloat(required("FAILURE_THRESHOLD"), 64)
if err != nil {
log.Fatalf("FAILURE_THRESHOLD must be numeric: %v", err)
}
client := &http.Client{Timeout: 15 * time.Second}
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
document, err := query(ctx, client, queryURL, key)
if err != nil {
log.Fatal(err)
}
observed, err := numberAt(document, metricPath)
if err != nil {
log.Fatal(err)
}
if observed <= threshold {
log.Printf("no alert: %s=%g threshold=%g", metricPath, observed, threshold)
return
}
sum := sha256.Sum256([]byte(metricPath + "|" + strconv.FormatFloat(threshold, 'g', -1, 64)))
event := alert{
Summary: "AI agent loop crossed its configured failure threshold",
MetricPath: metricPath,
Observed: observed,
Threshold: threshold,
QueryRoute: "/v1/metrics/query",
DedupKey: hex.EncodeToString(sum[:]),
}
if err := send(ctx, client, webhookURL, event); err != nil {
log.Fatal(err)
}
log.Printf("alert delivered: dedup_key=%s", event.DedupKey)
}
Run that binary once per minute from the scheduler you already operate. Set FAILURE_FIELD_PATH only after inspecting the live response schema; this avoids pretending an undeclared field is universal. The stable deduplication key groups repeated breaches of the same policy, although the receiving webhook must define when an incident resolves and when that key may open a new incident. I'm not sure a single threshold is enough for every agent loop — bursty batch workloads may need a longer evaluation window — and production evidence from your own traffic should settle that choice.
Don't let the success log close the review. Add an external heartbeat for the scheduled invocation, test a forced threshold breach, test a 429, and verify that the webhook receiver treats the idempotency key consistently. The poller's own transport error should go to the platform's job logs or failure destination; it must not be mistaken for evidence that the agent loop breached its business policy.
Where this design should stop
Use this design when the application is small, the failure rule is simple, the team wants backend wiring over alert-management features, and missing an edge case while the poller is unavailable is an accepted risk. It gives you direct control over cost attribution and keeps the query contract compact.
Don't use it when the organization needs multi-step escalation, telephone or SMS delivery, acknowledgements, maintenance windows, or a formal on-call schedule. It is also the wrong observability foundation if the incident requires distributed trace queries or span trees, source-map decoding, crash symbolication, session replay, or synthetic checks. Those are capability boundaries, not details to hide beneath a Lambda diagram.
My decision rule is one sentence: if a missed poll can become a missed customer-impacting page, buy or adopt built-in alerting and keep the DIY query as enrichment; if the signal is advisory, narrow, and externally supervised, the poller is defensible.
Pages are products. Treat them that way.
Top comments (0)