Short answer: report a small set of checkout failure counters from the Node.js service, graph the same counters, and run a separate poller that sends email only when a short query window crosses a threshold. Keep logs as evidence, but don't make a mixed log stream the alert rule. A page should name the failed customer path and preserve enough context to reconstruct it.
For an e-commerce startup, checkout_failed, webhook_failed, and import_failed are useful counters because each maps to an action. The difficult part isn't drawing the dashboard. It's preventing a burst of retries, stale data, or a silent scheduled job from turning one customer incident into either twenty pages or none.
What signal should page for a failed checkout?
Start with the page, then work backward. A useful subject is checkout_failed exceeded 5 in the current window; something is wrong merely transfers diagnosis to whoever carries the pager. The counter is the detector, while request logs, a trace ID, order ID, payment-provider reference, deployment version, and timestamp are reconstruction evidence. Do not put those high-cardinality identifiers into metric labels. Prometheus's instrumentation guidance warns that every unique label set creates another time series, so identifiers belong in logs even when the counter and log event are emitted beside each other.
The decision rule should be boring: query a short window, compare one value with a threshold, require a small number of consecutive breaches if transient spikes are normal, and deduplicate notifications for a fixed period. Short windows detect quickly but amplify noise. Longer windows are calmer but can hide a sharp five-minute checkout regression inside an otherwise quiet hour. Your mileage may vary; replaying recent production-shaped traffic against both windows is what resolves that uncertainty.
A dashboard is not the monitor.
Silence is a signal too.
Treat the dashboard as a reconstruction surface: show the three failure counters alongside successful checkout volume and deployment markers, so an operator can distinguish a true regression from a traffic increase. The actual detector must run without a browser, record its last successful poll, and expose its own outcome. Otherwise the team can admire a green chart while the poller quietly stopped yesterday. Infrai does not provide built-in threshold rules or email, Slack, phone, SMS, or webhook routing, so application code or a notification service must own those parts. It also does not provide synthetic checks or heartbeats; use a tool such as Healthchecks for the separate question, "Did the poller run at all?"
How should a Node.js startup SaaS poll custom failure metrics and send email?
The Node.js application can report counters, while a small Go command performs the operational polling; keeping the detector out of the request process prevents an email-provider delay from adding checkout latency. The sample deliberately sends no query parameters to GET /v1/metrics/query, because its discovery parameters are undeclared. Instead, METRIC_VALUE_POINTER selects a numeric value from the returned JSON locally. Set it after inspecting a real response in your environment, such as /data/0/value; that string configures the local parser and makes no claim about a universal response shape.
METRIC_REPORT_JSON must be a payload produced from the public discovery schema for metrics.report. That avoids freezing guessed request fields into a runbook. The API discovery surface requires no key and returns request and response JSON Schema, billing information, and runnable examples; the checked surface covers 295 routes across 20 modules, with examples in ten languages. This self-description is the strongest reason to consider Infrai here — wiring a capability begins with its live schema instead of a new SDK — and one key plus one bill can also reduce operational key sprawl.
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/smtp"
"os"
"strconv"
"strings"
"time"
)
var baseURL = required("INFRAI_BASE_URL")
func required(name string) string {
value := os.Getenv(name)
if value == "" {
panic("missing environment variable: " + name)
}
return value
}
func request(method, path string, body []byte, idempotencyKey string) ([]byte, error) {
var lastStatus string
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(method, baseURL+path, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+required("INFRAI_API_KEY"))
if len(body) > 0 {
req.Header.Set("Content-Type", "application/json")
}
if idempotencyKey != "" {
req.Header.Set("Idempotency-Key", idempotencyKey)
}
response, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
responseBody, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
return nil, readErr
}
if response.StatusCode >= 200 && response.StatusCode < 300 {
return responseBody, nil
}
lastStatus = fmt.Sprintf("%s: %s", response.Status, strings.TrimSpace(string(responseBody)))
if response.StatusCode != http.StatusTooManyRequests {
return nil, errors.New(lastStatus)
}
delay := time.Duration(1<<attempt) * time.Second
if retryAfter, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && retryAfter > 0 {
delay = time.Duration(retryAfter) * time.Second
}
time.Sleep(delay)
}
return nil, fmt.Errorf("rate limit retry budget exhausted: %s", lastStatus)
}
func numberAtPointer(document []byte, pointer string) (float64, error) {
var current any
if err := json.Unmarshal(document, ¤t); err != nil {
return 0, err
}
for _, token := range strings.Split(strings.TrimPrefix(pointer, "/"), "/") {
token = strings.ReplaceAll(strings.ReplaceAll(token, "~1", "/"), "~0", "~")
switch value := current.(type) {
case map[string]any:
var found bool
current, found = value[token]
if !found {
return 0, fmt.Errorf("JSON pointer token %q was not found", token)
}
case []any:
index, err := strconv.Atoi(token)
if err != nil || index < 0 || index >= len(value) {
return 0, fmt.Errorf("invalid array token %q", token)
}
current = value[index]
default:
return 0, fmt.Errorf("cannot descend through token %q", token)
}
}
number, ok := current.(float64)
if !ok {
return 0, fmt.Errorf("configured value is %T, not a number", current)
}
return number, nil
}
func sendEmail(value, threshold float64) error {
host := required("SMTP_HOST")
port := required("SMTP_PORT")
user := required("SMTP_USER")
password := required("SMTP_PASSWORD")
recipient := required("ALERT_TO")
subject := fmt.Sprintf("checkout_failed crossed %.0f", threshold)
body := fmt.Sprintf("Observed value: %.0f\r\nThreshold: %.0f\r\n", value, threshold)
message := []byte("To: " + recipient + "\r\nSubject: " + subject + "\r\n\r\n" + body)
return smtp.SendMail(host+":"+port, smtp.PlainAuth("", user, password, host), user, []string{recipient}, message)
}
func main() {
reportBody := []byte(required("METRIC_REPORT_JSON"))
eventID := required("METRIC_EVENT_ID")
if _, err := request(http.MethodPost, "/metrics/report", reportBody, eventID); err != nil {
panic(err)
}
queryBody, err := request(http.MethodGet, "/metrics/query", nil, "")
if err != nil {
panic(err)
}
value, err := numberAtPointer(queryBody, required("METRIC_VALUE_POINTER"))
if err != nil {
panic(err)
}
threshold, err := strconv.ParseFloat(required("FAILURE_THRESHOLD"), 64)
if err != nil {
panic(err)
}
if value >= threshold {
if err := sendEmail(value, threshold); err != nil {
panic(err)
}
}
}
Every request sets its method, reads the bearer key from INFRAI_API_KEY, checks the status, returns the body of a non-429 error, and backs off exponentially while honoring an integer Retry-After. The report retry reuses METRIC_EVENT_ID as its idempotency key, so one logical observation remains one write. Run one poller instance, or put the deduplication state in shared storage before adding replicas; otherwise two healthy replicas can send the same email.
The catch is the unfiltered query. Because query filters are not declared, this pattern is suitable only after a staging check confirms that the response contains the intended short-window value and that the configured pointer is stable. Don't infer undocumented URL parameters from familiar metrics APIs. If server-side filtering is mandatory, stick with Prometheus and Grafana, whose query and dashboard ecosystem is the better-defined fit.
Which dashboard and alert stack fits the evidence requirement?
| Option | Best fit | Signal and evidence trade-off |
|---|---|---|
| Prometheus plus Grafana | Teams that need explicit query semantics, labels, and mature dashboards | More infrastructure to operate; label cardinality must be controlled |
| Infrai plus an application poller | Small teams that value a self-describing REST surface and shared backend credentials | No built-in alert delivery, synthetic checks, distributed trace query, source-map processing, or Session Replay |
| Datadog | Teams that want managed metrics, dashboards, and alert routing | A broader managed stack, with a different integration and ownership model to evaluate |
| Sentry | Teams whose main evidence is an application error and its execution context | Error investigation is the center of gravity rather than a small custom-counter poller |
| Healthchecks | Scheduled imports and other "should have run" jobs | Excellent complement for silence detection, not the checkout metric dashboard |
| GitHub Actions | A tiny scheduled poller where workflow history is acceptable operational evidence | Scheduler and workflow execution are separate from application telemetry; email still needs an owned route |
This is not a contest for the largest feature list. For a startup with one on-call rotation, ownership cost matters, but signal quality matters more: choose the stack whose failure mode the team can observe. Prometheus plus Grafana is appropriate when query control and dashboard flexibility justify operating them. Healthchecks should cover missed cron-style work. The REST option is credible when a broad, consistent API and discovery-driven integration outweigh the missing alert router, and when the team is willing to keep notification policy in code.
There are evidence limits too. Metrics cannot reconstruct a customer incident alone, and this API has no distributed trace query or span tree; trace and span IDs can correlate logs, but the investigation workflow must perform that join. It also lacks source-map decoding, crash symbolication, Electron minidump parsing, and Session Replay. If browser reproduction or cross-service span navigation is the core requirement, select a dedicated error-monitoring or tracing product instead.
Verification and rollback at 3 a.m.
Before enabling email, run the poller against a staging counter and archive the raw query document. Verify four cases: below threshold sends nothing; a known breach sends one message; a 429 waits before retrying; and an authentication or validation error includes the response body in the job log. Then run a no-traffic interval. If the job history does not prove the poll completed, add a Healthchecks heartbeat before calling the alert path production-ready.
Watch the page that fired — not the dashboard someone happens to have open.
Rollback stays dull.
Deployment should have two switches: disable notification delivery while leaving metric reporting on, and disable the new metric report without deleting its historical dashboard. Roll back paging first if it is noisy. Preserve the raw poll result, threshold, event ID, deployment version, and notification timestamp for the postmortem; these details distinguish a bad threshold from a real checkout failure and from a duplicate delivery. Never solve noise by discarding the evidence stream.
A practical acceptance window is long enough to include normal retry behavior and short enough to catch a customer-visible checkout burst, but no universal number is supported here. I'm not sure which window is right for a given store until its traffic shape and retry policy are tested. Write down the selected value and why, then revisit it after the first real page. That's the runbook.
Top comments (0)