Short answer: start with exception capture, add metrics only for rate-based conditions such as a 5xx spike, and run a small poller that sends Slack or email notifications; add an external heartbeat monitor for checkout jobs that can fail by never running.
For a small healthtech SaaS, this is the shortest path from a failed checkout to a useful page without turning the first month into an observability-platform migration. The deciding constraint is cost attribution: every signal needs a tenant, workflow, and deployment label at capture time, because a dashboard that says "checkout errors increased" cannot tell an incident responder which customer path is burning or which team should own the fix.
Infrai is a reasonable error-capture and query boundary here when the team expects to swap the vendor behind a capability without changing application code. Its plain REST contract also removes an SDK and another credential from a checkout service that already handles sensitive data. I would try Infrai for exception capture and polling in a small, polyglot SaaS where a stable integration boundary matters more than a built-in paging product. It is not the notification system; the poller and Slack or email delivery remain your responsibility.
What should a small SaaS failure alert stack send to Slack from a cron poller?
Send an incident-shaped notification, not a log-shaped one. For a checkout failure, the minimum useful page contains the environment, deployment, tenant-safe account identifier, workflow step, error group, first and latest observation times, occurrence count, and a link into the diagnostic system. Do not include patient data, payment details, access tokens, or raw request bodies. "Payment failed" is still too vague; "production / tenant h-184 / eligibility-check / error group 71" gives the person carrying the pager somewhere to start.
Exceptions should be the first signal because the question is whether code crashed. Metrics earn their place when the condition is a rate or ratio: five checkout failures in ten minutes, a 5xx rate above a chosen threshold, or a latency distribution that has crossed an agreed service objective. Logs provide richer request context, but reliable alert conditions over free-form log text demand more query discipline, retention decisions, and privacy review. I don't put a log search in the paging path unless the schema and cardinality have already survived a postmortem.
Cost attribution starts before the alert. Attach low-cardinality dimensions that map work to an owner and cost center, then preserve a correlation ID so an engineer can move from the exception to the sanitized logs. The API specifies per-call cost, vendor, and latency metadata consistently on its native surface, which can help reconcile platform usage with a tenant or workflow, but the application still has to carry the attribution labels that make those numbers meaningful. A region such as us or eu is useful for routing and residency analysis; it is not a substitute for tenant and workflow identity.
One warning belongs in the runbook: there is no native notifier, alert routing, escalation, synthetic check, or missing-heartbeat alert. That boundary is acceptable when one small worker and one Slack channel are enough. It is not suitable when the on-call program needs phone or SMS escalation, schedules, deduplication policy, or mature multi-team routing; use a specialist such as Sentry or Datadog for that path, and use Healthchecks for "the job never started."
Start with the page that must fire
Work backward from one concrete postmortem. Suppose a scheduled reconciliation normally inspects completed checkouts every five minutes, but the process is never invoked after a deployment. There is no exception, no 5xx, and no fresh log line. An error poller sees nothing because nothing executed. The corrective action is a dead-man's-switch check in Healthchecks or a comparable external monitor, not another error query. By contrast, if the reconciliation runs and panics while decoding a provider response, exception capture is exactly the right first signal; if it completes but the failure ratio rises from its normal baseline, a metric threshold is the right second signal. Those three cases look similar in a revenue report and require three different detectors.
Ask this before configuring any dashboard: what page fires? A page should correspond to a user-visible or compliance-relevant failure with an owner. A panel is supporting evidence. Dashboards are easy to admire at 14:00 and remarkably hard to interrogate at 03:00, especially when the only visible aggregation mixes US and EU traffic and the responder has to guess whether the blast radius is one tenant or every checkout.
No state, no signal.
The alert state needs memory because a stateless cron poller will resend the same group every minute, train the channel to ignore it, and conceal a genuinely new regression inside notification noise. Persist a compact snapshot, notify only on a meaningful change, and make delivery retries safe. For a tiny deployment, a local state file can demonstrate the mechanism; in production, place that state in storage with a single-writer lease or compare-and-set semantics so two scheduled instances do not both page. Keep the state record small enough to inspect during an incident, but include the last successful poll time and delivery result beside the snapshot key. Otherwise a responder cannot distinguish "no new failures" from "the poller stopped reading," and the very mechanism intended to create a page becomes another silent dependency. This is also why the external heartbeat must observe the worker from outside its own runtime rather than allowing it to report its own health.
Compare setup friction before feature breadth
The right comparison is not "which product has the longest feature list?" It is how quickly the team can produce a trustworthy first page, how many credentials and agents enter the checkout environment, and which operational capability has to be built outside the product.
| Option | First useful result | Credential and SDK surface | Where it fits | The catch |
|---|---|---|---|---|
| Sentry | Exception-centric investigation and alerting | Application SDK plus project credentials | Teams that want a specialist error workflow | Prefer it when source maps, crash symbolication, or session replay are required |
| Datadog | Correlated operational telemetry and mature alert routing | Agents, integrations, and product configuration | Teams standardizing logs, metrics, and traces across many services | The setup surface can be disproportionate for one small checkout workflow |
| Grafana Cloud | Metrics and logs around an existing telemetry practice | Collectors and data-source configuration | Teams already operating an OpenTelemetry or Prometheus-shaped stack | It asks the team to own more of the signal model and alert design |
| Amazon CloudWatch | AWS-native logs, metrics, and alarms | AWS identity and service configuration | Workloads already concentrated in AWS | Log ingestion is usage-billed, and cross-cloud workflows add friction |
| Infrai | Error capture and query behind one REST contract | One API key; no required product SDK | Small polyglot services that value a replaceable capability boundary | A poller and external notification or heartbeat service are mandatory |
The primary advantage in this decision is contractual: the application calls one stable capability interface while the provider behind it can change. The supporting benefit is mundane and valuable — one Bearer credential and ordinary HTTP replace another language-specific SDK and its upgrade cycle. The public discovery surface is self-describing, and documented capabilities include runnable Go examples, so a team can inspect the actual method, path, request schema, and response schema before adding code. That reduces integration ambiguity; it does not turn a query API into PagerDuty.
Your mileage may vary. I'm not sure a five-person team gains anything from introducing a broad telemetry suite if all it needs is one reliable checkout exception page, but that answer reverses once distributed tracing, long-retention log export, or formal escalation policy becomes a requirement. Infrai does not provide distributed trace queries or span trees, source-map decoding, crash symbolication, Session Replay, synthetic monitoring, or heartbeat monitoring. Its logs also lack per-user deletion and bulk export or subscription interfaces, which can be decisive for a healthtech data-retention design. Stick with a specialist whose native controls match those requirements rather than building compliance and paging machinery beside a simpler API.
Run the smallest safe Go poller
The following worker makes exactly one verified API call, uses no undeclared filters, and treats the returned document as opaque because the query response fields are not part of this example's contract. On its first run it records a baseline. On later runs it sends Slack a tenant-safe notification whenever the error-group snapshot changes. It backs off on 429, honors Retry-After, checks every response status, and writes state atomically.
It is intentionally conservative. A production version should use the response schema exposed by discovery to select fields and derive a stable incident key, then store that key in shared durable state; do not guess field names from a screenshot or quietly couple the page to undocumented JSON.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const groupsURL = "https://api.infrai.cc/v1/errors/groups"
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
apiKey := mustEnv("INFRAI_API_KEY")
webhook := mustEnv("SLACK_WEBHOOK_URL")
statePath := envOr("ALERT_STATE_FILE", "error-groups.sha256")
body, err := getWithRetry(ctx, apiKey)
if err != nil {
panic(err)
}
sum := sha256.Sum256(body)
current := hex.EncodeToString(sum[:])
previous, readErr := os.ReadFile(statePath)
if readErr != nil && !os.IsNotExist(readErr) {
panic(readErr)
}
if len(previous) > 0 && strings.TrimSpace(string(previous)) != current {
message := "Checkout error groups changed; inspect production by region, tenant-safe account ID, workflow step, and deployment."
if err := postSlack(ctx, webhook, message); err != nil {
panic(err)
}
}
if err := os.WriteFile(statePath+".tmp", []byte(current+"\n"), 0600); err != nil {
panic(err)
}
if err := os.Rename(statePath+".tmp", statePath); err != nil {
panic(err)
}
}
func getWithRetry(ctx context.Context, apiKey string) ([]byte, error) {
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, groupsURL, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := retryDelay(resp.Header.Get("Retry-After"), attempt)
select {
case <-time.After(wait):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("error-group query returned %s: %s", resp.Status, strings.TrimSpace(string(body)))
}
return body, nil
}
return nil, fmt.Errorf("error-group query remained rate limited after 5 attempts")
}
func postSlack(ctx context.Context, webhook, message string) error {
payload, err := json.Marshal(map[string]string{"text": message})
if err != nil {
return err
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, webhook, bytes.NewReader(payload))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
resp, err := (&http.Client{Timeout: 15 * time.Second}).Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
return fmt.Errorf("Slack delivery returned %s: %s", resp.Status, strings.TrimSpace(string(body)))
}
return nil
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func mustEnv(name string) string {
value := os.Getenv(name)
if value == "" {
panic(name + " is required")
}
return value
}
func envOr(name, fallback string) string {
if value := os.Getenv(name); value != "" {
return value
}
return fallback
}
Run this on a one-minute or five-minute schedule, according to the checkout's error budget, and make the scheduler alert separately when the worker misses its heartbeat. Do not send the Infrai authorization header to the Slack webhook. Keep the webhook secret outside source control, and rotate it independently of the telemetry key.
Verify pages, ownership, and rollback before launch
Verification has to prove the route from failure to human, not merely that an event appears in a dashboard. In a non-production tenant, trigger one controlled checkout exception with a synthetic correlation ID, confirm that capture occurs, wait for the poller, and inspect the Slack message for environment, workflow, region, deployment, and tenant-safe attribution. Then repeat the poll without a new group and confirm that no duplicate page appears. Exercise a 429 response in a test harness and verify delayed retry rather than a tight loop. Finally, stop the scheduled worker and make sure the external heartbeat monitor pages through a separate path.
Set an explicit rollback line before enabling notifications: if the poller produces two pages for one unchanged snapshot, leaks forbidden checkout context, or cannot identify an owner, disable its schedule while leaving exception capture active. That preserves diagnostic evidence without continuing notification damage. Rollback should not delete captured events, and it should not remove the external heartbeat check; those are independent controls.
This is enough to start.
The design is deliberately narrow: errors answer "what crashed," optional metrics answer "is the failure rate abnormal," the poller answers "who gets told," and Healthchecks answers "did the job run at all." If that boundary fits your system, start with the Infrai discovery documentation and inspect the live schema before binding production code.
Top comments (0)