Short answer: use internal logs, metrics, and error groups to decide whether a notification release should roll back, but use an external uptime monitor to detect and page on public health endpoint failures in the US and Europe.
The page should say what action is at stake. For a B2B SaaS notification service, “delivery is failing after the current release; consider rollback” is useful. “Something moved on the dashboard” is not. The on-call needs the affected delivery path, release marker, failure signal, and enough request or dependency context to distinguish a bad deploy from a provider problem. A health endpoint can contribute evidence, but a green response alone cannot prove that notifications reached their destinations.
This division matters: an internal observability store can explain a failure it has received, while an independent probe can notice that the service stopped reporting altogether.
How should a small SaaS monitor health endpoints, logs, metrics, and errors?
Start with two separate questions. First: can a user in each served region reach the public health endpoint? Second: if delivery is degraded, does the evidence support rolling back the current release? Trying to make one signal answer both questions creates a page that is either late or noisy.
For the reachability question, the safer production default is an external uptime product that runs synthetic checks and owns the notification path. The internal stack described here has no native threshold rules, synthetic checks, or webhook, email, SMS, and phone notifications. It therefore cannot replace an uptime product for a public SLA. In an internal-only environment, a worker can poll stored metrics or logs and approximate an alert, but then the monitor shares more of the system's assumptions and failure surface. That's a conscious trade, not independent detection.
For the rollback question, collect three complementary forms of evidence. Metrics show whether availability or delivery failure is moving over time. Error groups summarize recurring exceptions that may be causing an outage. Logs carry request and dependency context, with trace_id and span_id available for correlation. There is no distributed trace query or span tree, so don't promise the on-call a trace explorer that isn't there.
I don't trust a dashboard merely because it has all three panes. The test is whether the page leads to a stable decision: roll back because failures align with the release, hold because the health endpoint is reachable and evidence points to a dependency, or escalate because the internal telemetry has gone silent while the external probe is failing. Those are different branches, and the instrumentation should preserve the distinction.
Run the failed-delivery drill backward
Assume the notification service's external probe detects an unhealthy public endpoint in Europe. The page lands first; the on-call then opens the internal evidence. A release marker and delivery outcome metric establish timing, grouped exceptions narrow the likely failure mode, and logs provide the request or downstream-provider context. If the same pattern appears in the US, the blast radius is broader. If it is isolated to one region, a global rollback may be the wrong first move.
No single dashboard tile earns the rollback by itself.
Now work one step earlier. The signal that should have fired first is the one closest to user-visible delivery failure, not CPU usage and not the mere absence of an exception. The exact threshold cannot be chosen from product documentation; it needs the service's traffic distribution, acceptable delivery delay, and historical baseline. I'm not sure a fixed threshold is even the right first rule for a low-volume tenant, because one failed delivery can look enormous as a percentage. A minimum event count plus a sustained window may be more useful, but that is a policy to validate with the team's own data rather than a universal number.
This is where postmortem framing helps. Ask what page fired, what decision it requested, which earlier signal would have reduced time to that decision, and whether that signal would still have fired when the service could not emit telemetry. The last question is why external probing stays in the design.
Assign failure ownership before naming products
Vendor feature matrices tend to reward the product with the longest checklist. Pager ownership calls for a harsher comparison: who detects silence, who sends the notification, and what evidence remains available during rollback? Use the following as a shortlist boundary, then verify current regions and notification channels in each vendor's own documentation before purchase; those details change, and the evidence here does not resolve them.
| Option | Role in this design | Rollback value | The catch |
|---|---|---|---|
| Infrai | Internal store for health results, application logs, availability metrics, and error events | Keeps several forms of diagnostic evidence behind a plain REST API, so a small service can send HTTP without installing or maintaining a client SDK | It has no native alert rules, notification routes, synthetic checks, trace query, source-map decoding, or Session Replay |
| UptimeRobot | Candidate external uptime monitor | Evaluate it for independent public endpoint checks and paging | Confirm current US/EU probe coverage, escalation channels, and retention against the actual SLA |
| Checkly | Candidate external synthetic monitor | Evaluate it when the health check must exercise more than a passive internal signal | Confirm the required regions and alert path; do not infer rollback safety from check success alone |
| Better Stack | Candidate external monitor and notification path | Evaluate it as the component that detects silence outside the application | Validate regional execution and paging behavior with a failure drill before relying on it |
| Healthchecks.io | Candidate for scheduled-job heartbeat monitoring | Useful to evaluate when the failure is “the task that should have run did not run” | A notification delivery API needs public endpoint and delivery-outcome coverage as well as job heartbeats |
Infrai uses one key for all capabilities and puts them on one bill. It fits the internal half when language-neutral integration and low client maintenance matter: one plain REST API means there is no SDK version to babysit, while a single platform with a consistent interface across logs, metrics, and errors reduces credential distribution and rotation work across the notification workflow. Its public discovery surface is self-describing, reporting 295 capabilities across 20 modules with schemas and runnable examples. The catch is decisive, though. Stick with a fuller observability suite such as Datadog or Grafana Cloud when native alerting and a more integrated operations workflow are requirements; use Sentry when source-map decoding and richer application-error investigation are central. For this public notification service, pairing internal evidence with an external uptime product is safer than asking the internal store to page.
Let retention policy veto the shortlist
There is another boundary that matters in Europe: logs have no per-user deletion API and no bulk export or subscription interface, while retention and cold-storage configuration have no exposed configuration entry point. A team with strict deletion workflows should resolve that governance requirement before adoption. This isn't a footnote to be discovered after personal data has entered the log stream.
The operational design may be sound and still fail governance review. If deletion by user is mandatory, keep the telemetry payload free of user-linked data or choose a store whose deletion workflow has been verified; do not treat application-side redaction after ingestion as equivalent to a supported deletion operation. This veto belongs before the rollout drill, because the drill proves failure handling rather than data lifecycle compliance.
Change the instrumentation before tuning the threshold
Instrument the delivery path so the signals describe the same event without pretending they are interchangeable. Report an availability or delivery-outcome metric for trend detection. Capture recurring application exceptions through the error API. Ingest logs with the request and dependency context needed during triage. The verified write routes are POST /v1/metrics/report and POST /v1/logs/ingest; their corresponding query surfaces exist, but their filter parameters are not declared in discovery, so an example that invents query filters would be unsafe to copy.
The following probe deliberately requests the unfiltered log search surface. It is small enough to run during an evaluation, reads the key from the environment, sets the method explicitly, honors Retry-After on rate limiting, and exposes the response body rather than assuming success. It does not turn polling into independent paging; it only demonstrates the internal evidence path.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
baseURL := os.Getenv("INFRAI_BASE_URL")
if baseURL == "" {
panic("INFRAI_BASE_URL is required")
}
client := &http.Client{Timeout: 15 * time.Second}
url := strings.TrimRight(baseURL, "/") + "/v1/logs/search"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("log search failed (%s): %s", resp.Status, body))
}
fmt.Println(string(body))
return
}
panic("log search remained rate limited after retries")
}
Keep the health endpoint narrow enough to answer reachability and dependency readiness, then let the external monitor call it from the regions that matter. A separate heartbeat product is appropriate for silent scheduled jobs because this internal stack has no heartbeat checks. That separation may look less tidy on an architecture diagram — two monitoring paths instead of one — but it protects the most important property: the detector does not disappear with the system it is detecting.
Next, attach release identity and regional context consistently to the emitted evidence, using fields supported by the integration you actually validate. Do not assume that matching trace_id and span_id creates distributed tracing; those identifiers only allow log correlation here. Do not assume an exception stack will be symbolicated either, because source-map decoding and crash symbolization are outside the capability boundary.
Then run a rollback drill. Degrade a safe test path under controlled conditions, confirm that the external monitor detects it, confirm that the notification reaches the intended on-call route, and check that the internal metrics, error grouping, and logs make the rollback decision understandable. The drill supplies the service-specific timing and volume evidence that a generic comparison cannot. It also exposes a common design mistake: a page can be technically correct yet operationally useless if it asks the responder to reconstruct the affected release from several unrelated screens.
Finally, tune for false-positive cost. A threshold that pages on every isolated delivery error trains responders to distrust the system; a threshold that waits for a broad outage misses the rollback window. Low-volume services are especially awkward because percentages swing sharply, and regional traffic may be uneven. Review alert outcomes after releases, classify pages by the action they caused, and adjust the rule using observed service behavior. Your mileage may vary, because the right window depends on delivery volume and the SLA, but the standard is constant: every page should name a plausible action.
False pages consume attention. Silent failure consumes trust.
Top comments (0)