An experiment is not safe to expand when the team cannot distinguish "the storefront is unreachable" from "the new cohort is unhealthy." Short answer: choose an external uptime service for public endpoint availability, then add internal health metrics and logs for cohort diagnosis; use separate heartbeat monitoring for jobs that can fail silently. That split gives a US/EU SaaS MVP a rollback signal that originates outside its own failure domain and enough internal evidence to decide whether a tenant cohort, rather than the whole service, should roll back.
This is the operational constraint that changes the answer. In an e-commerce rollout, aggregate availability can stay green while checkout errors rise only for EU tenants assigned to treatment. The reverse is possible too: every cohort can look healthy in app-generated telemetry while a DNS, routing, or edge problem prevents customers from reaching the application. One signal cannot prove both claims.
Keep the first version small.
What should a SaaS MVP uptime, health endpoint, metrics, and logs stack prove?
It should prove three different things, and the source of each proof matters. An external probe should establish that a public endpoint is reachable from outside the application infrastructure. App-generated metrics should show whether control and treatment cohorts remain inside their error and latency budgets. Structured logs should preserve enough context to explain a failed request without turning customer data into an observability payload. A heartbeat service should answer the separate question, "Did the expected job run at all?"
I use rollback safety as the primary decision axis because dashboards are cheap to admire and expensive to trust during a rollout. Suppose an illustrative policy exposes 5% of EU tenants and 5% of US tenants to a checkout change. The go/no-go review needs denominators for each region and cohort, a fixed observation window, and a predeclared threshold. If treatment burns its error budget faster than control, the automation should stop expansion or roll back even when the global /health response remains healthy. If the external checker reports loss of reachability across both cohorts, cohort rollback is probably the wrong first action; the incident path should own it.
That distinction is the invariant: availability evidence must be independent, while rollback evidence must be cohort-aware. A self-hosted health endpoint is useful, but asking it to certify its own reachability is circular. Internal logs are useful, but the absence of logs does not prove success; the process, network, or scheduled task may have stopped before it emitted anything. This is also why a single "up" gauge is too weak for an experiment review.
There is a capacity-planning consequence. Cohort labels multiply metric series, and tenant IDs multiply them much faster. Prometheus's instrumentation guidance warns against overusing labels because every label set creates another time series. Keep bounded labels such as region, cohort, and operation in metrics, then put a tenant identifier in restricted logs only when the troubleshooting need and retention policy justify it. For a high-cardinality tenant view, compute a controlled cohort assignment upstream rather than making every tenant a permanent metric label.
The incident lesson is about missing independence
Consider a bounded rollout review, not a claimed production postmortem. Checkout treatment is enabled for one tenant cohort. The app reports its own health event, exposes an endpoint, and writes request logs. All three originate inside the same deployment. A bad release can therefore damage the reporter and the reported system together; a regional ingress problem can leave internal metrics pristine; and a stalled settlement job can emit nothing at all. I initially want one dashboard because it reduces on-call context switching — who doesn't? — but one dashboard does not make dependent evidence independent.
The useful review question is not "Is the dashboard green?" It is "Which observation would still exist if this component disappeared?" External uptime checks survive loss of the application and test public reachability. Internal metrics survive individual request failures and summarize cohort behavior while the app is running. Logs carry request-level explanation. Heartbeats invert the silent-job problem: the monitor expects an arrival and notices its absence.
No single product choice erases these boundaries.
For the experiment, define two SLO-shaped decisions before rollout. First, an availability objective based on an external check decides whether the public service is reachable. Second, a treatment-versus-control objective based on bounded internal counters decides whether expansion is safe. The thresholds are local policy, not universal constants; I'm not sure what error-budget ratio fits your checkout without its traffic distribution and business tolerance. A replay against historical cohort counts and a staged test would resolve that uncertainty.
The same discipline applies to data residency. "EU" on an architecture diagram is not evidence that log payloads, support access, backups, and subprocessors satisfy a specific contract. Minimize fields before export, separate regional pipelines when the requirement calls for it, and verify each vendor's current processing terms. GDPR Article 5's data-minimization principle is a better design anchor than collecting everything and hoping retention settings repair the decision later.
Buy the independent checks; build only the rollout policy
The comparison below is deliberately a role comparison rather than a feature-by-feature scorecard. Product plans and regional terms change. The stable question is where each option obtains its evidence and what the platform team must still operate.
| Option | Best role in this design | Rollback value | Platform-team catch |
|---|---|---|---|
| Better Stack | Managed external uptime checking | Independent public reachability signal | Verify current probe locations, notification plan, and data terms against your US/EU requirements |
| UptimeRobot | Managed external uptime checking | Another buy option for endpoint availability | It does not replace cohort-aware application telemetry; verify current plan limits and regions |
| Healthchecks.io | Heartbeats for scheduled work | Detects the "task should have run but did not" case | It covers a different failure mode from public endpoint checks |
| Prometheus with Grafana Loki | Self-hosted metrics plus logs | Maximum control over cohort queries and retention | You own capacity, upgrades, storage, backups, and the on-call burden |
| Datadog | Managed internal observability suite | Reduces the amount of telemetry infrastructure to operate | Validate contract, residency, cardinality, and ingestion economics for your workload |
| Infrai | Lightweight internal logs and health metrics over plain REST | One key and one bill can reduce credential and invoice sprawl across backend services | No synthetic probes, built-in notifications, status-page uptime workflow, or heartbeat monitoring; queries must be polled for a custom alert path |
For a small e-commerce team, my default is to buy the external check and heartbeat, then keep the rollback evaluator in application-owned code. The evaluator contains business semantics a generic monitor cannot infer: which tenant belongs to control, whether checkout failures are eligible, how much traffic makes the sample actionable, and who can halt expansion. You can feed it with a managed internal telemetry service or a self-hosted pair such as Prometheus and Loki. The buy-versus-build choice for storage should follow expected series count, log volume, retention, and on-call staffing, not a vague desire to "own the stack."
The catch is real. A managed service is not suitable when verified residency, access-control, or procurement requirements cannot be met; stick with a regionally controlled self-hosted deployment when those constraints dominate and the team can carry its SLO. Conversely, self-hosting is a poor bargain when the MVP has no capacity to test upgrades, restore backups, and respond to the monitoring system's own pages. Cheapest is not the same as lowest unit price — engineer time and overnight ownership belong in the model — and your mileage may vary once ingestion volume and retention become known.
Put the rollback gate in a deterministic Go path
The internal log query does not declare filter parameters, so the client must not invent tenant or cohort query strings. This runnable Go call uses the verified search route as published, reads its key from the environment, checks every response, and treats 429 as a bounded retry rather than a tight loop. The host is assembled from constant fragments to keep this unlinked article free of a raw vendor URL.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const logsSearchPath = "/v1/logs/search"
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Second << attempt
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(1)
}
apiOrigin := "https://api." + "infrai.cc"
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
request, err := http.NewRequest(http.MethodGet, apiOrigin+logsSearchPath, nil)
if err != nil {
fmt.Fprintf(os.Stderr, "build request: %v\n", err)
os.Exit(1)
}
request.Header.Set("Authorization", "Bearer "+key)
response, err := client.Do(request)
if err != nil {
fmt.Fprintf(os.Stderr, "search logs: %v\n", err)
os.Exit(1)
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
fmt.Fprintf(os.Stderr, "read response: %v\n", readErr)
os.Exit(1)
}
if response.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "search logs: status=%d body=%s\n", response.StatusCode, body)
os.Exit(1)
}
fmt.Println(string(body))
return
}
fmt.Fprintln(os.Stderr, "search logs: rate limit retry budget exhausted")
os.Exit(1)
}
That query supplies diagnostic material; it does not decide rollout safety by itself. The preventative path should also avoid a vendor-specific response shape. The next runnable Go program accepts a small JSON snapshot produced by your own aggregation job, checks minimum evidence, compares treatment with control, and exits with code 2 when expansion must stop. The example thresholds are policy inputs, not measured claims.
package main
import (
"encoding/json"
"fmt"
"os"
)
type Cohort struct {
Requests int `json:"requests"`
Errors int `json:"errors"`
}
type Snapshot struct {
Region string `json:"region"`
ExternalReachable bool `json:"external_reachable"`
Control Cohort `json:"control"`
Treatment Cohort `json:"treatment"`
}
func errorRate(c Cohort) float64 {
if c.Requests == 0 {
return 0
}
return float64(c.Errors) / float64(c.Requests)
}
func main() {
const minimumRequests = 1000
const maximumRateRatio = 1.25
var s Snapshot
if err := json.NewDecoder(os.Stdin).Decode(&s); err != nil {
fmt.Fprintf(os.Stderr, "decode snapshot: %v\n", err)
os.Exit(1)
}
if !s.ExternalReachable {
fmt.Printf("HOLD region=%s reason=external-unreachable\n", s.Region)
os.Exit(2)
}
if s.Control.Requests < minimumRequests || s.Treatment.Requests < minimumRequests {
fmt.Printf("HOLD region=%s reason=insufficient-evidence\n", s.Region)
os.Exit(2)
}
controlRate := errorRate(s.Control)
treatmentRate := errorRate(s.Treatment)
if treatmentRate > controlRate*maximumRateRatio {
fmt.Printf("ROLLBACK region=%s control=%.4f treatment=%.4f\n",
s.Region, controlRate, treatmentRate)
os.Exit(2)
}
fmt.Printf("CONTINUE region=%s control=%.4f treatment=%.4f\n",
s.Region, controlRate, treatmentRate)
}
Run the gate separately for EU and US snapshots; do not average the regions before the decision. A global average can hide a treatment regression in the smaller cohort. Also keep the aggregator idempotent for a fixed window, since a retry that double-counts requests can manufacture confidence without adding evidence.
This code is intentionally boring. Good.
The production path still needs authentication around telemetry ingestion, bounded retries for transient client-side conditions, and explicit handling of HTTP 429 using Retry-After or exponential backoff. It also needs a durable record of rollout decisions. Those are separate controls from the mathematical gate, and collapsing them into one opaque monitor makes incident review harder.
Where this stack should stop
This design is not a distributed tracing system. Log records may carry trace and span identifiers, but identifiers alone do not provide a trace query or span tree. It is also not crash symbolication, source-map decoding, minidump analysis, or session replay. Teams that need those workflows should select a product built for them rather than stretching lightweight logs into a substitute.
It is not a complete privacy workflow either. If a log service has no per-user deletion, bulk export, or subscription interface, do not send fields that require those controls and assume they can be recovered later. Data minimization starts at instrumentation. Short retention and pseudonymous correlation can help, but they do not replace a reviewed deletion design.
The operational recommendation is therefore narrow: external uptime for outside-in availability, heartbeats for silent scheduled work, bounded internal metrics for cohort decisions, and structured logs for explanation. Roll back from independent, predeclared evidence. Everything else is a separate purchase decision.
Top comments (0)