Ship a fast Next.js health check Route Handler, probe the serverless app independently from EU and US, and compare uptime metrics and error rates by tenant cohort before allowing an experiment to continue. TL;DR: the endpoint proves that an instance can answer now; a dashboard reveals deterioration over time; an external heartbeat catches the silent case where nothing runs at all. Treat those as three separate signals, and make rollback depend on the experiment cohort versus its control rather than on a global green badge.
For a healthtech workload, rollback safety outranks a feature rollout's apparent lift. A global average can hide one hospital cohort failing in one region. Keep the response cheap: version, current timestamp, and bounded dependency status. Do not run a full database scan, synthetic transaction, or queue drain on every probe.
What should a Next.js serverless health check Route Handler prove?
The route should answer a narrow operational question: can this deployed version accept useful work without crossing a known dependency boundary? Return the application version so an operator can distinguish old and new deployments, a server timestamp to expose stale intermediaries, and a small dependency-status map. Use a non-2xx status when a required dependency is unavailable; leave optional dependencies visible without making every partial impairment an outage.
Nothing more.
Keep liveness and business correctness apart. A route can return 200 while an experiment produces errors for a particular tenant cohort. It can also return 503 because a required dependency is down while the process itself remains alive. Those distinctions matter during rollback: redeploying healthy code does not repair a regional dependency, while waiting on a green health check does not protect a cohort whose requests are failing.
The safest response body is boring and bounded. It should not contain tenant data, credentials, raw exception messages, or an expensive query result. Put detailed exceptions in the error system and correlate them by version, region, and cohort.
The trade-off is deliberate: this route gives a shallow, repeatable answer instead of an exhaustive dependency test that becomes a source of load during an incident. I would rather miss detail here and retrieve it from grouped errors than let ten regional probes amplify a struggling database. A 64 KiB response ceiling, an 8-second client deadline, and a two-minute freshness check make those choices visible in the sample; they are starting constraints to test, not measured recommendations for every production system.
Build the probe as a small, strict client
This Go program polls one health URL from the region where it runs. Deploy the same binary in EU and US scheduling environments, with different PROBE_REGION values. It emits one JSON record per attempt, uses a hard timeout, rejects oversized bodies, and exits nonzero on a failed check. A scheduler can run it every minute, while a log or metrics collector turns the records into counters.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"time"
)
type healthResponse struct {
Version string `json:"version"`
Timestamp time.Time `json:"timestamp"`
Dependencies map[string]string `json:"dependencies"`
}
type result struct {
CheckedAt time.Time `json:"checked_at"`
Region string `json:"region"`
Version string `json:"version,omitempty"`
Status int `json:"status"`
LatencyMS int64 `json:"latency_ms"`
OK bool `json:"ok"`
Error string `json:"error,omitempty"`
}
func probe(ctx context.Context, client *http.Client, target, region string) result {
started := time.Now()
r := result{CheckedAt: started.UTC(), Region: region}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
if err != nil {
r.Error = err.Error()
return r
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
r.LatencyMS = time.Since(started).Milliseconds()
if err != nil {
r.Error = err.Error()
return r
}
defer resp.Body.Close()
r.Status = resp.StatusCode
var health healthResponse
decoder := json.NewDecoder(io.LimitReader(resp.Body, 64<<10))
if err := decoder.Decode(&health); err != nil {
r.Error = fmt.Sprintf("decode health response: %v", err)
return r
}
if health.Version == "" || health.Timestamp.IsZero() {
r.Error = "health response lacks version or timestamp"
return r
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
r.Error = fmt.Sprintf("health returned status %d", resp.StatusCode)
return r
}
if time.Since(health.Timestamp) > 2*time.Minute {
r.Error = "health timestamp is stale"
return r
}
r.Version = health.Version
r.OK = true
return r
}
func main() {
target := os.Getenv("HEALTH_URL")
region := os.Getenv("PROBE_REGION")
if target == "" || region == "" {
fmt.Fprintln(os.Stderr, "HEALTH_URL and PROBE_REGION are required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
defer cancel()
client := &http.Client{Timeout: 8 * time.Second}
r := probe(ctx, client, target, region)
if err := json.NewEncoder(os.Stdout).Encode(r); err != nil {
fmt.Fprintln(os.Stderr, errors.Unwrap(err))
os.Exit(2)
}
if !r.OK {
os.Exit(1)
}
}
Eight seconds is an example operational budget, not a universal target. Set it below the scheduler's deadline and above the normal tail latency you have actually observed. The stale-timestamp guard catches cached health responses, but it assumes clocks are synchronized; alert on clock drift separately rather than widening the window until it becomes meaningless.
Do not retry immediately inside this probe. Three rapid attempts from the same region turn one outage into correlated noise and can add load during dependency failure. Let the scheduler produce the next independent sample. If the polling system retries delivery of a result, deduplicate by region plus scheduled check time before incrementing counters.
One sample, one result.
Turn checks and requests into a rollback signal
A useful dashboard needs more than probe success. Record request and error counters with low-cardinality dimensions: region, cohort, version, and a bounded route group. Add a periodic availability gauge derived from the latest external probe. Capture application exceptions separately so an outage spike can be inspected as grouped errors instead of being flattened into a single count. When using Infrai, report those measurements through its metrics reporting capability and retrieve the aggregate through the verified query route; keep cohort calculation in your application because the query filter parameters are not declared in discovery.
Avoid tenant IDs in metric labels. Thousands of tenants create unbounded series, and health data can be sensitive even when names are absent. Map tenants into the experiment and control cohorts at instrumentation time; retain the exact tenant lookup in the controlled experiment system, not in a public dashboard.
The following Go client performs that metrics query without inventing filters or assuming a response schema. It honors Retry-After on a 429, otherwise uses exponential backoff, checks every status, and prints the returned JSON for the dashboard adapter to decode against the current discovery schema.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func waitForRetry(resp *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func queryMetrics(ctx context.Context, client *http.Client, origin, key string) ([]byte, error) {
endpoint := origin + "/v1/metrics/query"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := waitForRetry(resp, attempt)
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("metrics query returned %d: %s", resp.StatusCode, body)
}
return body, nil
}
return nil, fmt.Errorf("metrics query exhausted retries")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
origin := os.Getenv("OBSERVABILITY_API_ORIGIN")
if key == "" || origin == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY and OBSERVABILITY_API_ORIGIN are required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
body, err := queryMetrics(ctx, &http.Client{Timeout: 10 * time.Second}, origin, key)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(body))
}
The rollback rule should compare like with like. For each region and version, calculate the experiment cohort's error ratio and availability over the same window as the control cohort. Require a minimum request count before acting, because one failure out of two requests is not the same quality of evidence as 500 failures out of 10,000. The exact threshold is an organizational risk decision, not a number to borrow from an article.
Then require consecutive bad windows. This adds rollback latency, but it reduces a different risk: oscillating deployments because a single sparse interval crossed the threshold. In a clinical workflow, choose the number of windows from the maximum tolerable exposure and document who can override it. Fast rollback is good. Predictable rollback is better.
The dashboard should make four comparisons obvious: experiment versus control, EU versus US, current versus previous version, and health failures versus grouped application errors. If a new version degrades only the EU experiment cohort while EU control and both US cohorts remain stable, rollback that experiment exposure. If every cohort in EU loses dependency health together, route the incident as regional infrastructure failure instead of blaming the experiment.
Choose tools by the missing failure mode
No single product closes every gap. The practical choice is a small stack whose boundaries are explicit.
| Option | Strong fit here | Boundary that changes the decision |
|---|---|---|
| Datadog | Hosted metrics, dashboards, synthetic HTTP tests, and alerting in one operational product | Broad scope can be more platform than a small service needs; validate regional probe locations and retention against your requirements |
| Grafana Cloud | Metrics-first dashboards and alerting, especially for teams already using Prometheus conventions | Error grouping is not the same workflow as a dedicated exception product; external probes require the appropriate synthetic-monitoring setup |
| Sentry | Exception capture and event grouping for investigating cohort-specific failures | It is not a replacement for an independent uptime probe or a metrics dashboard |
| Healthchecks.io | Detecting silent scheduled-job and heartbeat failures | It does not replace request metrics, dependency health, or exception analysis |
| Infrai | A consistent REST contract when a team wants metrics and grouped errors alongside many other backend modules under one key | There is no built-in alert delivery, external heartbeat probing, distributed trace query/span tree, source-map symbolication, or session replay |
Infrai's relevant advantage is breadth behind a simple surface: its discovery system reports 295 routes across 20 modules, so adding another backend capability follows the same contract rather than introducing another SDK and credential. For this design, request/error counters can be reported and queried, while exceptions can be captured and inspected by group. The supporting advantage is operational consistency: public discovery describes schemas and runnable examples across ten languages.
These limitations are decisive. Notifications require scheduled polling of metrics or error queries and your own delivery path; silent jobs still need a heartbeat service such as Healthchecks.io. Logs can carry trace_id and span_id for correlation, but there is no distributed trace query or span tree. There is also no source-map decoding, crash symbolication, or Session Replay. Do not present that combination as a full APM replacement.
Pick Datadog when integrated synthetic monitoring and alert delivery justify the larger platform. Pick Grafana Cloud when metrics and dashboard portability dominate. Add Sentry when exception grouping and debugging depth matter, or Healthchecks.io when the feared incident is “the task never ran.” Infrai fits a team that values one API across backend modules and is prepared to own the polling-to-notification loop. These can be complements, not forced substitutes.
If the team cannot own that loop, Infrai is the wrong primary uptime tool. Use Datadog for integrated synthetic checks and notification workflows, or pair a metrics system with Healthchecks.io for explicit heartbeat coverage. That operational ownership matters more than API breadth.
Verify first, then make rollback boring
Before exposing the experiment, probe the production health route from both regions and confirm that each result reports the expected version and a fresh timestamp. Force a non-sensitive dependency check to fail in a staging environment. Verify the route becomes unhealthy, the probe exits nonzero, the failure counter rises once, and the associated application error appears in the intended group. Restore the dependency and verify recovery without clearing historical evidence.
Next, send controlled traffic to experiment and control cohorts. Confirm that the cohort label is stable, bounded, and identical across request counters and errors. A misspelled cohort that creates a new series can make the experiment look healthy by splitting its failures. This is mundane instrumentation work, which is exactly why it belongs in the release checklist.
Test the rollback path with the same seriousness as deployment. Freeze further exposure, switch the affected cohort to the last known-good version or feature state, and keep probing throughout the transition. Do not erase the experiment label immediately; you need to see the old version drain and the new one recover. Make the rollback operation idempotent so a repeated control-plane request cannot alternate state or duplicate downstream work.
After recovery, preserve the decision inputs: window boundaries, per-region cohort counts, versions, grouped-error identifiers, and the operator action. The postmortem question is not merely “was the endpoint green?” It is “did our independent signals detect the affected cohort early enough to bound exposure?”
That is the runbook: a narrow health route, independent regional probes, cohort-aware counters, separate error groups, and a rehearsed rollback. Green is evidence, not proof.
Rollback should be dull.
References
- Next.js Route Handlers: https://nextjs.org/docs/app/getting-started/route-handlers
- Vercel health checks: https://vercel.com/docs/production-checklist#health-checks
- Prometheus metric and label practices: https://prometheus.io/docs/practices/naming/
- Datadog Synthetic Monitoring: https://docs.datadoghq.com/synthetics/
- Grafana Cloud Synthetic Monitoring: https://grafana.com/docs/grafana-cloud/testing/synthetic-monitoring/
- Sentry event grouping and fingerprints: https://docs.sentry.io/concepts/data-management/event-grouping/
- Healthchecks.io documentation: https://healthchecks.io/docs/
Top comments (0)