Startup Uptime Monitoring API: Rollback-Safe Healthcheck Endpoints for Cohort Experiments

Ship the monitor that proves a cohort experiment can be rolled back safely, not the monitor with the smallest advertised bill. For an e-commerce startup, that means separating customer-facing health checks from cron heartbeats, retaining tenant and experiment context without creating unbounded labels, and testing the status path before the experiment reaches every cohort.

TL;DR: treat uptime, scheduled-job completion, and public incident communication as three related but distinct contracts. A green health endpoint cannot prove that yesterday's comparison job finished. A received heartbeat cannot prove that its writes were correct. Keep rollback state outside the monitoring service, make every scheduled comparison idempotent, and choose an EU/GDPR posture only after mapping the data that actually leaves your system. Price belongs in the decision, but after failure coverage and rollback evidence.

What should a startup uptime monitoring API prove before expansion?

Suppose a commerce team compares a checkout experiment across tenant cohorts. Cohort A receives the current flow, cohort B receives a new flow, and a scheduled job aggregates outcomes before the next rollout step. The operational question is narrower than “is the API up?” The team needs to know whether requests are being served, whether the comparison window closed, and whether the result is trustworthy enough to continue.

Those signals fail on different clocks. An external probe may run every minute while the aggregation job runs hourly. The probe can remain green through a stuck queue, a missed scheduler invocation, or a job that exits successfully after processing only part of its assigned tenants. Conversely, a late probe from one region should not automatically reverse a sound experiment.

Use an explicit rollout gate. For example, keep a cohort at its current exposure when any of these conditions is true:

• the serving health check cannot complete its bounded dependency checks;
• the expected job completion event is absent after its deadline and grace period;
• the run reports an incomplete tenant set, a conflicting attempt, or an unknown experiment revision;
• the rollback target or last known-good configuration cannot be identified.

This is conservative by design. Rollback safety is the primary axis, so an ambiguous result means “hold,” not “advance.” The monitor raises evidence; the deployment controller owns the change. That boundary prevents a transient observation from becoming an uncontrolled configuration write.

Hold means hold.

Model three contracts, not one green light

The first contract is availability. A health endpoint should answer a specific question in bounded time. A shallow endpoint can show that the process accepts traffic. A readiness-style endpoint may include dependencies required to serve the request path. Combining every downstream system into one check makes the result hard to interpret and lets a slow optional dependency turn a useful probe into noise.

The second contract is freshness. A cron heartbeat is evidence that a named run reached a defined point. “Started,” “completed,” and “failed” are different events. For the cohort comparison, completion should carry a run identifier, the experiment revision, the expected window, and aggregate counts that allow the receiver to reject obvious partial work. Do not put tenant IDs into metric label values. Prometheus instrumentation guidance warns against overusing labels because every unique label combination creates a new time series. Tenant-level detail belongs in bounded logs or a queryable run record, with access and retention controls appropriate to that data.

The third contract is communication. A status page describes customer impact and recovery; it is not the source of truth for rollout automation. Publishing an internal tenant identifier, raw endpoint payload, or job token there creates exposure without improving the rollback decision. Feed it a deliberately small incident state: affected capability, scope, start time, and current mitigation status.

One light is tempting. It is also lossy.

Implement an idempotent heartbeat receiver

The receiver below uses Go's standard library and a storage interface. It accepts one completion event for each run and treats a byte-for-byte-equivalent retry as success. A conflicting retry is rejected. In production, authenticate the request before decoding it, cap the body size at the edge as well as in the handler, and implement PutOnce with a database uniqueness constraint on run_id rather than an in-memory check.

package heartbeat

import (
    "encoding/json"
    "errors"
    "fmt"
    "net/http"
    "time"
)

type Completion struct {
    RunID              string    `json:"run_id"`
    ExperimentRevision string    `json:"experiment_revision"`
    WindowEnd          time.Time `json:"window_end"`
    ExpectedTenants    int       `json:"expected_tenants"`
    ProcessedTenants   int       `json:"processed_tenants"`
}

var ErrConflict = errors.New("run already recorded with different data")

type Store interface {
    PutOnce(Completion) error
}

type Handler struct {
    Store Store
    Now   func() time.Time
}

func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        w.Header().Set("Allow", http.MethodPost)
        http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
        return
    }

    var c Completion
    dec := json.NewDecoder(http.MaxBytesReader(w, r.Body, 16<<10))
    dec.DisallowUnknownFields()
    if err := dec.Decode(&c); err != nil {
        http.Error(w, "invalid completion event", http.StatusBadRequest)
        return
    }

    if err := validate(c, h.Now().UTC()); err != nil {
        http.Error(w, err.Error(), http.StatusUnprocessableEntity)
        return
    }
    if err := h.Store.PutOnce(c); err != nil {
        if errors.Is(err, ErrConflict) {
            http.Error(w, "conflicting run", http.StatusConflict)
            return
        }
        http.Error(w, "storage unavailable", http.StatusServiceUnavailable)
        return
    }

    w.WriteHeader(http.StatusNoContent)
}

func validate(c Completion, now time.Time) error {
    if c.RunID == "" || c.ExperimentRevision == "" {
        return errors.New("missing run identity")
    }
    if c.ExpectedTenants <= 0 || c.ProcessedTenants < 0 {
        return errors.New("invalid tenant counts")
    }
    if c.ProcessedTenants != c.ExpectedTenants {
        return fmt.Errorf("incomplete run: processed %d of %d tenants",
            c.ProcessedTenants, c.ExpectedTenants)
    }
    if c.WindowEnd.After(now) {
        return errors.New("window end is in the future")
    }
    return nil
}

The 16 << 10 limit is 16 KiB, intentionally larger than this small event but small enough to reject an accidental bulk upload. The exact limit is a local interface decision, not a universal recommendation. What matters is that it is explicit, tested, and enforced before unbounded decoding.

There is a subtle sequencing trap here. Do not send “complete” and then commit the cohort results. Commit the result and an outbox record in one transaction, then deliver the heartbeat from that durable record. Retries use the same run ID. Otherwise a process crash can produce the worst kind of green signal: completion observed, data absent.

That ordering is the safety mechanism.

Choose the service by failure coverage

The original buying question often starts with “cheapest API.” A useful evaluation starts with a failure matrix instead. Run the same trial against any managed service or self-hosted implementation; do not infer coverage from a feature label.

Test	Expected evidence	Rollout action
Serving process stops	Probe failure with timestamp and region	Hold expansion; inspect impact
Scheduler never starts the run	Missing-heartbeat alert after deadline plus grace	Hold; invoke the runbook
Worker retries after timeout	One accepted run ID, no duplicate result	Continue only after completeness checks
Job processes 48 of 50 expected tenants	Incomplete-run signal with aggregate counts	Hold and repair the run
Monitor cannot deliver an alert	Secondary delivery or explicit delivery failure	Treat monitor path as degraded
Experiment must be reversed	Last known-good revision remains addressable	Roll back independently of monitor availability

Then assess EU/GDPR fit from the data flow. Inventory probe locations, event fields, logs, notification destinations, retention, deletion behavior, subprocessors, and access paths. A regional marketing label alone does not answer where heartbeat payloads or alert messages travel. Minimize the payload first. It reduces the amount of sensitive operational context that any processor needs to handle.

Cost comparison comes last and uses your workload: number of endpoints, probe regions, check interval, scheduled jobs, notification volume, retained history, status-page needs, and engineering time for maintenance. Avoid a plan that becomes operationally blind at the first quota boundary. Also avoid paying for dimensions the rollback decision does not use. The cheapest valid option is the least expensive one that passes the failure drill and data-flow review, not the smallest number on a pricing page.

There are real limitations on both sides of the build-or-buy choice. A minimal hosted uptime API may be unsuitable when the decision needs transactional evidence from the cohort result, because an external probe cannot inspect an internal commit without widening access. A self-hosted receiver can keep that evidence close to the source and enforce a precise idempotency key, but the team then owns its availability, upgrades, storage recovery, alert delivery, and on-call runbook. A feature-rich hosted service may cover probes, heartbeats, and status communication in one place, while also increasing the amount of operational metadata sent to a processor and coupling several failure signals to one control plane. None of those trade-offs settles the choice in advance. Use an external probe for independent reachability, keep authoritative experiment state in the commerce system, and select hosted or self-hosted heartbeat handling according to the failure drill and reviewed data flow.

Verify the page, the alert, and the rollback

Before expanding beyond the first tenant cohort, stage a controlled failure. Stop a disposable serving instance and confirm that the external check changes state. Suppress one scheduled completion and confirm the missing-heartbeat deadline. Replay the same completion event twice, then send a conflicting payload with the same run ID. The first retry should be harmless; the conflict should remain visible.

Check the human path too. The alert needs the experiment revision, comparison window, run ID, and a link to the internal run record. It should not require an operator to reconstruct those fields from a generic “cron failed” page at 03:00. The public status update should contain only customer-relevant scope, while the internal timeline preserves the evidence needed for review.

Rollback is a separate drill. Pin the prior experiment revision, stop further cohort expansion, and verify that serving behavior returns without depending on the monitoring provider. Leave the failed run record immutable; issue a new run ID for recomputation. Reusing the old ID blurs the audit trail and can make a legitimate retry indistinguishable from changed input.

Finally, define ownership. One team owns probe semantics, one owns the scheduler deadline, and the experiment owner decides whether evidence permits expansion. Those may be the same small group, but the responsibilities still need names. Monitoring without an operator and a decision rule is storage.

References

• https://prometheus.io/docs/practices/instrumentation/
• https://logback.qos.ch/manual/appenders.html