Short answer: A startup should choose the least elaborate monitoring setup that can still reconstruct an incident across its US and EU tenant cohorts: external regional health checks, one heartbeat per scheduled job class, and a status page fed by confirmed incidents rather than raw probe failures. Cheap is a constraint, but evidence quality is the decision axis. A low monthly bill is wasted if the retained data cannot show whether an experiment harmed one cohort, one region, or everyone.
For a fintech application, the hard part is not making a green page. It is preserving enough independent evidence to compare an experiment without turning every tenant ID, request ID, and feature flag into an expensive permanent label. The five choices below derive the setup from that constraint.
1. Give each signal one job
An uptime check, a cron heartbeat, and an incident page answer different questions. An external check asks whether a user can reach a narrow public path from a region. A heartbeat asks whether scheduled work reported completion within its expected interval. An incident page communicates an assessed service condition. Combining them into one synthetic signal makes setup look easier, yet it weakens reconstruction: a late settlement job is not proof that the public API is unavailable, and one failed probe is not enough evidence for a customer-facing incident.
The minimum useful design has two regional probes against the same shallow endpoint, a heartbeat keyed to each materially different job schedule, and an incident workflow that requires corroboration. Keep tenant-cohort context in deploy and experiment metadata, not in the public health response. That boundary matters. If /health/ready emits tenant IDs or experiment variants, a bounded availability test quietly becomes a cardinality generator.
A probe can stay deliberately boring:
code=$(curl --silent --show-error --output /dev/null --write-out '%{http_code}' \
--connect-timeout 3 --max-time 8 \
https://api.example.test/health/ready)
test "$code" = "200"
This checks reachability and readiness. It does not perform a transfer, mutate a ledger, or depend on one tenant's data. Don't put destructive or financially meaningful work in a health check.
2. How should US/EU health checks, cron heartbeats, and an incident page divide evidence?
Start with a signal-to-question matrix. It prevents a common evaluation mistake: comparing plans by feature count before deciding which evidence must survive an incident.
| Signal | Question answered | Useful dimensions | Dimensions to avoid |
|---|---|---|---|
| US external check | Can a US client reach the public edge? | region, endpoint class, result | tenant ID, request ID |
| EU external check | Can an EU client reach the public edge? | region, endpoint class, result | tenant ID, request ID |
| Cron heartbeat | Did a job class finish on schedule? | job class, environment, result | run UUID as a metric label |
| Experiment event | Which cohort received which behavior? | experiment, variant, cohort definition version | raw account number |
| Incident record | What was declared, when, and why? | affected region, service, phase | every underlying probe sample |
The experiment join should happen during investigation, not in the uptime metric's label set. Preserve a deployment marker and the cohort-definition version beside the experiment timeline. Then retain high-level probe outcomes long enough to cover the incident-review window, while keeping detailed request evidence for a shorter period or sampling it. This makes the comparison possible without multiplying every time series by every tenant.
Consider an illustrative experiment with 2 regions, 3 endpoint classes, 2 result states, 2 variants, and 400 tenants. A region-by-endpoint-by-result metric has 2 × 3 × 2 = 12 combinations before ordinary operational labels. Adding variant makes 24. Adding tenant makes 9,600. The last multiplication rarely improves detection, because the external probe is not acting as each tenant; it mainly enlarges storage and query work. Keep tenant-level outcomes in a controlled event dataset where access, sampling, and retention can be managed separately.
This is also where feature-toggle discipline helps. Cohorts change over time, so an incident record needs the cohort rule or its version, not merely the friendly variant name. Martin Fowler's feature-toggle taxonomy distinguishes release, experiment, ops, and permissioning concerns; treating every toggle as equivalent loses the operational context needed to interpret an exposure.
3. Spend the telemetry budget on reconstruction
Price tables change. The durable comparison is a unit model: check executions, heartbeat events, retained incident records, status-page subscribers, and any label or series limits. Ask each candidate to map its bill to those units, then model the traffic you intend to send. I'm not sure a single universal retention period exists; the right answer depends on how long finance, support, and engineering need to dispute or reproduce a cohort result. The requirement should be written down before a plan is selected.
Use retention math, not instinct. At a one-minute interval, two regions probing three endpoint classes produce 2 × 3 × 60 × 24 = 8,640 check executions per day. That is an input count, not a claim about any provider's billing. If the application has 12 job classes and each emits one terminal heartbeat per hour, that adds 288 events per day. Retries deserve a separate result field or event type; otherwise a noisy retry loop can look like healthy throughput.
Storage needs a similar budget. Suppose, purely for capacity planning, a normalized result averages 300 bytes before indexing. The six one-minute checks produce about 2.6 MB of raw result payload per day. Indexing and platform overhead are implementation-dependent, so your mileage may vary. Measure an exported day from the trial rather than multiplying that estimate into a purchasing claim. A useful trial records raw bytes ingested, indexed series, retained event count, and query latency for one incident reconstruction.
Sampling belongs after the invariants. Keep every state transition and every incident declaration. Sample repetitive successful probes if the platform permits it, but retain failed probes and the successful observations on both sides of a failure window. For experiment events, deterministic sampling by a pseudonymous cohort key is more useful than independent random sampling when the goal is comparison, because the same cohort remains represented across the window. The catch is that rare failures can disappear under aggressive sampling. When missed rare events carry material risk, retain all qualifying error events and sample only the high-volume success path.
Shorter is often better.
4. What should a team test before publishing an incident?
The easiest setup is the one whose failure behavior the team can explain at 03:00. Run a controlled rollout that disables one non-financial test endpoint, delays a test heartbeat, and changes an experiment cohort definition in a staging environment. Confirm that the probes remain regionally distinguishable, the late heartbeat does not automatically claim total application downtime, and the timeline retains the cohort-definition version.
Avoid wiring one failed check directly to the public page. Require a small state machine: suspected, confirmed, monitoring, resolved. Confirmation can use consecutive failures, agreement across vantage points, or an operator decision according to the service's risk. The exact threshold is a policy choice; record it with the incident so a reviewer knows why publication occurred. A stringent threshold reduces noisy declarations but can delay communication. A permissive threshold reports sooner but can publish transient network noise.
Customer experience belongs in the test as a separate layer. Core Web Vitals uses the 75th percentile as the assessment threshold for LCP, INP, and CLS. That does not turn web-vitals data into an uptime signal. It offers a useful pattern for cohort comparison: examine a distribution and a defined percentile rather than declaring success from an average. Compare the experiment variants by region and cohort, while keeping those browser measurements separate from the binary availability checks.
A candidate is not suitable when it cannot export incident history and underlying observations in a form the team can retain. A hosted page is also the wrong choice when policy requires the entire communication surface to run inside a controlled network; use a self-hosted incident page and monitoring components in that case. Conversely, self-hosting is a poor bargain for a small team that cannot own upgrades, independent delivery, and on-call operation of the monitoring path. The operational burden is part of cost.
5. Roll out with parallel evidence, then remove noise
Run the old and new paths in parallel for one review window. Do not dual-page the on-call team; send the new path to a non-paging destination, then compare timestamps, regional attribution, heartbeat lateness, and incident transitions. One representative experiment should cross the window so reviewers can reconstruct variant exposure from the retained metadata.
After the review, remove labels that were never used in detection or reconstruction, set explicit retention by signal class, and document the incident confirmation rule. Then switch paging, keep the previous history read-only for its required retention period, and rehearse an export. The final acceptance test is compact: given an incident timestamp, can an engineer identify the affected region, the relevant job class, the deployed experiment version, and the evidence that justified each public status transition?
If the answer is yes, the setup is easy enough.
Top comments (0)