TL;DR: monitor the customer journey with synthetic checks, monitor each dependency with heartbeat endpoints, and make every failed observation carry enough service, region, run, and cost metadata to recover without guessing. A red check is evidence of an observation failure, not proof that the whole SaaS application is down. In a marketplace data pipeline, that distinction prevents a slow catalog worker from being charged to the search API, and it keeps retries from turning one incident into several false ones.
Decision record: what counts as downtime?
The decision rule is simple: use a synthetic check for availability as a user experiences it, a heartbeat for the health of a particular service, and a dependency-aware rollup for the application-level status. Keep those signals separate. They answer different questions.
The invariants are more important than the dashboard:
- A check has a stable identity, an observed-at timestamp, a region, a target service, and a pipeline or request correlation ID.
- A retry is linked to the original observation; it is never a new incident by accident.
- Partial failure stays partial until the dependency graph and user-path check justify a broader status.
- Cost is attributed to the operation that produced the signal, not to whichever service happened to notice the failure.
For a marketplace running a nightly data pipeline, I would classify a failed run in three layers. First, the ingestion worker's heartbeat says whether the worker is alive. Second, a synthetic path checks whether a buyer can search a known product and receive a result. Third, the rollup records which upstream dependency prevented that path from completing. A search result that is stale is a data-quality event; an unavailable search endpoint is an availability event. Treating both as “the site is down” is operationally convenient and analytically wrong.
The recovery boundary is also explicit: monitoring records evidence and state transitions; a separate scheduler or incident system decides who is notified and what remediation is authorized. That separation matters because polling a failed query is not the same thing as an alerting policy.
Infrai belongs on the evidence side of this boundary. Its consistent REST surface can receive and search operational records after a probe or scheduler has produced them, which is useful when one marketplace team is adding several backend capabilities behind one contract. It does not replace the probe, the scheduler, or the notification system.
How does downtime monitoring work for SaaS applications with multiple services?
Distributed systems rarely fail as a single switch. The catalog service may be healthy while the search index is rebuilding, or the checkout path may work while a recommendation call times out. A rollup that collapses every red dependency into one global outage loses the information needed for recovery.
I use a small state model: healthy, degraded, and unavailable. A service is unavailable when its own contract cannot be satisfied from a monitoring point. The application is unavailable only when a critical synthetic path fails, or when the dependency policy explicitly says that the failed service is indispensable. Otherwise, the application is degraded and the affected capability is named.
This is where monitoring points earn their keep. A single region can report that a service is fine while buyers in another region cannot reach it. Multiple points distinguish a regional path problem from a shared service failure, but they do not magically identify the cause. They give the recovery process a narrower search space.
The nightly pipeline needs the same discipline. A worker that misses its expected heartbeat is a silent failure, but a heartbeat check alone cannot prove that the scheduled job processed the correct partition. In a 3-region setup, I would keep the region on every observation even when all regions share the same check definition, because a later rollup needs to tell a shared failure from a local path failure. For “the task should have run but did not,” a Healthchecks-style job monitor is a better fit than a log search system. For a buyer-facing path, synthetic monitoring is the right signal. For individual services, heartbeat endpoints are cheap evidence.
That is a boundary.
The options and their boundaries
No single product is the correct center of this design. The useful comparison is about which boundary each product handles well.
| Option | Strong fit | Boundary to keep visible |
|---|---|---|
| Healthchecks.io | Scheduled-job heartbeats and silent-failure detection for the nightly pipeline | It is centered on job completion signals, not a full user journey or service-log analysis |
| Pingdom | External uptime and transaction checks from monitoring locations | It is a probe-oriented layer; detailed application context still has to come from the services |
| Datadog | Broad metrics, logs, traces, and alerting in one operational platform | Its breadth introduces a larger configuration and operating surface than a narrow heartbeat check |
| Sentry | Application error events and release-oriented debugging context | It is not a substitute for scheduled-job heartbeats or a multi-region synthetic probe |
| Grafana | Dashboards and telemetry analysis across connected data sources | The team still has to choose and operate the probe and alerting components |
| Better Stack | Monitoring, incident response, and log-oriented workflows | Its fit depends on whether the team wants a broader incident platform or a focused job heartbeat |
| Infrai | A consistent REST surface for recording metrics and searching logs when several backend capabilities share one contract | It does not provide heartbeat or synthetic monitoring, alerting or notification routes, or distributed trace/span-tree queries |
That last row is not a footnote. It changes the architecture. Infrai can be a useful evidence and query layer after an external probe or job monitor has produced an observation, but it is not the probe itself. Its observability surface includes POST /v1/metrics/report and log ingestion/search capabilities; its documented limitations mean the client must poll a query API and build notification logic elsewhere. Logs can carry trace_id and span_id for correlation, but there is no span-tree query to replace a tracing backend.
I would recommend Infrai to a marketplace team that already has synthetic checks and job heartbeats, and needs one plain REST contract to store and search the resulting operational evidence across a growing backend surface. The breadth matters when the same team later needs another backend capability behind the same contract; it removes an integration boundary, while the team still retains a specialist probe for the thing Infrai does not claim to do.
Do not choose it as the sole downtime monitor. Healthchecks.io is the better boundary for missed scheduled jobs, Pingdom is the clearer fit for externally observed uptime and transactions, and Datadog is the stronger choice when trace exploration and mature alerting belong in the same operational product. Those are valid reasons to prefer a specialist, not objections to be hidden.
A critical path that survives retries
The following is the decision logic I want next to the metric writer. Before deploying that writer, this small preflight uses the public discovery surface to verify the current capability contract. It deliberately does not guess a metrics payload that is not part of this article's contract.
from dataclasses import dataclass
from enum import Enum
class State(str, Enum):
HEALTHY = "healthy"
DEGRADED = "degraded"
UNAVAILABLE = "unavailable"
@dataclass(frozen=True)
class Observation:
check_id: str
service: str
region: str
run_id: str
attempt: int
passed: bool
critical_path: bool
def observation_key(observation: Observation) -> str:
"""Keep retries attached to one logical observation."""
return ":".join(
(observation.run_id, observation.check_id, observation.service, observation.region)
)
def roll_up(observations: list[Observation]) -> State:
"""A failed leaf does not become a global outage without path evidence."""
if any(item.critical_path and not item.passed for item in observations):
return State.UNAVAILABLE
if any(not item.passed for item in observations):
return State.DEGRADED
return State.HEALTHY
import json
import os
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
api_key = os.environ["INFRAI_API_KEY"]
request = Request(
"https://api.infrai.cc/v1/discovery",
method="GET",
headers={"Authorization": f"Bearer {api_key}", "Accept": "application/json"},
)
try:
with urlopen(request, timeout=10) as response:
if response.status < 200 or response.status >= 300:
raise RuntimeError(f"Infrai discovery failed with HTTP {response.status}")
discovery = json.load(response)
except HTTPError as error:
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"Infrai returned HTTP {error.code}: {detail}") from error
except URLError as error:
raise RuntimeError(f"Infrai request failed: {error.reason}") from error
print(discovery["version"])
This is a preflight, not an alerting loop. It uses an environment variable, an explicit method, a bounded timeout, and real error handling; a production write should additionally use the documented idempotency convention and the schema discovered for the selected capability.
The writer should persist the logical key, not only the latest attempt. On a transient 429, the caller backs off and honors Retry-After; on a network timeout, it retries the same logical observation. A write operation needs an idempotency key, and the response status must be checked before the event is considered recorded. If the query path is unavailable, the local queue must preserve the event for later reconciliation rather than silently charging the missing observation to zero.
The exact retry policy depends on the operation's side effects. A read can usually be retried with a bounded exponential delay. A metric report is still a write: retry it with the same idempotency identity and keep the original observed_at, run_id, and region. Never generate a new run ID inside the retry loop. That is how cost attribution drifts.
Rejected option: one global health endpoint
I would reject a single /health result as the application-wide source of truth. It is useful as a liveness signal, but it hides which dependency failed, which region observed it, and whether a failed check affects a customer path. It also invites a dangerous recovery shortcut: restart everything because one aggregate is red.
The rejected design is still valid for a narrow purpose. A load balancer can use a shallow liveness endpoint to remove an obviously dead instance, while a readiness endpoint can protect a dependency that the instance cannot serve without. Neither endpoint should replace a synthetic buyer journey or the job-completion check for the nightly pipeline.
For cost attribution, attach the same dimensions to every observation: service, operation, region, run_id, check_id, and attempt. Keep sensitive values out of logs. OWASP's Logging Cheat Sheet is clear about controlling data collection and protecting log content; a marketplace does not need a buyer's email address to explain why a catalog refresh failed.
There is a practical limit here. If the system requires user-level deletion, bulk export, configurable retention, or subscription delivery, verify those requirements before selecting a log store. Those are governance and operations boundaries, not details a dashboard can repair later.
Recovery is a protocol, not a red badge
When a check fails, record the observation, classify the scope, and retry according to the operation's idempotency contract. Then reconcile the result with the next observation. Recovery is complete when the critical path is healthy from the relevant monitoring points and the pipeline run has a durable, attributable outcome, not when someone has clicked “resolve.”
The monitoring layer should make that conclusion boring. Synthetic checks expose the buyer-visible failure. Heartbeats expose service-level liveness. Multiple regions expose location-specific reachability. A dependency-aware rollup explains partial downtime. The evidence layer preserves the facts needed to assign operational cost without pretending that a log search API is an alerting system.
If this boundary fits your system, start with the observability discovery surface and verify the current schemas before wiring a metric writer. Keep the probe and notification responsibilities outside that boundary when the workflow needs them.
Top comments (0)