DEV Community

KillianBerg5391
KillianBerg5391

Posted on

Downtime Monitoring for SaaS Applications with Multiple Microservices — Signal Design

Short answer: Downtime monitoring for SaaS applications with multiple microservices works best as layered evidence: service heartbeats, a synthetic checkout journey, and probes from multiple regions, interpreted together before anyone declares the application down.

The least complex design that gives a trustworthy answer is layered. A checkout monitor should answer one operational question: can a customer complete the purchase right now? Treat each result as a signal, then interpret the signals together before declaring the checkout down.

That last step matters. A payment service can be healthy while an API gateway, inventory dependency, or browser-facing path is failing. A single green endpoint is not proof that checkout works; a single red dependency is not automatically a customer-visible outage.

How does downtime monitoring work for SaaS applications with multiple microservices?

Start with a service map small enough to reason about. For a B2B SaaS checkout, that may be the edge or gateway, catalog, cart, inventory, payment authorization, and order creation paths. Each service exposes a heartbeat endpoint that checks its essential dependencies without performing a real purchase. A synthetic check then performs the user-shaped path: load the checkout, submit a test cart, reach the payment handoff, and confirm the expected result using a safe test account.

Heartbeats are cheap and local. Synthetic actions reveal routing, authentication, browser, and integration failures that health endpoints can miss. Multiple monitoring points expose a regional network or deployment problem that a single probe might misclassify as a global outage.

I initially thought the synthetic journey should be the whole monitor. It is easy to explain and close to the user's experience. In practice, that creates noisy diagnosis: one failed journey tells you checkout is broken, but not which dependency caused it. The better trade-off is to keep the synthetic test as the customer-impact signal and use heartbeats to narrow the search.

How do you separate a partial outage from a checkout outage?

Use an explicit decision rule instead of letting every service emit an independent page.

Evidence Interpretation Action
Synthetic checkout fails from several regions and a critical heartbeat fails Likely customer-visible outage Page the failed dependency owner and preserve run evidence
Synthetic checkout fails in one region while heartbeats remain healthy Regional path or probe issue is plausible Compare another monitoring point before declaring a global incident
Heartbeat fails but synthetic checkout succeeds Dependency degradation or an over-sensitive check Investigate, but keep it as a warning until impact is confirmed
All heartbeats pass and synthetic checkout fails User-path, gateway, browser, or test-data problem Inspect the synthetic step and request evidence
One non-critical service fails while checkout succeeds Partial degradation Record the failure without creating an outage page

Distributed architecture changes the meaning of “down.” A dependency can be unavailable without making every customer action impossible. Conversely, checkout can return ordinary HTTP success while a downstream authorization or order-creation step silently fails. The monitor needs a small dependency model and a signal-quality policy, not a larger pile of pings.

That distinction is the whole job.

Keep the rule visible.

Store at least the check name, service, region, timestamp, outcome, duration, and correlation identifiers. Keep sensitive request data out of logs. OWASP's logging guidance is a useful guardrail: monitoring data should help an operator reconstruct an event without becoming a second place where payment or identity data leaks.

A small Python collector that keeps the handoff visible

This example shows the seam between account-platform and observability data behind one REST API: it reads the available key inventory, then polls log search with the same base URL and bearer key. The local join preserves the first response as incident context; the second response supplies evidence for the same review. The search endpoint is intentionally called without invented filter parameters because its discovery contract does not declare them.

import os
import time
from typing import Any

import requests

BASE_URL = os.environ.get("OBSERVABILITY_BASE_URL", "https://api.example.invalid/v1")
API_KEY = os.environ["INFRAI_API_KEY"]


def get_json(path: str) -> Any:
    headers = {"Authorization": f"Bearer {API_KEY}"}
    delay = 1.0

    for _ in range(4):
        response = requests.request(
            method="GET",
            url=f"{BASE_URL}{path}",
            headers=headers,
            timeout=10,
        )
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else delay)
            delay *= 2
            continue
        if not response.ok:
            raise RuntimeError(
                f"GET {path} failed with {response.status_code}: {response.text}"
            )
        return response.json()
    raise RuntimeError(f"GET {path} was rate-limited after retries")


def collect_incident_context() -> dict[str, Any]:
    key_inventory = get_json("/account/keys/list")
    log_evidence = get_json("/logs/search")
    return {
        "key_inventory": key_inventory,
        "log_evidence": log_evidence,
        "same_base_url": BASE_URL,
        "same_credentials": True,
    }


if __name__ == "__main__":
    print(collect_incident_context())
Enter fullscreen mode Exit fullscreen mode

This is a collector, not an alerting system. There is no notification route in this setup, so a scheduler must poll it and apply the decision rule. It also does not pretend to provide distributed span trees: correlation IDs in log records can be linked, but the collector cannot reconstruct a trace graph.

The code does not revoke a key, invent a log filter, or claim that a log query is an alert. A real collector would normalize returned records into the check schema, attach the synthetic run ID, and send the result to the polling process that owns notification policy. Account rotation or compromise review and the log evidence used to estimate blast radius can still be examined with one key and one base URL.

The same workflow built from a vendor console plus Datadog logs would mean two signups, two credential sets, and glue that copies an identifier from the account system into the logging system. That split is workable, and Datadog has a mature log-search and alerting experience, but the operator owns the correlation contract. A single REST surface reduces that integration seam while introducing its own cost: one platform to trust, one bill, and one outage surface.

Where the common tools fit

No tool wins every layer. Better Uptime is a sensible fit when the primary need is straightforward uptime checks, incident status, and notifications around external URLs. It is less suited to explaining a multi-service dependency without additional instrumentation.

Checkly is a strong choice for synthetic monitoring because browser and API checks can represent a customer journey more closely than a heartbeat. Synthetic coverage still needs service-level health signals and a policy for distinguishing a failed probe from a regional or test-account problem.

Datadog covers broader infrastructure observability, including logs, metrics, traces, and alerting, so it fits teams that want an integrated operations platform and can accept its configuration surface. Its breadth does not remove the need for a checkout-specific decision rule; more telemetry can produce more noise when every dependency can page independently.

Sentry is useful when the strongest signal is an application exception or error group, especially for finding the code path behind a failed request. It is not a substitute for a regional synthetic checkout check. Grafana is a good fit for teams that already own metrics, dashboards, and alert rules across their stack; its flexibility also means the team must define the service dependency policy itself.

Infrai is a reasonable option when the desired capability is a plain REST contract across account and observability work, especially when keeping the handoff behind one key matters. Its discovery surface and runnable examples make the contract inspectable before wiring a client. The boundaries are material: there is no built-in alert or notification route, no heartbeat or synthetic monitoring, and no distributed trace or span-tree query. A team still needs a polling scheduler, a synthetic-monitoring product such as Checkly, or a heartbeat service such as Healthchecks to cover those gaps.

The selection rule is simple: choose the smallest combination that can produce a trustworthy customer-impact signal and explain it during an incident. A focused uptime tool may be enough for one public endpoint. A synthetic platform earns its place when the user journey is the risk. A broad observability platform fits teams that need deep cross-signal investigation. A REST capability surface fits teams that want to own the decision policy and keep the integration contract portable.

Operating the monitor without creating alert fatigue

Run heartbeats frequently enough to catch a real failure, but require more than one failed sample before treating a transient network error as a service outage. Run the synthetic checkout less often than the heartbeats if it is expensive or touches scarce test resources. Keep the test account and payment path isolated from production customer data, and rotate credentials through the same operational process as other sensitive keys.

Record region with every result. A global decision should require corroboration from multiple monitoring points, while a regional result should remain visible as a regional incident rather than being flattened into “all checkout is down.”

Review false positives. If a heartbeat pages repeatedly while synthetic checkout remains healthy, the check is probably measuring an internal dependency too aggressively or using a timeout that does not match its responsibility. If synthetic checks fail while every heartbeat is green, inspect the browser path, gateway behavior, authentication, and test data. Signal quality improves when the team changes checks based on evidence.

Further reading

Top comments (0)