DEV Community

BrennThorn8571
BrennThorn8571

Posted on

SaaS Downtime Monitoring: How Multiple Microservices Reconstruct Application Workflows

#observability #microservices #sre #healthchecks

A useful downtime monitor for a multi-service SaaS application should answer one operational question: can we reconstruct what stopped the nightly data pipeline, in order, from signals that were captured before the alert? The decision rule is simple: treat endpoint availability as a symptom, and preserve a time-correlated record of dependency health, pipeline stages, and customer-visible impact.

Short answer: probe the externally meaningful workflow, collect shallow health signals from each dependency, attach a stable trace or run identifier to structured events, and alert on an SLO-relevant condition rather than on every failed check. A service returning 200 is not proof that a pipeline can read, transform, and publish its data.

That is the whole point.

How does downtime monitoring work across multiple SaaS services?

For a B2B SaaS nightly pipeline, the monitor has two jobs that are easy to confuse. A liveness check asks whether a process is running. A readiness check asks whether that process can accept useful work. The incident-reconstruction monitor needs a third signal: did the scheduled run produce the expected business result within its latency objective?

That distinction matters during partial failure. The API may be reachable while the scheduler cannot enqueue a run; the worker may be ready while the object store is returning authorization errors; or the export stage may finish while a downstream index remains stale. Each condition produces a different recovery action. A single global uptime percentage hides that choice.

I would define the nightly workflow as a small state machine: scheduled, accepted, reading, transforming, publishing, and verified. Each transition emits a structured event with run_id, service, stage, observed_at, duration_ms, and a bounded error category. The monitor then correlates events by run_id, not by adjacent log lines. Clock skew and concurrent runs make adjacency an unreliable join key.

The SLO should describe the user-relevant outcome. For example, the objective might require 99.5% of scheduled runs to reach verified before the reporting deadline over a rolling window. The exact target belongs to the service owner and its contract; the important part is that the alert measures completion and freshness, while component checks explain the breach.

How should signals survive a partial outage?

Start at the edge with a synthetic transaction that submits a small, isolated pipeline request. It should exercise authentication, scheduling, one representative read, a transformation with a known result, and publication to a test destination. Keep this probe separate from customer data and make its side effects disposable. A DNS or load-balancer check can run more frequently, but it is only an availability signal.

Inside the workflow, health endpoints should be cheap and explicit. Liveness must not perform a full dependency cascade, because making every probe call every dependency turns an outage into a probe storm. Readiness can report dependency state, but it should distinguish degraded from not_ready and include a machine-readable reason. The scheduler's run record remains the authority for whether work was accepted.

Logs provide the reconstruction detail, provided they are designed as records rather than prose. Do not place access tokens, session identifiers, or raw customer payloads in them. OWASP's logging guidance calls out sensitive data management, event attributes, and the need to protect logs as operational data, so redaction belongs in the logging library and review process, not in an after-the-fact dashboard filter.

A compact Go event type makes the contract visible to every stage:

type PipelineEvent struct {
    RunID      string `json:"run_id"`
    Service    string `json:"service"`
    Stage      string `json:"stage"`
    Outcome    string `json:"outcome"`
    OccurredAt time.Time `json:"occurred_at"`
    DurationMs int64 `json:"duration_ms"`
    ErrorCode  string `json:"error_code,omitempty"`
}
Enter fullscreen mode Exit fullscreen mode

The event is intentionally boring. That is a feature. A bounded error_code such as dependency_timeout or invalid_input is more useful for aggregation than an exception string containing a tenant name or a variable request body. Keep the full diagnostic detail in a protected trace or incident attachment with an explicit retention policy.

Which failure modes deserve an alert?

Alert on a condition that can change an operator's decision. A failed edge probe should page only when it persists long enough to exceed the agreed availability window, or when it is corroborated by the workflow state. A growing queue, repeated dependency timeouts, and missed publication deadlines may be warning signals before the customer-facing SLO is violated; they should not all create independent pages for the same incident.

For incident reconstruction, preserve three timestamps: when the scheduler intended to start, when each stage began and ended, and when verification observed the result. The difference between those timestamps separates queue delay from execution delay and execution delay from publication delay. This is where capacity planning becomes concrete: if queue delay grows while worker duration is stable, adding workers may not address the bottleneck; if worker duration grows with input volume, the run-time budget and concurrency limit need review. The practical trap is to aggregate by service before preserving the run identifier: that produces a convincing chart of worker errors while hiding that the scheduler rejected a subset of runs earlier, and it can make a retry look like a second outage. Keep the raw transition sequence long enough to reconstruct the reporting window, then derive service-level counters from it. That ordering costs storage and query discipline, but it prevents a convenient summary from becoming the only evidence available during review.

A small correlation function can make the monitor's decision deterministic:

func completedBeforeDeadline(events []PipelineEvent, deadline time.Time) bool {
    verified := false
    for _, event := range events {
        if event.Stage == "verified" && event.Outcome == "success" && !event.OccurredAt.After(deadline) {
            verified = true
        }
    }
    return verified
}
Enter fullscreen mode Exit fullscreen mode

This does not replace a metrics backend or tracing system. It defines the business assertion that those systems must make queryable. A monitor that cannot distinguish a late successful run from a failed run will produce noisy pages and weak post-incident evidence.

Buy or build the reconstruction path?

The sensible boundary is usually mixed. Managed telemetry can reduce the on-call burden of storage, indexing, retention, and alert evaluation. Self-hosted components can preserve control over sensitive payloads, network placement, and query shape. Neither choice removes the need to define event contracts, sampling rules, ownership, and an SLO.

Concern Managed service Self-hosted path
Storage and upgrades Less platform work; retention and export limits must be reviewed More control; capacity, upgrades, and disaster recovery remain yours
Incident queries Fast to start; query semantics and portability vary Tailored joins and retention; the team owns performance tuning
Sensitive data boundary Requires a clear data-classification and redaction policy before export Keeps placement under team control, while access security is still required
On-call load Fewer infrastructure failure modes More operational ownership, including saturation and recovery testing

The decision should follow the incident evidence. If the team cannot answer which stage missed the deadline because identifiers were absent, buying another dashboard will not fix the gap. If the identifiers exist but queries routinely time out during an incident, storage layout and retention are the next engineering problem.

This method has limitations. It is not suitable as the only signal for tenant-specific authorization, regional behavior, or high-volume backfills. A synthetic workflow cannot represent every tenant's permissions, data volume, or regional dependency, and a structured event stream cannot prove that an uninstrumented side effect succeeded. The trade-off is deliberate: use it for the questions its evidence can answer, then pair it with service-specific tests where the risk justifies the extra operational load.

How do we verify and roll back the monitor?

Test the monitor against recorded timelines before paging anyone. Include a clean run, a scheduler rejection, a read timeout followed by retry success, a duplicate run_id, a late publication, and a clock-skew case. Assert the alert state and the evidence returned to the operator. A green dashboard is insufficient if the incident view cannot show the first failed transition.

Deploy alert-rule changes gradually. Run the new rule in observation mode, compare its classifications with existing incident records, then enable notification for one workflow or schedule. Keep the previous rule available until the new rule has covered a representative reporting window. Rollback should mean disabling the new evaluation and retaining emitted events, not deleting the evidence that explains why the change was made.

The final check is operational: can an engineer identify the failing stage, its dependency, the customer-visible consequence, and the next safe action without opening raw customer data? Yes. Then the monitor is doing more than reporting downtime. It is preserving a defensible reconstruction of the failure while the system is still changing.

The alert is not the incident.

References

Top comments (0)