DEV Community

Cover image for Reliability isn't something you claim. It's something you show
Yuriy for Adal

Posted on

Reliability isn't something you claim. It's something you show

Every online service claims to be reliable. Far fewer publish enough data for anyone to verify that claim.

“Reliable” appears on so many SaaS landing pages that the word has almost lost its meaning. It is usually backed by familiar promises: resilient infrastructure, continuous monitoring, high availability. Users are still expected to take the company's word for it.

But no system runs without failures forever. Bugs reach production, data centers lose connectivity, databases fail over, and bad deployments have to be rolled back. Even if every component is well designed, external dependencies and human error remain part of the system.

Trust, then, should not begin with a promise that incidents will never happen. It should begin with a willingness to measure them—and to show the results.

What do we actually mean by reliability?

“The service was available 99.9% of the time” sounds precise. Without a measurement model, it still tells us very little.

What counts as available? A successful response from the home page? The ability to sign in? The completion of the product's core operation? Is the service available if it responds, but takes 30 seconds? What if only one region is affected? Are maintenance windows excluded? Over what period is the number calculated?

Site Reliability Engineering uses three related concepts to make those questions explicit:

  • SLI, or Service Level Indicator: a measured property of the service, such as the percentage of requests completed successfully or within a latency threshold.
  • SLO, or Service Level Objective: the target for that indicator, such as at least 99.9% of requests completing successfully over a rolling 30-day period.
  • SLA, or Service Level Agreement: a customer agreement that defines the commitment, measurement period, calculation method, exclusions, and consequences if the promised level is missed.

The distinction matters. An SLA is not just an impressive percentage, and it is not the probability that a server will randomly go down. It is a contractual promise whose meaning depends on both the measurement and the terms around it. Google's Site Reliability Engineering book offers a useful explanation of these concepts.

Why a few extra nines make such a big difference

One practical way to think about availability is through an error budget: the amount of time, or the share of requests, a system can lose before it misses its objective.

If we simplify the calculation to time-based availability, with no exclusions and a 365-day year, the numbers look like this:

Availability target Allowed downtime per 30 days Allowed downtime per year
99% 7 hours 12 minutes 3 days 15 hours 36 minutes
99.9% 43 minutes 12 seconds 8 hours 45 minutes 36 seconds
99.99% about 4 minutes 19 seconds about 52 minutes 34 seconds

The difference between 99% and 99.99% is only 0.99 percentage points. Yet the downtime budget becomes 100 times smaller: from 5,256 minutes to 52.56 minutes per year.

Those extra nines require more than larger servers. They require redundancy, safe deployment practices, observability, tested recovery procedures, on-call coverage, and a disciplined incident-review process. That is why higher availability usually costs significantly more.

The percentage alone does not describe the impact, either. Five minutes of downtime in the middle of the night and five minutes during a major ticket sale look identical in an availability report, but they are not remotely equivalent for users. In a webhook system, even a brief interruption can coincide with a payment notification, an order update, or another request that cannot simply disappear.

Monitoring feels optional—until the first incident

When an application is working, monitoring can feel like secondary work. No one is complaining, the server responds, and the team has a backlog full of features with more visible product value. It is tempting to assume that, if something goes wrong, the logs will be enough.

Imagine that a payment provider sends a webhook, but the receiving application never processes it. By the time someone investigates, every component is healthy again. Without measurements collected in advance, the team cannot tell whether the request was accepted, persisted, delayed in a queue, lost during delivery, or rejected by the destination itself. Individual services may contain fragments of the answer, but those fragments still have to be correlated by timestamp and request.

Monitoring tells you when a known condition has gone wrong: error rates have increased, a component is unavailable, or latency has crossed a threshold. Observability helps you reconstruct what happened using metrics, logs, and traces.

Their value becomes obvious during the first real incident. By then, however, it is too late to collect the data you did not record. Observability has to exist before you need it.

A green status indicator is not enough

A public status page answers a simple and important question: is the service working right now?

One global indicator is rarely sufficient. If the marketing site loads but the API cannot accept requests, users will not consider the service available. If a failure affects only one region, an overall green status hides information that matters to the people in that region.

A useful status page breaks the system into components that reflect real user journeys. Depending on the product, it might report the health of:

  • the public website and control panel;
  • the API or request-ingestion layer;
  • queue processing;
  • data storage;
  • delivery to external systems;
  • individual regions.

A short history window adds useful context. It lets users see not only what is happening this minute, but also whether there were recent disruptions. Still, 24 hours of history cannot demonstrate reliability over a month, a quarter, or a year. A green indicator now and a green chart for the previous day say nothing about the incidents that may have happened over the previous three months.

That requires an incident history: when each incident started and ended, which components and regions were affected, what users experienced, how recovery progressed, and what the team learned afterward. A cumulative availability figure is meaningful only when it includes both a time period and a published calculation method.

In other words, current status tells users what is happening now. A short history tells them what happened recently. Long-term metrics and incident records tell them how the service has performed over time.

Automate the measurement, not the accountability

If a team manually decides which failures belong in its statistics, it will eventually be tempted to omit a short or isolated event. A few seconds can seem insignificant, especially when the problem affected only a small number of users and disappeared quickly.

That is how trustworthy statistics gradually become marketing statistics. One inconvenient event is excluded, then another, until the published number no longer reflects the user experience.

The underlying data collection should therefore be automatic. Checks should run continuously, results should be stored whether or not they are flattering, and metrics should follow rules defined in advance.

This does not mean that every failed check should immediately be announced as a major incident. A single request can fail because of a local network route, DNS resolution, or a problem with the monitoring node itself. Automated monitoring still needs thresholds, retries, and—where appropriate—comparison across multiple locations.

It helps to separate two layers:

  1. Measurement. Automated checks and metrics record events without manual selection.
  2. Communication. People explain the scope, user impact, cause, recovery process, and changes intended to prevent a recurrence.

Automation reduces the opportunity to massage the numbers. It does not remove the team's responsibility to explain what happened clearly.

What an honest status page should include

Simply publishing a /status URL does not create transparency. Users need enough information to make a decision.

At a minimum, a useful status page should include:

  • the state of user-facing components and regions;
  • the time of the latest update;
  • active incidents and their user impact;
  • a history of resolved incidents;
  • clear distinctions between an outage, partial outage, performance degradation, and planned maintenance;
  • the method used to calculate availability;
  • the period covered by the published metric.

For a serious incident, a blameless postmortem is also valuable. It should describe the sequence of events, user impact, detection, recovery, and the concrete changes made to the system or process afterward.

Measurement boundaries matter as well. A technical health endpoint may prove that a process is running. It does not prove that a user can complete the product's core operation. The closer a check is to a real user journey, the more useful its result becomes.

How we started doing this at Adal

Adal receives and delivers webhooks, so availability is not an abstract quality for us. If the system cannot accept an incoming request, the sending service may receive an error. If delivery is disrupted, a request may remain pending or require another attempt. If the control panel is unavailable, users lose visibility into what is happening.

We published an Adal status page. It reports health separately for our regions in Almaty, Stockholm, and Nuremberg. Within each region, it distinguishes public services—the web interface, socket server, and request ingestion—from infrastructure components such as the regional database, queue, and storage connection.

The page shows more than the system's current state. For each region, it publishes hourly statistics for the previous 24 hours in UTC, from the earliest interval through the current one. The incomplete hour is marked separately because checks are still running and its final status may change.

The data comes from automated checks with a dynamic frequency. Under normal conditions, every service in every region is checked once every 30 seconds. When a check detects an error, monitoring for that service and region switches to once per second so that the system can confirm the incident and follow the recovery closely. After recovery is confirmed, the interval returns to 30 seconds.

The system keeps a history of detected errors, while the public page groups the results into hourly intervals that are easier to read. This means a brief failure does not disappear simply because the service is healthy again by the time someone opens the page.

The page also distinguishes between status reported by Adal's own infrastructure and an availability check performed from the user's browser. That distinction matters: local DNS, a VPN, firewall rules, CORS, or the network route can affect an external check even when the component itself is working.

At the same time, one failed check does not necessarily represent a user-facing incident. Our next step is to analyze the stored history, correlate failures across components and regions, determine their actual impact, and publish confirmed incidents. For serious cases, that communication should explain not only the duration, but also the cause, consequences, and measures taken after recovery.

We also do not treat the current status and a 24-hour chart as proof that we have already achieved a particular SLA. Evaluating availability over a month or a year requires observations covering that period, a history of confirmed incidents, and calculation rules published in advance. It would be strange to write about transparency while presenting a short observation window as statistical evidence of long-term reliability.

An incident does not have to destroy trust

Most users understand that complex systems can fail. Uncertainty and silence are far more damaging: does the team know about the problem? Which features are affected? Is data safe? When will the next update arrive?

Honest incident communication does not make a service less reliable. It gives users enough information to assess the risk, take temporary measures, and see how the team responds when something goes wrong.

The most revealing moment for a status page is not when every indicator is green. Its real value appears during an incident: how quickly did the first update arrive, how accurately did it describe the impact, and did the team publish a useful review after recovery?

Reliability cannot be established by a sentence on a landing page. It has to be measured over time—and the results have to include the minutes no one is proud of.

Top comments (0)