When it comes to data systems, there are three foundational things that I believe are important to look into: reliability, scalability, and maintainability. In this article, I want to take a deeper look at one of these concepts: reliability.
To some extent, everyone has an intuitive idea of what it means for a software system to be reliable. We expect the system to perform the function it was designed to perform, tolerate users making mistakes or using it in unexpected ways, provide acceptable performance for its intended use case, and prevent unauthorized access. If all of these things can be considered part of working correctly, then we can understand reliability as the ability of a system to continue working correctly even when things go wrong.
The things that can go wrong in a system are generally referred to as faults. A system that is designed to continue operating despite certain faults is known as a fault-tolerant system. However, the term fault tolerance can sometimes be misleading because it may suggest that a system can be made tolerant of every possible type of fault. In reality, this is not feasible. It makes more sense to identify the specific types of faults that a system is expected to tolerate and design around them.
A fault can be understood as when one component of a system deviates from its expected specification, while a failure occurs when the system as a whole stops providing the required service to the user. For example, a disk developing a hardware problem is a fault in one component. If that fault eventually causes the application to become unavailable or data to become inaccessible, the user experiences a failure.
It is usually impossible to reduce the probability of faults to zero. Hardware will eventually fail, software will contain bugs, networks will become unavailable, and humans will make mistakes. Therefore, rather than assuming that faults will never occur, reliable systems should be designed with the assumption that certain faults will occur. The goal is to ensure that an individual fault does not necessarily become a system-wide failure.
Deliberately Triggering Faults
One approach that makes a lot of sense when building reliable systems is to deliberately introduce faults and observe how the system responds.
A well-known example of this approach is Netflix's Chaos Monkey. Netflix built a tool that deliberately crashes or terminates servers to test how their systems respond to unexpected failures. Instead of waiting for a real server failure to happen, engineers intentionally create the failure and observe whether the system can recover without interrupting the user's experience.
The purpose is not simply to crash servers. The larger goal is to force engineers to build systems that can recover automatically from failures. If a server suddenly disappears, the system should ideally be able to detect the problem, redirect work to another instance, and continue operating.
This leads to an important idea about reliability: the goal is not necessarily to prevent every failure from happening. Instead, the goal is to limit the impact of failures and allow the system to recover from them.
Hardware Faults
Another major cause of system failures is hardware failure. Hardware components such as disks, RAM, power supplies, and entire machines can fail.
One common response to hardware faults is to introduce redundancy. For example, RAID, which stands for Redundant Array of Independent Disks, can be used to provide redundancy at the disk level. Instead of depending on a single disk, data can be distributed or duplicated across multiple disks so that the failure of one disk does not necessarily cause the entire system to stop working.
This approach cannot completely prevent hardware problems from causing failures, but redundancy can significantly reduce the impact of an individual component failing. It is possible for a machine with redundant hardware to continue operating even when one of its components has failed.
However, as the amount of data and the number of machines in a system increase, relying only on hardware redundancy becomes insufficient. Entire machines can fail, not just individual components. This is one of the reasons modern data systems increasingly rely on software-based fault-tolerance techniques.
These techniques include redundancy, checkpointing, error detection and correction, replication, and automatic recovery. Instead of assuming that a particular machine will always be available, the software is designed so that the system can continue operating when a machine becomes unavailable.
This is an important shift in thinking: rather than trying to make every individual component perfectly reliable, we can build a system that remains reliable even when some of its components fail.
Software Errors
Hardware is not the only source of faults. Software itself can introduce serious problems.
For example, an application server may contain a bug that causes it to crash whenever it receives a particular type of unexpected input. If the same bug exists in every instance of that application server, then simply having multiple servers may not solve the problem. Every server could potentially fail when it receives the same input.
Software faults can also result in cascading failures, where a small problem in one component triggers problems in another component, which then triggers further problems throughout the system.
For example, imagine that one service becomes slow. Another service depends on it and begins waiting for responses. As requests continue to accumulate, resources such as memory, CPU, or connection pools become exhausted. Eventually, the second service may also fail, causing other services that depend on it to fail as well.
This shows why reliability is not simply about making individual components work correctly. We also have to think about how components interact with each other and what happens when one of them behaves unexpectedly.
There is no single or quick solution to software faults. Instead, reliable systems require a combination of techniques and careful thinking about the assumptions and interactions within the system.
Some of these techniques include thorough testing, process isolation, allowing processes to crash and restart, measuring system behavior, monitoring production systems, and analyzing failures when they occur.
For example, allowing a process to crash may seem counterintuitive when the goal is reliability. However, if a process becomes corrupted or enters an unhealthy state, restarting it may be safer than allowing it to continue running incorrectly. The important thing is to design the surrounding system so that the failure and restart of one process does not bring down the entire system.
Monitoring is also important because a system cannot respond effectively to failures that it cannot detect. By continuously measuring system behavior, engineers can identify unusual patterns, investigate problems, and understand how the system behaves under real-world conditions.
Human Error
Another important source of system failures is human error.
Humans build systems, deploy software, configure infrastructure, respond to incidents, and maintain production environments. Even when people have the best intentions, mistakes can happen.
Therefore, reliability should not only account for hardware and software faults; it should also account for the possibility that humans will make mistakes.
One way to address this is to design systems in a way that minimizes opportunities for error. If a potentially dangerous operation can be performed with a single command or click, we should consider whether additional safeguards should be introduced.
Another approach is to decouple the places where people are most likely to make mistakes from the places where those mistakes can cause failures. A mistake made by a developer should ideally be caught through code review, automated testing, staging environments, or deployment safeguards before it reaches production.
Finally, systems should be tested thoroughly at different levels. Testing can help identify problems before they reach users and can also help engineers understand how the system behaves when something goes wrong.
The goal is not to assume that humans will never make mistakes. Instead, we should design systems with the understanding that mistakes will happen and ensure that those mistakes have limited consequences.
Building Reliable Systems from Unreliable Parts
When we put all of these ideas together, we can see that reliability is not about creating a system in which nothing ever goes wrong. That is unrealistic.
Disks will fail. Servers will crash. Software will contain bugs. Networks will become unavailable. Users will provide unexpected input. Engineers and administrators will make mistakes.
The more realistic approach is to anticipate these problems and design the system to tolerate them.
This can involve redundancy at the hardware level, replication at the software level, checkpointing, error detection and correction, process isolation, automatic recovery, monitoring, testing, and deliberately introducing failures to verify that the system can recover.
The fundamental idea is that a reliable system should not depend on every individual component behaving perfectly all the time. Instead, the system should be designed so that the failure of one component does not necessarily cause the entire system to fail.
Reliability, therefore, is not simply about preventing failures. It is about expecting failures, limiting their impact, detecting them, and recovering from them.
This is what allows us to build reliable data systems from components that are, individually, not perfectly reliable.
Top comments (0)