How to Build Fault-Tolerant Systems: 5 Engineering Lessons for Resilient Software

WiWo — Tue, 12 Aug 2025 03:04:32 +0000

My father, a turner by profession, pursued excellence and quality throughout his career. His motto, "No errors, no defects in the product," inspired me to adopt the same standard in software development. However, as much as I strived for perfection, bugs and issues persisted, unearthed by QA engineers or end-users. Each failure felt personal, challenging my commitment to excellence. Over time, as systems grew more complex, I realized the inevitability of errors. This revelation, reinforced by lessons from the aviation industry, transformed my approach to system design.
In the 1960s and early 1970s, aircraft design focused on creating fail-free systems. However, as systems became increasingly complex, this approach introduced new points of failure, making maintenance more challenging. Following high-profile incidents, such as the Turkish Airlines Flight 981 crash in 1974, the industry shifted its focus to fault tolerance. Instead of striving for perfection, engineers have come to embrace the reality of failure and prioritize resilience.
This paradigm shift was eye-opening. If the aircraft industry could accept failure as inevitable, why couldn't we in software development? Here, I share some of my key principles for building robust, fault-tolerant systems that can continue functioning in the face of failures.

Key Principles for Fault-Tolerant Software Design

1. Quality

Nothing improves software quality more than rigorous testing. Testing must be done correctly, and adopting the Test-Driven Development (TDD) approach is vital. By writing tests before implementation, we ensure they align with requirements rather than the code itself.

2. Fault Tolerance

Retry Patterns with Exponential Backoff: For transient failures, such as temporary network outages, retrying with increasing delays prevents overwhelming the system. For example, an API call might retry after 2 seconds, then 4 seconds, and so on. However, an idempotent design is crucial to prevent unintended consequences, such as duplicate transactions in payment systems.
Circuit Breakers: When a service consistently fails, a circuit breaker halts and retries until the service is restored. This prevents cascading failures. A common example is a load balancer that monitors server health checks and reroutes traffic to healthy nodes when a server is down.
Container Orchestrators: Tools like Kubernetes can automatically restart failed services, maintaining availability and reducing downtime.
Disaster Recovery: Backup solutions and database replication are critical for recovering from catastrophic failures. For instance, maintaining a geographically distributed backup site can minimize downtime during a regional outage

3. Monitoring and Logging

A system is only as good as its observability. Logs and monitoring tools provide critical insights into system activity.
Logs: Writing clear, concise, and structured logs is essential. For example, correlation IDs allow tracing a single user transaction across multiple services, enabling developers to identify the root cause of failures. Tools like ElasticSearch and Kibana, Splunk make it easier to analyze logs and spot anomalies, such as unexpected spikes in error rates.
Real-Time Monitoring: Platforms like AWS CloudWatch or Google Cloud Observability can monitor system metrics, such as CPU usage and queue lengths, to detect and respond to potential issues in real-time. Auto-scaling mechanisms, such as AWS Auto Scaling Groups, allow systems to adjust resources based on demand dynamically

4. Fail Fast

Imagine heading out to buy lunch, only to discover that you forgot your wallet at the restaurant. The task was destined to fail from the start. Similarly, in software, inputs and preconditions must be validated early to avoid unnecessary resource usage.
For example, an order processing system should validate inventory availability before payment. Failing fast saves time and helps developers quickly pinpoint issues

Conclusion

Building fault-tolerant systems requires striking a balance between robust design, proactive error handling, and continuous monitoring. By embracing failure as inevitable and focusing on resilience, we can create software that not only survives but thrives in the face of adversity. Principles such as " fail fast," retry patterns, and real-time monitoring ensure systems remain reliable, even under pressure.
Let's strive for excellence not by avoiding failure altogether but by building systems prepared to recover gracefully when failure inevitably occurs

HTML popover

WiWo — Fri, 03 May 2024 04:16:51 +0000

Introduction

You know what popover (popup) is, so feel free to skip this section. A popover or popup is a usually small window, which can appear in any place on the page, to provide the user with some information. Unlike the dialog, popover cannot be modal, it is always modeless, so usually the information, provided by the popup is less critical, than the one supplied by the dialog, and does not require user interaction.
There are two ways to display popover(popup) using native HTML: declarative and JavaScript.

Declarative

Step 1
Declare a popover markup(please pay attention to the ID of the "div", "my_popover"):

 <div id="my_popover" popover>
    <h2>An HTML PopOver</h2>
    <button popovertargetaction="hide">Close</button>
 </div>

Step 2
Add a button with the attribute "popovertarget", as the value we will provide an ID of the "div" element with the "popover" attribute:

<button popovertarget="my_popover" popovertargetaction="show">
  Show popup
</button>

The "popovertarget", "popovertargetaction" and "popover"
The "popovertarget" binds between the button, which applies actions on the popover (like showing, hiding, etc.), and the "div", which will be used as a popover.
The "popovertargetaction" declares the action should be applied on the popover as an "onclick" event of the button. The "popovertargetaction" may have the following values:
"show" - will show the popover
"hide" - will hide the popover
"toggle" - will toggle between the states "shown" and "hidden"

The "popover" attribute defines the behavior mode of the popover. There are two modes :
"auto" - the default value of the "popover" (when no value was provided) attribute and instructs the browser to hide the popup once a "click" event occurs outside of the popup ("popover")
"manual" - instructs the browser to hide the popup only if the dedicated action closes the popover ("hide" or "toggle"). Important: the popover, which has a "manual" behavior can coexist on the screen with other popovers, meaning it will not be closed automatically, by showing another popover.

Showing and closing popover using JavaScript

Step 1
Let's declare three functions: "show" - to show the popover, "close" - to hide the popover, and "toggle" - toggle the popover. As a parameter, all the functions will accept the id of the popover:

const show = (id) => {
     const popover = document.getElementById(id);
     popover.showPopover && popover.showPopover();
}

const close = (id) => {
     const popover = document.getElementById(id);
     popover.hidePopover && popover.hidePopover();
}

const toggle = (id) =>{
     const popover = document.getElementById(id);
     const popoverOpen = popover.togglePopover && popover.togglePopover();
}

Step 2
Let's add the popover div with the id "my_popover". The popover behavior will be declared as manual to demonstrate the toggling, otherwise, the popover will be hidden automatically once there is a click event anywhere outside the popover area

<div id="my_popover" popover="manual">
       <h2>An HTML PopOver 1</h2>
       <button autofocus>Send</button>
       <button  onclick="close('my_popover')">Close</button> 
 </div>

Step 3
Let's add two buttons: to show and toggle the popover. The close button appears in the popover itself
Show
Toggle

Styling

Positioning
Popover can be shown at every location on the screen. The "popover-open" pseudo-class can be used to specify the popover position and size. The next CSS snapshot assumes the popover has id "my_popover" from the examples above

 #my_popover:popover-open {
        width: 300px;
        height: 150px;
        position: absolute;
        bottom: 5px;
        right: 5px;
        margin: 0;
 }

The popover will be shown with a height of 150px and a width of 300px will be displayed in the bottom right corner of the browser

Backdrop
The "::backdrop" pseudo-element, as in the case of the "dialog" element, is a full-screen element placed behind the popover. In the following example, we will define a "::backdrop" element that will color the content behind the popover in an ugly purple color. Just as in the example above, we will assume, that the popover has the id "my_popover"

#my_popover::backdrop {
         background-color: #673ab752;
 }

Summary

Popover has been supported in all browsers since April 2024. It can be used in a declarative way and/or using JavaScript. Adding the "popover" attribute to the "div" element converts it to popover/popup. There are two modes popover can appear: "auto" - when it disappears once a click happens anywhere outside the popover, and "manual" - when the popover should be closed explicitly. The popover exposes an interface with three methods: "showPopover", "hidePopover" and "togglePopover"

DEV Community: WiWo