All articles in this series: github.com/epicdeepak/designing-data-intensive-applications
Why response times vary
Even in a scenario where you'd think all requests should take the same time, you get variations. Some causes:
- Random additional latency introduced by a context switch to a background process
- The loss of a network packet and TCP retransmission
- A garbage collection pause
- A page fault forcing a read from disk
- Mechanical vibrations in the server rack
…and many more.
Why the slowest requests matter
Amazon describes response time requirements for internal services in terms of the 99.9th percentile, even though it only affects 1 in 1,000 requests.
Why? The customers with the slowest requests are often the ones with the most data in their accounts, because they've made many purchases. In other words, they're the most valuable customers. It's important to keep them happy by making sure the website is fast for them.
The numbers back this up:
- Amazon observed that a 100 ms increase in response time reduces sales by 1%.
- Others report that a 1-second slowdown reduces a customer satisfaction metric by 16%.
Percentiles in SLOs and SLAs
Percentiles are often used in Service Level Objectives (SLOs) and Service Level Agreements (SLAs), contracts that define the expected performance and availability of a service.
For example, an SLA may state that the service is considered up if it has:
- a median response time under 200 ms, and
- a 99th percentile under 1 s,
and the service may be required to be up 99.9% of the time.
These metrics set expectations for clients of the service and let customers demand a refund if the SLA isn't met.
Tail latency amplification
High percentiles become especially important in backend services that are called multiple times as part of serving a single end-user request.
Even if those calls are made in parallel, the end-user request still has to wait for the slowest of them to complete.
So even if only a small percentage of backend calls are slow, the chance of hitting a slow call goes up when one end-user request needs several backend calls. As a result, a larger share of end-user requests end up being slow.
Approaches for coping with load
- Vertical scaling (scaling up): moving to a more powerful machine.
- Horizontal scaling (scaling out): distributing the load across multiple smaller machines.
- Distributing load across multiple machines is also known as a shared-nothing architecture.
Maintainability
Most of the cost of software is not in its initial development, but in its ongoing maintenance:
- Fixing bugs
- Keeping its systems operational
- Investigating failures
- Adapting it to new platforms
- Modifying it for new use cases
- Repaying technical debt
…and so on.
Three design principles for software systems
- Operability: make it easy for the operations team to keep the software running.
- Simplicity: make it easy for new engineers to understand the system, by removing as much complexity as possible.
- Evolvability: make it easy for engineers to change the system in the future, adapting it for unanticipated use cases as requirements change. Also known as extensibility, modifiability, or plasticity.
Symptoms of complexity
- Explosion of the state space
- Tight coupling of modules
- Tangled dependencies
- Inconsistent naming and terminology
- Hacks aimed at solving performance problems
- Special-casing to work around issues elsewhere
Why complexity is dangerous
In complex software, there's also a greater risk of introducing bugs when making a change:
- The system is harder for developers to understand and reason about
- Hidden assumptions
- Unintended consequences
- Unexpected interactions are easily overlooked
Accidental complexity
Moseley and Marks define complexity as accidental if it is not inherent in the problem the software solves, but arises only from the implementation.
Day 3 takeaway
Your slowest requests often belong to your most valuable users, and when one request fans out to many backend calls, slowness multiplies. On the maintainability side, most of what software costs comes after launch. Complexity that comes only from how you built it, not from the problem itself, is the kind worth removing.
See you tomorrow for Day 4. 👋
Top comments (0)