Reading DDIA, Day 2: Human Errors, Scalability, and Why Latency Response Time

Day 2 of reading Designing Data-Intensive Applications (DDIA). Yesterday covered hardware faults and software errors. Today finishes reliability with the third kind of fault, human errors, and then starts on scalability: how to describe load, how to describe performance, and the difference between latency and response time.

📓 My original handwritten notes: Day 2 notes on GitHub

3. Human errors

Humans are known to be unreliable. So the question is:

How do we make our systems reliable, in spite of unreliable humans?

a) Design systems in a way that minimises opportunities for errors.
For example, well-designed abstractions, APIs, and admin interfaces make it easy to do "the right thing" and discourage "the wrong thing".

b) Decouple the places where people make the most mistakes from the places where they can cause failures.
In particular, provide fully featured non-production sandbox environments where people can explore and experiment safely, using real data, without affecting real users.

c) Test thoroughly at all levels, from unit tests to whole-system integration tests and manual tests.

d) Allow quick and easy recovery from human errors, to minimise the impact in case of failures.
For example: make it fast to roll back configuration changes, roll out new code gradually, and provide tools to recompute data.

e) Set up detailed and clear monitoring, such as performance metrics and error rates. In other engineering disciplines this is called telemetry.
Monitoring can show us early warning signals and let us check whether any assumptions or constraints are being violated. When a problem occurs, metrics can be invaluable in diagnosing the issue.

f) Implement good management practices and training.

Scalability

Scalability is the term we use to describe a system's ability to cope with increased load.

But it is not a one-dimensional label you can attach to a system. It is meaningless to say "X is scalable" or "Y doesn't scale".

Discussing scalability means asking questions like:

"If the system grows in a particular way, what are our options for coping with the growth?"
"How can we add computing resources to handle the additional load?"

Describing load

Load can be described with a few numbers called load parameters.

The best choice of parameters depends on the architecture of your system. It might be:

• Requests per second to a web server
• The ratio of reads to writes in a database
• The number of simultaneously active users in a chat room
• The hit rate on a cache

…and so on.

Describing performance

You can look at it in two ways:

1. When you increase the load and keep system resources unchanged (CPU, memory, network bandwidth, etc.), how is the performance of your system affected?
2. When you increase a load parameter, how much do you need to increase the resources to keep performance unchanged?

Both questions need performance numbers, so we need a way to describe the performance of a system:

• In a batch processing system such as Hadoop, we usually care about throughput: the number of records we can process per second, or the total time it takes to run a job on a dataset of a certain size.
• In online systems, service response time is critical: the time between a client sending a request and receiving a response.

Latency vs. response time

These two are often used synonymously, but they are not the same.

• Response time is what the client sees. Besides the actual time to process the request, it includes network delays and queueing delays.
• Latency is the duration that a request is waiting to be handled, during which it is latent, awaiting service.

Day 2 takeaway

You can't call a system "scalable" in general. Scalability only makes sense in terms of a specific load parameter, and you can only judge it by measuring performance as that load grows.

See you tomorrow for Day 3. 👋