DEV Community: Deepak Verma

Reading DDIA, Day 3: Tail Latency, Scaling Up vs. Out, and the Real Cost of Software

Deepak Verma — Sun, 18 Oct 2026 06:51:23 +0000

All articles in this series: github.com/epicdeepak/designing-data-intensive-applications

Why response times vary

Even in a scenario where you'd think all requests should take the same time, you get variations. Some causes:

Random additional latency introduced by a context switch to a background process
The loss of a network packet and TCP retransmission
A garbage collection pause
A page fault forcing a read from disk
Mechanical vibrations in the server rack

…and many more.

Why the slowest requests matter

Amazon describes response time requirements for internal services in terms of the 99.9th percentile, even though it only affects 1 in 1,000 requests.

Why? The customers with the slowest requests are often the ones with the most data in their accounts, because they've made many purchases. In other words, they're the most valuable customers. It's important to keep them happy by making sure the website is fast for them.

The numbers back this up:

Amazon observed that a 100 ms increase in response time reduces sales by 1%.
Others report that a 1-second slowdown reduces a customer satisfaction metric by 16%.

Percentiles in SLOs and SLAs

Percentiles are often used in Service Level Objectives (SLOs) and Service Level Agreements (SLAs), contracts that define the expected performance and availability of a service.

For example, an SLA may state that the service is considered up if it has:

a median response time under 200 ms, and
a 99th percentile under 1 s,

and the service may be required to be up 99.9% of the time.

These metrics set expectations for clients of the service and let customers demand a refund if the SLA isn't met.

Tail latency amplification

High percentiles become especially important in backend services that are called multiple times as part of serving a single end-user request.

Even if those calls are made in parallel, the end-user request still has to wait for the slowest of them to complete.

So even if only a small percentage of backend calls are slow, the chance of hitting a slow call goes up when one end-user request needs several backend calls. As a result, a larger share of end-user requests end up being slow.

Approaches for coping with load

Vertical scaling (scaling up): moving to a more powerful machine.
Horizontal scaling (scaling out): distributing the load across multiple smaller machines.
Distributing load across multiple machines is also known as a shared-nothing architecture.

Maintainability

Most of the cost of software is not in its initial development, but in its ongoing maintenance:

Fixing bugs
Keeping its systems operational
Investigating failures
Adapting it to new platforms
Modifying it for new use cases
Repaying technical debt

…and so on.

Three design principles for software systems

Operability: make it easy for the operations team to keep the software running.
Simplicity: make it easy for new engineers to understand the system, by removing as much complexity as possible.
Evolvability: make it easy for engineers to change the system in the future, adapting it for unanticipated use cases as requirements change. Also known as extensibility, modifiability, or plasticity.

Symptoms of complexity

Explosion of the state space
Tight coupling of modules
Tangled dependencies
Inconsistent naming and terminology
Hacks aimed at solving performance problems
Special-casing to work around issues elsewhere

Why complexity is dangerous

In complex software, there's also a greater risk of introducing bugs when making a change:

The system is harder for developers to understand and reason about
Hidden assumptions
Unintended consequences
Unexpected interactions are easily overlooked

Accidental complexity

Moseley and Marks define complexity as accidental if it is not inherent in the problem the software solves, but arises only from the implementation.

Day 3 takeaway

Your slowest requests often belong to your most valuable users, and when one request fans out to many backend calls, slowness multiplies. On the maintainability side, most of what software costs comes after launch. Complexity that comes only from how you built it, not from the problem itself, is the kind worth removing.

See you tomorrow for Day 4. 👋

Reading DDIA, Day 2: Human Errors, Scalability, and Why Latency Response Time

Deepak Verma — Thu, 15 Oct 2026 18:21:53 +0000

Day 2 of reading Designing Data-Intensive Applications (DDIA). Yesterday covered hardware faults and software errors. Today finishes reliability with the third kind of fault, human errors, and then starts on scalability: how to describe load, how to describe performance, and the difference between latency and response time.

📓 My original handwritten notes: Day 2 notes on GitHub

3. Human errors

Humans are known to be unreliable. So the question is:

How do we make our systems reliable, in spite of unreliable humans?

a) Design systems in a way that minimises opportunities for errors.
For example, well-designed abstractions, APIs, and admin interfaces make it easy to do "the right thing" and discourage "the wrong thing".

b) Decouple the places where people make the most mistakes from the places where they can cause failures.
In particular, provide fully featured non-production sandbox environments where people can explore and experiment safely, using real data, without affecting real users.

c) Test thoroughly at all levels, from unit tests to whole-system integration tests and manual tests.

d) Allow quick and easy recovery from human errors, to minimise the impact in case of failures.
For example: make it fast to roll back configuration changes, roll out new code gradually, and provide tools to recompute data.

e) Set up detailed and clear monitoring, such as performance metrics and error rates. In other engineering disciplines this is called telemetry.
Monitoring can show us early warning signals and let us check whether any assumptions or constraints are being violated. When a problem occurs, metrics can be invaluable in diagnosing the issue.

f) Implement good management practices and training.

Scalability

Scalability is the term we use to describe a system's ability to cope with increased load.

But it is not a one-dimensional label you can attach to a system. It is meaningless to say "X is scalable" or "Y doesn't scale".

Discussing scalability means asking questions like:

"If the system grows in a particular way, what are our options for coping with the growth?"
"How can we add computing resources to handle the additional load?"

Describing load

Load can be described with a few numbers called load parameters.

The best choice of parameters depends on the architecture of your system. It might be:

Requests per second to a web server
The ratio of reads to writes in a database
The number of simultaneously active users in a chat room
The hit rate on a cache

…and so on.

Describing performance

You can look at it in two ways:

When you increase the load and keep system resources unchanged (CPU, memory, network bandwidth, etc.), how is the performance of your system affected?
When you increase a load parameter, how much do you need to increase the resources to keep performance unchanged?

Both questions need performance numbers, so we need a way to describe the performance of a system:

In a batch processing system such as Hadoop, we usually care about throughput: the number of records we can process per second, or the total time it takes to run a job on a dataset of a certain size.
In online systems, service response time is critical: the time between a client sending a request and receiving a response.

Latency vs. response time

These two are often used synonymously, but they are not the same.

Response time is what the client sees. Besides the actual time to process the request, it includes network delays and queueing delays.
Latency is the duration that a request is waiting to be handled, during which it is latent, awaiting service.

Day 2 takeaway

You can't call a system "scalable" in general. Scalability only makes sense in terms of a specific load parameter, and you can only judge it by measuring performance as that load grows.

See you tomorrow for Day 3. 👋

Reading DDIA, Day 1: Data Systems, Reliability, and Why a Fault Isn't a Failure

Deepak Verma — Wed, 14 Oct 2026 02:48:40 +0000

I've started reading Designing Data-Intensive Applications (DDIA), and I'm going to write up what I learn each day as I go. My original handwritten notes go on GitHub, and this is the cleaned-up version.

Day 1 covers the building blocks of data systems, the questions to ask when you design one, and the first big topic: reliability.

The building blocks

Most data-intensive applications are built from a few standard pieces:

Databases: store data so that they, or another application, can find it again later.
Caches: remember the result of an expensive operation, to speed up reads.
Search indexes: let users search data by keyword or filter it in various ways.
Stream processing: send a message to another process, to be handled asynchronously.
Batch processing: periodically crunch a large amount of accumulated data.

The lines between these categories are blurry:

Some datastores are also used as message queues. Example: Redis.
Some message queues come with database-like durability guarantees. Example: Apache Kafka.

A few more tools that show up again and again:

Memcached: an application-managed caching layer.
Elasticsearch / Solr: full-text search servers.

Questions to ask when designing a data system or service

How do you make sure data stays correct and complete, even when things go wrong internally?
How do you keep performance consistently good for clients, even when parts of your system are degraded?
How do you scale to handle an increase in load?
What does a good API for the service look like?

The three concerns that matter most

Reliability
Scalability
Maintainability

Today was all about the first one.

Reliability

For software, reliability means:

The application performs the function the user expected.
It can tolerate the user making mistakes or using the software in unexpected ways.
Its performance is good enough for the required use case, under the expected load and data volume.
The system prevents unauthorised access and abuse.

In one line: it should work correctly, even when things go wrong.

Faults vs. failures (remember this one)

The things that can go wrong are called faults. Systems that anticipate faults and can cope with them are called fault-tolerant or resilient.
A fault is not the same as a failure.
- A fault is one component of the system deviating from its spec.
- A failure is when the system as a whole stops providing the required service to the user.

📌 On my reading list: Netflix's Chaos Monkey.

Types of faults

1. Hardware faults

A hard disk crashes
RAM becomes faulty
The power grid has a blackout
Someone unplugs the wrong network cable

Related term: MTTF (Mean Time To Failure).

2. Software errors

A software bug that causes every instance of an application server to crash when given a particular bad input.
- Example: the leap second on June 30, 2012 made many applications hang at the same time, because of a bug in the Linux kernel.
A runaway process that uses up a shared resource: CPU time, memory, disk space, or network bandwidth.
A service the system depends on slows down, becomes unresponsive, or starts returning corrupted responses.
Cascading failures, where a small fault in one component triggers a fault in another component, which then triggers more faults.

Day 1 takeaway

A reliable system isn't one where nothing goes wrong. It's one where faults happen and the system as a whole keeps working anyway.

📓 Original handwritten notes: Github link

See you tomorrow for Day 2. 👋