I've started reading Designing Data-Intensive Applications (DDIA), and I'm going to write up what I learn each day as I go. My original handwritten notes go on GitHub, and this is the cleaned-up version.
Day 1 covers the building blocks of data systems, the questions to ask when you design one, and the first big topic: reliability.
The building blocks
Most data-intensive applications are built from a few standard pieces:
- Databases: store data so that they, or another application, can find it again later.
- Caches: remember the result of an expensive operation, to speed up reads.
- Search indexes: let users search data by keyword or filter it in various ways.
- Stream processing: send a message to another process, to be handled asynchronously.
- Batch processing: periodically crunch a large amount of accumulated data.
The lines between these categories are blurry:
- Some datastores are also used as message queues. Example: Redis.
- Some message queues come with database-like durability guarantees. Example: Apache Kafka.
A few more tools that show up again and again:
- Memcached: an application-managed caching layer.
- Elasticsearch / Solr: full-text search servers.
Questions to ask when designing a data system or service
- How do you make sure data stays correct and complete, even when things go wrong internally?
- How do you keep performance consistently good for clients, even when parts of your system are degraded?
- How do you scale to handle an increase in load?
- What does a good API for the service look like?
The three concerns that matter most
- Reliability
- Scalability
- Maintainability
Today was all about the first one.
Reliability
For software, reliability means:
- The application performs the function the user expected.
- It can tolerate the user making mistakes or using the software in unexpected ways.
- Its performance is good enough for the required use case, under the expected load and data volume.
- The system prevents unauthorised access and abuse.
In one line: it should work correctly, even when things go wrong.
Faults vs. failures (remember this one)
- The things that can go wrong are called faults. Systems that anticipate faults and can cope with them are called fault-tolerant or resilient.
-
A fault is not the same as a failure.
- A fault is one component of the system deviating from its spec.
- A failure is when the system as a whole stops providing the required service to the user.
📌 On my reading list: Netflix's Chaos Monkey.
Types of faults
1. Hardware faults
- A hard disk crashes
- RAM becomes faulty
- The power grid has a blackout
- Someone unplugs the wrong network cable
Related term: MTTF (Mean Time To Failure).
2. Software errors
- A software bug that causes every instance of an application server to crash when given a particular bad input.
- Example: the leap second on June 30, 2012 made many applications hang at the same time, because of a bug in the Linux kernel.
- A runaway process that uses up a shared resource: CPU time, memory, disk space, or network bandwidth.
- A service the system depends on slows down, becomes unresponsive, or starts returning corrupted responses.
- Cascading failures, where a small fault in one component triggers a fault in another component, which then triggers more faults.
Day 1 takeaway
A reliable system isn't one where nothing goes wrong. It's one where faults happen and the system as a whole keeps working anyway.
📓 Original handwritten notes: Github link
See you tomorrow for Day 2. 👋
Top comments (0)