Episode 1 established the mindset: failure is normal, not a sign of bad engineering. Episode 2 gets specific — you can't detect or handle a failure you can't even name.
Saturday, Round 2
👦 Nephew: Uncle, last time you convinced me failure is basically guaranteed. Fine, I accept it. So what actually fails?
👨🦳 Uncle: You tell me. Start listing things that could go wrong in your app right now.
👦 Nephew: Uh... the server could crash. The database could go down. My code could have a bug.
👨🦳 Uncle: Keep going.
👦 Nephew: The network? Someone could deploy the wrong thing? Payment gateway dies mid-checkout?
👨🦳 Uncle: You just named six of the seven categories without trying. You already know this. You've just never sorted it.
1. Hardware Failure
2. Software Failure
3. Network Failure
4. Database Failure
5. Third-Party Failure
6. Human Error
7. Resource Exhaustion
👦 Nephew: Then why do we need the list at all, if I already know it instinctively?
👨🦳 Uncle: Because "instinctively" isn't fast enough at 2 AM. Let's trace each one properly.
Part 1 — Hardware Failure
👦 Nephew: This one's obvious anyway — I deploy to AWS. The cloud hides hardware failure from me.
👨🦳 Uncle: Does it?
👦 Nephew: ...doesn't it? That's the whole point of paying for EC2 instead of buying a server.
👨🦳 Uncle: Let's trace it. Your app sits on an EC2 instance. What's underneath the instance?
👦 Nephew: Virtual machine stuff, I guess?
👨🦳 Uncle: And underneath that?
👦 Nephew: ...an actual physical machine somewhere. In a data center.
👨🦳 Uncle: There it is.
Your app
|
"Virtual" server (EC2/Droplet)
|
ACTUAL physical hardware somewhere in a data center
|
Still capable of failing — just less visible to you
👦 Nephew: So it's not hidden. It's just one layer further away than I thought.
👨🦳 Uncle: Exactly. AWS absorbs a lot of it — that's part of what you're paying for — but disks still fail, instances still get abruptly terminated, whole availability zones still go down. That's Hardware Failure.
Hardware Failure Examples
--------------------------
- SSD/HDD fails, data on that disk is unreadable
- RAM stick develops a fault, random crashes
- A physical server loses power
- A whole data center rack fails
👦 Nephew: This is the Chaos Monkey thing from last time.
👨🦳 Uncle: That's exactly it. Chaos Monkey exists because Netflix can't opt out of hardware failing. They just refused to be surprised by it.
Part 2 — Software Failure
👨🦳 Uncle: Look at this and tell me what's wrong.
app.get('/user/:id', async (req, res) => {
const user = await db.findUser(req.params.id);
res.json(user.profile);
});
👦 Nephew: If user is null, user.profile crashes.
👨🦳 Uncle: Right. What category is that?
👦 Nephew: Easy — software failure. My bug.
👨🦳 Uncle: Now — remember the memory leak we covered in the Node Internals series? Something holding a reference so GC can't collect it, and the process slowly OOMs?
👦 Nephew: Yeah.
👨🦳 Uncle: Same category. So is a null check crash the same severity as a slow memory leak that kills your process three days later?
👦 Nephew: ...no. Not even close.
👨🦳 Uncle: Right. Same label. Wildly different scale. A missing null check takes down one request, maybe your process if nobody caught it. A leak takes down the whole thing, just slower. Worth remembering when you're deciding how much time a fix actually deserves.
Software Failure Examples
---------------------------
- Null/undefined reference crashes a function
- Infinite loop consumes 100% CPU
- A dependency's new version silently breaks your code
- Unhandled promise rejection kills the process
- Memory leak grows until the process OOMs
Part 3 — Network Failure
👦 Nephew: My request timed out yesterday, on the Pilooopu admin panel. Was that the server being down?
👨🦳 Uncle: Was it?
👦 Nephew: I mean... I assumed so. It just hung.
👨🦳 Uncle: Let's trace what "hung" actually means. Client sends a request. What has to happen before the server even sees it?
👦 Nephew: DNS resolves the domain, then... it goes through routers, I guess, to reach the server.
👨🦳 Uncle: And if one of those routers drops the packet?
👦 Nephew: The server never even gets the request.
👨🦳 Uncle: So from where you're sitting —
👦 Nephew: — it looks exactly the same as the server being down. A timeout either way.
Client Network Server
| | |
| ---- request ----------->| |
| X packet dropped here |
| | |
| (client sees a timeout, server never even saw the request)
👨🦳 Uncle: That's Network Failure — and that's exactly why it's miserable to debug. Same symptom, four possible systems responsible, and none of them are the one you're staring at.
Network Failure Examples
--------------------------
- DNS lookup fails or times out
- Packet loss between your server and the database
- A load balancer misroutes or drops connections
- TLS handshake fails
- Latency spikes without an outright failure
Part 4 — Database Failure
👦 Nephew: Speaking of hanging — remember my app just froze under load a few months back? No errors, nothing in the logs?
👨🦳 Uncle: I remember. What did you eventually find?
👦 Nephew: Connection pool, I think. But I never really confirmed why.
👨🦳 Uncle: Let's confirm it now. Pool size ten. Ten requests come in and hold their connections a bit too long. Eleventh request arrives —
👦 Nephew: — no connection left for it. It just waits.
Connection Pool (size: 10)
+----------------------------------+
| [used][used][used]...[used] | ← all 10 in use
+----------------------------------+
|
Request #11 arrives
|
No free connection available
|
Request waits... and waits... eventually times out
👦 Nephew: So that means MySQL actually crashed, right? That's what caused it?
👨🦳 Uncle: No — that's exactly the mistake everyone makes. MySQL was probably sitting there completely healthy the entire time.
👦 Nephew: Then what was actually broken?
👨🦳 Uncle: Your app. Holding connections too long, not releasing them properly. Database Failure, but the database itself is often innocent.
Database Failure Examples
----------------------------
- Connection pool exhausted — no free connections left
- A slow query locks a table, backing up every other query
- Replication lag — replica serves stale data
- Disk fills up, writes start failing
- Primary node crashes, no failover configured
👦 Nephew: So the database's failure was never the database's fault.
👨🦳 Uncle: Almost never is.
Part 5 — Third-Party Failure
👦 Nephew: Here's a theory. If I didn't write the code — Stripe, SendGrid, whatever — it's not really my problem when it breaks. That's on them.
👨🦳 Uncle: Is it?
👦 Nephew: ...I feel like you're about to ruin this theory.
👨🦳 Uncle: Let's trace it. Razorpay goes down. Your server calls Razorpay and waits. What does "waits" mean for the requests behind it?
👦 Nephew: They queue up behind the one that's stuck.
👨🦳 Uncle: And if Razorpay takes ninety seconds to time out?
👦 Nephew: Then every user behind that one call is stuck for ninety seconds too. Even the ones who have nothing to do with payment.
Your Server ---calls---> Third-Party API
|
Third party goes down
|
Your server, if not careful, hangs waiting on that call too
|
Your OWN healthy users start getting stuck behind it
👦 Nephew: So their outage becomes my outage. For free.
👨🦳 Uncle: That's Third-Party Failure. You genuinely can't fix their five minutes of downtime — but you control whether it becomes your five minutes too. That's the entire reason Circuit Breaker exists, which we'll get to in Module 3.
Third-Party Failure Examples
--------------------------------
- Payment gateway (Stripe/Razorpay) times out mid-transaction
- Email service (SendGrid) rate-limits or goes down
- A third-party auth provider (Google/Facebook login) is slow
- An external weather/maps/data API changes its response format
👦 Nephew: Great. So now even my database has trust issues, and so does everything outside it.
Part 6 — Human Error
👦 Nephew: This category feels unfair to even call a "type of failure." It's not the system. It's just someone screwing up.
👨🦳 Uncle: Is it, though? Say someone on your team pushes a bad env variable to production by accident. Who's at fault?
👦 Nephew: ...the person who pushed it?
👨🦳 Uncle: Or — was there no review step that would've caught it?
👦 Nephew: ...there wasn't, actually.
👨🦳 Uncle: Then who's really at fault — the person having a bad Friday at 5 PM, or the process that let one bad Friday reach production?
Junior framing
↓
Someone made a mistake, fire them.
Senior framing
↓
The SYSTEM allowed one honest mistake
to cause this much damage. Fix the system.
👦 Nephew: So the fix isn't "be more careful."
👨🦳 Uncle: "Be more careful" isn't a fix. It's a hope. The real fix is code review, staging environments, a confirmation step before anything destructive.
Human Error Examples
------------------------
- Wrong environment variable in production
- Accidentally deployed to the wrong environment
- A bad migration deletes/corrupts data
- Force-pushed over someone else's changes
- Misconfigured a firewall rule, blocked legitimate traffic
Part 7 — Resource Exhaustion
👦 Nephew: Isn't this basically the event loop thing from Node Internals? Getting overwhelmed under a traffic burst?
👨🦳 Uncle: Close. Let's actually trace where it breaks, because "overwhelmed" is doing a lot of work in that sentence. Traffic climbs. What runs out first?
👦 Nephew: Memory? CPU?
👨🦳 Uncle: Could be either, or disk, or file descriptors, or even a rate limit on an API you call. Point is — it's rarely instant.
Normal: memory/CPU usage stays flat
Rising load: usage climbs steadily
Exhaustion: usage hits the ceiling → OS kills the process,
or the event loop can't keep up with new requests
👦 Nephew: So this is the category where success is what kills you. Too many users show up, and that's the failure.
👨🦳 Uncle: That's Resource Exhaustion, and exactly why it's last on this list — and honestly the one worth planning for earliest, the moment your product starts actually growing.
Resource Exhaustion Examples
--------------------------------
- Out of memory (OOM) — process gets killed by the OS
- CPU pegged at 100%, event loop can't keep up
- Disk full — logs or temp files fill the disk
- Too many open file descriptors / socket connections
- Rate limit hit on an external API you depend on
Part 8 — Putting the Seven Together
👨🦳 Uncle: Try something. Razorpay goes slow on you — third party failure. Walk me through what happens next, using what you now know.
👦 Nephew: Okay... my requests to Razorpay start piling up. That eats into my connection pool, so — database failure?
👨🦳 Uncle: Keep going.
👦 Nephew: All those pending requests are sitting in memory, waiting. So memory climbs. Resource exhaustion.
👨🦳 Uncle: And then?
👦 Nephew: ...the process just dies. Software failure, technically, since the crash itself is code failing to handle it.
Example chain:
Third-party payment API is slow (Third-Party Failure)
|
Your requests to it start piling up
|
Connection pool gets exhausted (Database Failure)
|
Memory usage climbs holding pending requests (Resource Exhaustion)
|
Process crashes (Software Failure)
👨🦳 Uncle: You just traced a real incident shape without me saying a word. One small external failure, left unhandled, cascades through your own system until it takes the whole thing down. Timeout, Circuit Breaker, Bulkhead — Module 3 — each one exists to break that chain at a different link.
👦 Nephew: Basically everything hates me.
👨🦳 Uncle: Welcome to production.
What we covered in Episode 2
- Hardware Failure — physical machines still fail, even in the cloud
- Software Failure — bugs, unhandled errors, memory leaks; broadest category, variable blast radius
- Network Failure — the path between two healthy machines can still break
- Database Failure — often the app's fault (connection pool misuse), not the database's
- Third-Party Failure — you can't control it, but you control your reaction to it
- Human Error — reframed as a process/guardrail failure, not a personal one
- Resource Exhaustion — memory, CPU, disk, file descriptors; usually a ramp, not a cliff
- How a real incident often chains across several categories at once
Next up — Episode 3: "Failure Detection" — timeouts, health checks, heartbeats, logs, and monitoring: how a system finds out something broke before a user has to tell you.
Top comments (0)