Episode 2 gave you the seven categories of failure. Episode 3 answers the first real lifecycle question: once one of those seven happens, how does your system even find out?
Saturday, Round 3
👦 Nephew: Uncle, quick question. If my database connection drops at 2 AM tonight — how would I actually find out?
👨🦳 Uncle: Think about it honestly. Would you?
👦 Nephew: ...no. Not until a user complains in the morning, probably.
👨🦳 Uncle: That's the whole problem, right there.
Worst case: Failure happens → user notices → user complains → you find out
Goal: Failure happens → system notices → you find out → user never notices
👦 Nephew: So detection is just closing that gap.
👨🦳 Uncle: Shrinking the time between "something broke" and "someone who can fix it knows about it." Six tools do that. One at a time.
1. Timeouts
2. Health Checks
3. Heartbeats
4. Logs
5. Monitoring / Metrics
6. Error Responses
Part 1 — Timeouts
👦 Nephew: If my database hangs, Express just throws an error back automatically, right?
👨🦳 Uncle: Does it?
👦 Nephew: ...I actually don't know. I assumed.
👨🦳 Uncle: Query goes out. Database never responds. What happens to that line of code?
👦 Nephew: It just... waits?
👨🦳 Uncle: For how long?
👦 Nephew: ...forever?
👨🦳 Uncle: Forever.
Nephew: Wait. So my code could literally hang. Forever. No error, nothing?
👨🦳 Uncle: Nothing. Just a request quietly holding a connection, doing nothing, for as long as the process lives.
👦 Nephew: That's terrifying.
👨🦳 Uncle: Exactly. So you don't wait and hope. You force a decision.
Without timeout:
Request sent → ??? → ??? → ??? → (hangs indefinitely)
With timeout:
Request sent → waits up to 3s → no response → ERROR THROWN → you know NOW
👦 Nephew: So a timeout doesn't fix the database being slow.
👨🦳 Uncle: Correct — it just guarantees you find out fast. That's this whole episode, honestly. None of these six tools fix anything. They just surface the problem quickly enough that something else — Episode 4 — can act on it.
Part 2 — Health Checks
👦 Nephew: Isn't checking "is the process alive" enough, though? If Node's running, the server's up.
👨🦳 Uncle: Is it, though?
👦 Nephew: ...you're going to trace this too, aren't you.
👨🦳 Uncle: Every time. Your Node process is alive, accepting connections, fully responsive — but its database connection died an hour ago. Is that server "up"?
👦 Nephew: Technically yes. Practically... no, it can't actually do anything.
👨🦳 Uncle: Right. "Process is running" and "process can do its job" are two different questions.
Process alive? ✅ Yes
Can reach database? ❌ No
|
"Is the server up?" → misleadingly YES
"Is the server HEALTHY?" → correctly NO
👦 Nephew: So who's actually asking that second question?
👨🦳 Uncle: You build an endpoint that answers it honestly, and Kubernetes asks it — every few seconds, on its own.
Kubernetes calls GET /health every few seconds
|
200 OK → keep sending traffic here
|
503 → stop sending traffic, maybe restart the container
👦 Nephew: So a bad health check can get my own container restarted, without a human ever noticing?
👨🦳 Uncle: Correct. That's the system detecting and reacting on its own — no one woken up at 3 AM for it. We'll go deeper into exactly that in the Recovery episode.
Part 3 — Heartbeats
👨🦳 Uncle: New scenario. You have 300 background workers. No HTTP. No API. No endpoint to knock on. One of them just died. How do you know?
👦 Nephew: ...I genuinely don't know. There's nothing to ask it.
👨🦳 Uncle: Right — that's exactly why heartbeats exist. If nobody can ask it, it has to tell you, on its own, on a schedule.
Health Check: Heartbeat:
Monitor ---"are you okay?"--> Service ---"I'm alive"---> Monitor
Service <-------"yes"------- (repeats every N seconds, unprompted)
👦 Nephew: So it just... pings out into the void every few seconds, whether anyone's listening or not?
👨🦳 Uncle: Exactly. And when it stops?
👦 Nephew: Silence is the signal.
Uncle: You got there yourself.
Heartbeat every 10s: ping... ping... ping... ping... ping...
Worker dies: ping... ping... ...silence...
↑
monitoring system notices the gap
Part 4 — Logs
👦 Nephew: Logs feel almost too basic to count as "failure engineering." I've used console.log since day one.
👨🦳 Uncle: Fine on your laptop. At real production traffic — quick guess, what breaks first?
👦 Nephew: Uh... it gets slow?
👨🦳 Uncle: More basic. You've got a million lines of plain text. Find me the ones about order 1001, from an hour ago, at error level only.
👦 Nephew: ...I'd grep and pray.
👨🦳 Uncle: That's the actual problem. No structure, nothing to query. A structured logger fixes exactly that — every log becomes a searchable object instead of a sentence.
👦 Nephew: So structured logs turn "grep and pray" into an actual query.
👨🦳 Uncle: Exactly. Quick test — a retry that succeeded on the second attempt. What level?
👦 Nephew: Error?
👨🦳 Uncle: Did anything actually fail, though? It succeeded.
👦 Nephew: ...guess not. Warn, then. Unexpected, but not broken.
👨🦳 Uncle: There you go.
| Level | Use for |
|---|---|
debug |
Detailed internal state, only useful while actively debugging |
info |
Normal events worth recording — user signed up, order placed |
warn |
Something unexpected but not broken — retrying a request, slow response |
error |
Something actually failed and needs attention |
👨🦳 Uncle: Get this right, and your logs are searchable evidence. Get it wrong, and your real error logs are buried under a thousand info lines nobody reads.
Part 5 — Monitoring & Metrics
👦 Nephew: If logs already tell me what happened, why do I need something else on top?
👨🦳 Uncle: Does a log tell you a request was 120ms yesterday, and 900ms right now?
👦 Nephew: ...no. That's two separate log lines. I'd have to notice the pattern myself.
👨🦳 Uncle: That's the gap. Logs tell you about one event. You need something that shows the shape of things over time — the climb, before it becomes an outage. That's what metrics are for, and the tool most Node teams reach for is Prometheus — it scrapes numbers like response time on a schedule, and Grafana turns them into a graph.
👦 Nephew: So instead of one data point, I get the whole trend line.
👨🦳 Uncle: Exactly.
Response time trend:
10:00 ▂ 120ms
10:15 ▂ 135ms
10:30 ▃ 180ms
10:45 ▅ 310ms ← rising, nobody's paged yet
11:00 ▇ 900ms ← this is where users start noticing
11:15 █ timeout ← this is where it becomes an incident
👦 Nephew: So if I only open the dashboard after something breaks, I've already missed the entire point.
👨🦳 Uncle: Completely missed it. The value is catching 120ms turning into 900ms before it hits timeout — not confirming the outage after the fact.
Part 6 — Error Responses
👦 Nephew: Honestly, I used to just throw 500 at everything. Does the specific code actually matter that much?
👨🦳 Uncle: Put yourself on the other side. Something calls your API, gets a 500. What should it do?
👦 Nephew: ...retry, I guess?
Uncle: Should it? What if the 500 was actually a bad request that'll fail identically every time?
👦 Nephew: Oh. Then retrying is pointless. Same failure, forever.
👨🦳 Uncle: Right. A lazy 500 erases that distinction completely.
| Code | Meaning | What the caller should do |
|---|---|---|
400 |
Bad request — caller's fault | Don't retry, fix the request |
401 / 403
|
Auth failure | Don't retry, re-authenticate |
404 |
Not found | Don't retry |
429 |
Rate limited | Retry later, with backoff |
500 |
Unexpected server error | Maybe retry, log it |
503 |
Service unavailable (you know it's down) | Retry shortly |
👦 Nephew: So the status code is detection too — just pointed outward, at whoever's calling me instead of at myself.
👨🦳 Uncle: Exactly that.
Part 7 — All Six, Working Together
👨🦳 Uncle: Last one. I'll give you a scenario, you tell me the tool. A background worker silently died six hours ago. Nobody noticed.
👦 Nephew: Heartbeat. No endpoint to health-check, so it needed to announce itself.
👨🦳 Uncle: A request to a third-party API is hanging forever.
👦 Nephew: Timeout.
👨🦳 Uncle: Response times have been quietly climbing for two hours. No errors yet.
👦 Nephew: Metrics. Nothing's actually broken yet, so logs wouldn't even have anything to say.
👨🦳 Uncle: You just built the map yourself.
Timeout → catches a single hanging request, fast
Health Check → tells orchestrators if this instance is usable
Heartbeat → tells monitors if a background worker is alive
Logs → gives you the detailed story of what happened
Metrics → shows the trend before it becomes an incident
Error Codes → tells OTHER services what kind of failure this was
👦 Nephew: So none of these six actually fix anything.
👨🦳 Uncle: Not one. And remember this line, because it's the whole episode in a sentence: if your users are the first to discover your outage, your monitoring has already failed.
Practical Node.js Implementation
The six tools from today, as real code, in one place.
Timeout
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), 3000);
try {
const result = await fetch('https://api.example.com/data', {
signal: controller.signal
});
} catch (err) {
if (err.name === 'AbortError') {
console.error('Request timed out after 3s');
}
} finally {
clearTimeout(timeoutId);
}
Health check
app.get('/health', async (req, res) => {
try {
await db.query('SELECT 1');
res.status(200).json({ status: 'ok' });
} catch (err) {
res.status(503).json({ status: 'unhealthy', reason: err.message });
}
});
Heartbeat
setInterval(() => {
redis.set('worker:heartbeat', Date.now(), 'EX', 30);
}, 10000); // "I'm alive" every 10 seconds, expires if not refreshed
Structured logging
const logger = require('pino')();
logger.info({ userId: 42 }, 'User logged in');
logger.error({ err, orderId: 1001 }, 'Payment failed');
Metrics
const promClient = require('prom-client');
const httpDuration = new promClient.Histogram({
name: 'http_request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
labelNames: ['method', 'route', 'status']
});
app.use((req, res, next) => {
const end = httpDuration.startTimer();
res.on('finish', () => {
end({ method: req.method, route: req.path, status: res.statusCode });
});
next();
});
Error response
res.status(503).json({
error: 'DATABASE_UNAVAILABLE',
message: 'Unable to reach the database. Please retry shortly.',
retryable: true
});
👦 Nephew: So now I know something broke.
👨🦳 Uncle: Right.
👦 Nephew: How do I stop it from taking the whole system down?
👨🦳 Uncle: That's where real engineering begins.
👦 Nephew: Saturday?
👨🦳 Uncle: Saturday.
Top comments (0)