DEV Community

Cover image for Failure Engineering Explained by Uncle to Nephew — Episode 4: Failure Handling
surajrkhonde
surajrkhonde

Posted on

Failure Engineering Explained by Uncle to Nephew — Episode 4: Failure Handling

#softwareengineering #distributedsystems #backend #node

Episode 3 covered detection — how a system finds out something broke. Episode 4 is the next link: detection told you something's wrong, now what?

Saturday, Round 4

👦 Nephew: Uncle, I set up timeouts and logs on my project like you said. Yesterday a payment gateway call actually timed out. I detected it. Logged it. Moved on.

👨‍🦳 Uncle: And?

👦 Nephew: That's it. Nothing else happened.

👨‍🦳 Uncle: Was it an analytics call, or a charge?

👦 Nephew: ...a charge.

👨‍🦳 Uncle: Then "log it and move on" wasn't handling. That was just watching it happen.

👦 Nephew: So detection alone isn't enough.

👨‍🦳 Uncle: Detection tells you something's wrong. Handling decides what you actually do. Six tools today.

1. Retry
2. Exponential Backoff
3. Fallback
4. Queue
5. Circuit Breaker (preview — full episode later)
6. Graceful Degradation
7. Dead Letter Queue
Enter fullscreen mode Exit fullscreen mode

Part 1 — Retry

👨‍🦳 Uncle: Simplest idea in the list. A call fails, you just try it again. What's wrong with that?

👦 Nephew: Nothing? If it fails, try again.

👨‍🦳 Uncle: Always?

👦 Nephew: ...I feel like you're setting a trap.

👨‍🦳 Uncle: Your payment gateway call — the one that timed out yesterday. If you'd retried it immediately, what actually happened on the gateway's side?

👦 Nephew: It timed out, so... it failed?

👨‍🦳 Uncle: Did it? Or did you just not get the response in time?

👦 Nephew: Wait. Those are different things.

Uncle: Very different. A timeout tells you nothing about whether the charge went through.

You                    Payment Gateway
 | -- charge request --> |
 |                       | (processes it successfully)
 |  <-- X response lost--| (network drops the response)
 | -- times out, retry-->|
 |                       | (charges again!)
Enter fullscreen mode Exit fullscreen mode

👦 Nephew: So I could've charged the customer twice, and my "handling" would've been the thing that caused it.

Uncle: That's the trap. So — what's actually safe to retry?

👦 Nephew: ...something that doesn't change anything if it runs twice?

👨‍🦳 Uncle: Exactly. That property has a name — idempotency, its own pattern in Module 3. For today, just remember: retry the safe stuff, and be paranoid about anything that touches money, inventory, or state.

Part 2 — Exponential Backoff

👨‍🦳 Uncle: Say retrying is safe here. Should you retry instantly, three times in a row?

👦 Nephew: Why not? Faster recovery.

👨‍🦳 Uncle: If 10,000 clients are all hitting a struggling service, and all 10,000 retry instantly — what happens to that service?

👦 Nephew: ...it gets hit even harder. I'd be making it worse.

👨‍🦳 Uncle: Right. So you wait a little longer after each failure, giving the service room to breathe.

Attempt 1 fails → wait ~2s
Attempt 2 fails → wait ~4s
Attempt 3 fails → wait ~8s
Attempt 4 fails → wait ~16s
Attempt 5 fails → give up, surface the error
Enter fullscreen mode Exit fullscreen mode

👦 Nephew: And if all 10,000 clients failed at the same second — don't they all back off on the exact same schedule too? Wouldn't they just retry together again?

👨‍🦳 Uncle: You just found the reason jitter exists — a small random delay added on top, so the retries land in a trickle instead of a second flood.

Without jitter:  1,000 clients retry at EXACTLY the same millisecond
With jitter:     1,000 clients retry spread across a small random window
Enter fullscreen mode Exit fullscreen mode

Part 3 — Fallback

👨‍🦳 Uncle: Sometimes the right response isn't "try again." It's "do something else instead."

👦 Nephew: Like what?

👨‍🦳 Uncle: Your recommendation service goes down. What should the user see?

👦 Nephew: An error, I guess? "Recommendations unavailable."

👨‍🦳 Uncle: Would you rather see an error, or a generic "popular items" list?

👦 Nephew: ...the popular list, obviously. Nobody wants to see an error for something that small.

👨‍🦳 Uncle: That's a fallback — trading perfect functionality for continued functionality.

Ideal path:    User → Personalized Recommendations   (best experience)
Failure path:  User → Popular Items fallback          (good enough experience)
Worst case:    User → Error page                      (avoid this if at all possible)
Enter fullscreen mode Exit fullscreen mode

👦 Nephew: Does everything deserve a fallback though?

👨‍🦳 Uncle: Would you want a fallback on a failed payment — "couldn't verify the charge, so we charged you a random amount instead"?

👦 Nephew: God, no.

👨‍🦳 Uncle: Right. Fallbacks are for things where "imperfect" beats "nothing." Some failures need to just fail, loudly and correctly.

Part 4 — Queue

👨‍🦳 Uncle: Your signup flow sends a welcome email synchronously, inside the request. Email service goes down for five minutes. What happens to signups?

👦 Nephew: They'd... fail? Or hang, waiting on the email service?

👨‍🦳 Uncle: Does signing up really need to wait on an email being sent?

👦 Nephew: ...no. Not really. The account's already created by that point.

👨‍🦳 Uncle: So don't make it wait. Push the email job somewhere safe, respond to the user immediately, and let a worker send it whenever the email service is ready again.

Without a queue:
Signup request → wait for email service → respond    (slow, and fails if email is down)

With a queue:
Signup request → push job to queue → respond immediately
                        |
                   Worker processes it whenever the email service is ready
Enter fullscreen mode Exit fullscreen mode

👦 Nephew: So the queue turns "must succeed right now" into "will succeed eventually."

👨‍🦳 Uncle: Exactly — and that shift alone eliminates an entire category of failures. This is the BullMQ + Redis pairing from the roadmap — BullMQ handles the queue and retry logic, Redis holds the job data.

Part 5 — Circuit Breaker (Preview)

👨‍🦳 Uncle: One more, quick preview — full episode later in Module 3. A downstream service is completely dead. Every request still tries it, waits, times out, fails. What's wrong with that picture?

👦 Nephew: You're... wasting the wait every single time, for every request?

👨‍🦳 Uncle: Right. So instead, after a few failures in a row, the system just stops trying — for a while.

First few requests → try, fail, fail → breaker "trips" (OPEN)
        |
All further requests → fail IMMEDIATELY, no wasted waiting
        |
After a cooldown → breaker allows ONE test request through
        |
If it succeeds → breaker closes, normal traffic resumes
If it fails → breaker stays open, wait longer
Enter fullscreen mode Exit fullscreen mode

👦 Nephew: So it gives up on purpose, temporarily, instead of endlessly hoping?

👨‍🦳 Uncle: Giving up fast and cheap beats failing slow and expensive, for every single request. Full states and implementation — its own episode.

Part 6 — Graceful Degradation

👨‍🦳 Uncle: This one isn't really a new tool. It's the philosophy underneath the last three. When something fails, what's the smallest thing you're willing to lose?

Full system:
[ Core Checkout ] [ Recommendations ] [ Reviews ] [ Live Chat Support ]

If Recommendations service fails:
[ Core Checkout ] [ (fallback: skip it) ] [ Reviews ] [ Live Chat Support ]
   ↑ still works, users can still buy things
Enter fullscreen mode Exit fullscreen mode

👦 Nephew: So this is the "why" behind fallbacks — keep the important 80% alive even if the nice-to-have 20% breaks.

👨‍🦳 Uncle: Exactly. Fallbacks, timeouts, circuit breakers — they're all just tools in service of this one question: "if this piece fails, what's the least damaging way my system can keep going?"

Part 7 — Dead Letter Queue

👨‍🦳 Uncle: Back to your queue. A job fails. Retries once, twice, three times. Still fails. What happens to it now?

👦 Nephew: ...does it just try forever?

👨‍🦳 Uncle: Would you want it to?

👦 Nephew: No — that'd waste resources forever on something that's clearly never going to work.

👨‍🦳 Uncle: So does it just get dropped, silently?

👦 Nephew: That feels worse. You'd lose the job and never know it happened.

👨‍🦳 Uncle: Right — neither option is acceptable. So it goes somewhere specific: a Dead Letter Queue. Not deleted, not endlessly retried — set aside for a human to actually look at.

Job fails → retry 1 → retry 2 → retry 3 → still failing
        |
Moved to Dead Letter Queue (not deleted, not silently dropped)
        |
Engineer reviews later: "why did THIS specific job keep failing?"
Enter fullscreen mode Exit fullscreen mode

👦 Nephew: So a DLQ is basically a detection tool wearing a queue's clothes.

👨‍🦳 Uncle: That's exactly it. It closes the loop between handling and detection.

Part 8 — Matching the Tool to the Failure

👨‍🦳 Uncle: Last one. I'll give you the failure, you give me the tool. A single network blip on a call that's safe to repeat.

👦 Nephew: Retry with backoff.

👨‍🦳 Uncle: A downstream service that's completely, entirely down.

👦 Nephew: Circuit breaker. No point hammering something that's dead.

👨‍🦳 Uncle: A non-critical feature — recommendations, say — just failed.

👦 Nephew: Fallback. Or graceful degradation, really the same idea.

👨‍🦳 Uncle: Work that doesn't need to happen this millisecond.

👦 Nephew: Queue it.

👨‍🦳 Uncle: A job that keeps failing no matter what you throw at it.

👦 Nephew: Dead letter queue — stop retrying blindly, let a human look.

👨‍🦳 Uncle: You just built the whole table yourself.

Situation Right tool
A single network blip on a safe-to-repeat call Retry + backoff
A downstream service is completely down Circuit Breaker
A non-critical feature fails Fallback / Graceful Degradation
Work that doesn't need to happen instantly Queue
A job that keeps failing no matter what Dead Letter Queue
An operation that changes money/state and might get retried Idempotency (Module 3)

👨‍🦳 Uncle: And that's really the whole lesson today, in one line: the scariest retry isn't the one that fails again — it's the one that quietly succeeds twice.

Practical Node.js Implementation

The six tools from today, as real code, in one place.

Retry

async function callPaymentGateway(payload) {
  for (let attempt = 1; attempt <= 3; attempt++) {
    try {
      return await paymentApi.charge(payload);
    } catch (err) {
      if (attempt === 3) throw err;
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Exponential backoff with jitter

async function retryWithBackoff(fn, maxAttempts = 5) {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (attempt === maxAttempts) throw err;
      const delay = Math.min(1000 * 2 ** attempt, 30000);
      const jitter = Math.random() * 300;
      await new Promise(r => setTimeout(r, delay + jitter));
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Fallback

async function getRecommendations(userId) {
  try {
    return await recommendationService.getFor(userId);
  } catch (err) {
    logger.warn({ userId }, 'Recommendation service down, using fallback');
    return getPopularItems();
  }
}
Enter fullscreen mode Exit fullscreen mode

Queue

const { Queue } = require('bullmq');
const emailQueue = new Queue('emails', { connection: redisConnection });

app.post('/signup', async (req, res) => {
  const user = await createUser(req.body);
  await emailQueue.add('welcome-email', { userId: user.id });
  res.status(201).json(user);
});
Enter fullscreen mode Exit fullscreen mode

Dead letter queue (via BullMQ's built-in failed state)

const emailQueue = new Queue('emails', {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 2000 }
  }
});

const failedJobs = await emailQueue.getFailed();
Enter fullscreen mode Exit fullscreen mode

👦 Nephew: Okay. So we've detected it, and now we've handled it.

👨‍🦳 Uncle: Right.

👦 Nephew: So what's left? The danger's contained — how does the system actually get back to fully healthy?

👨‍🦳 Uncle: That's where recovery begins.

👦 Nephew: Saturday?

👨‍🦳 Uncle: Saturday.

What we covered in Episode 4

  • Retry — powerful but dangerous on non-idempotent operations like payments
  • Exponential backoff + jitter — retrying without making an overloaded service worse
  • Fallback — trading perfect functionality for continued functionality
  • Queue (BullMQ + Redis) — turning "must succeed now" into "will succeed eventually"
  • Circuit Breaker — a preview of stopping wasted calls to a service that's already down
  • Graceful Degradation — the design principle underneath fallbacks and circuit breakers
  • Dead Letter Queue — where permanently failing jobs go instead of vanishing or retrying forever
  • Matching the right handling tool to the right type of failure

Top comments (0)