Retries, Timeouts & Circuit Breakers for Data Pipelines

retries timeouts and circuit breakers are the three controls that decide whether a data pipeline degrades gracefully or falls over the first time a dependency hiccups — and they are the single set of patterns senior data engineers are expected to reach for by name the moment an interviewer says "the upstream API started returning 503s at 2 a.m." Every non-trivial pipeline is a chain of remote calls: an HTTP extractor pulling from a partner API, a warehouse COPY staging a batch to Snowflake, a streaming sink flushing to Kafka, an enrichment step calling a geocoding service. Each hop can fail transiently, hang forever, or brown out under load, and the difference between a pipeline that self-heals and one that pages you is entirely in how you handle those failures — whether you retry blindly and cause a retry storm, whether you call without a timeout and hang a worker, whether you keep hammering a dead dependency instead of failing fast.

This guide is the walkthrough you wished existed the first time you had to defend a resilience design on a whiteboard. It builds the failure model in layers: the failure taxonomy that decides whether a call is even safe to retry, exponential backoff with jitter to spread the retries out and stop the thundering herd, the timeout budget that carries one deadline down a multi-hop call chain, the circuit breaker state machine and the bulkhead that isolates pools so one slow dependency can't sink the whole worker, and finally the correct order to nest all four primitives around a single idempotency-safe call. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works. All examples are Python against typical data-pipeline dependencies, but the mental model carries over to any language and any orchestration engine.

When you want hands-on reps immediately after reading, drill the defensive-coding practice library →, rehearse failure paths on the exception-handling practice library →, and wire the patterns into real jobs on the ETL practice library →.

---

On this page

• The pipeline failure taxonomy
• Retries with backoff and jitter
• Timeouts and deadline budgets
• Circuit breakers and bulkheads
• Composing resilience in a pipeline
• Cheat sheet — resilience recipes
• Frequently asked questions
• Practice on PipeCode

---

1. The pipeline failure taxonomy

Classify the failure before you react — retryable, fatal, or poison decides everything downstream

The one-sentence invariant: resilience engineering starts by classifying every failure into transient (retry with backoff), persistent (fail fast and alert), or poison (quarantine the input), because the wrong reaction to a failure class — retrying a fatal error, aborting on a transient blip, or replaying a poison record forever — is more damaging than the original failure. Data engineers get this wrong most often by treating "the call failed" as a single event when it is really a family of events with wildly different correct responses. A 503 from an overloaded API wants a backed-off retry; a 401 from an expired credential wants an immediate abort and a page; a malformed record that crashes your parser wants to be shunted to a dead-letter queue so the other 999,999 records in the batch still land.

The three failure classes that decide your reaction.

• Transient. The call would likely succeed if you tried again in a moment: HTTP 429/503, connection reset, DNS blip, a brief network partition, a warehouse "too many concurrent queries" throttle. These are the only failures you should retry. The correct reaction is backoff + jitter, bounded by a retry budget and a deadline.
• Persistent. The call will fail identically no matter how many times you retry: HTTP 401/403 (bad credentials), 400 (malformed request), 404 (wrong URL), a schema mismatch, a NOT NULL violation on a required column. Retrying these wastes time and hides the real problem. The correct reaction is fail fast, surface the error, and alert.
• Poison. The input is the problem, not the dependency: a single record whose payload crashes the transform, an un-decodable byte sequence, a value that violates a downstream constraint. Retrying reprocesses the same bad record forever (a "poison pill" that blocks the queue). The correct reaction is to quarantine the record to a dead-letter store and continue with the rest of the batch.

The four resilience primitives — one job each.

• Retries convert a transient failure into a success by trying again, spaced out by backoff and jitter. They do nothing for persistent or poison failures.
• Timeouts bound how long any single call may take, converting a hang (the worst failure mode — it consumes a worker indefinitely) into a fast, retryable failure.
• Circuit breakers stop calling a dependency that is persistently failing, converting a slow cascade of timeouts into an instant fail-fast, and giving the dependency room to recover.
• Bulkheads isolate resources (connection pools, thread pools, worker slots) per dependency so that one saturated dependency cannot consume all the capacity and starve the healthy ones.

The 2026 reality — partial failure is the default, not the exception.

• Distributed pipelines fail partially. A pipeline touching five services has, at any instant, a non-trivial probability that at least one is degraded. Designing for "everything is up" is designing for a state that rarely holds.
• Blind retries amplify outages. The single most common self-inflicted incident is the retry storm: a dependency slows down, every client retries immediately, the added load pushes the dependency fully over, and the retries now guarantee it can never recover. Retries without backoff, jitter, and a budget are a loaded gun pointed at your own infrastructure.
• Idempotency is the precondition. You may only safely retry a call whose repetition has no extra effect — a GET, an upsert keyed by a natural key, a COPY into an idempotent staging table. Retrying a non-idempotent POST that charges a card or appends a duplicate row is a correctness bug wearing a resilience costume.

What interviewers listen for.

• Do you classify the failure before choosing a reaction? — senior signal. Weak candidates say "add retries"; strong candidates say "is it transient, persistent, or poison?"
• Do you name idempotency as the precondition for retries without being prompted? — required answer.
• Do you describe a retry storm and how backoff + jitter + a budget prevent it? — senior signal.
• Do you distinguish a timeout (bounds one call) from a circuit breaker (bounds calling a dead dependency) — not conflate them? — required answer.
• Do you send poison records to a dead-letter queue rather than failing the whole batch or retrying forever? — senior signal.

Worked example — classify a batch of pipeline failures

Detailed explanation. The most useful artifact for a resilience interview is a failure-classification table you can build on the spot. Every senior discussion converges on "which of these do we retry?" within the first few minutes; having a crisp mapping in your head is what separates a fluent answer from a hand-wave. Walk through classifying the failures a typical ingestion task actually sees in a week.

• The task. An hourly extractor pulls JSON from a partner REST API, transforms each record, and upserts into a Postgres staging table.
• The failures observed. A mixed bag: throttling, expired tokens, one un-parseable record, a network reset, a NOT NULL violation.
• The goal. Assign each failure a class and a reaction, so the retry logic never fires on the wrong thing.

Question. Classify each observed failure and state the correct reaction for the extractor.

Input.

Observed failure	HTTP / error	Class	Correct reaction
API returned "rate limit exceeded"	429	transient	retry with backoff
Auth token expired overnight	401	persistent	refresh token or fail + alert
One record has a non-UTF-8 byte	ValueError in parse	poison	dead-letter the record
Connection reset mid-response	ConnectionResetError	transient	retry with backoff
Upsert hit a NOT NULL on email	IntegrityError 23502	persistent (bad data mapping)	fail + alert

Code.

# Classify a raised exception into a resilience action.
import httpx
import psycopg2

RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504}

class FailureClass:
    TRANSIENT  = "transient"    # retry with backoff
    PERSISTENT = "persistent"   # fail fast + alert
    POISON     = "poison"       # dead-letter the record

def classify(exc: Exception) -> str:
    # Network-level blips are transient by nature.
    if isinstance(exc, (httpx.ConnectError, httpx.ReadError, ConnectionResetError, TimeoutError)):
        return FailureClass.TRANSIENT

    # HTTP status: split retryable server/throttle errors from client errors.
    if isinstance(exc, httpx.HTTPStatusError):
        code = exc.response.status_code
        if code in RETRYABLE_STATUS:
            return FailureClass.TRANSIENT
        if 400 <= code < 500:
            return FailureClass.PERSISTENT      # bad request/creds; retry won't help
        return FailureClass.TRANSIENT

    # A record we cannot even parse is a poison pill, not a dependency fault.
    if isinstance(exc, (UnicodeDecodeError, ValueError, KeyError)):
        return FailureClass.POISON

    # A constraint violation is bad data mapping; retrying replays the same failure.
    if isinstance(exc, psycopg2.IntegrityError):
        return FailureClass.PERSISTENT

    # Unknown: be conservative — do not retry something you don't understand.
    return FailureClass.PERSISTENT

Step-by-step explanation.

1. Network-level exceptions (ConnectError, ReadError, ConnectionResetError, TimeoutError) are transient by definition — the request never got a definitive answer, so trying again is both safe (for an idempotent call) and likely to succeed.
2. HTTP status errors split on the code. The RETRYABLE_STATUS set holds throttles (429), request timeouts (408), and server errors (500/502/503/504) — all "the server is struggling, back off and retry." Any other 4xx is a client error: retrying an expired token or a malformed body reproduces the failure exactly.
3. A record you cannot parse — a bad byte, a missing key, an un-coercible value — is a poison failure. The dependency is fine; the input is broken. Retrying reprocesses the same bad record forever, so it must be dead-lettered.
4. A Postgres IntegrityError (a NOT NULL or FK or unique violation) is classified persistent here because it signals a mapping bug: the same row will violate the same constraint on every retry. Some teams route it to poison instead, dead-lettering the offending row; the key point is that it is not transient.
5. The default branch is deliberately conservative: an unknown exception is treated as persistent (fail fast), never as transient. Retrying something you do not understand is how a small bug becomes a retry storm.

Output.

Failure	classify() returns	Downstream action
429 Too Many Requests	transient	enqueue for backed-off retry
401 Unauthorized	persistent	stop, refresh creds, alert
non-UTF-8 record	poison	write to dead-letter table
connection reset	transient	enqueue for backed-off retry
NOT NULL violation	persistent	stop, fix mapping, alert

Rule of thumb. Never wire a retry decorator to "any exception." Build a classify() function first; retry only the transient class, fail fast on persistent, and dead-letter poison. The classifier is the load-bearing decision — everything else is mechanism.

Worked example — the resilience-primitive decision tree

Detailed explanation. Given a failing call, the senior engineer runs a short decision tree to pick which primitive applies. Codifying the tree makes the interview answer reproducible: an interviewer can hand you any failure scenario and you can name the right primitive in seconds. Walk the tree over three canonical scenarios.

• Q1. Is the call idempotent (safe to repeat)? → no = do not retry; make it idempotent first (idempotency key / upsert). yes = go to Q2.
• Q2. Is the failure transient? → no = fail fast (persistent) or dead-letter (poison). yes = go to Q3.
• Q3. Could the call hang? → yes = wrap in a timeout first. Always, in practice.
• Q4. Is the dependency failing repeatedly across many calls? → yes = add a circuit breaker so you stop hammering it. no = plain retry + backoff is enough.
• Q5 (parallel). Does this dependency share a pool with others that must stay healthy? → yes = put it behind a bulkhead.

Question. Walk the tree for three scenarios and record the primitives each needs.

Input.

Scenario	Idempotent?	Transient?	Can hang?	Repeated failure?
GET partner API (occasional 503)	yes (GET)	yes	yes	sometimes
POST charge to payment API	no	yes	yes	rare
Warehouse load sharing one pool with 3 sinks	yes (COPY to stage)	yes	yes	during brownouts

Code.

# Decision-tree helper (illustrative) — returns the primitives to apply.
def pick_primitives(idempotent: bool,
                    transient: bool,
                    can_hang: bool,
                    repeated_failure: bool,
                    shares_pool: bool) -> list[str]:
    primitives: list[str] = []

    if not idempotent:
        primitives.append("make-idempotent-first")   # prerequisite, not optional

    if can_hang:
        primitives.append("timeout")                 # always, in practice

    if transient and idempotent:
        primitives.append("retry+backoff+jitter")

    if repeated_failure:
        primitives.append("circuit-breaker")

    if shares_pool:
        primitives.append("bulkhead")

    return primitives


print(pick_primitives(True,  True, True, False, False))
# → ['timeout', 'retry+backoff+jitter']

print(pick_primitives(False, True, True, False, False))
# → ['make-idempotent-first', 'timeout', 'retry+backoff+jitter']

print(pick_primitives(True,  True, True, True,  True))
# → ['timeout', 'retry+backoff+jitter', 'circuit-breaker', 'bulkhead']

Step-by-step explanation.

1. Scenario 1 — an idempotent GET that occasionally 503s and can hang. The tree yields timeout + retry+backoff+jitter. No breaker needed because the failures are occasional, not sustained; no bulkhead because it has its own client.
2. Scenario 2 — a non-idempotent POST that charges a card. The tree first demands make-idempotent-first: attach an idempotency key so a retried charge is deduped server-side. Only then are timeout and retry safe.
3. Scenario 3 — a warehouse load that shares one connection pool with three other sinks and browns out under load. The tree yields the full stack: timeout, retry, circuit breaker (sustained brownouts), and bulkhead (shared pool must not be starved).
4. The tree makes explicit that primitives compose: most real calls need at least a timeout and a retry; the breaker and bulkhead are added when the failure is sustained or the resources are shared.
5. The most important branch is the first one. If a call is not idempotent, no amount of retry machinery is safe — the answer is to make it idempotent, not to skip the retry.

Output.

Scenario	Primitives to apply
GET partner API	timeout, retry+backoff+jitter
POST charge	idempotency key, timeout, retry+backoff+jitter
Shared-pool warehouse load	timeout, retry, circuit breaker, bulkhead

Rule of thumb. Walk the tree in order: idempotency first, then timeout, then retry, then breaker, then bulkhead. The order is not arbitrary — it is the same order you will nest the primitives in code (section 5).

Worked example — the cost of a naive retry loop

Detailed explanation. The classic anti-pattern is for _ in range(5): try call except: continue with no delay, no jitter, and no idempotency check. It looks like resilience and is actually an outage amplifier. Quantify what it does to a struggling dependency so you can argue against it with numbers, not opinions.

• The setup. 200 pipeline workers each call the same API. The API slows down (a GC pause, a deploy). Every worker's call fails.
• The naive loop. Each worker immediately retries up to 5 times with zero delay.
• The effect. Instead of 200 requests, the API now receives up to 1,000 requests in a tight burst — precisely when it is least able to serve them.

Question. Quantify the request amplification of a naive tight retry loop versus a single attempt.

Input.

Parameter	Value
Concurrent workers	200
Naive retries per worker	5
Delay between retries	0 (tight loop)
Dependency state	overloaded, ~0% success

Code.

# Naive tight retry loop — DO NOT ship this.
def fetch_naive(client, url):
    for _ in range(5):
        try:
            return client.get(url)          # no timeout, no backoff, no jitter
        except Exception:
            continue                        # immediately hammer again
    raise RuntimeError("all retries failed")

# Amplification model
WORKERS      = 200
RETRIES      = 5
success_rate = 0.0     # dependency is down

# With ~0% success, every worker exhausts all attempts.
total_requests_naive  = WORKERS * (1 + RETRIES)   # first try + 5 retries
total_requests_single = WORKERS * 1

print(f"naive  : {total_requests_naive} requests in a tight burst")
print(f"single : {total_requests_single} requests")
print(f"amplification: {total_requests_naive / total_requests_single:.0f}x")
# naive  : 1200 requests in a tight burst
# single : 200 requests
# amplification: 6x

Step-by-step explanation.

1. When the dependency is healthy, the naive loop is invisible — the first attempt succeeds and the retries never fire. This is exactly why the anti-pattern survives code review: it looks harmless in the happy path.
2. When the dependency is down, success rate drops to ~0%, so every worker burns all six attempts (one initial + five retries). Two hundred workers become 1,200 requests — a 6× amplification — delivered in a tight burst with no spacing.
3. The burst arrives precisely when the dependency is weakest. A dependency that might have recovered from 200 requests is now guaranteed to stay down under 1,200. The retries have converted a brownout into an outage.
4. There is no jitter, so all 200 workers retry in lockstep — a synchronized "thundering herd" that hits the dependency in coordinated waves rather than a smooth trickle.
5. There is no timeout, so if the dependency hangs (rather than erroring), each worker blocks indefinitely — the retry count never even gets a chance to matter, and 200 workers are consumed forever.

Output.

Retry strategy	Requests to a down dependency	Recovery odds
Single attempt	200	dependency may recover
Naive tight loop (×6)	1,200	dependency pinned down
Backoff + jitter + budget	~200–400, spread over time	dependency recovers

Rule of thumb. Every retry loop must add three things the naive version lacks: a delay that grows (backoff), randomness so clients de-synchronize (jitter), and a ceiling on total retries across the fleet (a budget). Without all three, "adding retries" makes outages worse, not better — which is exactly what section 2 fixes.

Python
Topic — defensive-coding
Defensive-coding problems on failure classification

Practice →

Python Topic — exception-handling Exception-handling problems on retryable vs fatal errors

Practice →

---

2. Retries with backoff and jitter

exponential backoff grows the gap between attempts and jitter scatters them — together they turn retries from an outage amplifier into a recovery mechanism

The mental model in one line: a correct retry policy multiplies the wait between attempts (exponential backoff), adds randomness so independent clients stop retrying in lockstep (jitter), caps the total retries across the fleet (a retry budget), and fires only on idempotent, transient failures — the four elements together let a struggling dependency recover instead of being pinned down by a synchronized herd of immediate retries. Every senior data engineer has shipped a retry loop; every senior data engineer has, at least once, watched a naive one turn a five-minute blip into a two-hour incident.

Why fixed-delay retries synchronize into a thundering herd.

• Fixed delay is deterministic. If every client retries exactly 1 second after a failure, and all clients failed at the same instant (because the dependency went down at once), they all retry at the same instant — a coordinated wave that hits the dependency as hard as the original burst.
• Exponential backoff spreads waves apart in time. Waiting 1s, then 2s, then 4s, then 8s means the fleet's retries thin out: fewer clients are still retrying by attempt 4, and they are spaced further apart. The dependency gets breathing room that grows with each round.
• But backoff alone still synchronizes. Even with exponential backoff, if all clients start their backoff at the same moment, attempt 2 for everyone lands ~2s later, attempt 3 ~6s later, and so on — the waves are further apart but still waves. Jitter is what breaks the synchronization within each wave.

The three jitter recipes — from the AWS backoff-and-jitter playbook.

• Full jitter. sleep = random_between(0, min(cap, base * 2**attempt)). Each client waits a uniformly random amount between zero and the current exponential ceiling. This maximally de-correlates clients and is the recommended default for most workloads.
• Equal jitter. sleep = half + random_between(0, half) where half = min(cap, base * 2**attempt) / 2. Keeps a guaranteed minimum wait (half the ceiling) plus a random half — useful when you want backoff to still grow visibly but with spread.
• Decorrelated jitter. sleep = min(cap, random_between(base, prev_sleep * 3)). The next wait is a random value up to 3× the previous wait, which climbs quickly but stays random and self-limits at the cap. AWS's testing found decorrelated jitter competitive with full jitter and often better at draining a backlog fast.

The retry budget — capping the blast radius.

• The problem. Per-call retries (e.g. "retry up to 5 times") bound one call but not the fleet. Under a broad outage, thousands of calls each retrying 5× is still a storm.
• The token bucket. Maintain a shared budget (a token bucket) that refills slowly and is debited on every retry. When the bucket is empty, retries are skipped — the call fails fast instead of adding to the storm. A common rule of thumb is a budget of ~10–20% of the success traffic: retries may add at most 20% load on top of normal requests.
• The effect. During a broad outage the budget drains quickly and most retries are suppressed, so the fleet's total load on the dependency stays bounded — the single most effective retry-storm defense.

Idempotency — the precondition for any retry.

• The rule. You may only retry a call whose repetition is harmless. GET, HEAD, an upsert keyed by a natural key, a COPY into a staging table that dedupes on load — all safe. A raw INSERT that appends, a POST that charges, an "increment counter" — not safe.
• The idempotency key. For unsafe operations, attach a client-generated idempotency key (a UUID per logical operation). The server stores processed keys and returns the original result on a duplicate — turning a non-idempotent call into a safely-retryable one.
• The interview line. "I only retry idempotent operations; for non-idempotent ones I add an idempotency key so the retry is deduped server-side." Say this unprompted.

Common interview probes on retries.

• "Why add jitter?" — required answer: to de-synchronize independent clients and prevent a thundering herd.
• "What is a retry storm and how do you prevent it?" — backoff + jitter + a fleet-wide retry budget.
• "Which failures do you retry?" — only idempotent, transient ones.
• "How many retries?" — bounded by both a max-attempts count and the remaining deadline budget (section 3).

Worked example — exponential backoff with full jitter

Detailed explanation. The canonical retry helper: try a call up to N times, sleeping a full-jittered exponential delay between attempts, retrying only on classified-transient failures, and giving up when attempts or a max elapsed time run out. Build it from scratch so every knob is explicit.

• Base. 0.5s initial backoff.
• Cap. 30s ceiling on any single sleep.
• Attempts. 6 maximum.
• Jitter. Full jitter — uniform between 0 and the current ceiling.

Question. Implement a retry wrapper with exponential backoff and full jitter that retries only transient failures.

Input.

Parameter	Value
base	0.5 s
cap	30 s
max_attempts	6
jitter	full (uniform 0..ceiling)
retry predicate	classify(exc) == transient

Code.

import random
import time

def retry_full_jitter(fn, *, base=0.5, cap=30.0, max_attempts=6):
    """Call fn(); retry transient failures with full-jitter exponential backoff."""
    attempt = 0
    while True:
        try:
            return fn()
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                raise                      # non-transient, or out of attempts
            # Full jitter: uniform between 0 and the exponential ceiling.
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            sleep_s = random.uniform(0, ceiling)
            print(f"attempt {attempt} failed ({exc!r}); sleeping {sleep_s:.2f}s "
                  f"(ceiling {ceiling:.1f}s)")
            time.sleep(sleep_s)

Step-by-step explanation.

1. The loop calls fn() and returns immediately on success — the happy path pays nothing. All retry machinery lives on the exception path.
2. On failure it increments the attempt counter, then consults classify(exc). If the failure is not transient, or attempts are exhausted, it re-raises — the wrapper never swallows a fatal error or loops forever.
3. ceiling = min(cap, base * 2**(attempt-1)) computes the exponential ceiling: 0.5s, 1s, 2s, 4s, 8s… clamped at the 30s cap. This is the upper bound on the wait, not the wait itself.
4. random.uniform(0, ceiling) applies full jitter — the actual sleep is a uniform random draw between 0 and the ceiling. Two workers that failed at the same instant now sleep different amounts, so their next attempts land at different times.
5. time.sleep(sleep_s) waits, then the loop tries again. The combination of a growing ceiling and per-attempt randomness is what turns a synchronized herd into a smooth, thinning trickle of retries.

Output.

Attempt	Ceiling	Example jittered sleep
1	0.5 s	0.31 s
2	1.0 s	0.74 s
3	2.0 s	0.12 s
4	4.0 s	3.55 s
5	8.0 s	2.90 s
6	(raise — attempts exhausted)	—

Rule of thumb. Default to full jitter with an explicit base, cap, and max_attempts, and gate every retry on a transient classification. The jitter is not optional polish — it is the difference between retries that help and retries that synchronize into a storm.

Worked example — a fleet-wide retry budget (token bucket)

Detailed explanation. Per-call max_attempts bounds one call; it does nothing to stop ten thousand calls each retrying during a broad outage. A shared token-bucket budget caps the fleet's retry load: retries spend tokens, the bucket refills slowly, and when it is empty retries are suppressed. Build a thread-safe budget and wire it into the retry decision.

• Refill. Tokens accrue at a rate tied to success traffic (e.g. 0.2 tokens per successful call → retries may add at most 20% load).
• Cost. Each retry costs 1 token.
• Suppression. No token → skip the retry, fail fast.

Question. Implement a retry budget and modify the retry loop to consult it before each retry.

Input.

Parameter	Value
Budget ratio	0.2 (retries ≤ 20% of success traffic)
Max bucket	100 tokens
Cost per retry	1 token
On empty	skip retry, raise

Code.

import threading

class RetryBudget:
    """Token bucket capping fleet-wide retries to a fraction of success traffic."""
    def __init__(self, ratio=0.2, max_tokens=100.0):
        self.ratio      = ratio
        self.max_tokens = max_tokens
        self.tokens     = max_tokens
        self._lock      = threading.Lock()

    def on_success(self):
        # Every success earns partial credit toward future retries.
        with self._lock:
            self.tokens = min(self.max_tokens, self.tokens + self.ratio)

    def try_spend(self) -> bool:
        # Spend one token for a retry; return False if the budget is exhausted.
        with self._lock:
            if self.tokens >= 1.0:
                self.tokens -= 1.0
                return True
            return False


BUDGET = RetryBudget(ratio=0.2, max_tokens=100.0)

def retry_with_budget(fn, *, base=0.5, cap=30.0, max_attempts=6):
    attempt = 0
    while True:
        try:
            result = fn()
            BUDGET.on_success()            # replenish on every success
            return result
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                raise
            if not BUDGET.try_spend():     # fleet-wide brake
                raise RuntimeError("retry budget exhausted — failing fast") from exc
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            time.sleep(random.uniform(0, ceiling))

Step-by-step explanation.

1. The bucket starts full (100 tokens) and every successful call adds ratio (0.2) tokens back, capped at max_tokens. In steady state, plenty of successes keep the bucket topped up, so occasional retries always find a token.
2. Each retry calls try_spend(), which atomically debits one token or returns False. The lock makes it safe across the many threads a worker pool runs.
3. During a broad outage, successes stop, so the bucket stops refilling. The first ~100 retries drain it; after that, try_spend() returns False and every further retry is suppressed — the loop raises immediately instead of adding to the storm.
4. This is the crucial fleet-level property: no matter how many calls are failing, the total retry load is bounded by the bucket size plus the (now-zero) refill rate. The dependency sees normal traffic plus a small, bounded retry overhead, never a 6× amplification.
5. When the dependency recovers, successes resume, the bucket refills, and retries are re-enabled automatically — no manual intervention, no config flip.

Output.

Fleet state	Successes	Bucket behavior	Retries allowed?
Healthy	many	stays near full	yes
Brief blip	mostly succeeding	drains slowly, refills	yes
Broad outage	~0	drains to 0, no refill	suppressed after ~100
Recovering	resuming	refills at 0.2/success	re-enabled gradually

Rule of thumb. Bound retries in two dimensions: per-call (max_attempts) and fleet-wide (a token-bucket budget of ~10–20% of success traffic). The per-call cap stops one call from looping; the budget stops the fleet from storming.

Worked example — idempotency keys make an unsafe call retryable

Detailed explanation. Retries are only safe on idempotent operations. When you must retry something that mutates state exactly-once (a payment, a "create shipment", an append), attach a client-generated idempotency key so the server dedupes duplicates. Walk through the pattern on a non-idempotent "register a payout" call.

• The key. A UUID generated once per logical operation and reused across all retries of that operation.
• The server contract. On first sight of a key, process and store the result under it; on a duplicate key, return the stored result without re-processing.
• The result. The client can retry freely; the effect happens at most once.

Question. Make a non-idempotent payout call safely retryable with an idempotency key.

Input.

Component	Value
Operation	POST /payouts (non-idempotent)
Idempotency key	UUID per logical payout
Server store	processed_keys table
Retry policy	full jitter + budget

Code.

import uuid

def create_payout(client, account_id, amount_cents, idem_key):
    """Non-idempotent server call made safe by an idempotency key header."""
    resp = client.post(
        "/payouts",
        json={"account_id": account_id, "amount_cents": amount_cents},
        headers={"Idempotency-Key": idem_key},   # the safety mechanism
    )
    resp.raise_for_status()
    return resp.json()

def payout_once(client, account_id, amount_cents):
    # Generate the key ONCE, outside the retry loop, so every retry reuses it.
    idem_key = str(uuid.uuid4())
    return retry_with_budget(
        lambda: create_payout(client, account_id, amount_cents, idem_key)
    )

-- Server-side dedupe table backing the Idempotency-Key contract
CREATE TABLE processed_payouts (
    idem_key     UUID        PRIMARY KEY,
    account_id   BIGINT      NOT NULL,
    amount_cents BIGINT      NOT NULL,
    result       JSONB       NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);

-- On each request: try to claim the key; if it already exists, return the stored result.
INSERT INTO processed_payouts (idem_key, account_id, amount_cents, result)
VALUES (%s, %s, %s, %s)
ON CONFLICT (idem_key) DO NOTHING
RETURNING result;

Step-by-step explanation.

1. payout_once generates the idempotency key once, outside the retry loop. This is the single most common mistake: generating the key inside the retried lambda would produce a new key per attempt, defeating the whole mechanism.
2. Every retry of create_payout sends the same Idempotency-Key header. From the server's perspective, all attempts of one logical payout are tagged identically.
3. Server-side, the INSERT ... ON CONFLICT (idem_key) DO NOTHING atomically claims the key. On the first attempt the row is inserted and the payout is processed; on any retry the conflict fires, no duplicate payout is created, and the stored result is returned.
4. This converts a non-idempotent operation into one that is safe to retry: the money moves at most once, no matter how many times the network eats the response and the client retries.
5. With idempotency in place, the payout call can now flow through the same retry_with_budget wrapper as any read — the earlier failure classes and budget apply unchanged. Idempotency is the key that unlocks retries for mutating operations.

Output.

Attempt	Idempotency-Key	Server action	Payouts created
1 (response lost)	k-abc	insert key, process	1
2 (retry)	k-abc	conflict → return stored	still 1
3 (retry)	k-abc	conflict → return stored	still 1

Rule of thumb. Generate the idempotency key once per logical operation, reuse it across every retry, and back it with an ON CONFLICT DO NOTHING claim server-side. This is the bridge that lets you retry mutating calls without double-effects.

Data-engineering interview question on resilient retries

A senior interviewer might ask: "You own an hourly extractor that pulls from a partner REST API. During their nightly maintenance the API returns bursts of 503s and occasionally hangs. Your current code retries immediately five times and has twice caused their on-call to page you back for 'hammering us during maintenance.' Redesign the retry logic: which failures you retry, the backoff strategy, how you stop the fleet from storming, and how you keep the extractor's own SLA."

Solution Using decorrelated jitter with a retry budget and per-attempt deadline check

import random
import time

def retry_decorrelated(fn, *, base=0.5, cap=20.0, max_attempts=6,
                       deadline_s=None, budget: "RetryBudget" = BUDGET):
    """Retry transient failures with decorrelated jitter, a fleet budget,
    and an overall deadline so retries never overrun the task SLA."""
    start   = time.monotonic()
    sleep_s = base
    attempt = 0
    while True:
        try:
            result = fn()
            budget.on_success()
            return result
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                raise
            if not budget.try_spend():
                raise RuntimeError("retry budget exhausted") from exc
            # Decorrelated jitter: next wait is random up to 3x the previous.
            sleep_s = min(cap, random.uniform(base, sleep_s * 3))
            # Never sleep past the overall deadline.
            if deadline_s is not None:
                remaining = deadline_s - (time.monotonic() - start)
                if remaining <= 0:
                    raise TimeoutError("deadline exceeded before retry") from exc
                sleep_s = min(sleep_s, remaining)
            time.sleep(sleep_s)

# Wire it to the extractor with a 90s task deadline.
def extract_hourly(client, url):
    return retry_decorrelated(
        lambda: client.get(url, timeout=httpx.Timeout(connect=1.0, read=5.0, timeout=8.0)),
        base=0.5, cap=20.0, max_attempts=6, deadline_s=90.0,
    )

Step-by-step trace.

Step	Value	Reasoning
Retry predicate	classify(exc) == transient	503/reset retried; 401/400 fail fast
Backoff	decorrelated jitter (random up to 3× prev)	climbs fast, stays de-synchronized
Cap	20 s per sleep	one slow retry can't eat the whole deadline
Fleet brake	token budget (20% of success)	suppresses retries during broad outage
Overall deadline	90 s	retries never overrun the task SLA
Per-call timeout	connect 1s / read 5s / total 8s	a hang becomes a fast retryable failure

After the redesign, a 503 burst triggers backed-off, jittered retries that spread across the fleet instead of arriving in lockstep; the shared budget suppresses retries once the partner is broadly down, so the extractor stops hammering during maintenance; and the 90-second deadline guarantees the hourly task still finishes (or fails cleanly) within its slot instead of retrying into the next hour.

Output:

Metric	Before (naive)	After (decorrelated + budget + deadline)
Requests to a down partner	6× amplification, bursty	~1× + bounded budget overhead
Retry timing	synchronized waves	de-synchronized spread
Behavior during maintenance	pages partner on-call	suppressed by budget
Task overrun risk	retries into next hour	capped at 90 s deadline
Hang handling	worker blocked forever	per-call timeout → fast retry

Why this works — concept by concept:

• Transient-only retry — the classify() gate ensures only 503s, throttles, and network resets are retried; a 401 or a malformed request fails fast and pages a human instead of looping uselessly.
• Decorrelated jitter — random.uniform(base, prev * 3) climbs quickly (draining a backlog fast) while keeping every client's timing independent, so the fleet never re-synchronizes into a herd.
• Retry budget — the shared token bucket caps fleet-wide retry load to ~20% of success traffic; during a broad outage it drains and suppresses retries, which is the actual retry-storm defense (per-call caps alone are not).
• Overall deadline — clamping each sleep to the remaining budget and aborting when it hits zero guarantees the hourly extractor finishes within its slot; resilience must never come at the cost of the task's own SLA.
• Cost — O(max_attempts) calls in the worst case per task, bounded further by the deadline; the budget adds one atomic token op per call. Compared to the naive loop's 6× burst, this is bounded load that lets the dependency recover — the difference between a self-healing pipeline and a self-inflicted outage.

Python
Topic — exception-handling
Exception-handling problems on retry and backoff

Practice →

Python Topic — defensive-coding Defensive-coding problems on idempotent retries

Practice →

---

3. Timeouts and deadline budgets

A timeout bounds one call; a timeout budget carries one deadline down the whole chain so a slow hop can't blow the end-to-end SLA

The mental model in one line: every remote call must carry a timeout — separate connect, read, and total tiers — and a multi-hop pipeline must carry a single end-to-end deadline that each hop decrements and propagates downstream, so that no call hangs forever, no downstream timeout exceeds the time actually left, and the whole chain fails fast the instant the budget is spent. The missing timeout is the most dangerous bug in resilience engineering because it is invisible until a dependency hangs — and then it consumes a worker indefinitely, which no retry, breaker, or bulkhead can fix after the fact.

The three timeout tiers — set all of them, always.

• Connect timeout. How long to wait to establish the connection (TCP + TLS handshake). Should be short — 1–2 seconds — because a healthy endpoint connects fast; a slow connect means the endpoint or network is unhealthy and you want to fail quickly to a retry.
• Read timeout. How long to wait for the response body after the request is sent (the server's time to produce and stream data). Tuned to the operation: a few seconds for a lookup, longer for a report query. This is the tier that catches a server that accepted the request but is grinding.
• Total (overall) timeout. A hard ceiling on the entire call including connect, all reads, and any redirects/retries within the client. This is the backstop that guarantees the call cannot exceed a known wall-clock bound even if the finer tiers are misconfigured.

The missing-timeout trap.

• The default is often "wait forever." Many clients and drivers default to no timeout. A single call to a hung dependency then blocks its worker thread indefinitely.
• Hangs are worse than errors. An error is fast and retryable; a hang silently consumes capacity. A pool of 50 workers all calling one hung dependency is 50 workers gone — the pipeline stalls with no error to alert on.
• The rule. Never make a network call — HTTP, database, queue, RPC, DNS — without an explicit timeout. "No timeout" is a latent outage waiting for the dependency to hang.

The deadline budget — one clock for the whole chain.

• The idea. Instead of each hop having an independent timeout, the entry point sets one end-to-end deadline (e.g. 10 seconds). Each hop computes how much of that budget remains, uses it (minus a margin) as its timeout, and passes the reduced budget to the next hop.
• The invariant. A downstream call's timeout must be less than the remaining budget. Calling a downstream service with a 30-second timeout when only 2 seconds of budget remain is pointless work — the caller will have already given up.
• Propagation. The remaining deadline travels with the request (an absolute timestamp in a header, a gRPC deadline, a context object). Downstream services honor it and refuse work they can't finish in time.

Timeout × retry interaction — retries live inside the deadline.

• The trap. A per-call timeout of 8s plus 5 retries can consume 40+ seconds — far past a 10-second task deadline. The retry count and the timeout must be reconciled against the budget.
• The fix. The retry loop checks the remaining budget before each attempt (as in section 2's retry_decorrelated), and never sleeps or retries past it. Retries are opportunistic within the deadline, not additive to it.
• The math. With a 10s budget and an 8s per-call timeout, there is room for at most one attempt plus a short retry — not five. The budget, not the retry count, is the real limit.

Common interview probes on timeouts.

• "What timeouts do you set on an HTTP call?" — required answer: connect, read, and total — not one blanket value.
• "What happens if you don't set a timeout?" — the call can hang forever and consume the worker.
• "What is a deadline budget?" — one end-to-end deadline decremented and propagated per hop.
• "How do retries fit inside a deadline?" — retries only run while budget remains; the deadline caps total time, not the retry count.

Worked example — connect / read / total timeout tiers

Detailed explanation. The canonical HTTP client setup for a data-pipeline extractor: distinct connect, read, and total timeouts tuned to the operation, so a slow connect fails in a second while a legitimately long report read gets the time it needs — all under a hard total ceiling. Configure it explicitly.

• Connect. 1s — a healthy endpoint connects fast.
• Read. 5s for a normal API; 30s for a heavy report endpoint.
• Total. 8s (normal) — hard ceiling regardless of the finer tiers.

Question. Configure an httpx client with tiered timeouts and show how each tier fires.

Input.

Tier	Normal endpoint	Report endpoint
connect	1 s	1 s
read	5 s	30 s
total	8 s	35 s
pool wait	2 s	2 s

Code.

import httpx

# Tiered timeouts — never a single blanket value.
NORMAL = httpx.Timeout(connect=1.0, read=5.0, write=5.0, pool=2.0, timeout=8.0)
REPORT = httpx.Timeout(connect=1.0, read=30.0, write=5.0, pool=2.0, timeout=35.0)

def fetch(client: httpx.Client, url: str, *, heavy: bool = False):
    tier = REPORT if heavy else NORMAL
    try:
        resp = client.get(url, timeout=tier)
        resp.raise_for_status()
        return resp.json()
    except httpx.ConnectTimeout:
        raise RuntimeError(f"connect > {tier.connect}s — endpoint/network unhealthy")
    except httpx.ReadTimeout:
        raise RuntimeError(f"read > {tier.read}s — server accepted but is grinding")
    except httpx.PoolTimeout:
        raise RuntimeError(f"pool wait > {tier.pool}s — client out of connections")

Step-by-step explanation.

1. httpx.Timeout takes independent connect, read, write, pool, and overall timeout values. Setting them separately lets each catch a distinct failure mode instead of one blanket number that is either too tight for reads or too loose for connects.
2. connect=1.0 fails fast when the endpoint can't be reached — a slow connect almost always means the network or endpoint is unhealthy, and you want that to become a quick, retryable error rather than a long stall.
3. read=5.0 (or 30 for reports) bounds the server's time to produce the body. A ReadTimeout means the server accepted the request but is grinding — distinct from a connect failure and often worth a different reaction.
4. pool=2.0 bounds how long the call waits for a free connection from the client's pool. A PoolTimeout means your side is out of connections (a local bottleneck), which is exactly the signal a bulkhead (section 4) acts on.
5. The overall timeout=8.0 is the hard ceiling — even if the finer tiers are misconfigured, no call exceeds it. Distinguishing the exception types lets the caller log precisely which tier fired, which is invaluable when debugging a flaky dependency.

Output.

Failure mode	Exception raised	Meaning
Endpoint unreachable	ConnectTimeout	network/endpoint down → retry
Server accepted, grinding	ReadTimeout	slow response → retry / breaker
No free client connection	PoolTimeout	local pool exhausted → bulkhead
Everything slow	overall timeout	hard ceiling hit

Rule of thumb. Always set connect, read, and total timeouts as separate values tuned to the operation, and distinguish the exception types so logs tell you which tier fired. A single blanket timeout is either too tight for slow reads or too loose to catch a hung connect.

Worked example — a deadline budget propagated down a call chain

Detailed explanation. A three-hop enrichment pipeline (extract → enrich → load) must finish within one end-to-end deadline. Instead of three independent timeouts that can sum past the SLA, carry one deadline: each hop computes the remaining budget and uses it (minus a margin) as its own timeout. Build the propagation with an absolute-deadline object.

• Entry deadline. 10 seconds total.
• Per-hop margin. 200 ms reserved for local processing between hops.
• Propagation. Each hop passes the remaining budget to the next.

Question. Implement a Deadline that each hop consults to derive its own timeout, and abort the chain when the budget is spent.

Input.

Hop	Nominal cost	Timeout derived from
extract	~4 s	remaining budget (10 s)
enrich	~4 s	remaining budget (~6 s)
load	~2 s	remaining budget (~2 s)

Code.

import time
import httpx

class Deadline:
    """One end-to-end deadline, propagated and decremented per hop."""
    def __init__(self, budget_s: float, margin_s: float = 0.2):
        self.expiry_at = time.monotonic() + budget_s
        self.margin    = margin_s

    def remaining(self) -> float:
        return self.expiry_at - time.monotonic()

    def timeout_for_hop(self) -> float:
        # A hop may use the remaining budget minus a margin — never more.
        left = self.remaining() - self.margin
        if left <= 0:
            raise TimeoutError("deadline exceeded before hop could start")
        return left


def run_chain(client, record, budget_s=10.0):
    dl = Deadline(budget_s)

    # Hop 1 — extract: timeout bounded by the whole remaining budget.
    raw = client.get(f"/source/{record}",
                     timeout=httpx.Timeout(dl.timeout_for_hop())).json()

    # Hop 2 — enrich: less budget remains; downstream timeout shrinks accordingly.
    enriched = client.post("/enrich", json=raw,
                           timeout=httpx.Timeout(dl.timeout_for_hop())).json()

    # Hop 3 — load: whatever is left; if <= margin, we abort instead of calling.
    client.post("/load", json=enriched,
                timeout=httpx.Timeout(dl.timeout_for_hop()))
    return {"loaded": record, "budget_left_s": round(dl.remaining(), 2)}

Step-by-step explanation.

1. Deadline records an absolute expiry (monotonic() + budget) rather than a duration. Absolute deadlines propagate correctly across hops because "how much time is left" is always expiry - now, regardless of how long earlier hops took.
2. timeout_for_hop() returns the remaining budget minus a small margin. This is the invariant in code: a hop's timeout is derived from what's left, so a downstream call can never be given more time than the whole chain has.
3. If a hop has already overrun (remaining ≤ margin), timeout_for_hop() raises TimeoutError before making the call — refusing to start work that cannot finish in time. This is deadline-aware load shedding: you don't pay for a call you'll abandon.
4. As the chain progresses, each hop sees a smaller budget: extract gets ~10s, enrich gets ~6s, load gets ~2s. The downstream timeout automatically shrinks — no per-hop constant to keep in sync, no risk of the sum exceeding the SLA.
5. monotonic() (not time.time()) is used throughout so the deadline is immune to wall-clock adjustments (NTP steps, leap seconds) that could otherwise make "remaining" jump.

Output.

Hop	Remaining on entry	Timeout used	Remaining on exit
extract	10.0 s	9.8 s	6.1 s
enrich	6.1 s	5.9 s	2.0 s
load	2.0 s	1.8 s	0.3 s
(if enrich overran to 0.1 s)	0.1 s	raise TimeoutError	chain aborts

Rule of thumb. Propagate one absolute deadline down the chain and derive each hop's timeout from the remaining budget minus a margin. Independent per-hop timeouts that can sum past the SLA are a latent breach; a shared, decrementing deadline makes the SLA a hard guarantee.

Worked example — reconciling retries with the remaining budget

Detailed explanation. Retries and deadlines interact: a generous per-call timeout plus several retries can silently overrun the task SLA. The fix is to let the retry loop and the timeout both read the same deadline, so retries only happen while budget remains and each attempt's timeout is clamped to what's left. Walk through the math and the code.

• Budget. 10s task deadline.
• Per-call timeout. min(8s, remaining budget).
• Retries. Allowed only while remaining > 0.

Question. Combine the retry loop and the deadline so total time never exceeds the budget.

Input.

Parameter	Value
Task deadline	10 s
Nominal per-call timeout	8 s
Backoff	full jitter, base 0.5 s
Constraint	total wall-clock ≤ 10 s

Code.

def call_within_deadline(client, url, *, budget_s=10.0, nominal_timeout=8.0,
                         base=0.5, max_attempts=6):
    dl      = Deadline(budget_s, margin_s=0.1)
    attempt = 0
    while True:
        # Each attempt's timeout is the smaller of nominal and what's left.
        per_call = min(nominal_timeout, dl.timeout_for_hop())
        try:
            r = client.get(url, timeout=httpx.Timeout(per_call))
            r.raise_for_status()
            return r.json()
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                raise
            ceiling  = min(4.0, base * (2 ** (attempt - 1)))
            sleep_s  = random.uniform(0, ceiling)
            # Do not sleep past the deadline; if no budget remains, give up.
            if dl.remaining() - sleep_s <= dl.margin:
                raise TimeoutError("no budget left for another attempt") from exc
            time.sleep(sleep_s)

Step-by-step explanation.

1. A single Deadline governs both the per-call timeout and the retry decision, so there is one source of truth for "how much time is left" — the loop can never disagree with the timeout.
2. per_call = min(nominal_timeout, dl.timeout_for_hop()) clamps each attempt's timeout to the remaining budget. Early attempts get up to the nominal 8s; later attempts get less as the budget shrinks. No attempt can be granted more time than the task has.
3. Before sleeping for backoff, the loop checks dl.remaining() - sleep_s: if sleeping would leave no time for another attempt, it raises immediately rather than burning the budget on a wait that leads nowhere.
4. The effect is that the retry count becomes secondary — the deadline is the real limit. With a 10s budget and an 8s first attempt, there is only room for one retry with a short backoff, and the code enforces exactly that.
5. This is the correct reconciliation the interview probes for: retries are opportunistic within the deadline, never additive to it. The task finishes within its SLA whether it succeeds on attempt one or exhausts its budget.

Output.

Attempt	Budget left	per_call timeout	Outcome
1	10.0 s	8.0 s	fails at 8 s
backoff	2.0 s	sleep 0.4 s	ok, budget remains
2	1.6 s	1.5 s	fails at 1.5 s
backoff	0.1 s	—	raise (no budget)

Rule of thumb. Feed the retry loop and the per-call timeout from one shared deadline. When they read the same clock, retries automatically fit inside the SLA and the task never overruns — the number of retries you actually get is dictated by the budget, not a constant.

Data-engineering interview question on deadline budgets

A senior interviewer might ask: "You have a multi-hop enrichment pipeline — extract from a source API, call a geocoding service, then load to the warehouse — with a strict 12-second per-record SLA because it runs inline in a streaming consumer. Occasionally the geocoder gets slow and the whole record processing blows past 12 seconds, backing up the consumer lag. Design the timeout and deadline strategy so no record ever exceeds 12 seconds end-to-end, including retries."

Solution Using a propagated deadline budget with tiered timeouts and deadline-aware retries

import time
import httpx

class Deadline:
    def __init__(self, budget_s, margin_s=0.15):
        self.expiry_at = time.monotonic() + budget_s
        self.margin    = margin_s
    def remaining(self):
        return self.expiry_at - time.monotonic()
    def timeout_for_hop(self, cap=None):
        left = self.remaining() - self.margin
        if left <= 0:
            raise TimeoutError("deadline exceeded")
        return min(left, cap) if cap else left

def process_record(client, record, budget_s=12.0):
    dl = Deadline(budget_s)

    # Hop 1 — extract (retryable within the shared deadline).
    raw = _hop(client, "GET", f"/source/{record}", None, dl, cap=5.0)

    # Hop 2 — geocode (the slow one): capped and deadline-bounded.
    geo = _hop(client, "POST", "/geocode", {"addr": raw["addr"]}, dl, cap=4.0)

    # Hop 3 — load (idempotent COPY into staging).
    _hop(client, "POST", "/load", {**raw, **geo}, dl, cap=3.0)
    return {"record": record, "ms_left": round(dl.remaining() * 1000)}

def _hop(client, method, url, body, dl: Deadline, cap):
    attempt = 0
    while True:
        per_call = dl.timeout_for_hop(cap=cap)     # min(remaining, cap)
        try:
            r = client.request(method, url, json=body,
                               timeout=httpx.Timeout(connect=1.0, read=per_call,
                                                     write=per_call, pool=1.0,
                                                     timeout=per_call))
            r.raise_for_status()
            return r.json() if r.content else {}
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= 3:
                raise
            sleep_s = random.uniform(0, min(0.5, dl.remaining() / 4))
            if dl.remaining() - sleep_s <= dl.margin:
                raise TimeoutError(f"{url}: no budget for retry") from exc
            time.sleep(sleep_s)

Step-by-step trace.

Step	Value	Reasoning
Entry deadline	12 s (absolute, monotonic)	matches the streaming SLA
Per-hop cap	extract 5 s / geocode 4 s / load 3 s	keeps any one hop from eating all budget
Per-call timeout	min(remaining − margin, cap)	never exceeds what the chain has left
Retry within hop	≤ 3, transient-only, budget-checked	retries fit inside the deadline
Geocoder slow path	capped at 4 s, then fail/degrade	slow hop can't back up the consumer
Abort condition	remaining ≤ margin	shed load rather than overrun

After the redesign, each record carries one 12-second monotonic deadline. The slow geocoder is capped at whatever is left up to 4 seconds; if it can't answer in time the hop aborts and the record is handled by the degraded path (e.g. loaded without geocoding, flagged for backfill) — the consumer never blocks past 12 seconds, so lag stays flat even during a geocoder brownout.

Output:

Metric	Before	After
Worst-case per-record time	unbounded (geocoder-driven)	≤ 12 s hard
Consumer lag during brownout	grows	flat
Slow-hop handling	blocks the chain	capped + degraded path
Retry overrun	possible	impossible (budget-checked)
Timeout coverage	partial	connect/read/total on every hop

Why this works — concept by concept:

• Absolute monotonic deadline — one Deadline object per record expresses the 12-second SLA as an absolute instant; every hop and every retry reads remaining() from the same clock, so nothing can independently overrun.
• Per-hop cap plus remaining budget — timeout_for_hop(cap) uses min(remaining − margin, cap), so no hop hogs the budget and none is ever granted more time than the chain has left.
• Deadline-aware retries — retries are transient-only, capped at three, and each checks that a backoff sleep still leaves budget; retries live inside the deadline rather than adding to it.
• Degraded path on abort — when the budget is spent the hop raises instead of blocking, letting the consumer fall back (load without geocode, flag for backfill) so lag never grows — availability over completeness under time pressure.
• Cost — O(hops × attempts) calls bounded hard by 12 s wall-clock; monotonic arithmetic is O(1) per check. Compared to independent per-hop timeouts that can sum to 30s+, the shared deadline converts a soft target into a hard guarantee — the property a streaming SLA actually requires.

Python
Topic — defensive-coding
Defensive-coding problems on timeouts and deadlines

Practice →

ETL Topic — etl ETL problems on SLA-bounded multi-hop pipelines

Practice →

---

4. Circuit breakers and bulkheads

A circuit breaker stops calling a dependency that keeps failing; a bulkhead isolates pools so one sick dependency can't drown the healthy ones

The mental model in one line: a circuit breaker is a three-state machine (closed → open → half-open) that watches the recent failure ratio for a dependency, trips open to fail fast when failures exceed a threshold, waits a reset timeout, then admits a single probe on half-open to test recovery — and a bulkhead is a bounded resource pool per dependency so that a saturated or slow dependency consumes only its own slice of capacity and cannot starve the rest of the pipeline. Retries and timeouts protect a single call; breakers and bulkheads protect the whole worker from a dependency that is broadly, persistently sick.

The three breaker states.

• Closed. Normal operation — calls pass through to the dependency. The breaker records outcomes (success/failure) in a rolling window. As long as the failure ratio stays below the threshold, it stays closed.
• Open. Tripped — calls fail immediately without touching the dependency (a fast CircuitOpenError, optionally a fallback). This is the key behavior: instead of every call timing out slowly against a dead dependency, they fail in microseconds, freeing workers and giving the dependency room to recover.
• Half-open. After a reset timeout, the breaker admits a single trial call (a probe). If it succeeds, the breaker closes (recovery confirmed); if it fails, the breaker re-opens and the reset timer restarts. Half-open prevents a stampede of calls the instant the timer expires.

The trip condition — ratio over a rolling window, not a raw count.

• Why ratio, not count. "Trip after 5 failures" trips on 5 failures out of 5 or 5 out of 5,000 — very different situations. A failure ratio over a rolling window (e.g. "> 50% of the last 20 calls failed") trips only when the dependency is genuinely, broadly unhealthy.
• The minimum-volume guard. Require a minimum number of calls in the window before the ratio can trip (e.g. at least 10 calls). This stops a single early failure from tripping the breaker before there is enough signal.
• The window. A rolling count-based window (last N calls) or time-based window (last T seconds). Count-based is simpler and common; time-based adapts better to bursty traffic.

Bulkheads — isolation so one leak doesn't sink the ship.

• The metaphor. A ship's hull is divided into sealed compartments (bulkheads); a breach floods one compartment, not the whole vessel. In software, a bulkhead is a bounded pool (connections, threads, semaphore permits) per dependency.
• The failure it prevents. Without bulkheads, one slow dependency's calls pile up and consume every worker/connection in a shared pool; healthy dependencies then can't get a connection and the whole pipeline stalls — a cascading failure from one sick component.
• The mechanism. Give each dependency its own bounded pool (or a semaphore capping concurrent calls). When dependency B is slow and its pool fills, calls to B are rejected fast (or queued briefly), but dependencies A and C still have their own capacity and keep working.

Fallbacks and load shedding when the breaker is open.

• Fallback. When the breaker is open, return a degraded-but-useful result instead of an error where possible: a cached value, a default, a "skip enrichment and flag for backfill" path. Availability over completeness under stress.
• Load shedding. If there is no useful fallback, fail fast and cheaply — an open breaker is load shedding, shedding load off the struggling dependency so it can recover.
• The interview line. "An open breaker fails fast; where a degraded result is acceptable I attach a fallback (cache, default, skip-and-backfill) so the pipeline stays available."

Common interview probes on breakers and bulkheads.

• "How does a circuit breaker differ from a retry?" — retry re-attempts one call; a breaker stops attempting a broadly-failing dependency.
• "What are the three states?" — closed, open, half-open.
• "What trips it?" — a failure ratio over a rolling window, guarded by a minimum call volume.
• "What is a bulkhead?" — per-dependency resource isolation so one sick dependency can't starve the others.
• "What does half-open do?" — admits one probe to test recovery without a stampede.

Worked example — a rolling-window circuit breaker

Detailed explanation. Build a thread-safe circuit breaker with the three states, a rolling-window failure ratio, a minimum-volume guard, and a reset timeout with a half-open probe. This is the exact shape interviewers expect you to be able to sketch.

• Threshold. Trip at > 50% failures.
• Window. Last 20 outcomes.
• Min volume. At least 10 calls before tripping.
• Reset timeout. 30 seconds open before a half-open probe.

Question. Implement the breaker and its call() guard.

Input.

Parameter	Value
failure_threshold	0.5
window_size	20
min_volume	10
reset_timeout_s	30

Code.

import time
import threading
from collections import deque

class CircuitOpenError(Exception):
    pass

class CircuitBreaker:
    def __init__(self, failure_threshold=0.5, window_size=20,
                 min_volume=10, reset_timeout_s=30.0):
        self.threshold   = failure_threshold
        self.min_volume  = min_volume
        self.reset_after = reset_timeout_s
        self.window      = deque(maxlen=window_size)   # True=failure
        self.state       = "closed"
        self.opened_at   = 0.0
        self._lock       = threading.Lock()

    def _failure_ratio(self):
        if len(self.window) < self.min_volume:
            return 0.0                                 # not enough signal
        return sum(self.window) / len(self.window)

    def call(self, fn):
        with self._lock:
            if self.state == "open":
                if time.monotonic() - self.opened_at >= self.reset_after:
                    self.state = "half-open"           # admit one probe
                else:
                    raise CircuitOpenError("circuit open — failing fast")

        try:
            result = fn()
        except Exception:
            self._record(True)                         # failure
            raise
        else:
            self._record(False)                        # success
            return result

    def _record(self, failed: bool):
        with self._lock:
            if self.state == "half-open":
                # A probe result decides the outcome definitively.
                if failed:
                    self.state, self.opened_at = "open", time.monotonic()
                else:
                    self.state, self.window = "closed", deque(maxlen=self.window.maxlen)
                return
            self.window.append(failed)
            if self.state == "closed" and self._failure_ratio() > self.threshold:
                self.state, self.opened_at = "open", time.monotonic()

Step-by-step explanation.

1. The window is a bounded deque of booleans (True = failure). _failure_ratio() returns 0 until at least min_volume outcomes exist — the minimum-volume guard that stops one early failure from tripping the breaker prematurely.
2. call() first checks state under the lock. If open and the reset timeout has elapsed, it transitions to half-open to admit a single probe; if open and still cooling down, it raises CircuitOpenError immediately — the fast-fail that defines the open state.
3. On the actual call, success and failure are both recorded via _record(). In the closed state, each outcome appends to the window and the ratio is re-checked; crossing the threshold trips the breaker open and stamps opened_at.
4. In the half-open state, the probe's result is decisive: a success closes the breaker (and resets the window for a clean slate); a failure re-opens it and restarts the reset timer. Only one probe is admitted because the state flips off "half-open" as soon as the probe runs.
5. All state transitions happen under the lock, so a pool of concurrent workers sees a consistent breaker — critical because breakers are inherently shared across the callers of one dependency.

Output.

Event sequence	State after
12 calls, 7 fail (58% > 50%)	open
calls during 30 s window	CircuitOpenError (fast fail)
30 s elapse, next call	half-open (probe admitted)
probe succeeds	closed (window reset)
probe fails	open (timer restarts)

Rule of thumb. Trip on a failure ratio over a rolling window with a minimum-volume guard, fail fast while open, and admit exactly one half-open probe per reset interval. A count-based "trip after N failures" breaker misfires on both high- and low-traffic dependencies.

Worked example — a bounded bulkhead pool

Detailed explanation. A bulkhead caps how many calls to a given dependency can be in flight at once, so a slow dependency consumes only its own slice of workers. Implement it as a semaphore with a bounded acquire timeout: if no permit is available quickly, the call is rejected fast rather than queuing forever.

• Max concurrent. 10 in-flight calls to this dependency.
• Acquire timeout. 100 ms to get a permit, else reject.
• Effect. Dependency B saturating its bulkhead cannot touch A's or C's permits.

Question. Implement a bulkhead and wrap a dependency call in it.

Input.

Parameter	Value
max_concurrent	10
acquire_timeout_s	0.1
on rejection	raise BulkheadFullError

Code.

import threading

class BulkheadFullError(Exception):
    pass

class Bulkhead:
    """Cap concurrent in-flight calls to one dependency."""
    def __init__(self, max_concurrent=10, acquire_timeout_s=0.1):
        self._sem     = threading.BoundedSemaphore(max_concurrent)
        self._timeout = acquire_timeout_s

    def call(self, fn):
        acquired = self._sem.acquire(timeout=self._timeout)
        if not acquired:
            raise BulkheadFullError("bulkhead full — shedding load fast")
        try:
            return fn()
        finally:
            self._sem.release()


# One bulkhead PER dependency — isolation is the whole point.
BULKHEADS = {
    "source_api": Bulkhead(max_concurrent=20, acquire_timeout_s=0.1),
    "geocoder":   Bulkhead(max_concurrent=5,  acquire_timeout_s=0.05),
    "warehouse":  Bulkhead(max_concurrent=10, acquire_timeout_s=0.2),
}

def call_dependency(name, fn):
    return BULKHEADS[name].call(fn)

Step-by-step explanation.

1. Each Bulkhead wraps a BoundedSemaphore sized to the maximum concurrent calls that dependency should ever have in flight. The semaphore is the hard cap on concurrency for that one dependency.
2. acquire(timeout=...) tries to take a permit but waits at most acquire_timeout_s. If the bulkhead is full (all permits held by slow in-flight calls), the acquire fails fast and raises BulkheadFullError instead of queuing indefinitely — fast rejection is the point.
3. The try/finally guarantees the permit is released even if fn() raises, so a failing dependency never leaks permits and permanently shrinks its own pool.
4. Crucially, there is one bulkhead per dependency. The geocoder gets only 5 permits; even if all five are stuck on a slow geocoder, the source API's 20 permits and the warehouse's 10 are untouched — those dependencies keep flowing. This is the isolation that prevents a cascade.
5. Sizing is a capacity decision: set each bulkhead to the concurrency the dependency can actually handle (or that you're willing to spend on it), not to the worker count. A too-large bulkhead defeats the isolation; a too-small one throttles healthy traffic.

Output.

Dependency	Permits	State when geocoder is slow
source_api	20	flowing normally
geocoder	5	full → BulkheadFullError (shed)
warehouse	10	flowing normally
worker pool overall	—	not starved; cascade prevented

Rule of thumb. Give every dependency its own bounded bulkhead sized to what that dependency can handle, and reject fast when it's full. Shared pools let one slow dependency consume all capacity; per-dependency bulkheads contain the damage to one compartment.

Worked example — breaker plus fallback for a degraded path

Detailed explanation. An open breaker fails fast — but "fail" doesn't have to mean "error to the caller." Where a degraded result is acceptable, attach a fallback so the pipeline stays available while the dependency recovers. Walk through a geocoder call that falls back to "no geocode, flag for backfill" when the breaker is open.

• Primary. Call the geocoder through the breaker.
• Fallback. On CircuitOpenError (or a call failure), return a record marked geocoded=False for later backfill.
• Effect. Records keep flowing; geocoding is filled in later.

Question. Wrap the geocoder in a breaker with a backfill fallback.

Input.

Component	Value
Dependency	geocoder
Breaker	50% / 20-window / 30 s reset
Fallback	geocoded=False, needs_backfill=True

Code.

GEOCODER_BREAKER = CircuitBreaker(failure_threshold=0.5, window_size=20,
                                  min_volume=10, reset_timeout_s=30.0)

def geocode_with_fallback(client, record):
    def primary():
        r = client.post("/geocode", json={"addr": record["addr"]},
                        timeout=httpx.Timeout(connect=1.0, read=3.0, timeout=4.0))
        r.raise_for_status()
        return {**record, **r.json(), "geocoded": True, "needs_backfill": False}

    try:
        return GEOCODER_BREAKER.call(primary)
    except CircuitOpenError:
        # Breaker open — skip fast, degrade gracefully.
        return {**record, "geocoded": False, "needs_backfill": True}
    except Exception:
        # Call failed (already recorded by the breaker) — degrade too.
        return {**record, "geocoded": False, "needs_backfill": True}

Step-by-step explanation.

1. The primary() closure is the real geocoder call, wrapped with tiered timeouts so a slow geocoder becomes a fast failure the breaker can count.
2. GEOCODER_BREAKER.call(primary) runs the call through the breaker. While closed, it passes through and records the outcome; once the failure ratio trips it open, subsequent calls raise CircuitOpenError in microseconds without touching the geocoder.
3. On CircuitOpenError, the fallback returns the record marked geocoded=False, needs_backfill=True. The pipeline keeps moving — records are loaded without geocoding and picked up by a later backfill job — instead of stalling on a dead dependency.
4. A raw call failure (breaker still closed but this call errored) is also degraded to the fallback, so the caller has one consistent contract: it always gets a record back, geocoded or flagged.
5. The net effect is graceful degradation: during a geocoder outage the pipeline's throughput is unaffected, only the completeness of geocoding is temporarily reduced — exactly the trade a resilient pipeline should make under stress.

Output.

Geocoder state	Breaker	Result
Healthy	closed	geocoded=True
Failing (below trip)	closed	this record degraded; failure recorded
Broadly down	open	geocoded=False, fast (no call)
Recovering	half-open probe	one probe; closes on success

Rule of thumb. Pair a circuit breaker with a fallback wherever a degraded result is acceptable. The breaker sheds load off the sick dependency; the fallback keeps the pipeline available — throughput preserved, completeness backfilled later.

Systems interview question on circuit breakers and bulkheads

A senior interviewer might ask: "Your ingestion worker pool calls three dependencies — a source API, a geocoder, and the warehouse — all sharing one HTTP connection pool. Last week the geocoder had a brownout: its calls went from 50 ms to 8 seconds, the shared pool filled with stuck geocoder calls, and the entire worker fleet stalled — even source-API and warehouse calls couldn't get a connection. Design the fix so one sick dependency can never again take down the whole worker."

Solution Using per-dependency bulkheads plus a circuit breaker with fallback

# Per-dependency isolation: each dependency gets its own bulkhead AND breaker.
BULKHEADS = {
    "source_api": Bulkhead(max_concurrent=20, acquire_timeout_s=0.1),
    "geocoder":   Bulkhead(max_concurrent=5,  acquire_timeout_s=0.05),
    "warehouse":  Bulkhead(max_concurrent=10, acquire_timeout_s=0.2),
}
BREAKERS = {
    "source_api": CircuitBreaker(0.5, 20, 10, 30.0),
    "geocoder":   CircuitBreaker(0.5, 20, 10, 30.0),
    "warehouse":  CircuitBreaker(0.6, 20, 10, 20.0),
}

def guarded_call(name, fn, fallback=None):
    """Bulkhead (outer) → circuit breaker (inner) → the call."""
    def breakered():
        return BREAKERS[name].call(fn)
    try:
        return BULKHEADS[name].call(breakered)
    except (BulkheadFullError, CircuitOpenError):
        if fallback is not None:
            return fallback()
        raise

def process(client, record):
    raw = guarded_call("source_api",
                       lambda: _get(client, f"/source/{record}"))
    geo = guarded_call("geocoder",
                       lambda: _post(client, "/geocode", {"addr": raw["addr"]}),
                       fallback=lambda: {"geocoded": False, "needs_backfill": True})
    return guarded_call("warehouse",
                        lambda: _post(client, "/load", {**raw, **geo}))

Step-by-step trace.

Layer	Component	Effect during geocoder brownout
Bulkhead (per dep)	geocoder capped at 5 permits	stuck calls consume only 5 slots, not the fleet
Circuit breaker (per dep)	geocoder trips open at 50%	calls fail fast in µs after the trip
Fallback	geocoded=False + backfill flag	records keep flowing
source_api bulkhead	20 permits, untouched	source calls unaffected
warehouse bulkhead	10 permits, untouched	warehouse loads unaffected
worker fleet	not starved	no cascade; throughput preserved

After the fix, the geocoder brownout is contained: its bulkhead caps stuck calls at 5 in-flight, its breaker trips open within a rolling window so further calls fail in microseconds, and the fallback flags records for backfill. The source-API and warehouse dependencies keep their own permits and breakers and flow normally. The single-dependency brownout that previously stalled the whole fleet now degrades only the geocoding completeness for a few minutes.

Output:

Metric	Before (shared pool)	After (bulkhead + breaker)
Blast radius of one slow dep	entire worker fleet	one dependency's slice
Geocoder brownout effect	full stall	geocode degraded, rest flows
Time wasted on dead dep	8 s per call × many	µs fast-fail after trip
Recovery	manual restart	automatic via half-open probe
Throughput during incident	~0	near-normal (minus geocode)

Why this works — concept by concept:

• Per-dependency bulkhead — each dependency's bounded semaphore means stuck geocoder calls can occupy at most 5 slots; the source API's 20 and the warehouse's 10 are physically separate, so one leak floods one compartment only.
• Per-dependency circuit breaker — the geocoder's breaker trips on its own failure ratio and fails its calls fast, converting 8-second hangs into microsecond rejections that free workers immediately.
• Bulkhead outer, breaker inner — the bulkhead admits the call into the dependency's capacity slice first; the breaker then decides whether to attempt or fast-fail. This ordering keeps rejected calls from ever consuming a permit's worth of time.
• Fallback for graceful degradation — where a degraded result is acceptable (geocoding), the fallback keeps records flowing; where it isn't (warehouse load), the error propagates. Availability is chosen per dependency, deliberately.
• Cost — one semaphore acquire and one breaker state-check per call, both O(1); a small amount of per-dependency config. Compared to a shared pool with no isolation, this converts an unbounded cascade into a bounded, self-recovering degradation — the exact property that keeps one sick dependency from taking down the fleet.

Python
Topic — defensive-coding
Defensive-coding problems on circuit breakers

Practice →

Python Topic — exception-handling Exception-handling problems on fallbacks and fail-fast

Practice →

---

5. Composing resilience in a pipeline

Nest the primitives in the right order — bulkhead → circuit breaker → timeout → retry → idempotent call — or they fight each other

The mental model in one line: the four resilience primitives compose in a strict nesting order — the bulkhead is outermost (admit into the dependency's capacity slice), then the circuit breaker (fail fast if the dependency is broadly sick), then the timeout (bound this attempt), then the retry loop (re-attempt transient failures within the deadline), wrapping the innermost idempotent call — and getting the order wrong makes the primitives undermine each other, e.g. retrying outside the breaker so retries keep a tripped breaker from ever resetting. Composition is where junior implementations fall apart: each primitive is correct in isolation but wired in the wrong order, so they cancel out.

The nesting order and why each layer sits where it does.

• Bulkhead (outermost). Admission control first: if the dependency's capacity slice is full, reject before spending any breaker or retry work. A rejected call should cost almost nothing.
• Circuit breaker (next). If the dependency is broadly failing, fail fast before the timeout and retry machinery runs. No point timing out and retrying a dependency the breaker already knows is dead.
• Timeout (next). Bound this attempt. The timeout sits inside the breaker so that each timed-out attempt is recorded as a failure the breaker can count toward tripping.
• Retry (inner). Re-attempt transient failures, each attempt getting its own timeout, all within the shared deadline. Retry sits inside the breaker so a tripped breaker stops retries — and inside the deadline so retries never overrun the SLA.
• Idempotent call (innermost). The actual operation, which must be idempotent (or carry an idempotency key) for any of the above to be safe.

Why retry is inside the breaker, not outside.

• Retry outside the breaker. If retries wrap the breaker, then when the breaker is open, the retry loop keeps re-invoking it — each invocation immediately raises CircuitOpenError, the retry catches it, waits, and tries again, hammering a breaker that is trying to stay open to let the dependency rest. The retries prevent the reset from ever helping.
• Retry inside the breaker. With retries inside, a tripped breaker raises once and the caller sees the open circuit; retries only happen while the breaker is closed and calls are actually reaching the dependency. This is the correct coupling.

Why retry is inside the deadline, outside the timeout.

• Inside the deadline. The retry loop must read the shared deadline (section 3) so its total time — attempts plus backoff sleeps — fits the SLA. Retries are opportunistic within the budget.
• Outside the (per-call) timeout. Each individual attempt gets its own timeout; the retry loop wraps those attempts. So "timeout" bounds one attempt and "retry" governs how many attempts — the timeout is inside the retry loop, the deadline is outside it.

Observability — one metric per layer.

• Retry metrics. Retry count, retry-budget-exhausted count, final outcome (success-after-retry vs exhausted).
• Timeout metrics. Per-tier timeout counts (connect/read/total), deadline-exceeded count.
• Breaker metrics. State (closed/open/half-open) as a gauge, trip count, half-open probe outcomes.
• Bulkhead metrics. In-flight gauge, rejection count, queue/acquire wait.
• The rule. Every layer must emit a metric, because when an incident happens you need to know which layer fired. A resilient stack with no observability is a black box during the exact moment you need to see inside it.

Failure injection — prove it works before production does.

• Why. Resilience code runs on the unhappy path, which normal testing rarely exercises. Untested resilience is often subtly broken (a swallowed exception, a wrong nesting order) and you find out during the real outage.
• How. Inject faults deliberately: a test double that returns 500s, adds latency, or hangs; a chaos step in staging that kills a dependency. Assert the breaker trips, the fallback fires, the deadline holds.
• The interview line. "I test the resilience stack with failure injection — I assert the breaker trips at the threshold, retries respect the budget, and the deadline is never exceeded."

Common interview probes on composition.

• "In what order do you nest the primitives?" — bulkhead → breaker → timeout → retry → idempotent call.
• "Why is retry inside the breaker?" — so a tripped breaker stops retries instead of being hammered.
• "How do retries and the deadline relate?" — retries live inside the deadline; the deadline caps total time.
• "How do you know the resilience works?" — failure injection + a metric per layer.

Worked example — the full resilience decorator stack

Detailed explanation. Assemble all four primitives into one composable wrapper in the correct nesting order, around an idempotent call, reading a shared deadline. This is the artifact a senior candidate should be able to produce end-to-end.

• Order. bulkhead(breaker(retry(timeout(idempotent call)))).
• Deadline. Shared across the retry loop and per-attempt timeout.
• Fallback. Optional, applied when the outer layers reject.

Question. Compose bulkhead, breaker, retry, and timeout around an idempotent call with a shared deadline and optional fallback.

Input.

Layer	Responsibility
bulkhead	admission / capacity isolation
breaker	fail fast on broad failure
retry	re-attempt transient, within deadline
timeout	bound each attempt
call	idempotent operation

Code.

def resilient_call(name, make_call, *, budget_s=10.0, max_attempts=4,
                   fallback=None):
    """Compose the four primitives in the correct nesting order."""
    dl = Deadline(budget_s, margin_s=0.1)

    def one_attempt():
        # timeout is applied by make_call using the remaining budget.
        per_call = dl.timeout_for_hop(cap=budget_s)
        return make_call(per_call)                 # innermost: idempotent call + timeout

    def with_retry():
        attempt = 0
        while True:
            try:
                return one_attempt()
            except Exception as exc:
                attempt += 1
                if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                    raise
                if not BUDGET.try_spend():
                    raise
                sleep_s = random.uniform(0, min(2.0, 0.5 * 2 ** (attempt - 1)))
                if dl.remaining() - sleep_s <= dl.margin:
                    raise TimeoutError("deadline") from exc
                time.sleep(sleep_s)

    def with_breaker():
        return BREAKERS[name].call(with_retry)     # retry INSIDE the breaker

    try:
        return BULKHEADS[name].call(with_breaker)  # bulkhead OUTERMOST
    except (BulkheadFullError, CircuitOpenError, TimeoutError) as exc:
        if fallback is not None:
            return fallback()
        raise

Step-by-step explanation.

1. A single Deadline is created once at the top and read by both one_attempt() (for the per-call timeout) and with_retry() (for the backoff budget check), so every layer agrees on the remaining time.
2. one_attempt() is the innermost layer: it derives the per-call timeout from the deadline and invokes the idempotent make_call. Timeout wraps the call directly.
3. with_retry() wraps one_attempt — retries transient failures, checks the retry budget, and never sleeps past the deadline. This is the retry layer, sitting outside the per-call timeout but inside the breaker.
4. with_breaker() wraps with_retry — so the breaker sees the net result of the retry loop, and, crucially, when the breaker is open it raises before any retry runs. Retries are inside the breaker, as required.
5. BULKHEADS[name].call(with_breaker) is the outermost layer: admission control first. If any outer layer rejects (bulkhead full, breaker open, deadline exceeded), the optional fallback fires. The nesting reads inside-out exactly as bulkhead → breaker → timeout → retry → call.

Output.

Condition	Which layer acts	Result
Dependency healthy	call	success
Transient blip	retry (inside breaker)	success after retry
Attempt hangs	timeout → retry	fast retry
Dependency broadly down	breaker	fast-fail → fallback
Capacity slice full	bulkhead	fast-reject → fallback
Budget spent	deadline	abort → fallback

Rule of thumb. Compose inside-out: idempotent call, wrapped by timeout, wrapped by retry, wrapped by breaker, wrapped by bulkhead. Reading the code from the innermost function outward should spell the nesting order — if it doesn't, the layers will fight.

Worked example — the nesting-order trace under a brownout

Detailed explanation. To cement why the order matters, trace one call through the stack during a dependency brownout, then trace the same call with retry mistakenly placed outside the breaker to show the failure. This contrast is the interview's favorite "gotcha."

• Scenario. The dependency has been failing for 40 seconds; the breaker is open.
• Correct order. Retry inside breaker.
• Wrong order. Retry outside breaker.

Question. Trace a call under both orderings and show how the wrong order defeats the breaker.

Input.

Setup	Value
Breaker state	open (tripped 40 s ago, 30 s reset)
Retry policy	4 attempts, backoff
Dependency	still down

Code.

CORRECT: bulkhead( breaker( retry( timeout(call) ) ) )
  1. bulkhead: permit acquired (dependency idle, plenty free)
  2. breaker.call(): state == open, reset not elapsed
        -> raises CircuitOpenError immediately
  3. retry NEVER runs (it's inside the breaker)
  4. caller catches CircuitOpenError -> fallback
  => 1 fast rejection, dependency untouched, breaker keeps resting

WRONG: retry( bulkhead( breaker( timeout(call) ) ) )
  1. attempt 1: breaker open -> CircuitOpenError
  2. retry catches it, sleeps, attempt 2: breaker open -> CircuitOpenError
  3. retry catches it, sleeps, attempt 3: ... attempt 4: ...
  => 4 invocations hammering a breaker that is TRYING to stay open;
     retries burn the deadline re-poking a dependency meant to rest

Step-by-step explanation.

1. In the correct order, the breaker sits outside the retry loop. When the breaker is open, breaker.call() raises CircuitOpenError once, the retry loop never executes (it is the argument to breaker.call), and the caller degrades to a fallback. One fast rejection; the dependency is left alone to recover.
2. In the wrong order, retry sits outside the breaker. Each retry attempt calls into the breaker, which is open and raises CircuitOpenError; the retry loop treats that as a failure to retry, sleeps, and tries again — up to four times.
3. The wrong order defeats the breaker's entire purpose: the breaker trips open precisely to stop traffic and let the dependency rest, but the outer retry loop keeps re-invoking it, spending backoff sleeps and deadline budget on calls guaranteed to fast-fail.
4. There is a subtler harm: if CircuitOpenError were (incorrectly) classified as transient, the retries would also count as breaker-adjacent churn and delay the caller's fallback, worsening latency during the exact incident the breaker exists to handle.
5. The lesson generalizes: the breaker must be able to suppress retries, which is only possible when it wraps them. Any ordering where retry is outside the breaker reintroduces the storm the breaker was added to prevent.

Output.

Ordering	Invocations while open	Deadline burned	Breaker helped?
Retry inside breaker (correct)	1 fast-fail	~0	yes — dependency rests
Retry outside breaker (wrong)	4 + backoff sleeps	most of it	no — breaker hammered

Rule of thumb. Retry must live inside the circuit breaker. If a tripped breaker doesn't stop your retries, your nesting order is wrong — and you've rebuilt the retry storm the breaker was supposed to prevent.

Worked example — a failure-injection test for the stack

Detailed explanation. Resilience code lives on the unhappy path, so it must be tested with injected faults. Write a test that drives the composed stack against a fault-injecting dependency and asserts the breaker trips, the fallback fires, and the deadline holds. This is what "I test my resilience" concretely means.

• Fault double. A fake dependency that returns 503s, then recovers.
• Assertions. Breaker trips within the window; fallback returns; total time ≤ deadline.

Question. Write a failure-injection test that proves the stack degrades correctly.

Input.

Injected behavior	Assertion
15 consecutive 503s	breaker trips to open
calls while open	fallback returned, fast
recovery after reset	half-open probe closes breaker

Code.

import time

class FaultyDependency:
    """Injects failures, then recovers — for testing the resilience stack."""
    def __init__(self, fail_first=15):
        self.calls = 0
        self.fail_first = fail_first
    def __call__(self, timeout):
        self.calls += 1
        if self.calls <= self.fail_first:
            raise httpx.HTTPStatusError("503", request=None,
                                        response=httpx.Response(503))
        return {"ok": True, "call": self.calls}

def test_stack_degrades_and_recovers():
    dep  = FaultyDependency(fail_first=15)
    name = "geocoder"
    BREAKERS[name] = CircuitBreaker(0.5, 20, 10, reset_timeout_s=1.0)

    fallbacks = 0
    start = time.monotonic()
    for _ in range(20):
        try:
            resilient_call(name, dep, budget_s=2.0, max_attempts=2,
                           fallback=lambda: (_ for _ in ()).throw(CircuitOpenError()))
        except CircuitOpenError:
            fallbacks += 1

    # 1. The breaker must have tripped open under sustained 503s.
    assert BREAKERS[name].state in ("open", "half-open")
    # 2. Once open, calls fast-fail into the fallback.
    assert fallbacks > 0
    # 3. No run exceeded its 2s deadline.
    assert time.monotonic() - start < 20 * 2.0

    # 4. After the reset timeout, a probe should close the breaker.
    time.sleep(1.1)
    result = resilient_call(name, dep, budget_s=2.0, max_attempts=2)
    assert result["ok"] is True
    assert BREAKERS[name].state == "closed"

Step-by-step explanation.

1. FaultyDependency is a deterministic fault double: the first 15 calls raise a 503, then every subsequent call succeeds. This lets the test drive the breaker across all three states on demand.
2. The loop makes 20 calls through the full resilient_call stack. The early calls fail and feed the breaker's window; once the failure ratio crosses 50% with enough volume, the breaker trips open and later calls fast-fail into the fallback.
3. Assertion 1 checks the breaker actually tripped (state open or half-open) — proving the trip condition works under sustained failure, not just in theory.
4. Assertion 3 checks the deadline held: 20 calls each bounded by a 2-second budget must finish well under 40 seconds, proving retries never overran the SLA even while everything was failing.
5. After sleeping past the 1-second reset timeout, one more call exercises the half-open probe against the now-recovered dependency; assertion 4 confirms the probe succeeds and the breaker closes — proving automatic recovery. The test covers trip, fast-fail, deadline, and recovery: the whole state machine.

Output.

Assertion	Proves
breaker state open/half-open	trip condition fires under load
fallbacks > 0	open breaker fast-fails to fallback
total time < N × budget	deadline holds during failure
final state closed + ok	half-open probe recovers automatically

Rule of thumb. Ship a failure-injection test with every resilience stack. If you have not watched your breaker trip, your fallback fire, and your deadline hold in a test, you do not know they work — and the first real outage is a terrible place to find out.

Systems interview question on composing resilience

A senior interviewer might ask: "Design an end-to-end resilient ingestion task for an Airflow DAG that pulls from a partner API, enriches via a geocoder, and loads to Snowflake. It must survive transient blips, dependency brownouts, and hangs, finish within a 60-second task SLA, never cause a retry storm, and be observable enough that on-call can tell which failure mode fired. Walk me through the full composition and how you'd verify it."

Solution Using the full composed stack with shared deadline, per-dependency isolation, and per-layer metrics

def ingest_record(client, record, metrics, budget_s=60.0):
    """End-to-end resilient ingestion: bulkhead > breaker > timeout > retry > idempotent call."""
    dl = Deadline(budget_s, margin_s=0.5)

    def make(name, method, url, body, cap, fallback=None):
        def one(per_call):
            r = client.request(method, url, json=body,
                               timeout=httpx.Timeout(connect=1.0, read=per_call,
                                                     write=per_call, pool=1.0,
                                                     timeout=per_call))
            r.raise_for_status()
            return r.json() if r.content else {}

        def with_retry():
            attempt = 0
            while True:
                try:
                    return one(dl.timeout_for_hop(cap=cap))
                except Exception as exc:
                    attempt += 1
                    metrics.incr(f"{name}.attempt")
                    if classify(exc) != FailureClass.TRANSIENT or attempt >= 3:
                        raise
                    if not BUDGET.try_spend():
                        metrics.incr(f"{name}.budget_exhausted"); raise
                    sleep_s = random.uniform(0, min(2.0, 0.5 * 2 ** (attempt - 1)))
                    if dl.remaining() - sleep_s <= dl.margin:
                        metrics.incr(f"{name}.deadline_exceeded"); raise TimeoutError from exc
                    time.sleep(sleep_s)

        def with_breaker():
            br = BREAKERS[name]
            metrics.gauge(f"{name}.breaker_state", {"closed":0,"half-open":1,"open":2}[br.state])
            return br.call(with_retry)

        try:
            return BULKHEADS[name].call(with_breaker)
        except (BulkheadFullError, CircuitOpenError, TimeoutError) as exc:
            metrics.incr(f"{name}.shed.{type(exc).__name__}")
            if fallback is not None:
                return fallback()
            raise

    raw = make("source_api", "GET",  f"/source/{record}", None, cap=20.0)
    geo = make("geocoder",   "POST", "/geocode", {"addr": raw["addr"]}, cap=10.0,
               fallback=lambda: {"geocoded": False, "needs_backfill": True})
    # Idempotent load: MERGE/COPY keyed on record id — safe to retry.
    make("warehouse", "POST", "/load", {**raw, **geo, "idem_key": record}, cap=15.0)
    metrics.gauge("record.ms_left", dl.remaining() * 1000)
    return {"record": record, "geocoded": geo.get("geocoded", False)}

Step-by-step trace.

Layer	Component	Observability signal
Deadline	one 60 s monotonic budget	record.ms_left gauge
Bulkhead	per-dependency semaphore	<dep>.shed.BulkheadFullError
Circuit breaker	per-dependency state machine	<dep>.breaker_state gauge (0/1/2)
Timeout	tiered, derived from remaining budget	<dep>.deadline_exceeded
Retry	transient-only, budget + deadline bounded	<dep>.attempt, <dep>.budget_exhausted
Idempotent load	MERGE/COPY keyed on record id	safe to retry

After deployment, each record carries one 60-second deadline down all three hops. The source API and warehouse use their own bulkheads and breakers; the geocoder degrades to a backfill flag when its breaker opens. Every layer emits a metric, so on-call sees exactly which mode fired — a geocoder.breaker_state = 2 gauge plus rising geocoder.shed counts says "geocoder brownout, breaker open, degrading gracefully" at a glance. A failure-injection test in CI proves the trip, fallback, and deadline before the code ever ships.

Output:

Metric	Value / behavior
Worst-case task time	≤ 60 s (shared deadline)
Retry storm risk	none (budget + backoff + jitter)
One-dependency brownout	contained (bulkhead + breaker + fallback)
Hang handling	tiered timeouts → fast retry
Diagnosability	one metric per layer per dependency
Pre-production confidence	failure-injection test in CI

Why this works — concept by concept:

• One shared deadline — a single 60-second monotonic budget threads through every hop and retry, so the composed stack respects the task SLA as a hard bound rather than a hopeful sum of per-call timeouts.
• Correct nesting order — bulkhead → breaker → timeout → retry → idempotent call means admission control runs first, a sick dependency fast-fails before retry work, and retries sit inside the breaker so a trip actually suppresses them.
• Per-dependency isolation — separate bulkheads and breakers keep a brownout in one dependency from starving the others; the fallback chooses availability over completeness where that trade is acceptable (geocoding) and not where it isn't (the load).
• A metric per layer — retry counts, breaker-state gauges, shed counts, and deadline-exceeded counters make the stack a glass box during an incident, so on-call diagnoses the failure mode in seconds instead of guessing.
• Cost — O(hops × attempts) calls, hard-bounded by 60 s, plus O(1) semaphore, breaker, and budget checks per call and a handful of metric emissions. Compared to an unguarded task that hangs, storms, and cascades, this is the difference between a pipeline that pages on-call at 2 a.m. and one that degrades gracefully and self-recovers — verified by failure injection before it ever runs in production.

ETL
Topic — etl
ETL problems on end-to-end resilient ingestion

Practice →

Python
Topic — defensive-coding
Defensive-coding problems on composing resilience

Practice →

---

Cheat sheet — resilience recipes

• Failure classification first. Split every failure into transient (429/503/reset → retry with backoff), persistent (401/400/404/schema → fail fast + alert), or poison (un-parseable record → dead-letter). Build a classify(exc) function; wire retries to the transient class only. Never retry "any exception."
• Exponential backoff with full jitter. ceiling = min(cap, base * 2**(attempt-1)); sleep = random.uniform(0, ceiling). Grows the gap and scatters the timing so independent clients de-synchronize. Defaults: base 0.5 s, cap 20–30 s, max 4–6 attempts. Jitter is mandatory, not optional.
• Decorrelated jitter (backlog drain). sleep = min(cap, random.uniform(base, prev_sleep * 3)). Climbs faster than full jitter while staying de-correlated — AWS's recommended variant for draining a backlog quickly after an outage.
• Retry budget token bucket. Shared bucket refilling at ~10–20% of success traffic, debited 1 token per retry; when empty, suppress retries and fail fast. This is the actual retry-storm defense — per-call max_attempts bounds one call, the budget bounds the fleet.
• Idempotency is the retry precondition. Only retry idempotent calls (GET, upsert, keyed COPY). For non-idempotent ones, generate an idempotency key once per logical op, reuse it across retries, and dedupe server-side with INSERT ... ON CONFLICT (idem_key) DO NOTHING RETURNING result.
• Timeout tiers — always all three. httpx.Timeout(connect=1.0, read=5.0, timeout=8.0) (+ pool). Connect short (unhealthy endpoint fails fast), read tuned to the operation, total as the hard ceiling. Never make any network call without an explicit timeout — the missing timeout is a latent hang.
• Deadline budget propagation. Set one absolute monotonic() deadline at the entry point; each hop uses min(remaining − margin, cap) as its timeout and passes the reduced budget down. A downstream timeout must be < the remaining budget. Retries live inside the deadline, never additive to it.
• Circuit breaker state machine. Closed → (failure ratio > threshold over a rolling window, min-volume guarded) → Open → (after reset timeout) → Half-open → (probe succeeds → Closed / probe fails → Open). Trip on a ratio, not a raw count; admit exactly one half-open probe.
• Bulkhead pool sizing. One bounded semaphore per dependency, sized to what that dependency can handle (not the worker count), with a short acquire timeout that rejects fast when full. Isolation means one slow dependency floods one compartment, never the whole worker fleet.
• Nesting order mnemonic. bulkhead → breaker → timeout → retry → idempotent call (outer to inner). Retry lives inside the breaker (so a trip suppresses retries) and inside the deadline (so retries fit the SLA); the per-call timeout is inside the retry loop.
• Fallbacks and load shedding. When a breaker opens or a bulkhead rejects, return a degraded-but-useful result where acceptable (cache, default, skip-and-backfill); otherwise fail fast. An open breaker is load shedding — it sheds load off the sick dependency so it can recover.
• Observability + failure injection. Emit one metric per layer per dependency — retry count, budget-exhausted, breaker-state gauge (0/1/2), bulkhead rejections, deadline-exceeded. Prove the stack with a fault-injecting test double: assert the breaker trips, the fallback fires, and the deadline holds before production does.

Frequently asked questions

What are retries, timeouts, and circuit breakers in one sentence?

They are the three core resilience controls for remote calls in a data pipeline: a retry re-attempts a transient failure (spaced out by exponential backoff and jitter, gated on idempotency), a timeout bounds how long any single call may take so a hung dependency can't consume a worker forever, and a circuit breaker stops calling a dependency that is persistently failing so you fail fast instead of piling up slow timeouts. They solve different problems — retries recover from blips, timeouts cap latency, breakers stop cascades — and a resilient pipeline composes all three (plus bulkheads) around every idempotent call. Every senior data-engineering interview probes them because they are the load-bearing patterns for any pipeline that touches a network.

Why add jitter to exponential backoff?

Because exponential backoff alone still lets independent clients retry in lockstep. If a dependency goes down and a hundred workers all fail at the same instant, plain exponential backoff has them all wait 1 s, then all wait 2 s, then all wait 4 s — the retries arrive in synchronized waves that hammer the dependency exactly when it's trying to recover (the "thundering herd"). Jitter adds randomness to each wait so the waves smear out into a smooth trickle. Full jitter (random(0, ceiling)) maximally de-correlates clients and is the common default; decorrelated jitter (random(base, prev*3)) drains a backlog faster. Without jitter, "adding backoff" only spaces the herds apart in time; it doesn't break the herd itself.

What is a timeout budget / deadline propagation?

A timeout budget (or deadline budget) is a single end-to-end time limit set at a pipeline's entry point and shared across every downstream hop, rather than each hop having an independent timeout. The entry point records an absolute deadline (e.g. monotonic() + 10 s); each hop computes the remaining budget, uses it (minus a small margin) as its own timeout, and passes the reduced budget to the next hop — this is deadline propagation. The invariant is that a downstream call's timeout must be less than the remaining budget, so the chain fails fast the instant the budget is spent instead of letting three independent 8-second timeouts sum to 24 seconds and blow a 10-second SLA. Retries also read the same deadline, so they fit inside the budget rather than adding to it.

How does a circuit breaker differ from a retry?

A retry re-attempts one call, assuming the failure is transient and the next attempt might succeed. A circuit breaker watches many calls to a dependency and, when the recent failure ratio crosses a threshold, stops attempting that dependency entirely for a cooldown period — it "trips open" and fails calls fast without touching the dependency. The distinction is scope and intent: retries are optimistic ("try again, it might work"); the breaker is pessimistic ("this dependency is broadly down, stop hammering it and let it recover"). They compose — retry inside the breaker — so occasional blips are retried while a sustained outage trips the breaker and short-circuits the retries. Retrying outside the breaker is a classic bug: the retries keep re-poking a breaker that is trying to stay open.

What is a bulkhead and when do I need one?

A bulkhead is per-dependency resource isolation — a bounded pool (connections, threads, or semaphore permits) dedicated to each dependency — named after a ship's sealed compartments that stop one breach from flooding the whole hull. You need one whenever multiple dependencies share a resource pool and one of them can get slow: without isolation, a slow dependency's calls pile up, consume every connection or worker in the shared pool, and starve the healthy dependencies too, turning one component's brownout into a full pipeline stall (a cascading failure). With a bulkhead, each dependency gets its own bounded slice; when one saturates, its calls are rejected fast while the others keep flowing. The classic trigger is exactly the interview scenario: "one slow dependency stalled the entire worker fleet" — the fix is per-dependency bulkheads.

In what order should I nest the resilience primitives?

From outermost to innermost: bulkhead → circuit breaker → timeout → retry → idempotent call. The bulkhead is outermost so admission/capacity control runs first and a rejected call costs almost nothing. The circuit breaker is next so a broadly-failing dependency fast-fails before any timeout or retry work happens. The timeout bounds each individual attempt. The retry loop wraps those attempts — and it must sit inside the breaker (so a tripped breaker suppresses retries instead of being hammered) and inside the shared deadline (so retries fit the SLA rather than adding to it). The innermost call must be idempotent, or carry an idempotency key, or none of the retrying above is safe. Getting the order wrong — most commonly putting retry outside the breaker — makes the primitives undermine each other.

Practice on PipeCode

• Drill the defensive-coding practice library → for the retry-policy, timeout, circuit-breaker, and bulkhead problems senior interviewers love.
• Rehearse on the exception-handling practice library → for classifying transient vs fatal failures, backoff loops, and fallback paths.
• Wire the patterns into real jobs on the ETL practice library → for SLA-bounded multi-hop pipelines, deadline budgets, and end-to-end resilient ingestion.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the nesting-order decision against real graded inputs.

Lock in resilience muscle memory

Docs explain the patterns. PipeCode drills explain the decision — when a naive retry loop becomes a retry storm, when a missing timeout hangs a worker, when a circuit breaker earns its place, and in what order the four primitives must nest around an idempotent call. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice defensive-coding problems →
Practice ETL problems →