Failing Gracefully: Robust REST API Requests for Data Engineering

Not every failed API call deserves a retry. A 404 and a 503 look similar in your logs — both show up as an exception or a bad status code — but one means "try again" and the other means "you have a bug." Most retry logic gets this wrong by treating every failure the same way.

That's the thing about data engineering: our pipelines sit at the mercy of other people's servers. Networks blip, rate limits get hit, upstream services restart mid-deploy. We can't prevent any of that. What we can do is make our requests polite, patient, and hard to kill — and that starts with telling failures apart before deciding what to do about them.

In this article, we'll build each layer of robust request handling by hand — timeouts, retry policies, exponential backoff, jitter, and friends — so every concept gets a face before a library gives it a name. Then we'll hand the same job over to packages like tenacity, urllib3's Retry, backoff, and stamina, and you'll recognize every knob they expose: they're doing exactly what we just did, only with fewer lines and better testing.

Sound good? Grab a coffee. Let's make your pipeline unbotherable.

---

The Stack of Robustness

Before we dive in, here's the mental model. Robust request handling is a layered thing — each layer assumes the one below it:

We'll walk this stack bottom-up. By the end, you'll have all six in your toolbox (and we'll mostly use the first five — circuit breakers get an honorable mention).

---

A Field Guide to API Failures

Not all failures are alike. The single most important skill here is triage: which failures are temporary, and which are your fault?

Roughly, everything that can go wrong falls into three buckets:

• Network-level failures — DNS won't resolve, the connection resets, the read takes forever. These are almost always blips. Retry.
• Server-side failures — 429 Too Many Requests, 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout. The server is struggling right now, but it will probably recover. Retry — politely.
• Client-side failures — 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 422 Unprocessable Entity. Retrying a 404 is like knocking on the same door repeatedly and hoping a different house appears. The problem is your request (or your credentials). Fix the code; don't retry.

Burn that decision tree in. Every robust client you'll ever write — by hand or via a library — is just that diagram translated into code.

---

Technique 1: Timeouts — Never Hang Forever

Here's a fun fact that has ruined many evenings: requests has no default timeout. None. If the server accepts your connection and then goes silent, your pipeline will sit there waiting until the heat death of the universe (or your orchestrator kills the task, whichever comes first).

import requests

# ❌ Will happily hang forever
resp = requests.get("https://api.example.com/v1/orders")

# ✅ Always, always pass a timeout
resp = requests.get("https://api.example.com/v1/orders", timeout=(5, 30))

That tuple is (connect_timeout, read_timeout):

• Connect timeout (5s) — how long we'll wait just to shake hands with the server. If the TCP connection can't be established in 5 seconds, something is wrong.
• Read timeout (30s) — once connected, how long we'll wait for the server to produce a response. Big payloads and slow endpoints warrant a bigger number.

Two separate clocks, two separate failure modes — a slow DNS lookup and a slow response body are different problems, and a single blanket timeout=30 conflates them.

A timeout is what turns "mysteriously stuck pipeline" into "error you can actually handle." It's the foundation everything else builds on — retries can't rescue a call that never technically failed.

httpx users: same idea, httpx.Client(timeout=5.0), or for finer control: httpx.Timeout(connect=5, read=30).

---

Technique 2: Retries — But Only for Failures That Deserve It

Once a request can fail, the next step is trying again. The naive version looks like this:

# The "just loop it" approach
for attempt in range(5):
    try:
        resp = requests.get(url, timeout=(5, 30))
        break
    except requests.RequestException:
        continue  # immediately try again

There are two problems here, and they're both about manners.

Problem one: it retries everything, including 404s. Retrying a permanent error just burns time and API quota before failing anyway.

Problem two: the retry is instant. Imagine the server is a bartender who just dropped a tray of glasses. Is the best response to immediately shout your order again? No — you give them a second to sweep up the shards.

So retries need two companions: a policy (which failures are retryable) and patience (a delay before trying again). The policy we covered in the field guide above. Patience is next.

---

Technique 3: Exponential Backoff — Fail Politely

Instead of retrying instantly, we wait a bit before each attempt — and we double that wait every time:

delay = base * 2^attempt     (capped at some maximum)

With base = 1 second, capped at 60:

Total patience expended: 15 seconds — and the server got increasingly generous breathing room between hits.

Why exponential instead of a fixed 1-second wait? Because of what usually went wrong:

• If the failure is a tiny blip, attempt 2 succeeds — the extra delay costs you almost nothing.
• If the failure is a big deal (server overloaded, deploying, melting down), hammering it every second is actively harmful. You'd be contributing to the very overload you're waiting out.

Compare the two strategies side by side:

Attempt    Linear (1s, 2s, 3s…)    Exponential (1s · 2^n, capped)
────────   ────────────────────    ───────────────────────────────
   1              1s                        1s
   2              2s                        2s
   3              3s                        4s
   4              4s                        8s
   5              5s                       16s
   6              6s                       32s
   7              7s                       60s ← capped

Both start gentle. But linear stays annoying, while exponential quickly transitions to "you know what, I'll check back later." That's the energy you want.

The cap matters too — without it, attempt #10 would have you waiting 8+ minutes on math alone. Pick a ceiling (30–60s is a common choice) so your worst case stays sane.

---

Technique 4: Jitter — Don't Stampede

Exponential backoff has one embarrassing weakness, and it shows up precisely in data engineering: we rarely run one worker.

Say your Airflow DAG spawns 100 concurrent tasks, all calling the same API. The API hiccups. All 100 tasks fail at roughly the same moment, all wait exactly 4 seconds (thanks, deterministic backoff), and all 100 hit the API at the same instant. You've just DDoS'd your data source by accident:

This is the classic thundering herd problem, and the fix is beautifully cheap: add a random offset to each delay. The standard recipe (from a beloved AWS architecture blog post) is full jitter:

delay = random.uniform(0, min(base * 2**attempt, cap))

Instead of "sleep exactly 4 seconds," every client sleeps somewhere between 0 and 4 seconds. Deterministic backoff synchronizes your fleet; jitter de-synchronizes it. One line of randomness, catastrophe averted.

Seeing it happen makes the effect more visceral than any box diagram — 80 workers all failing together, then either slamming the server as one wall of traffic or trickling in over four seconds:

---

Technique 5: Listen to the Server — Retry-After and Friends

Here's a life lesson disguised as an HTTP header: when a server returns 429 or 503, it often includes a Retry-After header telling you exactly how long to wait.

HTTP/1.1 429 Too Many Requests
Retry-After: 30

Your cleverly computed exponential backoff says "wait 4 seconds." The server explicitly says "wait 30." The server wins. It knows how many neighbors you have in the queue and how full its buffers are. Your formula is a guess; its header is a weather forecast.

(Caveat: Retry-After can legally contain either seconds or an HTTP-date. Most APIs use seconds; parse defensively if you don't trust the source.)

While we're being good citizens: if you know an API's rate limits, respect them proactively rather than discovering them via 429s. Some APIs hand you usage counters in response headers like X-RateLimit-Remaining — a well-behaved pipeline can throttle itself before getting slapped. Check the docs of whatever you're ingesting; the good ones document this.

---

Technique 6: Idempotency — Retrying Without Regrets

One more concept before we build, and it's the sneaky one: retries replay your request. Sometimes that's harmless. Sometimes it double-charges a credit card.

HTTP methods have a personality trait called idempotency — "can I repeat this without changing the outcome?"

• GET, HEAD — pure reads. Retry freely; the server just shrugs.
• PUT, DELETE — sending the same thing twice lands you in the same place. Generally safe to retry.
• POST — not idempotent. If a timeout happens after the server received your POST /orders but before you saw the response, a retry might create two orders. Retry #1 worked — you just never heard about it.

As data engineers we mostly do GETs, so we're naturally in the safe zone. But the moment you push data back to an API (triggering exports, acknowledging webhooks, writing back ML predictions), keep this in mind:

1. Many APIs support idempotency keys — send a unique token in a header (Idempotency-Key: <uuid>), and retries with the same token get deduplicated server-side. If the API offers this, use it. It's free safety.
2. If not, make retries of POSTs a conscious decision, not a default.
3. And here's the data-engineering twist: even on the destination side you can design for this — load into staging and MERGE/upsert on a natural key, so a partially-retried batch can never duplicate rows in your warehouse. Idempotency all the way down.

---

Assembling the Full Machinery (By Hand)

You now know every component. Let's bolt them together — no retry libraries, just the concepts:

import random
import time

import requests

RETRYABLE_STATUSES = {429, 500, 502, 503, 504}   # "the server is having a moment"
MAX_ATTEMPTS = 5
BASE_DELAY = 1.0     # seconds; doubles every attempt
DELAY_CAP = 60.0     # never sleep longer than this


def fetch(url: str) -> requests.Response:
    resp = None
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            resp = requests.get(url, timeout=(5, 30))    # ① timeouts
        except requests.RequestException:                # ② network errors
            if attempt == MAX_ATTEMPTS:
                raise                                    # out of chances — let it burn
        else:
            if resp.status_code not in RETRYABLE_STATUSES:
                resp.raise_for_status()                  # ③ our bug: fail fast, no retry
                return resp
            if attempt == MAX_ATTEMPTS:
                resp.raise_for_status()                  # ④ server's bug, but we're done trying

        if resp is not None and "Retry-After" in resp.headers:
            delay = float(resp.headers["Retry-After"])   # ⑤ server knows best
        else:
            ceiling = min(BASE_DELAY * 2 ** (attempt - 1), DELAY_CAP)
            delay = random.uniform(0, ceiling)           # ⑥ backoff + full jitter

        time.sleep(delay)

Read it top to bottom and you'll see the whole stack: timeouts (①), a retry policy that distinguishes network errors and server-side failures from our own mistakes (②③④), Retry-After awareness (⑤), and exponential backoff with jitter (⑥). About 25 lines. This is what the packages do — they just do it with nicer ergonomics.

One thing to watch: resp isn't reset between iterations, so if attempt 1 fails with a 503 (giving you a real response with headers) and attempt 2 then raises a RequestException (say, a dropped connection), the Retry-After check on the next pass would read the stale response from attempt 1, not anything related to the current failure. Worth guarding against explicitly if you adapt this for production — reset resp = None at the top of each loop body before the request.

Drop it into a real pipeline task:

def fetch_all_orders(base_url: str) -> list[dict]:
    """One bad page no longer takes the whole DAG down with it."""
    orders = []
    url = f"{base_url}/v1/orders"
    while url:
        page = fetch(url).json()
        orders.extend(page["results"])
        url = page["next"]
    return orders

Now — having built this ourselves and earned the right to be lazy — let's meet the packages.

---

Testing This Without Hitting a Real API

Retry logic is exactly the kind of code that's easy to write and easy to leave untested, because triggering the failure paths means either waiting for a real outage or manually taking a dependency down. Neither is something you want in CI.

The fix is mocking at the HTTP layer, not the function layer — you want to fake what the server does, not stub out your own fetch:

import responses  # pip install responses

@responses.activate
def test_retries_on_503_then_succeeds():
    responses.add(responses.GET, "https://api.example.com/v1/orders", status=503)
    responses.add(responses.GET, "https://api.example.com/v1/orders", status=503)
    responses.add(responses.GET, "https://api.example.com/v1/orders", json={"results": []}, status=200)

    resp = fetch("https://api.example.com/v1/orders")

    assert resp.status_code == 200
    assert len(responses.calls) == 3   # two failures, one success

This confirms the policy (retry 503s) and the attempt count, without sleeping through real exponential delays — responses and similar libraries (httpx's MockTransport, respx) let you assert on call counts and headers without your test suite taking sixty seconds to run.

Worth testing explicitly: that a 404 does not retry, that Retry-After is honored when present, and that the function gives up after MAX_ATTEMPTS. Those three are the actual policy — the parts most likely to have a bug, and the parts a code reviewer will care about most.

Knowing it's happening in production

A retry that works silently is invisible until it isn't — until the day it's retrying constantly and nobody notices because nothing "failed" from the orchestrator's point of view. Two cheap additions fix that:

import logging

logger = logging.getLogger("pipeline.http")

# inside the retry loop, right before time.sleep(delay):
logger.warning(
    "retrying %s (attempt %d/%d, status=%s, delay=%.1fs)",
    url, attempt, MAX_ATTEMPTS, resp.status_code if resp else "network_error", delay,
)

That single log line, shipped to whatever you already aggregate logs with, turns "the pipeline is slow today for no reason" into a searchable pattern. If you're using tenacity, this is even less code — before_sleep=before_sleep_log(logger, logging.WARNING) gives you the same thing as a one-line argument.

The metric worth having, if you're tracking any: a counter of retries per endpoint per day. A sudden spike is often the earliest signal that an upstream API is degrading — well before it degrades enough to fail your MAX_ATTEMPTS and page anyone.

---

Now Meet the Packages

Each of these gives you the same machinery with less plumbing. Knowing what the knobs do makes their APIs read like plain English.

tenacity — the composable classic

tenacity works via decorators and tiny, mixable building blocks — one per concept you just learned:

from tenacity import (
    retry,
    retry_if_exception,
    stop_after_attempt,
    wait_exponential_jitter,
)

import requests

RETRYABLE_STATUSES = {429, 500, 502, 503, 504}


def is_retryable(exc: BaseException) -> bool:
    if isinstance(exc, (requests.ConnectionError, requests.Timeout)):
        return True                                    # network blips: retry
    if isinstance(exc, requests.HTTPError):
        status = exc.response.status_code if exc.response else None
        return status in RETRYABLE_STATUSES            # 5xx/429: retry
    return False                                       # 4xx: our bug


@retry(
    retry=retry_if_exception(is_retryable),            # the retry policy
    wait=wait_exponential_jitter(initial=1, max=60),   # backoff + jitter, one call
    stop=stop_after_attempt(5),                        # and a stopping condition
    reraise=True,  # raise the original error, not tenacity's wrapper
)
def fetch(url: str) -> requests.Response:
    resp = requests.get(url, timeout=(5, 30))
    resp.raise_for_status()
    return resp

The predicate style shown here handles both exception types and status codes — the most realistic setup. If you only need exception-based retries, retry_if_exception_type((requests.ConnectionError, requests.Timeout)) is simpler.

Where tenacity shines is composition: wait_combine to chain wait strategies, before_sleep to log every retry, before/after hooks for metrics, retry=retry_any(...) to OR conditions together. For "I want precise control of everything," this is the one.

urllib3's Retry — the invisible layer

This one takes a completely different approach: instead of decorating functions, it configures the transport itself. Every call made through the session gets retries for free — no changes to any call site:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry_strategy = Retry(
    total=5,
    backoff_factor=1,                               # sleeps ≈ 0s, 1s, 2s, 4s, 8s
    status_forcelist=[429, 500, 502, 503, 504],     # which statuses to retry
    allowed_methods=["GET", "HEAD"],                # POST excluded by default (idempotency!)
    respect_retry_after_header=True,                # default, but let's be explicit
)

session = requests.Session()
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("https://", adapter)
session.mount("http://", adapter)

resp = session.get("https://api.example.com/v1/orders", timeout=(5, 30))

Notice what it got right without being asked: POST is excluded from retries by default, because urllib3's authors clearly thought about the idempotency problem too. (You can add it via allowed_methods, but now you'd be doing it consciously.)

Two quirks worth knowing:

• The first retry fires immediately (urllib3 assumes many failures resolve on a quick second try), then delays grow exponentially.
• When retries are exhausted, you get a requests.exceptions.RetryError wrapping the story — catch that, or set raise_on_status=False to get the final failed response back instead.

If you're retrofitting robustness onto a large existing codebase that uses requests everywhere, this is the lowest-friction option in this article.

backoff — the minimalist

backoff is tenacity's laid-back cousin. Two decorators, sane defaults, done:

import backoff
import requests

@backoff.on_exception(
    backoff.expo,                                    # exponential waits
    (requests.ConnectionError, requests.Timeout),
    max_tries=5,
    max_time=60,          # whole-operation deadline — lovely for pipelines
    jitter=backoff.full_jitter,                      # default, but see: no herd
)
def fetch(url: str) -> requests.Response:
    resp = requests.get(url, timeout=(5, 30))
    resp.raise_for_status()
    return resp

That max_time parameter deserves a wink — a wall-clock budget for the whole retry sequence ("give up after 60 seconds total") is exactly the semantics a pipeline task wants, and not every library offers it this cleanly. For status-code-based retries, backoff.on_predicate(backoff.expo, my_status_checker) retries on response values instead of exceptions. First-class async support, too.

stamina — the opinionated newcomer

From the "boring, correct defaults" school of engineering. stamina configures almost nothing, and what it does by default is right:

import stamina
import requests

@stamina.retry(on=requests.RequestException, attempts=5, timeout=60.0)
def fetch(url: str) -> requests.Response:
    resp = requests.get(url, timeout=(5, 30))
    resp.raise_for_status()
    return resp

Exponential backoff with jitter — built in. Retry budget with a hard deadline — built in. Clean structured logging of every retry — built in. Async support — built in. If you're starting a fresh project and don't need tenacity's advanced hooks, this is a very pleasant default.

One caveat about httpx's retries

httpx accepts a retries argument on its transports, but read the fine print:

client = httpx.Client(transport=httpx.HTTPTransport(retries=2))

That retries connection errors only — it will not retry a 503. For full behavior in httpx, either pass a urllib3.util.retry.Retry instance to HTTPTransport (yep, the same one from earlier) or wrap calls with tenacity/backoff/stamina. A trap worth knowing about.

Which one?

Package	Style	Sweet spot
tenacity	function decorator	Fine-grained control, composable hooks
urllib3 Retry	transport layer	Drop-in for existing requests code
backoff	function decorator	Simplicity, max_time budgets, async
stamina	function decorator	Modern defaults, logging, async

Honest answer: all four are good. Pick the one whose ergonomics match your codebase, and spend the saved time on tests.

---

What About Your Orchestrator?

"Wait," you might say, "my Airflow DAG already has retries." Right — and they operate at a different altitude:

from airflow.decorators import task
from datetime import timedelta

@task(retries=3, retry_delay=timedelta(minutes=5))
def ingest_orders(): ...

(Dagster offers the same idea via RetryPolicy(max_retries=3).)

Orchestrator-level retries re-run the entire task: re-fetch all 40,000 records, re-do all the parsing, re-run everything because one request failed on page 400. Request-level retries — the whole topic of this article — patch the hole in seconds, in place, without redoing the rest.

Use both:

• Request-level retries handle transient blips (the 99% case): fast, surgical, cheap.
• Task-level retries handle bigger failures — an API down for 10 minutes, a bad deploy upstream. Coarse, slow, rare.

One more distinction: the two layers want different Retry-After attitudes. Inside a single task, a 30-second server-mandated wait is fine. Inside an orchestrator retry? If the server says "come back in 6 hours," fail the task and let the scheduler's own backoff handle it — don't sleep inside a worker slot for six hours.

---

Going Further: The Circuit Breaker

One honorable mention for the curious. Retries ask "is the server back yet?" — but if an API is properly down, retry loops from many workers keep asking a dead server for attention. A circuit breaker tracks consecutive failures and, past a threshold, "opens the circuit": requests fail immediately for a cool-down period before a single probe request checks if the service recovered.

For single-pipeline work, retries + backoff + jitter cover you fine. Circuit breakers earn their keep when many services call many APIs and you want failures contained. If that's your world, look at pybreaker — now you know exactly where it fits in the stack.

---

The Pocket Checklist

Here's the whole article compressed into something you can paste into a PR description or a runbook. Before your next pipeline calls an API:

• [ ] Timeouts on every request. (connect, read). Non-negotiable. No default in requests means forever.
• [ ] Retry only what's transient. Network errors, 429, 5xx. Never 400/401/404.
• [ ] Exponential backoff, capped. Start at ~1s, double it, top out at 30–60s.
• [ ] Full jitter. random.uniform(0, delay). Your 100 concurrent tasks will thank you.
• [ ] Respect Retry-After. The server's forecast beats your formula.
• [ ] Think about POST + idempotency. Retry reads freely; think twice before replaying writes.
• [ ] Cap total effort. Max attempts and/or a wall-clock budget, then fail loudly and let the orchestrator escalate.
• [ ] Test the policy, not just the happy path. Mock the failures; assert on retry counts and give-up behavior.
• [ ] Log every retry. A silent retry loop today is a mystery slowdown next week.
• [ ] Let a package do the plumbing — tenacity, urllib3 Retry, backoff, or stamina — now that you know what each knob means.

None of this is exotic engineering. It's mostly judgment: knowing which failures are temporary, giving the server room to recover, and not replaying writes you can't take back. Twenty-five lines, or one decorator — either way, the pipeline stops needing you to babysit it.

---