DEV Community

Nico Reyes
Nico Reyes

Posted on

Tried 3 ways to handle retries in Python. This one actually makes sense.

Tried 3 ways to handle retries in Python. This one actually makes sense.

Scraping APIs that randomly fail taught me retries matter.

Tried a bunch of approaches. Most were garbage.

Manual While Loop (My First Attempt)

Started obvious:

attempts = 0
max_attempts = 3

while attempts < max_attempts:
    try:
        response = requests.get(url)
        response.raise_for_status()
        break
    except requests.exceptions.RequestException:
        attempts += 1
        if attempts >= max_attempts:
            raise
Enter fullscreen mode Exit fullscreen mode

Worked for one endpoint.

Copy pasted this everywhere. Had 15 files with the same loop. Changed retry logic once, had to update 15 files.

Dumb.

Tenacity Library (Overkill)

Found the tenacity library. Looked professional:

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), 
       wait=wait_exponential(multiplier=1, min=2, max=10))
def fetch_data(url):
    response = requests.get(url)
    response.raise_for_status()
    return response.json()
Enter fullscreen mode Exit fullscreen mode

This worked fine but felt like using a forklift to move a chair.

Had to install extra deps. Exponential backoff config was confusing for simple cases. Team kept asking what multiplier meant. Not worth it honestly.

Simple Decorator (What I Use Now)

Built my own tiny decorator:

import time
from functools import wraps

def retry(times=3, delay=1):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(times):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    if attempt == times - 1:
                        raise
                    time.sleep(delay)
            return None
        return wrapper
    return decorator

@retry(times=3, delay=2)
def fetch_api(url):
    response = requests.get(url)
    response.raise_for_status()
    return response.json()
Enter fullscreen mode Exit fullscreen mode

This is like 15 lines. No deps. Anyone reading my code understands it instantly.

Want different retry counts per function? Just change the decorator params. Want exponential backoff? Add delay * (attempt + 1) to the sleep.

Used this pattern building scrapers with the ParseForge API. Sometimes endpoints hiccup and need a second attempt. This decorator makes that brain dead simple.

When to Use What

Manual loop: Never. Just don't.

Tenacity: If you need complex retry logic like jitter or conditional retries based on error type. Or if your company already uses it.

Simple decorator: Pretty much everything else. APIs that occasionally fail. Network requests. Anything where just trying again solves it

Top comments (0)