My Python script hung for 6 hours on one HTTP request. Here is the one-line fix

#webdev #beginners #python #tutorial

I left a small Python script running overnight to check a few URLs. In the morning it had done nothing for six hours. No error, no crash, no log line. It was just stuck on one request.

The cause: requests has no default timeout. If the server accepts the connection and then never answers, requests.get() waits forever. The official docs say it plainly: nearly all production code should use this parameter.

The broken version

import requests

def check(url):
    r = requests.get(url)          # no timeout: can block forever
    return r.status_code

The fix: always pass a timeout

import requests

def check(url):
    r = requests.get(url, timeout=(3.05, 10))
    return r.status_code

The tuple is (connect timeout, read timeout) in seconds:

connect: how long to wait to open the connection (3 seconds is plenty).
read: how long to wait between bytes of the response (10 seconds here).

Note that the read timeout is not a limit on the whole download. A server that trickles one byte every 9 seconds will still keep you waiting. For a hard overall limit, see the streaming approach at the end.

Handle the exception, then retry sensibly

A timeout raises requests.exceptions.Timeout. Catch it so one slow site does not kill the whole run:

import time
import requests

def check(url, attempts=3):
    for i in range(attempts):
        try:
            r = requests.get(url, timeout=(3.05, 10))
            return r.status_code
        except requests.exceptions.Timeout:
            wait = 2 ** i            # 1s, 2s, 4s
            print(f"timeout on {url}, retry in {wait}s")
            time.sleep(wait)
        except requests.exceptions.RequestException as e:
            print(f"failed on {url}: {e}")
            return None
    return None

Timeout is a subclass of RequestException, so the order of the except blocks matters: put the specific one first.

Set it once for the whole script

Passing timeout= on every call is easy to forget. A small session wrapper makes the safe behaviour the default:

import requests

class TimeoutSession(requests.Session):
    def request(self, method, url, **kwargs):
        kwargs.setdefault("timeout", (3.05, 10))
        return super().request(method, url, **kwargs)

s = TimeoutSession()
print(s.get("https://example.com").status_code)

A hard total limit (optional)

If you need a wall-clock cap on a large download, stream it and check the clock yourself:

import time, requests

def download(url, max_seconds=30):
    start = time.monotonic()
    chunks = []
    with requests.get(url, stream=True, timeout=(3.05, 10)) as r:
        for chunk in r.iter_content(8192):
            chunks.append(chunk)
            if time.monotonic() - start > max_seconds:
                raise TimeoutError("download took too long")
    return b"".join(chunks)

Quick checklist

Every network call has a timeout. No exceptions.
Catch Timeout separately from other errors.
Retry with backoff, and cap the number of attempts.
Log the URL on failure so the 3 a.m. you knows what happened.

The standard library has the same trap: urllib.request.urlopen(url, timeout=10) also needs the argument, see the Python docs.

This article was written with AI assistance and checked by the author.

What is the worst "it just hung and told me nothing" bug you have hit, and how did you finally find it?

DEV Community