My AI Agent Hung on a Single HTTP Request for 14 Hours. I Told Everyone It Was "Running a Long Task."

For most of a Tuesday, my agent dashboard showed one job in state running. I glanced at it three times and felt a small, smug satisfaction — look at it go, grinding through a big batch while I do other things.

It wasn't grinding. It had been sitting on one requests.get() call to a flaky third-party API since 2:47 AM. Fourteen hours earlier. No timeout. No retry. No failure. Just a socket quietly waiting for bytes that would never arrive, holding a worker slot hostage, while everything downstream assumed work was happening.

The worst part isn't the bug. The worst part is that I designed this failure without knowing it, and my monitoring was perfectly happy with it.

The failure, step by step

Here's the exact code path. It will look familiar if you've written any agent tooling:

def fetch_vendor_prices(vendor_id):
    resp = requests.get(f"{VENDOR_API}/prices", headers=AUTH)
    resp.raise_for_status()
    return resp.json()

Every line here is "correct." Linters love it. Code review passes it. And it contains one of the most common latent bugs in unattended software: requests has no default timeout. None. Zero. If the remote server accepts your TCP connection, then stalls mid-response — a dying load balancer, a half-closed socket, a vendor deploy going sideways — your call blocks forever.

At 2:47 AM the vendor's API did exactly that. My agent called the tool, the socket hung, and the whole job froze in a state that looked, from the outside, like legitimate long-running work.

Why my monitoring didn't catch it

This is the part that stings. I had "monitoring." Specifically:

1. A process check. The agent process was alive (blocked on I/O is still alive), so systemd was happy.
2. A heartbeat. My heartbeat was emitted between jobs. The job never finished, so no missed-heartbeat signal — the heartbeat just… wasn't due yet.
3. A job-duration alert. I had one. It was set to fire at 24 hours, because I'd once had a legitimate batch job run 9 hours and I'd gotten tired of false alarms. Fourteen hours slid right under it.

So I had built a monitoring system that could detect a dead agent with near-perfect accuracy and a hung agent not at all. Dead agents are actually the easy case — they announce themselves. Hung agents lie to you by looking busy.

I only found it because I got curious at 4 PM about why the job was taking so long, SSH'd into the Pi, and ran py-spy dump against the process. One thread, parked in sock_recv, since before sunrise.

The fix: timeouts at every layer

There is no single fix, because "hung" can happen at any layer. Here's what I actually changed, in order of impact.

1. A default timeout on every HTTP call, enforced. I stopped trusting myself to add timeout= per call and centralized it:

import requests

_session = requests.Session()
_adapter = requests.adapters.HTTPAdapter(max_retries=0)
_session.mount("https://", _adapter)
_session.mount("http://", _adapter)

def get(url, **kw):
    kw.setdefault("timeout", (5, 60))  # (connect, read)
    return _session.get(url, **kw)

Two numbers, not one: connect timeout (5s — if a server won't even accept the connection quickly, it's having a day) and read timeout (60s — the max gap between bytes). The read timeout is the one that would have saved my Tuesday: it fires on a stalled response even after the connection succeeds.

Then I added a grep to CI, of all places:

grep -rnE "requests\.(get|post|put|delete|patch)\(" src/ | grep -v "timeout" && exit 1

Crude, zero false confidence, catches the exact bug class. It's flagged three real omissions since.

2. A wall-clock deadline per job. The HTTP timeout protects one call. But an agent loop can also hang on a sequence of slow calls that never individually time out, or on something with no timeout at all (a subprocess, a DB lock). So every job now gets a hard deadline:

deadline = time.monotonic() + MAX_JOB_SECONDS
while not queue.empty():
    if time.monotonic() > deadline:
        raise JobTimeoutError(f"exceeded {MAX_JOB_SECONDS}s")
    ...

and the job runner wraps the whole thing in signal.alarm() as a backstop for code that ignores cooperative checks. Belt and suspenders, because hung jobs are exactly the case where you want redundancy.

3. Heartbeats during work, not between jobs. My old heartbeat proved "the agent finished something recently." Useless for hangs. The new one proves "the agent is making progress right now": the worker touches a heartbeat file every loop iteration, and a separate cron checks its mtime:

# check_heartbeat.sh — cron, every 10 min
AGE=$(( $(date +%s) - $(stat -c %Y /var/run/agent/heartbeat) ))
[ "$AGE" -gt 1800 ] && curl -s -X POST "$HEALTHCHECK_URL/fail"

A job hung inside a single blocking call stops touching the file, the age climbs, and Healthchecks.io (or a self-hosted equivalent) pages me within 30 minutes instead of "whenever I get curious."

4. Duration alerts sized per job type, not globally. My 24-hour alert existed to accommodate one legitimately slow job. Instead of loosening the threshold for everything, I tag jobs with a type and alert at 3× the p95 duration for that type. My "long batch" job alerts at ~10 hours; the pricing-sync job that hung? Its p95 is four minutes. It would have alerted at twelve.

The honest lesson

The bug wasn't the missing timeout=. Bugs like that are inevitable and cheap to fix. The real failure was a monitoring design that could only see death, not stuckness — plus my own willingness to look at a job running 14× longer than usual and narrate it as productivity instead of investigating it. I had even joked in my dev log that morning: "agent's been heads-down all day."

Unattended systems fail in two directions: they die loudly, or they lie quietly. Most guides cover the first. If you run agents on a Pi, in a closet, overnight — go look at your longest-running job right now and ask whether anything you've built would notice if it never finished. For me, the answer was no.

Six weeks since the fix: two vendor outages, one ISP blip, and a rogue DNS failure — all caught within minutes, all self-recovered or paged cleanly. Zero silent hangs.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.