DEV Community

rene
rene

Posted on

My Python script crashed and the error message never printed. The bug was output buffering.

#python #webdev #beginners #tutorial

I spent two hours debugging a script that was "silently" failing. Every time it crashed, the terminal showed nothing — no traceback, no print output, nothing. The script had just... stopped.

The bug wasn't in my logic. It was in how Python buffers output.

The setup

I had a script that printed progress as it ran, then crashed partway through:

import time

for i in range(10):
    print(f"Step {i}: processing...")
    time.sleep(1)
    if i == 5:
        raise RuntimeError("something broke")
Enter fullscreen mode Exit fullscreen mode

When I ran it normally (python script.py), I saw every print() line fine, then the traceback. No mystery there.

But the moment I redirected output to a file or piped it into another program — exactly what you do when a script runs unattended (cron job, background task, CI step, an AI agent loop) — the output vanished:

python script.py > log.txt &
# log.txt stays EMPTY until the process exits or the buffer fills
Enter fullscreen mode Exit fullscreen mode

If the script crashed hard (killed, OOM, timeout), the buffered lines never got flushed to disk. I was staring at an empty log file trying to figure out where it died.

Why this happens

When stdout is a terminal, Python flushes on every newline (line-buffered). The moment stdout is a file or a pipe, Python switches to block-buffering — it holds output in memory (usually 4–8 KB) and only writes it out when the buffer fills or the process exits cleanly. A hard crash skips that final flush, so your last minutes of output just disappear.

This is exactly the kind of thing that bites you the first time you run something unattended — a scheduled script, a Docker container, or an AI agent loop writing its own log file.

The fix (three ways, pick one)

1. Flush explicitly where it matters:

print(f"Step {i}: processing...", flush=True)
Enter fullscreen mode Exit fullscreen mode

2. Force the whole script unbuffered from the command line:

python -u script.py > log.txt
Enter fullscreen mode Exit fullscreen mode

3. Set it as an environment variable (useful for Docker/cron where you can't change the launch command easily):

export PYTHONUNBUFFERED=1
Enter fullscreen mode Exit fullscreen mode

Any one of these and your log file updates in real time, line by line — so when the process dies, you see exactly the last thing it was doing instead of a blank file.

Why this matters more than it looks

If you're running anything unattended — a cron job, a background worker, or an autonomous script that's supposed to log its own crashes — this bug means your logging gives you zero signal exactly when you need it most: at the moment of failure. I only caught it because I was staring at a script that writes its own safety log before it enforces a budget limit, and the log was empty right when the limit should have tripped.

That's actually what pushed me to package the budget-guard pattern as a tiny free script — a drop-in check that stops a runaway loop before it burns your API budget, with flush=True baked in so the log is never empty when you need it: https://renevibe76.gumroad.com/l/kwgtni (free, name-your-price).

Has output buffering ever eaten your logs right before a crash? What's the weirdest "silent failure" bug you've debugged?

Top comments (0)