DEV Community

Taylor Wang
Taylor Wang

Posted on

The Job Looked Frozen for 48 Hours. It Was Waiting for stdin.

Have you ever stared at a remote job that stayed in running state until a timeout killed it? I did that for forty-eight hours, and I blamed a generated CLI that had never actually crashed. The laptop run finished in seconds because I was sitting at a real keyboard the whole time. The free server run looked frozen because the process was blocked on stdin nobody could type into.

Why this looked like a model problem

I had asked a coding model for a small maintenance CLI that confirmed before it deleted leftover build artifacts. On my laptop the script printed a warning, I typed yes, and the rest of the pipeline continued without drama. On the remote box the same entrypoint printed one line and then produced nothing until the job was killed. Was the model stuck in a retry loop, or was the box swapping, or had logging simply vanished again?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access to draft the CLI helpers, then replayed the same script on MonkeyCode's free server option. That pairing kept my laptop keyboard from accidentally answering prompts the remote job would never receive. If you generate a tool locally and never rerun it without a TTY, you keep shipping a conversation instead of a command.

Hours 0–8: I blamed latency that was not there

I watched the process list and saw a Python interpreter that was not burning CPU, which I treated as idle generation. I added more logging around the cleanup function, reran the job, and waited like someone who refuses the obvious. The new prints did not show up until the job was killed, which made the hang look even more mysterious. I still wrote "the remote runtime is slow" in my notes, because that story felt easier than reading file descriptors.

Commands I kept repeating, none of which named the real bug yet:

PID=$(pgrep -n -f 'cleanup_cli.py')
ps -o pid,stat,wchan,etime,cmd -p "$PID"
ls -l /proc/$PID/fd
tr '\0' '\n' < /proc/$PID/environ | sort
Enter fullscreen mode Exit fullscreen mode

wchan was the clue I ignored for most of the first night. The process was sitting in I/O wait, not in a busy loop that would have implicated the model. fd/0 was already a pipe, not a terminal, and I still did not connect that fact to input().

Hours 8–24: extra prints made the silence worse

Python block-buffers stdout when it is not attached to a terminal, and a server job is usually not a terminal. My extra print() calls sat in a buffer, so remote logs stayed empty while the process was either working or blocked on a prompt. I then decided the logger was misconfigured, which wasted an evening on handlers and formatters that were never the bottleneck. Have you ever "fixed" a hang by adding more logs, only to make the hang look complete?

The smallest reproduction I wish I had run first is boring on purpose:

# save as buffered_prompt.py
import time

print("starting cleanup preview")
time.sleep(2)
answer = input("Type yes to continue: ")
print(f"got {answer!r}")
Enter fullscreen mode Exit fullscreen mode

On a laptop with a TTY, this is a conversation. You see the first line, you type yes, and the process exits. On a remote job with an inherited pipe, it is a trap that looks like a freeze.

# TTY path — this is the lie your laptop tells you
python buffered_prompt.py

# unbuffered prints, but stdin still closed immediately
python -u buffered_prompt.py < /dev/null
Enter fullscreen mode Exit fullscreen mode

< /dev/null fails fast with EOFError, which is annoying and therefore honest. The hang I actually hit is closer to an open pipe that nobody writes to and nobody closes:

# save as hang_demo.py — run locally; this is a recipe, not a benchmark
import subprocess
import sys
import time

proc = subprocess.Popen(
    [sys.executable, "-u", "buffered_prompt.py"],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True,
)
time.sleep(8)
print("still running?", proc.poll())
print("stdout so far:", proc.stdout.read() if False else "<not reading yet>")
proc.kill()
Enter fullscreen mode Exit fullscreen mode

If you never write to stdin and never close it, input() waits forever. The job is not frozen. It is polite. It is waiting for a human who does not exist on that box.

Hours 24–36: stdin was the waiter, not the model

When I finally read /proc/$PID/fd/0, the link target was a pipe. input() had not gotten slow, and the free server had not gotten stuck in some mysterious runtime pause. The generated CLI had a confirmation prompt because destructive cleanup feels scary, and scary tools ask questions. Remote jobs do not answer questions. They inherit file descriptors and then sit on them.

I also found the usual cousins of input() hiding in the same file:

  • getpass.getpass() also blocks, and it is even quieter about what it wants.
  • Library helpers such as confirmation prompts will do the same thing unless you pass an explicit flag.
  • A print() without flush=True can hide the prompt entirely when stdout is a pipe.
  • yes | python cleanup_cli.py on my laptop had been answering for me without my noticing.

That last one still makes me wince. I had a shell alias that piped yes into maintenance scripts, which is why the laptop path never taught me anything useful. The remote job did not have my alias. Why would it?

Hours 36–48: a contract that fails closed

I stopped asking the process to chat. The CLI now requires an explicit --yes for destructive work, and it refuses to start if stdin looks like a conversation the environment cannot finish. The guard is small enough to paste, and it is mean on purpose.

# save as noninteractive.py
from __future__ import annotations

import os
import sys


class InteractiveStdinError(RuntimeError):
    """Raised when a CLI would block on a prompt nobody can answer."""


def require_noninteractive(*, allow_env: str = "ALLOW_TTY_PROMPTS") -> None:
    if os.environ.get(allow_env) == "1":
        return
    if sys.stdin is None or not sys.stdin.isatty():
        return
    raise InteractiveStdinError(
        "stdin is a TTY; refusing to start because this command must run "
        "without prompts. Pass --yes, or set ALLOW_TTY_PROMPTS=1 only in a "
        "real terminal you intend to sit at."
    )


def require_explicit_yes(argv: list[str]) -> None:
    if "--yes" not in argv:
        raise SystemExit(
            "refusing destructive work without --yes "
            "(this process will not call input())"
        )
Enter fullscreen mode Exit fullscreen mode

Wire it at the top of the entrypoint, before any preview logging that you still want to see in captured output:

# save as cleanup_cli.py
import sys
from noninteractive import require_explicit_yes, require_noninteractive

def main(argv: list[str]) -> int:
    require_noninteractive()
    require_explicit_yes(argv)
    print("cleanup start", flush=True)
    # ... do the work with no input() calls ...
    print("cleanup done", flush=True)
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1:]))
Enter fullscreen mode Exit fullscreen mode

The flush=True is not decoration. If you only remember one remote-logging habit from these notes, remember that one. PYTHONUNBUFFERED=1 and python -u are the process-wide version of the same idea, and I now set them on every non-interactive job:

export PYTHONUNBUFFERED=1
python -u cleanup_cli.py --yes
Enter fullscreen mode Exit fullscreen mode

A test that fails if somebody sneaks input() back in

This is the artifact I would keep even if I threw the rest of the notes away. It does not measure speed. It only proves the process cannot sit on a pipe.

# save as test_cleanup_cli_noninteractive.py
import subprocess
import sys
import time
from pathlib import Path

CLI = Path(__file__).with_name("cleanup_cli.py")

def run(args, *, write_stdin=None, timeout=3):
    proc = subprocess.Popen(
        [sys.executable, "-u", str(CLI), *args],
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
    )
    try:
        out, _ = proc.communicate(input=write_stdin, timeout=timeout)
    except subprocess.TimeoutExpired:
        proc.kill()
        raise
    return proc.returncode, out

def test_refuses_without_yes():
    code, out = run([])
    assert code != 0
    assert "--yes" in out

def test_does_not_hang_on_open_pipe():
    started = time.monotonic()
    code, out = run(["--yes"], write_stdin=None, timeout=3)
    elapsed = time.monotonic() - started
    assert elapsed < 3
    assert code == 0
    assert "cleanup done" in out
Enter fullscreen mode Exit fullscreen mode

If a future edit reintroduces input(), that second test should trip TimeoutExpired instead of going green. That is the whole point. Green and silent is how I lost the first day.

Decision table I wish I had on day one

What you observe What I assumed What I would check first
Job stays running, CPU near zero Model or server is slow wchan, /proc/$PID/fd/0, any input() / confirm helper
No logs until the process dies Logging stack is broken TTY vs pipe buffering, python -u, flush=True
Works on the laptop only Path, deps, or "the free server" Aliases like `yes \
{% raw %}EOFError immediately The script is buggy stdin is /dev/null; this is the honest failure
Hang with an open pipe Deadlock in my own locks Somebody opened stdin=PIPE and never wrote or closed

What broke, and what I would repeat

Here is the short list I actually reused the next time a generated CLI touched a remote box:

  1. Treat every input(), getpass(), and confirm helper as a production incident waiting for a missing TTY.
  2. Require --yes for destructive work, and keep the flag out of default generated snippets if you can.
  3. Run the entrypoint once with stdin=PIPE and a three-second timeout before you trust a remote job.
  4. Force unbuffered stdout on non-interactive runs, because delayed prints impersonate a freeze.
  5. Read /proc/$PID/fd before you invent a story about model latency.

Would I still let a model draft the helper functions? Yes, because the draft is cheap and the review is the real work. Would I ship that draft after a laptop run where I typed yes like a helpful protagonist? Not again. The laptop is an accomplice. It answers questions the server will never hear.

Limitations, and who should skip this

This approach is rude to tools that are supposed to be conversations. If you are building a REPL, a debugger frontend, or a break-glass command that must ask a human in a real terminal, do not fail closed on a TTY. isatty() is also not a moral truth: some CI systems attach a pseudo-terminal, and some local runs redirect stdin without meaning any harm.

The guard does not prove the cleanup is correct. It only proves the process will not sit forever waiting for a keystroke. It will not save you from a missing dependency, a wrong working directory, or a prompt hidden inside a library you did not read. Windows console behavior differs from /proc, so copy the tests, not the /proc folklore, if that is your environment.

If your job is actually stuck in a network wait, this whole narrative will waste your time. Check wchan and the file descriptors first, then decide whether stdin is even in the story.

I would repeat the fail-closed contract, the unbuffered stdout flag, and the three-second pipe test. I would not repeat the part where I narrated a frozen model while a prompt sat on a pipe. If you already replay generated CLIs on a remote box, steal the guard and the timeout test; the rest of these notes can stay in the graveyard with my first-day theories.

Top comments (0)