Your SSH Command Timeout Is Probably a Budget, Not an Inactivity Window

TL;DR — I spent an evening chasing a deploy that kept dying with "command produced no output for 30s and was abandoned." The command was producing output the whole time. Turns out phpseclib's curTimeout is a wall-clock budget for the whole exec() call, not an inactivity window — it only ever counts down, and arriving data never resets it. The fix is two-fold: run exec() in callback mode and re-arm the timeout on every chunk, and wrap genuinely silent remote commands in a heartbeat.

The setup

I'm building a control plane that provisions Linux nodes over SSH — install packages, install a Node toolchain, install Docker, build an app on the box. All of it runs from PHP through phpseclib, because a long-lived agent on every node is a much bigger thing to own than an SSH connection.

Long-running remote commands have one nasty property: you can't tell "still working" from "hung" without output. So the agent has a rule I like — silence is a failure, never a success:

// A read timeout is a FAILURE, never a success. phpseclib returns the
// partial output and no exit status when the channel goes quiet.
if ($ssh->isTimeout()) {
    throw new NodeCommandTimeout("Command produced no output for {$this->timeout}s and was abandoned.");
}

That rule exists because of an older bug where a quiet apt-get install got abandoned half-way and the pipeline recorded it as success. Failing loudly on silence was the right call.

Failure one: silence really does happen

First live failure: "Installing the node toolchain failed: command produced no output for 30s."

My model at the time was "apt is chatty, its own progress lines keep the channel alive, so just don't pass -qq". That model is wrong, and it's worth spelling out why, because the same reasoning shows up everywhere:

• apt waiting on the dpkg lock is completely silent. Not "quiet" — silent.
• One large download can go 30+ seconds between progress lines.
• dnf -q prints nothing at all until it's done.

A tool's own chatter is a side effect, not a contract. If your liveness signal is something you don't control, you don't have a liveness signal.

So: generate one. A small POSIX-sh wrapper that backgrounds a heartbeat, runs the command, kills the heartbeat, and — this is the part people forget — preserves the wrapped command's exit status:

final class ShellKeepalive
{
    public static function wrap(string $command): string
    {
        return '( __i=0; while :; do sleep 1; __i=$((__i+1)); [ "$__i" -ge 10 ] || continue; '
            .'__i=0; echo "[keepalive] still running"; done ) & __ka=$!; '
            .'{ '.$command.'; }; __st=$?; '
            .'kill "$__ka" 2>/dev/null; '
            .'exit "$__st"';
    }
}

Two details that cost me more thought than the rest of it:

POSIX sh, not bash. These strings run under sudo -n sh -c on freshly provisioned boxes. Assuming bash on a minimal image is how you get a "syntax error near unexpected token" at 2am.

One-second ticks, echo every tenth — not sleep 10. kill reaches the subshell, but it does not reach a sleep already running inside it. If the heartbeat is sleep 10, a stray sleep can hold the channel's stdout open for up to ten extra seconds after every wrapped command. Tick at one second and the worst-case tail is one second.

That's the kind of thing you want a real test for, not a mental model:

it('returns promptly — the heartbeat does not hold the channel open after the command', function () {
    $start = microtime(true);

    $p = Process::fromShellCommandline(ShellKeepalive::wrap('true'));
    $p->setTimeout(10)->run();

    expect(microtime(true) - $start)->toBeLessThan(4.0);
});

it('preserves the wrapped command exit status under real sh', function () {
    $fail = Process::fromShellCommandline(ShellKeepalive::wrap('false'));
    $fail->setTimeout(10)->run();

    expect($fail->getExitCode())->toBe(1);
});

Note these run the string under a real shell via Symfony Process. Asserting str_contains($command, 'sleep') would have passed happily on a wrapper that swallows the exit code. If you're generating shell, execute the shell in your tests.

Failure two: the timeout was never about silence

Heartbeat shipped. Same deploy. Same error. Different command — this time a perfectly healthy npm ci running at --loglevel=info, streaming output continuously, abandoned mid-stream with a message insisting it had produced no output.

Which was the clue. The message and the reality disagreed, so the message was built on a false premise.

Here's the phpseclib mechanic, and I don't think it's widely known:

curTimeout is set once, when exec() opens the channel, from whatever you passed to setTimeout(). From then on it only depletes. Data arriving on the channel does not reset it.

So $ssh->setTimeout(30) doesn't mean "abort after 30 seconds of inactivity". It means "abort this exec() call after 30 seconds, full stop." It's a total-runtime budget wearing an inactivity window's error message. Every command longer than 30 seconds was dead on arrival, however loud it was — which also explains why the heartbeat changed nothing. The clock was never listening.

The fix is that setTimeout() resets curTimeout. Switch to exec()'s callback mode and re-arm on every chunk:

private function execOnce(string $host, string $command): CommandResult
{
    $ssh = $this->connect($host);

    // phpseclib's curTimeout is a WALL-CLOCK BUDGET that depletes even while
    // output is streaming in. setTimeout() resets it, so re-arming on every
    // chunk is what turns the budget into the inactivity threshold the error
    // message has always claimed: only {timeout}s of genuine SILENCE aborts.
    $stdout = '';

    $ssh->exec($command, function (string $chunk) use (&$stdout, $ssh): void {
        $stdout .= $chunk;
        $ssh->setTimeout($this->timeout);
    });

    // ...
}

Now 30 seconds means thirty seconds of actual silence. And the heartbeat stays — it isn't redundant, it's what feeds the re-armed window when the remote command genuinely has nothing to say.

The bit worth keeping

Three failures, one root cause, and the thing that made it take an evening instead of ten minutes was that I had written the wrong model down and then trusted my own note. "apt's chatter is the keepalive" was in my project docs, in a code comment, load-bearing. Every subsequent debugging step inherited it.

Two takeaways I'd hand to anyone doing remote execution from an app:

1. Check what your timeout actually measures. "Timeout" is one word for at least three different clocks: connect timeout, total-runtime budget, inactivity window. Libraries mix them freely and the error message is often aspirational. Read the source, or prove it with a long chatty command.
2. Don't rely on someone else's output as your liveness signal. If a stalled command and a working command look identical on the wire, generate the difference yourself.

And when a bug's error message contradicts what you observed — the message is a claim, not evidence. Go check the claim first.