On September 14 Temporal announced a $550M Series E at a $12.55B valuation. The product is Durable Execution: workflow state outlives a crashed process, so a multi-step agent resumes where it stopped. The announcement quotes OpenAI's VP of Infrastructure, Venkat Venkataramani, calling it "more than ever a core requirement for modern AI systems", and CEO Samar Abbas: "Reliability has never been optional, but AI has quickly raised the cost of skipping it."
Durable execution answers one question: what happens to the work when the process dies. It leaves a second one to everybody else. How do the observers find out? A process that exits cleanly can say so. A machine that loses power says nothing, and from upstream that silence is identical to a dropped Wi-Fi connection.
We shipped shell.online 0.17.0 on September 18, and most of the release is about that second question. shell.online puts a running terminal (a coding agent, a build, a remote shell) behind a browser link, and the browser sorts sessions into a Write column, meaning live and typeable, and a Finished column.
Nobody closed a session whose machine died
Until 0.17.0 a session was closed by the CLI as its process exited. A reboot or a power cut never runs that step. The relay only sees the host socket go away, which is also what a network drop looks like. So the session sat in Write, offered as something you could type into, until the relay expired it, which the 0.17.0 changelog entry puts at twelve hours.
The fix lands in three places, and each one uses only what that place can know.
The machine. The CLI keeps a JSON record per session. A record whose control socket stopped answering used to be deleted on the spot, which threw away the only local evidence the session existed. Now scanLocalSessions stamps the record with AbandonedAt, blanks the stored browser password (closing needs the id and nothing else), and keeps it for up to 24 hours. On the next daemon start, before taking any work, reclaimAbandonedSessions in cmd/shell/reclaim.go closes each one in the account. The whole catch-up is bounded at 15 seconds so it cannot become the reason a daemon fails to start. It also sends no exit code. The comment in the source: "Nobody saw the process end, and inventing a 0 or a 1 would put a result in the record that no one observed."
The relay. /api/sessions/:id now returns host_last_seen_at, the last moment the relay held the host socket. That one timestamp is what lets a caller tell a blip from a machine that is gone.
The service and the browser. When the relay answers missing or exited, the accounts service writes closedAt, dated from hostLastSeenAt when it has one, because the moment the service noticed is always later than the moment it happened. A disconnected session gets different treatment. The browser computes "Machine gone" once the host has been silent for HOST_GONE_MS, which is ten minutes. That state is derived on the client and never stored, so a laptop that wakes up and reconnects goes live again on the next poll. Persistent sessions are excluded, since waiting to be resumed is what they are for.
A 401 outranks the local clock
A machine shows as online for one reason: the daemon polls /api/agent/commands every two seconds, and the service treats 15 seconds without a poll (AGENT_ONLINE_MS) as offline. Two bugs stopped that poll on healthy machines.
Renewal starts 60 seconds before the access token expires. A renewal that failed, say on a laptop waking into a flaky network, stood the poll down for the retry backoff, which caps at 60 seconds, even though the token in hand still worked. The backoff was only reset by a successful renewal, so one blip left the machine renewing late, and dropping offline, every cycle afterwards. In 0.17.0 the poll continues while renewal retries on its own schedule, and a successful poll clears the backoff.
The second bug is the more general lesson. Only the local clock could start a renewal. A machine with a slow clock, or one whose sign-in had been retired by a later login, presented the same dead token every two seconds for as long as it stayed up. In cmd/shell/agent_loop.go a 401 from the poll now sets a refused flag and renews immediately. The comment on that flag: "The expiry it was handed is a claim about a clock, and this is what actually happened." The loop slows down only when both the poll and the renewal have been refused, and it prints the shell login instruction once instead of going quiet.
A fixed tiebreak is a session that never gets checked
One more fix is on main but not yet tagged (PR #186, under Unreleased in the changelog). The commit message describes what was seen in production: three open sessions, one reported "Status unavailable" by every request, while curl against the relay for the same id returned "connected".
The service seeded its relay view two sessions per request. Each request may land on a worker instance with an empty cache, and the order was decided by session start time. Every cold instance therefore picked the same two, and the third was checked by nobody. In app/server/lib/session-liveness.ts a batch now covers eight, and equally stale sessions are ordered by Math.random(). The limit that actually protects the relay is unchanged: 30 checks a minute, against the 120 per address the relay allows. The source comment states the rule plainly: "Any fixed tiebreak is a session that is never chosen."
The same ambiguity one layer down
Pilot Protocol runs encrypted UDP tunnels through NATs, where silence is just as hard to read. In pkg/daemon/tunnel.go, TunnelKeepaliveInterval is 25 seconds, set below the 30-second lower bound of consumer NAT UDP idle timeouts. Because peers with an established crypto session exchange keepalives both ways at that interval, the path watchdog can treat sustained inbound silence as a dead path and not as idleness. It is the same move as host_last_seen_at: guarantee a rhythm, date the last beat, and silence becomes evidence.
0.17.0 also covers the restart itself. The first time you agree to browser-started sessions during shell login, the CLI installs the daemon as a user service (a LaunchAgent on macOS, a systemd user unit on Linux). A machine that reboots comes back without anyone logging in, runs the reclaim step, and closes the sessions it dropped.
Top comments (0)