We run SSH traffic through a Cloudflare Durable Object. Each user's machine holds one outbound control WebSocket to the DO; when a phone wants in, the DO tells the machine to dial back. This is a standard hub pattern built on the WebSocket Hibernation API.
It worked, then it started failing, and every signal we had said the system was healthy.
The symptom
Phone connects to /connect/:machineId. The socket opens, then immediately closes with 1013. Meanwhile the machine that is supposedly offline believes it is connected:
- TCP is
ESTABLISHED - The WebSocket's protocol-level ping (
pingInterval = 15s) succeeds, every time - The app has no error to report
Only restarting the machine's app fixed it.
There is exactly one place in our Worker that emits 1013:
ws.close(1013, "machine agent offline");
That line runs when getWebSockets("agent") returns nothing for that machine. So the DO did not have the socket.
Why
Under Hibernation, the Cloudflare runtime answers protocol-level ping/pong on your behalf, without waking the Durable Object.
This is documented, intentional behavior. The entire point of Hibernation is to stop billing GB-seconds for an idle object, and waking it up to say "yes, still here" would defeat that.
But it means this state is reachable:
| Layer | State |
|---|---|
| TCP connection | alive |
| WebSocket protocol ping/pong | succeeds (answered at the edge) |
| DO instance still holding this socket | lost |
The machine decided it was connected based on row 2. To deliver a dial, the DO needs row 3.
This is not the TCP blackhole case, where the connection is dead and everything stalls. Here TCP is genuinely fine and the protocol ping is succeeding. The health check is correct. It is answering the wrong question.
The fix
The fix is an application-level round trip, where the pong is produced by your own code:
async webSocketMessage(ws: WebSocket, msg: string | ArrayBuffer) {
const role = this.roleOf(ws);
if (role === "agent") {
// Control frames on this socket are text. Do not hand a non-string to
// JSON.parse. `JSON.parse(typeof msg === "string" ? msg : "")` throws
// SyntaxError on the empty string — ask me how I know.
if (typeof msg !== "string") return;
try {
const control = JSON.parse(msg) as { op?: unknown };
if (control.op === "ping") {
// The point is that THIS code ran. That is the signal.
safeSend(ws, JSON.stringify({ op: "pong" }));
}
} catch {
// Drop malformed control frames silently. A heartbeat that becomes a new
// reason to disconnect is worse than no heartbeat.
}
return;
}
this.forward(role, msg);
}
The machine sends {op:"ping"} every 45s and reconnects after two missed cycles (100s, so two periods plus slack).
Do not reach for setWebSocketAutoResponse() here:
// This puts the blind spot right back.
this.ctx.setWebSocketAutoResponse(
new WebSocketRequestResponsePair("ping", "pong")
);
That API exists specifically to answer without waking the DO, which makes it exactly wrong as a liveness probe, for the same reason protocol ping was: the reply proves the edge is up, not that your object still holds the socket.
Shipping the heartbeat before the Worker
We shipped the machine agent with the new heartbeat while the old Worker was still deployed. The old Worker ignores {op:"ping"}, so no pong ever arrives. An agent that treats silence as death then severs a perfectly healthy connection every 100 seconds and keeps reconnecting until the Worker rolls out.
The guard is one clause:
bool controlHeartbeatExpired({
required bool pongSeen,
required Duration sinceLastInbound,
}) => pongSeen && sinceLastInbound > kControlPingTimeout;
pongSeen && reads as redundant: if it timed out, surely it is dead. Delete it and you get the reconnect storm above, but only during the window where client and server versions disagree. Staging, where you deploy both together, will not reproduce it. We pinned it with a test so nobody deletes it for looking useless.
If your client and server deploy independently, decide explicitly what happens when the peer does not answer your new protocol.
What protocol ping is still good for
We did not remove it. Data connections still run pingInterval = 15s.
Mobile NAT reclaims idle mappings within tens of seconds, and after that nothing arrives and nothing errors. A terminal just sits on its last frame forever. Edge-proxied pongs detect this perfectly well, because if TCP were actually dead the proxied reply could not reach you either.
| Question | Right probe |
|---|---|
| Is the TCP/NAT path alive? | protocol ping is enough |
| Does the DO still hold this socket? | application round trip required |
Data connections kept pingInterval = 15s. Only the control socket got the application round trip.
This came out of building Otter Beam, which lets you reach the tmux sessions on your own machine from a phone. The relay only forwards the encrypted SSH stream; it cannot read it.
Top comments (0)