DEV Community

RescueShell
RescueShell

Posted on Originally published at rescueshell.com

A reconnect schedule is a thundering-herd problem

Our device agent has two reconnect schedules stacked on top of each other, and one guard behind
them. They look like tuning, but they set how the whole fleet meets the relay after an outage.

We have not published a measurement of a real relay restart, and this post does not claim one.
What follows is the design, read from the agent's code and checked with a simulation of the
library's backoff.

Schedule one: the socket library, with jitter

When the relay drops the connection, the socket library reconnects on its own first. The agent
asks for a 5 second first delay and a 60 second cap (session_manager.go:498-499). The delay
doubles on each attempt up to the cap. Each delay is randomised by up to half either way (the
library's default factor of 0.5, third_party/socket.io-client/constants.go:15-16), and then
clamped back into 5 to 60 seconds (Duration() in the library's backo2.go, v3.0.6).

That clamp is the part we got wrong. Every delay that jitter pulls below 5 seconds becomes exactly
5 seconds. We ran the library's own backo2.go 10,000 times with our settings. The first retry
landed between 5.0 and 7.5 seconds, and 5,005 of the 10,000 landed on exactly 5,000 ms. A relay
restart drops every socket at about the same moment, so about half the fleet's first attempts are
scheduled for the same 5 second delay, within whatever spread there was in noticing the drop.

The same thing happens from the other side at the cap. From the fifth retry on, the doubled delay
is already 60 seconds, jitter spreads it over 30 to 60, and the clamp puts about half on exactly
60 seconds: 5,011, 4,954 and 4,995 of 10,000 for the fifth, sixth and ninth retry. The second to
fourth retries are not clamped and are spread out.

There is one more random delay, and this one has no clamp. If the relay is shedding /init load
it answers temporarily_unavailable to an exchange, registration or token refresh
(init.socket.server.ts:243). The agent then waits a random 5 to 60 seconds and tries again, up
to three times (initBackoff, throttledRetries and retryWhileThrottled in
session_manager.go:47, :59-60, :811). That wait only matters for a device that runs an
/init exchange, and after a relay restart not every device does.

Which devices go back through /init

A device holds a session token for ten days (deviceJwtExpirationSeconds is 864000 in
src/config.json:52). The agent gets one in each Start() and hands it to the socket library
once. The library re-sends that same token on every reconnect, and the agent has no way to give
it a newer one (session_manager.go:501-506).

A connected device does also go back to /init every 24 hours
(refreshTokenTickerInterval, :84, scheduledRefresh, :630). But that rotates the stored
refresh credential only. The session token on the wire stays the one from the last Start(). The
relay checks its signature and expiry on every handshake (session.socket.server.ts:224), and a
rotation does not invalidate it, because the row the token names keeps its identity
(init.socket.server.ts:592-596). So after a relay restart the fleet splits in two:

  • Session token under ten days old. The library reconnects /session with the token it has. No /init exchange runs, so the clamped library delay is its only spread.
  • Session token older than that. The relay refuses it. The device shows as not connected, and the next 60-second tick runs Start() (main.go:272). That refreshes over /init inside retryWhileThrottled (session_manager.go:461-463), which is where the relay sheds load and the random 5 to 60 second wait spreads those devices out.

The age that counts is the time since the process last ran Start(), not since the device first
connected.

Schedule two: the agent's own loop, without jitter

Behind the library sits a plain loop in main.go (:21, :231-232):

  • 60 seconds. The agent checks its connection once a minute.
  • 5 quick retries. If the connection is still gone, it tears it down and rebuilds it from scratch on each of the next ticks, up to five times.
  • 30 minutes. After five rebuilds it waits until thirty minutes have passed since the last one, then starts over.

No jitter here, no exponential curve. Each agent's ticker starts when the agent starts, so ticks
land at different seconds for different devices. That is the only spread this loop has.

What each schedule protects

Now say the relay comes back slow, or falls over again under load.

The library schedule is the one the relay meets first. Its first wave is half a wall: about half
the fleet scheduled for exactly 5 seconds, the rest spread over the next 2.5. After that the doubling thins
it out, until the cap gives half of the stragglers the same 60 second delay again.

The outer loop guards against something else: a device rebuilding its whole connection every
minute, forever. Each rebuild closes the previous connection first, so the library's own retries
on it end there. After the fifth, the loop leaves the last rebuild standing and stays quiet for
thirty minutes. What happens next depends on how far that rebuild got. If it had already created
its /session socket, that socket keeps retrying at the capped, jittered pace and brings the
device back by itself. If the relay was unreachable at /init, setup never finished. The /init
socket may reconnect, but it does nothing more, because the connect handler returns until setup
is done (handleInitConnect, session_manager.go:717-719). That device waits for the
thirty-minute mark.

In short: after a few quick tries in the first minute, a device starts at most one
rebuild a minute, and five of them. The library path does move partly in step, because of the
clamp, and the outer loop has no jitter at all. Those are the two things we would change first if
the fleet grew: a lower floor or our own jitter for the library, and jitter for the loop.

A failure that never retries

One case skips the loop. If the relay says the device's credential is no good, the agent exits
with code 2 instead of retrying (ExitCodeAuthFailure, main.go:16, :223-225, :275-278).
Only a refusal that is not also marked transient counts (requiresCredentialReset,
session_manager.go:120). A device with a dead credential hammering the relay every minute would
be a herd of its own.

Why you cannot just tune it

Change a sixty-second cap to ten and you have not tuned one device. You have changed the load on
your relay from every device, at once, on the next release. You cannot roll it back from the relay,
because the devices you would need to reach are the ones running the change.

That is why we treat these numbers as a fleet contract, not a setting. Changing them is a
load test you plan.


RescueShell: €0.10 a device a month, billed yearly, at least €6 a year, plus VAT. Thirty-day
trial, card required, and the clock starts when your first device connects. Hosted in the EU,
exceptions listed.

Try it

First published on the RescueShell blog.

Top comments (0)