The setup
We recently added continuous peer syncing to Quantum-Lattice's node — instead of checking for missed blocks only once at startup, nodes now re-check periodically in the background, so they recover automatically after any brief network interruption.
Testing it live went well at first: a test node correctly caught up on a real gap, pulling and applying dozens of missing blocks over a genuine internet connection. Then... nothing. No errors. No crash. Just silence, for 15 minutes straight, while the real chain kept advancing without it.
The investigation
The node was still alive — CPU usage was normal, no panic in the logs. That ruled out a crash. The periodic task was clearly supposed to run every 60 seconds indefinitely:
\rust
tokio::spawn(async move {
loop {
catch_up(...).await;
tokio::time::sleep(Duration::from_secs(60)).await;
}
});
\\
Nothing about that loop looks wrong on its own. The bug wasn't in the loop — it was in what the loop was waiting on.
The actual bug
Inside catch_up(), the function that opens a connection to a peer and asks "what's your current height?" had no timeout at all:
\rust
let mut stream = TcpStream::connect(peer).await.ok()?;
\\
If that connection attempt ever hangs — doesn't cleanly succeed, doesn't cleanly fail, just sits there — the .await never resolves. The loop isn't broken; it's just permanently parked on one single call that will never return. It worked once because that first connection happened to succeed quickly. Some time later, one connection attempt didn't, and the entire background task quietly stopped forever.
The fix
Wrap the connection attempt (and the subsequent read) in an explicit timeout:
\rust
let mut stream = tokio::time::timeout(
Duration::from_secs(10),
TcpStream::connect(peer),
).await.ok()?.ok()?;
\\
Now a hung attempt fails cleanly after 10 seconds instead of blocking forever, the loop proceeds to its next sleep-and-retry cycle, and a temporary network hiccup becomes a temporary hiccup — not a permanent, silent failure.
The lesson
Every .await in a loop meant to run forever is a place where "forever" can quietly mean "until this one specific call never returns." We'd already added timeouts to every inbound connection this node handles — but the outbound calls this same node makes to check on its peers had been overlooked entirely. Worth explicitly auditing every unattended, long-running loop for exactly this: not "can this fail?" but "can this specific await simply never resolve at all?"
Confirmed fixed with a real, live re-test — the same node now recovers correctly and repeatedly, watched over several consecutive cycles rather than assumed from a single pass.
Top comments (0)