DEV Community

Cover image for It didn't crash. It just wouldn't wait.
Kane
Kane

Posted on

It didn't crash. It just wouldn't wait.

Summer Bug Smash: Smash Stories 🐛🛹

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

StableRoute is a liquidity router on Stellar. The frontend is a Next.js app: quotes, pairs, admin controls, webhooks, audit logs. Most of it is forms and lists. The stats page is different. It is the room where you look at the protocol and ask a simple question: is it alive?

That page used to look like this.

useEffect(() => {
  const id = setInterval(refetch, POLL_MS);
  return () => clearInterval(id);
}, [refetch]);
Enter fullscreen mode Exit fullscreen mode

Five seconds. Hit /api/v1/stats. Paint the pair count. Paint whether the router is paused. Do it forever. It is the kind of code you write when you want the dashboard to feel live, and you do not yet know what "live" costs.

I found out on a bad afternoon, not a dramatic one. The backend got slow. Not down, just sick. Latency climbed past five seconds. The stats page did not wait. setInterval does not care whether the last request came home. It fires on the clock.

Every five seconds refetch bumped a reload key. The fetch effect tore down the previous request, marked it cancelled, and started another. If the server needed eight seconds, the client never kept a request long enough to finish. The dashboard sat there asking, cancelling, asking again. From the server's side it looked like a client that would not stop poking a bruise.

When the API actually failed, it got worse. A 500 came back fast. The interval did not slow down. It kept hitting every five seconds. The UI flipped from error to loading to error. The alert unmounted and remounted. The page flickered. Anyone with the tab open became part of the outage.

That is the bug. It did not throw. It did not panic a contract. It just refused to be kind when the system needed kindness.

The fix was to stop treating time as the thing that drives the network.

I threw out setInterval and wrote useBackoffInterval. The idea is small enough to hold in your head. A poll is allowed to schedule the next poll only after the current one has settled. Settled means success or error. While status is idle or loading, nothing is scheduled. One request in flight. No pile-up. No cancelled work eating the only response you were going to get.

Failures remember themselves. A ref counts them, not React state, so counting a failure does not itself retrigger the effect. First error waits ten seconds. Then twenty. Then forty. Then it stops at sixty. The formula is the boring one everyone should have used from the start:

delay = Math.min(baseMs * 2 ** failureCount, maxMs)
Enter fullscreen mode Exit fullscreen mode

The first success zeros the counter. The cadence comes back to five seconds without anyone pressing Retry. The page still has a Retry button, because a human should be able to say "try now" without waiting for the backoff clock. Automatic recovery is not the same as trapping someone in a delay.

There was a quieter bug inside the quieter bug. If you put callback in the effect deps, every new function identity kills the timer and starts another. Polling becomes jitter. If you freeze the callback, you poll with a stale closure. The way through is the unfashionable one: keep the latest callback in a ref, update it every render, and let the effect depend on status and timing, not on the function you plan to call.

const callbackRef = useRef(callback);
callbackRef.current = callback;

useEffect(() => {
  if (status !== 'success' && status !== 'error') return;
  // ... count, compute delay, schedule callbackRef.current
  return () => cancel(timeoutId);
}, [baseMs, cancel, maxMs, schedule, status]);
Enter fullscreen mode Exit fullscreen mode

Unmount clears the timeout. I tested that by leaving, advancing fifteen seconds, and counting fetch calls. It stayed at one. That is the whole point of cleanup. Most people write it. Fewer people prove it.

The tests are the part I am actually proud of. I did not want to sit in a browser and wait for a minute of backoff. The hook takes schedule and cancel as arguments. In production those are setTimeout and clearTimeout. In Jest they are mocks. The test can say: you were told to wait 10,000ms, then 20,000, then 40,000, then 60,000, then 60,000 again. It can say: after an error, a success puts you back on 5,000. It can say: if the callback identity changes before the timeout fires, you call the new one, not the old one. It can say: idle and loading schedule nothing.

That is how the page stopped being a liability.

When the router is healthy, the tiles still update every five seconds. The "Updated just now" label still ticks. When the router is not healthy, the page tells you so once, holds the alert still, and backs away. The copy is honest: retrying automatically, with a longer delay, while the service is unavailable. It is not pretending everything is fine. It is also not kicking the thing that is already down.

I would not call this legendary. I would call it the work that makes software feel like it was written by someone who has been on both sides of a slow endpoint. The stats page is a small surface. It is also the first place an operator looks when they are worried. If that page panics, the rest of the product feels like it is panicking with it.

The win was not a clever algorithm. The win was that the dashboard learned how to wait.

Top comments (0)