Our retry ladder was keyed by the wrong thing

Last week I wrote about polling rental sites every thirty seconds without getting walled off, and I showed the backoff we used when a site pushed back:

const _backoff: Record<string, { failures: number; nextRetryAt: number }> = {};
const BACKOFF_BASE_MS = 10 * 60_000; // first backoff: 10 minutes
const BACKOFF_MAX_MS  = 60 * 60_000; // capped at 60 minutes

I described it as keyed by site URL rather than global, which is better than global, and left it there. Since then the whole thing has been replaced, because it was wrong in three separate ways and each one showed up as a user visible symptom.

This is the rewrite, and what each mistake actually cost.

Mistake one: keying by search URL

Record<string, ...> keyed by the search URL looks fine until you remember that a user watches several searches on the same site. One search for a two bedroom, one for a studio, one for a different neighbourhood. Same host, three keys.

A refusal does not come from a search. It comes from the site. So when one of those three got refused, the other two carried on knocking at full rate, which is both why the site stayed unhappy and why recovery took so long: whichever search happened to retry first got refused again and restarted its own counter, while the other two were mid-wait, out of phase, doing the same thing a few seconds later.

The fix is one line of intent and it changes the behaviour completely:

/**
 * Keyed by host, not by search URL. A refusal comes from the site, so it
 * applies to every search on it.
 */
export class BackoffRegistry {
  private entries = new Map<string, BackoffEntry>();
  ...
}

Key your circuit breaker on the thing that failed, not on the thing you were doing when it failed. The site refused. The site is the key.

Mistake two: one ladder for every kind of failure

The old code had a single base and a single multiplier, so four unrelated things got the same treatment: an interstitial that clears itself, a site asking us to slow down, a flat refusal that needs a human, and our own network dropping.

They have nothing in common. Now each has its own ladder:

export const LADDERS: Record<BackoffKind, number[]> = {
  challenge:    [60_000, 180_000, 480_000, 900_000, 1_800_000],
  rate_limited: [120_000, 300_000, 900_000, 1_800_000],
  denied:       [300_000, 900_000, 1_800_000, 3_600_000],
  error:        [30_000, 120_000, 300_000],
};

An explicit array rather than a formula, because the thing I want to be able to answer is "how long will we wait the third time this happens", and reading that off a list is easier than evaluating base * 2 ** (n - 1) in my head and then remembering where the cap lands. The last entry is the ceiling, so the array is also the documentation.

The reasoning per row is worth spelling out, because the numbers are policy rather than maths:

• error is short because it is most likely our end, and there is nothing to be gained by punishing a site for our own network blip.
• challenge starts at a minute because by the time one is recorded, the scraper has already tried to wait it out, and these usually clear on their own.
• denied is the slow one, because it needs the user to go and pass a check by hand. Retrying hard achieves nothing except looking worse.
• rate_limited always yields to the site's own Retry-After when that asks for longer:

return Math.min(Math.max(jittered, retryAfterMs ?? 0), MAX_WAIT_MS);

A different kind of failure also starts its own count rather than inheriting one:

const failures = previous && previous.kind === kind ? previous.failures + 1 : 1;

A dropped connection should not be met with the thirty minute wait earned by three refusals.

Mistake three: no jitter, in a system that batches

Every search is checked inside one cycle. So if two hosts refuse us in the same cycle, and the ladder is deterministic, they come back at the same instant. That synchronised burst is a worse traffic shape than the one that got us refused in the first place, and it stays synchronised on every subsequent retry.

const jittered = Math.round(step * (0.85 + random() * 0.3));

Plus or minus 15%, and random is a parameter with a default so the ladders can be tested by running them rather than by reasoning about them:

export function delayFor(
  kind: BackoffKind,
  failures: number,
  retryAfterMs?: number,
  random: () => number = Math.random
): number

The module has no imports beyond a type. No browser, no Electron, no clock except the one passed in. That is the reason it is a module at all: the retry policy is the piece most worth being able to read in one place, and a pure one is a piece you can assert on.

The number that was really wrong

The old first step was ten minutes.

Ten minutes of not looking at a rental site is ten minutes of rooms going to somebody else, in a product whose entire promise is being early. That number existed because the old code treated any challenge page as a full block, so it had to be pessimistic about how often it would fire. Once challenges were handled as their own thing rather than as refusals, the pessimism had no reason to exist, and the first rung could drop to a minute.

It is worth checking your own constants for this shape. A number chosen to be safe under an assumption you have since fixed is not safe any more, it is just slow.

The user can overrule all of it

The retry clock is on screen, per search, counting down. And there is a button next to it:

/**
 * Stop waiting on a blocked site and try it again now.
 *
 * The user can see the countdown, and they are often looking at this because
 * they have just gone and passed the check in their own browser, so making
 * them wait out a timer we chose for them would be perverse.
 */
app.post('/api/monitor/retry', (req, res) => {
  monitor.clearBackoff(url);
  const started = monitor.checkNow();
  res.json({ ok: true, started });
});

This is the part I would push hardest on in a review of anyone's backoff code. Your ladder is a guess about a situation you cannot observe. The user sometimes can observe it, because they just went and dealt with it by hand. A guess that cannot be overridden by somebody who knows better is not caution, it is stubbornness.

Success clears the entry outright rather than decaying it, for the same reason:

/** Forget this host's failures. Returns whether there were any. */
clear(host: string): boolean {
  return this.entries.delete(host);
}

A backoff that keeps punishing you after the problem is gone is a backoff that has stopped measuring anything.

See it running

The per search retry countdown and the retry button are part of the monitoring UI, described on the help page. Each of the supported sites has its own page covering what watching it involves, and the app itself is at notifio.app/download.

If you read the earlier post and copied the ten minute constant, this is your correction. Sorry about that.