The backoff ladder was not the bug, the key was: per host, not per search

Notifio watches rental search result pages from the user's own machine and emails them when something new appears. Sites that do not want to be read say so, and the app has to decide how long to leave one alone afterwards.

I have written about the ladders themselves before, in One backoff ladder became four, and the first retry went from ten minutes to sixty seconds. That post was about the waits being too long. This one is about something I got wrong in the same module and did not notice until the waits were short enough for the real problem to become visible: the registry was keyed by the wrong thing.

What the key was

A user adds searches, not sites. The settings file is a list of URLs, the poll loop iterates that list, and so the first version of the backoff registry did the obvious thing and keyed its entries by search URL. One entry per thing the loop knows about.

It is a very natural mistake, and here is the module doc comment that replaced it:

/**
 * Keyed by host, not by search URL. A refusal comes from the site, so it
 * applies to every search on it. Keying by URL meant five searches on one site
 * each kept their own counter, and each went back to knocking while the others
 * were standing down: both why we stayed blocked and why recovery was slow.
 */

The failure has two halves and they pull in opposite directions, which is why it took a while to see as one bug.

Half one: we stayed blocked. Users do not spread fifteen searches across fifteen sites. They pick a city and watch one portal with five different price ceilings and neighbourhoods. When that portal refused search one, search one stood down and searches two through five walked straight into the same wall on the same cycle, each earning its own first rung. The site saw four more requests immediately after refusing us, which is precisely the behaviour the backoff exists to prevent.

Half two: recovery was slow. The counters desynchronised. Search one was on rung three while search five was on rung one, so there was no moment at which the host was being left alone and no moment at which it was all tried again. Searches trickled back in at staggered times, each one capable of refreshing the refusal for itself while the others were still waiting, and the host never cleanly cleared.

Keying by host collapses both: one entry, one counter, one nextRetryAt, one shared decision for every search on that domain. A refusal is a fact about the site, so it belongs in a map keyed by the site.

The four ladders

With the key fixed, the ladders can be per kind of refusal rather than one pessimistic ladder for everything:

export const LADDERS: Record<BackoffKind, number[]> = {
  challenge:    [60_000, 180_000, 480_000, 900_000, 1_800_000],
  rate_limited: [120_000, 300_000, 900_000, 1_800_000],
  denied:       [300_000, 900_000, 1_800_000, 3_600_000],
  error:        [30_000, 120_000, 300_000],
};

The last entry in each is the ceiling, so the ladder is a list and not a formula. Four kinds, four different shapes, and the reasoning for each is written next to it in the source rather than left for a future reader to reverse engineer from the numbers.

A detail that matters more than it looks: a change of kind resets the count rather than inheriting it.

const previous = this.entries.get(host);
const failures = previous && previous.kind === kind ? previous.failures + 1 : 1;

/**
 * A different kind of failure starts its own ladder rather than inheriting a
 * count from an unrelated one: a network blip should not be punished with the
 * wait earned by three refusals.
 */

Without that, a host that refused us three times and then had a dropped connection would be handed the fourth rung of the error ladder for something that was our fault, not theirs.

Jitter, and why the key change made it necessary

export function delayFor(
  kind: BackoffKind,
  failures: number,
  retryAfterMs?: number,
  random: () => number = Math.random
): number {
  const ladder = LADDERS[kind];
  const step = ladder[Math.min(Math.max(failures, 1) - 1, ladder.length - 1)];
  const jittered = Math.round(step * (0.85 + random() * 0.3));
  return Math.min(Math.max(jittered, retryAfterMs ?? 0), MAX_WAIT_MS);
}

Plus or minus fifteen percent, and the reason is specific to how the poll loop works:

/**
 * Jitter matters because several searches usually share a cycle: without it,
 * every host blocked in the same cycle comes back in the same instant, which
 * is the burst that got them blocked in the first place.
 */

Note the asymmetry in that return statement. Math.max against retryAfterMs means the site's own Retry-After wins whenever it asks for longer than our ladder would, and never when it asks for less. We will wait longer than we planned if asked politely, and we will not be talked into waiting less. Then Math.min caps the whole thing:

/** Never wait longer than this, whatever a site's Retry-After claims. */
export const MAX_WAIT_MS = 60 * 60_000;

An hour. A header is remote input, and a Retry-After of eighty six thousand seconds is not something a monitoring product can honour and still be a monitoring product. The product promise is thirty seconds; an hour of silence is already the worst thing we are prepared to do to a user, and there is no value in being more obedient than that.

One function the poll loop asks for the next deadline

/** When the next waiting host comes due, or null if none are waiting. */
soonest(): number | null {

This is what lets the UI render a countdown per search rather than a spinner that means nothing, and it is only answerable cheaply because the entries are per host. With per URL entries, "when does this search come due" had to consider every sibling search on the same domain, because any of them could refresh the refusal.

The reason it is its own module

/**
 * Its own module for two reasons. It is the policy most worth being able to
 * read in one place, and it is pure: no browser, no Electron, no clock beyond
 * what is passed in, so the ladders below can be checked by running them
 * rather than by reasoning about them.
 */

delayFor takes its randomness as a parameter and record takes now as a parameter, both with defaults, so the whole policy can be driven from a script with a fixed clock and a fixed random:

// Mid-range jitter, so a step comes back as exactly itself.
const noJitter = () => 0.5;
const NOW = 1_700_000_000_000;

That runs as pnpm test:backoff and prints the four ladders before it asserts anything, which turned out to be the most useful part. A list of numbers in a source file is hard to sanity check; the same numbers printed as 1m → 3m → 8m → 15m → 30m are immediately either reasonable or not. The constraint that makes this possible is that testable files in the app may not import Electron, which I wrote about in Our Electron app has no test framework, and the only rule is that testable files may not import electron.

What the user sees

The user side of the host key is covered in When a block clears, only one search goes back in: once the entry is per host, a host coming out of a wait can be given a single probe search instead of all five at once, which is a thing that is simply not expressible when each search owns its own counter.

Which sites this actually comes up on is written per site, with citations, on the supported sites pages. Facebook Marketplace needs a logged in session and is the one that drops you most often, Idealista and ImmobilienScout24 are the ones where this policy earns its keep, and the help page covers what the app asks you to do when a session expires rather than when a site refuses us.