When a block clears, only one search goes back in

Notifio watches rental search pages from the user's own machine and emails them the moment a new listing appears. A user can have up to fifteen searches, and in practice several of those live on the same site: three Kamernet searches for three neighbourhoods, two Pararius searches at different price caps.

I wrote about keying the retry ladder by host rather than by search URL a few days ago. That post is about how long to wait after a site refuses us. This one is about the ten lines that decide what happens the moment that wait is over, which turned out to be the harder half.

The obvious version is a thundering herd you built yourself

Three searches on one host, one block, one ladder shared between them. The wait expires. All three are due. All three go out in the same cycle, to the same site, within a second or so of each other.

That is the exact request pattern that got us blocked. We waited five minutes specifically so we could do it again, harder, because now we have a backlog.

So the scheduler does not ask "which searches are due". It asks "which hosts are due", and then sends exactly one search per recovering host:

const byHost = new Map<string, Site[]>();
for (const site of enabledSites) {
  const host = hostOf(site.url);
  const list = byHost.get(host);
  if (list) list.push(site);
  else byHost.set(host, [site]);
}

const due: Site[] = [];
const probing = new Set<string>();
let waitingCount = 0;

for (const [host, list] of byHost) {
  const backoff = _backoff.get(host);
  if (!backoff) {
    due.push(...list);
    continue;
  }
  if (backoff.nextRetryAt <= now) {
    probing.add(host);
    due.push(list[0]);
    for (const site of list.slice(1)) siteState.setNextRetry(site.url, backoff.nextRetryAt);
    continue;
  }
  waitingCount += list.length;
  for (const site of list) siteState.setNextRetry(site.url, backoff.nextRetryAt);
}

A host with no backoff entry contributes all of its searches, which is the normal path and stays untouched. A host whose wait has expired contributes one. That single request is the cheapest possible way to ask the only question that matters: are we welcome again?

If the answer is yes, the host's entry is deleted and every search on it runs normally from the next cycle. If the answer is no, one refusal is recorded instead of three, so the ladder advances one step rather than three, and the site saw one request rather than three while it was busy refusing us.

list[0] is deliberate rather than random. It is the first search the user configured on that host, so the probe is the same page every time and the log reads consistently. Rotating the probe would have spread the risk across pages for no gain, since a refusal applies to the whole host anyway.

The searches that are not probing still have to say something

The loop above does not skip the other searches. It calls siteState.setNextRetry on each of them.

That matters because the app shows one row per search, and each row shows a status:

kamernet.nl / Leiden      48 listings   checked 12s ago
kamernet.nl / Utrecht     blocked       retry in 45s
kamernet.nl / Delft       blocked       retry in 45s

The block belongs to the host, but the user reads their searches. A row that went quiet with no explanation looks like a bug in our app rather than a wall on somebody else's site, and "retry in 45s" is the difference between a user who waits and a user who starts clicking things. Two of those three rows are waiting on a countdown that no request of their own will ever satisfy, and they still display it, because the honest thing to show is when the app will next try.

When every host is waiting, there is nothing to poll at all, and that gets said out loud rather than looking like a crash:

if (due.length === 0) {
  const soonest = _backoff.soonest();
  log(
    `All ${enabledSites.length} site(s) waiting` +
      (soonest ? `. Next try in ${formatWait(soonest - now)}` : '')
  );
  emitStatus('idle');
  return;
}

idle is a distinct status from polling and from stopped. An app that says "running" while doing nothing for twenty minutes is lying, and an app that says "stopped" when it is waiting on purpose invites the user to restart it, which throws away the very state that was keeping them out of trouble.

Two strikes before the ladder starts, for errors only

The other half of not overreacting is deciding when a failure counts at all.

A refusal from the site is recorded immediately, because it is a deliberate answer from the server and repeating the request will not change it. A plain error is different: a renderer crash, a dropped wifi connection, a navigation timeout on a page that is merely slow today.

/**
 * Consecutive hard errors on a host before it is made to wait.
 *
 * One failed scrape is usually a blip (a renderer crash, a dropped wifi
 * connection) and the next cycle recovers, so backing off immediately would
 * turn a two-second hiccup into minutes of not looking.
 */
const ERROR_GRACE = 2;

So the first error on a host logs "retrying next cycle" and does nothing else. Only the second consecutive one starts the error ladder. The cost of being wrong in this direction is one extra request thirty seconds later. The cost of being wrong in the other direction is thirty seconds of bad wifi turning into minutes of not looking at a page where rooms are going to other people.

That asymmetry is the whole design rule for this file. Every wait we impose on ourselves is time in which a listing can appear and be gone before we look, so a wait has to earn its place by preventing a real refusal, not by being cautious in general.

What I would take to another codebase

Three things generalise past rental sites:

Group by the thing that refused you, then act on the group, not the queue. Our failures come from a host, so the host is the unit of both the wait and the recovery. If your rate limits are per API key, the key is the unit, and retrying every queued job for that key at once is the same mistake in a different shape.

Recover with one request, not with the backlog. Coming back from a block is a question, and a question only needs to be asked once. The backlog can wait one more cycle for the answer.

Distinguish "we chose not to act" from "nothing happened". The countdown on a row that is not even probing, and the idle status when nothing is due, both exist because a deliberate pause and a hang look identical from outside unless you go out of your way to make them different.

See it running

The per-search rows, the retry countdowns and the manual retry button are described on the help page. Each supported site has its own page covering what watching it actually involves, including Kamernet and Pararius, and the full list is here. The app itself is at notifio.app/download.