Every thirty seconds it fails, and you should get one email, not 2,880

Notifio checks your rental searches every thirty seconds. Most of the time a check succeeds and either finds something or does not.

Sometimes it fails in a way you need to hear about. Your session on a site expired, so the app is looking at a login page instead of your search results and it will keep doing that forever unless you go and sign in. Or the site put a verification wall in front of us that needs a human to clear.

Those are real, and staying quiet about them is its own bug: an app that silently stops working is worse than one that tells you it stopped.

So: send an email when a check fails. Except a search that is failing is not failing once. At one check every thirty seconds it fails 2,880 times a day, and the second email has already done all the useful work the first one did not.

Between a failed check and an email there are four gates in our monitor. They look like a pile of rate limiting. They are not. Each one answers a different question, and only the first is about the failure itself.

Gate 1: is this kind of failure ever the user's business?

Our scrape outcome distinguishes three things that all look like "it did not work":

export type BlockKind = 'challenge' | 'rate_limited' | 'denied';

export interface ScrapeOutcome {
  /** null = hard error (network / crash). [] + authFailure = login wall detected */
  listings: Listing[] | null;
  authFailure?: boolean;
  blocked?: boolean;
  blockKind?: BlockKind;
  retryAfterMs?: number;
  // ...
}

A hard error never sends an email. Not after one, not after fifty. If your wifi dropped or our browser process crashed, there is no action for you to take, and "Notifio could not reach the internet" is a notification that makes the product look broken while telling you something your laptop already made obvious.

Hard errors do not even count against the site immediately:

/**
 * Consecutive hard errors on a host before it is made to wait.
 *
 * One failed scrape is usually a blip (a renderer crash, a dropped wifi
 * connection) and the next cycle recovers, so backing off immediately would
 * turn a two-second hiccup into minutes of not looking.
 */
const ERROR_GRACE = 2;

That is the whole first gate: sort failures by what the user could do about them, before you sort them by anything else. Two of the categories have an action attached (go and log in, go and clear a check). One does not, and it never emails.

Gate 2: has it survived long enough to be real?

A verification interstitial is often gone by the next check. By the time the monitor records one, the scraper has already tried to read past it and to wait it out. So a single challenge is not evidence of anything except that the internet is the internet:

// A challenge that clears on the next try is not worth an email. Only tell
// the user once it has survived a couple of retries, because by then it needs
// them to go and pass the check by hand.
if (kind === 'challenge' && entry.failures < 3) return;

Note this gate is conditional on the kind. rate_limited and denied do not get a grace period, because they are not transient in the same way. A site that flatly refused us is telling us something about the state of the world that three more attempts will not change.

This is the gate that most often gets written as a global "wait n failures before alerting" constant, applied to everything. That version is either too slow for the failures that need you now or too chatty for the ones that clear on their own, and tuning the number just moves which of those two you get.

Gate 3: have we already told them?

The plain hourly limit, per search, per kind:

const lastAlert = _lastBlockAlerts[siteUrl] || 0;
if (now - lastAlert < 60 * 60 * 1000) return;
_lastBlockAlerts[siteUrl] = now;

The interesting part is not the hour. It is what happens on recovery:

function resetBackoff(host: string): void {
  _backoff.clear(host);
  delete _hostErrors[host];
  for (const key of Object.keys(_lastBlockAlerts)) {
    if (hostOf(key) === host) delete _lastBlockAlerts[key];
  }
}

When a host starts working again, the alert clock is cleared along with the backoff ledger. So the rule is not "at most one email per hour". It is "at most one email per hour of continuous trouble". A site that breaks, recovers, and breaks again two hours later gets to tell you twice, which is correct, because that is two separate things going wrong. Without that line, the second outage inherits the first one's silence, and your quietest hour is the one right after you thought the problem was over.

If you keep a "last notified at" timestamp anywhere, go and check whether anything clears it on success. In my experience it usually does not, and the failure mode is invisible because the bug is an email that never arrives.

Gate 4: do they already know?

This one I would never have designed up front. It came from using the app.

You log out of a rental site on purpose, in the login window the app itself gave you. Thirty seconds later the next poll hits a login page, correctly identifies an auth failure, and emails you to say your session on that site has expired and you should log in.

Technically accurate. Completely stupid.

// After a manual logout we suppress auth-failure alerts for that host for a
// while, so the user doesn't get an immediate "login required" email/notification
// for a logout they just performed on purpose. Keyed by hostname → expiry ms.
const _authSuppressedUntil: Record<string, number> = {};

Set on logout, and the symmetric half matters just as much, because a suppression with only one way out is a way to go permanently quiet:

export function clearBackoff(siteUrl: string): void {
  // ...
  // A successful login lifts any post-logout auth-alert suppression for the host.
  delete _authSuppressedUntil[host];
}

Gate 4 is the only one that is not about the failure at all. It is about what the user did ten seconds ago in a different window. That information exists nowhere in the scrape result, and no amount of thinking about the scrape result gets you to it.

The thing I noticed while writing this

I have written before about the retry ladder being keyed by the wrong thing. Backoff used to be keyed per search URL, which meant five searches on one site each kept their own counter and each went back to knocking while the others were standing down. It is keyed by host now, because a refusal comes from the site and applies to everything on it.

Look at gate 3 again. _lastBlockAlerts[siteUrl]. Keyed by the search URL.

Five searches on the same site, all walled by the same wall, can produce five emails in the same hour. It is the same mistake, one layer up, sitting directly above the fix for it. The auth suppression in gate 4 is keyed by host and the recovery sweep in resetBackoff iterates URLs to clear a host, so the code already knows which key is right in two places out of three.

I am writing it down rather than quietly fixing it before posting, because the interesting bit is not the bug. It is that fixing something at one layer does not propagate upward on its own, and "we already learned this lesson" is exactly the belief that stops you looking.

Go and look

• notifio.app and the download page
• The help page covers what the app does when a site stops cooperating
• The sites it monitors, most of which have some form of wall
• Polling someone else's website every 30 seconds without getting banned is the layer underneath all of this
• When a block clears, only one search goes back in is the recovery side

The general shape, if you take one thing: the test for sending a notification is not "did something fail". It is "is there an action the user can take, that they do not already know about, that we have not already asked them for". Failure is a precondition for that question, not an answer to it.