Polling someone else's website every 30 seconds without getting banned

Our app watches rental search pages and emails the user the moment a new listing appears. The whole value proposition is latency: if you hear about a room an hour late, you are the fiftieth message in that landlord's inbox and you will not get a viewing.

So we poll. Every 30 seconds, per search. And the interesting engineering is not the polling — it is everything we had to build so that polling that hard does not get us walled off.

The loop is boring on purpose

let _running = false;
let _timer: NodeJS.Timeout | null = null;
let _pollCount = 0;

Module-level state, one timer, no queue, no worker pool. Each cycle walks the user's searches one at a time, scrapes each, diffs against the last snapshot, and emails anything new.

Sequential, not parallel. That is deliberate: three concurrent page loads against the same host is a much louder traffic signature than three sequential ones, and the wall-clock saving buys us nothing a user can perceive.

Randomised delays, because a metronome is a fingerprint

A request landing at exactly :00 and :30 of every minute, forever, is not what a human browsing looks like. It is what a bot looks like, and it is trivially detectable in a request log — you do not need a machine learning model for that, you need a histogram.

So we jitter both the gap between polls and the gap between searches inside a poll. The average stays around 30 seconds; the variance is what buys the cover.

Backing off when a site pushes back

This is the part most scraping tutorials skip. Every site eventually shows you a challenge page, a rate-limit page, or a "verify you are human" wall. The wrong response is to retry immediately, which is the exact behaviour the wall exists to punish.

const _backoff: Record<string, { failures: number; nextRetryAt: number }> = {};
const BACKOFF_BASE_MS = 10 * 60_000; // first backoff: 10 minutes
const BACKOFF_MAX_MS  = 60 * 60_000; // capped at 60 minutes

Three properties worth stealing:

It is keyed by site URL, not global. One site putting up a wall must not stop the other fourteen searches. A global circuit breaker turns one site's bad afternoon into a total outage of the product.

It resets on success, not on a timer. resetBackoff(url) deletes the entry entirely the moment a scrape comes back clean. A backoff that decays on a schedule keeps punishing you after the problem is already gone.

It is capped. Exponential backoff with no ceiling eventually means "retry in nine days", which is indistinguishable from broken. An hour is the longest a user would tolerate not hearing about a site, so that is the ceiling.

Telling the user, but not fifteen times

When a site blocks us, the user needs to know — a silently walled scraper is worse than no scraper, because they think they are covered. But a loop that notices the block every 30 seconds will happily send 120 emails an hour.

const _lastBlockAlerts: Record<string, number> = {}; // hourly rate-limit

One notification per site per hour. The same shape shows up for login-expiry alerts:

const _authSuppressedUntil: Record<string, number> = {};
const AUTH_SUPPRESS_MS = 60 * 60 * 1000;

That one is subtler. If a user deliberately logs out of a site inside the app, the very next poll correctly notices they are logged out — and would fire a "your session expired, please log in" alert about an action they took on purpose two seconds ago. So a manual logout suppresses auth alerts for that host for an hour.

The general rule: any state your loop can observe, it will observe on every iteration. Every user-visible reaction to observed state needs a rate limit attached at the moment you write it, not after the first angry support email.

A config file re-read every cycle

function loadConfig(): Config {
  try {
    return JSON.parse(fs.readFileSync(CONFIG_PATH, 'utf8'));
  } catch {
    return { email: { to: '' }, sites: [] };
  }
}

Two decisions in nine lines. We re-read from disk every poll rather than caching in memory, so a user adding a search takes effect on the next cycle with no restart and no cache-invalidation logic. And a corrupt or missing config degrades to an empty config instead of throwing — a parse error must never be able to kill the loop, because the loop not running is the one failure mode a user cannot detect for themselves.

See it running

The sites we currently poll, and the quirks of each, are listed at notifio.app/alerts — click into any site to see which search-URL shapes it accepts and what the site does to make watching it awkward.

If you would rather watch the loop than read about it, grab the app from notifio.app, paste any rental search URL into it, and leave it open. The number to watch is the gap between a listing going live and the email landing. It should be under a minute.