Change detection is an identity problem, not a checksum problem

The naive version of "tell me when this page changes" is a hash. Fetch the page, hash the body, compare to last time, alert on difference. It takes about nine lines and it is wrong for almost every real use.

We learned this building a rental-listing watcher. Hash the HTML of a search results page and you will alert on: a rotating ad slot, a "12 people viewed this today" counter, a CSRF token, a relative timestamp ticking from "2 minutes ago" to "3 minutes ago", and a session id baked into every link. You will alert every single poll, forever, and none of those alerts are a new listing.

The question is not did the page change. It is did the set of things on the page change.

Extract entities, then diff the set

So the scraper does not return a document. It returns a Listing[] — title, price, URL, per item. The diff runs over that array, keyed by a stable identifier, and the only event we care about is "an id appeared that was not in the previous set".

That reframing kills every false positive above in one move. Ads are not listings. View counters are not listings. A relative timestamp is a property of a listing, not its identity.

It also hands you the hard question that the hash approach was hiding: what is the identity of one of these things?

For us it is the listing URL, after normalisation. That is not free. Plenty of sites append tracking parameters that vary per page load, which means the raw href is unstable and a naive keying alerts on the same room every 30 seconds. Some sites re-issue a different URL when a landlord edits the price, so you have to accept a duplicate alert or accept a missed one. There is no third option, and choosing is a product decision, not an engineering one: we would rather tell you about the same flat twice than miss a flat once.

Storing the previous set

One snapshot file per watched search, named by hashing the URL:

function snapshotPath(url: string): string {
  const hash = crypto.createHash('sha1').update(url).digest('hex').slice(0, 16);
  return path.join(DATA_DIR, `${hash}.json`);
}

SHA-1 is doing zero security work here. It is a filename-safe fixed-length function over an arbitrary string, because a search URL contains slashes, ampersands, encoded commas and occasionally non-ASCII, and none of that belongs in a path. Sixteen hex characters is 64 bits, which is comfortably collision-free for the fifteen searches a user is allowed.

The write has to be atomic

This is the part that bites people who ship desktop software, where "the process was killed mid-write" is not a rare event — it is what happens every time a user closes their laptop lid or force-quits.

function saveSnapshot(url: string, listings: Listing[]): void {
  const file = snapshotPath(url);
  const tmp = `${file}.tmp`;
  try {
    // Atomic write (temp + rename) so a crash can't corrupt the snapshot.
    fs.writeFileSync(tmp, JSON.stringify(listings, null, 2), 'utf8');
    fs.renameSync(tmp, file);
  } catch (err) {
    console.error(`[monitor] Failed to save snapshot for ${url}:`, err);
  }
}

Write to a temp file, then rename over the target. rename within a filesystem is atomic: a reader sees either the entire old file or the entire new one, never a half-written one. Write in place and a crash at the wrong microsecond leaves you with truncated JSON.

And the failure mode of truncated JSON here is nasty. On the next poll the snapshot fails to parse, the loop treats the search as never-seen, and every listing on the page is "new" — so the user gets forty emails about flats they already saw. A corrupted cache does not degrade into missing alerts. It degrades into spam.

The read side is paranoid to match:

function loadSnapshot(url: string): Listing[] | null {
  const file = snapshotPath(url);
  if (!fs.existsSync(file)) return null;
  try {
    const parsed = JSON.parse(fs.readFileSync(file, 'utf8'));
    return Array.isArray(parsed) ? parsed : null;
  } catch {
    return null;
  }
}

Note Array.isArray. JSON.parse succeeding tells you the bytes were valid JSON, not that they were your JSON — null, 4, and {} all parse fine and all explode differently three functions downstream. Validate the shape at the boundary where it enters your program.

The first poll is a special case

A null snapshot means we have never seen this search. If you treat that as "an empty previous set", every listing currently on the page is new and the user's first experience of your product is an inbox with fifty emails in it.

So the first poll records and stays silent. You only get told about things that appeared after you started watching. That is also the honest reading of what the user asked for.

Why any of this matters

We wrote up what the timing actually looks like on real rental sites — how long a listing stays available before it is gone — at notifio.app/guides/how-fast-do-rental-listings-go. It is the number that determines whether a 30-second poll is over-engineering or the bare minimum.

If you want to see the diff engine work on a page of your choosing, point the app at any search URL: notifio.app.