DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Our page check returns a score out of four, and the decision does not use it

Notifio watches rental listing sites and, if you have the auto-reply upgrade, replies to new listings for you by replaying a flow you recorded once. That means a program opens a page in your logged-in session and sends a message to a stranger with your name and phone number on it.

The expensive failure here is not missing a listing. It is sending a well-formed enquiry about something that is not a listing. The scraper finds candidate listings with a generic anchor sweep, which is the right design for coverage and the wrong design for certainty: a blog post, a category page, a "how to rent in Amsterdam" explainer or an agency's team page can all slip past a keyword filter and arrive at the reply engine looking like work to do.

So before anything is typed, the page has to prove what it is. The function that does it looks like a scoring model and then decides with a rule that ignores the score.

The whole check

const score = (price ? 2 : 0) + (address ? 1 : 0) + (contact ? 1 : 0);
return {
  looksLikeListing: price && (address || contact),
  score,
  signals: { price, address, contact },
};
Enter fullscreen mode Exit fullscreen mode

Four points available. Price is worth two, an address is worth one, a contact affordance is worth one. And then looksLikeListing does not look at score at all.

That is deliberate, and it took a moment to be comfortable with. A threshold on the score (score >= 3) would accept address plus contact with no price, which is the exact shape of an agency contact page: a street address in the footer, a "send us a message" form, no price anywhere. Meanwhile a sparse listing with a price and a contact button but a vague area instead of a postcode scores 3 as well, and is genuinely a listing. One number cannot separate those, because the signals are not interchangeable: a price is near-proof, and the other two are corroboration.

So the rule is a sentence, not a sum. A price, plus at least one of an address or a way to contact somebody.

The score still gets computed, because it goes in the refusal:

return {
  ok: false,
  block: 'not_listing',
  detail: `Page does not look like a listing (score ${check.score})`,
};
Enter fullscreen mode Exit fullscreen mode

A refusal that says "this did not look like a listing" is unactionable. A refusal that says it scored 2 tells you, months later, whether the page had nothing at all or whether it had a price and we failed to find the rest of it. The number is diagnostics, not a decision, and keeping those two jobs separate is why the threshold temptation is worth resisting.

What counts as a price

const PRICE_MARKERS = [
  /[€£$]\s?\d{2,5}([.,]\d{3})*([.,]\d{2})?/,
  /\b\d{3,5}\s?(euro|eur|gbp|pounds?)\b/i,
  /\b(per month|per maand|p\/m|pro monat|kaltmiete|warmmiete|al mes|par mois|huurprijs|rent)\b/i,
];
Enter fullscreen mode Exit fullscreen mode

The third one is the interesting entry. kaltmiete and warmmiete are German for rent excluding and including utilities, and a German listing will often print those words next to a figure in a layout the first two patterns do not catch. huurprijs is the Dutch equivalent. These are not translations added for completeness: the sites Notifio covers are mostly Dutch, British, German and Spanish, and a check written only in English would refuse to reply on most of them. You can see the spread on the alerts index.

What counts as an address

const ADDRESS_MARKERS = [
  /\b\d{4}\s?[A-Z]{2}\b/,                       // NL postcode
  /\b\d{5}\b/,                                   // DE / ES / FR postcode
  /\b[A-Z]{1,2}\d{1,2}[A-Z]?\s?\d[A-Z]{2}\b/,    // UK postcode
  /\b(street|straat|laan|weg|kade|gracht|plein|strasse|straße|calle|avenida|rue|avenue|road)\b/i,
];
Enter fullscreen mode Exit fullscreen mode

Three postcode patterns covering five countries, and a vocabulary of street-type nouns in five languages. gracht is there because a very large number of Amsterdam addresses are canals.

The second pattern is the loosest thing in the file. \b\d{5}\b matches a German postcode and also matches any five-digit number on any page in the world. I left it in anyway, and the reason is the direction of the error: an address is corroboration, never proof. It cannot pass a page on its own, because the price is mandatory. A false positive on \d{5} only ever gets you from "has a price, no corroboration" to "has a price, corroborated", and a page with a rent on it is already most of the way to being a listing. If the address signal were load-bearing I would not accept that pattern for a second.

Where a keyword betrayed us

One guard earlier in the chain is a hard stop on anything touching money, because a reply flow that wanders into a payment page is the one failure with a number attached to it. The obvious implementation is to look for payment-method words. That implementation does not survive contact with the market:

/**
 * Payment-provider and card-field markers. Deliberately narrow: a bare "ideal"
 * keyword would match Idealista (a real listing site), so we key off PSP script
 * hosts and card input attributes instead of payment-method words.
 */
const PAYMENT_HTML_MARKERS = [
  /js\.stripe\.com|checkout\.stripe\.com|hooks\.stripe\.com/i,
  /\.mollie\.com|mollie\.nl/i,
  /(checkoutshopper|live\.adyen|test\.adyen)\./i,
  /paypal\.com\/(sdk|smart)/i,
  /js\.braintreegateway\.com/i,
  /x\.klarnacdn\.net|klarna\.com\/(sdk|web-sdk)/i,
  /autocomplete=["']?cc-(number|exp|csc)/i,
  /name=["']?(cardnumber|card_number|cc-number|creditcard)/i,
  /id=["']?(card-number|cardNumber|card-element)/i,
  /data-(stripe|adyen|mollie)=/i,
];
Enter fullscreen mode Exit fullscreen mode

iDEAL is the dominant Dutch payment method, so "ideal" is an obvious keyword for a Dutch-market product. Idealista is one of the largest property portals in Spain, and we have a page for it. A substring match would have made every Idealista listing look like a checkout page and silently disabled the feature for an entire country.

Keying off script hosts and card input attributes instead is more code and more maintenance, and it is also the only version that can tell a payment page from a page whose brand contains a payment word. The same shape of problem shows up in the paywall guard, where a two-digit cap is what separates a subscription price from a rent, which I wrote about in €1450 per month is rent, €9.99 per month is a paywall.

The order is the design

The listing check runs last, out of seven:

export function evaluatePageSafety(input: SafetyInput): SafetyVerdict {
  // 1. Domain lock, the aggregator hand-off case.
  // 2. Anything touching money, hard stop, no overrides.
  // 3. Anti-bot wall, back off rather than fight it.
  // 4. Session expired.
  // 5. Needs a new account, we never create accounts.
  // 6. Needs a paid plan on the site itself.
  // 7. Is this even a listing?
Enter fullscreen mode Exit fullscreen mode

Severity descending, and the first refusal wins. "This is not a listing" is last because it is the least informative answer: a login wall is also not a listing, and reporting it as not_listing would send the user looking for a bug in the scraper when what they actually need to do is sign in again. Each refusal maps to its own recorded status, skipped_offsite, skipped_payment, skipped_captcha, and so on down to skipped_not_listing, so the reason survives in the ledger rather than collapsing into "failed".

The domain lock at position one is the one I wrote about separately in The guard that stops our automation from following a link.

There is also one escape hatch:

/** Skip the listing-shape check (e.g. when already on a contact form). */
skipListingCheck?: boolean;
Enter fullscreen mode Exit fullscreen mode

A contact form that has opened in a modal, or on its own URL, frequently has no price on it. It is the same page in the user's journey, the domain lock and the money and captcha guards still apply, and the listing check has already passed on the page we came from. Without this flag the check refuses the second half of every successful flow.

The general version

If you are building something that acts on pages you do not control:

  • Decide what the expensive failure is, then put the guard in front of that specific thing rather than in front of "errors".
  • Keep the score and the decision apart. Scores are for explaining a refusal to a human later. Decisions want a rule you can say out loud.
  • Know which of your signals is proof and which is corroboration, and let the corroborating ones be sloppy. \b\d{5}\b is fine when it cannot pass anything on its own.
  • Every keyword list is a bet about a language. Someone has named their company after your keyword.

The product these refusals live in is on the download page, what it does is on the help page, and if you want the human version of the thing the reply engine is trying to write, that is the first message to a landlord.

Top comments (0)