Our crawler check is one regex with "bot" in it, so a phone called Cubot is quoted pounds

Notifio shows every visitor a price in their own currency, and charges exactly that. A visitor in the Netherlands reads €24.99 on the home page and pays €24.99. I wrote up how the app gets told its price earlier today.

Search engines are the exception, and they get pounds. Here is why, measured on the live home page from a machine in the euro area:

$ curl -s https://notifio.app/ | grep -o "€24.99" | wc -l
       7
$ curl -s https://notifio.app/ | grep -o '"@type":"Offer".\{0,60\}'
"@type":"Offer","price":"20.00","priceCurrency":"GBP","url":"https://notifio.app/pricing"

Seven euro figures in the body, and a structured-data offer that says twenty pounds. The SoftwareApplication node is built from the list price in lib/pricing/constants, which is GBP pence, and it has no business varying by visitor: it is an identity for the product, merged across every page by a stable @id. The <title> and meta description are the same, static, GBP strings.

So a crawler that read the body would index a page whose visible price disagreed with its own structured data, and which price it decided to show in a result would not be up to me. With the same request as Googlebot:

$ curl -s -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://notifio.app/ | grep -o "€24.99" | wc -l
       0
$ curl -s -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://notifio.app/ | grep -o "£20" | wc -l
      15

Zero euros, fifteen pounds, offer still GBP. The page agrees with itself.

The rule is fourteen lines

/**
 * Shared bot detection.
 *
 * Currently used by the currency resolver, which quotes GBP to crawlers so
 * search engines index the GBP list price rather than the currency of whichever
 * datacentre happened to fetch the page.
 */
export const BOT_PATTERN =
  /vercel|bot|crawler|spider|googlebot|bingbot|slackbot|twitterbot|facebookexternalhit|linkedinbot|whatsapp|headless/i;

export function isBotUserAgent(userAgent: string | null | undefined): boolean {
  if (!userAgent) return false;
  return BOT_PATTERN.test(userAgent);
}

and it has exactly one caller:

export function currencyForVisitor(
  country: string | null | undefined,
  userAgent: string | null | undefined
): DisplayCurrency {
  if (isBotUserAgent(userAgent)) return BASE_CURRENCY;
  return currencyForCountry(country);
}

Count the alternatives: twelve. Now count the ones that can ever fire. googlebot, bingbot, slackbot, twitterbot and linkedinbot all contain bot, which is already an alternative on its own, so five of the twelve are unreachable. They are documentation written in the syntax of code, which is a thing I do more often than I would like, and the regex engine never consults them.

The seven that do work are vercel, bot, crawler, spider, facebookexternalhit, whatsapp and headless. Six of those are specific enough to mean what they say.

The seventh

bot is a three-letter substring with no boundary around it, matched case-insensitively, against a string that device manufacturers are allowed to put their brand name into. Cubot is a real Android phone maker. Its user agents look like this, and so:

$ curl -s -A "Mozilla/5.0 (Linux; Android 13; CUBOT NOTE 21) AppleWebKit/537.36 Chrome/120 Mobile Safari/537.36" \
  https://notifio.app/pricing | grep -o "Prices in [A-Z]\{3\}"
Prices in GBP

$ curl -s https://notifio.app/pricing | grep -o "Prices in [A-Z]\{3\}"
Prices in EUR

Same IP, same country header, same second. The phone with "BOT" in its model name is sold the British price.

What that costs is worth being precise about, because it is smaller than it first looks. The same resolver runs in three places: the page that renders the price, the checkout route for the licence, and the checkout route for the upgrade. All three read the same two headers from the same request, so a Cubot owner is quoted £20 and charged £20. Nobody is shown one figure and billed another, which is the failure the whole module exists to prevent, and the line under the price correctly reads "Prices in GBP, including VAT where it applies."

What it does cost is a worse offer. A Dutch customer on that phone pays in a foreign currency, their bank takes its cut of the conversion, and the price they see is not the one their neighbour sees on a Samsung. For a £20 product that is pennies, and it is still the kind of thing I would rather know about than not.

There is no clean regex

The obvious repair is a word boundary. \bbot\b does not match Googlebot/2.1, so that breaks the main case. Requiring a delimiter in front, something like [^a-z]bot, happily matches CUBOT NOTE because the space before the model name is a delimiter too. A negative lookbehind for cu fixes today's phone and nothing else.

The structural options are the usual three, in increasing cost:

1. Drop the bare bot and keep a curated list of agents, accepting that a new crawler is quoted euros until somebody adds it.
2. Keep the regex and stop treating the answer as important, on the grounds that the title, the description and the structured data are already GBP for everyone, and they are what a search result quotes.
3. Verify crawlers properly, by reverse DNS on the request address, which is what Google documents and which costs a lookup on every page render.

Option 2 is tempting and I think it is wrong, because a rendered body that contradicts its own Offer node is the sort of thing that gets a page flagged rather than ranked. Option 3 buys accuracy nobody is paying for. So the honest answer for now is option 1 plus an uncomfortable truth: the list has to include the bare bot, because most crawlers that matter are called something-bot and will never be enumerable in advance.

One last thing that regex now decides, which it was not written for. The desktop app's licence check runs through the same resolver, user agent included, and returns the price label the app displays. That means the user agent string Electron happens to send is an input to our pricing, and the day it starts containing one of seven tokens, every customer in the world is quoted pounds. It is not something a fourteen-line file with "Shared bot detection" at the top should be deciding on its own, and that, rather than the phone, is the part I will be moving.

If you want to see both answers for yourself, both commands above run against the real site: the pricing page is rendered per request, so changing the user agent on your own machine changes the number, and the help page is static, so it never changes at all.