Googlebot is not in your customers' country, and your pricing page just told it so

We localise prices from the CDN's country header on pub-trivia.app. Visitors in Ireland see euros, visitors in Japan see yen, visitors in the UK see the pounds we actually charge.

Then you remember that a crawler is also a visitor, and that it is sitting in a datacentre.

Try it before you read on

Two requests to the same URL, from your machine, right now:

curl -s https://pub-trivia.app/pricing | grep -o 'Approximate [A-Z]*' | head -1

curl -s -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://pub-trivia.app/pricing | grep -o 'Approximate [A-Z]*' | head -1

If you are outside the UK, the first prints your currency and the second prints nothing at all, because the crawler is served plain GBP and GBP has no conversion notice to print. Same URL, same server, different answer. That is deliberate, and here is the reasoning.

What goes wrong without it

Googlebot's crawl traffic mostly originates in the United States. Bingbot likewise. Whatever header your CDN stamps on their requests is the country of a datacentre, not the country of a customer.

So an uncorrected geo-pricing page hands a US-located crawler dollars, and four things break at once.

The indexed price is wrong for the audience. Our customers are UK pubs. The snippet that ends up in the index quotes a converted, rounded-up dollar figure that no customer will ever be charged.

The structured data contradicts the page. Our Product JSON-LD carries priceCurrency: "GBP", because that is the currency of the offer in the legal sense. A crawler that reads $42.99 in the visible HTML and GBP in the markup has found a mismatch, and mismatched structured data is exactly the signal rich-result validation is designed to catch.

The price is an approximation with a footnote the crawler drops. Our converted figures are deliberately rounded up past the real conversion, with the caveat rendered next to them. Snippets keep the number and lose the caveat.

It is unstable. The crawler's exit region can change between crawls. Your canonical price then flaps between currencies with nothing in your deploy history to explain it.

The fix is one branch

export async function getDisplayCurrency(): Promise<DisplayCurrency> {
  const headerList = await headers()

  if (isBotUserAgent(headerList.get('user-agent'))) {
    return BASE_CURRENCY
  }

  return currencyForCountry(headerList.get('x-vercel-ip-country'))
}

BASE_CURRENCY is GBP, the amount actually debited. A crawler gets the true price with no conversion, no buffer and no rounding, which is also the only figure that agrees with the JSON-LD.

Is this cloaking?

This is the question everyone asks next, and the answer matters enough to get precise about.

Cloaking is showing a crawler different content from users in order to manipulate ranking. What we do is show the crawler the canonical, unconverted price, the one in the contract, the one in the structured data, and the one every UK visitor sees. The converted figure is an approximate convenience label for humans, presented as approximate.

Two tests I would hold any such branch to:

• Would you be comfortable explaining it in a blog post? (Evidently.)
• Does the crawler get the least favourable version, or the most? Ours gets the plain, lowest, un-buffered number. If the bot branch were handing crawlers a better price than customers, that would be a different conversation.

The same reasoning is why the branch keys on user agent rather than on a list of crawler IPs. We are not trying to detect crawlers reliably enough to hide something from them. A regex that catches the honest ones is sufficient, because there is nothing to hide.

export const BOT_PATTERN =
  /vercel|bot|crawler|spider|googlebot|bingbot|slackbot|twitterbot|facebookexternalhit|linkedinbot|whatsapp|headless/i

vercel is in there for our own build and preview fetches. slackbot, twitterbot and facebookexternalhit are there for a reason worth its own paragraph.

Link unfurlers are crawlers too

When someone pastes your pricing URL into Slack, Slack fetches it from Slack's servers and renders a preview card. Without the bot branch, a link shared between two people in Manchester produces a card quoting Californian dollars.

Same for open-graph images if they are generated per request. Ours are prerendered at build time and carry no price at all, which sidesteps the problem rather than solving it, and I would recommend that shape: keep prices out of anything you cache or prerender.

Where the bot check should live

Ours is in lib/http/bots.ts, not inline at the one place that calls it, and that placement is the point:

The answer is a product decision, not a detail of any one page. A crawler must be quoted GBP so search engines index the currency we actually charge in, rather than the currency of whichever datacentre happened to fetch the page.

Inline, it reads as a hack in a pricing file. Extracted and documented, the next person who adds a price to another surface can find it, and the next person who adds a second geo-dependent feature can ask whether crawlers should see that one too. Usually they should not.

The cost you are accepting

Reading the user agent, like reading the country header, makes the page render per request. You lose static caching on your pricing page either way. That trade is worth taking: a cached page that shows the wrong currency is worse than an uncached one that shows the right one, and a pricing page is not where your traffic volume lives.

If your homepage also shows prices, as ours does, it inherits the same constraint. Worth knowing before you put a pricing table above the fold on a page you expected to serve statically.

Go and check ours

pub-trivia.app/pricing in a browser, with a VPN if you have one, shows the localised behaviour. The two curl commands at the top of this post show the crawler behaviour. And if you would like to see what the software does once you are past the pricing page, the free tier needs no card.