seotto.com

Posted on Nov 4 Edited on Nov 4

Postmortem: an automatic geo-IP redirect hid our UK and German sites from search

#seo #i18n #webdev #postmortem

Hypothetical postmortem. The company and numbers are invented, but this failure mode is common on multi-country sites.

Summary

The store runs three locales: /us/, /uk/ and /de/. To improve conversion, the team shipped middleware that redirected visitors to the locale matching their IP's country. A visitor in Germany who opened /us/shoes was 302-redirected to /de/schuhe.

Search crawlers crawl largely from US-based IP addresses. Every time a crawler requested a UK or German URL, it was redirected to the US version. Over the following weeks, most /uk/ and /de/ pages dropped out of the index, and UK and German organic traffic fell to a fraction of normal. Humans in each country saw the right site, so nobody internally noticed anything wrong.

Timeline

Week 0: geo-redirect middleware deployed behind a feature flag, enabled for 100% of traffic.
Week 1: conversion on country-matched sessions ticks up. The change is declared a success.
Weeks 2–4: Search Console for the UK and DE properties shows "Page with redirect" growing and indexed pages falling. Nobody owns those properties day to day.
Week 5: the UK marketing lead asks why organic traffic is down sharply. Investigation begins.
Week 5, day 2: root cause found by requesting a UK URL from a US server and seeing the redirect.

Root cause

// middleware.ts (simplified version of what shipped)
export function middleware(req: Request) {
  const country = req.headers.get("cf-ipcountry") ?? "US";
  const locale = COUNTRY_TO_LOCALE[country] ?? "us";
  const url = new URL(req.url);
  const current = url.pathname.split("/")[1];
  if (current !== locale) {
    url.pathname = translatePath(url.pathname, locale);
    return Response.redirect(url, 302);
  }
}

The logic assumed every visitor belongs on their IP's locale. Two groups break that assumption:

Crawlers, which may visit from a single country and need to see every locale.
Humans who deliberately chose another locale: expats, travelers, people using a VPN, a UK customer shopping for a gift for someone in the US.

The hreflang annotations were correct, but they couldn't help. Crawlers rarely reached the pages that carried them.

Why we didn't catch it

Testing happened from the office. Each regional team only ever saw their own locale behave "correctly".
The success metric was conversion, and conversion went up.
Search Console ownership was split by country, and nobody was watching the smaller properties.

The fix

1. Never force-redirect based on IP

We replaced the redirect with a suggestion banner: "Looks like you're in Germany. Go to the German store?" Every locale URL now serves its own content with a 200, to everyone.

export function middleware(req: Request) {
  const country = req.headers.get("cf-ipcountry") ?? "US";
  const suggested = COUNTRY_TO_LOCALE[country];
  const res = NextResponse.next();
  if (suggested) res.headers.set("x-suggested-locale", suggested); // UI shows a banner
  return res;
}

The user's choice is stored in a cookie and respected afterwards. The cookie is set only by the user clicking, never automatically.

2. Only redirect the root, if at all

A bare https://example.com/ with no locale can reasonably redirect to a locale, ideally based on Accept-Language and with an x-default hreflang pointing to a locale-picker or default page. Deep locale URLs must never be redirected away.

3. Test from other countries, as a crawler would

We added a CI check that requests sample URLs from each locale and fails on any redirect:

#!/usr/bin/env bash
# fail if any locale URL redirects (run from a single region, like a crawler)
fail=0
for u in https://example.com/uk/shoes https://example.com/de/schuhe https://example.com/us/shoes; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -A "Mozilla/5.0 (compatible; Googlebot/2.1)" "$u")
  echo "$code $u"
  [ "$code" = "200" ] || fail=1
done
exit $fail

For the full picture, use the URL Inspection tool in each Search Console property. It shows what Google's crawler actually received.

4. One owner for all locales' search health

A single person or team now reviews indexing reports for every property weekly. A scheduled crawl of each locale (with an SEO crawler or a service like seotto.com) flags redirects and status changes per locale, so a drop in one country can't hide behind stable numbers in another.

How to check your own site in 10 minutes

You don't need a full audit to know whether you have this problem. Run these three checks against one deep URL per locale:

Request it with no cookies and a neutral Accept-Language. A curl -sI should return a 200 for the URL you asked for, not a 301 or 302 to another country's site.
Request it from another region. A cheap VPS or a cloud shell in a different country works. If the status code or the Location header changes with the IP address, crawlers will see the same thing.
Inspect it in each Search Console property. Use URL Inspection on a page from each locale. If "Google-selected canonical" points to another country's URL, or the crawled page shows the wrong language, the redirect (or something like it) is still in the way.

If all three pass for every locale, geo-redirects aren't your problem. If any fails, check the CDN and edge rules too, not just the application. In our case the redirect lived in an edge function that nobody on the SEO side knew existed.

Lessons

Personalization by redirect is a crawlability risk. Personalize content or show a suggestion. Don't move people (or bots) to a different URL.
hreflang only works if crawlers can reach each version.
Test from outside your geography. Your office is not a representative sample of the internet.
Conversion wins can hide acquisition losses. Look at both before calling a change successful.

Action items

[x] Replace geo-redirect with suggestion banner
[x] CI check for redirects on locale URLs
[x] Single owner for all Search Console properties
[ ] Add x-default hreflang and a locale picker at the root
[ ] Document "no IP-based redirects" in the frontend guidelines

DEV Community