The plan was to find out how much of the web is geo-blocked. I had eight servers booting in eight countries and a list of the 500 most popular domains, and I was expecting to come out with a map of the splinternet: this site is closed in India, that one redirects Germany somewhere else, here is the wall.
What I got instead was a flat line, and it took me a while to work out that the flat line was the interesting part.
The setup was simple enough. One HTTPS request per domain from each of New York, San Francisco, Toronto, London, Frankfurt, Singapore, Bangalore and Sydney, with a Chrome user agent, following redirects, recording the status code, the hostname I ended up at, the DNS answers, the timing and any cookies. Then the same list again from my own flat in Sofia, on ordinary home broadband, as a control. Around 4,500 requests in total, all of them identical except for where they came from.
345 of the 500 answered from every vantage point. The missing 155 are almost all things like gtld-servers.net and app-measurement.com, which earn their place in a traffic ranking without ever serving a page to a human being, so they are excluded from everything below.
The geography, then:
| difference between the eight countries | domains | share |
|---|---|---|
| different HTTP status code | 22 | 6.4% |
| different final hostname | 12 | 3.5% |
| either of those | 29 | 8.4% |
Twenty-nine sites out of 345. I went through all of them by hand expecting to find something worth writing about, and mostly found marketing departments. Yahoo has a separate front door for every country I probed from, so my eight machines landed on uk.yahoo.com, ca.yahoo.com, de.yahoo.com, au.yahoo.com, sg.yahoo.com and the plain www one. Kaspersky does the same thing with five. CNN pushes anyone outside the United States to edition.cnn.com, Business Insider has a German edition and sent Frankfurt there, and USA Today runs a separate eu.usatoday.com which I assume exists so their lawyers can sleep. None of that is a wall. It is a company deciding which of its own pages to show you, which is what companies have always done, and it is not the splinternet anybody is worried about.
The flat line
The number I kept staring at was a different one. About 27% of the sites refused my request, which sounded enormous until I looked at the breakdown by region: 26.7% in New York, 27.5% in London, 27.0% in Bangalore, 26.7% in Sydney, and so on down the list. Geographic blocking does not produce a flat line across five continents. Whatever I had found, it had nothing to do with geography, and I had very nearly written it up as though it did.
So I ran the same script from my flat, which is the one vantage point that is unambiguously a person sitting at a desk rather than a machine in a rack.
| domains | share | |
|---|---|---|
| served to home and to all eight datacentres | 243 | 70.6% |
| refused my client everywhere, home included | 71 | 20.6% |
| served to home, refused by every datacentre | 14 | 4.1% |
| refused to home, served to the datacentres | 1 | 0.3% |
That middle row is a fifth of the top 500 turning away an ordinary programmatic HTTPS request wherever it comes from. The row under it is smaller but cleaner, because the only thing that changed between my flat and the eight machines was the address range the request arrived from, and fourteen sites treated that as reason enough to refuse. Reddit is in that list. So are Patreon, Imgur, Dell, Zendesk, DeviantArt, MediaFire, Investopedia, Criteo, AppsFlyer, Wiley, Ring, att.net and one adult site. Where the datacentres got a response body at all it was a 403 in 96 cases, a 429 in eight and a 406 in eight.
I want to be careful about what that does and does not prove, because there is an obvious hole in it. My client was Python's urllib sending a Chrome user agent, and the TLS handshake underneath that looks nothing like Chrome's. Bot detection services are built precisely to notice that mismatch. So when I checked a sample of the always-refused sites by driving a real Chromium at them, Cisco loaded perfectly, and Bloomberg came back with a page titled "Are you a robot?", and eBay, Etsy, Cambridge University Press and Behance all refused as well. But Chromium under Playwright is automated, and automation is detectable too, so that test does not separate "this site blocks non-browsers" from "this site blocks anything automated". It only tells me both categories get turned away. The measurement I can actually stand behind is the narrow one: a fifth of the top 500 will not answer a scripted request, no matter which continent it comes from.
Toronto
There was one result I went back and re-ran because I did not believe it. IMDb, un.org and oup.com each refused seven of my eight vantage points and served exactly one, and the one was Toronto every time. Britannica served Toronto and Sydney and refused the other six. Canva served London, Singapore and Sydney and refused New York, San Francisco, Toronto, Frankfurt and Bangalore.
There is no version of international law in which Oxford University Press serves Canada and refuses Germany. What that pattern actually shows is that none of this is a policy about countries. It is a reputation score attached to ranges of IP addresses, and the particular range my Toronto droplet was handed had not yet been marked the way the others had. If I had created those eight machines an hour later I would probably have got a different set of winners, which is worth remembering the next time you conclude that some site has blocked your country.
The timings came along for free, and they are a reminder that the web still has a physical location no matter how much we pretend otherwise. Median time to a complete response was 244ms from New York, 275ms from Frankfurt, 286ms from London, 321ms from Toronto, then 506ms from Singapore, 680ms from Bangalore and 690ms from Sydney. Identical hardware, identical software, same 250-odd sites, and a person in Sydney waits nearly three times as long as a person in New York. Twenty years of CDN buildout and the top 500 still mostly live somewhere else if you are in the southern hemisphere.
One more thing fell out of the data that I was not looking for. Across those 345 sites, 162 of them set a cookie on the very first request, before any click, any consent banner, any interaction of any kind. From New York that added up to 541 cookies for 345 page requests.
What it cost, and what I would change
The whole exercise cost about seven cents. That is the part I would underline for anyone who fancies doing something similar: the eight machines were DigitalOcean's cheapest droplets, created with a loop over their API, answering SSH about two minutes later, and destroyed the same afternoon. Measuring the web from several continents at once used to mean emailing universities to beg for shell accounts, which is why most of the published research in this area comes out of academic consortia with funding attached. It now costs less than a coffee, and the probe itself is sixty lines of the Python standard library.
What I would do differently, if I run it again, is fix the client. Sending Chrome's user agent over Python's TLS stack is the exact contradiction that bot detection exists to catch, and it means my 20.6% is a measurement of how the web treats one specific clumsy client rather than programs in general. Doing the same sweep with a real browser engine in all eight regions would split that number into "blocks automation" and "blocks this particular thing I wrote", and I would genuinely like to know the ratio.
The detail that has stuck with me since is Reddit, Patreon and Imgur sitting in that list of fourteen. Those three get pasted into chats and tickets and documents all day long, and every time that happens something has to fetch the page to build the little preview card with the title and the picture. That fetcher lives in a datacentre. It is getting the same 403 my eight machines got, not the page my flat got. For years I assumed blank preview cards were a bug in whatever app I was using, and it never crossed my mind that the app was being refused on purpose.
Top comments (0)