DEV Community

Devil Scrapes
Devil Scrapes

Posted on

A fake 404 and a 403 that isn't a ban — the UK food ratings API

#api #python #webscraping #debugging

We probed the UK Food Standards Agency's ratings API and got a 404 that said the API doesn't exist. It exists. Then we fixed that and got a 403 that looked exactly like an IP ban. It wasn't. Both responses are the API working as designed — and both designs are invisible until you've lost an afternoon to them.

Quick answer

Building on api.ratings.food.gov.uk? Two headers-and-habits rules:

  1. Send x-api-version: 2 on every request. Without it the API answers 404 "The API 'Establishments' doesn't exist" — a message engineered to make you re-check your URL forever. The URL is fine; the header is missing.
  2. Never query /Establishments bare. An unfiltered query returns 403 "This is a CPU intensive query: please use one of the documented filters". That's not a ban and not rate limiting — it's a query-cost guard, and it means a page-through-everything scraper design is dead on arrival. Filter by local authority, name or address on every request.

The 404 that means "missing header"

Here's the trap in its natural habitat:

GET https://api.ratings.food.gov.uk/Establishments?name=pizza
→ 404 {"Message": "The API 'Establishments' doesn't exist"}

GET https://api.ratings.food.gov.uk/Establishments?name=pizza
    x-api-version: 2
→ 200, establishments with ratings
Enter fullscreen mode Exit fullscreen mode

Same URL, same method, opposite outcome. A 404 with a body claiming the API doesn't exist reads like a retired endpoint — we initially logged it as one. Every dead-endpoint conclusion about this API that doesn't mention the version header is wrong.

Why does the FSA API return 403 on a valid query?

Because the query is valid and expensive. The bare Establishments listing would walk every food business in the UK, so the API refuses with:

403 {"Message": "This is a CPU intensive query: please use one of
     the documented filters in your query (e.g. filter by LocalAuthority)."}
Enter fullscreen mode Exit fullscreen mode

The failure mode this creates for scrapers is nasty: a 403 pattern-matches to "we've been blocked", which sends you down the proxy-rotation rabbit hole. No proxy fixes this, because nothing is blocked — the request shape is the problem. Narrow the query (local authority ID, business name, address, rating value) and the same client on the same IP gets a 200.

What data is actually in there?

Every food business establishment in England, Wales and Northern Ireland (plus Scotland's parallel scheme): business name and type, full address, the hygiene rating (0–5, or Scotland's pass/fail), the three inspection sub-scores (hygiene, structural, confidence in management), the rating date and the local authority. It's the dataset behind the green sticker in every UK café window — and it's genuinely useful for hospitality lead-gen, franchise diligence and food-safety monitoring.

What the scraper actually does

UK Food Hygiene Ratings Scraper queries the FSA ratings register with the version header, the mandatory filters and paging handled for you, and returns one flat row per establishment — name, type, address, rating, sub-scores, inspection date, authority — as JSON, CSV or Excel. Pay-per-result at $2.05 per 1,000 establishments; an empty match costs only the start fee.

We're not going to call this API easy: it hides behind a lying 404, refuses honest queries with a scary 403, and paginates region by region. Absorbing that is the product. 😈

For neighbouring lead-gen sources we also run UK company data via GLEIF LEI records, Norway's company register and BBB business leads.

The one-line version

On the FSA ratings API: x-api-version: 2 or every response is a fake 404, and always send a filter or the 403 you get isn't the ban it impersonates.

Top comments (0)