DEV Community

Cover image for Scrape Tesco Data with Python: How Much Do Clubcard Prices Really Save?
Happy Endpoint
Happy Endpoint

Posted on Originally published at Medium

Scrape Tesco Data with Python: How Much Do Clubcard Prices Really Save?

Every Tesco shelf edge has the yellow Clubcard Price tags, and the question I wanted answered is simple: how much do they actually take off, and how many products have one? I collected 14 aisles of Tesco UK on 24 October 2026 with Python. The median single-item Clubcard Price is 20.9% below the shelf price. But if you only scrape the first page of each aisle, as most tutorials do, you will think offers are about twice as common as they are.

The data

Tesco has no public product API, so I used the Tesco Data API, which returns Tesco's category pages as JSON: shelf price, unit price, isForSale and a promotions list with a description, an offerType, a startDate and an endDate for each offer. Keys come from RapidAPI.

I picked 14 aisles across food, drink, household and toiletries, from Fresh Fruit to Laundry. Together Tesco lists 8,231 products in them. Category pages hold 24 products each, and a big aisle like Wine runs to 72 pages, so I did not pull everything. For each aisle I took page 1, the middle page and the last page. That turned out to matter more than anything else in this analysis.

The run cost 54 API calls, including the category tree and one exploratory search. It returned 940 product rows. After removing duplicates, products that are not for sale and one row with no price, 920 products are left.

The code

This is the whole script. It looks up aisle IDs from the category tree (/get-category-ids-for-uk), pulls three pages per aisle from /category-page, caches every response to disk, and turns each promotion's text into a price per item.

# Scrape Tesco UK aisles with the Tesco Data API and measure Clubcard Prices.
# pip install requests pandas; export RAPIDAPI_KEY=...; python clubcard_core.py
import json
import math
import os
import re
from pathlib import Path

import requests

HOST = "tesco-data-api.p.rapidapi.com"
CACHE = Path(__file__).resolve().parent.parent / "data" / "raw"
HEADERS = {"X-RapidAPI-Key": os.environ.get("RAPIDAPI_KEY", ""), "X-RapidAPI-Host": HOST}


def get(path, params, cache_name):
    """Call the API once per page and keep the JSON, so reruns are free."""
    cache = CACHE / cache_name
    if cache.exists():
        return json.loads(cache.read_text())
    resp = requests.get("https://" + HOST + path, headers=HEADERS, params=params, timeout=60)
    if resp.status_code != 200:
        print("skipped", cache_name, resp.status_code)
        return None
    CACHE.mkdir(parents=True, exist_ok=True)
    cache.write_text(json.dumps(resp.json(), ensure_ascii=False))
    return resp.json()


def aisle_ids():
    tree = get("/get-category-ids-for-uk", {}, "explore-categories-uk.json")["data"]["taxonomy"]
    return {(d["name"], a["name"]): a["id"]
            for sd in tree for d in sd.get("children", []) for a in d.get("children", []) if "id" in a}


def load_aisle(cat_id, name):
    """Page 1, the middle page and the last page of one aisle."""
    get_page = lambda n: get("/category-page", {"categoryId": cat_id, "pageNo": n}, "uk-%s-p%d.json" % (name, n))
    first = get_page(1)
    if first is None:
        return []
    last = max(1, math.ceil(first["data"]["pageInformation"]["totalResults"] / 24))
    pages = [first]
    for n in sorted({math.ceil(last / 2), last} - {1}):
        page = get_page(n)
        if page is None and n == last and n - 1 > 1:
            page = get_page(n - 1)  # the computed last page sometimes fails
        if page is not None:
            pages.append(page)
    return [dict(p, pageNo=pg["data"]["pageNo"], position=i, aislePages=last)
            for pg in pages for i, p in enumerate(pg["data"]["products"])]


MONEY = r"(?:£(\d+(?:\.\d+)?)|(\d+)p)"


def pounds(m, i=1):
    return float(m.group(i)) if m.group(i) else int(m.group(i + 1)) / 100


def parse_offer(description, shelf):
    """Turn a promotion description into (kind, min_qty, member price per item)."""
    text = description.split(" - ")[0]
    if "Meal Deal" in text:
        return "meal deal", None, None
    m = re.match(r"Any (\d+) for " + MONEY, text) or re.match(r"(\d+) for " + MONEY, text)
    if m:  # Any 2 for £4, or 2 for £5 or 4 for £8 (first tier)
        qty = int(m.group(1))
        return "multibuy", qty, pounds(m, 2) / qty
    m = re.match(r"Any (\d+) for (\d+) ", text)
    if m:  # Any 3 for 2: cheapest free, so equal prices give this per item
        qty = int(m.group(1))
        return "multibuy", qty, shelf * int(m.group(2)) / qty
    m = re.match(r"Buy (\d+) or more Save (\d+)%", text)
    if m:
        return "multibuy", int(m.group(1)), shelf * (1 - int(m.group(2)) / 100)
    m = re.match(MONEY + r" .*Clubcard Price", text)
    if m:  # £3.00 Clubcard Price, 75p Clubcard Price, £3.50 Half Price Clubcard Price
        return "single", 1, pounds(m)
    return "other", None, None


AISLES = [
    ("Fresh Food", "Fresh Fruit"), ("Fresh Food", "Fresh Meat & Poultry"),
    ("Fresh Food", "Cheese"), ("Fresh Food", "Ready Meals"), ("Bakery", "Bread"),
    ("Treats & Snacks", "Chocolate, Sweets, Mints & Chewing Gum"),
    ("Treats & Snacks", "Crisps, Snacks, Nuts & Popcorn"), ("Food Cupboard", "Cereals"),
    ("Drinks", "Fizzy & Soft Drinks"), ("Drinks", "Beer & Cider"), ("Drinks", "Wine"),
    ("Drinks", "Coffee"), ("Household", "Laundry"),
    ("Toothpaste, Mouthwash & Toothbrush", "Toothpaste"),
]


def collect():
    ids, rows, seen = aisle_ids(), [], set()
    for dept, aisle in AISLES:
        name = "".join(c if c.isalnum() else "-" for c in aisle.lower()).strip("-")[:40]
        for p in load_aisle(ids[(dept, aisle)], name):
            price = (p.get("price") or {}).get("price")
            if p["id"] in seen or not p["isForSale"] or not price:
                continue  # duplicates, off-sale items and missing prices
            seen.add(p["id"])
            offers = [(o,) + parse_offer(o["description"], price) for o in p["promotions"]]
            singles = [unit for o, kind, qty, unit in offers if kind == "single"]
            multis = [unit for o, kind, qty, unit in offers if kind == "multibuy"]
            rows.append({
                "aisle": aisle, "id": p["id"], "title": p["title"], "page": p["pageNo"],
                "position": p["position"], "aisle_pages": p["aislePages"], "price": price,
                "unit": p["price"].get("unitOfMeasure"), "n_offers": len(offers),
                "kinds": sorted({kind for o, kind, qty, unit in offers}),
                "single_off": 1 - min(singles) / price if singles else None,
                "multi_off": 1 - min(multis) / price if multis else None,
                "offers": [{"description": o["description"], "kind": kind, "qty": qty,
                            "start": o["startDate"], "end": o["endDate"]}
                           for o, kind, qty, unit in offers],
            })
    return rows


def aisle_share(g, col):
    """Page 1 is one page of the aisle, not a third of it: weight it that way."""
    pages = g.aisle_pages.iloc[0]
    first, rest = g[g.page == 1][col].mean(), g[g.page > 1][col].mean()
    return (first + rest * (pages - 1)) / pages


if __name__ == "__main__":
    import pandas as pd

    df = pd.DataFrame(collect())
    df["any_offer"] = df.n_offers > 0
    df["clubcard_price"] = df.single_off.notna()
    summary = pd.DataFrame({
        "products": df.groupby("aisle").size(),
        "any_offer": df.groupby("aisle").apply(aisle_share, "any_offer"),
        "clubcard_price": df.groupby("aisle").apply(aisle_share, "clubcard_price"),
        "median_saving": df.groupby("aisle").single_off.median(),
    })
    print(summary.sort_values("clubcard_price", ascending=False).round(2).to_string())
Enter fullscreen mode Exit fullscreen mode

The output, as shares (0.44 is 44%):

                                        products  any_offer  clubcard_price  median_saving
aisle
Chocolate, Sweets, Mints & Chewing Gum        50       0.61            0.44           0.21
Fizzy & Soft Drinks                           74       0.62            0.43           0.24
Toothpaste                                    67       0.36            0.36           0.48
Crisps, Snacks, Nuts & Popcorn                74       0.42            0.25           0.20
Bread                                         66       0.21            0.20           0.13
Beer & Cider                                  62       0.30            0.18           0.17
Cereals                                       73       0.15            0.13           0.19
Ready Meals                                   62       0.66            0.12           0.25
Fresh Fruit                                   73       0.19            0.10           0.23
Fresh Meat & Poultry                          64       0.24            0.08           0.17
Coffee                                        58       0.22            0.05           0.36
Laundry                                       60       0.07            0.04           0.16
Cheese                                        65       0.35            0.03           0.21
Wine                                          72       0.01            0.01           0.16
Enter fullscreen mode Exit fullscreen mode

The part that does the real work is parse_offer. Tesco does not give you a member price as a number. It gives you a sentence like £2.95 Better than Half Price Clubcard Price or Any 2 for £4.50 Clubcard Price - Selected Sliced And Grated Cheese, and you have to turn that into a price per item yourself. I split the offers into three kinds: a single Clubcard Price (one item, lower price), a multibuy (you must buy 2, 3, 4 or 6) and a meal deal (a bundle price across several products).

Offers crowd onto page 1

This was the finding I did not expect. In the order the category pages come back, promoted products sit near the top of most aisles. In my sample, 55.8% of page-1 products had some promotion, against 37.0% on the middle page and 18.7% on the last page. The first three slots of page 1 were even more loaded: 54.8% of those 42 products had a single-item Clubcard Price.

55.8% of products on page 1 of a Tesco aisle carry a promotion, against 37.0% on the middle page and 18.7% on the last page.

The effect is strong but not universal. Page 1 had the higher share in 10 of the 14 aisles: coffee 74.1% against 19.4% deeper in the aisle, fresh fruit 59.3% against 15.2%, and wine 92.3% against 0.0%. In chocolate, fizzy drinks, ready meals and fresh meat, offers are spread through the whole aisle and page 1 had slightly fewer than the deeper pages. So there is no single correction factor; it has to be done aisle by aisle.

That means the raw sample says 39.8% of products have an offer and 22.8% have a Clubcard Price, and both numbers are too high, because page 1 is 39.1% of my sample but only one of the 6 to 72 pages in each aisle. Weighting page 1 as one page and letting the middle and last pages stand for the rest, the estimate across the 14 aisles is 26.5% with any offer and 14.1% with a single-item Clubcard Price. Roughly 1 product in 7.

That is an estimate from two deep pages per aisle, not a census. Treat the per-aisle figures as "a lot", "some" or "very few" rather than as exact numbers.

Where the Clubcard Prices are

An estimated 43.6% of products in the chocolate and sweets aisle and 43.1% in fizzy and soft drinks carry a single-item Clubcard Price, against 3.1% in cheese and 0.6% in wine.

Chocolate and sweets (43.6%) and fizzy drinks (43.1%) lead, with toothpaste close behind at 35.9%. Cheese (3.1%), laundry (3.7%) and wine (0.6%) sit at the bottom. Wine is the clearest case of the page-1 effect: a third of the products I sampled had an offer, all of them on page 1, and the middle and last pages of a 72-page aisle had none at all (0 of 46 products).

Ready meals look odd at first: 66.3% of products carry a promotion but only 12.3% a Clubcard Price. Most of the rest are meal deals, which I come back to below.

How much a Clubcard Price takes off

Across 210 single-item Clubcard Prices, the median saving is 20.9%. The middle half of offers sit between 16.7% and 30.2%, and 47.6% of them save less than a fifth. That lines up with the CMA's 2024 review of loyalty pricing, which found average member savings of 17% to 25% across five supermarkets.

Deep cuts are rare. Only 18 offers (8.6%) were half price or better. The biggest was 60%, Oral-B kids' toothbrush refills at £30.00 with a £12.00 Clubcard Price, followed by Costa instant coffee at £6.80 with a £2.95 Clubcard Price (56.6% off).

Toothpaste aisle Clubcard Prices save a median 48.5% and coffee 36.3%, against 13.3% for bread and about 15% to 17% for laundry, wine, meat and beer.

Depth varies by aisle more than frequency does. Toothpaste aisle offers save a median 48.5% and coffee 36.3%. Bread saves 13.3%. Coffee shows that frequency and depth are separate questions: only about 5% of the coffee aisle carries a Clubcard Price, but where there is one, it is big. If you build a price index, track both, or a handful of deep cuts at the top of the aisle will look like a cheap category.

The three offer kinds behave differently:

Offer kind Offers Median saving Items you must buy Median length
Single Clubcard Price 210 20.9% 1 42 days
Multibuy Clubcard Price 135 25.0% 2 to 6 (median 3) 110 days
Meal deal 52 not per product a bundle 210 days
Not a Clubcard offer 1 not computed 1 10 days

Multibuys save a little more per item, 25.0% at the median and 33.3% at most, but you have to buy at least two, and 22 of the 135 need six (the wine "Buy 6 or more Save 25%" deal). They also run far longer than single Clubcard Prices: a median 110 days against 42.

Pitfalls in the Tesco data

offerType tells you almost nothing. 397 of the 398 offers I collected were CLUBCARD_PRICING. The only other value was one STANDARD_OFFER ("Save £4.10" on a laundry basket). Meal deals, 3 for 2s and straight member prices all share the same type, so you must parse description. Count anything with CLUBCARD_PRICING as a discount and Ready Meals looks like Tesco's most discounted aisle, when most of its offers are bundles like "WASABI DINE Meal Deal for £10 Clubcard Price - Buy 1 Main, 1 Sauce & 1 Side" on £6.00 items.

Compute the saving; do not read it from the label. The labels are accurate but rounded down. All 11 "Half Price" offers were exactly 50%. "Better than Half Price" ranged from 52.9% to 60.0%, "Save 25%" from 25.0% to 27.0%, and "Save 1/3" from 33.3% to 34.8%. Most offers have no label at all, only a price, so you need the arithmetic anyway.

Multibuy savings depend on the basket. "Any 2 for £4.50" on Tesco Grated Mozzarella 250g at £2.65 works out at £2.25 each, 15.1% off. But "any" means you can mix products from the offer range, and if the other item is cheaper your saving is smaller. "Any 3 for 2" gives the cheapest item free, so it only equals 33.3% when the prices match. And tiered offers like "2 for £5 or 4 for £8" need a choice; I used the first tier.

Page 1 returns more than 24 products. resultsPerPage says 24, but page 1 returned 24, 26 or 27 rows depending on the aisle. Two of the three duplicate IDs in my 940 rows were page-1 top slots that showed up again deeper in the same aisle. A third product, a Fit Foods ready meal, sat in both Fresh Meat & Poultry and Ready Meals. Deduplicate on id before counting anything.

Unit prices come in five units. Of 940 rows, 554 were priced per kg, 267 per litre and 111 per item, plus 4 per kg DR.WT (drained weight, for things like burrata in brine) and 3 per 100g. The toothpaste aisle alone mixes 43 products priced per litre with 24 priced per item, and the per-item group includes tubes of toothpaste, not only brushes and refills. Compare unitPrice only within the same unitOfMeasure.

Offers are a snapshot with a clock on them. Single Clubcard Prices run in short cycles: 37 of them ended on 4 November and 27 on 27 October. 67 of the 398 offers ended within a week of my collection date. A one-day scrape tells you about this week's cycle, not about Tesco in general. Some member prices are not short at all: 30 single Clubcard Prices ran for more than 90 days.

Off-sale products still appear. 17 rows had isForSale set to false, five of them in Bread. They still carry a price and sometimes promotions, so filter them before averaging.

Failed pages are gaps, not empty pages. A few of my requests for deep pages came back with a server error instead of products. The script prints and skips a failed page, and steps back one page when the last page fails; in a scheduled job, retry failures with a short backoff. If you treat a failed page as "no products", you will quietly bias every share towards page 1.

Some labels are not in the data at all. None of the 940 rows mention Tesco's Aldi Price Match, so if you need that badge, this endpoint will not give it to you.

What I would do next

One day of data shows the cycle, not the trend. The next step is to run the same aisles daily for a month, which would show how often products move in and out of Clubcard Prices and whether the shelf price moves when they do. The same script works on Tesco Ireland by swapping in /ireland-category-page and the Irish category tree.

The API and its endpoints are on the Tesco Data API page. If you want the whole catalog in one file instead, there is a Tesco UK products dataset, and the product side of the API is covered in our LinkedIn guide to Tesco grocery data. For a custom basket or daily tracking, get in touch.

Top comments (0)