DEV Community: Russell Yapp

What a data-quality page for scraped price data looks like (and the numbers behind ours)

Russell Yapp — Tue, 13 Oct 2026 12:25:28 +0000

A scraper's README usually ends with a field list: price, unitPrice, ean, inStock, and so on. It tells you what a row can hold. It does not tell you how often the field is filled, whether a join between two shops lands on the same product, or whether last Tuesday's run worked at all. Those are the questions that decide whether the data is usable, and a field list cannot answer them.

So for the scrapers I run on Apify (UK supermarkets, Australian supermarkets, Google Hotels prices and UK property listings) I published a data-quality page with five measures per data source, each with its sample and date. This post explains what each measure catches and what ours show, including the parts that are not flattering.

The five measures

1. Identifier coverage. The share of rows that carry a barcode (EAN/GTIN), or for sources with no barcode, the identifier that plays the same role. It catches the difference between "the schema has an ean field" and "you can actually join on it". In the 4 October 2026 weekly snapshot, Tesco rows carried a barcode on 99.1% of 23,352 rows, Sainsbury's on 100.0% of 26,511 and Asda on 98.9% of 20,196. Aldi and Morrisons publish no barcodes at all, and the page says so rather than leaving a blank. For hotels the identifier is Google's hotel id, which every row carries by construction, so the page says that is not a measured figure. For property it is the UPRN, which only one portal gives.

2. Cross-retailer match accuracy, with its method. If you join the same product across shops, how often is the join right? Ours is measured on every pair of rows from two different UK retailers with an identical matchKey (a normalised brand, name and pack size) in the same 4 October snapshot: 4,251 pairs. Where both rows carry a barcode (4,225 pairs), the pair is scored by whether the barcodes agree: 4,124 do, 97.6%. That is conservative, because a product listed under a second barcode at one shop counts as wrong. The 127 pairs a barcode cannot settle were read by hand on brand, name, pack size and price; counting those, 4,228 of 4,251 are the same product, 99.5%.

The caveat matters as much as the number: exact matchKey joins find only about 14.8% of the pairs that a barcode proves are the same product. Shops word the same product differently. So the key is precise but finds few matches, and an accuracy figure without that caveat would mislead anyone planning a join on it.

3. Run success. The share of scheduled canary runs that succeeded over a stated window. For UK grocery that is 36 runs from 25 September to 13 October 2026 from two daily canaries, all 36 marked SUCCEEDED. Two things about that definition are stated on the page. A run counts as succeeded when the platform marks it SUCCEEDED, so warnings inside a succeeded run are not counted as failures. And a canary is a small smoke test: it shows that a run can reach its sources, not that every field is complete. The hotels canary shows what a real failure looks like in this measure: 13 of 14 runs (92.9%), with the 1 October run failing on a place lookup, fixed that week.

4. Per-field completeness. The share of rows with a value for each field that matters. For the 73,745 UK rows in the 4 October snapshot, shelf price is 100.0%, unit price 99.6% and pack size 89.9%. The per-retailer split is where it gets useful: pack size is 100.0% for Asda but 84.1% for Tesco and 86.9% for Sainsbury's, and Aldi's unit price is 94.1%. An overall figure hides which shop is short.

5. Known gaps, stated plainly. The limits a buyer would otherwise find the hard way, in one list per source.

What the numbers turned up

Writing the page down forced some findings into the open.

Aldi rows mostly cannot be joined automatically. Aldi publishes no barcodes and no stock status, and its products rarely share an exact name with another retailer's. The page's advice is to compare Aldi by unit price instead.
Woolworths stock is always true. In a 7,194-row Woolworths sample (store 1101, 7 October 2026), stock status is present on 100% of rows, but every value is true, because the listing shows only available products. A completeness figure alone would have called that field perfect.
Zoopla's UPRN depends on the listing type. With details on, 47% of rows (71 of 150) carried a UPRN in a London for-sale sample of 24 September 2026 that included 67 new-homes listings, against 70% (7 of 10) in a separate 10-listing sample with new homes excluded. Both are small, dated samples, and the page says so; the known gap is that UPRN and EPC rating are often missing on new-homes listings.
The hotels bookingUrl is null by design. Listing it as a known gap is cheaper than answering the same question from every new user.
Morrisons has no barcode measure, because it publishes none. The page shows that as a stated "None", not a missing number.

How the numbers are produced

Three sources, no estimates:

Weekly snapshot files. For UK grocery, barcode coverage, completeness and match accuracy are counted from the latest complete weekly snapshot, one row per product per retailer, with the week and date stated next to each figure.
Canary run metadata. Run success is read from the scheduled canaries' run records: the platform status, the dates and the count, nothing inferred.
Hand checks. Where a measure needed a person, as with the 127 match pairs a barcode could not settle, the sample size and the method are stated with the figure.

Every figure is dated, and where there is no measurement the row is left out rather than filled with a guess. The page carries its last-updated date at the top (13 October 2026 for the figures here); each grocery figure names the weekly snapshot it came from, so check both before quoting one.

If you want to see the raw material, the free daily UK basket is on Kaggle with a first-look notebook, and the scraper itself is the UK Supermarket Price Scraper on Apify.

Reading anyone's quality claims

When a data vendor, including me, tells you the data is accurate, ask four things: accurate on what sample, measured how, as of when, and what does the method miss? A match rate with no recall figure, a success rate with no definition of success, or a completeness figure with no per-source split is a number you cannot use yet. The useful version is usually less impressive and much more helpful.

Prices are online shelf prices as published by each retailer on the day shown; they are not store-specific unless stated. This article is not affiliated with, endorsed by, or partnered with any of the retailers, portals or sites named; their names are used only to describe public data, and product names, images and descriptions remain theirs.

29 pack-size changes in a week: what a shrinkflation scan of 71,000 UK grocery products found

Russell Yapp — Mon, 12 Oct 2026 17:05:59 +0000

Shrinkflation is easy to suspect and hard to prove: a pack gets smaller, the price stays put, and the only trace is a number on the label. I wanted to know how often it shows up in UK supermarket data in a single week, so I compared two weekly snapshots of the online catalogues of Tesco, Sainsbury's, Asda and Aldi, taken on 27 September and 4 October 2026.

What the scan found

About 71,000 products were on sale at the same retailer in both weeks (Tesco 22,574, Sainsbury's 25,369, Asda 19,868, Aldi 3,588). Of those, 29 changed their pack-size text between the two snapshots:

6 got smaller while the shelf price held or rose (the shrinkflation pattern);
3 got larger;
5 were the same pack written differently (175g becoming 0.18kg, for example);
15 could not be compared, usually because one week gave a count ("8 pack") and the other a weight, or no size at all.

Of the six, three were confirmed on the retailers' own product pages:

Asda, Kenco Millicano instant coffee, both jars. The Decaf and the Original (renamed Americano) went from 100g to 90g, and the shelf price from £3.50 to £4.00. Per kilo that is £35.00 to £44.44, up 27%. Asda shows the £4.00 as a Rollback price, with a was-price of £6.92.
Sainsbury's Cheese & Garlic Stuffed Mushrooms. 200g to 190g at an unchanged £2.10. The site still showed the old unit price (£10.50 per kg) the following week; at 190g it works out at about £11.05.

The other three need a human. In each the product's name changed along with its size: a seasonal chocolate figure whose listed weight fell with its new name although its unit price implied the same weight both weeks, a multipack whose bar count dropped while its unit price suggests the old count was a labelling error, and a sauce jar replaced under the same product id by a different, smaller kit. Those are exactly the cases an automatic scan gets wrong, so they are flagged rather than reported.

Three confirmed cuts in 71,000 products is a footnote, not a trend. It is one week of data; the scan runs again each month.

Method

Each retailer's whole online range is captured once a week into a dataset, one row per product with the shelf price, loyalty price, pack size and unit price as the retailer publishes them. The scan pairs each product with itself in the previous week by the retailer's own product id, and falls back to a shared EAN barcode when the id changed. It reads the pack size into grams, millilitres or a count, and treats a change within the rounding of the shorter form as a rewrite, not a new pack. Unit prices are converted to per kilo or per litre, because one retailer switches between per 100g and per kilo from week to week.

A product is flagged as shrinkflation only when the quantity fell and the shelf price held or rose. Any product whose name changed as well is marked for a hand check, because a new product under an old id is the commonest false alarm. Promotions are kept as they are: a price shown as a Rollback or a loyalty price is reported as such, not removed.

The same weekly snapshots feed a free index, the UK online shelf-price index. The newsletter first reported these finds in the week 41 issue.

Prices are online shelf prices as published by each retailer on the day shown; they are not store-specific. This article is not affiliated with, endorsed by, or partnered with any of the retailers named; their names are used only to describe public product data, and product names, images and descriptions remain the retailers'.

Coles, Woolworths and Aldi prices in one schema: what a weekly snapshot actually captures

Russell Yapp — Wed, 07 Oct 2026 12:39:25 +0000

The problem

Coles, Woolworths and Aldi Australia have three sites, three schemas and no shared product id. We wanted one weekly snapshot of all three in a single row shape, with a key to join rows on. The Australia Supermarket Price Scraper, an Apify Actor, returns exactly that. This post covers what a real weekly snapshot of all three looks like, measured on Wednesday 7 October 2026 (ISO week 2026-W41), and the one hard constraint we hit: Coles.

The snapshot schedule

Coles' and Woolworths' weekly specials start on Wednesday, so the snapshot runs every Wednesday, in five slots from 07:00 UTC. This week it made seven runs, all completed: Coles in 2 runs, Woolworths in 4, Aldi in 1.

Retailer	Runs	Products	Requests	Notes
Woolworths	4	16,430 (7,194 + 2,515 + 5,582 + 1,139)	947	No blocks. Store 1101 (the national default) on every run. Everyday Market (third-party marketplace) products excluded
Aldi	1	2,560	102	Plain direct connection, no proxy needed
Coles	2	6,628 (3,350 + 3,278)	81 to 82 per run	Stopped at Coles' request limit, as designed

That is 25,618 rows in total.

The join keys

Every row carries matchKey: a normalised brand, name and pack size. It is the recommended way to match the same product across all three retailers.

Rows can also carry ean, the GTIN-13 barcode, validated by its check digit. Woolworths rows carry one on nearly every row. Aldi publishes no barcode anywhere, so Aldi rows carry an empty ean array and join on matchKey. Coles listing rows, which is what the weekly snapshot reads, carry no barcode either; they also join on matchKey.

The hard constraint: Coles

Coles' bot protection answers about 80 requests per session and then stops. The Actor never evades or rotates around a block. When Coles stops answering, the Actor stops cleanly for Coles and keeps the rows it has already returned.

So the question was how much of Coles fits into about 80 requests. Two choices:

The snapshot reads listing pages only: 48 products per page, with no per-product barcode request.
Categories are ordered staples first: dairy, eggs, bread and bakery, meat, fruit and vegetables, pantry basics, then the rest.

The result this week was 6,628 Coles products in 2 runs (3,350 and 3,278, each stopping at 81 to 82 requests as designed). Coles lists 30,609 products, so that is 21.7% of the range, against a planned 22.9%. It is a partial snapshot, and the staples are the part covered.

A finding that changed the product

From 6 October, Coles' per-product data pages grew several times over, to about 40 to 50 KB each. A listing page carries 48 products in about 48 KB, roughly 1 KB per product, so a Coles row with a barcode lookup now takes dozens of times the proxy transfer of a listing row.

So from version 0.1.8, Coles barcode lookup (colesProductDetails) is opt-in and off by default. Product mode always looks barcodes up. Left off, Coles search and category rows have an empty ean and a run gets far more Coles products within the limit. Turn it on if you need barcodes and can accept far fewer Coles products per run.

What a row looks like

One row, from a Coles product page captured on 24 September 2026 (with unitPrice, categoryPath, imageUrl and scrapedAt illustrative, not real):

{
  "retailer": "coles",
  "retailerProductId": "1902454",
  "ean": ["9310036048985"],
  "matchKey": "pauls zymil lactose free full cream milk 2000ml",
  "name": "Pauls Zymil Lactose Free Full Cream Milk",
  "brand": "Pauls",
  "packSize": "2L",
  "price": 5.95,
  "wasPrice": 6.9,
  "unitPrice": { "value": 2.98, "measure": "l" },
  "loyaltyPrice": null,
  "promotionText": "save $0.95",
  "promotionEndsAt": null,
  "inStock": true,
  "categoryPath": ["Dairy, Eggs & Fridge", "Milk", "Long Life Milk"],
  "url": "https://www.coles.com.au/product/pauls-zymil-lactose-free-full-cream-milk-2l-1902454",
  "imageUrl": "https://cdn.productimages.coles.com.au/productimages/...",
  "rating": null,
  "currency": "AUD",
  "market": "au",
  "storeId": "7674",
  "onSaleDate": null,
  "scrapedAt": "2026-09-24T10:15:00Z"
}

This row has an ean because it came from a product page. A Coles row from the listing-only snapshot would have an empty ean.

Cost of the snapshot

The whole weekly snapshot (7 runs, 25,618 rows) cost $0.49 of platform usage. Of that, 0.039 GB of residential proxy ($0.31) went on Coles and Woolworths. Per 1,000 rows: Woolworths about $0.022, Coles listing-only about $0.016, Aldi about $0.010.

Those are our platform costs, not what you pay. Your price is below.

How to run it

Search mode, one keyword across all three retailers:

{
  "mode": "search",
  "queries": ["milk"],
  "maxItems": 60
}

Category mode, from a category URL copied from the retailer's site (the retailer is inferred from the URL):

{
  "mode": "category",
  "categoryUrls": ["https://www.coles.com.au/browse/dairy-eggs-fridge/milk"],
  "maxItems": 200
}

With Apify's Python client (apify-client 3.x, where call() returns a run object), then grouping rows on matchKey to see one product across retailers:

from collections import defaultdict
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("yappman/au-supermarket-price-scraper").call(
    run_input={"mode": "search", "queries": ["milk"], "maxItems": 60}
)

by_product = defaultdict(list)
for row in client.dataset(run.default_dataset_id).iterate_items():
    by_product[row["matchKey"]].append((row["retailer"], row["price"]))

for key, prices in by_product.items():
    if len(prices) > 1:
        print(key, prices)

Pack sizes are written differently across retailers, so a matchKey join will miss some products that are the same in the shop. Use ean where it is present.

Special Buys

Version 0.1.8 also adds a Special Buys mode ("mode": "specialBuys") for Aldi Australia's dated Special Buys drops, which Aldi puts in store on Wednesdays and Saturdays. For example, the 7 October drop was 94 products. Rows carry a new onSaleDate field with the drop date; the field is null for Coles and Woolworths rows.

Pricing and limits

Pricing is pay-per-event: $2.00 per 1,000 products returned on the Free tier, $1.20 per 1,000 at Gold, plus $0.005 per run start. Cap spend with Apify's maximum-charge setting on the run, and with the maxItems input.

The data is public and logged-out: no login, basket or account data. Woolworths prices are for the store Woolworths assigns the session (usually 1101); there is no store selection. Aldi publishes no barcode and no online stock.

Where to find it

The Australia Supermarket Price Scraper is on the Apify Store: https://apify.com/yappman/au-supermarket-price-scraper. This Actor is not affiliated with, endorsed by, or partnered with Coles, Woolworths or Aldi; retailer names are used only to describe the public product data it returns.

Planning applications and constraints next to UK property listings (Rightmove, Zoopla, OnTheMarket)

Russell Yapp — Mon, 05 Oct 2026 16:00:52 +0000

The Actor "UK Property Listings" returns Rightmove and Zoopla rows, with OnTheMarket as an opt-in third portal, on one schema (every row has a portal field). It now has two optional planning add-ons for each listing, beside a separate sold-prices mode. This post describes what they return, what they cost and where they stop.

Store page: https://apify.com/yappman/uk-property-listings

The base: three portals, one schema

Run it from a pasted search or listing URL, from place names or postcodes with filters, or from a list of listing URLs. portals defaults to Rightmove and Zoopla; add "onthemarket" to include OnTheMarket. On OnTheMarket the price, bedroom and type filters are applied by the Actor to the rows it fetches, and a search shows at most 1,020 results. Apify Proxy is used as needed and is included in the price. Runs are capped at 1,000 rows, and maxItems defaults to 40.

Planning constraints

includePlanningConstraints looks up official constraints at each listing's coordinates from the UK government's planning data service (planning.data.gov.uk). The fields are conservationArea, listedBuilding, floodRiskZone, greenBelt, article4 and treePreservationZone.

England only. Outside England, planning.coverage.constraints is "not-covered" and nothing is charged for that listing.
The datasets are incomplete. A missing constraint is not proof that none exists.
listedBuilding is the nearest listed building or structure within 25 m. It does not mean the property itself is listed.

Nearby planning applications

includePlanningApplications returns applications near each listing: reference, council, address, description, decision state, type, received and decided dates, distance, and a link to the council's page. The data comes from a nightly copy of PlanIt's UK planning data.

planningRadiusMetres: 25 to 500, default 150.
planningYears: 1 to 3, default 1.
Up to the 100 nearest are returned, nearest first. planning.coverage.applicationsTruncated says when there were more.
Descriptions are the council's text with titled names (Mr, Mrs, Dr) removed. Applicant and agent name fields are never kept.

Sold prices

mode: "sold-prices" returns HM Land Registry sale history, read via Rightmove's sold-price pages. It is a separate mode from listings, and the planning add-ons do not apply to it.

Rightmove only, and England and Wales only.
Postcode level. A row has no street address, listing link or Rightmove id. This is a licensing choice: the addresses attached to Land Registry prices come from Royal Mail's address file.
Each row has the postcode, the postcode centroid as coordinates, lastSoldPrice, lastSoldDate and every recorded sale since 1995, newest first.

A worked input

The run behind the output below (the launch build's first smoke run): Leeds, Rightmove and Zoopla, six rows, planning applications only.

{
  "locations": ["Leeds"],
  "portals": ["rightmove", "zoopla"],
  "maxItems": 6,
  "includePlanningApplications": true
}

The planning block of one output row, from run XjlFAz1GCf8ihFZou on build 0.2.8, 5 October 2026 — the row with the most applications, as returned:

{
  "planning": {
    "applications": [
      {
        "reference": "26/00553/TR",
        "authority": "Leeds",
        "address": "Rear Of Beechwood House Ling Lane Scarcroft Leeds LS14 3HX",
        "description": "T1 Oak: (located in garden of 8 fern croft scarcroft) Reduce the lateral branches by 2.5m leaving a 2m overhang. The trees canopy is shading the boundary leylandii hedge with early signs the hedge is starting to fail . This would allow more light onto the south facing hedge.",
        "state": "Unresolved",
        "type": "Trees",
        "receivedAt": "2026-01-29",
        "decidedAt": "2026-03-23",
        "distanceMetres": 75,
        "url": "https://publicaccess.leeds.gov.uk/online-applications/applicationDetails.do?keyVal=T9M6L7JBJPA00&activeTab=summary",
        "sourceUpdatedAt": "2026-10-02T01:52:45Z"
      },
      {
        "reference": "26/01612/TR",
        "authority": "Leeds",
        "address": "Rear Of 10 Fern Chase Scarcroft Leeds LS14 3JL",
        "description": "T1 Beech: To cut back all overhanging branches by 4m to the boundary hedge to suitable growing points.",
        "state": "Unresolved",
        "type": "Trees",
        "receivedAt": "2026-03-20",
        "decidedAt": "2026-05-18",
        "distanceMetres": 100,
        "url": "https://publicaccess.leeds.gov.uk/online-applications/applicationDetails.do?keyVal=TC5DN5JBG5Y00&activeTab=summary",
        "sourceUpdatedAt": "2026-10-01T02:15:27Z"
      }
    ],
    "constraints": {
      "conservationArea": null,
      "listedBuilding": null,
      "floodRiskZone": null,
      "greenBelt": null,
      "article4": null,
      "treePreservationZone": null
    },
    "coverage": {
      "applications": "planit",
      "constraints": "not-requested",
      "constraintsCoverageNote": "England only; official datasets are incomplete, so absence is not proof of no constraint; listedBuilding means a listed building or structure within 25 m, not that this property is listed",
      "applicationsCoverageFrom": "2025-07-02",
      "applicationsTruncated": false,
      "applicationsNote": "PlanIt mirror, copied nightly (decisions on older applications can lag; see sourceUpdatedAt); council text, lightly redacted; covers applications received since applicationsCoverageFrom; applicant and agent names are never kept"
    },
    "checkedAt": "2026-10-05T15:39:14Z"
  }
}

To read it: coverage.applications is "planit" when the copy answered, "not-covered" for an area PlanIt does not currently collect, or "unavailable"; applicationsCoverageFrom is the date before which no applications are included, and applications is a list that may be empty. It is null when applications were not requested, the area is not covered, or the copy was unavailable.

Pricing

Pay per event. Platform usage is included.

Event	Charged when	Price
run-start	Once per run, only when rows arrive	$0.005 per run
listing	Each listing row	$3.00 per 1,000
listing-detail	Each detail page fetched	$5.00 per 1,000
planning-constraints	Each listing where the lookup succeeds for a point in England	$5.00 per 1,000
planning-applications	Each listing where the lookup answers, including "none nearby"; not for not-covered or unavailable	$10.00 per 1,000

Sold-price rows have their own event, sold-price, at $3.00 per 1,000.

Worked example from the README: 200 listings with details and planning constraints is 200 × $0.013 + $0.005, about $2.61.

"None nearby" is charged, because it is an answer: a listing with no applications in the radius costs the same as one with many. Not charged: a run that returns nothing; a listing outside England for constraints; a listing in an area PlanIt does not currently collect (planning.coverage.applications: "not-covered"); and an applications lookup when the copy is out of date or cannot be read, or the grid cells around the listing hold no applications from a council that counts ("unavailable"). Higher Apify plans get tiered discounts, and the maximum charge per run setting caps spend.

Limits

Constraints are England only. Applications follow where PlanIt collects: the councils of the applications in the roughly 1 km of grid cells around a listing decide, and an area whose councils PlanIt no longer collects is answered not-covered, free. Near a council boundary, a collected neighbour's applications in those cells make a listing count as covered. Sold prices are England and Wales.
Undated applications are left out. Without a received date they cannot be shown to fall inside the window.
The copy has a start date. Applications received before it are never included, whatever planningYears is set to. Each answered row carries the date as planning.coverage.applicationsCoverageFrom. At launch the copy holds about 12 months, and it deepens nightly.
The newest days arrive a few days late. If the copy's newest complete day is more than 5 days old, the add-on counts the copy as out of date. Decisions on applications received more than two months ago can lag by up to a few weeks; sourceUpdatedAt shows when each record was last copied.
Searches are user-directed, with no whole-site crawl mode. Each run is capped at 1,000 rows, and each portal caps a single search: Rightmove and Zoopla at about 1,000 results, OnTheMarket at 1,020.

Data sources

Planning constraints: public sector information under the Open Government Licence v3.0 (planning.data.gov.uk).
Planning applications: courtesy of PlanIt (planit.org.uk).
Sold prices: HM Land Registry data © Crown copyright and database right 2026, under the Open Government Licence v3.0. Postcode centroids contain OS data and Royal Mail data © Crown copyright and Royal Mail copyright and database right 2026, with source Office for National Statistics under the OGL v3.0.

The Actor is not affiliated with or endorsed by Rightmove, Zoopla or OnTheMarket. You are responsible for your own use of the data.

Comparing UK supermarket prices programmatically: Tesco, Sainsbury's, Asda and Aldi in one schema

Russell Yapp — Mon, 05 Oct 2026 09:21:21 +0000

The problem

Four supermarkets, four sites, four different schemas, and no shared product id between them. If you want to compare Tesco, Sainsbury's, Asda and Aldi prices programmatically, what you actually need is one row shape that fits all four, and a key you can join rows on. We built the UK Supermarket Price Scraper, an Apify Actor, to return exactly that: one schema across all four retailers, with the join keys built in.

The join keys

The join key we use is ean, the GTIN-13 barcode, validated by its check digit. Tesco carries an EAN on every row we have seen. Sainsbury's carries one on nearly every row (codes that fail the check digit are dropped). Asda carries one on about 97% of rows, recovered from Asda's own product image id; the rest have none. Aldi publishes no barcode anywhere in its listing API, product API or product pages, so Aldi rows carry an empty ean array. Morrisons, added on 5 October 2026 as an opt-in fifth retailer (retailers: ["morrisons"]), also publishes no barcode, so its rows join on matchKey like Aldi's.

Every row, from all four retailers, also carries matchKey: a normalised brand, name and pack size. It is the join key for Aldi, and the recommended way to match the same product across retailers generally, since every row carries it whether or not an EAN is present.

One honest caveat: pack sizes are written differently across retailers, for example Asda's "4 pint" against Tesco's "2.272L" for the same four-pint milk. A matchKey join on own-label milk needs a normalising step of its own to catch that. An ean join does not, because the barcode identifies the exact product regardless of how its pack size is written on the page.

A worked comparison

All prices below are national online shelf prices as published on each retailer's own site on 24 September 2026. They will have moved since; treat the table as an illustration of the method, not today's prices.

Product	Tesco	Sainsbury's	Asda	Aldi
Semi-skimmed milk, 4 pints (2.27 L)	£1.65	£1.65	£1.65	£1.65
Own-label salted butter, 250 g	£1.85	£1.85	£1.85	£1.85
Granulated sugar, 1 kg	£1.09	£1.09	—	£1.09
Cravendale filtered semi-skimmed milk, 1 L	—	£1.75 (Nectar £1.40)	£1.64	—
Free-range large eggs, 6	—	£2.10	£2.10	£1.69

Asda's granulated sugar comes in a 500 g pack at £0.89, not the 1 kg pack the other three sell, so it is left out of the row above rather than compared directly.

Two things stand out. First, on plain staples the big four price-match each other to the penny: milk, butter and sugar are identical, or near enough, across Tesco, Sainsbury's, Asda and Aldi. The interesting signal is not the staple; it is loyalty pricing and the branded lines, where retailers actually diverge, as the Cravendale and egg rows above show. Second, a comparison has to use the effective price, meaning the loyalty price when one is shown, or it misreads Sainsbury's and Tesco: Sainsbury's £1.75 shelf price for Cravendale becomes £1.40 with a Nectar card, and any Tesco row carrying a Clubcard price in loyaltyPrice is cheaper than its price field alone suggests.

Code: cheapest price per product, across all four retailers

One run across all four retailers in search mode, then group the rows by ean (falling back to matchKey where ean is empty) and keep the cheapest effective price per product:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("yappman/uk-supermarket-price-scraper").call(
    run_input={
        "mode": "search",
        "retailers": ["tesco", "sainsburys", "asda", "aldi"],
        "queries": ["semi skimmed milk"],
        "maxItems": 200,
    }
)

def effective_price(row):
    loyalty = row.get("loyaltyPrice")
    price = row["price"]
    return loyalty if loyalty is not None and loyalty < price else price

cheapest = {}
for row in client.dataset(run.default_dataset_id).iterate_items():
    key = row["ean"][0] if row["ean"] else row["matchKey"]
    price = effective_price(row)
    if key not in cheapest or price < cheapest[key]["price"]:
        cheapest[key] = {"row": row, "price": price}

for entry in cheapest.values():
    row = entry["row"]
    print(row["retailer"], row["name"], entry["price"])

effective_price treats a loyalty price as the real price whenever it is lower than the shelf price. Asda and Aldi rows always fall through to price, because those two retailers never set loyaltyPrice.

History and trend

For a longer view than one run gives you, the Actor also publishes a free daily dataset: a fixed basket of 212 staple products, up to three products per retailer for each of 20 staples (milk, bread, eggs, butter, cheddar, chicken breast, bananas, apples, potatoes, onions, tomatoes, pasta, rice, baked beans, cereal, coffee, tea bags, sugar, orange juice, olive oil), across Tesco, Sainsbury's, Asda and Aldi. It is fetched every morning and appended to one dataset, on the same schema as the Actor's own output, so the same products build a price history from 24 September 2026. Filter or group on scrapedAt for a given day's snapshot.

JSON: https://api.apify.com/v2/datasets/ynAT9NPps2EdjMOJa/items?format=json
CSV: https://api.apify.com/v2/datasets/ynAT9NPps2EdjMOJa/items?format=csv

Neither URL needs an account or token. The same basket is also on Kaggle as a CSV under CC BY 4.0, refreshed monthly: UK Supermarket Prices: Daily 212-Item Basket.

For a market-level view rather than a basket, there is also the UK Online Shelf-Price Index: a weekly index of posted online prices at the same four retailers, with the week of 27 September 2026 (2026-W39) as 100. It is published as aggregates only (a headline, each retailer and each product class, with the number of matched products behind every figure). Each week is appended and never revised, and the four-week change appears from W43.

JSON: https://api.apify.com/v2/datasets/AvRCWqFinCiNjx3sP/items?format=json
CSV: https://api.apify.com/v2/datasets/AvRCWqFinCiNjx3sP/items?format=csv

It measures posted online shelf prices at these four retailers, not what shoppers pay, and it is not the ONS CPI. The method is written up at https://data.yappman.com/shelf-price-index. One row per index value:

import pandas as pd
import requests

INDEX = "https://api.apify.com/v2/datasets/AvRCWqFinCiNjx3sP/items"
rows = requests.get(INDEX, params={"format": "json"}, timeout=60).json()
retailers = pd.DataFrame([r for r in rows if r["recordType"] == "retailer"])
print(retailers.pivot(index="vintage", columns="retailer", values="level"))

A week-on-week change per product, once the rows are in a dataframe:

import pandas as pd
import requests

DATASET = "https://api.apify.com/v2/datasets/ynAT9NPps2EdjMOJa/items"
rows = requests.get(DATASET, params={"format": "json"}, timeout=60).json()
df = pd.DataFrame(rows)
df["scrapedAt"] = pd.to_datetime(df["scrapedAt"])
# Keep the last seven days: from the start of the day a week before the newest run.
start = df["scrapedAt"].max().normalize() - pd.Timedelta(days=7)
df = df[df["scrapedAt"] >= start].sort_values("scrapedAt")
weekly = (
    df.groupby(["retailer", "retailerProductId"])
    .agg(first_price=("price", "first"), last_price=("price", "last"))
)
weekly["change"] = weekly["last_price"] - weekly["first_price"]

Cost and limits

Pricing is pay-per-event: $0.005 once per run, plus $2.50 per 1,000 products returned ($0.0025 each) on the Free plan, with discounts on higher Apify plans. Cap spend with Apify's maximum-charge setting on the run, and with the maxItems input. The data itself is public and logged-out: no basket, account or delivery-slot information. Asda's stock flag is for a single store, set by asdaStoreId, though Asda's prices themselves stay national. Aldi publishes neither stock nor a barcode for any product, which is why its rows join on matchKey instead of ean. Morrisons is opt-in for now and will join the default retailers once its daily canaries have run clean for two weeks.

Where to find it

The UK Supermarket Price Scraper is on the Apify Store: https://apify.com/yappman/uk-supermarket-price-scraper. This Actor is not affiliated with, endorsed by, or partnered with Aldi, Sainsbury's, Asda or Tesco; retailer names are used only to describe the public product data it returns.

How to get Google Hotels prices with an API in 2026

Russell Yapp — Wed, 30 Sep 2026 08:32:03 +0000

The short answer

Google has no public API for the prices it shows in Google Hotels. The old hotel-price programmes were closed to new users years ago, and nothing replaced them. If you need those prices as structured data now — for a rate-shopping tool, a travel app, a competitor check or an AI agent — you have two options: build and maintain your own scraper, or call one that is already maintained.

I built the second option. The Google Hotels Price Scraper is an Apify Actor. You give it a destination and dates, or specific hotels, and it returns one row per property with the lead nightly price, the per-vendor rates behind it (Booking.com, Expedia, Agoda and the rest), star class, guest rating, review count, amenities and coordinates. You call it through Apify's API, its client libraries, or the console.

Why the existing scrapers keep failing

Google's hotel pages don't serve tidy data. Prices, ratings and vendor offers sit in positional arrays, where a field is identified by its index rather than its name. When Google shifts an index, a scraper doesn't crash. It quietly returns the wrong number in the right-looking field, or nothing at all. Keeping one working means fixtures for every response type, golden tests that catch a shifted field, and a daily check that fails loudly when the shape changes.

I can only speak to what is public. The most-used Google Hotels scraper on the Apify Store showed 22,296 successful runs out of 53,949 in its last 30 days, a 41.3% success rate (checked 28 September 2026). Its description documents one input form while the live form is another, and it leaves the proxy setting for the user to choose. Whatever the cause, it means a large share of runs come back empty, and you pay for the attempt.

The design here is the opposite. The Actor manages its own network access, so there is no proxy setting to get wrong. A run that gets nothing back fails with a clear status instead of returning placeholder rows. A daily check runs the public build and fails if the row shape changes.

What the Actor returns

Each row carries:

hotelId, name, propertyType, hotelClass
rating and reviewCount
address, latitude, longitude
checkIn, checkOut, nights, adults, rooms — the searched itinerary, on every row
leadPrice — nightly price, total for the stay, currency, and, with vendor rates on, which vendor offers it
vendorRates — one entry per vendor with its cheapest total for the stay and how many offers it covers
amenities and amenityCodes, url, scrapedAt

Prices are pre-tax, as Google's nightly figure is, and they are in the currency you ask for. The Actor never converts them.

A worked example

A search for hotels in Lisbon for 12 to 14 October 2026, with Apify's Python client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("yappman/google-hotels-prices").call(
    run_input={
        "mode": "search",
        "searches": [{"destination": "Lisbon", "checkIn": "2026-10-12", "checkOut": "2026-10-14", "adults": 2}],
        "maxItems": 20,
        "includeVendorRates": True,
    }
)
for row in client.dataset(run.default_dataset_id).iterate_items():
    lead = row["leadPrice"]
    print(row["name"], lead["perNight"], lead["total"], lead["currency"])

The same call as one curl request that waits for the run and returns CSV:

curl -X POST "https://api.apify.com/v2/acts/yappman~google-hotels-prices/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN&format=csv" \
  -H "content-type: application/json" \
  -d '{"mode":"search","searches":[{"destination":"Lisbon","checkIn":"2026-10-12","checkOut":"2026-10-14"}],"maxItems":20}'

A 20-hotel search takes about 10 seconds. Two real rows from a run on 29 September 2026:

Name	Stars	Rating (reviews)	Per night	Total, 2 nights
Independente Principe Real	3	4.3 (1,020)	$120.76	$242
Lisbon Wine House - Rooms & Suites	3	3.7 (125)	$74.99	$150

With includeVendorRates on, the first hotel came back with 16 vendors. The four cheapest for the same two nights:

Vendor	Per night	Total
Vio.com	$121	$242
goseek.com	$121	$242
Super.com	$126	$253
Evendo	$134	$267

Each entry also carries vendorDomain (the vendor's site, for example super.com) and offerCount, the number of Google offers folded into that vendor's cheapest. There is no booking link: Google's links carry session identifiers, so the Actor never passes them on. Use vendorDomain together with the row's url, which is the Google hotel page.

These prices were captured on 29 September 2026 and will have changed by the time you read this. Treat them as illustrative.

Cost

Pricing is pay-per-event, and platform usage is included for normal runs. On the Free plan:

$0.002 per hotel row ($2.00 per 1,000)
$0.0025 per row that carries vendor rates ($2.50 per 1,000), only when you ask for them
$0.01 per hotel row in history mode with two or more snapshots
$0.005 per run as Apify's start fee, for a default 1 GB Actor

The Bronze, Silver and Gold plans get 10%, 20% and 30% off the row events. The search above, 20 hotels, costs 20 × $0.002 + $0.005 = $0.045. Five hotels with vendor rates cost 5 × $0.002 + 5 × $0.0025 + $0.005 = $0.0275. A blocked or empty run charges no row events. You can cap spend with Apify's maximum-charge setting and the maxItems input.

Rate history for tracked stays

Once a week the Actor re-reads a set of tracked hotels and stores a dated lead-price snapshot. The tracked set is 50 cities across the US, UK and Europe, each with two stays: one 14 days out and one 42 days out, for 2 adults and 2 nights. history mode returns those snapshots for hotels you name, oldest first, with the change since the previous snapshot. It makes no Google request. A hotel and dates that aren't tracked return no row and cost nothing. A free tracked mode lists exactly which hotel ids and dates are covered, so run that first.

Limitations

No browser is needed and none is used. The Actor reads the same data Google serves to any visitor.
No personal data. There are no reviewer names and no review text. The tracked list holds hotel ids and stay dates only, and nothing about who runs the Actor.
It's public, logged-out data: what any visitor sees for those dates and that party size.
One room per search, because Google ignores the field. Filters on price, rating and star class work; free cancellation and amenity-label filters aren't supported yet.
Vendor rates work in search mode. Looking up specific hotels by id returns their details without a vendor breakdown.
Very narrow filters can end a search early. The run summary's stoppedEarly says when that happened.
History covers tracked stays only, and no more than 100 hotels per run.

Where to find it

The Google Hotels Price Scraper is on the Apify Store as yappman/google-hotels-prices. It is not affiliated with, endorsed by or partnered with Google; Google and Google Hotels are named only to say where the data comes from. Its terms restrict automated access, so you are responsible for how you use the data. This is not legal advice.

How to get Tesco prices with an API in 2026

Russell Yapp — Thu, 24 Sep 2026 16:37:53 +0000

The short answer

There is no public Tesco price API today. Tesco ran a developer programme in the past, but it was retired years ago and nothing has replaced it. If you need Tesco's shelf prices as structured data now, for a price-comparison tool, competitor monitoring, an inflation tracker or an AI shopping agent, the practical choices are the same as for any other major UK retailer: scrape tesco.com yourself, or use a maintained scraper that already exposes the data through an API.

We built the second option. The UK Supermarket Price Scraper is an Apify Actor that reads Tesco, Sainsbury's, Asda and Aldi and returns one schema of structured product rows, callable through Apify's API, its client libraries, or the Apify console.

Why doing it yourself is expensive to keep working

Keeping a Tesco scraper working is not a one-off job. Tesco's site is protected by a bot-management service, so a plain HTTP client needs serious engineering to get a consistent response. Tesco's front end also changed its URL scheme and rendering during 2026, which breaks any scraper written against the old pages. Clubcard pricing adds a further wrinkle: single-item Clubcard prices and multibuy or meal-deal Clubcard offers look similar on the page, but only the single-item price is a number anywhere in the data, and the rest only ever appears as promotion text, so a naive parser silently drops it. Keeping up with all of that, in practice, means fixtures for every page type, a canary run that catches breakage before customers do, and someone on call for when the site next changes.

What the Actor returns for Tesco

The Actor runs in three modes for Tesco: search (keywords, run against Tesco's own search), category (a tesco.com browse URL, copied from the site, for example .../shop/en-GB/browse/<department>/<aisle>/... including the /all form), and product (a Tesco product page URL, for example https://www.tesco.com/shop/en-GB/products/257265436). All three modes return the same row shape.

For Tesco, each row carries:

retailer: always tesco
retailerProductId: Tesco's own product id, the tpnc
ean: the barcode, present on every Tesco row we have seen
name, brand, packSize: as shown on the product page (own-label rows carry Tesco as the brand)
price: the current GBP shelf price
loyaltyPrice: the single-item Clubcard price, when one is shown
promotionText: multibuy and meal-deal Clubcard offers, as free text (Tesco's wasPrice is always null)
inStock: Tesco's own online availability flag, national rather than per store
categoryPath, url, imageUrl, rating, scrapedAt: as elsewhere in the schema

A worked example

A search for milk on Tesco, using Apify's Python client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("yappman/uk-supermarket-price-scraper").call(
    run_input={"mode": "search", "retailers": ["tesco"], "queries": ["semi skimmed milk"], "maxItems": 50}
)
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(row["name"], row["price"], row["loyaltyPrice"], row["ean"])

The same call as a single curl request to Apify's run-and-wait endpoint, returning CSV directly:

curl -X POST "https://api.apify.com/v2/acts/yappman~uk-supermarket-price-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN&format=csv" \
  -H "content-type: application/json" \
  -d '{"mode":"search","retailers":["tesco"],"queries":["semi skimmed milk"],"maxItems":50}'

Two real rows from that search, captured on 24 September 2026:

Name	tpnc	Price	Loyalty price	Promotion text	EAN
Tesco British Semi Skimmed Milk 2.272L, 4 Pints	254656543	£1.65	none	none	5000436589457
Arla LactoFREE Semi Skimmed Milk Drink 2L	309449739	£3.15	£2.50	£2.50 Clubcard Price	5000181042139

All prices here are national online shelf prices as published on tesco.com on 24 September 2026. They will have changed by the time you read this; treat them as illustrative, not current.

Cost

Pricing is pay-per-event, and platform usage is included in it. Two charges apply: $0.005 once per run as a start fee, and $2.50 per 1,000 products returned ($0.0025 each) on the Free plan, with discounts on higher Apify plans. A run that searches 5 terms across all four retailers and returns 300 products costs 300 × $0.0025 + $0.005 = $0.755. You can cap spend with Apify's maximum-charge setting on the run, and with the maxItems input.

Limitations

This is public, logged-out data, with the same limits as the rest of the Actor. Tesco prices and stock are both national figures; there is no store-level stock and no store selection for Tesco. Tesco's own search mixes in marketplace-seller and F&F clothing listings, and the Actor skips both, returning only grocery products. There is no basket, account or delivery-slot data, and no Clubcard account is used; everything comes from pages Tesco serves to any visitor, logged out.

Two more things worth knowing

If you want a look at real numbers before writing any code, there is a free daily dataset: a fixed basket of 212 staple products across Tesco, Sainsbury's, Asda and Aldi, fetched every morning and appended to one dataset since 24 September 2026, no account or token needed:

https://api.apify.com/v2/datasets/ynAT9NPps2EdjMOJa/items?format=csv (CSV; format=json gives JSON).

And if you want an alert rather than a one-off pull, there is an n8n workflow for a daily price-drop alert on a shopping basket, built on this Actor: import basket-price-drop-alert.json into n8n.

Where to find it

The UK Supermarket Price Scraper is on the Apify Store: https://apify.com/yappman/uk-supermarket-price-scraper. This Actor is not affiliated with, endorsed by, or partnered with Tesco; retailer names are used only to describe the public product data it returns.