Visas pay my bills — I run an agency in Bali. The other half of my week goes into Apify Actors, and one of them reads the three Australian supermarkets in one run: ask for full cream milk and you get what Coles sells, what Woolworths sells and what ALDI sells, in the same columns.
I assumed the hard part would be getting in. It partly is — Coles sits behind a bot check — but Woolworths answers its own search service in a fraction of a second and ALDI runs a plain product API. A 46-product run finishes in six seconds. The hard part turned out to be the question that sounds trivial: are these two rows the same product?
The run that showed the problem
One keyword, three chains, 20 products each: Coles returned 20, Woolworths 20, ALDI 8. The comparison output is supposed to collapse those into one row per product with three prices side by side.
What came out: 38 comparison rows, of which only 7 had more than one chain on them. Everything else was a "single" — one chain, no comparison. For a tool whose entire reason to exist is putting three prices on one row, that is close to a silent failure: the rows look fine, the schema is right, and the field you came for is null.
Three causes, all in the data rather than the network.
1. One brand, three spellings
Here is the brand field for the same 2L bottle of a2 full cream milk, as the three chains publish it:
| Chain | brand |
name |
price |
|---|---|---|---|
| Coles | a2 Milk |
a2 Milk Full Cream Milk | 7.30 |
| Woolworths | a2 |
a2 Milk Full Cream 2L | 7.30 |
| ALDI | A2 THE A2 MILK COMPANY |
A2 THE A2 MILK COMPANY A2 Full Cream Milk 2L | 7.29 |
My matcher required the brands to agree before it would put two rows together, and "agree" meant this:
const brandKey = (row) => String(row.brand || '').toLowerCase().replace(/[^a-z0-9]/g, '');
// 'a2milk' · 'a2' · 'a2thea2milkcompany'
Three different strings, so three groups of one. The one deliberate exception — house brands, where Coles Full Cream Milk, Woolworths Full Cream Milk and ALDI's Farmdale are treated as comparable — was doing all the visible work. Everything branded fell apart.
The fix is to compare brands as sets of words instead of strings, with the corporate boilerplate dropped:
const BRAND_NOISE = new Set(['the', 'company', 'co', 'pty', 'ltd', 'group', 'brand', 'holdings']);
const brandTokens = (b) => [...new Set(String(b || '').toLowerCase()
.replace(/[^a-z0-9]+/g, ' ').trim().split(/\s+/).filter((w) => w && !BRAND_NOISE.has(w)))];
export function brandsAgree(a, b) {
const ta = brandTokens(a); const tb = brandTokens(b);
if (!ta.length || !tb.length) return false;
const [shorter, longer] = ta.length <= tb.length ? [ta, tb] : [tb, ta];
return shorter.every((w) => longer.includes(w)); // {a2} ⊆ {a2, milk}
}
{a2} is a subset of {a2, milk}, and A2 THE A2 MILK COMPANY reduces to {a2, milk} once the and company go. One row, three prices: $7.30, $7.30, $7.29.
The same fix caught something I was not looking for: Coles files the 750 mL Farmhouse Gold bottle under brand Farmhouse, Woolworths under Farmhouse Gold. Subset, so they now meet — $3.90 at Coles against $3.40 at Woolworths, a 13% gap, because Woolworths had it marked down that day. Invisible in every earlier run.
Where I stopped. Same brand, 1.5 L size: Coles calls it Pauls, Woolworths calls it Farmhouse Gold. No shared word, so those stay two single rows unless you ask for the looser equivalent mode. I could close it by reading the brand out of the name — Coles' is "Pauls Farmhouse Gold Full Cream Milk", so the words are right there. I decided not to: Woolworths also sells "Farmhouse Gold Unhomogenised Milk 1.5L", and a matcher that infers brand from names will pair that with the wrong jug. A missing comparison is a gap a buyer can see; a wrong comparison is a number they will believe.
2. Word overlap pairs two different jugs
Which brings me to the opposite failure, one row down in the same output: colesName "Pauls Zymil Lactose Free Low Fat Milk", woolworthsName "Pauls Zymil Lactose Free Full Cream Milk 2L", matchConfidence 0.81, priceGapPercent 14.
Same brand, same normalised 2000 ml, four words in common out of eight, so the overlap score cleared the threshold and the Actor announced a 14% saving between two products that are not the same product. There was a second one: Coles' plain long-life milk paired with ALDI's lactose-free long-life, reported as 12%.
Groceries declare their differences in the title, and the declarations come in two shapes that need opposite treatment:
// Exclusive: conflict only when BOTH names declare a value and the values differ.
// A chain that does not write "full cream" in the title is normal.
const ATTR_EXCLUSIVE = {
fat: { full: /\b(full cream|whole milk)\b/, light: /\b(light|lite)\b/,
lowfat: /\b(low fat|hi lo)\b/, skim: /\b(skim|no fat|fat free)\b/ },
base: { soy: /\bsoy\b/, oat: /\boat\b/, almond: /\balmond\b/, /* … */ },
flavour: { chocolate: /\b(chocolate|choc)\b/, strawberry: /\bstrawberry\b/, /* … */ },
};
// Flags: conflict when one name carries it and the other does not. Plain milk is
// not lactose-free milk, and the silent name is the plain product.
const ATTR_FLAGS = {
lactoseFree: /\b(lactose free|free from lactose|zymil)\b/, organic: /\borganic\b/,
highProtein: /\bprotein\b/, longLife: /\b(long life|uht)\b/, barista: /\bbarista\b/,
powder: /\bpowder\b/, unhomogenised: /\bunhomogenised\b/, preserved: /\b(condensed|evaporated)\b/,
};
Treating silence correctly is the whole trick. If a missing "full cream" counted as a conflict, ALDI's terse titles would stop matching anything; if a missing "lactose free" counted as agreement, every lactose-free product pairs with its ordinary twin at a fictional discount.
3. The chain lists one product twice and you pay for both
Not a matching problem — a billing one. Woolworths' search API accepts only a few page sizes, so a 20-row request is served as two pages of 12, sorted by relevance, and relevance is not stable across pages:
rows 20
DUP 2 ('208064', 'a2 Milk Full Cream 2L')
DUP 2 ('797407', 'a2 Milk Full Cream 3L')
Stockcodes 208064 and 797407 came back at positions 1 and 2, then again at 19 and 20. Twenty rows, eighteen products, charged as twenty — this Actor bills per row. And a second identical Woolworths row cannot join the group its twin already occupies, so it becomes a bogus single: two of my 38 rows were that. The fix goes where rows are counted and charged:
const idKey = row.productId ? `${row.chain}:${row.productId}` : null;
if (idKey && seenIds.has(idKey)) { stats.duplicates++; continue; }
The run summary reports duplicatesDropped, so nobody has to take my word for it.
Same input, after the three fixes
productsSeen: 46 · duplicatesDropped: 2 · rowsDelivered: 33 · 6 seconds
38 rows became 33, multi-chain rows went 7 → 9, four of them with all three chains:
| Size | Coles | Woolworths | ALDI | Match |
|---|---|---|---|---|
| 1000 ml house brand | 1.75 | 1.75 | 1.75 | own-brand |
| 2000 ml house brand | 3.40 | 3.40 | 3.39 | own-brand |
| 3000 ml house brand | 4.95 | 4.95 | 4.95 | own-brand |
| 2000 ml a2 | 7.30 | 7.30 | 7.29 | same-brand |
Fewer rows, more answers. And the answer is worth noticing: on house-brand full cream milk the three chains sit within one cent of each other at every size, so the cheapest-chain question is settled by a rounding error. The money is in brands and in whatever is marked down that morning — Farmhouse Gold at 13%, Zymil 2L at 14% once it meets the right Coles product.
Fields that are empty on purpose
Three chains, three different silences. ALDI publishes no online stock — inStock is null and availability reads In stores, so filtering on inStock === true drops every ALDI row. availableQuantity exists only at Coles, and with no store number it is a warehouse figure (2,224 for the house-brand 3L), not shelf stock. A row can have no price at all: Woolworths returned "Dairy Farmers Full Cream Milk 1L" with price: null, wasPrice: 3.00, inStock: false, and Coles returned "Geelong Full Cream Milk" with no price while still saying InStoreAndOnline — so a comparison row can legitimately carry cheapestChain: null.
One more thing that is not a bug: ALDI's search is a text match over the whole catalogue. Ask it for milk and 8 of its 30 rows are Maltesers, a Cadbury block, milk arrowroot biscuits and a milk frother. Coles returned 30 of 30 inside Milk and Long-Life Milk, Woolworths 30 of 30 inside DAIRY - MILK and BEVERAGES. The relevance filter you want is the chain's own category, in three vocabularies: Milk / Full Cream Milk, DAIRY - MILK, Dairy, Eggs & Fridge / Milk.
Calling it
No key for the sources, no browser:
import requests
rows = requests.post(
"https://api.apify.com/v2/acts/lergassy~au-grocery-scraper/run-sync-get-dataset-items",
params={"token": TOKEN},
json={"keywords": ["full cream milk"], "chains": ["coles", "woolworths", "aldi"],
"output": "comparison", "maxProductsPerKeyword": 20},
).json()
It lives here: Coles, Woolworths & ALDI Scraper. Give a run a watchName and schedule it, and later runs return only what moved — the mode I actually use.
When not to reach for it. Prices are the national online prices each chain shows a visitor with no store selected; store-level and delivery-zone pricing is not in there. Woolworths' category service refuses anonymous requests, so whole-aisle sweeps work at Coles and ALDI only. And if you only ever need one chain, a single-chain scraper is less machinery.
The lesson I keep relearning: when you join sources that never agreed on a schema, the join key is the product. Everything else is plumbing — and the plumbing was never what was broken.
Top comments (1)
This is a classic entity-resolution problem, so I would keep both the normalized match and the evidence that produced it. Brand, pack size, unit price, and normalized quantity can form a deterministic first pass; uncertain pairs should remain reviewable rather than being silently collapsed. That also makes it easier to detect when one retailer changes its product naming scheme.