If you want Australian supermarket prices as rows, the awkward part is not the first read. It is the second one: next week you do not want 60 products again, you want the ones that moved. This post shows a ready-made way to get that for Aldi Australia with an Apify Actor I publish, and what it costs.
One limit up front. Coles and Woolworths are not available at the moment. The Actor reads Aldi Australia only, so everything below is Aldi. I am not describing any Coles or Woolworths feed because there is not one to describe right now.
The Actor is Aldi Australia Price Scraper & Tracker: Special Buys, published by Yappman Data: https://apify.com/yappman/au-supermarket-price-scraper
The four example Tasks
An Apify Task is a saved set of inputs for an Actor. I published four for Aldi Australia, each with its own page:
- Aldi milk price changes, weekly watch: https://apify.com/yappman/au-supermarket-price-scraper/examples/aldi-milk-price-changes-weekly-watch
- Aldi staples: weekly price changes: https://apify.com/yappman/au-supermarket-price-scraper/examples/aldi-staples-weekly-price-changes
- Aldi Special Buys: new products each week: https://apify.com/yappman/au-supermarket-price-scraper/examples/aldi-special-buys-new-products-weekly
- Aldi whole range, all 14 departments: https://apify.com/yappman/au-supermarket-price-scraper/examples/aldi-whole-range-all-14-departments
The milk, staples and Special Buys Tasks use watch mode. The whole-range Task is a plain read.
As of Actor version 0.1.20 you can read the whole range with one input instead of pasting department URLs. Set fullRange to true in category mode, leave categoryUrls empty, and the Actor reads every department of Aldi Australia's online range in a fixed order:
{
"mode": "category",
"fullRange": true,
"maxItems": 500
}
maxItems still caps what is returned and charged. The default is 60, which fills mostly or entirely from the first department or two, so raise maxItems if you want a spread across the range. The run summary shows fullRange: true. It works with a watch too. Prices and the other inputs are unchanged.
How watch mode works
Give a run a watchId and it becomes a watch. The first run is a baseline: it returns the products in scope. Each later run with the same watchId and the same search returns only products that are new to the watch or whose price changed. Rows carry a watchEvent field (baseline, added or changed) and, for changes, the old price and the move as a percentage.
The milk watch input is:
{
"mode": "search",
"retailers": ["aldi_au"],
"queries": ["full cream milk"],
"maxItems": 60,
"watchId": "milk-weekly"
}
The watch keeps its memory in a key-value store named grocery-watch-<watchId> in your own Apify account, so your baseline is yours and nobody else's runs affect it. Delete that store and the next run is a first run again.
There is also minMovePct, which ignores price moves under a percentage. Small moves add up and are reported once they reach it.
What each Task costs
Pay per event. On the Free plan a product is $0.0015 and the run start is $0.005. Paid Apify plans pay less per product.
| Task | First run | Later runs |
|---|---|---|
| Milk watch | up to about $0.095 (60 x $0.0015 + $0.005) | $0.005 start fee plus any changed rows |
| Staples watch | up to about $0.91 (600 x $0.0015 + $0.005) | $0.005 start fee plus any changed rows |
| Special Buys | at most about $0.46 (300 products); about $0.34 for the 224 listed on 5 Nov | $0.005 start fee plus any new or changed rows |
| Whole range | 500 products about $0.76 | on demand, no watch |
The whole range is about 2,600 products when last counted, which comes to about $4 on the Free plan ($0.0015 per product plus the $0.005 start fee). The Task is set to 500 on purpose, so one click costs about $0.76. Change Max items if you want more.
A run that returns nothing costs only the start fee. If no Special Buys drop is listed, that Task returns no rows.
Use Apify's maximum cost per run setting if you want a hard ceiling on any run.
Setting up the weekly schedule
- Open the milk watch Task and run it once. That is the baseline.
- Keep the
watchIdand the search words. Changing either on the samewatchIdis refused, so use a new id for a new scope. - In the Console open Schedules, add the Task and pick a time. The README example uses
0 21 * * 0, which is Sunday 21:00 UTC. - Read each week's rows where
watchEventischangedoradded.
If you want to be told when something moves, the README describes a webhook you can add to the Task.
Special Buys works the same way. Each drop is listed by Aldi a week or two ahead, and the Task's first run returns the products of today's and later drops as a baseline. After that you get new or changed products.
What a row holds
One row per product: price, unit price, was-price where Aldi shows one, promotion text (on Special Buys rows, the on-sale line), a matchKey and a scrapedAt time. Aldi publishes no barcode, so the ean field is empty and matchKey, a normalised brand, name and pack size, is the join key. Aldi reports no stock online, so inStock is null.
Prices are as shown to a visitor with no store selected. You cannot choose a store.
Limits worth knowing
- Aldi Australia only. Coles and Woolworths are not available at the moment.
-
fullRangeis for Aldi Australia in category mode withcategoryUrlsempty. The range is about 2,600 products when last counted, and that count changes. - A watch scope of more than 3,000 products, a read that comes back much smaller than last time, or a read that fails part-way is not compared. Nothing is charged beyond the start fee, and the run summary says why.
- Changes that do not fit under
maxItemscome back in the next run. That includes the first run: a scope bigger thanmaxItemsis returnedmaxItemsat a time, each charged. - A product that leaves the scope and comes back is returned again as
added, and charged. - A watch run that crashes part-way can miss changes for good. It never reports or charges a change twice.
- Special Buys mode reads only the drops Aldi lists at the time of the run. An earlier drop that is no longer listed cannot be read.
- Special Buys rows have no end date and no stock level.
A free overview
If you only want weekly index numbers rather than rows, our free index is at https://data.yappman.com/au-shelf-price-index/. It covers Coles, Woolworths and Aldi Australia shelf prices, with partial Coles coverage.
Independent tool. Not affiliated with, endorsed by or sponsored by Coles, Woolworths or Aldi.
Top comments (1)
Filtering deltas server-side before returning results is a smart design for a weekly feed — it keeps downstream pipelines clean and means users only pay to move data that actually changed. For other Actor builders thinking through how pricing models interact with feed designs like this one, there is some useful context here: dev.to/ispyhumanfly/proxy-bills-ar.... Are the deltas computed against a snapshot you persist in Apify's key-value store between runs, or do you diff against the previous dataset output?