DEV Community: Russell Yapp

Tracking changes to the UK sponsor register with a change feed

Russell Yapp — Tue, 03 Nov 2026 17:57:02 +0000

The UK Home Office publishes a register of licensed sponsors for the Worker and Temporary Worker routes. It publishes only the latest file. If an organisation was added last month, or dropped off last week, the register does not tell you when.

I have built an Apify Actor, UK Sponsor Register Changes, that keeps that history from the day we started collecting. It has been public since 3 November 2026. This post explains what it returns, how to run it, and what it will not tell you.

Why a change feed rather than a full list

The register is useful as a lookup. It is less useful if your question is "what changed since I last looked?". Compliance, HR and immigration advisers, and analysts who follow the sector, usually want that second question answered:

which organisations have been added;
which are no longer listed;
which have had a rating changed;
which have had a route added or removed.

Comparing two large files by hand each time is slow and easy to get wrong. A feed gives you only the rows that differ, each with a date, so you pay for and read only the changes.

What a change row contains

One row per change. According to the README, each row has the date, the organisation, town, county, route, the old and new type and rating, where it came from, and its id and sequence. The README's sample row (an invented organisation, not a real one):

date	changeType	organisation	town	county	route	oldTypeRating	newTypeRating
2026-01-05	added	Example Org Ltd	Exampleton	Examplesire	Worker	null	Worker (A rating)

The change types are an organisation added, no longer listed, a rating changed, or a route added or removed. "Removed" means only that the organisation is not in the file of that date. It does not say why.

How to run it

The Store page is here: https://apify.com/yappman/uk-sponsor-register-changes

The simplest input caps the run at three rows:

{"maxItems": 3}

The Console form starts at 3 rows, so a default run costs up to $0.015. To narrow a run, the README's example input uses a start date, a change type and a county:

{"changedSince": "2026-10-19", "changeTypes": ["added"], "counties": ["West Yorkshire"], "maxItems": 50}

Set maxItems to cap what a run can cost. A call that leaves it out returns up to 100 rows.

What it will not tell you

It is not advice. It is a dated record of changes to a public register. Being on the register says nothing about the status of any job or any visa outcome.
It is not a compliance check on a named employer. I do not offer it as a way to confirm an employer's current standing.
It does not have the register's full history. The register publishes only its latest file, so history starts on the day we began collecting. Collection is intended each working day, and any day we missed is listed in RUN_SUMMARY (daysMissed).
It does not give names of individuals, by design. Entries that look like an individual's name are designed to be left out and counted in RUN_SUMMARY (withheldByType; counts under 5 show as <5). The check is rule-based and can be wrong both ways. It can leave out a genuine organisation, and it can let through a company, school or trading name that belongs to one person.
It holds back unusual days. If a comparison shows more than 3% of the register changing in one day, we hold it back for checking rather than publish it.
Your use is your responsibility. We cannot control what happens to rows once they are delivered.

Cost

Pay per event: $0.005 for each change row returned, which is $5.00 per 1,000. The price is the same on every plan, with no start fee and no run fee. A run that is refused before any data is read, that finds no changes in your window, or that finds the change log unavailable costs nothing.

Some figures from the README: a run capped at 3 rows costs up to $0.015; a run with no cap returns up to 100 rows, so up to $0.50. For comparison with other tools, check other sponsor-register Actors on the Store yourself; I make no claim about relative price.

Privacy

The rows are organisation names from a public register, and some organisations are sole traders. The privacy notice for this feed, including how to object or ask for an entry to be removed, is at https://data.yappman.com/privacy/. A removal changes our record and later runs. It cannot recall rows already delivered.

Licence and attribution

Contains public sector information licensed under the Open Government Licence v3.0. Source: Register of licensed sponsors: workers, UK Visas and Immigration, GOV.UK.

Not affiliated with or endorsed by the Home Office or UK Visas and Immigration.

I would like to hear if a field is missing or a row looks wrong. Ratings and reviews on the Store page also help other buyers judge whether it fits.

Russell Yapp, Yappman Data

Week 44 shelf-price indices: UK down 0.24%, Australia up 0.48%

Russell Yapp — Mon, 02 Nov 2026 11:56:23 +0000

I publish two weekly indices of posted online supermarket prices, one for the UK and one for Australia. Both for 2026-W44 went out on 2 November 2026. This post covers the headline moves, what the indices do not measure, and how to read the data yourself.

What the indices are

Each week I read the online catalogues of the retailers, match products week on week by the retailer's own product id, and chain the geometric mean (a Jevons index) of the price ratios. Retailers get equal weights. There are no spending weights. The indices are intended weekly, normally on Monday.

They are not CPI and not an official statistic. They measure posted online shelf prices, not prices paid.

UK, 2026-W44

The headline for Tesco, Sainsbury's, Asda and Aldi is 99.72, down 0.24% on the week. The base is 100 at 2026-W42, when the series restarted after a skipped week 41. It rests on 74,335 matched products. Food and soft drinks is 99.81, down 0.21%.

By retailer, week on week:

Tesco: +0.05%
Sainsbury's: +0.04%
Asda: -0.17%
Aldi: -0.88%

I am not giving a cause for Aldi's fall. The index shows the movement only.

Australia, 2026-W44

The like-for-like headline for Coles, Woolworths and Aldi Australia is 100.64, up 0.48%, from a base of 100 at 2026-W41. It rests on 21,029 matched products, in Australian dollars. Food and soft drinks is 100.23, up 0.05%.

Across all matched products, by retailer: Coles +0.15%, Woolworths +1.19%, Aldi Australia -0.10%.

Personal care is the class that moved most, up 7.15% on the page. Shelf prices include promotions by design, so a change in how many products are on offer can move a class even when regular prices do not.

One limit matters here: Coles is a partial slice, with 6,425 products used (21.0% of the Coles lists) this week.

Read the data

The full UK series is a public CSV under CC BY 4.0. It is long format, with one row per vintage, record type, retailer and class.

import pandas as pd

url = ("https://api.apify.com/v2/datasets/AvRCWqFinCiNjx3sP/items"
       "?format=csv&clean=true")
df = pd.read_csv(url)
head = df[df["recordType"] == "headline"]
print(head[["vintage", "level", "weekOnWeekPct", "matched"]]
      .sort_values("vintage"))

The column names come from the data dictionary. Levels from different series (base W39 and base W42) cannot be compared.

Where to get it

UK W44: DOI 10.5281/zenodo.23881940, and the Hugging Face dataset yappman/uk-online-shelf-price-index.
AU W44: DOI 10.5281/zenodo.23882013, and the Hugging Face dataset yappman/au-online-shelf-price-index.
Pages and methods: UK index, AU index.

The indices publish aggregates only. Product-level shelf prices can be read on demand with the UK Supermarket Price Scraper, a pay-per-event Actor on the Apify Store. For Australia, the Australia Supermarket Price Scraper reads Aldi Australia; the AU index page says Coles and Woolworths are read only on runs started by the Actor owner's account. Check each Store page for current pricing.

Cite as "Yappman Data, UK Online Shelf-Price Index" or "Yappman Data, AU Online Shelf-Price Index", CC BY 4.0, with the vintage.

Australian Government contract notices from AusTender as fixed rows, with supplier names left out

Russell Yapp — Sun, 01 Nov 2026 10:43:41 +0000

Australian Government agencies publish the contracts they award on AusTender, the government's procurement site, which the Department of Finance runs. Every contract at or above the reporting threshold appears there as a contract notice, and notices can later be amended. The data is open, and AusTender has an API that returns it in the Open Contracting (OCDS) format.

I built an Apify Actor that reads those contract notices and returns one fixed row per notice: AusTender Contract Notices. You can look up notices by id, search by date and filters, or run a daily alert that returns only notices it has not sent before.

It is independent: not affiliated with or endorsed by the Department of Finance or AusTender.

What the data is

AusTender materials are released by the Department of Finance under the Creative Commons Attribution 3.0 Australia licence (CC BY 3.0 AU). Only contracts at or above the reporting threshold are published on AusTender: A$10,000 for most agencies, higher for corporate Commonwealth entities (Commonwealth Procurement Rules, paragraph 7.18). These are contract awards and their amendments. They are not open tenders.

What a row holds

One row per notice or amendment, in a fixed field order. This is the README's sample, shortened to the main fields:

cnId	agency	value	currency	contractStart	contractEnd	publishedAt	procurementMethod
CN4280221	National Indigenous Australians Agency	44490.6	AUD	2026-09-22T14:00:00Z	2027-06-29T14:00:00Z	2026-10-01T23:51:13Z	Limited tender

The full row also has these fields:

parentCnId (the parent notice for an amendment, otherwise null), ocid and releaseId;
title, the agency's contract reference as published, and unspscCodes;
limitedTenderExempt, the notice's exemption flag as published;
dateSigned, lastModifiedAt and amendments (the number of amendments);
noticeUrl, which opens the notice on the AusTender API;
alertFingerprint (alert mode only), plus attribution and licence, which carry the credit line on every row.

What is left out

Version 1 outputs no supplier's name, legal name, ABN or party id, and has no supplier filter. The notice link (noticeUrl) opens the notice on the AusTender API, which does name the supplier.

A notice's description (its subject line) can name a person, so it is never output and there is no keyword search. The title field is the agency's contract reference, not a subject. It is output only when it is reference-shaped: it contains a digit, contains no whitespace and is at most 40 characters. Otherwise it is null. Contact details and addresses are never read.

If a notice names you, or your business as a sole trader, the README says you can ask for it to be left out by emailing info@yappman.com with the CN id; the Actor acts on a request within one month.

Three modes

Lookup (cnIds): up to 20 CN ids, such as CN4280221, or an amendment id such as CN4271813-A1. Each returns the latest version of its notice, with its amendment count. An id AusTender does not know is reported as not found and costs nothing.
Search: a date range of up to 92 days (default 31) on one of four dates (dateType: contractPublished, contractLastModified, contractStart or contractEnd). Filters are agencies, unspscCodes, valueMin and valueMax (in AUD), procurementMethods (Open tender, Limited tender, Prequalified tender), and contract start and end dates.
Alert (alertId): run it daily. Each run reads the last 14 days of contract last-modified dates, so amendments are caught, and returns only the notices this alert has never sent before. An alert keeps what it has sent in your own key-value store, named after the alert. Do not put a person's name in alertId.

An alert sends at most maxItems new notices a run, newest first, so narrow it with filters and set maxItems for its expected volume. The README says unfiltered daily volume is in the high hundreds.

Example input

A search for one agency's larger contracts published in October 2026:

{
  "mode": "search",
  "dateType": "contractPublished",
  "dateFrom": "2026-10-01",
  "dateTo": "2026-10-31",
  "agencies": ["National Indigenous Australians Agency"],
  "valueMin": 100000,
  "procurementMethods": ["Limited tender"],
  "maxItems": 50
}

And from Python, with apify-client and an Apify token:

import os
from decimal import Decimal

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("yappman/au-contract-notices").call(
    run_input={
        "mode": "lookup",
        "cnIds": ["CN4280221"],
    },
    max_total_charge_usd=Decimal("0.01"),
)
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(row["cnId"], row["agency"], row["value"], row["currency"], row["procurementMethod"])

What it costs

Pay per event: one notice event for each row returned, in every mode. On the Apify Free plan that is $0.002 per notice, which is $2.00 per 1,000. It is $0.0018 on Bronze, $0.0016 on Silver and $0.0012 on Gold and above. There is no start or run fee, so a run that returns no notices costs nothing.

Worked examples on the Free plan, from the README:

a Console default run (the form starts maxItems at 3) costs up to $0.006;
one notice looked up costs $0.002;
through the API, maxItems defaults to 1,000, so a run with no maxItems costs up to $2.00. Set it;
an unfiltered alert with maxItems 1000 costs up to $2.00 a day.

When not to use it

This is not legal or procurement advice, and the Actor does not claim to be complete. Only contracts at or above the reporting threshold are on AusTender at all. AusTender materials are subject to change and should be verified on the AusTender website (www.tenders.gov.au). Because supplier names are left out and there is no keyword search, it will not tell you who won a contract; the notice link on AusTender does.

Source and credit

The output carries this credit on every row:

AusTender materials are released by the Department of Finance. AusTender materials are subject to change and should be verified on the AusTender website to ensure the information is up to date and correct: www.tenders.gov.au. Derived and modified by Yappman Data from AusTender contract notices: fields selected, renamed and redacted, and supplier names and identifiers left out.
Licensed under CC BY 3.0 AU: https://creativecommons.org/licenses/by/3.0/au/

The Store page has the full input reference: yappman/au-contract-notices.

Australian education providers (CRICOS) as clean rows, with the names that look like people left out

Russell Yapp — Sat, 31 Oct 2026 12:16:54 +0000

Australia keeps a public register of education providers that may enrol overseas students: the Commonwealth Register of Institutions and Courses for Overseas Students, known as CRICOS. The Department of Education publishes it on data.gov.au as CSV files under the Creative Commons Attribution 2.5 Australia licence (CC BY 2.5 AU). The files are open, but they come as separate tables with long address blocks, and a provider can be a company, a university, a TAFE or a sole trader.

I built an Apify Actor that reads two of the files, Institutions and Courses, and returns one fixed row per provider: CRICOS Register Lookup (Australia). It is ready for a spreadsheet, a script or an AI agent.

Not affiliated with or endorsed by the Department of Education.

What a row holds

One row per provider, in a fixed field order:

providerCode, the CRICOS provider code (five digits and a capital letter, for example 00001K);
providerName, when the name rule shows it (see below), and nameWithheld, which says whether the name is left out;
institutionType as the register spells it (for example Government or Private);
state, postcode and city of the provider's postal address;
institutionCapacity, as a whole number;
courseCount (rows in the Courses file for the provider, expired courses included) and activeCourseCount (the ones the register marks as not expired);
website, when neither name is left out;
snapshotDate, from the Last-Modified header of the Institutions file the run read.

There are no postal address lines, location names, contact details or course fees in the output. The Institutions file does contain addresses and websites; the Actor reads it in memory for the run and returns only the fields above.

Two modes

search (the default) returns the providers that pass your filters, sorted by provider code. Filters are states (ACT, NSW, NT, QLD, SA, TAS, VIC, WA), institutionTypes, up to 50 postcodes, and nameContains (2 to 80 characters, matched against the name in the output). maxItems caps the rows returned: 100 by default through the API, 10 in the Console form, at most 2,000.

lookup takes 1 to 20 provider codes. Codes that are not in the register are listed in the run summary and the rest are returned.

The name rule

Some providers are sole traders, and a register of businesses can still carry a person's name. So the Actor does not simply print every name. It leaves a name out unless the name rule finds an organisation cue and no person shape. Where the name is left out, the row still has the provider code, type, state, postcode and city, and nameWithheld is true.

The rule is a pattern test, not a judgement about any person. It can be wrong both ways: it can let through a college named after a person, and it leaves out a name made of what looks like a person's name plus a word such as College, even when it belongs to an organisation. A search by name only matches the names that are shown, so it cannot be used to find a name that was left out.

The share left out is about 24% of providers (365 of 1,547, measured 28 October 2026). A row whose name is left out still gives the postal state, postcode and city, which for a sole trader can be a home locality. If you appear in the register and want your entry left out, the privacy page on the Yappman Data website describes how to ask.

Price

$0.002 per row returned, which is $2.00 per 1,000 rows, flat on every Apify plan. There is no start fee and no run fee. Refused, empty and source-problem runs are not charged. The run's maximum charge stops it at the last row it can pay for.

When not to use it

Do not use it to decide whether a provider may enrol a student, or as proof of a provider's registration. The register is the authority; the Actor returns a dated copy of two of its files. A provider that is missing from a result has not necessarily closed.

Source and credit

The data is CC BY 2.5 AU. The Actor's output carries this line:

Contains data from the Commonwealth Register of Institutions and Courses for Overseas Students (CRICOS), Department of Education, Australian Government, licensed under Creative Commons Attribution 2.5 Australia (CC BY 2.5 AU). This Actor filters and reshapes the data; changes were made.

It makes two requests to data.gov.au per run, builds them itself (the input takes no URLs, headers or query strings) and stops if the source answers with a rate-limit refusal.

The Store page is here: yappman/cricos-register. Ratings and reviews there help other buyers judge whether this Actor fits their use.

Aldi Netherlands, Aldi Nord and Aldi Süd shelf prices in EUR

Russell Yapp — Fri, 30 Oct 2026 13:15:35 +0000

I have published an Apify Actor that reads shelf prices from three Aldi shops: Aldi Netherlands (aldi.nl), Aldi Nord Germany (aldi-nord.de) and Aldi Süd Germany (aldi-sued.de). All three return the same row shape, in euros. This post covers what a row holds, the three ways to run it, what it costs and what it does not do.

The Actor is Aldi Netherlands & Germany Price Scraper: Shelf Prices (EUR) on the Apify Store. It is an independent tool, not affiliated with, endorsed by or sponsored by ALDI Nederland, ALDI Nord, ALDI SÜD or any ALDI company.

Both German Aldi chains

Germany has two Aldi chains, with their own shops and prices. This Actor reads both: Aldi Nord (aldi-nord.de, retailer aldi_nord_de) and Aldi Süd (aldi-sued.de, retailer aldi_sued_de). They are different sites, and each row names the shop it came from.

What a row holds

Each row is one product. Here is a sample from aldi.nl (from a capture of 27 Oct 2026):

{
  "retailer": "aldi_nl",
  "retailerProductId": "1235974",
  "ean": [],
  "name": "Romige kwark",
  "brand": "MILSANI",
  "packSize": "500g",
  "price": 1.89,
  "wasPrice": null,
  "unitPrice": { "value": 3.78, "measure": "kg" },
  "deposit": null,
  "promotionText": null,
  "categoryPath": ["Zuivel, eieren en boter", "Verse zuivel"],
  "url": "https://www.aldi.nl/product/romige-kwark-1235974.html",
  "matchKey": "milsani romige kwark 500g",
  "currency": "EUR",
  "market": "nl"
}

A few fields need a word:

unitPrice is the shop's own figure, as the shelf shows it. aldi.nl gives one on about three quarters of products, aldi-nord.de gave one on every record I have seen, and aldi-sued.de on every record I have seen except loose fruit and vegetables. Where a shop shows none, the field is null. The Actor never calculates one.
deposit is the bottle deposit (Pfand) in euros, as the shop shows it, for example 0.25. It is filled on aldi-nord.de and aldi-sued.de where the shop shows one, and is null otherwise. On aldi.nl it is always null. price stays the shelf price as the site gives it.
wasPrice and promotionText are filled while a reduction is shown (for example -10%). On aldi-nord.de a reduced price has not yet been seen in a real capture. On aldi-sued.de one was seen in a capture of 28 Oct 2026; that shop gives no end date for a reduction, so promotionEndsAt is null there.
ean is always empty, because Aldi shows no barcode on these sites. matchKey is a name and pack-size key you can join on.
categoryPath is in Dutch or German, as each shop names it.
Not returned: images, product descriptions, ingredients, nutrition, reviews and ratings. imageUrl is always null, and inStock is null because stock is per store and not shown online.

Three modes

Mode	Input	What it reads
`category` (default)	`categoryUrls`	Every product on a category listing page
`product`	`productUrls`	Named product pages
`catalogue`	`retailers`, `maxItems`	The listing pages a shop's sitemap names (about 135 for `aldi.nl`, 120 for `aldi-nord.de`); for `aldi-sued.de`, the 49 top-level category pages

All three shops work in all three modes. In category mode, aldi.nl and aldi-nord.de need a subcategory page, because their top-level category pages carry no product list and are refused with a message. On aldi-sued.de a top-level category page lists products too, and a listing is read page by page (30 products a page) until maxItems or the end of the listing. In catalogue mode retailers defaults to ["aldi_nl"]; use aldi_nord_de and aldi_sued_de for Germany.

There is no search mode. Category and product modes take the shop from each page address, so you can mix the shops in one run. maxItems caps the rows saved (default 60).

The default input reads one small aldi.nl page (eggs, about 4 products) and finishes in under a minute.

Running it from Python

With apify-client and an Apify token:

import os
from decimal import Decimal

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("yappman/nl-de-supermarket-price-scraper").call(
    run_input={
        "mode": "category",
        "categoryUrls": [
            "https://www.aldi.nl/producten/zuivel-eieren-boter/verse-zuivel.html",
            "https://www.aldi-nord.de/sortiment/milchprodukte/milch-milchgetraenke.html",
            "https://www.aldi-sued.de/produkte/kaese/frischkaese/k/1588161425467086",
        ],
        "maxItems": 30,
    },
    max_total_charge_usd=Decimal("0.055"),
)
for row in client.dataset(run.default_dataset_id).iterate_items():
    unit = row["unitPrice"] or {}
    print(row["retailer"], row["name"], row["packSize"], row["price"], unit.get("value"), unit.get("measure"))

That run returns at most 30 rows in total. max_total_charge_usd stops the run if its charges would pass the cap.

Each run also produces a run summary, which counts the records skipped for having no current price.

What it costs

The Actor is pay per event. There is a start fee of $0.005 per run, plus one product event for each row returned, and no other event. On the Apify Free plan a product costs $0.0015, which is $1.50 per 1,000. On higher plans it costs less, down to $0.90 per 1,000.

Worked examples on the Free plan:

the default run (about 4 rows) costs $0.005 + 4 x $0.0015 = $0.011;
the three-shop run above costs at most $0.005 + 30 x $0.0015 = $0.05;
1,000 rows cost $1.505.

A run that returns nothing costs only the start fee. The Store page has the current prices.

How it reads the shops

At the start of every run the Actor requests each selected shop's robots.txt. Requests go out one at a time, about one every two to three seconds, from the Apify platform's own network. If a shop answers 403 or 429, the run stops for that shop at once, with no retry. A server error is retried a few times, then stops that shop too, and so does any other answer that is not a normal page. You pay only for the rows already returned.

Limits worth knowing

Prices are those shown to a visitor with no store chosen, and they are current at the row's scrapedAt.
price is the shelf price as the site gives it, and deposit is returned separately. The Actor makes no claim about whether a shelf price includes the deposit.
Records without a current price, or not on sale yet, are skipped and counted in the run summary. That covers a promotion that has not started and an aldi-sued.de offer dated later than the run.
Aldi Nord and Aldi Süd are separate shops with their own prices, so the same product can differ between them.

The Store page has the full field list, the input reference and the changelog: Aldi Netherlands & Germany Price Scraper: Shelf Prices (EUR).

Ireland rent and house price index data as clean rows (RTB / CSO)

Russell Yapp — Thu, 29 Oct 2026 15:30:25 +0000

Ireland publishes two useful housing tables through the Central Statistics Office (CSO): the RTB Average Monthly Rent Report (table RIQ02) and the Residential Property Price Index (table HPM09). Both are open data under CC BY 4.0. Both come as JSON-stat cubes, which are compact but awkward to filter: one long array of values, with dimensions you have to decode before you can ask "what is the average rent for a two-bed apartment in Carlow?".

I built an Apify Actor that reads those two tables and returns one fixed row per figure, ready for a spreadsheet, a script or an AI agent: Ireland Rent and House Price Index (RTB / CSO).

Not affiliated with or endorsed by the Residential Tenancies Board (RTB) or the Central Statistics Office (CSO).

The two tables

Rent (RIQ02, quarterly). Average monthly rent by place, number of bedrooms, property type and quarter. The places run from counties down to towns and local areas. Where the source marks a cell as insufficient data, that cell is empty and the Actor leaves it out (and does not charge for it).

House price index (HPM09, monthly). The CSO's Residential Property Price Index, base 2015 = 100, monthly from January 2005, in 20 categories that combine an area and a type of home: National - houses, Dublin - apartments, Dublin City - houses, the regions and so on. No category is finer than a Dublin local authority or a region. It is an index of price changes, not the price of any one home.

Rents are averages of registered tenancies for an area, not the rent of any one home. The CSO tables are the authority, and the data is published some months after the period it covers.

Two modes

Choose the table with table (RIQ02 or HPM09) and the job with mode.

search returns rows for what you filter. For rent: locations, bedrooms, propertyTypes, quarterFrom and quarterTo (for example 2024Q1). For the price index: category names in locations, statistics (index, change1m, change3m, change12m, which are the CSO's own published changes), monthFrom and monthTo.
lookup gives the latest figure and the change on a year earlier, for 1 to 20 places (rent) or categories (price index).

Values are spelled as the table spells them, for example Carlow, Two bed, Apartment. A bedrooms or property type value the table does not have refuses the run before anything is charged.

What a row looks like

A rent lookup for Carlow, all bedrooms and all property types, run on 28 Oct 2026, returned one row. Selected fields:

Field	Value
`location`	Carlow
`bedrooms`	All bedrooms
`propertyType`	All property types
`latestQuarter`	2025Q4
`latestAverageMonthlyRentEur`	1349.13
`yearEarlierQuarter`	2024Q4
`changePercent`	10.3
`table`	RIQ02

changePercent is worked out from the two published averages and rounded to one decimal place.

A price index lookup for Dublin - all residential properties in the same test returned a latest index of 178.4 for August 2026, against 171.8 a year earlier, with the CSO's published 12-month change of 3.9%.

How many rows a lookup returns

This is the one thing to know before you run it. A rent lookup returns one row per place, bedrooms group and property type. That is up to 42 rows per place (7 bedrooms groups x 6 property types), fewer where the source has no figure. In a test on 28 Oct 2026, a lookup of 2 places with no filters returned 68 rows.

If you want one row per place, set the filters to the totals:

{
  "table": "RIQ02",
  "mode": "lookup",
  "locations": ["Carlow", "Carlow Town"],
  "bedrooms": ["All bedrooms"],
  "propertyTypes": ["All property types"]
}

A price index lookup returns one row per category.

Price

One event, row: $0.001 per row returned, from either table, flat on every Apify plan. There is no start fee and no run fee. A refused run, an empty result and a source problem are not charged. maxItems (default 100) caps the rows you can be charged for, and the run's maximum charge stops it at the last row it can pay for.

Worked examples:

A rent lookup of 2 places with no filters: 68 rows, $0.068.
The same lookup filtered to all bedrooms and all property types: 2 rows, $0.002.
A price index lookup of 2 categories: 2 rows, $0.002.

Calling it from Python

With the apify-client package (3.x), set a maximum charge on the run so a wide query cannot cost more than you expect:

from decimal import Decimal
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("yappman/ireland-rent-index").call(
    run_input={
        "table": "RIQ02",
        "mode": "lookup",
        "locations": ["Carlow"],
        "bedrooms": ["All bedrooms"],
        "propertyTypes": ["All property types"],
    },
    max_total_charge_usd=Decimal("0.05"),
)
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(row["location"], row["latestQuarter"], row["latestAverageMonthlyRentEur"], row["changePercent"])

Each run also saves a RUN_SUMMARY record in its key-value store: the status, the table, the rows matched and returned, the empty cells left out, any places not found in a lookup, and the attribution lines below.

Limits

One request to the CSO per run (the whole table), at most one request every 1.5 seconds, a 60-second network timeout, and reading the source stops after 5 minutes.
A rate-limit refusal (HTTP 429 or 503) stops the run with no retry and nothing charged; RUN_SUMMARY.retryAfter says when to try again.
The input takes no URLs, headers or query strings: the Actor builds the request itself.
Nothing is read from the RTB website; only the CSO's published tables.

Credit

The data is CC BY 4.0, and the Actor's output carries these lines:

Source: RTB Average Monthly Rent Report (table RIQ02), copyright Residential Tenancies Board (RTB), published by the Central Statistics Office, Ireland, licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Changes were made: cells with no figure in the source (insufficient data) are not returned, and rows are filtered and reshaped.
Source: Residential Property Price Index (table HPM09), copyright Central Statistics Office, Ireland, licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Changes were made: the table is filtered and reshaped into one row per month, category and statistic; cells with no figure are not returned.

The rows hold area averages and index values: a place or category, a period, bedrooms and property type, and the figure. There are no fields for people, addresses, landlords, tenants or properties.

If you try it, the Store page is here: yappman/ireland-rent-index. Ratings and reviews there help other buyers judge whether it fits their use.

Six months, 6,314 real bets, +1.2%: what a betting bot taught me about data

Russell Yapp — Thu, 29 Oct 2026 11:32:34 +0000

I spent six months building a bot that placed bets on the Betfair Exchange. On 29 October 2026 I stopped live betting, paused development and switched the bot off. This is what I learned, and it is mostly not about betting.

The short version: the bot's results looked good in testing, did not follow in live running, and almost every reason I found was a data or measurement error. The models were rarely the problem.

What I built

A Betfair Exchange bot across many sports: tennis, horse racing, football, darts, cricket, golf, basketball and more. It had paper trading, an optimiser that generated thousands of model variants, and live execution at small stakes (£1 to £5). I built it with AI coding agents (Claude Code).

What happened

From 22 April to 29 October 2026 the live book settled 6,314 bets with £16,792 staked, for a net +£205 (+1.2%).

By month: April −£99 (−16.7%), May +£487 (+5.8%), June −£221 (−4.3%), July −£11 (−1.0%), August +£60 (+8.7%), September −£36 (−11.5%), October +£26 (+5.5%).

All of the profit came from May. Without May the total is about −£280. After I reset to small flat stakes in July, the result was about +£38 on roughly £2,600 staked. That is inside one standard deviation of luck, about ±£144, so it is indistinguishable from zero. The last 60 days came to about −£23, and tennis, the core of the book, ran −7.6%.

I did not find a live edge. So I stopped.

The lessons

1. Paper beats live by a lot, and it is not commission

For the same lane, the gap between paper and live return on stake (ROI) measured 6 to 19 points. Commission explained only about 0 to 1 point. Almost all of the rest was adverse selection: you get filled when you are wrong. Paper ROI also shrank by 60% or more out of sample.

2. Execution was the edge, not prediction

Across the book, only 18.5% of live orders were filled. For tennis, crossing the spread lifted the fill rate from 37.8% to 92.5%. The price advantage vanished: median closing-line value went to 0.00%, and live P&L ran −£61 in six days. The apparent edge had been queue position, not selection.

3. Phantom settlements

Bets that never matched on the exchange were settled anyway, using the market result. That created fake live P&L. I excluded 25 of them in July. It came back in October through a different code path: 31 bets that lapsed on Betfair were settled (−£9.38).

4. Survivor bias from a clock race

12.3k bets got stuck because settlement compared against an event clock that raced the market closing. The bets that did settle were a biased sample. One Formula 1 lane showed +242% on 6 bets. Its true figure was +1% on 52, and I had already put it live on the strength of the inflated one.

5. Counting the same bet many times

Hundreds of model variants placed the same bet on the same selection, and I counted each result as independent evidence. When I de-duplicated by market, selection and direction, apparently significant results collapsed. One "strong" cell fell to 48 bets with a confidence interval that spanned zero.

6. Optimisers find lucky days, not edges

71% of the high-ROI lanes that had a live record lost money live. Their backtest profit was concentrated in a handful of days. Screening by how concentrated the profit was across days worked better than screening by ROI or t-statistics.

7. Closing-line value did not predict profit

The usual rule says that if you beat the closing price, profit follows. Across my whole book the correlation between closing-line value and ROI was about −0.10. The one lane that made money live did not beat the sharp close. It was harvesting a market inefficiency.

8. A config save silently deleted 1,306 lanes

A configuration save removed 1,306 strategy lanes, including 12 of the 20 live ones, and left no log trail. I found it weeks later because performance had changed.

9. Growing stakes on winning streaks buys high

Raising stakes during winning runs meant the drawdown arrived at peak stake size. Flat stakes were safer.

10. A horse's name broke the API client

A runner with "ssl" in its name made a parse error look like a network error. That triggered bursts of "API paused" back-off.

11. The tests wrote to the live database

Running the test suite in the production environment leaked test bets into the real ledger.

12. Nearly everything tested negative

I tested 12 conditioning filters (surface, time of day, fatigue, venue and others): 11 were dead and 1 was untestable. I tried 25 more sports and markets, and every one was refuted. An LLM "advisor" that picked bets came out significantly negative (t = −2.91), and it simply copied the favourite 99.35% of the time. Switching venue was not an option either: Polymarket and Kalshi are blocked in the UK.

What it was really about: what was known when

Look back over that list. Most of it is a timing or provenance failure. Using information the bot could not have had at decision time. Clocks racing each other. UTC against BST. Dead heats. Partial updates. Duplicates counted as independent evidence.

Results that look like an edge are very often a data artefact. Whatever you are modelling, whether it is prices, property or sport, you need data that records what was known when, not just what is true now.

That is why I publish dated, point-in-time data at data.yappman.com: UK and Australian shelf prices (daily and weekly snapshots, corrected only for errors), plus property, hotel and public-contract data through Apify.

The bot is switched off; the code and data are archived.

Nothing here is betting advice.

Publishing data from UK public sources without publishing people

Russell Yapp — Wed, 28 Oct 2026 15:28:41 +0000

UK public bodies publish a lot of open data. An open licence covers the dataset's copyright. It does not make personal data in that dataset safe to republish. Both the HMRC trade lists and the Global LEI Index contain records that are really people. This article looks at how three of my Apify Actors deal with that. It describes what each Store README says, and nothing more. This is not legal advice.

The problem

A sole trader trades under their own name. A register that lists "importers and exporters" or "legal entities" will therefore list some individuals next to companies. A tender notice can name a contact. A name can sit inside a company name. If you collect these sources into one tidy table, you can end up publishing people without meaning to.

The three Actors take different routes. All three are pay per event, and the Store pages carry the current prices.

UK Importers and Exporters (HMRC Trade Info)

This Actor returns one row per trader, flow and month. HMRC publishes sole traders' names next to companies', and the README says the Actor does not return them.

Organisations only. A name is returned only if it carries an organisation marker, such as Limited, Ltd, PLC or LLP. A name without one is withheld. So is a name with a title such as Mr or Dr, unless a legal form follows. So is a name that holds a contact or an address.
Counted, not silent. In search and alert modes, withheld names are left out and counted in RUN_SUMMARY.withheldPersons.
No addresses. Address lines are never output or stored. Only the outward part of the postcode is returned, and there is no address field.
Nothing kept. Every run reads HMRC's current publication, and no trader name or address is kept between runs. An alert's store holds hashes only.

The licence is the Open Government Licence v3.0. The README adds that the licence does not cover personal data, so sole traders' details are not ours to republish.

UK Public Contracts Lookup

This Actor returns notices from Find a Tender and Contracts Finder (and Public Contracts Scotland if you choose it) in one schema. The README describes which fields a row can hold and which it never holds. That is a design choice at the field level.

No supplier names. A row holds the title, buyer, value, CPV codes, regions, stage, dates and a link. It has no winner, contact, address or description field, and the buyer field is the organisation name only. A suppliers field is reserved for organisation supplier names, pending a review; it is null in this version, and no supplier name is read.
Nothing else that identifies a person. Rows hold no contact names, emails, phone numbers, addresses, supplier names or descriptions. Keywords are matched against the description, but the description is never returned.
Redaction in titles. Email addresses, UK phone numbers and a name written straight after words such as "contact", "speak to" or "FAO" are replaced with [redacted].

The README is open about the limit: the redaction works by pattern, so a name written some other way in a title is not caught.

LEI Lookup

This Actor returns Legal Entity Identifier records from the Global LEI Index, one row per entity. The data is CC0. The README notes that CC0 applies to the dataset's copyright and does not remove data-protection duties for personal data.

Sole proprietors withheld. A recognised sole proprietor is a record with the category SOLE_PROPRIETOR, or one registered under a sole-trader legal form (98 codes in the ISO 20275 list). Arrangements named after a person are treated the same way: settlements, family, will, discretionary and life-interest trusts, and "estate of".
Two behaviours. In lei mode, the row keeps the LEI, statuses and dates, but the names, registration number and addresses are null, and the withheld field says which. In name and status modes, these entities are left out and counted in RUN_SUMMARY.withheldPersons.
City-level addresses. For everyone, an address is country, region and city only. No street, number or postcode is read.

The README states two limits. A company whose name contains a person's name is returned as published. And the sole-proprietor and arrangement tests can miss a person. The record link opens the source record, which the Actor does not filter.

What the three have in common

Each one decides at the field level what a row can hold, rather than cleaning up afterwards. Each one counts what it withheld, so a result with fewer rows than expected is explained. And each README says where its method stops.

Removal and objection

The Trade Info README sets out a route. HMRC removes a trader who asks, from the following month's publication, and the Actor stops returning it from then. You can also ask us directly. Email info@yappman.com with the trader's name as it appears in the results, or its HMRC trader id. The README says requests get a reply within one month, and that each accepted request is a reviewed change to our own removal list. Traders on it are never returned, and are counted in RUN_SUMMARY.removedByRequest.

The Contracts and LEI READMEs give the same route: email info@yappman.com with the notice id or the LEI, or, without one, the product and the name as it appears.

Who owns the output

Our position is that output belongs to the account that runs the Actor, under Apify's terms. That account is then responsible for what it does with the rows. Again, this is not legal advice. If you are unsure whether a use is lawful, ask a qualified adviser.

The Store pages

UK Importers and Exporters: https://apify.com/yappman/uk-importers-exporters
UK Public Contracts Lookup: https://apify.com/yappman/contracts-lookup
LEI Lookup: https://apify.com/yappman/lei-lookup

Drafted with an AI assistant. The behaviour described comes from the three Store READMEs as they stood on the afternoon of 28 October 2026.

Notify search engines on every GitHub Pages deploy with IndexNow

Russell Yapp — Tue, 27 Oct 2026 15:29:47 +0000

When I looked on 27 October 2026, Bing Webmaster Tools no longer showed a URL Submission page. It pointed to IndexNow instead. So I added a small GitHub Actions workflow to the repository behind data.yappman.com. After each GitHub Pages deploy, it tells IndexNow which pages changed. This article walks through it. It is our own code.

What IndexNow needs

IndexNow lets a site tell participating search engines that URLs have changed, instead of waiting for them to find out. You publish a key as a text file at your site root. The file is named after the key and contains the key.

That key is public by design. It is not a secret and does not belong in repository secrets. Each request names the file's address in a keyLocation field, so the receiver can fetch it. In our repository the workflow finds the key by looking for a 32-character hex name ending in .txt.

Why only the changed pages? Sending every URL on every deploy tells a search engine nothing about what is new. A diff keeps each submission to what changed. The workflow also needs no secret and no token. Its permissions are contents: read and nothing else.

The trigger and the gate

on:
  workflow_run:
    workflows: ["pages-build-deployment"]
    types: [completed]
  workflow_dispatch:
    inputs:
      base:
        description: "Git ref to compare with HEAD (default: the commit before HEAD)"
        required: false

The workflow runs when GitHub's own pages-build-deployment workflow completes. You can also start it by hand, with an optional base ref to compare against.

if: ${{ vars.INDEXNOW_ENABLED == 'true' && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}

The job runs only if the repository variable INDEXNOW_ENABLED is true. Within that, a manual start always passes, and a deploy-triggered start passes only if the deploy succeeded. The file's own header calls it a draft that does nothing until you set the variable (Settings, Secrets and variables, Actions, Variables).

The job checks out the deployed commit with full history (fetch-depth: 0), because the next step needs to compare commits. A concurrency group keeps two pings from running at once.

Work out the changed pages

BASE="$BASE_INPUT"
[ -n "$BASE" ] || BASE="HEAD~1"
git diff --name-only "$BASE" HEAD -- '*.html' > changed.txt
node docs/release/indexnow-urls.js $(cat changed.txt) > urls.txt

With no base given, it compares against the commit before HEAD. git diff --name-only lists the changed .html files. A small Node script then turns file paths into public URLs.

The script is short. It accepts only .html paths. It skips _held/, docs/, assets/ and .github/, the 404 page and the Google verification file. index.html becomes the site root. folder/index.html becomes https://data.yappman.com/folder/. It removes duplicates, sorts the list and prints one URL per line. The step then writes the count of URLs to the step output.

Ping IndexNow

KEY_FILE=$(ls | grep -E '^[0-9a-f]{32}\.txt$' | head -n 1)
KEY="${KEY_FILE%.txt}"
test -n "$KEY"
test "$(curl -fsS "https://data.yappman.com/$KEY_FILE")" = "$KEY"

This step runs only if the count is not zero. It finds the key file, then fetches it from the live site and checks the content matches the key. If the file is not served, the step fails before anything is sent.

curl -fsS -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  --data @body.json -w "\nHTTP %{http_code}\n"

A short inline Node command builds body.json with the host, the key, the key location and the URL list. Then curl posts it to https://api.indexnow.org/indexnow and prints the HTTP status.

Testing it

Set INDEXNOW_ENABLED to true, then start the workflow by hand from the Actions tab. Leave base empty to compare with the previous commit. Or give it an older ref, such as a tag, to cover several deploys in one go. Read the log. The changed-pages step prints the URL list, and the ping step ends with the HTTP status.

First run

Our first run submitted 20 changed URLs and got HTTP 202. I read that as accepted for processing. It does not mean the pages are indexed. Check your own search engine tools later to see what they did with the URLs.

Limits

IndexNow is a hint, not a guarantee. A search engine decides whether and when to crawl and index a page.
Google has not announced support for IndexNow. For Google, the sitemap and Search Console remain the routes.
The workflow only sees .html files that follow the folder/index.html pattern. Changes to other file types are not submitted.
git diff --name-only also lists deleted files, and the script does not check that a file still exists. A removed page would be submitted as a changed URL.
If a deploy changes nothing in HTML, nothing is sent.

About the site

data.yappman.com publishes a free weekly UK and Australian shelf-price index. Product-level shelf prices on demand come from the UK Supermarket Price Scraper on the Apify Store: https://apify.com/yappman/uk-supermarket-price-scraper

Drafted with an AI assistant. The workflow described is our own code, and the figures come from our own first run on 27 October 2026.

Get UK supermarket prices into an AI assistant via MCP: a free index server and a paid full-data route

Russell Yapp — Tue, 27 Oct 2026 12:07:00 +0000

If you ask an AI assistant what groceries cost, it will usually guess or search the web. I would rather it read a dataset. The Model Context Protocol (MCP) lets an assistant call tools on a remote server, so a price dataset can be one of those tools.

This article covers two routes. One is a free MCP server I run that serves a weekly shelf-price index and a small fixed basket. The other is Apify's hosted MCP server, which lets an assistant run two pay-per-event Actors that return product-level shelf prices on request. The free route is static snapshots and cannot answer every question. The paid route exists for the questions it cannot.

The free server has its own page, with the endpoint, tools and licence: https://data.yappman.com/mcp/.

The free index and basket are a small sample. Product-level data, on request, comes from the two paid Actors covered further down.

The problem

An assistant asked "did UK shelf prices go up this week?" has three bad options: recall something from training, search the web and quote a news article, or make up a figure. None of them gives you a dated source you can cite.

A tool call fixes the first part. The assistant asks a server for a specific week, gets rows back, and can say which week and where it came from. That is the whole idea.

For background on the wider choice of UK supermarket price sources, free and paid, I compared them in an earlier article: Where to get UK supermarket price data: free and paid, compared.

The free server

It is a small read-only server on Cloudflare Workers. It uses streamable HTTP and needs no key or account.

The address is:

https://yappman-grocery-mcp.blue-bar-ca29.workers.dev/mcp

It is also listed in the MCP Registry as io.github.yappmaniac/grocery-price-snapshots, and described at https://data.yappman.com/mcp/.

What it serves, all under CC BY 4.0:

The weekly UK and Australian online shelf-price index. The Australian index includes Coles and Woolworths rows from my own collection runs.
A fixed 212-item UK basket, with price, was-price, loyalty price, unit price, pack size and stock flag per retailer, on the days it was collected.

Product names, brands, descriptions and images are not served. Those belong to the retailers.

Add it in Claude

On a Free, Pro, Max, Team or Enterprise plan, Claude can use a remote MCP server as a custom connector. On Pro or Max:

Go to Customize, then Connectors.
Choose Add, then Add custom connector.
Enter a name (for example "UK shelf prices") and the address above, then Continue.
Under Authentication, Claude should detect "No sign-in"; leave it, then Add.
In a chat, use the "+" button at the lower left, choose Connectors, and switch the new one on for that conversation.

On Team and Enterprise plans an owner adds the connector under Organization settings first, and each person then connects it. Free accounts are limited to one custom connector, which matters later in this article.

These steps follow Anthropic's help page "Get started with custom connectors using remote MCP", which I read on 27 October 2026. Menu labels change, so if yours differ, go by that page: https://support.claude.com/en/articles/11175166-get-started-with-custom-connectors-using-remote-mcp

If you use Claude Code, this command should add it (I tested the Claude.ai route above, not this one):

claude mcp add --transport http supermarket-prices https://yappman-grocery-mcp.blue-bar-ca29.workers.dev/mcp

Add it in ChatGPT

OpenAI documents how to connect your own MCP server to ChatGPT in its developer mode help article: https://help.openai.com/en/articles/12584461-developer-mode-apps-and-full-mcp-connectors-in-chatgpt-beta

I have not tried it, and which plans and workspaces can do it, and the menu names, are set by OpenAI and may change, so I am not giving steps here. Follow that page. The free server's address is the one given above.

Example questions and the tool that answers each

You ask	Tool	Notes
Which weeks of the UK index are published?	`list_index_vintages` with `market: "uk"`	Newest first, with publication time, revision and base week.
How did UK posted shelf prices move this week?	`get_shelf_price_index` with `market: "uk"` and `recordType: "headline"`	Omit `vintage` for the latest week. Returns the index level and week-on-week change.
Which Australian retailer's shelf prices moved most last week?	`get_shelf_price_index` with `market: "au"`, then compare the retailer rows	Retailer ids are `coles`, `woolworths` and `aldi_au`.
Show dairy and eggs for one UK retailer.	`get_shelf_price_index` with `market: "uk"`, `retailer: "tesco"` and `class: "dairy-eggs"`	`class` takes a product class id.
What did this barcode cost at the UK retailers that carry it on a given day?	`lookup_basket_prices` with `eans` and `date`	Up to 20 barcodes per call. Only the 212 basket products.
What did a particular Aldi product cost?	`lookup_basket_prices` with `products: [{"retailer": "aldi", "retailerProductId": "..."}]`	The basket holds no barcodes for Aldi, so use the retailer's product id.
Can I cite this? What is the licence and which days are missing?	`about_the_data`	Returns the licence, what is not served, the published weeks and the basket date range with short and missing days.

Each result from get_shelf_price_index and lookup_basket_prices carries a credit line. A well-behaved assistant will repeat it. You can ask it to.

A prompt to try: "Use the UK shelf prices connector. Call about_the_data first, then get the latest headline index and tell me the week, the week-on-week change and the credit line."

When you need more: the paid Actors via Apify's MCP server

The free server has no product names, no search and no live lookups. Questions like "what does this product cost at four retailers right now" or "price my shopping list" need product-level data. I publish two Actors on the Apify Store for that:

UK Supermarket Price Scraper covers Tesco, Sainsbury's, Asda and Aldi.
AU Supermarket Price Scraper covers Aldi Australia. (It can also read Coles and Woolworths, but only for runs on my own account; for any other account those two are skipped and not charged.)

The Store pages are the place for current pricing and what each mode does. Both are pay per event, charged to the Apify account that runs them. The Actors return current product-level data on request: the run happens when your assistant asks for it.

Apify hosts an MCP server at https://mcp.apify.com. A tools parameter in the address limits which Actors it offers as tools. It does not limit what your sign-in can reach: the client can read other runs, datasets and stores in your Apify account. If that matters, use a separate Apify account for this. Actors are written as owner/name and separated by commas. For the UK Actor:

https://mcp.apify.com?tools=yappman/uk-supermarket-price-scraper

I have connected to that address from Claude as a custom connector: it signs in through Apify and lists the Actor and four helper tools. Apify's documentation shows several Actors in one tools list separated by commas, so the address for both is ?tools=yappman/uk-supermarket-price-scraper,yappman/au-supermarket-price-scraper. I have connected with the Australian Actor in the list, but not with this exact address.

The four helper tools are Get Actor run, Get dataset items, Get key-value store record and Abort Actor run. Claude asks for approval before each tool call unless you change that setting. That is a client setting: if you allow a tool permanently, or use a client that runs tools automatically, Actor runs will start, and be charged, without asking.

The about_the_data tool on the free server points to the two Actors and, in a fullerData field, to that combined address, so an assistant that hits the limits of the free data can point you there.

Steps

Create or sign in to an Apify account. Before connecting, set a monthly usage limit on that Apify account. Every run is charged to your account, not mine. Running these Actors costs money, so check the pricing on each Store page first.
Add the address above to your client as a remote MCP server. In Claude, that is the same custom connector route as before, using the Apify address. Free Claude accounts are limited to one custom connector, so you would need to swap out the free server or use a paid plan to have both at once.
Sign in with Apify when asked. Apify's documentation says OAuth is the recommended way to connect: your browser opens so you can sign in and approve access, and your API token stays out of the client configuration. Clients without OAuth support can send an Apify API token in an Authorization: Bearer header instead; use a scoped token and keep it out of shared or committed config.
Ask for what you want: "Use the UK supermarket Actor to find current shelf prices for semi-skimmed milk at Tesco, Asda and Aldi." Check the Actor call your client shows before you approve it.

Apify's MCP page (read 27 October 2026) gives no steps for Claude web connectors, so the Claude steps here come from my own test and Anthropic's help page.

Limits and licence

Posted online shelf prices, not CPI, not prices paid. The index is not a government statistic and does not reflect what shoppers paid at the till.
Static snapshots, not live. The free server serves compiled files. It may be incomplete or out of date. Check the retailer before relying on a price.
Free hosting. The free server runs on a free plan. Heavy use may be turned away for the rest of the day. Please do not crawl it or build production systems on it, and I may change or remove it.
Coverage. The basket is 212 products. The index has no product names. Anything wider is the paid route.
Licence. My compilation is CC BY 4.0. I claim no rights in retailer content, and none is served. Credit it as: "Source: Yappman Data, [dataset title], [vintage or date], CC BY 4.0." The index and basket tools return the exact line. Say if you changed anything.
Output of the paid Actors belongs to the Apify account that runs them, under Apify's terms. Retailer content in it (images, descriptions) is not mine to license. This is not legal advice.
Independence. Yappman Data is not affiliated with or endorsed by any retailer, government body or Apify. Provided as is, with no warranty.

Finding UK importers of a product by HS code with HMRC's open data

Russell Yapp — Tue, 27 Oct 2026 09:27:48 +0000

Not affiliated with HMRC. The data comes from HMRC UK Trade Info, published under the Open Government Licence v3.0.

If you sell something, source something or research a market, a common question is who in the UK already imports or exports it. HMRC publishes monthly lists of importers and exporters by commodity code, but the raw files are awkward: one file per month, codes at several lengths, and names mixed with sole traders.

This post goes from a product to a list of UK traders using HS and CN8 codes. It covers choosing a code, searching by code, flow and month, and reading the output in Python. The rows come from an Apify Actor I built, UK Importers and Exporters: HMRC Trade Data. It returns one row per trader, flow and month, with the outward code of the postcode only and no address. A name is kept only if it carries an organisation marker such as Limited, Ltd, PLC or LLP, so sole traders are left out. Pricing is pay per event: $0.005 a run plus $0.002 per trader row.

This is a list of companies from public records. It tells you nothing about volumes or values.

To follow along without running anything, there is a free 20-row sample on Kaggle (HS 8419, August 2026). The rows are in name order from the start of the list, so it is not a random or representative sample of the month. If you want to run a search yourself, there is a ready-made example Task: HS 8419, imports, 20 rows.

1. What the rows contain, and what they do not

Each row says that a trader dealt in goods under some commodity codes in a given month, in one flow: imports or exports. The row carries the 8-, 6- and 4-digit commodity codes that trader dealt in, and the outward code of its postcode (for example the first half of a postcode, not the whole thing).

There are no address lines, no full postcodes, and no trade values or volumes. The rows say who traded under a code, not how much.

2. From a product to a code

HS codes classify goods. The first two digits give a chapter, four digits a heading, six digits a subheading. UK trade statistics use eight digits (CN8), which narrow the subheading further. The sample above uses the four-digit heading 8419.

Pick the longest code you are sure of. A short code is broader, so it brings back more traders; a long one is narrower. If you search on a shorter code, traders whose rows carry a longer code beneath it are included. That is why each row lists its codes at several lengths (hs4Codes, hs6Codes, cn8Codes): you can check which code a trader actually matched on.

To find a code for a product, look it up in the UK Trade Tariff and note it at the length you can defend.

3. Running a search

The input takes the commodity codes, whether you want imports, exports or both, and a month range. The quickest way to see it work is the example Task linked above, which is set up for HS 8419 imports and 20 rows. Run it, open the dataset, and you have the same kind of rows described here. The Kaggle sample has both flows; the example Task is imports only, so the two will not be identical.

A run is charged at $0.005 plus $0.002 per trader row, so a 20-row search comes to about $0.045.

4. Reading the rows in Python

Export the dataset as JSON from the Apify Console and load it. This snippet counts rows by flow, lists traders that appear in both flows, and shows the most common postcode districts.

import json
from collections import Counter

with open("traders.json", encoding="utf-8") as f:
    rows = json.load(f)

print(Counter(r["flow"] for r in rows))

imports = {r["name"] for r in rows if r["flow"] == "imports"}
exports = {r["name"] for r in rows if r["flow"] == "exports"}
print("In both flows:", sorted(imports & exports))

print(Counter(r["postcodeDistrict"] for r in rows).most_common(10))

A row looks like this (illustrative only: the name is an invented example and the code lists are shortened):

{"name": "Example Trading Limited", "postcodeDistrict": "EX2", "flow": "imports", "month": "2026-09", "hs6Codes": ["..."]}

Other fields include traderId, traderKey, cn8Codes, hs4Codes, otherCodes, source, publishedAt and matchType.

5. Looking up a named company, and watching a code

The same Actor can look up a named company and can alert you to traders that are new to a code. The Actor page describes how to set either up.

6. Limits

Postcode district only: no address lines and no full postcodes.
Sole traders are left out, but the name rule is not a guarantee. If you see a name that should not be there, or you want a business removed, email info@yappman.com with the name as it appears in the data. I reply within one month.
No trade values or volumes.
HMRC publishes the lists monthly, so the data trails the month it describes. The Kaggle sample covers August 2026.
Names and codes are as published; I do not correct them.

7. Licence and credit

The data is published under the Open Government Licence v3.0, which does not cover personal data; that is why names without an organisation marker are withheld. Keep this credit line with any copy or use:

Contains public sector information licensed under the Open Government Licence v3.0 (HMRC, UK Trade Info).

Compiled by Yappman Data. Not affiliated with HMRC.

8. Cost

$0.005 a run plus $0.002 per trader row. The 20-row example costs about $0.045. Pay-per-event pricing means a larger search costs in proportion to the rows it returns.

Try it

Free 20-row sample (HS 8419, August 2026): https://www.kaggle.com/datasets/russellyapp/uk-importers-and-exporters-hs-8419-aug-2026
Example Task (HS 8419 imports, 20 rows): https://apify.com/yappman/uk-importers-exporters/examples/uk-importers-of-heating-and-cooling-machinery
The Actor: https://apify.com/yappman/uk-importers-exporters

I built the Actor and publish the sample. Corrections and questions: info@yappman.com.

After an Actor reboot, the Apify SDK forgets what it has already charged

Russell Yapp — Fri, 23 Oct 2026 15:11:15 +0000

I build pay-per-event Actors for the Apify Store. This week I tested what happens to charging when an Actor reboots in the middle of a run, because Actors can be rebooted or migrated during long runs. I tested a forced reboot; I have not tested a platform migration. The result surprised me, so here it is, with the fix I use.

What I tested

I wrote a small private Actor that charges one event per dataset row, then forces a reboot part-way through with Actor.reboot(). I ran it with a low "maximum cost per run" so the limit would matter, and checked the run metadata afterwards. I used Apify SDK for Python 4.0.2.

What I found

After the reboot the SDK reported zero events charged, even though a few had already been charged before the reboot. As far as I can tell, the SDK takes its charged counts from environment variables set when the container starts, so a restarted container starts from the old numbers instead of the current ones.

In one test, the run metadata ended up showing more charged events than the maximum I had set. Apify's documentation says the platform enforces the limit and that charges above the charged count are not billed, so I can't say a customer would have been billed for the extra events. What I can say is that the run's own record disagreed with the limit I had set, and my Actor had no way to notice. I'd rather not rely on the platform to quietly absorb that.

I haven't seen an existing SDK issue describing this exact behaviour, so it may be worth a report.

The fix I use

Three small pieces, all inside the Actor:

Keep your own record of charging. In my Actors, a CHARGE_STATE record in the run's key-value store is written as "pending" before each charge and updated to "charged" afterwards. After a restart you can read it and know what really happened, even if the SDK's counts say zero.
Dedupe against what you've already stored. Before pushing a row, check whether it is already in the dataset. A restart then can't charge for the same row twice.
Keep your own spend limit. Work out how many events you may still charge as: the user's maximum, minus the larger of your own record of earlier spend and the SDK's total, minus what you've charged since this start. Use Decimal for the money and round down. Never use floats for prices: (0.011 - 0.005) / 0.002 gives 2.9999999999999996 in floats, so rounding down would lose an event.

With those in place my test Actor stopped at exactly the limit I set ($0.011, three rows) in a run that rebooted, and a control run with a higher limit produced no duplicate rows.

How to check your own Actor

Use a throwaway copy, a low per-run cost limit, and a forced Actor.reboot() part-way through. Then read the run metadata and compare the charged counts with the maximum you set, and with the number of distinct rows you expect to have been charged. If they disagree, you have the same gap.

I'm happy to answer questions in the comments.