TradingLookup for TradingLookup

Posted on Oct 28

Reading one legally required sentence from 31 broker websites

#python #webscraping #opendata #datascience

If you visit a UK or EU broker that offers contracts for difference (CFDs), you'll find a sentence like this somewhere on the page:

70% of retail investor accounts lose money when trading spread bets and CFDs with this provider.

Regulators require it. Each provider has to work out its own figure and print it as part of a standard risk warning. That makes it one of the few numbers in retail trading that every firm states in the same shape, on its own site, in public.

We built an open dataset of those sentences: 31 providers, read from the UK on 6 October 2026 (7 updated after a second UK read on 13 October). The figures run from 51% to 84%, with a median of 73.46%. This post covers how we read them, what went wrong, and how we pulled eight years of earlier figures out of the Wayback Machine.

A strict match, not "any percentage"

Broker homepages are full of percentages: margin rates, spreads, promotional rates. The first version of our extractor grabbed the first percentage it found near the words "retail investor". It picked up a 5% margin rate that happened to sit just above the warning.

The fix was to anchor the wording: the percentage only counts if the mandated phrase follows it directly.

import re

RATE = re.compile(
    r'(\d{1,2}(?:[.,]\d+)?)\s*(?:–|-)\s*(\d{1,2}(?:[.,]\d+)?)\s?%'  # a range: "74-89%"
    r'|(\d{1,2}(?:[.,]\d+)?)\s?%'                                     # or one figure: "73.6%"
)
VALID = re.compile(
    r'\s*(?:of\s+(?:all\s+)?retail\s+(?:investor|client|account|cfd)'
    r'|der\s+Kleinanlegerkonten)',                                    # German wording, IG Europe
    re.I,
)

def find_rate(text):
    for m in RATE.finditer(text):
        if VALID.match(text, m.end()):          # wording must start right after the %
            return m
    return None

A few details mattered more than expected:

Comma decimals and a space before %. EU pages write "74 %" and "73,5 %".
Ranges. Some groups print a range across their entities. We keep both ends and take the midpoint.
More than one warning on a page. A site serving several group companies can show a UK and an EU figure side by side. We record every distinct match, and a page with more than one is flagged for a check by hand.
The sentence, not just the number. We store the warning text as it was printed, from the percentage to the next full stop. Then anyone can see exactly what was read.

A fingerprint for every read

Each row carries a SHA-256 hash of the page text we read, plus how it was fetched and from where:

import hashlib

text = page_text(html)                   # the text we matched against
fingerprint = hashlib.sha256(text.encode("utf-8")).hexdigest()

The hash doesn't prove the page said what we claim. It does pin down which version of the page we mean, so a later disagreement can be traced to "the page changed" or "we misread it".

An honest second read

On 13 October we ran an independent, automated second read of every source page, from a build environment in the US rather than the UK. Rules for that pass:

An honest user agent that names us and links our method page.
robots.txt is checked first. A 401 or 403 on robots.txt counts as "don't crawl".
No bot-wall bypass. If the page says "verify you are human", we log it as blocked and move on.

The result: 16 of 31 figures matched, 10 pages blocked us, 5 showed no figure, and none disagreed.

The "no figure" cases were instructive. eToro, for example, served our US-based reader its US site, which carries no CFD warning, so the UK page was never reached. A blocked or empty page isn't a contradiction, so we report the three outcomes separately instead of a single "verified" percentage.

Eight years of history from the Wayback Machine

The Internet Archive has captured many of these pages over the years, so we can go back to 2018. The CDX API lists captures for a URL:

cdx = (
    "https://web.archive.org/cdx/search/cdx"
    f"?url={url}&output=json&fl=timestamp,statuscode,original"
    "&filter=statuscode:200&collapse=timestamp:6"   # at most one capture per month
)

From those we pick about one capture per quarter and run the same strict match on the archived text. Two practical notes:

Go slowly. We wait 5 seconds between archive requests. Faster than about 15 a minute and archive.org blocks the IP for a while.
Gaps stay gaps. If there's no capture for a quarter, the series has a hole. We never fill it.

The history file holds 404 quarterly rows for 18 providers, from 2018-Q3 to 2026-Q4, each linked to its capture.

What you can use

Everything is CC BY 4.0, credited to TradingLookup:

The CSV is served with Access-Control-Allow-Origin: *, so you can load it straight into a notebook or a chart in the browser.

The series is provisional: the legal entity behind two of the figures isn't confirmed yet, and regulators' registers haven't been cross-checked. If you spot a figure that doesn't match a provider's current page, the method page says how to report it.

CFDs are complex instruments and come with a high risk of losing money rapidly due to leverage. Each figure is the provider's own statement about its client base, not anyone's personal odds.

DEV Community