DEV Community: Clear Fetch

Tennis, football and basketball results as clean JSON, without a sports data contract

Clear Fetch — Sat, 05 Sep 2026 19:10:43 +0000

Sports data is one of those areas where the gap between "public information" and "affordable to obtain" is
absurd. Scores and results are facts, printed in every newspaper. Getting them as structured data usually means
either a five-figure contract with a rights holder, or writing your own scraper and maintaining it forever.

If you are building a model, a scoreboard or a fantasy tool rather than a betting exchange, there is a middle
path. This post is about the details that separate usable sports data from a spreadsheet you cannot trust.

The status field is where most sports scrapers get it wrong

Every match has a state, and naive scrapers reduce it to three: upcoming, live, finished. Real fixtures are
messier. A tennis match can be retired mid-set. A football match can be abandoned at half time, replayed, awarded
to one side, or decided on penalties. Each of those needs a different treatment in any model you build.

I learned this the hard way. My tennis scraper inferred status codes from observed data, and the mapping looked
convincing: code 5 seemed to be a walkover, codes 17 and 18 seemed to be postponed and cancelled, because that
was consistent with the handful of matches I had checked.

Then I found the authoritative list. Codes 17 and 18 are FIRST_SET and SECOND_SET. They mean the match is
being played right now. My scraper was reporting live matches as cancelled. On one day's data, seven matches
were mislabelled, including one sitting at 6-4, 0-0 with a set already on the board.

The lesson generalises beyond this one feed: when you infer an enum from samples, you are fitting to whatever
happened to be in your sample. A match in its first set and a cancelled match both look like "no result yet" if
you squint. Find the real mapping, or at minimum, add an assertion that catches the contradiction. Mine now
refuses to label anything as postponed or cancelled if there are games on the board.

Scores are not a single number

The second thing that bites you is assuming a score is two integers.

Tennis needs set scores and tiebreak points. A set that ends 7-6 is meaningless without the 7-4 in the tiebreak, and a match at 6-6 in the first set has a live tiebreak count that looks exactly like a games score if you map it carelessly.
Basketball needs quarters and any overtime periods. The useful assertion here is that they must add up: 20 + 27 + 21 + 21 = 89 for the home team, or your mapping is wrong. That single check is what proved my field mapping rather than my confidence in it.
Football needs halves, extra time and shootouts, and there is a trap. A match that finished 0-0 and was won 4-3 on penalties is commonly displayed as 1-0, because the display score absorbs the shootout point. If you derive the first-half score by subtracting the second half from the total, you will be wrong by exactly one goal on every shootout in your dataset. Better to leave that field null than to publish a confident wrong number.

Getting it

I published one Actor per sport, because a tennis-specific tool is more useful than a generic one and easier to
find:

Tennis: ATP, WTA, ITF and Challenger, with set and tiebreak scores, match statistics split by set, and full point-by-point sequences.
Football: every league, with goals and assists, cards and the offence behind them, substitutions, and statistics including expected goals.
Basketball: quarter and overtime scores plus the full statistics sheet.

All three cost $0.001 per match, so a full day of world football, roughly 1,700 matches, is under two dollars.

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("clearfetch/flashscore-football-scraper").call(
    run_input={"days": ["0"], "countries": ["England"], "status": ["finished"]}
)

for m in client.dataset(run["defaultDatasetId"]).iterate_items():
    ht = m["score"]["firstHalf"]
    print(f'{m["home"]["name"]} {m["score"]["home"]}-{m["score"]["away"]} {m["away"]["name"]}'
          f' (HT {ht["home"]}-{ht["away"]})' if ht else '')

A real row, from a Premier League match:

{
  "status": "finished",
  "stageDetail": "FINISHED",
  "home": { "name": "Newcastle" },
  "away": { "name": "Bournemouth" },
  "score": { "home": 2, "away": 2, "firstHalf": { "home": 1, "away": 2 } },
  "winner": "draw"
}

Ask for the details of that match and you get the 19 incidents behind the scoreline: who scored, who assisted,
which card was for what offence, and the substitutions, plus 120 statistic rows including expected goals.

Two things worth knowing

Draws exist. Obvious for football, but a surprising number of scrapers model winner as home or away and
quietly drop the third case. Mine returns draw, and 47 of the 1,759 matches on the day I tested were draws.

Everything unmapped is still there. Each row carries a raw object with every original field from the
source. If you need something I did not map, you do not have to wait for me to add it.

Scores and results are facts, and reading a public page is not the same as redistributing a rights-holder's
feed. What you do with the data afterwards, particularly if you republish it, is your responsibility to check.

How to find every Shopify store in a list of 10,000 domains

Clear Fetch — Sat, 05 Sep 2026 19:10:42 +0000

You have a list of domains. A conference attendee export, a directory scrape, a spreadsheet from a colleague.
You want to know which of them run Shopify, which run WordPress, which have HubSpot installed, and which are on
Cloudflare. That single question sits behind a lot of sales prospecting, competitor tracking and agency audits.

The mechanics are not complicated, and it is worth understanding them before you decide whether to build or buy.

What a website tells you without being asked

Every page you request hands over more than the HTML you asked for:

Response headers. Server: cloudflare and a cf-ray header are Cloudflare. X-Powered-By: PHP/8.2 is self-explanatory. Shopify sends x-shopid.
Cookies. _shopify_y, __utma, wp-settings-* each pin a platform.
HTML and meta tags. <meta name="generator" content="WordPress 6.9.7"> is about as direct as it gets.
Script sources. cdn.shopify.com, googletagmanager.com/gtm.js, js.hs-scripts.com.
DNS records. MX records name the mail provider. TXT records carry verification tokens for every SaaS tool the company has ever set up, which is a surprisingly complete picture of their stack.

Wappalyzer popularised a fingerprint format for exactly this, and the open
webappanalyzer project maintains those fingerprints publicly: at the
time of writing, 7,613 technologies with the patterns that identify each one.

The two things that make a naive implementation wrong

If you write this yourself over a weekend, you will hit both of these.

Technologies are injected at runtime. Take Cloudflare's own marketing site. It uses Google Tag Manager, but
there is no <script src="...gtm.js"> anywhere in the HTML. Instead an inline script does this:

const gtmContainerSrc = "https://www.googletagmanager.com/gtm.js?id=GTM-NDGPDFZ";
// ...
var s = document.createElement('script');
s.src = gtmContainerSrc;
document.head.appendChild(s);

A fingerprint matcher that only looks at <script src> attributes sees nothing. The same applies to headless
storefronts: Gymshark is a Shopify store, but its front end is Next.js and the only Shopify URLs on the page are
in an og:image tag and a srcset attribute.

The fix is to treat vendor URLs found anywhere in the document as evidence, at reduced confidence, rather than
only trusting script tags.

Fingerprint patterns carry metadata you have to parse correctly. Patterns look like this:

(?:cdn\.|\.my)shopify\.com\;confidence:50

That trailing \;confidence:50 is a tag, not part of the regex. Split on the wrong thing and the regex keeps a
trailing backslash, fails to compile, and gets silently dropped. In the database as it stands, 223 patterns carry
a confidence tag and 1,392 carry a version tag, so getting this wrong costs you every version number and a good
chunk of your detections. It is exactly the kind of bug that produces plausible-looking output that is quietly
missing a third of the answer.

Doing it without writing it

I published this as an Apify Actor so you can run it on a list without maintaining any of the above:
Tech Stack Detector. It costs $0.02 per website and
unreachable sites are free.

curl -X POST "https://api.apify.com/v2/acts/clearfetch~tech-stack-detector/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["gymshark.com", "techcrunch.com", "notion.so"]}'

Every detection comes back with the evidence behind it, so you can audit rather than trust:

{
  "name": "Cloudflare",
  "confidence": 100,
  "categories": [{ "id": 31, "name": "CDN" }],
  "evidence": [
    { "type": "headers", "key": "server", "pattern": "^cloudflare$", "match": "cloudflare" },
    { "type": "dns", "key": "ns", "pattern": "\\.cloudflare\\.com", "match": "hera.ns.cloudflare.com" }
  ]
}

Filtering a thousand domains down to "everything running Shopify" is then a one-liner:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("clearfetch/tech-stack-detector").call(run_input={"urls": domains})

for site in client.dataset(run["defaultDatasetId"]).iterate_items():
    if "Shopify" in site.get("techNames", []):
        print(site["url"], site["company"] if "company" in site else "")

Where this approach stops working

Be clear about the limits before you rely on it:

No JavaScript execution. Technologies that leave no trace in the HTML, headers, cookies or DNS, and are only visible as a window global after the page boots, will be missed unless another fingerprint implies them. Running a real browser would catch those, at roughly fifty times the cost per page.
One page per site. Something that only loads on a checkout or pricing page will not show up if you only scan the homepage. Pass deeper URLs when it matters.
Confidence is not certainty. A vendor URL inside an inline script is good evidence, not proof. Raise the minimum confidence if you would rather have fewer, harder detections.

Knowing those boundaries is the difference between a list you can act on and a list you have to re-check.