DEV Community

0xGollum
0xGollum

Posted on

I scraped live betting odds from 9 countries and never once hit a real anti-bot wall

#webscraping #python #api #buildinpublic

I just shipped a horse-racing odds tracker across 9 countries. Four totally different site architectures, zero paid proxies, zero headless browsers, zero CAPTCHAs solved. Every "protected" data source I hit turned out to be a public API hiding in plain sight. Here's what actually worked.

1. Curl before you Playwright

The instinct when you see a modern-looking racing site is to reach for a headless browser. Don't. First move is always a plain curl with a real User-Agent and a look at how much of the page is actual text vs markup:

import re
text = re.sub('<[^>]+>', ' ', html)
print(len(text), '/', len(html))
Enter fullscreen mode Exit fullscreen mode

A high ratio usually means server-side rendering — the data's already in the HTML, a JS engine was never going to be necessary. This alone ruled out headless browsers for 3 of the 4 sources I ended up using.

2. Official public APIs are more common than you'd think

One source runs on a pari-mutuel model, and it turns out the operator ships a fully public, key-less JSON API for its own web/app clients — no login, no proxy, nothing. If a market has a state-run or monopoly tote operator, check whether their own site calls a JSON backend before assuming you need to scrape HTML at all.

3. Read the NEXT_DATA / JSON-LD blob before you write a single CSS selector

Two of my four sources are Next.js apps. Both ship the entire page's data as a JSON blob in a <script> tag — __NEXT_DATA__ on one, schema.org SportsEvent JSON-LD on the other. No DOM parsing needed for the core fields; just find the tag and json.loads() it.

Gotcha: if the JSON has nested objects, don't reach for a regex like \{.*?\} to extract it — non-greedy matching stops at the first inner }, not the real end. I burned twenty minutes on this exact bug before switching to a proper brace-counting extractor:

def extract_json_object(text, start):
    depth = 0
    for i in range(start, len(text)):
        if text[i] == '{':
            depth += 1
        elif text[i] == '}':
            depth -= 1
            if depth == 0:
                return text[start:i + 1]
Enter fullscreen mode Exit fullscreen mode

4. Client-side config files sometimes ship real API keys

The most surprising find: one source is a GraphQL app, and the frontend's own JS config file contained a working AWS AppSync API key — meant for the public site to call directly. Introspection was even enabled, so I could enumerate the whole schema and find the exact query I needed (turned out to be getRaceForm, not the more obvious-looking getOdds, which silently returned nothing for reasons I never fully nailed down).

Lesson: view-source a site's own JS bundles before assuming an API is closed. If the frontend can call it with a static key, so can you.

5. Country/region filters on aggregator sites can lie to you

One aggregator site shows "next races" for a country page even when nothing's actually happening in that country right now — it silently falls back to showing whatever's soonest site-wide. First pass of my scraper mislabeled tracks from one country as belonging to another because of this. Fix was trivial once caught (cross-check against the actual structured-data field for country) but the failure mode was quiet — no error, just wrong data. Worth an explicit assertion for any "listing page defaults to something unexpected" pattern.

Built as a new Apify actor, World Turf Pulse — tracks steamer/drifter odds movement across nine countries. Happy to answer questions on any of the four scraping approaches above.

Top comments (0)