DEV Community

Harry
Harry

Posted on

Building an Enterprise Football Data Pipeline: Decoding Flashscore's Protocol for xG & Referee Analytics

#data #webscraping #apify #typescript

Most football data scrapers on the market only extract high-level final scores (e.g. 2-1). But quantitative sports analysts, data scientists, and predictive betting modelers need granular data: Expected Goals (xG), Official Referee Assignments, Goal Scorers paired with Assist Providers, and Half-Time vs Full-Time (1H/2H) statistical breakdowns.

When I set out to build a professional-grade Flashscore scraper on Apify, I ran into two major engineering challenges:

  1. The Memory Problem: Keeping Puppeteer running to scrape hundreds of historical matches consumes over 1.5GB of RAM per run.
  2. The Protocol Problem: Flashscore serves its deep statistical feeds using a proprietary pipe-delimited data format (~, ¬, ÷) over CDN endpoints, rather than standard REST APIs.

In this tutorial, I'll explain how I engineered the Flashscore Elite Statistics Extractor, how the hybrid Browser + HTTP/2 streaming pipeline drops RAM footprint from 1.5GB to 70MB, how to parse Flashscore's custom feed protocol, and how to pipe the resulting datasets directly into Python and Pandas.

🏛️ The Hybrid Pipeline Architecture

To achieve zero proxy reliance for standard runs and ultra-low compute costs, the Actor splits execution into a 2-Phase Hybrid Pipeline:

[ League & Season Selection ]
            │
            ▼
┌───────────────────────────────────────────┐
│ Phase 1: Browser Handshake (Puppeteer)    │
│  - Captures x-fsign security tokens       │
│  - Extracts countryId & tourId            │
└─────────────────────┬─────────────────────┘
                      │
           [ Immediate Browser Shutdown ]
            (RAM drops from 1.2GB -> 70MB)
                      │
                      ▼
┌───────────────────────────────────────────┐
│ Phase 2: Parallel HTTP/2 Feed Workers     │
│  - got-scraping with JA3 TLS matching     │
│  - Decodes df_st_1_ (Stats) & df_sui_1_   │
└─────────────────────┬─────────────────────┘
                      │
                      ▼
┌───────────────────────────────────────────┐
│ Self-Healing Recovery Pass                │
│  - Auto-retries skipped/failed matches    │
└─────────────────────┬─────────────────────┘
                      │
                      ▼
┌───────────────────────────────────────────┐
│ Dataset Output (JSON / Pandas / Excel)    │
└───────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Phase 1: Dynamic Handshake & Immediate Browser Teardown

Flashscore uses dynamic security tokens (x-fsign) and tournament identifiers embedded inside obfuscated scripts. Attempting to hardcode or reverse-engineer these tokens statically leads to frequent breakage whenever Flashscore deploys frontend updates.

Instead, the Actor launches a lightweight Puppeteer browser instance with stealth plugins to perform an initial handshake:

// src/main.ts (Handshake & Network Interception)

const onPageCreated = (page: Page) => {
    const listener = (request: HTTPRequest) => {
        const url = request.url();
        // Intercept dynamic background API calls (e.g. tr_1_countryId_tourId)
        const match = url.match(/_1_(\d+)_([A-Za-z0-9]{8})/);
        if (match) {
            countryId = match[1];
            tourId = match[2];
            page.off('request', listener);
            networkResolver(); // Instantly resolve network promise
        }
    };
    page.on('request', listener);
};

const { context, browser, page } = await SessionManager.startDiscovery(discoveryUrl, onPageCreated);

// Perform match discovery...
const matchesMetadata = await DiscoveryManager.discoverMatches(page, input.targetCount, context, tourId, countryId);

// CRITICAL: Close the browser immediately to save compute memory
console.log('[Main] Discovery complete. Closing browser...');
await browser.close();
Enter fullscreen mode Exit fullscreen mode

By closing the browser immediately after discovering the match IDs, we free up over 1GB of RAM before launching the data extraction phase.

Phase 2: Decoding Flashscore's Custom Feed Protocol

Flashscore delivers match statistics via raw text feeds from .ninja CDN servers. Fields are delimited using tilde (~), NOT (¬), and divide (÷) symbols.

For example, a raw statistics feed (df_st_1_{matchId}) looks like this:

SF÷Top stats~SG÷Expected goals (xG)¬SH÷1.42¬SI÷0.88~SG÷Ball possession¬SH÷58%¬SI÷42%~
Enter fullscreen mode Exit fullscreen mode

Building the Feed Parser

In src/parser.ts, we decode these raw delimited rows into strongly typed TypeScript objects and split stats by half-time and full-time:

// src/parser.ts

export class FlashscoreParser {
    private static KEY_MAP: Record<string, string> = {
        'Expected goals (xG)': 'expected_goals',
        'Ball possession': 'ball_possession',
        'Total shots': 'total_shots',
        'Shots on target': 'shots_on_target',
        'Corner kicks': 'corner_kicks',
        'Passes': 'passes',
        'Yellow cards': 'yellow_cards',
        'Fouls': 'fouls',
        'Offsides': 'offsides'
    };

    static parseRow(row: string): Record<string, string> {
        const obj: Record<string, string> = {};
        const cleanRow = row.startsWith('~') ? row.substring(1) : row;
        const fields = cleanRow.split('¬');

        for (const field of fields) {
            if (!field.includes('÷')) continue;
            const [key, value] = field.split('÷');
            if (key) obj[key] = value ?? '';
        }
        return obj;
    }

    static parseStatistics(raw: string): FlatMatchStatistics {
        const records = raw.split('~');
        const stages: any[] = [];
        let currentStage: any = null;

        for (const record of records) {
            const data = this.parseRow(record);
            if (record.startsWith('SF÷') && data['SF'] === 'Top stats') {
                currentStage = {};
                stages.push(currentStage);
                continue;
            }

            if ((record.startsWith('SD÷') || record.startsWith('SG÷')) && currentStage) {
                const rawName = data['SG'] || '';
                const key = this.KEY_MAP[rawName] || rawName.toLowerCase().replace(/\s+/g, '_');

                if (!currentStage[key]) {
                    currentStage[key] = {
                        home: this.sanitizeValue(data['SH']),
                        away: this.sanitizeValue(data['SI'])
                    };
                }
            }
        }

        return {
            fullTime: stages[0] || {},
            firstHalf: stages[1] || {},
            secondHalf: stages[2] || {}
        };
    }
}
Enter fullscreen mode Exit fullscreen mode

Extracting Scorers, Assists, and Referees

The incidents feed (df_sui_1_{matchId}) contains event streams for goals, cards, and match official assignments.

1. Goal Scorers & Assist Providers

We extract goal events and map both the scorer and the assist provider, including their unique Flashscore player URLs and IDs:

{
  "minute": "32'",
  "type": "Goal",
  "team": "home",
  "scorer": {
    "name": "Maldini D.",
    "url": "/player/maldini-daniel/txSCnXec/",
    "id": "txSCnXec"
  },
  "assist": {
    "name": "Bellanova R.",
    "url": "/player/bellanova-raoul/6BgYlqqU/",
    "id": "6BgYlqqU"
  }
}
Enter fullscreen mode Exit fullscreen mode

2. Match Official Referee (referee)

We parse the MIT÷REF tag to extract the assigned referee ("referee": "Colombo A."). This allows sports quant models to analyze referee card strictness and foul call frequency across historical seasons.

Self-Healing Recovery Pass for 100% Data Integrity

When running parallel HTTP workers across full season datasets (380+ matches), occasional network glitches or CDN rate limits can cause individual match requests to fail.

To ensure 100% dataset completeness, the Actor implements a Surgical Self-Healing Pass:

// src/main.ts (Self-Healing Recovery)

if (failedMatches.length > 0) {
    console.log(`[Main] 🔄 Self-Healing Final Pass: Retrying ${failedMatches.length} skipped match(es)...`);
    await new Promise(r => setTimeout(r, 3000));

    for (const meta of failedMatches) {
        try {
            const statsPath = `df_st_1_${meta.id}`;
            const rawStats = await Scraper.getRawFeed(statsPath, context);
            const flatStats = FlashscoreParser.parseStatistics(rawStats);
            // ... process and recover match record ...
            await Actor.pushData(fullMatch);
            console.log(`[Main] 🚀 Successfully recovered match: ${meta.homeTeam} vs ${meta.awayTeam}`);
        } catch (recoveryErr: any) {
            console.error(`[Main] Recovery failed for ${meta.id}: ${recoveryErr.message}`);
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Python & Pandas Integration Example

Once extracted, sports analysts can query the dataset API endpoint directly into Python:

import pandas as pd

# Fetch dataset directly from Apify JSON API endpoint
dataset_url = "https://api.apify.com/v2/key-value-stores/f1pLKNreZAd2R6acn/records/serie-a-2024-25-full.json?disableRedirect=true"
df = pd.read_json(dataset_url)

# Inspect Expected Goals (xG), Referee, and Scores
print(df[['homeTeam', 'awayTeam', 'referee', 'halfTimeScore', 'fullTimeScore', 'homeXg', 'awayXg']].head())
Enter fullscreen mode Exit fullscreen mode

Summary & Resources

By combining a transient Puppeteer handshake with parallel got-scraping HTTP/2 feed workers, you can build production football statistics extractors that run on 70MB RAM while capturing granular xG, referee, scorer/assist, and HT/FT breakdown metrics.

Top comments (0)