DEV Community: Harry

Building an Enterprise Football Data Pipeline: Decoding Flashscore's Protocol for xG & Referee Analytics

Harry — Fri, 28 Aug 2026 09:23:09 +0000

Most football data scrapers on the market only extract high-level final scores (e.g. 2-1). But quantitative sports analysts, data scientists, and predictive betting modelers need granular data: Expected Goals (xG), Official Referee Assignments, Goal Scorers paired with Assist Providers, and Half-Time vs Full-Time (1H/2H) statistical breakdowns.

When I set out to build a professional-grade Flashscore scraper on Apify, I ran into two major engineering challenges:

The Memory Problem: Keeping Puppeteer running to scrape hundreds of historical matches consumes over 1.5GB of RAM per run.
The Protocol Problem: Flashscore serves its deep statistical feeds using a proprietary pipe-delimited data format (~, ¬, ÷) over CDN endpoints, rather than standard REST APIs.

In this tutorial, I'll explain how I engineered the Flashscore Elite Statistics Extractor, how the hybrid Browser + HTTP/2 streaming pipeline drops RAM footprint from 1.5GB to 70MB, how to parse Flashscore's custom feed protocol, and how to pipe the resulting datasets directly into Python and Pandas.

🏛️ The Hybrid Pipeline Architecture

To achieve zero proxy reliance for standard runs and ultra-low compute costs, the Actor splits execution into a 2-Phase Hybrid Pipeline:

[ League & Season Selection ]
            │
            ▼
┌───────────────────────────────────────────┐
│ Phase 1: Browser Handshake (Puppeteer)    │
│  - Captures x-fsign security tokens       │
│  - Extracts countryId & tourId            │
└─────────────────────┬─────────────────────┘
                      │
           [ Immediate Browser Shutdown ]
            (RAM drops from 1.2GB -> 70MB)
                      │
                      ▼
┌───────────────────────────────────────────┐
│ Phase 2: Parallel HTTP/2 Feed Workers     │
│  - got-scraping with JA3 TLS matching     │
│  - Decodes df_st_1_ (Stats) & df_sui_1_   │
└─────────────────────┬─────────────────────┘
                      │
                      ▼
┌───────────────────────────────────────────┐
│ Self-Healing Recovery Pass                │
│  - Auto-retries skipped/failed matches    │
└─────────────────────┬─────────────────────┘
                      │
                      ▼
┌───────────────────────────────────────────┐
│ Dataset Output (JSON / Pandas / Excel)    │
└───────────────────────────────────────────┘

Phase 1: Dynamic Handshake & Immediate Browser Teardown

Flashscore uses dynamic security tokens (x-fsign) and tournament identifiers embedded inside obfuscated scripts. Attempting to hardcode or reverse-engineer these tokens statically leads to frequent breakage whenever Flashscore deploys frontend updates.

Instead, the Actor launches a lightweight Puppeteer browser instance with stealth plugins to perform an initial handshake:

// src/main.ts (Handshake & Network Interception)

const onPageCreated = (page: Page) => {
    const listener = (request: HTTPRequest) => {
        const url = request.url();
        // Intercept dynamic background API calls (e.g. tr_1_countryId_tourId)
        const match = url.match(/_1_(\d+)_([A-Za-z0-9]{8})/);
        if (match) {
            countryId = match[1];
            tourId = match[2];
            page.off('request', listener);
            networkResolver(); // Instantly resolve network promise
        }
    };
    page.on('request', listener);
};

const { context, browser, page } = await SessionManager.startDiscovery(discoveryUrl, onPageCreated);

// Perform match discovery...
const matchesMetadata = await DiscoveryManager.discoverMatches(page, input.targetCount, context, tourId, countryId);

// CRITICAL: Close the browser immediately to save compute memory
console.log('[Main] Discovery complete. Closing browser...');
await browser.close();

By closing the browser immediately after discovering the match IDs, we free up over 1GB of RAM before launching the data extraction phase.

Phase 2: Decoding Flashscore's Custom Feed Protocol

Flashscore delivers match statistics via raw text feeds from .ninja CDN servers. Fields are delimited using tilde (~), NOT (¬), and divide (÷) symbols.

For example, a raw statistics feed (df_st_1_{matchId}) looks like this:

SF÷Top stats~SG÷Expected goals (xG)¬SH÷1.42¬SI÷0.88~SG÷Ball possession¬SH÷58%¬SI÷42%~

Building the Feed Parser

In src/parser.ts, we decode these raw delimited rows into strongly typed TypeScript objects and split stats by half-time and full-time:

// src/parser.ts

export class FlashscoreParser {
    private static KEY_MAP: Record<string, string> = {
        'Expected goals (xG)': 'expected_goals',
        'Ball possession': 'ball_possession',
        'Total shots': 'total_shots',
        'Shots on target': 'shots_on_target',
        'Corner kicks': 'corner_kicks',
        'Passes': 'passes',
        'Yellow cards': 'yellow_cards',
        'Fouls': 'fouls',
        'Offsides': 'offsides'
    };

    static parseRow(row: string): Record<string, string> {
        const obj: Record<string, string> = {};
        const cleanRow = row.startsWith('~') ? row.substring(1) : row;
        const fields = cleanRow.split('¬');

        for (const field of fields) {
            if (!field.includes('÷')) continue;
            const [key, value] = field.split('÷');
            if (key) obj[key] = value ?? '';
        }
        return obj;
    }

    static parseStatistics(raw: string): FlatMatchStatistics {
        const records = raw.split('~');
        const stages: any[] = [];
        let currentStage: any = null;

        for (const record of records) {
            const data = this.parseRow(record);
            if (record.startsWith('SF÷') && data['SF'] === 'Top stats') {
                currentStage = {};
                stages.push(currentStage);
                continue;
            }

            if ((record.startsWith('SD÷') || record.startsWith('SG÷')) && currentStage) {
                const rawName = data['SG'] || '';
                const key = this.KEY_MAP[rawName] || rawName.toLowerCase().replace(/\s+/g, '_');

                if (!currentStage[key]) {
                    currentStage[key] = {
                        home: this.sanitizeValue(data['SH']),
                        away: this.sanitizeValue(data['SI'])
                    };
                }
            }
        }

        return {
            fullTime: stages[0] || {},
            firstHalf: stages[1] || {},
            secondHalf: stages[2] || {}
        };
    }
}

Extracting Scorers, Assists, and Referees

The incidents feed (df_sui_1_{matchId}) contains event streams for goals, cards, and match official assignments.

1. Goal Scorers & Assist Providers

We extract goal events and map both the scorer and the assist provider, including their unique Flashscore player URLs and IDs:

{
  "minute": "32'",
  "type": "Goal",
  "team": "home",
  "scorer": {
    "name": "Maldini D.",
    "url": "/player/maldini-daniel/txSCnXec/",
    "id": "txSCnXec"
  },
  "assist": {
    "name": "Bellanova R.",
    "url": "/player/bellanova-raoul/6BgYlqqU/",
    "id": "6BgYlqqU"
  }
}

2. Match Official Referee (`referee`)

We parse the MIT÷REF tag to extract the assigned referee ("referee": "Colombo A."). This allows sports quant models to analyze referee card strictness and foul call frequency across historical seasons.

Self-Healing Recovery Pass for 100% Data Integrity

When running parallel HTTP workers across full season datasets (380+ matches), occasional network glitches or CDN rate limits can cause individual match requests to fail.

To ensure 100% dataset completeness, the Actor implements a Surgical Self-Healing Pass:

// src/main.ts (Self-Healing Recovery)

if (failedMatches.length > 0) {
    console.log(`[Main] 🔄 Self-Healing Final Pass: Retrying ${failedMatches.length} skipped match(es)...`);
    await new Promise(r => setTimeout(r, 3000));

    for (const meta of failedMatches) {
        try {
            const statsPath = `df_st_1_${meta.id}`;
            const rawStats = await Scraper.getRawFeed(statsPath, context);
            const flatStats = FlashscoreParser.parseStatistics(rawStats);
            // ... process and recover match record ...
            await Actor.pushData(fullMatch);
            console.log(`[Main] 🚀 Successfully recovered match: ${meta.homeTeam} vs ${meta.awayTeam}`);
        } catch (recoveryErr: any) {
            console.error(`[Main] Recovery failed for ${meta.id}: ${recoveryErr.message}`);
        }
    }
}

Python & Pandas Integration Example

Once extracted, sports analysts can query the dataset API endpoint directly into Python:

import pandas as pd

# Fetch dataset directly from Apify JSON API endpoint
dataset_url = "https://api.apify.com/v2/key-value-stores/f1pLKNreZAd2R6acn/records/serie-a-2024-25-full.json?disableRedirect=true"
df = pd.read_json(dataset_url)

# Inspect Expected Goals (xG), Referee, and Scores
print(df[['homeTeam', 'awayTeam', 'referee', 'halfTimeScore', 'fullTimeScore', 'homeXg', 'awayXg']].head())

Summary & Resources

By combining a transient Puppeteer handshake with parallel got-scraping HTTP/2 feed workers, you can build production football statistics extractors that run on 70MB RAM while capturing granular xG, referee, scorer/assist, and HT/FT breakdown metrics.

Apify Actor: Flashscore Elite Statistics Extractor
Apify MCP Server: Accessible via https://mcp.apify.com/ for AI Agent tool calls.

Building a 90MB E-Commerce Price Intelligence Tool for AI Agents using Apify MCP Server

Harry — Fri, 28 Aug 2026 09:20:53 +0000

Running e-commerce competitor price monitoring in production usually forces a painful compromise: pay $200+/month for SaaS platforms like Prisync, or host custom Puppeteer/Playwright scrapers that chew through 1.5GB of RAM per instance just to extract basic price tags.

When I started building price tracking workflows, I wanted something faster and vastly cheaper: a lightweight HTTP scraper that runs on 128MB RAM, extracts Schema.org JSON-LD microdata in under 1 second, and connects directly to AI agents via the Apify Model Context Protocol (MCP) Server (https://mcp.apify.com/).

In this tutorial, I'll walk through how I built the Smart E-Commerce Price & Stock Monitor Actor, how it handles tricky real-world edge cases like SPA detection and 24-hour price bounce-back deduplication, and how you can expose it as an executable tool to LLM agents like Claude, Cursor, and n8n.

The Architecture: Why 90MB HTTP Beats 1.5GB Headless Browsers

Most modern direct-to-consumer (D2C) brand stores built on Shopify, WooCommerce, Magento, or Salesforce Commerce Cloud render product metadata on the server using Schema.org JSON-LD microdata.

Spinning up headless Chrome to render a full DOM when the price data already exists inside a <script type="application/ld+json"> tag is massive resource waste.

Here is the high-level architecture of the Actor:

[ E-Commerce Product URLs ]
          │
          ▼
┌─────────────────────────────────────────┐
│ got-scraping + CheerioCrawler (85MB RAM) │
└────────────────────┬────────────────────┘
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
┌──────────────────┐   ┌──────────────────────┐
│ Tier 1: JSON-LD  │   │ Tier 2: CSS Fallback │
│  Extractor       │   │   Selector Engine    │
└─────────┬────────┘   └──────────┬───────────┘
          └──────────┬────────────┘
                     ▼
┌─────────────────────────────────────────┐
│ Stateful Diffing & Bounce-Back Dedup    │
│  (Key-Value Store: STATE.json)          │
└────────────────────┬────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────┐
│ Output & Webhook (Telegram / n8n / MCP) │
└─────────────────────────────────────────┘

Key Engineering Decisions

got-scraping + CheerioCrawler: Replaces heavy browser automation. Uses HTTP/2 fingerprint matching to bypass basic anti-bot TLS fingerprinting while staying under 90MB RAM footprint.
2-Tier Extraction Engine:
- Tier 1 (JSON-LD): Parses @type: "Product" or "Offer" schemas automatically.
- Tier 2 (CSS Fallback): Pre-configured DOM selector engine for stores with incomplete microdata.
SPA Auto-Detection: When a site serves an empty client-side React/Angular skeleton (e.g. Zara), the Actor gracefully flags jsRenderingRequired: true rather than crashing or swallowing missing fields silently.

Connecting the Actor to AI Agents via Apify MCP Server

Apify recently launched the Model Context Protocol (MCP) Server (https://mcp.apify.com/). The Model Context Protocol is an open standard that allows LLMs (like Claude Desktop, Cursor, or n8n AI Agent nodes) to discover and run external tools dynamically.

Because any Actor published on Apify Store is automatically exposed through https://mcp.apify.com/, AI agents can invoke our price monitor as a native function call.

How the Input Schema Maps to MCP Tool Call Parameters

For an LLM to call an Actor reliably via MCP without syntax errors, the Actor's .actor/INPUT_SCHEMA.json must be strictly typed. Here is how our input configuration maps:

{
  "productUrls": [
    "https://skims.com/products/fits-everybody-t-shirt-bra-onyx",
    "https://www.allbirds.com/products/mens-tree-runners-fog"
  ],
  "minPriceChangePercentage": 0,
  "notifyOnFirstRun": true,
  "maxConcurrencyPerDomain": 2
}

Keeping Payload Output Compact (<500 Tokens)

A common mistake when building web scrapers for LLM consumption is returning raw HTML or unformatted JSON objects containing thousands of lines. This quickly floods the AI agent's context window and inflates API cost.

Our Actor returns an ultra-compact, token-efficient payload format:

{
  "url": "https://skims.com/products/fits-everybody-t-shirt-bra-onyx",
  "title": "FITS EVERYBODY T-SHIRT BRA | ONYX | 30 AA",
  "currency": "EUR",
  "currentPrice": 68.00,
  "previousPrice": null,
  "priceChanged": false,
  "inStock": true,
  "isNewProduct": true,
  "shouldAlert": true,
  "jsRenderingRequired": false,
  "checkedAt": "2026-08-11T10:36:46.935Z"
}

This record takes ~120 tokens to parse, allowing an AI agent to analyze dozens of product price alerts in a single prompt without blowing token budgets.

Real-World Production Gotchas & Code Solutions

Building a scraper that works in a local test script is easy; making it survive weeks of continuous production monitoring requires handling real-world edge cases.

Gotcha 1: The 24-Hour Price Bounce-Back Bug

During community testing on n8n, a community member (Cloudrocket) pointed out a subtle edge case in our deduplication logic:

Suppose a product price drops from $100 to $80 at 10 AM. An alert is sent and the hash md5(url|80) is saved for 24 hours. At 2 PM, the price reverts to $100 (alert sent). At 6 PM, another flash sale drops the price back to $80. Because md5(url|80) was saved 8 hours prior, the second drop gets swallowed!

To fix this without allowing threshold spam, I refactored stateStore.js to track price transitions rather than static target prices, and automatically invalidate outdated keys for that product URL upon any state change:

// src/services/stateStore.js

if (shouldAlert) {
    const currency = currentData.currency || 'USD';
    const prevPriceVal = prevItem && prevItem.price !== null ? prevItem.price : 'INIT';

    // Hash key combines product URL hash and transition signature (e.g. 100->80)
    const urlHash = crypto.createHash('md5').update(url).digest('hex').substring(0, 12);
    const transitionSignature = `${prevPriceVal}->${currentPrice}|${currency}|${currentInStock}`;
    const transitionHash = crypto.createHash('md5').update(transitionSignature).digest('hex').substring(0, 12);
    const dedupKey = `${urlHash}:${transitionHash}`;

    const DEDUP_WINDOW_MS = 24 * 60 * 60 * 1000; // 24 hours
    const lastSentAt = this.state.sentAlerts[dedupKey];

    if (lastSentAt && (Date.now() - lastSentAt) < DEDUP_WINDOW_MS) {
        shouldAlert = false;
        logger.debug(`Dedup: skipping duplicate alert for ${url} (key ${dedupKey} sent ${Math.round((Date.now() - lastSentAt) / 3600000)}h ago).`);
    } else {
        // Invalidate stale dedup keys for this product URL to allow legitimate bounce-backs
        const urlPrefix = `${urlHash}:`;
        for (const key of Object.keys(this.state.sentAlerts)) {
            if (key.startsWith(urlPrefix)) {
                delete this.state.sentAlerts[key];
            }
        }
        this.state.sentAlerts[dedupKey] = Date.now();
    }
}

Gotcha 2: Client-Side SPA Handling (React / Next.js)

When static scrapers encounter a client-side rendered Single Page Application like Zara, they typically fail silently or return null price fields.

In src/main.js, we detect this condition and attach actionable guidance to the dataset record:

// src/main.js (SPA Detection)

let jsRenderingRequired = false;
let userGuidance = null;

if (productData.price === null || productData.title === 'Unknown Product') {
    jsRenderingRequired = true;
    userGuidance = '⚠️ Product title or price could not be detected statically. This page may require JavaScript rendering (React/SPA) or custom CSS selectors in Actor input settings.';
    logger.warning(`Unable to extract price or title for ${url}. Page may require JavaScript rendering (SPA React/Next.js) or custom CSS selectors.`);
}

productData.jsRenderingRequired = jsRenderingRequired;
if (userGuidance) {
    productData.userGuidance = userGuidance;
}

Real Test Execution Benchmarks

Running a live benchmark test across 5 real D2C e-commerce stores (Skims, Velasca, Allbirds, Gymshark, and Zara):

INFO  System info {"apifyVersion":"3.7.2","crawleeVersion":"3.17.0","osType":"Windows_NT"}
INFO  Starting Smart E-Commerce Price & Stock Monitor for 5 product URL(s).
INFO  Processing product page: https://www.velasca.com/products/spigolatt-marrone-scuro
INFO  Processing product page: https://skims.com/products/fits-everybody-t-shirt-bra-onyx
INFO  Processing product page: https://www.allbirds.com/products/mens-tree-runners-fog
INFO  Processing product page: https://www.zara.com/it/it/t-shirt-strutturata-p00994300.html
WARN  Unable to extract price or title for https://www.zara.com/it/it/t-shirt-strutturata-p00994300.html. Page may require JavaScript rendering (SPA React/Next.js) or custom CSS selectors.
INFO  Processing product page: https://www.gymshark.com/products/gymshark-vital-seamless-2-0-leggings-black-marl-aw21
INFO  CheerioCrawler: Finished! Total 5 requests: 5 succeeded, 0 failed.

Benchmark Results

Execution Time: 3.64 seconds for 5 stores.
Memory Used: ~85 MB RAM.
Success Rate: 100% (5/5 HTTP requests finished cleanly).

Summary & Resources

By pairing lightweight HTTP static scraping with Schema.org JSON-LD microdata extraction, you can build e-commerce price monitoring systems that cost pennies to run and integrate seamlessly into AI agent workflows via Model Context Protocol.

Links & Code

Apify Actor: Smart E-Commerce Price & Stock Monitor
Apify MCP Server Documentation: mcp.apify.com
n8n Community Workflow Template: Available on the n8n community forum.

DEV Community: Harry

Building an Enterprise Football Data Pipeline: Decoding Flashscore's Protocol for xG & Referee Analytics

🏛️ The Hybrid Pipeline Architecture

Phase 1: Dynamic Handshake & Immediate Browser Teardown

Phase 2: Decoding Flashscore's Custom Feed Protocol

Building the Feed Parser

Extracting Scorers, Assists, and Referees

1. Goal Scorers & Assist Providers

2. Match Official Referee (referee)

Self-Healing Recovery Pass for 100% Data Integrity

Python & Pandas Integration Example

Summary & Resources

Building a 90MB E-Commerce Price Intelligence Tool for AI Agents using Apify MCP Server

The Architecture: Why 90MB HTTP Beats 1.5GB Headless Browsers

Key Engineering Decisions

Connecting the Actor to AI Agents via Apify MCP Server

How the Input Schema Maps to MCP Tool Call Parameters

Keeping Payload Output Compact (<500 Tokens)

Real-World Production Gotchas & Code Solutions

Gotcha 1: The 24-Hour Price Bounce-Back Bug

Gotcha 2: Client-Side SPA Handling (React / Next.js)

Real Test Execution Benchmarks

Benchmark Results

Summary & Resources

Links & Code

2. Match Official Referee (`referee`)