DEV Community: Goodness E. Eboh

How we scraped the 2026 FIFA World Cup data and what shocked us

Goodness E. Eboh — Tue, 08 Sep 2026 14:46:14 +0000

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/scrape-2026-fifa-world-cup-data-python

We built a Python pipeline over three sources (Sofascore, FIFA, and Wikipedia) to pull the full 2026 World Cup dataset. Each source delivered its data differently. Each needed a different extraction strategy. This is how we handled all three, and what the data revealed when we had it.

Full source on GitHub.

Before you start

Python 3.10 or later
Zenrows account for your API key
playwright, requests, beautifulsoup4, pandas, python-dotenv

pip install requests playwright beautifulsoup4 pandas python-dotenv

Three sources, three access patterns

No single source had everything. FIFA had team-level metrics. Sofascore had player-level statistics. Wikipedia had tournament metadata.

The access patterns were all different across the three sources.

Sofascore delivered data through a JavaScript-rendered JSON endpoint
FIFA delivered it through a backend API behind a per-session JWT
Wikipedia delivered server-rendered HTML

Standard requests handles the last one. It fails silently on the first two, returning an empty shell or a challenge page instead of data.

Retrieving Sofascore data from a JavaScript endpoint

Sofascore is a single-page application. The front end calls an internal API and JavaScript renders the result into the browser. requests alone returns the application shell with no data.

We used Zenrows Fetch with mode=auto. Because the endpoint returns JSON, there was no HTML parsing step. The response was already structured data.

import os
import time
from urllib.parse import urlencode

import requests

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

class SofascoreClient:
    BASE_URL = "https://www.sofascore.com/api/v1"

    def _get(self, endpoint: str, params: dict | None = None):
        base_url = f"{self.BASE_URL}/{endpoint.lstrip('/')}"
        target_url = f"{base_url}?{urlencode(params)}" if params else base_url

        retries = 3
        for attempt in range(retries):
            try:
                response = requests.get(
                    "https://api.zenrows.com/v1/",
                    params={
                        "apikey": ZENROWS_API_KEY,
                        "url": target_url,
                        "mode": "auto",  # handles JS rendering and proxy escalation per request
                    },
                    timeout=120,
                )
                response.raise_for_status()
                return response.json()
            except (
                requests.exceptions.ReadTimeout,
                requests.exceptions.ConnectionError,
                requests.exceptions.HTTPError,
            ) as e:
                print(f"Attempt {attempt + 1}/{retries} failed: {e}")
                if attempt == retries - 1:
                    raise
                wait = 2 ** attempt
                print(f"Retrying in {wait}s...")
                time.sleep(wait)

    def fetch_statistics(self, group, order, page=1, limit=20, accumulation="total"):
        endpoint = f"unique-tournament/{TOURNAMENT_ID}/season/{SEASON_ID}/statistics"
        params = {
            "group": group,
            "order": order,
            "page": page,
            "limit": limit,
            "accumulation": accumulation,
        }
        return self._get(endpoint, params=params)

Retrieving FIFA data from an authenticated API

FIFA's data came from a backend API called with a JWT bearer token issued for the current browsing session. Every visitor gets a temporary token; the front end attaches it to subsequent API calls. The endpoint couldn't be queried as a public URL.

We used Zenrows Browser Sessions with Playwright to follow the same authenticated flow a normal browser follows and intercepted the API responses.

import os
from playwright.async_api import async_playwright

class FIFAClient:
    def __init__(self):
        self.api_key = os.getenv("ZENROWS_API_KEY")
        self.connection_url = f"wss://browser.zenrows.com?apikey={self.api_key}"
        self.responses = {}

    async def _handle_response(self, response):
        if "gameday-prod.fifa.mangodev.co.uk" not in response.url:
            return
        self.responses[response.url] = await response.json()

    async def fetch_all(self):
        async with async_playwright() as p:
            browser = await p.chromium.connect_over_cdp(self.connection_url)
            context = browser.contexts[0]
            page = await context.new_page()

            page.on("response", self._handle_response)  # intercept all API responses
            await page.goto(self.URL)
            await self._remove_cookie_banner(page)

            for tab in self.TABS:
                await self._click_tab(page, tab)

            self._save_files()

Retrieving Wikipedia tournament metadata

Wikipedia is server-rendered. Fetch retrieved it with no extra capabilities. BeautifulSoup located the stadium summary table by its headers.

from bs4 import BeautifulSoup
import re

def normalize(self):
    with open(self.INPUT_FILE, encoding="utf-8") as f:
        soup = BeautifulSoup(f, "html.parser")

    target_table = None

    for table in soup.find_all("table", class_="wikitable"):
        headers = [th.get_text(" ", strip=True) for th in table.find_all("th")]
        if "Year" in headers and "Host" in headers and "Stadiums" in headers:
            target_table = table
            break

    records = []
    for row in target_table.find("tbody").find_all("tr")[1:]:
        cells = row.find_all(["th", "td"])
        if len(cells) < 4:
            continue
        year_match = re.search(r"\d{4}", cells[0].get_text())
        if not year_match:
            continue
        records.append({
            "year": int(year_match.group()),
            "hosts": [t.strip() for t in cells[1].stripped_strings if t.strip()],
            "cities": int(re.search(r"\d+", cells[2].get_text()).group()),
            "stadiums": int(re.search(r"\d+", cells[3].get_text()).group()),
        })

Normalizing three sources into one format

Three sources fetched three ways produce three shapes of data. Each had its own normalization layer. Sofascore returned duplicate players across statistic groups, so we keyed on player IDs to deduplicate.

def normalize_player(self, record):
    player = record.get("player", {})
    team = record.get("team", {})
    stats = {k: v for k, v in record.items() if k not in {"player", "team", "rating"}}
    return {
        "player_id": player.get("id"),
        "player": player.get("name"),
        "team": team.get("name"),
        "rating": record.get("rating"),
        "stats": stats,
    }

def normalize_dataset(self, dataset):
    with open(f"data/raw/sofascore/{dataset}.json") as f:
        data = json.load(f)

    players = {}
    for record in data:
        normalized = self.normalize_player(record)
        if normalized["player_id"]:
            players[normalized["player_id"]] = normalized  # keep one record per player

    return list(players.values())

What the data revealed

Ten findings from the dataset that challenged what we expected.

Dembélé scored six goals from just over two expected goals, close to three times his xG
The Golden Glove winner (Unai Simón) didn't appear in our top ten goalkeepers by rating
Bellingham, Rodri, and Tchouaméni dominated the defensive action rankings ahead of centre-backs
Japan recorded the highest shot conversion rate while taking the lowest share of shots from inside the box
Brian Brobbey recorded a 75% conversion rate without missing a single big chance
Michael Olise set a new World Cup record with seven assists and led the tournament in big chances created
More pressing volume didn't produce quicker ball recovery. Türkiye applied fewer pressures and recovered faster than France, England, and Argentina
Argentina reached the final with the most bookings and no VAR interventions against them
Spain led the tournament in defensive line breaks while holding around 90% passing accuracy
The 2026 three-country World Cup used 16 stadiums, fewer than Spain used alone in 1982

The full findings with charts are in the original article.

What's next

Zenrows Batch for large-scale collection across thousands of URLs
Zenrows Browser Sessions for any authenticated scraping workflow
Web data for LLM fine-tuning for feeding scraped data into training pipelines

How to Build a RAG Pipeline with Live Web Data Using ZenRows and LlamaIndex

Goodness E. Eboh — Mon, 13 Jul 2026 12:42:04 +0000

How do you make sure the documents entering your VectorStoreIndex contain actual content when building a LlamaIndex RAG (retrieval augmented generation) pipeline? This question comes up because the built-in SimpleWebPageReader struggles with many of the dynamic web pages it encounters.

Instead of valid page content, the reader can return Cloudflare challenge pages, empty React shells, or content with a poor signal-to-noise ratio relative to the tokens spent. This tutorial directly addresses this issue. It shows how replacing the built-in web reader with one that returns clean LlamaIndex documents from live URLs improves the quality of data available to agents, query engines, and vector indexes in RAG systems.

Why LlamaIndex web scraping loaders fail on real websites

There are three major failure modes that LlamaIndex's built-in web loaders show against modern web infrastructure.

The first is the Cloudflare/WAF (Web Application Firewall) block. When the SimpleWebPageReader makes a standard HTTP request using its underlying libraries like requests or urllib, it lacks capabilities such as browser fingerprints, TLS fingerprints, and human-like behavioral patterns. Security layers on websites protected by WAFs use these signals to identify automated requests and cause the reader to receive HTTP error pages or challenge screens instead.

The second failure mode is SPA (Single Page Application) empty shells. Many modern websites use frameworks like React, Angular, or Vue. When they are scraped, the initial HTML response is usually just an empty template shell containing an initialization script block. Since the reader does not execute JavaScript, it only sees the initial HTML shell.

The third is token bloat. Even for modern unprotected static sites, built-in readers pull raw HTML or raw text, which usually include navigation bars, headers, footers, sidebars, cookie banners, and tracking scripts. This unnecessary data, when indexed, costs tokens that would have been better spent pulling in more relevant content.

Using the SimpleWebPageReader to load data from an anti-bot challenge page shows exactly how output failures can occur.

from llama_index.readers.web import SimpleWebPageReader
documents = SimpleWebPageReader(html_to_text=True).load_data(
    ["https://www.scrapingcourse.com/cloudflare-challenge"]
)

print(documents[0].text[:200])

The reader was only able to pull the challenge page.

If a document like this is added to the VectorStoreIndex, LLM (Large Language Model) outputs would have no information related to the content that was meant to be pulled. At the same time, tokens are still consumed during both retrieval and generation. To avoid these failure modes, use a reader designed to access protected websites, execute JavaScript, and return cleaner content for indexing. The ZenRowsWebReader is one such reader.

Setting up `ZenRowsWebReader`

The ZenRowsWebReader is supported in LlamaIndex and can return valid Document objects from live pages that are ingestible by AI applications. It can access websites protected by WAF challenges, render JavaScript, and output markdown, all of which improve document quality. Here is how you can set it up.

To follow along, first generate a ZenRows API key from the ZenRows Dashboard and store it as the ZENROWS_API_KEY environment variable.

Next, import the reader and configure it with the following parameters to handle live web pages.

When reading sites protected by advanced anti-bot mechanisms like Cloudflare, enable the premium_proxy parameter to use residential IPs. Residential IPs are less likely to be identified as automated traffic by anti-bot systems.

from llama_index.readers.web import ZenRowsWebReader

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

Lastly, use the ZenRowsWebReader against the same website that the SimpleWebPageReader failed to scrape.

from llama_index.readers.web import ZenRowsWebReader
from dotenv import load_dotenv
import os

load_dotenv()

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

documents = reader.load_data([
    "https://www.scrapingcourse.com/cloudflare-challenge"
])

print(documents[0].text[:500])

Running it, the output shows that the reader successfully accessed the website, scraped its content, and returned it in Markdown.

Building a RAG pipeline with live web data

Having resolved the data quality issue with scraping web content from live pages using ZenRowsWebReader, the next step is to include the reader in a RAG pipeline. While the complete code is a single file, the implementation consists of four stages: configuring the pipeline to use OpenAI models, ingesting live web data with the reader, creating the vector index, and querying it to generate responses.

Step 1: Configure the OpenAI model and embedding model

Import all dependencies, load the API keys, and configure the OpenAI model and embedding model.

from dotenv import load_dotenv
import os

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

Settings.llm = OpenAI(
    model="gpt-4o-mini",
    api_key=OPENAI_API_KEY
)

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_key=OPENAI_API_KEY
)

This configuration uses gpt-4o-mini for fast, cost-effective response generation and text-embedding-3-small to create semantic embeddings that provide a good balance of retrieval quality, latency, and cost. Both models can be replaced with other supported OpenAI models depending on your specific requirements.

Step 2: Load live web data into LlamaIndex documents

Just like in the previous section, import the ZenRowsWebReader and configure it to load data from the target URLs into LlamaIndex Document objects.

from llama_index.readers.web import ZenRowsWebReader

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

documents = reader.load_data([
    "https://www.scrapingcourse.com/cloudflare-challenge",
    "https://www.scrapingcourse.com/javascript-rendering"
])

print(f"Loaded {len(documents)} document(s)")

Step 3: Create the vector index

Import VectorStoreIndex and pass the Document objects into it to create the index.

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents)

print("Index created successfully")

This example creates an in-memory VectorStoreIndex for simplicity. LlamaIndex also supports persisting indexes to disk or external vector stores.

Step 4: Query the index

Use the query engine to retrieve relevant content from the index and pass it to the OpenAI model for response generation.

query_engine = index.as_query_engine()

response = query_engine.query(
    "Summarize the information available across the indexed pages."
)

print("\nAnswer:\n")
print(response.response)

The result is that the LLM retrieved valid indexed data from the documents and generated output that matched the scraped content.

While this pipeline delivers valid data, websites frequently change layouts and content. So it’s important to have a layer that updates the index with fresh Document objects.

Keeping your knowledge base current

One-off loads deliver valid content, but they can quickly become outdated. Periodic refreshes allow current web content to be re-scraped from live pages as source content changes. But if the logic isn't right, it can cause the same content to be re-ingested repeatedly, increasing storage costs and reducing retrieval quality through duplicate content.

You can achieve efficient refreshes by generating hashes from the scraped page content and comparing them against previously stored values. This logic runs after loading documents with the ZenRowsWebReader and before passing the updated documents to the VectorStoreIndex.from_documents().

In addition to the existing reader setup, this implementation uses the json and hashlib modules to persist and compare content hashes.

import json
import hashlib

To set up the logic, create a hash store. In production, this is usually a database rather than a local file. Then create a set of functions that store hashes of scraped documents and compare them against newly scraped content.

HASH_STORE = "content_hashes.json"

def load_hashes():
    if os.path.exists(HASH_STORE):
        with open(HASH_STORE, "r") as f:
            return json.load(f)
    return {}

def save_hashes(hashes):
    with open(HASH_STORE, "w") as f:
        json.dump(hashes, f, indent=4)

def get_updated_documents(urls):
    documents = reader.load_data(urls)

    if not documents:
        print("No documents returned")
        return []

    stored_hashes = load_hashes()
    updated_documents = []

    for doc in documents:
        if not doc.text.strip():
            source_url = doc.metadata.get("source_url", "unknown")
            print(f"Skipping empty content: {source_url}")
            continue

        source_url = doc.metadata.get("source_url", "unknown")

        content_hash = hashlib.md5(
            doc.text.encode("utf-8")
        ).hexdigest()

        previous_hash = stored_hashes.get(source_url)

        if previous_hash == content_hash:
            print(f"No changes detected: {source_url}")
            continue

        print(f"Updated content detected: {source_url}")

        stored_hashes[source_url] = content_hash
        updated_documents.append(doc)

    save_hashes(stored_hashes)
    return updated_documents

documents = get_updated_documents([
    "https://www.scrapingcourse.com/cloudflare-challenge",
    "https://www.scrapingcourse.com/javascript-rendering"
])

During each refresh cycle, the function compares the latest scraped content against the previously stored hashes. Unchanged pages are skipped, while pages with updated content are returned for re-indexing.

At this point, the pipeline is fully functional. But even with clean web data and a working retrieval pipeline, you can still encounter issues with answer quality when querying.

Troubleshooting incomplete answers

Incomplete answers can result from retrieval configuration in LlamaIndex or from incomplete page content returned by the web reader. Here is how to troubleshoot both.

Insufficient retrieval context

During retrieval, LlamaIndex returns only a subset of the highest-scoring chunks from the vector index to provide relevant context for the LLM. By default, this naive top-k retrieval matches query embeddings with chunk embeddings. Since live webpages are broken down into multiple markdown chunks, important context can end up distributed across multiple document chunks.

The fix

Increase the top-k parameter to 10 so the query engine pulls more relevant chunks into the retrieval context, improving the likelihood of retrieving related information.

query_engine = index.as_query_engine(
    similarity_top_k=10
)

Missing dynamic JavaScript content

If your final output indicates missing information or returns placeholder strings such as "Enable JavaScript to see products," the scraper may be capturing the page before JavaScript-rendered elements have finished loading.

The fix

In the ZenRows reader parameters, verify that js_render=True is enabled and that a wait=2000 configuration has been added. These settings allow dynamic elements time to load so the target elements can fully render before the page is converted into a LlamaIndex document.

Conclusion

You have now built a RAG pipeline that uses the ZenRowsWebReader to ingest live web data into LlamaIndex. It can retrieve content from protected and JavaScript-driven websites, keep indexed content up to date through periodic refreshes, and generate responses from documents that standard web loaders often fail to retrieve.

DEV Community: Goodness E. Eboh

How we scraped the 2026 FIFA World Cup data and what shocked us

Before you start

Three sources, three access patterns

Retrieving Sofascore data from a JavaScript endpoint

Retrieving FIFA data from an authenticated API

Retrieving Wikipedia tournament metadata

Normalizing three sources into one format

What the data revealed

What's next

How to Build a RAG Pipeline with Live Web Data Using ZenRows and LlamaIndex

Why LlamaIndex web scraping loaders fail on real websites

Setting up ZenRowsWebReader

Building a RAG pipeline with live web data

Step 1: Configure the OpenAI model and embedding model

Step 2: Load live web data into LlamaIndex documents

Step 3: Create the vector index

Step 4: Query the index

Keeping your knowledge base current

Troubleshooting incomplete answers

Insufficient retrieval context

The fix

Missing dynamic JavaScript content

The fix

Conclusion

Setting up `ZenRowsWebReader`