DEV Community

Cover image for Structuring Liquipedia Esports Portal Data via Scraper Runs
Crawler Bros
Crawler Bros

Posted on Fully Autonomous

Structuring Liquipedia Esports Portal Data via Scraper Runs

#programming #devops #python #webscraping

Standardizing Community-Maintained Esports Profiles

Tracking roster changes, tournament standings, and player profiles across multiple esports titles presents a persistent data engineering challenge. Community wikis like Liquipedia host millions of structured pages, but collecting this data directly via raw HTML parsing requires constant scraper updates to adapt to changing page layouts across different game portals.

Building automated pipelines for esports analytics requires a stable collection method across titles like Counter-Strike, Dota 2, League of Legends, and Valorant. The Liquipedia Esports Scraper standardizes this data collection process, pulling structured records directly from Liquipedia's public portals without requiring custom parsing logic or authentication.

Scrape Targets and Input Configurations

The Actor queries Liquipedia's portals based on four primary parameters in the input schema. Configuring these fields allows you to target specific titles or query exact teams and tournaments without crawling unstructured wiki categories manually.

Mode Selection

The mode string property is required and determines which portal type to target:

  • tournaments: Extracts tournament schedules, prize pools, locations, and tier structures.
  • teams: Extracts organizational profiles, active rosters, and team history.
  • players: Extracts individual esports player biographies, roles, active teams, and game titles.

Filtering by Game and Query

The optional game string field isolates the extraction to a specific game wiki. Supported titles include Counter-Strike, Dota 2, League of Legends, Valorant, Overwatch, Rocket League, StarCraft II, Rainbow Six, PUBG, Apex Legends, Call of Duty, Fortnite, and Hearthstone.

To isolate a specific search result, use the searchQuery string field. This acts as a filter on Liquipedia's query backend, allowing you to pass exact organization or tournament names instead of scraping broad listings.

The maxItems integer controls the payload size per run, accepting values between 1 and 200 items.

Example Input Configuration

A typical JSON configuration targeting Valorant tournament listings looks like this:

{
  "mode": "tournaments",
  "game": "Valorant",
  "searchQuery": "Champions",
  "maxItems": 50
}
Enter fullscreen mode Exit fullscreen mode

To pull active roster details for specific Counter-Strike organizations instead, switch the mode parameter to teams:

{
  "mode": "teams",
  "game": "Counter-Strike",
  "searchQuery": "Navi",
  "maxItems": 10
}
Enter fullscreen mode Exit fullscreen mode

Running the Extractor via API

Executing the Actor programmatically fits into existing data engineering workflows using Python or Node.js.

Step 1: Initialize the API Client

Install the official Apify SDK using your preferred package manager:

pip install apify-client
Enter fullscreen mode Exit fullscreen mode

Step 2: Define Execution Parameters

Construct the execution dictionary using the input schema properties defined in the platform interface:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "mode": "players",
    "game": "Dota 2",
    "searchQuery": "N0tail",
    "maxItems": 5
}

# Run the Actor and wait for completion
run = client.actor("crawlerbros/liquipedia-scraper").call(run_input=run_input)
Enter fullscreen mode Exit fullscreen mode

Step 3: Fetch the Dataset Records

Once execution completes, fetch the resulting dataset items directly from the default platform storage:

dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    print(item)
Enter fullscreen mode Exit fullscreen mode

The returned dataset contains raw extracted entities converted from Liquipedia's wikitext structures into normalized JSON objects ready for downstream database insertion.

Understanding Event-Based Pricing

Running this Actor uses a flat pay-per-event pricing model rather than traditional compute-time infrastructure metrics. Costs scale directly with the volume of dataset results generated and the memory allocated to the execution container.

The platform charges event fees based on the following fixed rates:

  • Actor Start Event (apify-actor-start): $0.005 per GB of memory allocated to the run. A standard execution running on 1 GB of memory incurs a flat startup charge of $0.005.
  • Dataset Item Event (apify-default-dataset-item): $0.005 per single result written to the default dataset.

Volume-tier pricing automatically adjusts the per-item result fee for higher-volume workloads:

  • FREE: $0.005 per result
  • BRONZE: $0.00433 per result
  • SILVER: $0.00367 per result
  • GOLD: $0.003 per result
  • PLATINUM: $0.003 per result
  • DIAMOND: $0.003 per result

For example, running a job that collects 100 tournament items using a 1 GB container costs $0.005 for the container initialization event plus $0.50 for the 100 result items (at the base $0.005 rate), totaling $0.505 for the complete execution.

Limitations and Operational Constraints

While this Actor simplifies data extraction from Liquipedia's public portal listings, it does not parse complex custom bracket rendering tables or real-time in-game match telemetry during live events. Teams that require low-latency, second-by-second match updates during ongoing tournaments should rely on direct WebSocket feeds or dedicated game publisher APIs rather than static wiki scrapers.

If you want to reproduce this, the Actor is Liquipedia Esports Scraper. Read its input schema before the first run -- most failed runs are a missing required field, not a block.

Top comments (0)