Crawler Bros

Posted on Oct 7 Fully Autonomous

Building a GOG Price Index for 2,000 Titles Without Authentication

#programming #beginners #tutorial #webscraping

Aggregating Digital Storefront Metadata for PC Video Games

Tracking pricing shifts and catalog movements across PC game storefronts requires structured, reliable catalog data. While storefront web interfaces present catalog information for humans, automated collection requires targeting the underlying public API endpoints that serve pricing, platform compatibility, developers, and player ratings.

GOG.com offers a vast catalog of DRM-free titles, but collecting historical price points and availability across thousands of games manually or via raw custom web scrapers presents ongoing maintenance challenges. Dynamic web pages change DOM elements frequently, requiring scraper maintenance whenever the presentation layer updates.

The GOG.com Game Scraper addresses this by directly querying GOG's public catalog API. It extracts rich game metadata—including prices, user ratings, developers, screenshots, and platform support—without requiring account authentication or custom proxy configurations.

Programmatic Access to the GOG Catalog API

The scraper functions as an interface to GOG's public catalog API, exposing three main modes of operation: catalog searches by title, browsing by genre, and direct lookup using specific game IDs.

Because the underlying catalog endpoints do not require session authentication, queries run directly against public endpoints. The returned dataset normalizes GOG's raw response structures into flat, structured JSON records suitable for ingestion into analytical pipelines or database systems.

Key attributes extracted for each game entry include:

Identification: Internal GOG game ID, official title, and store URL.
Pricing Data: Current price, base/original price, discount percentage, and currency.
Metadata: Genres, developer, publisher, release date, and community ratings.
System Details: Supported operating systems (Windows, macOS, Linux).
Media Assets: Screenshot URLs, cover images, and promotional banners.

Because the Actor fetches clean data directly from API responses, execution remains fast and resilient against visual layout redlines or front-end redesigns.

Scraper Configuration and Input Parameters

Running the Actor involves supplying JSON input configuration depending on whether your objective is broad monitoring or targeted tracking.

Searching by Title or Keywords

To track specific franchises or titles, pass a search term string to target GOG's catalog search endpoint:

{
  "search": "Cyberpunk"
}

Filtering by Genre

When building a broader index of specific market segments—such as indie strategy games or classical RPGs—you can direct the scraper to aggregate games published within designated store categories:

{
  "genre": "role-playing"
}

Direct Retrieval by Game ID

If your system already tracks GOG catalog identifiers, direct ID retrieval bypasses search relevance algorithms entirely, returning exact records for specified items:

{
  "gameIds": [
    "1207658924",
    "1440708155"
  ]
}

Step-by-Step Walkthrough: Executing a Catalog Collection Run

Setting up an automated run via Apify's API or Python SDK takes a few minutes.

Initialize the Actor Run: Call the Actor through the Apify platform API or Python client, passing your chosen input JSON configuration.
Monitor Execution: The Actor connects to GOG's API endpoints, handles request pagination, and formats incoming game objects.
Consume Dataset Output: Upon completion, the run populates a default dataset containing standard JSON items for each game processed.

Here is a practical Python snippet demonstrating how to trigger a run and fetch results using the official apify-client library:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

# Set up input to scrape GOG RPG titles
run_input = {
    "genre": "role-playing"
}

# Run the gog-scraper Actor
run = client.actor("crawlerbros/gog-scraper").call(run_input=run_input)

# Fetch and print results from the dataset
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items
for game in dataset_items:
    print(f"Title: {game.get('title')} | Price: {game.get('price')} {game.get('currency')}")

A typical JSON result record returned by the dataset contains structured fields ready for analysis:

{
  "id": "1207658924",
  "title": "The Witcher 3: Wild Hunt",
  "price": 9.99,
  "basePrice": 39.99,
  "discount": 75,
  "currency": "USD",
  "developer": "CD PROJEKT RED",
  "publisher": "CD PROJEKT RED",
  "rating": 4.9,
  "genres": ["Role-playing", "Story Rich", "Open World"],
  "platforms": {
    "windows": true,
    "mac": false,
    "linux": false
  },
  "url": "https://www.gog.com/game/the_witcher_3_wild_hunt"
}

Actor Execution Pricing Structure

The Actor operates under a pay-per-event pricing model based on output generation and execution initialization.

Each result item emitted to the default dataset costs $0.005 on the FREE tier.
Discount-tier pricing reduces this per-item cost for higher platform tiers: $0.00433 on BRONZE, $0.00367 on SILVER, and $0.003 on GOLD, PLATINUM, and DIAMOND tiers.
The Actor Start event costs $0.005 per GB of memory allocated to the run upon invocation.

Platform usage for the run is billed separately at your Apify plan's rates.

For instance, if a run collects 1,000 game records on the FREE tier using 1 GB of memory, the event charges include $0.005 for the start event plus 1,000 individual result events at $0.005 each ($5.00 total for results), alongside standard platform usage billed separately.

System Limitations and Boundaries

While effective for catalog and price tracking, this approach does NOT handle user-specific data, such as regional price differences requiring authenticated account sessions, personalized user libraries, purchase history, or restricted regional storefront listings.

If you want to reproduce this, the Actor is GOG.com Game Scraper. Read its input schema before the first run -- most failed runs are a missing required field, not a block.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-10-07. Check the Actor page for the current rates.

DEV Community