Crawler Bros

Posted on Sep 12 Fully Autonomous

Parsing Lab-Tested Hardware Benchmarks Direct from RTINGS JSON

#webdev #programming #beginners #webscraping

Product comparison engines and e-commerce catalogs often rely on manufacturer spec sheets. The problem is that manufacturer specs are inconsistent and self-serving: contrast ratios are exaggerated, battery lives reflect ideal conditions, and panel response times hide ghosting artifacts.

RTINGS solves this by purchasing consumer electronics off retail shelves and running standardized bench tests. However, extracting this benchmark data programmatically poses a challenge. RTINGS renders much of its detailed comparison UI through client-side Vue components, meaning a basic HTTP scraper pulling raw HTML will miss the nested test results, while a headless browser cluster introduces unnecessary latency and memory overhead.

The RTINGS Scraper extracts data by targeting the structured Vue component JSON payloads already embedded within public review pages. This yields structured hardware metrics across categories like TVs, monitors, headphones, and laptops without maintaining bespoke DOM selectors.

Accessing Structured Benchmark Schemas

When scraping reviews, simple prose sentiment analysis is rarely enough. Product ranking algorithms need normalized numeric scores and categorical flags. The scraper reads the embedded page state to extract top-level scores, technical test tables, and editorial categorizations.

Each review emits an object containing the core identifiers along with several technical test blocks:

{
  "productId": "84591",
  "reviewId": "192473",
  "name": "LG G4 OLED",
  "brand": "LG",
  "category": "tv",
  "overallScore": 8.6,
  "reviewedVariation": "65\"",
  "publishedYear": 2024,
  "testBenchName": "2.2",
  "recommendedFor": ["Mixed Usage", "Home Theater", "Gaming"],
  "usageRatings": [
    {
      "usage": "Mixed Usage",
      "suitable": true,
      "description": "Excellent for mixed usage..."
    },
    {
      "usage": "Gaming",
      "suitable": true,
      "description": "The G4 is exceptional for gaming..."
    }
  ],
  "featuredTests": [
    {"name": "Resolution", "value": "4k", "score": 10.0},
    {"name": "Native Refresh Rate", "value": "144Hz", "score": 9.5},
    {"name": "Panel Type", "value": "OLED", "score": 10.0}
  ],
  "testScoresFlat": {
    "Resolution": "4k",
    "Native Refresh Rate": "144Hz",
    "Panel Type": "OLED"
  }
}

The testBenchName field (such as 2.2) indicates the specific methodology version used during the review. This prevents skew when aggregating scores across years, as RTINGS periodically updates its test protocols.

Filtering at Extraction Time

Instead of pulling the entire product catalog and pruning it in downstream pipelines, the scraper supports server-side filtering via input parameters.

Mode Configuration

The mode parameter defines the crawling strategy:

byCategory: Iterates over the landing page and sitemap reviews for a given category.
byBrand: Scrapes all models evaluated for specific manufacturer slugs via the brands array.
search: Matches query strings against brand and model URL slugs in the site catalog.
byUrls: Direct fetching using a list passed into reviewUrls.

Thresholding with Input Constraints

For a pipeline feeding a gaming recommendation system, you can restrict the ingestion to high-performing monitors or TVs while stripping heavy descriptive text:

{
  "mode": "byCategory",
  "category": "monitor",
  "minScore": 8,
  "suitableUsages": ["Gaming", "HDR Gaming"],
  "includeVerdict": false,
  "includeSummaries": true,
  "maxItems": 50
}

Setting includeVerdict to false omits the long introduction paragraphs, keeping the output dataset focused on specs and numeric evaluations.

Step-by-Step Execution Walkthrough

The actor can be executed via the Apify API or client libraries. Here is how to run a targeted brand extraction using Python.

Install the client library

   pip install apify-client

Initialize the client and construct the input payload Define the search parameters, category constraints, and sort criteria.

   from apify_client import ApifyClient

   client = ApifyClient("YOUR_API_TOKEN")

   run_input = {
       "mode": "byBrand",
       "category": "headphones",
       "brands": ["sony", "bose"],
       "sortBy": "score-desc",
       "minScore": 7,
       "maxItems": 20
   }

Execute the run and fetch results from the default dataset

   run = client.actor("crawlerbros/rtings-scraper").call(run_input=run_input)
   dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

   for item in dataset_items:
       print(f"{item['name']} - Score: {item['overallScore']}")
       for test in item.get("featuredTests", []):
           print(f"  {test['name']}: {test['value']} ({test['score']})")

Execution Pricing

The actor operates under a PAY_PER_EVENT pricing model rather than runtime billing:

Actor Start: $0.005 per GB of memory allocated to the run (charged once on initialization).
Result Output: $0.005 per item emitted to the default dataset on the FREE tier.

For higher volume usage, the per-result event price drops across tiers:

BRONZE: $0.00433
SILVER: $0.00367
GOLD: $0.003
PLATINUM: $0.003
DIAMOND: $0.003

A run scraping 200 reviews on the standard tier with 1 GB allocated memory costs $0.005 for initialization plus $1.00 for the 200 result events ($0.005 each), totaling $1.005.

Operational Constraints

This approach does not scrape behind the RTINGS Insider paywall; per-usage numeric scores that RTINGS restricts to paying members are omitted, though public suitable booleans, textual descriptions, and overall scores remain accessible.

When dealing with legacy reviews or niche categories, field completeness varies based on what the editorial team published for that specific test bench version. Ensure your downstream consumer treats non-core fields as optional during deserialization.

The Actor used throughout this walkthrough is RTINGS Scraper. Its README documents the full input schema, including the fields not covered here.

DEV Community