DEV Community

Minexa.ai
Minexa.ai

Posted on

Scraping SaaS review data for competitive intelligence: a developer's guide with Minexa API

Competitive intelligence tools live and die by data freshness. When your product tracks how software vendors are perceived across the market, manually checking review pages is not a workflow — it is a bottleneck.

SaaS review sites are one of the richest public sources for this kind of signal. Each product page contains reviewer sentiment, feature-level ratings, use case context, and version-specific feedback. The problem is that this data is spread across thousands of individual URLs, updated continuously, and not available via any official API.

This guide covers how to extract that data programmatically using the Minexa API — a structured web extraction API that removes the need to write CSS selectors, manage rendering infrastructure, or deal with inconsistent LLM output.

📋 What a review detail page typically contains

Field reference card

Field Example value
Reviewer name Anonymous / display name
Overall rating 4.2 / 5
Review title 'Great for mid-market teams'
Review body Full text content
Feature ratings Ease of use, support, value
Reviewer role 'Product Manager, 200-500 employees'
Verified purchase flag true / false
Review date 2024-11-03
Helpful votes 12
Vendor response Text + date

All of these fields live in the HTML of a single review page. Minexa extracts them by reading the DOM structure directly — no interpretation, no guessing.

🔧 How the Minexa API works for this use case

The developer workflow has two stages.

Stage 1 — Train the scraper once
Open a representative review URL in Chrome with the Minexa extension active. Minexa detects the page structure automatically and surfaces all available data points. Confirm the fields you want. The extension generates a stable scraper_id — for example 7431. That ID is reusable indefinitely across any structurally similar review page.

Stage 2 — Call the API at scale
Pass your list of review URLs to the API along with the scraper_id. Minexa handles JavaScript rendering, anti-bot layers, and geo-targeted content automatically — no additional configuration needed.

Minexa API request structure

import requests

response = requests.post(
  'https://api.minexa.ai/data',
  headers={
    'Content-Type': 'application/json',
    'Authorization': 'Bearer YOUR_API_KEY'
  },
  json={
    'scraper_id': '7431',
    'columns': ['top_25'],
    'urls': [
      'https://reviewsite.com/product/123/reviews',
      'https://reviewsite.com/product/456/reviews'
    ]
  }
)

data = response.json()
Enter fullscreen mode Exit fullscreen mode

You can batch up to 50,000 URLs in a single request. For competitive intelligence pipelines covering hundreds of products across multiple review platforms, this matters.

🔁 Handling paginated responses

When results span multiple pages, the API returns a next_token field. Use it to fetch subsequent pages until the token is absent.

all_results = []
token = None

while True:
  payload = {'scraper_id': '7431', 'columns': 'top_25',
             'urls': ['https://reviewsite.com/product/123/reviews']}
  if token:
    payload['next_token'] = token
  res = requests.post('https://api.minexa.ai/data',
                      headers=headers, json=payload).json()
  all_results.extend(res.get('data', []))
  token = res.get('next_token')
  if not token:
    break
Enter fullscreen mode Exit fullscreen mode

⚠️ Why LLM-based extraction breaks down here

Callout: the accuracy problem at scale
Review pages often contain multiple date fields (review date, vendor response date, last edited date) and multiple rating values (overall, feature-specific). LLM-based extractors have to infer which value maps to which field. At small volume this is manageable. Across tens of thousands of pages, misassigned values accumulate silently and corrupt downstream analysis.

Minexa binds each column to a specific DOM position. If a value is not present on a page, the output returns null — never a fabricated value.

⏱️ Scheduling and cron setup

The Minexa API does not manage scheduling internally. For recurring extraction — weekly review snapshots, daily sentiment checks — set up your own cron job and pass fresh URL lists to the API on each run. This gives you full control over cadence and URL scope without depending on any external scheduler.

👥 Who benefits from this

Persona takeaway — Competitive intelligence platforms
Your customers expect current data. Review sentiment shifts after product launches, pricing changes, and support incidents. A pipeline that pulls structured review data on a defined schedule gives your platform a live signal layer that static datasets cannot match. The scraper trains once per review site structure and runs indefinitely from that point.

Start building with the Minexa API

For related reading on extracting structured data at scale, see: Scraping e-commerce product pages for price monitoring

Top comments (0)