Competitor price tracking: why most setups break and what actually holds up

Tracking competitor prices sounds like a solved problem. Crawl a few sites once a day, pull product names and prices, diff yesterday versus today, send a summary. Simple enough to sketch on a whiteboard in five minutes.

In practice, the implementation is where things get complicated fast.

The actual problems, one by one

Static URL lists break immediately

The first instinct is to hardcode a list of product page URLs and hit them on a schedule. That works until a competitor adds new products, discontinues others, or restructures their catalog. Suddenly your snapshot is incomplete and you have no signal that anything changed.

The correct approach is to start from category or listing pages, not individual product URLs. You scrape the listing layer first to discover what products currently exist, then follow through to detail pages. This means your pipeline always reflects the live catalog, not a frozen snapshot from setup day.

JavaScript rendering is not optional for most sites

Many product pages load prices dynamically. A raw HTTP fetch returns a skeleton HTML with no price in it. You need a browser layer that actually executes JavaScript before extraction. This is where a lot of lightweight setups fall apart: the fetch succeeds, the extraction runs, and you get null or a stale cached value with no error to indicate anything went wrong.

Selector-based scrapers fail silently

If you write CSS or XPath selectors by hand, they work until the site does a minor layout update. After that, your selector either matches nothing (you get null) or matches the wrong element (you get a plausible but incorrect value). The second case is the dangerous one: your pipeline keeps running, your spreadsheet keeps filling up, and the data is wrong with no alert.

LLM extraction is expensive at any real volume

Using an LLM to parse product pages is tempting because it handles layout variation without selectors. The problem is cost. A full rendered HTML page can easily exceed half a million tokens. At that size, even the cheapest available models cost well over $0.02 per page. At 80,000 pages per month across five competitor sites, that becomes a significant recurring expense, and you still need validation logic because LLMs can swap sale price and original price when both appear on the same page.

Minexa's pricing is per page, not per token, so page size does not affect cost. The same extraction job that costs hundreds of dollars monthly with an LLM pipeline runs for a flat monthly rate with Minexa.

What a working pipeline actually looks like

Here is the structure that holds up in production:

• Listing page scrape: hit each competitor's category page to get the current product set and their detail URLs
• Detail page scrape: extract price, product name, availability, and any other relevant fields from each product page
• Diff logic: compare today's output against yesterday's stored snapshot
• Summary generation: format the changes and send a report

The extraction layer is where most of the engineering pain lives. Everything else is straightforward once you have reliable, consistent structured data coming out.

How Minexa fits into this

Minexa.ai is a DOM-based extraction platform. You train a scraper once using the Chrome extension by pointing at the HTML container that holds the data block you want. Minexa identifies all the data fields inside it automatically, no selector writing required. That scraper gets a stable scraper_id you reference in every API call going forward.

Training takes two to five minutes per site. After that, you call the API with your list of URLs and get structured JSON back.

A basic request looks like this:

import requests

url = "https://api.minexa.ai/data/"

data = {
  "batches": [{
    "scraper_id": 4731,
    "columns": ["top_30"],
    "urls": [
      "https://competitor-site.com/product/abc",
      "https://competitor-site.com/product/xyz"
    ],
    "scraping": {
      "js_render": True,
      "timeout": 30,
      "js_code": [
        {"wait_time": 2},
        {"page_init": True},
        {"wait_time": 4}
      ],
      "proxy": "verified",
      "retry": 3
    }
  }],
  "threads": 5
}

headers = {
  "Content-Type": "application/json",
  "api-key": "YOUR_API_KEY"
}

response = requests.post(url, json=data, headers=headers)
print(response.json())

Key parameters to understand:

• scraper_id: the ID generated when you train a scraper via the extension. One scraper per site structure, reused across all pages of that type.
• columns: "top_30" returns the 30 highest-ranked fields Minexa found on the page. You can also pass explicit column names like ["price", "product_name", "availability"] once you know which fields you need. Both options cost the same.
• js_render: set to true for any page that loads content dynamically.
• threads: controls how many pages are processed in parallel. Higher values mean faster jobs.
• retry: Minexa will automatically re-attempt failed fetches up to the number you set.

What you get back: consistent structured JSON for every URL, with the same field names across all pages processed by the same scraper. No normalization step needed.

Failure behavior worth knowing

Minexa is designed to fail loudly. If a page structure changes and the scraper no longer matches the HTML, affected fields return null or an explicit error, not a silently wrong value. If you accidentally pass a URL that does not match the scraper it was trained on, you get an error indicating the mismatch.

When a site does a significant redesign, you open an affected page in the extension, select the updated container, and create a new scraper. Same two-to-five minute process. The only change in your code is updating the scraper_id and checking whether the column names you rely on have shifted.

Getting started

Step 1: Install the Minexa Chrome extension

Step 2: Open a competitor product page and click 'Get Started' in the extension

Step 3: Hover over and select the HTML container that wraps the product data block

Step 4: Click 'Create Scraper' and wait a few minutes for field discovery to complete

Step 5: Click 'API Request' in the top right to get pre-generated Python code with your scraper_id already filled in

Step 6: Update the URL list, set your thread count, and run the script

Most setups reach first structured output in under ten minutes. Once the scraper exists, the engineering work is done. Your daily job is just passing the current URL list and processing the output.

For more on building extraction pipelines that hold up over time, see: 10 ways to build a price-tracking scraper that actually holds up