DEV Community

Martin Petrov
Martin Petrov

Posted on

Compare one product's price across online stores with Python (real output)

#webscraping #api #ecommerce #python

If you sell a product, resell it, or just buy it for an office, you keep asking the same question: which store has it cheapest right now, in my currency?

I wanted that answer as JSON from one call, so I built a small Cheapest Price Finder API. This post shows how to call it from curl and Python, the real output I got today (2026-11-10), how to run it every day and get only price changes, and what it does not do.

Why the usual approach hurts

The usual path for "compare a product across 4 shops" looks like this:

  1. Write a scraper per store, because every shop puts the price somewhere else.
  2. Find out that some shops put it in JSON-LD, some only in Shopify's product JSON, some in a meta tag, and some render it with JavaScript.
  3. Handle variants, sale prices (compareAtPrice) and "out of stock".
  4. Convert USD, EUR and GBP to one currency so the numbers are comparable.
  5. Sort, compute the difference to the cheapest, and save it somewhere.
  6. Fix it again when a store changes its theme.

None of it is hard. It is just a day of glue code per store that you then maintain forever.

What the API does

You give it product page URLs from any stores (or a product name plus store domains). For each page it reads the real price from the page's structured data (JSON-LD offers, Shopify product data or meta price tags), converts it with the daily ECB reference rates, and returns every offer sorted from cheapest to most expensive.

Each row has rankByPrice, isCheapest, price, currency, priceConverted, moreThanCheapest (amount and %), availability, brand, gtin, sku, rating and image. A RUN_SUMMARY record lists the cheapest offer and every page it could not read, with the reason.

It respects each store's robots.txt, sends an honest User-Agent and does not solve CAPTCHAs or bypass bot checks. A store that blocks automated requests is reported, not billed.

You can call it two ways:

Option 1: API.market with curl

Subscribe to the free plan on the API.market page, copy your key, then:

curl -X POST 'https://prod.api.market/api/v1/martindata/price-comparison-api/m/cheapest-price-finder' \
  -H 'x-api-market-key: YOUR_API_MARKET_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"productUrls": [
        "https://www.vervecoffee.com/products/aeropress-coffee-brewer",
        "https://www.intelligentsia.com/products/aeropress",
        "https://www.espressoparts.com/products/aeropress-coffee-maker",
        "https://www.stumptowncoffee.com/products/aeropress"],
      "convertTo": "EUR"}'
Enter fullscreen mode Exit fullscreen mode

The response is JSON with the sorted offers.

Option 2: Python through the Apify API

Get a token in Apify Console (Settings, API and Integrations). Then:

import os
import requests

TOKEN = os.environ["APIFY_TOKEN"]  # your own token
URL = ("https://api.apify.com/v2/acts/martin10291~cheapest-price-finder/"
       "run-sync-get-dataset-items")

def compare(urls, currency="EUR"):
    r = requests.post(URL, params={"token": TOKEN},
                      json={"productUrls": urls, "convertTo": currency},
                      timeout=180)
    r.raise_for_status()
    return r.json()

urls = [
    "https://www.espressoparts.com/products/aeropress-coffee-maker",
    "https://vervecoffee.com/products/aeropress-coffee-brewer",
    "https://stumptowncoffee.com/products/aeropress",
    "https://intelligentsia.com/products/aeropress",
]
for o in compare(urls):
    print(f"#{o['rankByPrice']} {o['store']:22} {o['price']:>6} {o['currency']}"
          f" = {o['priceConverted']:>6} EUR  +{o['moreThanCheapestPercent']}%  {o['availability']}")
Enter fullscreen mode Exit fullscreen mode

I tested it with the AeroPress Original coffee maker, because four coffee shops sell the same item. This is what I got today. The run took 2.9 seconds on Apify:

#1 vervecoffee.com         39.95 USD =  35.69 EUR  +0%     InStock
#2 intelligentsia.com      40    USD =  35.74 EUR  +0.1%   OutOfStock
#3 espressoparts.com       44.95 USD =  40.16 EUR  +12.5%  InStock
#4 stumptowncoffee.com     46    USD =  41.1  EUR  +15.2%  InStock
Enter fullscreen mode Exit fullscreen mode

So the same product costs 15% more in one shop than in another, and the second cheapest is out of stock. That second fact is why I return availability in every row: the cheapest offer is useless if you cannot buy it. Set "onlyInStock": true to drop those rows.

One full row (trimmed):

{
  "rankByPrice": 3,
  "isCheapest": false,
  "title": "AeroPress Coffee Maker - The Original 3-in-1 Coffee Brewer for Home, Travel & Adventure",
  "store": "espressoparts.com",
  "price": 44.95,
  "currency": "USD",
  "priceConverted": 40.16,
  "convertedCurrency": "EUR",
  "moreThanCheapest": 4.47,
  "moreThanCheapestPercent": 12.5,
  "availability": "InStock",
  "brand": "AeroPress",
  "gtin": "085276000824",
  "sku": "AEROPRESS",
  "rating": 4.89,
  "reviewCount": 9,
  "priceSource": "json-ld",
  "url": "https://www.espressoparts.com/products/aeropress-coffee-maker"
}
Enter fullscreen mode Exit fullscreen mode

gtin helps you check that two shops really sell the same item. In this run espressoparts.com and stumptowncoffee.com both show 085276000824. Not every shop publishes a GTIN, so treat it as a check, not a guarantee.

Searching by product name (best effort)

If you do not have the URLs yet, you can pass a name and store domains:

r = requests.post(URL, params={"token": TOKEN}, timeout=180, json={
    "searchQuery": "AeroPress coffee maker",
    "stores": ["aeropress.com", "walmart.com", "ebay.com"],
    "resultsPerStore": 2,
    "convertTo": "EUR",
})
Enter fullscreen mode Exit fullscreen mode

I will be honest about this mode: in my test today it found 0 offers. Walmart answered with a bot-check page (the tool does not solve those, it reports them), the search returned the AeroPress home and collection pages instead of a product page, and the search engine rate-limited the run. The RUN_SUMMARY said exactly that and suggested adding product URLs. Product URLs are the reliable input; use search only to discover candidates.

Run it every day and keep only price changes

Price checks are useful only when they repeat. Save the input above as a Task in Apify Console and add a daily schedule. Then compare each run with the previous one. The actor has no built-in price-change mode yet, so here is the small diff I use:

import json, pathlib

state = pathlib.Path("last_prices.json")
old = json.loads(state.read_text()) if state.exists() else {}
offers = compare(urls)
new = {o["url"]: o["priceConverted"] for o in offers}

for o in offers:
    before = old.get(o["url"])
    if before is not None and before != o["priceConverted"]:
        print(f"{o['store']}: {before} -> {o['priceConverted']} EUR")

state.write_text(json.dumps(new))
Enter fullscreen mode Exit fullscreen mode

Run it from cron on any small server, or from an Apify schedule with a webhook to your own endpoint, and you get a message only when a price moves. A built-in mode for this (only changed prices, per watch list) is the next thing I am adding.

Honest limits

  • Big marketplaces often block automated requests. Amazon, Walmart, Target, Best Buy and Newegg frequently answer with a bot-check page. You get a clear reason in RUN_SUMMARY.failed and you are not charged. It works best with brand stores and independent shops (Shopify, WooCommerce, Magento, BigCommerce).
  • Pages that build the price only with JavaScript and publish no product data return "no price found".
  • No shipping costs or coupon codes. It compares the listed product price.
  • Currency conversion uses daily ECB reference rates, not your card's rate.
  • It does not match products for you. You decide which URLs are the same item; gtin, sku and brand help.

Price

  • Apify: $3.00 per 1,000 results, and only offers with a price count as results (failed pages are free). The 4-store run above cost about one cent. Apify's free plan includes a small monthly credit, enough to try it.
  • API.market: a free plan to test, then the paid plans shown on the listing.

Wrap-up

I built this because I kept writing the same "find the price on this page" code for every new shop. If you run a small reseller business, a deal newsletter, procurement for an office, or a repricing script, try the free plan on API.market or the Apify actor.

Which stores do you need to compare? If one of them returns "no price found", post the URL in the comments and I will look at it.

Top comments (0)