Comparing prices across retailers is a unit-normalization problem, not a scraping problem

Disclosure: I'm the founder of Popgot, which I use as the example below. The problem and the approach apply regardless of what you build on.

Every price comparison project I've seen starts the same way: scrape a bunch of retailers, store the prices, sort ascending. And then it produces garbage rankings, because price is not a comparable field.

Here's the classic failure. Three listings for AA batteries:

Listing	Price	Count
Brand A	$5.99	16
Brand B	$6.99	20
Brand C	$11.94	40

Sort by price and Brand A "wins" at $5.99. Sort by cost per battery and the order flips completely: Brand C is ~29.9c per cell, Brand A is ~37.4c. The cheapest listing is the worst deal on the page.

Why this is hard

The naive fix is "just divide price by quantity." The problem is that quantity almost never exists as a clean number. It's buried in the title, and the title is written by whoever uploaded the listing:

• AA Batteries 24 Pack
• AA Alkaline Batteries, 1.5 Volts, 24 Count
• 48-Pack (2 x 24) Double A

So you end up writing a title parser. Then you discover the same product needs a different unit depending on the category: per fluid ounce for detergent, per serving for protein powder, per 100g for coffee, per sheet for paper towels. Then you discover that some categories need a spec filter before unit price is even meaningful. A fish oil at 20c per serving isn't cheaper than one at 34c per serving if the first one has half the EPA+DHA. You're comparing two different products.

That last part is the piece people underestimate. Normalization is only valid within a set of products that actually satisfy the same requirement, which means something has to read the label, not just the title.

What a normalized record looks like

This is the problem I ended up building Popgot around, so rather than describe it abstractly, here's the shape of the data. The developer API returns listings with the unit math already done:

GET /api/developer-api/products?query=aa+batteries&limit=10

{
  "products": [
    {
      "display_title": "ACDelco 40-Count AA Batteries",
      "source_type": "amazon",
      "price_cents": 1194,
      "unit_count": 40,
      "price_cents_per_unit": 29.85,
      "rating_average": 4.7,
      "review_count": 54696,
      "value_score": 27.413,
      "rank": 5
    }
  ]
}

The fields that matter for this problem are unit_count and price_cents_per_unit. source_type tells you which retailer the listing came from, so cross-retailer comparison is a single sort instead of a reconciliation job.

A minimal client:

const res = await fetch(
  "https://popgot.com/api/developer-api/products?" +
    new URLSearchParams({ query: "aa batteries", limit: 20 })
);

const { products } = await res.json();

const byUnitPrice = products
  .filter((p) => p.unit_count > 0)
  .sort((a, b) => a.price_cents_per_unit - b.price_cents_per_unit);

for (const p of byUnitPrice.slice(0, 5)) {
  console.log(
    `${(p.price_cents_per_unit / 100).toFixed(3)}/unit`,
    `${p.source_type.padEnd(8)}`,
    p.display_title
  );
}

Note the unit_count > 0 guard. Any dataset like this will have listings where the count couldn't be resolved, and you want those excluded from a unit-price sort rather than silently ranked at zero.

Things worth knowing before you build on it

I'd rather you hit these in a blog post than in production, so here are the sharp edges — including the ones in my own API.

value_score is opinionated. It blends unit price with rating signals, so it is not the same as "cheapest." In the sample above the top-ranked-by-value item is not the lowest price_cents_per_unit. If your product promises "cheapest," sort on the raw unit price yourself and ignore rank.

Cache aggressively, but treat cached prices as hints. Prices move, and the retailer's price at checkout is the one that actually applies. Never present a stored price as a guarantee.

Units are category-specific. Don't build UI copy that hardcodes "per item." Render whatever unit the category actually uses, or your detergent page will say "$0.06 per item" and mean nothing.

Spec filters belong upstream. If a user needs "at least 1000mg EPA+DHA," express that in the query rather than post-filtering on the title string. Title-based filtering will drop valid products and keep invalid ones.

The takeaway

If you're building anything that ranks products — a deals site, a budgeting tool, an internal procurement dashboard — the interesting engineering isn't collecting prices. It's deciding what the denominator is, and making sure the things you're dividing are genuinely substitutable. Get that wrong and you ship a sorted list that confidently recommends the worst option.

If you want to eyeball the output before writing any code, the search side of the same engine is at popgot.com — useful for sanity-checking your own unit math against ours, and for finding the cases where we get it wrong.

How are you handling this? I'm especially curious whether anyone has found a clean way to normalize multi-pack listings (2 x 24) without a pile of regex.