Competitor price tracking looks like a scraping problem when you scope it, and it stops looking like one about three weeks in. The fetching gets solved. What keeps slipping is everything that happens to the data after it arrives. Here is where the real work sits, based on how these pipelines are actually built.
The Collection Problem Is the One You Expect
Collection is the stage everyone plans for, and it is the stage with known answers. Competitor pages are increasingly JavaScript-rendered, so you need a real browser rather than an HTTP client, which means Playwright or Puppeteer driving headless Chrome. High request volume against the same origin gets you rate limited or IP banned, so you rotate proxies. Layout changes break selectors, so you build selector repair and alerting rather than assuming a scraper written once stays written.
Where an API exists, use it. Marketplace APIs give you structured data without parsing, at the cost of rate limits, restricted fields, and terms that constrain what you can do with the result. Most working setups are a mix: APIs where they cover the catalog, scraping where they do not.
None of this is easy, but all of it is understood. Budget for it and it gets done. The stages after it are the ones that surprise people.
Product Matching Is Where Projects Stall
Once the data is in, you have to answer a question that sounds trivial and is not: is their product the same as your product?
Competitor listings rarely share your SKUs. Product names differ. Unit sizes differ, so a three-pack on their site sits in the same result set as a single unit on yours. Bundles include accessories yours does not. You end up matching on some combination of UPC, normalized product name, attribute extraction, and sometimes image similarity, and you end up with a confidence score rather than a yes or no.
This matters more than it sounds. Every downstream number, price position, competitive index, MAP violation, is only as correct as the matching underneath it. A pipeline that collects flawlessly and matches sloppily produces confident, wrong dashboards. If the schedule is going to slip somewhere, it slips here, so it belongs on the plan as its own phase rather than as a line item inside processing.
A Price Is Not a Price
The second thing that breaks assumptions: the number on the page is not what the customer pays.
A competitor listed five dollars below you might charge more for shipping. They might be out of stock, in which case their price is not competing with anything. They might be bundling fewer accessories, or selling a different quantity, or showing a promotional price that expires tonight. On marketplaces there is a further layer, because the Buy Box winner captures roughly 80 percent of sales on a product page, and Buy Box ownership depends on seller rating and fulfillment as much as on price.
So the useful unit of comparison is not a price, it is a landed, availability-adjusted, quantity-normalized price with the context that produced it. Store the raw observation, price, shipping, stock state, seller, timestamp, and derive the comparable figure from that. Storing only the derived figure means you can never re-derive it when the rules change, and the rules always change. The wider version of this argument, including how it feeds category-level analysis, is laid out at https://www.webbrowserbot.com/price-monitoring/
Deciding What the Data Triggers
The last stage is the one that should have been decided first: what happens when a price moves?
There are three honest answers. A person reads a dashboard and decides. A rules engine adjusts automatically within guardrails, match the lowest competitor, stay within five percent of market average, never breach a margin floor. Or a model incorporates elasticity estimates and demand forecasts and optimizes toward revenue or share.
Each one demands a different level of confidence from everything upstream. A human reviewing a dashboard can spot a bad match and ignore it. An automated repricer cannot, so a single mismatched product becomes a real price change on a real listing. If the plan is automated repricing, the matching bar and the freshness bar both rise sharply, and that has to be known before the collection layer is designed, not after.
The Takeaway
Price monitoring projects are usually scoped as data collection projects and usually fail as data quality projects. Collection is the visible engineering. Matching is the hidden engineering. The comparability of the number is a modeling decision, and the action stage sets the tolerance for everything before it.
If you are scoping one, decide what the data will trigger first, then work backwards. The requirements fall out of that answer, and you will scope the middle two stages properly instead of discovering them in week three.
Top comments (0)