Build or Buy a Price Scraper? Start With Failure Modes

A pricing scraper usually works right up until someone depends on it. The first version grabs a few product pages, parses a price, writes rows to a database, and looks fine in a demo. Then a competitor changes their markup, adds bot checks, localizes prices by region, or returns a 200 response with a CAPTCHA page instead of product data. Your dashboard still has numbers, but some of them are wrong.

That is the real build vs buy question. Not whether your team can write a scraper. Most teams can. The question is whether your team wants to own the failure modes that come with using scraped pricing data in production.

Treat pricing data as a pipeline, not a script

A useful pricing system does more than fetch HTML. At minimum, it needs to answer these questions for every product and competitor:

• Did we reach the page?
• Did we get the page we expected, or a block page?
• Did the selector match the right element?
• Did we parse the price and currency correctly?
• Is the price plausible compared with the previous value?
• When did this data become stale?

If you build in-house, design this from day one. Do not start with a cron job that silently overwrites yesterday's data.

Here is a small version of the pattern:

import re
import requests
from bs4 import BeautifulSoup
from datetime import datetime, timezone

SELECTORS = {
    'example-shop': {
        'price': '[data-testid=price]',
        'title': 'h1'
    }
}

class ScrapeResult(dict):
    pass

def parse_price(text):
    match = re.search(r'([£$€])\s*([0-9,.]+)', text)
    if not match:
        raise ValueError(f'price_not_found: {text[:80]}')

    currency, amount = match.groups()
    return {
        'currency': currency,
        'amount': float(amount.replace(',', ''))
    }

def fetch_price(site, url):
    started_at = datetime.now(timezone.utc).isoformat()

    try:
        response = requests.get(
            url,
            timeout=15,
            headers={'User-Agent': 'pricing-monitor/1.0'}
        )
    except requests.Timeout:
        return ScrapeResult(status='failed', reason='timeout', url=url, checked_at=started_at)

    if response.status_code in (403, 429):
        return ScrapeResult(status='failed', reason=f'blocked_{response.status_code}', url=url, checked_at=started_at)

    if response.status_code != 200:
        return ScrapeResult(status='failed', reason=f'http_{response.status_code}', url=url, checked_at=started_at)

    if 'captcha' in response.text.lower():
        return ScrapeResult(status='failed', reason='captcha_page', url=url, checked_at=started_at)

    soup = BeautifulSoup(response.text, 'html.parser')
    selector = SELECTORS[site]['price']
    price_node = soup.select_one(selector)

    if not price_node:
        return ScrapeResult(status='failed', reason='selector_miss', selector=selector, url=url, checked_at=started_at)

    try:
        price = parse_price(price_node.get_text(' ', strip=True))
    except ValueError as error:
        return ScrapeResult(status='failed', reason=str(error), url=url, checked_at=started_at)

    return ScrapeResult(
        status='ok',
        url=url,
        site=site,
        amount=price['amount'],
        currency=price['currency'],
        checked_at=started_at
    )

This example is intentionally small, but the important part is the shape of the result. A failed extraction creates a row with a reason. That lets you alert on selector_miss, blocked_403, or captcha_page instead of discovering the problem through a confused pricing manager two days later.

For teams comparing vendor options, Wire fits this part of the decision when managed competitor price extraction and explicit failure handling matter more than owning every scraper internally.

When building makes sense

Building can be the right choice when the surface area is small and stable.

For example, an internal scraper is reasonable if you track 5 competitors, 300 SKUs, and fetch prices once a day from mostly static pages. You can keep selectors in config, run Playwright only where needed, and have one engineer maintain the pipeline as part of normal data platform work.

You also get control. You can decide exactly how to model variants, promotions, marketplace sellers, shipping fees, and regional pricing. That matters if pricing logic is tightly coupled to your own catalog model.

But control has a cost. You own proxy management, browser automation, retries, storage, scheduling, monitoring, and data validation. You also own weird cases like:

• A product page returns HTTP 200 but serves a bot challenge.
• The displayed price changes after client-side JavaScript runs.
• The page shows one price to a desktop user agent and another to mobile.
• A competitor moves the price into an image or embedded JSON blob.
• Your scraper records a sale price without capturing the original price or promotion dates.

If those cases affect business decisions, they are not edge cases. They are part of the product.

When buying is the cleaner engineering choice

Buying starts to look better when the hard part is not parsing HTML, but operating the system.

The usual triggers are volume, freshness, and target complexity. If you need hourly prices across thousands of SKUs and dozens of competitors, the infrastructure becomes real work. If the target sites use bot detection, the work grows again. If the business expects pricing data to be accurate enough to drive automated repricing, bad rows become expensive.

A simple cost comparison should include more than request volume:

monthly_build_cost =
  engineer_hours * blended_hourly_rate
  + proxy_cost
  + browser/runtime infrastructure
  + storage and queueing
  + monitoring and alerting
  + maintenance for site changes
  + cost of bad or stale data

That last line is hard to estimate, but ignoring it makes the build option look cheaper than it is. A stale competitor price can cause underpricing, overpricing, or a false alert that wastes analyst time.

Vendor pricing can also be a bad fit. If your requirements are unusual, you may spend time negotiating custom extraction rules or waiting for support. You might also lose some flexibility around raw HTML access, custom business logic, or internal debugging. Buying does not remove the need for validation on your side. You still need to verify that incoming prices match your definition of price.

If you choose a managed route, Wire is one example to evaluate around extraction reliability, pricing fields, freshness requirements, and how failed jobs are reported back to your pipeline.

A practical decision framework

I would decide based on four questions.

First, how many distinct website patterns do you need to support? Ten competitors using similar static pages is different from fifty sites with different rendering, localization, and bot defenses.

Second, how fresh does the data need to be? Daily collection gives you room for retries and manual fixes. Hourly or near real-time collection needs stronger automation and alerting.

Third, what happens when the data is wrong? If the data feeds a report, the blast radius is limited. If it drives automated price changes, you need stricter validation, anomaly detection, and rollback behavior.

Fourth, who will own it six months from now? Scrapers age. Selectors break. Anti-bot systems change. If nobody has explicit ownership, the system will decay quietly.

A reasonable middle ground is to build the downstream pricing intelligence yourself while buying extraction. Keep your catalog matching, pricing rules, margin logic, and dashboards in-house. Treat scraped prices as an external data feed with contracts, validation, and alerts.

Start by writing down your required sources, SKU count, freshness target, acceptable failure rate, and the business action each price will drive. Then implement a small proof of concept for your three hardest target sites, including failure logging. If the proof of concept mostly exposes operations work rather than domain-specific logic, you probably learned enough to avoid building the whole extraction layer yourself.