Competitive pricing is a data pipeline problem before it is a pricing strategy

You get asked to build a feature that “matches competitor prices.” It sounds simple until the first dataset arrives. One competitor includes shipping, another shows a coupon only after login, one page says “from $19.99,” and your scraper quietly captured the price for a used item instead of the new one.

That is the part people usually skip when they talk about competitive pricing. The pricing rule is rarely the hard part. The hard part is collecting prices you can trust, normalizing them, and making sure automation does not start a race to the bottom.

Competitive pricing is not always undercutting

Competitive pricing means setting your price based on market prices, not only your internal cost model. That can mean a few different things:

• Match the market price to reduce buyer friction.
• Undercut by a small amount when price is the main decision factor.
• Price above competitors when your product has brand, service, bundle, or availability advantages.
• Sell one item cheaply and make money later, like the classic razor-and-blades model.

For developers, the business question usually turns into this:

Given our SKU, our cost, margin rules, stock status, and competitor offers,
return a recommended price and explain why.

That “explain why” matters. If a buyer or category manager asks why a product dropped from $49.99 to $38.20, “the script did it” is not a useful answer.

Collect prices you can defend

A basic competitor price collector might start like this:

pip install httpx beautifulsoup4

from decimal import Decimal
import re
import httpx
from bs4 import BeautifulSoup

TARGETS = [
    {
        "sku": "USB-C-CHARGER-65W",
        "competitor": "example-shop",
        "url": "https://example.com/products/65w-usb-c-charger",
        "price_selector": ".product-price",
    }
]

PRICE_RE = re.compile(r"(\d+[\d,]*(?:\.\d{2})?)")


def parse_price(text: str) -> Decimal:
    match = PRICE_RE.search(text.replace(",", ""))
    if not match:
        raise ValueError(f"price_not_found: {text[:80]!r}")
    return Decimal(match.group(1))


def fetch_price(target: dict) -> dict:
    headers = {"User-Agent": "PriceMonitor/1.0 [e-mail]"}

    with httpx.Client(timeout=10, follow_redirects=True, headers=headers) as client:
        response = client.get(target["url"])
        response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    node = soup.select_one(target["price_selector"])

    if node is None:
        raise ValueError(
            f"selector_not_found: {target['competitor']} {target['sku']}"
        )

    return {
        "sku": target["sku"],
        "competitor": target["competitor"],
        "url": target["url"],
        "price": parse_price(node.get_text(" ", strip=True)),
    }


for target in TARGETS:
    try:
        print(fetch_price(target))
    except httpx.HTTPStatusError as exc:
        print(f"http_error: {exc.response.status_code} {target['url']}")
    except Exception as exc:
        print(f"collection_error: {exc}")

This is enough to show the pattern, but it is not enough to run pricing for a real catalog.

Here is how it fails in production:

• A page starts rendering prices with JavaScript, so selector_not_found spikes.
• The site returns 403 Forbidden or a CAPTCHA page, and your parser extracts no price.
• The captured price is “from $19.99,” not the actual matching variant.
• The competitor shows a marketplace offer from a third-party seller with slow shipping.
• The product is a 2-pack, while your SKU is a single unit.
• The page shows a coupon, but only for logged-in users or specific regions.

If you do not store these failures explicitly, you will treat missing data as cheap data, which is worse than having no data. Wire fits this part of the workflow when competitor-price extraction needs structured fields, retries, and clear failure states instead of silent scraper drift.

Normalize before comparing

Do not compare raw page prices directly. Compare landed, equivalent prices.

A useful normalized record usually needs fields like these:

CREATE TABLE competitor_prices (
  sku TEXT NOT NULL,
  competitor TEXT NOT NULL,
  observed_at TEXT NOT NULL,
  item_price NUMERIC NOT NULL,
  shipping_price NUMERIC DEFAULT 0,
  currency TEXT NOT NULL,
  quantity INTEGER DEFAULT 1,
  in_stock BOOLEAN NOT NULL,
  product_condition TEXT NOT NULL,
  url TEXT NOT NULL,
  collection_status TEXT NOT NULL
);

Then compare on a derived value:

landed_unit_price = (item_price + shipping_price - instant_discount) / quantity

You may also need to exclude offers where product_condition != 'new', in_stock = false, or shipping time exceeds your threshold. Otherwise your repricer may match an out-of-stock listing or a used item.

This is also where product matching matters. Matching by title similarity alone will burn you. “iPhone 15 case” and “iPhone 15 Pro case” are not interchangeable. For important SKUs, store a manually approved competitor URL or a product identifier such as GTIN, UPC, EAN, or manufacturer part number.

Put guardrails around the pricing rule

Once the data is clean enough, the repricing logic can stay boring. That is a good thing.

from decimal import Decimal


def recommend_price(
    cost: Decimal,
    current_price: Decimal,
    competitor_price: Decimal,
    min_margin: Decimal,
    max_price: Decimal,
    undercut_by: Decimal = Decimal("0.01"),
) -> dict:
    floor = cost * (Decimal("1") + min_margin)
    candidate = competitor_price - undercut_by

    if candidate < floor:
        return {
            "price": current_price,
            "action": "hold",
            "reason": "competitor_below_margin_floor",
            "margin_floor": floor,
            "competitor_price": competitor_price,
        }

    return {
        "price": min(candidate, max_price),
        "action": "update",
        "reason": "undercut_competitor_within_margin",
        "margin_floor": floor,
        "competitor_price": competitor_price,
    }


print(
    recommend_price(
        cost=Decimal("30.00"),
        current_price=Decimal("49.99"),
        competitor_price=Decimal("42.00"),
        min_margin=Decimal("0.20"),
        max_price=Decimal("59.99"),
    )
)

The important bit is not the one-cent undercut. It is the hold case. If the competitor sells below your margin floor, your system should refuse to follow and record why.

You probably also want:

• A maximum daily price change, for example 5%.
• A minimum observation count before acting.
• Human approval for high-revenue SKUs.
• A cooldown period after each change.
• Alerts when many competitors suddenly disappear from the dataset.

Without those controls, a bad scrape can become a bad price.

The real strategy depends on your business model

Amazon-style undercutting works when scale, logistics, and supplier terms support it. Best Buy uses price matching to stop customers from checking out elsewhere. Apple can price above similar hardware because the ecosystem and brand carry value. Aldi undercuts by redesigning operations, not by randomly accepting lower margins.

The implementation should reflect that strategy. A premium brand may monitor competitors only to avoid being wildly out of range. A commodity seller may update prices daily. A marketplace seller may need near-real-time monitoring, but only for SKUs where inventory and margin justify it.

A practical first version is small: pick 20 important SKUs, collect competitor prices twice a day, store failures as first-class records, calculate landed unit prices, and require manual approval before changing anything. After two weeks, inspect the failures and bad matches before you automate the next step.