DEV Community: Anakin

Price matching is a bad default: model the pricing decision instead

Anakin — Wed, 23 Sep 2026 10:00:01 +0000

A common retail pricing bug is not a syntax error. It is a pricing service that does exactly what you told it to do: scrape a competitor price, set your price to match it, and slowly erase your margin because the competitor price was out of stock, coupon-gated, bundled, or part of a weekend promo.

Price matching is a control loop

If you build pricing software, it helps to think of price matching as a control loop:

Observe competitor prices
Normalize those observations
Decide whether they matter
Apply a price change
Measure what happened

Most broken implementations spend all their effort on step 1 and almost none on steps 2 through 5.

A naive implementation looks like this:

def match_lowest_price(our_sku, competitor_offers):
    lowest = min(offer["price"] for offer in competitor_offers)
    return lowest

That function is easy to explain and dangerous to run. It treats every competitor offer as equivalent. It ignores shipping, stock, coupons, minimum advertised price rules, product condition, seller reputation, and whether you can make money at the returned price.

The failure mode is predictable. One competitor drops a SKU from $299 to $249 for a clearance sale. Your system matches it. Another competitor matches you. Your system sees the market at $249 and keeps the new price. Nobody encoded the fact that the original signal was temporary.

Store observations, not just prices

The first fix is to stop storing a single price field as if it represents the market. A competitor offer is a bundle of facts.

A useful schema starts closer to this:

CREATE TABLE competitor_offer_observations (
    id BIGSERIAL PRIMARY KEY,
    observed_at TIMESTAMPTZ NOT NULL,
    competitor TEXT NOT NULL,
    competitor_sku TEXT NOT NULL,
    matched_internal_sku TEXT NOT NULL,
    shelf_price NUMERIC(10, 2),
    checkout_price NUMERIC(10, 2),
    shipping_price NUMERIC(10, 2),
    coupon_value NUMERIC(10, 2),
    currency CHAR(3) NOT NULL,
    in_stock BOOLEAN,
    delivery_days INT,
    seller_name TEXT,
    product_condition TEXT,
    promo_label TEXT,
    scrape_status TEXT NOT NULL,
    raw_payload JSONB NOT NULL
);

This gives your pricing code enough context to reject bad comparisons. A price from an out-of-stock item should usually not trigger a reaction. A marketplace seller with a two-week delivery estimate may not matter if your value proposition is next-day shipping. A coupon visible only after login may matter a lot, but you need to model it explicitly.

This is also where scraping reliability becomes part of pricing correctness, not just data plumbing. If your extractor misses coupons or silently fails on checkout pages, your pricing engine receives false market data. Wire fits this part of the workflow when you need extraction jobs that return price, availability, promotion details, and failure states instead of only a shelf-price field.

Add guardrails before automation

Before you reach for machine learning, write down the rules a human pricing manager would enforce. The rules do not need to be perfect. They need to prevent obvious damage.

Here is a simple Python example:

from dataclasses import dataclass
from decimal import Decimal
from typing import Optional

@dataclass
class Offer:
    competitor: str
    checkout_price: Decimal
    shipping_price: Decimal
    in_stock: bool
    delivery_days: int
    product_condition: str
    promo_label: Optional[str]

@dataclass
class ProductPolicy:
    current_price: Decimal
    unit_cost: Decimal
    min_margin_pct: Decimal
    max_single_change_pct: Decimal
    ignore_delivery_over_days: int


def landed_price(offer: Offer) -> Decimal:
    return offer.checkout_price + offer.shipping_price


def is_comparable(offer: Offer, policy: ProductPolicy) -> bool:
    if not offer.in_stock:
        return False
    if offer.product_condition.lower() != "new":
        return False
    if offer.delivery_days > policy.ignore_delivery_over_days:
        return False
    return True


def floor_price(policy: ProductPolicy) -> Decimal:
    return policy.unit_cost / (Decimal("1.0") - policy.min_margin_pct)


def recommended_price(policy: ProductPolicy, offers: list[Offer]) -> Decimal:
    comparable = [o for o in offers if is_comparable(o, policy)]

    if not comparable:
        return policy.current_price

    market_price = min(landed_price(o) for o in comparable)
    margin_floor = floor_price(policy)

    target = max(market_price, margin_floor)

    max_drop = policy.current_price * policy.max_single_change_pct
    lowest_allowed_today = policy.current_price - max_drop

    return max(target, lowest_allowed_today)

This still matches prices, but it no longer does so blindly. It refuses to compare against irrelevant offers. It protects margin. It limits daily price movement so one bad scrape cannot cut a product by 40%.

The tradeoff is that rules create false negatives. You will sometimes ignore a competitor move that a human would care about. That is better than automatically following every noisy signal. You can tune rules after you inspect misses.

Separate signal from noise

A useful pricing system should ask questions like these before changing a price:

Did the competitor price persist across multiple observations?
Is the product actually in stock?
Is the lower price caused by a coupon, bundle, loyalty discount, or financing offer?
Does the competitor regularly discount this category?
Will matching violate margin, MAP, or brand constraints?
Are we seeing the same movement across several competitors?

The persistence check alone catches many bad updates. For example:

SELECT matched_internal_sku,
       competitor,
       MIN(checkout_price + shipping_price) AS min_landed_price,
       COUNT(*) AS observations
FROM competitor_offer_observations
WHERE observed_at > now() - interval '6 hours'
  AND scrape_status = 'ok'
  AND in_stock = true
GROUP BY matched_internal_sku, competitor
HAVING COUNT(*) >= 3;

That query does not prove a price is meaningful, but it filters out one-off observations. You can then combine it with category-specific rules. Electronics may need faster reaction times than furniture. Private-label products may not need direct matching at all.

Checkout price is another edge many teams underestimate. The shelf price may be $299, but the customer sees $279 after an auto-applied coupon and $14.99 shipping at checkout. If your data model cannot represent those fields separately, your decision code has to guess. Wire is relevant here because checkout-level extraction and explicit job errors make it easier to tell the difference between “no discount exists” and “the scraper never reached the discount step.”

Treat automation as execution, not strategy

Once you have clean observations and explicit guardrails, automation becomes much safer. A model can estimate elasticity, predict competitor behavior, or choose among pricing strategies, but it should not be the first layer of defense.

In practice, I would log every recommendation before applying it:

{
  "sku": "TV-55-OLED-123",
  "current_price": "899.00",
  "recommended_price": "879.00",
  "reason_codes": [
    "competitor_landed_price_lower",
    "price_seen_3_times",
    "above_margin_floor",
    "within_daily_change_limit"
  ],
  "ignored_offers": [
    {
      "competitor": "example-shop",
      "reason": "out_of_stock"
    }
  ]
}

Those reason codes matter when someone asks why revenue dropped in a category, or why you did not match a visible competitor price. Without them, pricing automation becomes hard to debug.

A good next step is to take one high-volume SKU category and build the observation table, comparability rules, margin floor, and recommendation log for that category only. Run it in shadow mode for two weeks, compare its recommendations with human decisions, then decide which rules deserve automation.

Model food delivery pricing as data, not a single scraped number

Anakin — Tue, 22 Sep 2026 10:00:01 +0000

A common mistake in food delivery pricing analysis is scraping the checkout total and treating it as the price. That number is useful, but it hides the reason the price changed. Was it the restaurant markup, delivery fee, service fee, a promotion, subscription eligibility, surge pricing, or a different address? If you only store the final total, every downstream dashboard turns into guesswork.

Food delivery pricing is a stack. Your data model needs to preserve the stack.

The price is not one field

For the same basket, two platforms can differ by 20% at checkout while showing similar menu prices. The difference usually comes from layers that move independently:

Restaurant item prices and markups
Platform service fees
Delivery fees
Small order fees
Promotions and credits
Subscription benefits
Driver or demand incentives that affect availability and delivery estimates
Taxes and local fees

If your scraper or ingestion job flattens this into total_price, you lose the ability to answer basic questions later.

A better raw table stores both the observed total and each component you can identify:

create table delivery_quotes (
  quote_id text primary key,
  platform text not null,
  restaurant_id text not null,
  restaurant_name text not null,
  captured_at timestamptz not null,
  country text not null,
  city text not null,
  postal_code text not null,
  latitude numeric,
  longitude numeric,
  basket_hash text not null,
  subscriber boolean not null,

  item_subtotal_cents integer,
  menu_markup_cents integer,
  service_fee_cents integer,
  delivery_fee_cents integer,
  small_order_fee_cents integer,
  promo_discount_cents integer,
  tax_cents integer,
  tip_cents integer,
  total_cents integer not null,

  delivery_eta_min integer,
  available boolean not null,
  raw_payload jsonb not null
);

The raw_payload column matters. Delivery platforms change labels, reorder fee rows, and sometimes collapse fees into vague names like other fees. If you only store parsed fields, parser bugs become permanent data loss.

Normalize baskets before comparing platforms

Competitive pricing data only works when the basket stays stable. Comparing a burger meal on one platform with a different combo on another platform creates noise that looks like pricing strategy.

One practical approach is to define canonical baskets, then map platform-specific item IDs into them.

from dataclasses import dataclass

@dataclass(frozen=True)
class BasketItem:
    canonical_sku: str
    quantity: int

@dataclass(frozen=True)
class Basket:
    basket_id: str
    items: tuple[BasketItem, ...]

LUNCH_BASKET = Basket(
    basket_id="burger_lunch_v1",
    items=(
        BasketItem("cheeseburger", 1),
        BasketItem("fries_regular", 1),
        BasketItem("coke_500ml", 1),
    ),
)

PLATFORM_SKU_MAP = {
    "ubereats": {
        "cheeseburger": "item_82716",
        "fries_regular": "item_19302",
        "coke_500ml": "item_77291",
    },
    "doordash": {
        "cheeseburger": "sku_cb_001",
        "fries_regular": "sku_fr_003",
        "coke_500ml": "sku_dr_014",
    },
}

def build_platform_cart(platform: str, basket: Basket) -> list[dict]:
    sku_map = PLATFORM_SKU_MAP[platform]
    return [
        {"platform_item_id": sku_map[item.canonical_sku], "quantity": item.quantity}
        for item in basket.items
    ]

This mapping looks boring, but it is where a lot of pricing analysis succeeds or fails. You need versioned baskets because menus change. If a restaurant replaces coke_500ml with coke_330ml, you should create burger_lunch_v2, not silently mutate historical comparisons.

Geography needs to be below city level

City-level averages hide the interesting part. Delivery fees and promotions often vary by postal code, neighborhood, traffic corridor, or even address. A platform can look competitive across New York while being consistently expensive in a high-value set of Manhattan postal codes.

For data collection, treat location as part of the primary key. These two observations are not duplicates:

platform=ubereats restaurant_id=abc basket=burger_lunch_v1 postal_code=10001 captured_at=2026-08-18T18:00Z
platform=ubereats restaurant_id=abc basket=burger_lunch_v1 postal_code=10011 captured_at=2026-08-18T18:00Z

They may return different delivery fees, ETAs, availability, and promotions.

This is also where scraping jobs become fragile. You need stable address fixtures, session handling, and checks that the platform actually priced the quote for the intended location. Tools like Wire sit in this part of the stack: geo-aware extraction, promotion capture, and failure handling for delivery platforms whose pages change often.

Promotions expire faster than weekly reporting

If you collect data once per week, you are not monitoring promotions. You are sampling history.

Flash discounts and cuisine-specific offers can appear for a few hours during peak demand. If a competitor runs 30% off sushi from 6pm to 9pm on Friday, a Monday morning scrape will miss the entire event.

For promotion monitoring, store every observed offer separately from the quote:

create table delivery_promotions (
  promo_observation_id text primary key,
  platform text not null,
  restaurant_id text,
  postal_code text not null,
  captured_at timestamptz not null,
  promo_type text not null,
  description text not null,
  discount_percent numeric,
  discount_cents integer,
  minimum_order_cents integer,
  funded_by text,
  expires_at timestamptz,
  raw_payload jsonb not null
);

Then query by demand window, not calendar week:

select
  platform,
  postal_code,
  count(*) as promo_count,
  avg(discount_percent) as avg_discount_percent
from delivery_promotions
where captured_at >= '2026-08-14 17:00:00+00'
  and captured_at <  '2026-08-14 22:00:00+00'
  and promo_type = 'percentage_discount'
group by platform, postal_code;

The failure mode here is quiet. Nothing crashes. Your dashboard just says there were no promotions because your crawler slept through them.

Segment subscriber and non-subscriber prices

Subscriptions change the meaning of price. A non-subscriber may see a $3.99 delivery fee while a subscriber sees zero. If you average those together, you get a number nobody actually paid.

Collect both views when possible:

Logged out or non-subscriber session
Subscriber session
Same basket
Same restaurant
Same address
Same time window

Store subscriber as a dimension, not as metadata. Analysts will need to filter on it constantly.

This also changes promotion analysis. Free delivery promotions barely matter to users who already get free delivery, but percentage discounts still do. If your model does not segment subscriptions, you can overestimate or underestimate the effect of a competitor promotion.

Build checks for bad quotes

Delivery pricing data has plenty of bad states. Some are obvious, like HTTP 403s. Others look valid unless you check them.

Examples:

Restaurant is closed, but the page still returns menu prices
Address geocodes to the wrong city
Basket item is unavailable and the cart silently drops it
Promotion requires a minimum order your basket does not meet
Currency changes because the platform redirects to another market
Subscriber session expires and starts returning non-subscriber fees

Add validation before writing analytics tables:

def validate_quote(q: dict) -> list[str]:
    errors = []

    if not q["available"]:
        errors.append("restaurant_unavailable")

    if q["currency"] != q["expected_currency"]:
        errors.append("currency_mismatch")

    if q["requested_postal_code"] != q["priced_postal_code"]:
        errors.append("location_mismatch")

    if q["cart_item_count"] != q["expected_item_count"]:
        errors.append("basket_incomplete")

    component_sum = sum(q.get(k, 0) or 0 for k in [
        "item_subtotal_cents",
        "service_fee_cents",
        "delivery_fee_cents",
        "small_order_fee_cents",
        "tax_cents",
    ]) - (q.get("promo_discount_cents", 0) or 0)

    if abs(component_sum - q["total_cents"]) > 2:
        errors.append("total_does_not_match_components")

    return errors

Keep invalid observations, but mark them. They are useful for debugging collection quality, not for pricing decisions. Wire is relevant here because reliable delivery extraction is less about fetching pages and more about detecting these bad quote states before they pollute the dataset.

The useful benchmark is explainable

A checkout total tells you who is cheaper. A layered quote tells you why.

That difference matters when someone asks what to do next. Lowering delivery fees, funding a restaurant promo, changing subscription benefits, and adjusting commission terms are different actions. They should not come from the same flattened metric.

If you are building this internally, start with one market, three postal codes, two baskets, and two collection windows per day. Store raw payloads, parsed fee components, subscriber state, and validation errors from the beginning. It is much easier to widen a clean dataset than to explain six months of totals that cannot be decomposed.

Mobile Proxies for Pricing Scrapers: Where They Help and Where They Hurt

Anakin — Mon, 21 Sep 2026 10:00:01 +0000

Your pricing scraper works fine for a week, then the numbers stop making sense. Some requests return 403. Some return a CAPTCHA page that your parser happily treats as an empty product. Some prices are valid, but for the wrong country. That last one is the dangerous failure: the job succeeds, the data is wrong, and nobody notices until a repricing rule acts on it.

Mobile proxies can help with pricing intelligence, but only in specific cases. They are not a magic fix for scraping. They are one tool for getting closer to how real mobile users see a site, especially when prices, stock, or promotions vary by region or device.

What mobile proxies actually change

A mobile proxy routes traffic through IPs assigned by mobile carriers, usually 4G or 5G networks. Target sites often treat those IPs differently from data center IPs because they look like normal consumer traffic.

That matters when a retailer varies content by location, carrier region, or mobile experience. It also matters when data center IP ranges get blocked quickly.

The important part is not just the proxy type. You still need consistent headers, cookies, pacing, retries, and validation. A request from a mobile IP with a desktop Chrome user agent, no cookies, and 200 concurrent requests to the same domain will still look strange.

Use mobile proxies when you have a reason:

You need country, region, or city-specific prices.
The site shows different prices or promos to mobile users.
Data center proxies hit 403, 429, or CAPTCHA too often.
You need to monitor fast-changing inventory without burning one IP range.

Do not use them by default. They cost more, latency varies more, and geo targeting can be less exact than people expect.

Validate the region before trusting the price

The easiest mistake is assuming the proxy location matches the price location. Mobile carrier IPs can geolocate to the carrier gateway instead of the user's actual city. If your scraper says it fetched a price from Madrid, but the site sees the request as Barcelona or even another country, your pricing data is contaminated.

Start every new proxy pool with a geolocation check:

curl -x http://USER:PASS@proxy.example:8000 https://ipinfo.io/json

Then compare that with what the target site thinks. Some sites expose region in page HTML, cookies, shipping selectors, or API responses. Log it.

For example, your product parser should not only emit this:

{
  "sku": "ABC-123",
  "price": 49.99,
  "currency": "EUR"
}

It should emit enough context to audit the result:

{
  "sku": "ABC-123",
  "price": 49.99,
  "currency": "EUR",
  "requested_region": "ES-MD",
  "detected_region": "ES-CT",
  "proxy_asn": "mobile-carrier-asn",
  "status_code": 200,
  "content_type": "text/html"
}

That mismatch should fail the record or at least mark it as suspect. A clean HTTP 200 does not mean the data is usable.

For teams that do not want to own the proxy rotation and regional extraction layer, Wire fits this part of the pricing workflow because it treats geo-aware extraction and failed collection attempts as job-level concerns rather than hidden network details.

Pace by domain, not just globally

Most scraper failures come from a bad traffic shape. A global limit like 100 requests per minute sounds safe until all 100 hit the same retailer in five seconds.

Use per-domain pacing and treat 403, 429, and CAPTCHA responses as signals. Here is a small Python example that shows the pattern. It is intentionally simple, but the structure matters.

import time
import random
import requests
from urllib.parse import urlparse

PROXY_URL = "http://USER:PASS@proxy.example:8000"
PROXIES = {
    "http": PROXY_URL,
    "https": PROXY_URL,
}

HEADERS = {
    "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1",
    "Accept-Language": "en-US,en;q=0.9",
}

last_request_at = {}
failure_count = {}

MIN_DELAY_SECONDS = {
    "example-retailer.com": 3.0,
    "default": 1.0,
}

BLOCK_STATUSES = {403, 429}


def wait_for_domain(url):
    domain = urlparse(url).netloc
    delay = MIN_DELAY_SECONDS.get(domain, MIN_DELAY_SECONDS["default"])
    elapsed = time.time() - last_request_at.get(domain, 0)

    if elapsed < delay:
        time.sleep(delay - elapsed)

    last_request_at[domain] = time.time()


def looks_like_captcha(response):
    text = response.text[:2000].lower()
    return "captcha" in text or "verify you are human" in text


def fetch_price_page(url, timeout=20):
    domain = urlparse(url).netloc

    if failure_count.get(domain, 0) >= 5:
        raise RuntimeError(f"circuit open for {domain}")

    for attempt in range(3):
        wait_for_domain(url)

        response = requests.get(
            url,
            headers=HEADERS,
            proxies=PROXIES,
            timeout=timeout,
        )

        if response.status_code in BLOCK_STATUSES or looks_like_captcha(response):
            failure_count[domain] = failure_count.get(domain, 0) + 1
            sleep_for = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(sleep_for)
            continue

        response.raise_for_status()
        failure_count[domain] = 0
        return response.text

    raise RuntimeError(f"failed after retries: {url}")

This does a few useful things:

It throttles per target domain.
It retries only on likely block conditions.
It opens a crude circuit breaker after repeated failures.
It avoids treating CAPTCHA HTML as a valid product page.

In production, you would also rotate sessions, persist cookies per region, track proxy ASN, and emit metrics for every request.

Track cost per successful page, not proxy price

Mobile proxies usually cost more than data center or residential proxies. That does not automatically make them too expensive. The metric that matters is cost per usable record.

If a data center proxy costs less but only returns valid pages 40% of the time, and a mobile proxy returns valid pages 92% of the time for a specific target, the mobile route may still be cheaper per collected SKU.

Track these numbers per domain and region:

Success rate for valid product pages, not just HTTP 200.
Block rate by status code and CAPTCHA detection.
Median and p95 latency.
Region mismatch rate.
Cost per successful page load.
Cost per unique SKU collected.

This is also where managed extraction can be easier to evaluate: with Wire, the useful comparison is whether each pricing job returns auditable results, retryable failures, and enough metadata to calculate cost per valid SKU.

The tradeoffs to expect

Mobile proxies have rough edges.

Geo accuracy can drift. Validate against target-site evidence, not only an IP database.

IP reputation can change quickly because many users may share one carrier IP through carrier-grade NAT. One bad burst from someone else can affect you.

Latency is less predictable than fixed networks. Set realistic timeouts and retry budgets.

Scaling too fast creates its own block pattern. Increase concurrency gradually and watch failure rates before adding more regions or threads.

Also, collect only data you are allowed to collect. Follow applicable law, site terms, and internal review processes. Pricing data pipelines often become business-critical, so compliance mistakes stick around longer than prototype code.

A practical starting point

Before switching a whole scraper fleet to mobile proxies, run a controlled test on one retailer, one region, and one product category. Log requested region, detected region, status code, latency, block reason, proxy ASN, and parser result. Compare the output against manually verified prices.

If the mobile proxy improves valid page rate or regional correctness enough to justify the cost, expand slowly. If it only hides missing retries, bad parsing, or weak observability, fix those first.

Implementing price matching without turning it into a pricing bug

Anakin — Fri, 18 Sep 2026 10:02:43 +0000

Price matching sounds like a simple business rule until someone asks you to implement it. Then you find out that cheaper competitor prices can be stale, out of stock, for a different SKU, locked behind coupons, or low enough to make your own order unprofitable.

If you treat price matching as if competitor.price < our.price then match, you will ship a margin bug.

Model the policy before writing the scraper

A price match policy is not just a discount. It is an eligibility decision.

At minimum, you need to answer these questions:

Is this the same product?
Is the competitor eligible?
Is the competitor offer currently available?
Does the price include shipping, tax, coupons, bundles, or membership discounts?
Is the matched price still above your margin floor?
Is this before purchase, at checkout, or after purchase as a refund?

Those details should live in data and code, not in customer support notes.

A basic schema might look like this:

create table competitor_offers (
  id bigserial primary key,
  product_id bigint not null,
  competitor text not null,
  competitor_url text not null,
  price_cents integer not null,
  currency char(3) not null,
  in_stock boolean not null,
  observed_at timestamptz not null,
  evidence jsonb not null default '{}'
);

create index competitor_offers_lookup
on competitor_offers (product_id, competitor, observed_at desc);

The evidence column is useful. Store things like page title, detected SKU, shipping text, and screenshot URL if you capture one. When a customer asks why a match was rejected, you do not want to reverse-engineer yesterday's page state from logs.

Treat competitor prices as untrusted input

Competitor price data usually comes from scraping, feeds, marketplace APIs, or customer-submitted URLs. All of those fail in boring ways.

A scraper may read the wrong DOM node after a layout change. A marketplace listing may show a used item next to a new one. A customer may submit a regional price you cannot verify from your server. A competitor page may return 200 OK while rendering an out-of-stock message in client-side JavaScript.

The symptom is not always an exception. Sometimes the job succeeds and gives you bad data.

That is why the extraction layer should produce structured records with confidence signals, not just a number. If you do not want to maintain per-retailer extraction, retries, and price evidence yourself, Wire is one option for turning competitor product pages into structured price records with failure handling.

Even if you build the whole pipeline yourself, keep the matching decision separate from the extraction decision. A scraped price should not automatically become a customer discount.

Encode rejection reasons explicitly

Here is a small TypeScript version of the core rule. It returns a decision with reasons instead of throwing or silently declining.

type Offer = {
  productId: number;
  competitor: string;
  priceCents: number;
  currency: 'USD' | 'EUR' | 'GBP';
  inStock: boolean;
  observedAt: Date;
};

type Product = {
  id: number;
  currentPriceCents: number;
  costCents: number;
  currency: 'USD' | 'EUR' | 'GBP';
};

type Policy = {
  eligibleCompetitors: Set<string>;
  maxOfferAgeMinutes: number;
  minMarginPercent: number;
};

type Decision =
  | { eligible: true; matchedPriceCents: number; marginPercent: number }
  | { eligible: false; reasons: string[] };

function evaluatePriceMatch(
  product: Product,
  offer: Offer,
  policy: Policy,
  now = new Date()
): Decision {
  const reasons: string[] = [];

  if (offer.productId !== product.id) reasons.push('different_product');
  if (!policy.eligibleCompetitors.has(offer.competitor)) reasons.push('competitor_not_eligible');
  if (offer.currency !== product.currency) reasons.push('currency_mismatch');
  if (!offer.inStock) reasons.push('competitor_out_of_stock');

  const ageMinutes = (now.getTime() - offer.observedAt.getTime()) / 60_000;
  if (ageMinutes > policy.maxOfferAgeMinutes) reasons.push('offer_too_old');

  if (offer.priceCents >= product.currentPriceCents) reasons.push('not_cheaper');

  const marginPercent = ((offer.priceCents - product.costCents) / offer.priceCents) * 100;
  if (marginPercent < policy.minMarginPercent) reasons.push('below_margin_floor');

  if (reasons.length > 0) {
    return { eligible: false, reasons };
  }

  return {
    eligible: true,
    matchedPriceCents: offer.priceCents,
    marginPercent: Math.round(marginPercent * 100) / 100
  };
}

This makes failures visible:

{
  "eligible": false,
  "reasons": ["offer_too_old", "below_margin_floor"]
}

That response is much easier to debug than a generic PRICE_MATCH_DENIED error.

Post-purchase matching is a different workflow

Pre-purchase price matching changes the cart price. Post-purchase matching creates a refund or credit. Do not implement both with the same code path.

For post-purchase requests, add constraints like:

purchase must be within 7 or 30 days
refund cannot exceed amount paid
one price adjustment per order line
original promotions must be included in the comparison
refund must be idempotent

The idempotency part matters. If a customer retries a request after a timeout, your API should not issue two refunds.

create unique index one_price_adjustment_per_order_line
on price_adjustments (order_line_id)
where reason = 'price_match';

You can still return a friendly message to the customer, but the database should enforce the business rule.

Avoid matching every price movement

The dangerous version of price matching is reacting to every competitor discount. Flash sales, clearance items, regional tests, and marketplace seller mistakes can all pull your price down for no good reason.

A safer system lets pricing teams configure boundaries:

match only selected competitors
ignore offers older than a set threshold
exclude marketplace sellers below a rating threshold
require the competitor item to be in stock
preserve a minimum gross margin
exclude categories where brand positioning matters more than price

Those rules are not just business preferences. They are production safeguards.

A practical next step

If you are building price matching, start by logging decisions without applying discounts. Run the evaluator in shadow mode for a week, store the accepted and rejected matches, and review the rejection reasons with support and pricing teams.

You will find bad product mappings, stale competitor offers, and margin edge cases before customers do.

Grocery price intelligence is mostly a data modeling problem

Anakin — Thu, 17 Sep 2026 10:00:02 +0000

The hard part of grocery price intelligence is not comparing $3.99 with $4.09. It is knowing whether those prices refer to the same product, in the same location, at the same time, under the same promotion rules. If you get any of those wrong, your pricing report looks precise but tells the business the wrong thing.

Grocery makes this worse than many other categories. Products expire. Prices vary by ZIP code. Delivery apps can change prices several times a day. A competitor may not discount the exact SKU you track, but they may discount a substitute that steals the same basket.

Here is the shape of a system I would build before trying to run any pricing model on top of it.

Store price observations, not just products

A common mistake is to model one row per product and keep updating the price. That loses the history you need for freshness, forecasting, and debugging.

Treat each scrape or feed record as an observation:

CREATE TABLE price_observations (
  id BIGSERIAL PRIMARY KEY,
  retailer TEXT NOT NULL,
  channel TEXT NOT NULL,          -- store, website, delivery_app
  location_id TEXT NOT NULL,      -- store id, ZIP, delivery zone, or geo hash
  observed_at TIMESTAMPTZ NOT NULL,

  external_sku TEXT,
  product_name TEXT NOT NULL,
  brand TEXT,
  size_text TEXT,                 -- 16 oz, 2 lb, 12 count
  category_path TEXT[],

  shelf_price NUMERIC(10, 2),
  promo_text TEXT,                -- 2 for $5, buy one get one, member price
  currency CHAR(3) DEFAULT 'USD',

  raw_payload JSONB NOT NULL
);

CREATE INDEX ON price_observations (retailer, location_id, observed_at DESC);
CREATE INDEX ON price_observations (external_sku, observed_at DESC);

Keep raw_payload. You will need it when a parser bug turns 12 count eggs into 12 oz eggs, or when a site changes promo wording and your downstream numbers jump for no business reason.

Normalize units before comparing prices

Unit normalization sounds boring until someone compares a 32 oz yogurt tub with a 5.3 oz single cup and reports a huge price gap. Store the shelf price, but compare on a normalized unit price where possible.

Here is a small version of the pattern:

from decimal import Decimal
import re

UNIT_TO_BASE = {
    'oz': ('oz', Decimal('1')),
    'lb': ('oz', Decimal('16')),
    'g': ('oz', Decimal('0.035274')),
    'kg': ('oz', Decimal('35.274')),
    'count': ('count', Decimal('1')),
}

SIZE_RE = re.compile(r'(?P<qty>\d+(?:\.\d+)?)\s*(?P<unit>oz|lb|g|kg|count|ct)', re.I)
PROMO_RE = re.compile(r'(?P<n>\d+)\s*for\s*\$(?P<price>\d+(?:\.\d+)?)', re.I)

def normalize_unit(unit: str) -> str:
    unit = unit.lower()
    return 'count' if unit == 'ct' else unit

def effective_price(shelf_price: Decimal, promo_text: str | None) -> Decimal:
    if not promo_text:
        return shelf_price

    match = PROMO_RE.search(promo_text)
    if match:
        return Decimal(match.group('price')) / Decimal(match.group('n'))

    return shelf_price

def unit_price(shelf_price: str, size_text: str, promo_text: str | None = None):
    match = SIZE_RE.search(size_text or '')
    if not match:
        return None

    qty = Decimal(match.group('qty'))
    unit = normalize_unit(match.group('unit'))
    base_unit, multiplier = UNIT_TO_BASE[unit]

    total_base_qty = qty * multiplier
    price = effective_price(Decimal(shelf_price), promo_text)
    return {
        'unit_price': price / total_base_qty,
        'base_unit': base_unit,
        'effective_price': price,
    }

print(unit_price('3.99', '32 oz'))
print(unit_price('5.00', '16 oz', '2 for $5'))

This will still fail on real grocery data. Bananas may be priced per pound online but sold per each in a delivery app. Meat may have estimated weights. Multipacks may say 6 x 4 oz. The important part is to make failures explicit. Return None, store a parse error, and exclude that row from automated comparisons until you fix the parser.

Location and freshness are part of the key

Two stores ten miles apart can have different competitors, customer behavior, and delivery fees. If your data model only has retailer and sku, you cannot answer whether a price is relevant to a specific customer.

Add freshness rules per channel. A weekly store scrape might be acceptable for shelf labels. For quick commerce, a six-hour-old price can already be stale.

SELECT *
FROM price_observations
WHERE retailer = 'example_market'
  AND location_id = '94107'
  AND observed_at > now() - interval '6 hours'
ORDER BY observed_at DESC;

When the extractor layer cannot consistently capture location-specific prices, every later stage inherits that uncertainty. In this context, Wire fits as the extraction layer: it returns grocery product, price, promotion, and location data that downstream pricing code can treat as inputs.

You should still track extraction quality yourself. At minimum, log coverage by category, freshness by location, parse error rate, and the share of rows with usable unit prices.

Match substitutes, not only exact SKUs

Exact SKU matching is useful, but grocery shoppers often compare needs rather than labels. If basmati rice gets expensive, jasmine rice may become the alternative. If a private-label cereal sits too close to the national brand price, customers may stop seeing it as a value option.

A practical approach is to split matching into tiers:

Tier 1: same UPC or retailer SKU
Tier 2: same brand, product type, size range
Tier 3: same product type, similar size, different brand
Tier 4: same need state, such as premium rice or kids cereal

Do not let an embedding model freely decide all matches. Use it to propose candidates, then apply hard filters like category, unit type, dietary attributes, and pack size range. Otherwise, you will eventually compare almond milk with almond butter because the names share tokens.

For pricing decisions, keep the match tier with the comparison. A 5 percent gap on exact UPCs means something different from a 5 percent gap between substitutes.

Measure the pipeline before trusting recommendations

Before you build forecasting or automated pricing rules, measure the data pipeline like a production system:

Freshness: percent of observations inside the allowed time window per channel
Coverage: tracked SKUs found per retailer, category, and location
Promo parse rate: percent of promo rows converted into effective prices
Unit-price availability: percent of rows with comparable base units
Match precision: sampled accuracy by match tier
Volatility: price changes per SKU per day, split by retailer and channel

A simple pricing model can tolerate missing data if it knows what is missing. It cannot tolerate silent bad comparisons.

Start with one category, one city, and three competitors. Build the observation table, normalize unit prices, tag stale records, and manually review substitute matches. Once those numbers look sane, add forecasting or automated recommendations on top.

Dynamic Pricing Without Creating a Pricing Incident

Anakin — Wed, 16 Sep 2026 10:00:01 +0000

Static prices are easy to reason about until the market moves faster than your admin panel. A competitor starts a flash sale, your inventory drops to 12 units, or a slow-moving SKU sits untouched for two weeks. If a human has to notice the change, decide what to do, and update prices manually, you are already late.

Dynamic pricing is just automated decision-making around price. The hard part is not changing the number. The hard part is changing it without violating margin rules, confusing customers, or teaching your system to chase bad data.

Start with constraints, not the algorithm

A common mistake is to start with “AI pricing” or a competitor-matching rule before defining what the system must never do.

For most e-commerce systems, those constraints look like this:

Never sell below minimum margin
Never exceed a maximum allowed price
Do not change the same SKU more than N times per day
Do not react to competitor data older than a set threshold
Do not move price by more than a fixed percentage per update
Respect MAP or channel-specific pricing rules where they apply

These constraints matter because pricing bugs are visible immediately. If your recommendation service returns a bad product sort order, users may not notice. If your repricer drops a $900 item to $90, they will.

Here is a simple rule-based repricer in TypeScript. It is not fancy, but it shows the shape of a safer implementation.

type PricingInput = {
  currentPrice: number;
  unitCost: number;
  inventory: number;
  salesLast24h: number;
  competitorPrice?: number;
  competitorSeenAt?: Date;
  now: Date;
};

type PricingConfig = {
  minMarginPct: number;
  maxPrice: number;
  maxMovePct: number;
  competitorFreshnessMinutes: number;
  lowInventoryThreshold: number;
  targetUndercutPct: number;
};

function clamp(value: number, min: number, max: number) {
  return Math.max(min, Math.min(max, value));
}

function priceSku(input: PricingInput, config: PricingConfig) {
  const minPrice = input.unitCost * (1 + config.minMarginPct / 100);
  let candidate = input.currentPrice;

  const competitorIsFresh =
    input.competitorPrice !== undefined &&
    input.competitorSeenAt !== undefined &&
    input.now.getTime() - input.competitorSeenAt.getTime() <=
      config.competitorFreshnessMinutes * 60_000;

  if (competitorIsFresh) {
    candidate = input.competitorPrice! * (1 - config.targetUndercutPct / 100);
  }

  if (input.inventory <= config.lowInventoryThreshold && input.salesLast24h > 0) {
    candidate *= 1.05;
  }

  if (input.salesLast24h === 0 && input.inventory > config.lowInventoryThreshold) {
    candidate *= 0.97;
  }

  const maxMove = input.currentPrice * (config.maxMovePct / 100);
  const boundedByMove = clamp(
    candidate,
    input.currentPrice - maxMove,
    input.currentPrice + maxMove
  );

  const finalPrice = clamp(boundedByMove, minPrice, config.maxPrice);

  return Number(finalPrice.toFixed(2));
}

This kind of function can run in a batch job every hour or respond to events such as inventory changes. The important part is that every suggested price passes through hard business limits before it reaches the catalog.

Competitor data is useful, but it fails in boring ways

Competitor-based pricing sounds straightforward: collect competitor prices, compare them to yours, then adjust. In practice, the failures are mostly data quality problems.

You will see cases like:

The competitor page returns HTTP 429 and your scraper stores the previous price as if it were current
A marketplace shows a coupon-adjusted price in one location but not another
The product match is wrong, so your 256 GB phone gets compared with the 128 GB model
Availability is missing, so you match a price from an out-of-stock listing
Shipping is excluded, even though it changes the effective price

That is why freshness and provenance should be part of the pricing input. A price without seenAt, source, availability, and product match confidence should not carry the same weight as a verified current price.

If you need competitor prices as an input rather than building scrapers for every marketplace, Wire is one way to treat product price, availability, and promotion data as a feed your repricer can consume.

Even with good data, avoid blindly matching the lowest price. The lowest visible price may come from a seller with slow delivery, limited stock, or a temporary promotion. Your repricer should know when to ignore a competitor.

Prevent price flapping

Price flapping happens when a SKU changes too often because the inputs bounce around. For example, you undercut a competitor by 2 percent, they undercut you again, and both systems keep walking down until one hits a floor.

You can reduce this with a few practical controls:

CREATE TABLE price_changes (
  id BIGSERIAL PRIMARY KEY,
  sku TEXT NOT NULL,
  old_price NUMERIC(10, 2) NOT NULL,
  new_price NUMERIC(10, 2) NOT NULL,
  reason TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX price_changes_sku_created_at_idx
  ON price_changes (sku, created_at DESC);

Before applying a new price, check recent changes:

async function canChangePrice(db: any, sku: string, maxChangesPerDay: number) {
  const result = await db.query(
    `SELECT count(*)::int AS count
     FROM price_changes
     WHERE sku = $1
       AND created_at > now() - interval '24 hours'`,
    [sku]
  );

  return result.rows[0].count < maxChangesPerDay;
}

This audit table also gives support and merchandising teams an answer when someone asks, “Why did this price change?” Store the rule name or model version in reason. Without that, debugging pricing behavior becomes guesswork.

Rule-based first, ML later

Machine learning can help with demand forecasting, elasticity estimates, and segment-level pricing. But many teams should start with rules because rules are easier to inspect.

A good first version might include:

Competitor price matching for known top SKUs
Inventory-based increases when stock is low
Discounts for aging inventory
Margin floors and maximum movement caps
Manual approval for high-value products

Once you have historical data, you can test whether a model improves outcomes. Do not just measure revenue. Track gross margin, conversion rate, units sold, refund rate, customer complaints, and price change frequency. A pricing model that increases revenue while crushing margin did not improve the business.

Roll it out like a risky production change

Dynamic pricing should ship behind controls. Start with a small SKU set. Run the repricer in shadow mode first, where it writes recommended prices but does not publish them. Compare recommendations against human decisions and existing rules.

For each recommendation, log:

Input prices and timestamps
Inventory level
Sales velocity
Rule or model version
Old price and recommended price
Final published price
Any constraint that modified the recommendation

Then review the outliers. The most useful test cases often come from rejected recommendations, not successful ones.

A practical next step: pick 20 SKUs, implement a rule-based repricer in shadow mode, and log every recommendation for two weeks before allowing it to publish prices automatically.

Dynamic Pricing Rules That Do Not Accidentally Burn Your Margins

Anakin — Tue, 15 Sep 2026 10:00:02 +0000

Dynamic pricing usually fails in boring ways. A competitor price is stale, inventory sync lags by ten minutes, or a rule lowers a product below contribution margin because nobody encoded the actual floor. The hard part is not changing prices. The hard part is deciding when a price change is safe.

Treat pricing as a decision system, not a cron job

A naive implementation often starts like this:

if (competitorPrice < myPrice) {
  myPrice = competitorPrice - 1;
}

That works until the competitor runs a loss-leading flash sale, their page parser reads the wrong variant, or your own inventory is almost gone and you should be raising price instead of chasing them down.

A better model separates inputs, rules, and guardrails:

Inputs: current price, cost, stock, recent sales, competitor prices, timestamps
Rules: when to increase or decrease price
Guardrails: absolute constraints that no rule can bypass
Observability: why the system chose the new price

For example, a gaming console might use demand-based pricing with competitor awareness:

Increase price by 12% if more than 50 units sold in 24 hours
Decrease price by 7% if fewer than 15 units sold in 24 hours
Do not go below $329
Do not go above $479
Ignore competitor prices older than 30 minutes
Do not change price more than once every 2 hours

Those last two rules matter more than they look. Without them, prices can oscillate all day or react to bad data.

Model the data you need before writing the rule

You need more than sku and price. Store enough context to explain the decision later.

create table price_observations (
  id bigserial primary key,
  sku text not null,
  source text not null,
  observed_price numeric(10,2) not null,
  observed_at timestamptz not null,
  metadata jsonb not null default '{}'
);

create table pricing_decisions (
  id bigserial primary key,
  sku text not null,
  old_price numeric(10,2) not null,
  new_price numeric(10,2) not null,
  reason text not null,
  inputs jsonb not null,
  decided_at timestamptz not null default now()
);

The price_observations table lets you distinguish between “Amazon is cheaper” and “our scraper last saw Amazon cheaper 4 hours ago.” Those are different facts.

Competitor extraction is also where many pricing systems get noisy: variant pages change, shipping gets included inconsistently, and blocked requests can look like missing products if you do not model failure separately. Wire is useful in this part of the pipeline when competitor prices need to be collected as explicit observations with inspectable failures instead of treated as always-fresh inputs.

Encode guardrails as code, not documentation

Here is a small TypeScript example. It is intentionally plain: no ML, no personalization, no magic. Just deterministic pricing with traceable reasons.

type PricingInput = {
  sku: string;
  currentPrice: number;
  unitCost: number;
  stock: number;
  unitsSold24h: number;
  competitorPrice?: number;
  competitorObservedAt?: Date;
  lastPriceChangeAt?: Date;
  now: Date;
};

type PricingDecision = {
  newPrice: number;
  reason: string;
  blocked?: string;
};

const MIN_MARGIN = 0.12;
const PRICE_FLOOR = 329;
const PRICE_CEILING = 479;
const MAX_COMPETITOR_AGE_MINUTES = 30;
const MIN_CHANGE_INTERVAL_MINUTES = 120;

function minutesBetween(a: Date, b: Date) {
  return Math.abs(a.getTime() - b.getTime()) / 60_000;
}

function roundPrice(price: number) {
  return Math.round(price * 100) / 100;
}

function clamp(price: number, min: number, max: number) {
  return Math.min(Math.max(price, min), max);
}

export function decidePrice(input: PricingInput): PricingDecision {
  if (
    input.lastPriceChangeAt &&
    minutesBetween(input.now, input.lastPriceChangeAt) < MIN_CHANGE_INTERVAL_MINUTES
  ) {
    return {
      newPrice: input.currentPrice,
      reason: "no_change",
      blocked: "price_changed_too_recently"
    };
  }

  const marginFloor = input.unitCost * (1 + MIN_MARGIN);
  const effectiveFloor = Math.max(PRICE_FLOOR, marginFloor);

  let candidate = input.currentPrice;
  let reason = "no_change";

  if (input.unitsSold24h >= 50 && input.stock > 20) {
    candidate = input.currentPrice * 1.12;
    reason = "high_demand_24h";
  } else if (input.unitsSold24h < 15 && input.stock > 50) {
    candidate = input.currentPrice * 0.93;
    reason = "low_demand_high_stock";
  }

  const competitorIsFresh =
    input.competitorPrice !== undefined &&
    input.competitorObservedAt !== undefined &&
    minutesBetween(input.now, input.competitorObservedAt) <= MAX_COMPETITOR_AGE_MINUTES;

  if (competitorIsFresh && input.competitorPrice! < candidate) {
    candidate = input.competitorPrice! - 1;
    reason = "fresh_competitor_undercut";
  }

  const guarded = clamp(candidate, effectiveFloor, PRICE_CEILING);

  return {
    newPrice: roundPrice(guarded),
    reason
  };
}

This code makes a few tradeoffs visible.

It refuses to change prices too often. That reduces reactivity, but it also prevents price bouncing when competitors and your own system keep reacting to each other.

It treats stale competitor data as unusable. That means you may miss a short sale, but you avoid changing prices based on a page snapshot that no longer reflects the market.

It calculates a margin-based floor in addition to a business-defined floor. If cost changes and nobody updates the static floor, the system still avoids selling below the configured minimum margin.

Test the ugly cases

Do not only test the expected demand spike. Test the cases that would create support tickets.

import { expect, test } from "vitest";
import { decidePrice } from "./pricing";

test("does not use stale competitor price", () => {
  const now = new Date("2026-01-10T12:00:00Z");

  const decision = decidePrice({
    sku: "console-1",
    currentPrice: 399,
    unitCost: 280,
    stock: 80,
    unitsSold24h: 10,
    competitorPrice: 299,
    competitorObservedAt: new Date("2026-01-10T10:00:00Z"),
    now
  });

  expect(decision.newPrice).toBe(371.07);
  expect(decision.reason).toBe("low_demand_high_stock");
});

Other tests worth adding:

Competitor price would push below cost plus margin
Current stock is low, so low-price rules should not fire
Price ceiling blocks a demand spike increase
Last price change happened 20 minutes ago
Competitor price is for the wrong variant or missing shipping

That last one is not purely a unit test problem. You need validation before observations enter the pricing engine. If the competitor page contains multiple variants, store the matched variant identifier and confidence score, or reject the observation.

Roll out with a kill switch

Start with one category and a small SKU set. Electronics, accessories, and seasonal products usually show pricing effects faster than low-volume premium products.

Track at least these metrics:

Conversion rate
Gross margin per order
Units sold
Cart abandonment
Price changes per SKU per day
Percentage of decisions blocked by guardrails

The last metric tells you whether your rules are fighting your constraints. If 60% of proposed changes hit the floor, your discount logic probably ignores cost. If many decisions are blocked because data is stale, fix ingestion before tuning pricing.

Also add a kill switch:

if (process.env.DYNAMIC_PRICING_ENABLED !== "true") {
  return { newPrice: input.currentPrice, reason: "disabled" };
}

Dynamic pricing is operational software. It touches revenue, customer trust, and sometimes marketplace compliance. Build it like something that can fail, because it will.

A good next step is to pick five SKUs, write down the exact inputs and guardrails for each, then run your pricing function in shadow mode for a week before allowing it to update the catalog.

Dynamic Pricing Is a Data Freshness Problem Before It Is a Model Problem

Anakin — Mon, 14 Sep 2026 10:00:02 +0000

If you have ever worked on marketplace pricing, the obvious problem is usually not the hardest one. The obvious problem is deciding the price. The harder problem is deciding whether the data you are using is fresh, complete, and safe enough to price from at all.

Food delivery makes this visible because the system changes every few minutes. Demand spikes in one neighborhood. Couriers move to another. A restaurant gets backed up. Rain changes ETAs. A competitor starts a promo. If your pricing service reacts to one signal but misses the others, users see weird fees, drivers get bad incentives, and support gets the tickets.

Price is an output of marketplace state

A delivery fee is not just a margin calculation. It is a control surface for the marketplace.

At minimum, a pricing service needs inputs like:

current order demand by zone
available couriers by zone
restaurant prep time estimates
recent cancellation and late delivery rates
traffic or ETA predictions
customer segment or subscription status
promo budget and experiment assignment
competitor price or promo observations, if you track them

The important detail is that these inputs age differently. Courier availability might be useless after 30 seconds. Restaurant prep time might be acceptable for 5 minutes. Customer segment can be cached for hours.

Treating all signals as equally fresh is how pricing systems make bad decisions confidently.

Put guardrails before the model

You can use a regression model, a rules engine, a bandit, or a hand-tuned formula. The pattern still needs the same safety checks around it.

Here is a simplified TypeScript version of a delivery-fee quote function:

type Signals = {
  zoneId: string;
  demandLast5Min: number;
  availableCouriers: number;
  avgPrepTimeMin: number;
  etaP50Min: number;
  updatedAtMs: number;
};

type Quote = {
  feeCents: number;
  reason: "NORMAL" | "SURGE" | "STALE_SIGNALS" | "CAPPED";
};

const MAX_SIGNAL_AGE_MS = 90_000;
const MAX_FEE_CHANGE_PCT = 0.25;

export function quoteDeliveryFee(
  baseFeeCents: number,
  previousFeeCents: number,
  signals: Signals,
  nowMs = Date.now()
): Quote {
  const ageMs = nowMs - signals.updatedAtMs;

  if (ageMs > MAX_SIGNAL_AGE_MS) {
    return {
      feeCents: previousFeeCents,
      reason: "STALE_SIGNALS"
    };
  }

  const courierCount = Math.max(signals.availableCouriers, 1);
  const pressure = signals.demandLast5Min / courierCount;

  let multiplier = 1;

  if (pressure > 4) multiplier += 0.35;
  else if (pressure > 2) multiplier += 0.15;

  if (signals.avgPrepTimeMin > 25) multiplier += 0.10;
  if (signals.etaP50Min > 40) multiplier += 0.10;

  const rawFee = Math.round(baseFeeCents * multiplier);

  const maxAllowed = Math.round(previousFeeCents * (1 + MAX_FEE_CHANGE_PCT));
  const cappedFee = Math.min(rawFee, maxAllowed);

  return {
    feeCents: cappedFee,
    reason: cappedFee !== rawFee ? "CAPPED" : multiplier > 1 ? "SURGE" : "NORMAL"
  };
}

This is not a pricing strategy. It is a safety shape.

The function does three things that matter in production:

It refuses to price from stale operational data.
It avoids division by zero when courier supply is empty.
It caps sudden fee changes so users do not see a fee jump from $2 to $9 in one refresh.

The division-by-zero case is not theoretical. In JavaScript, this can turn into a bad downstream payload:

JSON.stringify({ pressure: Infinity });
// {"pressure":null}

If another service treats null as 0, an outage in courier availability can become free delivery during peak demand. That is the kind of bug that looks like a finance problem until someone traces it back to serialization.

Discounts should behave like experiments

A discount is not useful because conversion went up. Conversion almost always goes up when you make something cheaper.

The question is whether the discount changed future behavior enough to justify the cost. Did the customer order again without a coupon? Did average order value increase? Did the promo mostly attract users who only buy when subsidized?

That means discounts need experiment IDs, holdouts, and post-promo measurement. A basic query might look like this:

select
  experiment_id,
  variant,
  count(distinct user_id) as users,
  avg(order_total_cents) as avg_order_value,
  avg(discount_cents) as avg_discount,
  sum(case when reordered_within_14d then 1 else 0 end)::float
    / count(*) as reorder_rate_14d
from promo_orders
where created_at >= now() - interval '30 days'
group by experiment_id, variant;

If treatment users reorder at the same rate as holdout users but cost $4 more per order, the discount did not create durable demand. It rented it.

External signals need timestamps and confidence

Competitor prices, restaurant menu changes, delivery estimates, and visible promos can all be useful inputs. They can also poison a pricing system if you treat scraped or third-party data as truth without metadata.

Every external observation should carry at least:

observed_at
source
confidence
entity_id
raw_value
normalized_value
parser_version

If a competitor promo page changes structure and your parser starts reading the wrong value, the worst outcome is not a failed job. The worst outcome is a successful job that writes believable garbage.

For teams that use external food delivery pages as pricing inputs, Wire is useful when you treat extracted prices, ETAs, and promos as timestamped observations with failure handling instead of silent facts.

The same rule applies if you build the extraction yourself. Store raw responses when possible, version your parsers, and alert on distribution shifts. If yesterday’s delivery fees ranged from 99 to 799 cents and today’s are all exactly 0, do not feed that into pricing.

Surge pricing is an operations mechanism

People often describe surge pricing as a way to increase revenue. In delivery marketplaces, it also coordinates supply.

A higher fee can reduce low-value demand. A higher courier payout can pull more drivers into a zone. But if you raise the customer fee without changing courier incentives, you may just reduce orders while leaving late deliveries unsolved.

That is why the pricing decision should emit more than a number. It should emit an explanation and an event:

{
  "quote_id": "q_123",
  "zone_id": "downtown",
  "fee_cents": 349,
  "reason": "SURGE",
  "pressure": 4.8,
  "signals_updated_at": "2026-08-18T12:04:30Z"
}

Operations, finance, support, and experimentation systems can all use that event. Without it, you only know what price the user saw. You do not know why they saw it.

What to build first

Do not start by making the pricing formula more complex. Start by measuring whether your current pricing inputs can be trusted.

Pick one pricing decision and add:

freshness checks per signal
caps on sudden price movement
explicit fallback behavior
experiment IDs for discounts
reason codes on every quote
alerts for impossible or suspicious input distributions

Once those pieces exist, improving the model becomes much safer. Without them, a smarter model just makes bad decisions faster.

Designing AI agents that can use live web data

Anakin — Fri, 11 Sep 2026 10:00:01 +0000

An AI agent that reads stale data is worse than one that admits it cannot answer. Pricing, listings, reviews, inventory, job posts, and product availability change constantly, but many agent workflows still depend on CSV exports, copied spreadsheets, or APIs that only cover part of the job.

The hard part is not asking the model to reason over web data. The hard part is getting current, structured data into the agent in a way that fails visibly when the source changes.

Treat web access as a data contract

Do not give an agent a vague tool called browse_web and hope it figures out the rest. Define the shape of the data you expect back.

For example, if you are collecting competitor hotel prices, the agent probably needs this:

{
  "source_url": "https://example.com/search?city=miami",
  "fetched_at": "2026-08-18T10:15:00Z",
  "currency": "USD",
  "results": [
    {
      "name": "Hotel Example",
      "nightly_rate": 184,
      "available": true,
      "rating": 4.6,
      "review_count": 812
    }
  ]
}

That schema matters because it gives you something to validate before the model touches the data. If results is empty, that might mean no availability. It might also mean the scraper hit a bot page, the DOM changed, or the site returned a 403. Those cases should not collapse into the same answer.

A catalogued extractor such as Wire can help here because the agent calls a named action with known parameters instead of inventing selectors at runtime.

Pick the extraction method based on how the site works

There are a few common ways to get live web data. They fail differently.

Plain HTTP works when the content is in the HTML response:

curl -L https://example.com/pricing

This is cheap and easy to cache, but it breaks when the page renders data client-side.

Browser rendering works when the site needs JavaScript:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/search?q=miami', {
  waitUntil: 'networkidle'
});

const cards = await page.locator('[data-testid="listing-card"]').allTextContents();
await browser.close();

This handles more sites, but it costs more per request and tends to fail under volume. You also inherit timing problems: a selector exists locally, then fails in production because a cookie banner, A/B test, or slow API call changed what rendered.

Network-level extraction is often better for listings and search pages. Many sites load data by calling JSON endpoints from their own frontend. If you can identify that request, you can call it directly or replay the same parameters. You avoid spinning up a browser and parsing HTML.

For high-volume listing pages, Wire is one example of this network-call approach: capture the background API calls the frontend already uses and return structured records instead of rendered markup.

Authenticated pages are a separate case. Do not pass user passwords through the agent. Store a browser session or token outside the model, then expose only the specific action the agent is allowed to perform, such as get_open_invoices or fetch_saved_search_results.

Make scraping asynchronous when it can take time

Live extraction often takes longer than a normal API request. A browser may need to load several pages. A crawl may fan out across a site. A target may rate limit you.

Use a job model instead of forcing the agent to wait on one long request.

import time
import requests

API_KEY = 'replace-me'
BASE_URL = 'https://scraper.internal.example'

headers = {
    'Authorization': f'Bearer {API_KEY}',
    'Content-Type': 'application/json'
}

start = requests.post(
    f'{BASE_URL}/jobs',
    headers=headers,
    json={
        'url': 'https://example.com/search?city=miami',
        'render_js': True,
        'output': 'json'
    },
    timeout=10
)
start.raise_for_status()

job_id = start.json()['id']
deadline = time.time() + 90

while time.time() < deadline:
    res = requests.get(f'{BASE_URL}/jobs/{job_id}', headers=headers, timeout=10)

    if res.status_code == 429:
        time.sleep(5)
        continue

    res.raise_for_status()
    payload = res.json()

    if payload['status'] == 'succeeded':
        data = payload['result']
        if not data.get('results'):
            raise RuntimeError('Scrape succeeded but returned zero results')
        print(data)
        break

    if payload['status'] == 'failed':
        raise RuntimeError(payload.get('error', 'Scrape failed without an error message'))

    time.sleep(2)
else:
    raise TimeoutError(f'Scrape job {job_id} did not finish within 90 seconds')

The important part is not the exact endpoint. It is the contract:

The first request returns a job ID.
Polling reports queued, running, succeeded, or failed.
Failures include a reason the agent can report.
Timeouts are explicit.
Empty output is treated as suspicious unless the query expected it.

If you skip this, your agent will eventually produce confident nonsense because the scraper returned a login page, a CAPTCHA, or an empty array.

Return evidence with the data

Agents need more than records. They need enough metadata to know whether the records are usable.

Include fields like:

{
  "source_url": "https://example.com/search?city=miami",
  "fetched_at": "2026-08-18T10:15:00Z",
  "http_status": 200,
  "extractor_version": "hotel-search-v3",
  "result_count": 42,
  "warnings": []
}

Warnings are useful. For example:

{
  "warnings": [
    "Only 8 results returned, expected at least 25 for this market",
    "Prices were missing for 3 listings",
    "Source page showed results in EUR, converted to USD"
  ]
}

That gives the agent room to answer carefully: I found 8 listings, but this looks incomplete, instead of treating partial data as complete.

Keep the agent on rails

The model should decide what data it needs and how to interpret it. It should not decide how to scrape a complex website from scratch on every run.

A better pattern is to expose narrow tools:

type FetchCompetitorRatesInput = {
  city: string;
  checkIn: string;
  checkOut: string;
  guests: number;
};

Behind that tool, you can change selectors, swap browser rendering for network extraction, add retries, or fix rate-limit handling without changing the agent prompt.

Start by picking one workflow where stale web data causes real manual work. Define the output schema, implement the extractor as an async job, and make the failure states visible before you connect it to an agent.

Data Quality Checks That Catch Real Web Scraping Failures

Anakin — Thu, 10 Sep 2026 10:00:00 +0000

A scraper can run successfully and still give you bad data. The HTTP request returns 200, your parser finds elements, the job writes rows, and nobody notices that price now contains the text from a promo banner because the site changed a CSS class.

That is the annoying part of scraping: failure often looks like success until someone uses the dataset.

Treat scraped data like an API contract

The first mistake is storing whatever the page gives you and cleaning it later. That works for one-off scripts. It breaks down when you run scrapers on a schedule or feed results into dashboards, pricing systems, search indexes, or ML jobs.

Define the shape of a valid record before you scrape too much:

{
  "product_id": "SKU-123",
  "url": "https://example.com/products/sku-123",
  "title": "USB-C Charger",
  "price_usd": 24.99,
  "currency": "USD",
  "in_stock": true,
  "scraped_at": "2026-08-18T10:15:00Z"
}

That contract should include types, required fields, accepted ranges, and any values that must be normalized. For example, do not allow price_usd to sometimes be "$24.99", sometimes 24.99, and sometimes "Contact us". Pick one representation and reject the rest.

Here is a small Pydantic model that catches common scrape issues at ingestion:

from datetime import datetime, timezone
from pydantic import BaseModel, HttpUrl, Field, field_validator, ValidationError

class ProductRow(BaseModel):
    product_id: str = Field(min_length=1)
    url: HttpUrl
    title: str = Field(min_length=1)
    price_usd: float = Field(gt=0, lt=10000)
    currency: str
    in_stock: bool
    scraped_at: datetime

    @field_validator("currency")
    @classmethod
    def currency_must_be_usd(cls, value):
        if value != "USD":
            raise ValueError("currency must be USD")
        return value

    @field_validator("scraped_at")
    @classmethod
    def scraped_at_must_have_timezone(cls, value):
        if value.tzinfo is None:
            raise ValueError("scraped_at must be timezone-aware")
        return value.astimezone(timezone.utc)

raw_row = {
    "product_id": "SKU-123",
    "url": "https://example.com/products/sku-123",
    "title": "USB-C Charger",
    "price_usd": "Contact us",
    "currency": "USD",
    "in_stock": True,
    "scraped_at": "2026-08-18T10:15:00Z"
}

try:
    row = ProductRow.model_validate(raw_row)
except ValidationError as exc:
    print(exc)

That bad row fails with a useful symptom:

price_usd
  Input should be a valid number, unable to parse string as a number

That error tells you something specific: either the product really has no public price, or your selector is no longer pointing at a numeric price. Both cases need handling, but they are different problems.

Validate before, during, and after scraping

You need checks at more than one point because different failures happen at different times.

Before scraping, validate inputs. Remove duplicate URLs, reject unsupported domains, and normalize URL parameters if they do not affect page content. A bad URL list creates duplicate records and retry noise.

During scraping, validate extracted fields before writing them as clean data. If a required field is missing, do not silently store null and hope downstream code understands it. Put the item into a re-scrape queue with the reason.

After scraping, validate aggregate behavior. One missing price might be normal. A 40 percent missing-price rate usually means the page template changed, your proxy received a bot-check page, or the site served a region-specific layout you did not expect.

A simple re-scrape queue can be enough:

valid_rows = []
rescrape_queue = []

for raw_row in scraped_rows:
    try:
        valid_rows.append(ProductRow.model_validate(raw_row).model_dump())
    except ValidationError as exc:
        rescrape_queue.append({
            "url": raw_row.get("url"),
            "product_id": raw_row.get("product_id"),
            "reason": exc.errors(),
            "raw": raw_row,
        })

print(f"valid={len(valid_rows)} rescrape={len(rescrape_queue)}")

The important part is not the queue technology. Use Redis, SQS, Postgres, or a file if the job is small. The important part is that invalid rows do not disappear and do not become trusted records.

Wire fits this kind of extraction workflow when selector failures, missing fields, retries, and validation status need to be tracked alongside the scraped records.

Watch for selector drift, not just exceptions

Many scraper bugs do not throw exceptions. BeautifulSoup, Cheerio, Playwright, and Selenium will happily return an empty list if a selector stops matching.

This code fails quietly:

price_el = soup.select_one("span.product-price")
price = price_el.text.strip() if price_el else None

If you accept None, the scraper keeps running. A dashboard might later show average price dropping because missing values were treated as zero, or a join might exclude products with null prices.

Make selector expectations explicit:

def require_text(soup, selector, field_name):
    el = soup.select_one(selector)
    if el is None:
        raise ValueError(f"missing selector for {field_name}: {selector}")

    text = el.get_text(strip=True)
    if not text:
        raise ValueError(f"empty text for {field_name}: {selector}")

    return text

raw_price = require_text(soup, "span.product-price", "price_usd")

This turns a silent data quality bug into an operational failure you can alert on. That is a good tradeoff. A failed job is easier to investigate than a clean-looking CSV full of wrong fields.

You should also store enough evidence to debug failures: URL, timestamp, scraper version, selector version, response status, content hash, and maybe a small HTML snapshot for failed pages. Do not store full pages forever unless you have a reason, since storage and privacy concerns add up quickly.

Use anomaly detection after basic rules

Start with simple rules. They catch most issues:

Required fields cannot be blank
Prices must be positive numbers
Timestamps must include a timezone
Product IDs must match the expected format
Duplicate product IDs within the same scrape should be investigated

Then add statistical checks for things that are valid individually but suspicious in bulk.

For example, a price of 499.99 might pass schema validation. But if the same product usually sells between 19.99 and 29.99, you should flag it.

import statistics

historical_prices = [21.99, 22.49, 20.99, 23.99, 21.49, 22.99]
new_price = 499.99

mean = statistics.mean(historical_prices)
stdev = statistics.stdev(historical_prices)
z_score = abs((new_price - mean) / stdev)

if z_score > 4:
    print(f"price anomaly: {new_price} z_score={z_score:.2f}")

This is not proof that the value is wrong. It is a signal. The site might have changed pack size, the item might be out of stock and listed by a marketplace seller, or your scraper might have captured a financing amount instead of a price.

That distinction matters. Do not automatically delete anomalies unless your business rule says to. Flag them, route them for review, or re-scrape from another source.

If you buy scraped datasets rather than run the crawlers yourself, Wire is most relevant when you need to understand how extraction failures, anomaly checks, and retry decisions are represented in the delivered data.

Keep schema changes visible

Scraped data changes because websites change. Your schema will change too. Add a version field to your output and keep validation code in version control.

{
  "schema_version": "products.v3",
  "product_id": "SKU-123",
  "price_usd": 24.99
}

When a field changes meaning, create a migration or a new version. Do not quietly reuse the old column name. availability meaning in_stock | out_of_stock is different from availability meaning ships_today | preorder | discontinued.

Your next practical step: take one existing scraper and add three things to it this week: a schema validator, a re-scrape queue for invalid rows, and a daily summary showing missing-field rates by domain and scraper version.

Competitor pricing data is only useful if you model its failure modes

Anakin — Tue, 08 Sep 2026 10:01:28 +0000

You build a quick scraper to watch competitor prices. It works for a week. Then a retailer changes its HTML, another starts showing member-only discounts, and your dashboard confidently says a $49 product is now $0. Bad pricing data is worse than missing pricing data because people trust the chart.

Competitive pricing analysis sounds like a business topic, but the hard part is usually engineering: collecting comparable observations from messy websites, detecting when extraction breaks, and avoiding comparisons that look precise but are wrong.

Store observations, not just current prices

A common mistake is keeping one row per competitor SKU and overwriting the price every time a scraper runs. That throws away the context you need when something looks wrong.

Use an append-only model instead:

CREATE TABLE price_observation (
  id BIGSERIAL PRIMARY KEY,
  competitor TEXT NOT NULL,
  sku TEXT NOT NULL,
  url TEXT NOT NULL,
  observed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  price_cents INTEGER,
  currency CHAR(3),
  availability TEXT,
  shipping_cents INTEGER,
  unit_quantity NUMERIC,
  extraction_status TEXT NOT NULL,
  error TEXT,
  raw_hash TEXT
);

CREATE INDEX ON price_observation (competitor, sku, observed_at DESC);

That extraction_status field matters. A missing price is not the same thing as a free product. Store failed_selector, blocked, out_of_stock, or currency_unknown rather than forcing everything into price_cents.

For this part of the stack, Wire is an extraction layer for competitor pricing data, so your application can consume structured price observations instead of page-specific HTML.

Fail loudly when a page changes

Here is a deliberately small scraper. The important part is not BeautifulSoup. The important part is that it validates what it extracted before writing a successful observation.

from decimal import Decimal
import hashlib
import re
import httpx
from bs4 import BeautifulSoup

PRICE_RE = re.compile(r'(\d+[\d,]*\.\d{2})')


def parse_price(text: str) -> int:
    match = PRICE_RE.search(text.replace(',', ''))
    if not match:
        raise ValueError(f'no price found in: {text[:80]}')

    amount = Decimal(match.group(1))
    if amount <= 0:
        raise ValueError(f'invalid price: {amount}')

    return int(amount * 100)


def fetch_price(url: str) -> dict:
    try:
        response = httpx.get(
            url,
            timeout=10,
            headers={'User-Agent': 'pricing-monitor/1.0'}
        )
        response.raise_for_status()

        html = response.text
        soup = BeautifulSoup(html, 'html.parser')
        node = soup.select_one('[data-testid="product-price"]')

        if node is None:
            return {
                'extraction_status': 'failed_selector',
                'error': 'missing [data-testid="product-price"]',
                'raw_hash': hashlib.sha256(html.encode()).hexdigest()
            }

        return {
            'price_cents': parse_price(node.get_text(' ', strip=True)),
            'currency': 'USD',
            'extraction_status': 'ok',
            'raw_hash': hashlib.sha256(html.encode()).hexdigest()
        }

    except httpx.HTTPStatusError as e:
        return {
            'extraction_status': 'blocked' if e.response.status_code in (403, 429) else 'http_error',
            'error': f'{e.response.status_code} {url}'
        }
    except Exception as e:
        return {
            'extraction_status': 'parse_error',
            'error': str(e)
        }

Without these states, a broken selector often turns into one of three bad symptoms: nulls silently disappear from aggregates, zeroes pull averages down, or yesterday's price gets reused and nobody notices the data is stale.

Normalize before comparing

The number on the page is not always the number you should compare.

A $40 two-pack is not cheaper than a $25 single item. A $99 product with $18 shipping is not equal to a $99 product with free shipping. A monthly subscription and annual subscription need different handling.

At minimum, calculate an effective unit price:

SELECT
  competitor,
  sku,
  observed_at,
  (price_cents + COALESCE(shipping_cents, 0)) / NULLIF(unit_quantity, 0) AS unit_price_cents
FROM price_observation
WHERE extraction_status = 'ok'
  AND currency = 'USD';

This is where many pricing dashboards start giving useful signals. You can spot products where you look overpriced but actually include shipping, or products where you look competitive only because you sell a smaller quantity.

Look for patterns, not single differences

One competitor undercutting you once is noise. A competitor discounting the same SKU every Friday for six weeks is a pattern.

For example, this query looks for competitors whose price changes unusually often:

WITH daily AS (
  SELECT
    competitor,
    sku,
    date_trunc('day', observed_at) AS day,
    min(price_cents) AS price_cents
  FROM price_observation
  WHERE extraction_status = 'ok'
  GROUP BY competitor, sku, day
), changes AS (
  SELECT
    competitor,
    sku,
    day,
    price_cents,
    lag(price_cents) OVER (
      PARTITION BY competitor, sku ORDER BY day
    ) AS previous_price_cents
  FROM daily
)
SELECT competitor, sku, count(*) AS price_change_days
FROM changes
WHERE previous_price_cents IS NOT NULL
  AND price_cents <> previous_price_cents
GROUP BY competitor, sku
HAVING count(*) >= 5
ORDER BY price_change_days DESC;

That output can tell you more than "they are cheaper." Frequent discounts might mean inventory pressure, weak retention, or automated repricing. It might also mean your scraper is alternating between a logged-out price and a promo price, so check the raw observations before drawing conclusions.

Wire fits this kind of workflow when the useful output is not a screenshot or one-off scrape, but repeated price observations you can diff over time.

The hard part is product matching

Most pricing mistakes come from comparing the wrong products.

Names are not enough. Retailers add bundles, change pack sizes, include accessories, or sell regional variants. If you match on title similarity alone, "USB-C charger 30W" and "USB-C charger 30W 2-pack" may look close enough while being commercially different.

Use a match table with confidence and review status:

CREATE TABLE competitor_sku_match (
  internal_sku TEXT NOT NULL,
  competitor TEXT NOT NULL,
  competitor_sku TEXT NOT NULL,
  match_confidence NUMERIC NOT NULL,
  reviewed BOOLEAN NOT NULL DEFAULT false,
  PRIMARY KEY (internal_sku, competitor, competitor_sku)
);

Let automation propose matches, but require review for high-impact SKUs. This is boring operational work, and it prevents expensive mistakes.

What the analysis is actually good for

Once the data is clean enough, competitive pricing analysis helps with more than repricing.

It can show financial leaks, like a product priced above the market without a matching conversion rate. It can reveal white space, like a missing budget tier where competitors only sell premium bundles. It can support product decisions, like whether customers pay extra for a feature or whether everyone treats it as table stakes.

The tradeoff is that none of this works from a single scrape. You need history, normalized units, extraction failure tracking, and reviewed SKU matches.

A practical next step: pick 10 important SKUs, track 3 competitors for 30 days, store every observation, and review every anomaly by looking at the raw page. If the data still supports the same conclusion after that, it is probably safe enough to use in pricing discussions.

Competitive pricing is a data pipeline problem before it is a pricing strategy

Anakin — Mon, 07 Sep 2026 10:00:01 +0000

You get asked to build a feature that “matches competitor prices.” It sounds simple until the first dataset arrives. One competitor includes shipping, another shows a coupon only after login, one page says “from $19.99,” and your scraper quietly captured the price for a used item instead of the new one.

That is the part people usually skip when they talk about competitive pricing. The pricing rule is rarely the hard part. The hard part is collecting prices you can trust, normalizing them, and making sure automation does not start a race to the bottom.

Competitive pricing is not always undercutting

Competitive pricing means setting your price based on market prices, not only your internal cost model. That can mean a few different things:

Match the market price to reduce buyer friction.
Undercut by a small amount when price is the main decision factor.
Price above competitors when your product has brand, service, bundle, or availability advantages.
Sell one item cheaply and make money later, like the classic razor-and-blades model.

For developers, the business question usually turns into this:

Given our SKU, our cost, margin rules, stock status, and competitor offers,
return a recommended price and explain why.

That “explain why” matters. If a buyer or category manager asks why a product dropped from $49.99 to $38.20, “the script did it” is not a useful answer.

Collect prices you can defend

A basic competitor price collector might start like this:

pip install httpx beautifulsoup4

from decimal import Decimal
import re
import httpx
from bs4 import BeautifulSoup

TARGETS = [
    {
        "sku": "USB-C-CHARGER-65W",
        "competitor": "example-shop",
        "url": "https://example.com/products/65w-usb-c-charger",
        "price_selector": ".product-price",
    }
]

PRICE_RE = re.compile(r"(\d+[\d,]*(?:\.\d{2})?)")


def parse_price(text: str) -> Decimal:
    match = PRICE_RE.search(text.replace(",", ""))
    if not match:
        raise ValueError(f"price_not_found: {text[:80]!r}")
    return Decimal(match.group(1))


def fetch_price(target: dict) -> dict:
    headers = {"User-Agent": "PriceMonitor/1.0 contact@example.com"}

    with httpx.Client(timeout=10, follow_redirects=True, headers=headers) as client:
        response = client.get(target["url"])
        response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    node = soup.select_one(target["price_selector"])

    if node is None:
        raise ValueError(
            f"selector_not_found: {target['competitor']} {target['sku']}"
        )

    return {
        "sku": target["sku"],
        "competitor": target["competitor"],
        "url": target["url"],
        "price": parse_price(node.get_text(" ", strip=True)),
    }


for target in TARGETS:
    try:
        print(fetch_price(target))
    except httpx.HTTPStatusError as exc:
        print(f"http_error: {exc.response.status_code} {target['url']}")
    except Exception as exc:
        print(f"collection_error: {exc}")

This is enough to show the pattern, but it is not enough to run pricing for a real catalog.

Here is how it fails in production:

A page starts rendering prices with JavaScript, so selector_not_found spikes.
The site returns 403 Forbidden or a CAPTCHA page, and your parser extracts no price.
The captured price is “from $19.99,” not the actual matching variant.
The competitor shows a marketplace offer from a third-party seller with slow shipping.
The product is a 2-pack, while your SKU is a single unit.
The page shows a coupon, but only for logged-in users or specific regions.

If you do not store these failures explicitly, you will treat missing data as cheap data, which is worse than having no data. Wire fits this part of the workflow when competitor-price extraction needs structured fields, retries, and clear failure states instead of silent scraper drift.

Normalize before comparing

Do not compare raw page prices directly. Compare landed, equivalent prices.

A useful normalized record usually needs fields like these:

CREATE TABLE competitor_prices (
  sku TEXT NOT NULL,
  competitor TEXT NOT NULL,
  observed_at TEXT NOT NULL,
  item_price NUMERIC NOT NULL,
  shipping_price NUMERIC DEFAULT 0,
  currency TEXT NOT NULL,
  quantity INTEGER DEFAULT 1,
  in_stock BOOLEAN NOT NULL,
  product_condition TEXT NOT NULL,
  url TEXT NOT NULL,
  collection_status TEXT NOT NULL
);

Then compare on a derived value:

landed_unit_price = (item_price + shipping_price - instant_discount) / quantity

You may also need to exclude offers where product_condition != 'new', in_stock = false, or shipping time exceeds your threshold. Otherwise your repricer may match an out-of-stock listing or a used item.

This is also where product matching matters. Matching by title similarity alone will burn you. “iPhone 15 case” and “iPhone 15 Pro case” are not interchangeable. For important SKUs, store a manually approved competitor URL or a product identifier such as GTIN, UPC, EAN, or manufacturer part number.

Put guardrails around the pricing rule

Once the data is clean enough, the repricing logic can stay boring. That is a good thing.

from decimal import Decimal


def recommend_price(
    cost: Decimal,
    current_price: Decimal,
    competitor_price: Decimal,
    min_margin: Decimal,
    max_price: Decimal,
    undercut_by: Decimal = Decimal("0.01"),
) -> dict:
    floor = cost * (Decimal("1") + min_margin)
    candidate = competitor_price - undercut_by

    if candidate < floor:
        return {
            "price": current_price,
            "action": "hold",
            "reason": "competitor_below_margin_floor",
            "margin_floor": floor,
            "competitor_price": competitor_price,
        }

    return {
        "price": min(candidate, max_price),
        "action": "update",
        "reason": "undercut_competitor_within_margin",
        "margin_floor": floor,
        "competitor_price": competitor_price,
    }


print(
    recommend_price(
        cost=Decimal("30.00"),
        current_price=Decimal("49.99"),
        competitor_price=Decimal("42.00"),
        min_margin=Decimal("0.20"),
        max_price=Decimal("59.99"),
    )
)

The important bit is not the one-cent undercut. It is the hold case. If the competitor sells below your margin floor, your system should refuse to follow and record why.

You probably also want:

A maximum daily price change, for example 5%.
A minimum observation count before acting.
Human approval for high-revenue SKUs.
A cooldown period after each change.
Alerts when many competitors suddenly disappear from the dataset.

Without those controls, a bad scrape can become a bad price.

The real strategy depends on your business model

Amazon-style undercutting works when scale, logistics, and supplier terms support it. Best Buy uses price matching to stop customers from checking out elsewhere. Apple can price above similar hardware because the ecosystem and brand carry value. Aldi undercuts by redesigning operations, not by randomly accepting lower margins.

The implementation should reflect that strategy. A premium brand may monitor competitors only to avoid being wildly out of range. A commodity seller may update prices daily. A marketplace seller may need near-real-time monitoring, but only for SKUs where inventory and margin justify it.

A practical first version is small: pick 20 important SKUs, collect competitor prices twice a day, store failures as first-class records, calculate landed unit prices, and require manual approval before changing anything. After two weeks, inspect the failures and bad matches before you automate the next step.