DEV Community: Anakin

Competitor pricing data is only useful if you model its failure modes

Anakin — Tue, 08 Sep 2026 10:01:28 +0000

You build a quick scraper to watch competitor prices. It works for a week. Then a retailer changes its HTML, another starts showing member-only discounts, and your dashboard confidently says a $49 product is now $0. Bad pricing data is worse than missing pricing data because people trust the chart.

Competitive pricing analysis sounds like a business topic, but the hard part is usually engineering: collecting comparable observations from messy websites, detecting when extraction breaks, and avoiding comparisons that look precise but are wrong.

Store observations, not just current prices

A common mistake is keeping one row per competitor SKU and overwriting the price every time a scraper runs. That throws away the context you need when something looks wrong.

Use an append-only model instead:

CREATE TABLE price_observation (
  id BIGSERIAL PRIMARY KEY,
  competitor TEXT NOT NULL,
  sku TEXT NOT NULL,
  url TEXT NOT NULL,
  observed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  price_cents INTEGER,
  currency CHAR(3),
  availability TEXT,
  shipping_cents INTEGER,
  unit_quantity NUMERIC,
  extraction_status TEXT NOT NULL,
  error TEXT,
  raw_hash TEXT
);

CREATE INDEX ON price_observation (competitor, sku, observed_at DESC);

That extraction_status field matters. A missing price is not the same thing as a free product. Store failed_selector, blocked, out_of_stock, or currency_unknown rather than forcing everything into price_cents.

For this part of the stack, Wire is an extraction layer for competitor pricing data, so your application can consume structured price observations instead of page-specific HTML.

Fail loudly when a page changes

Here is a deliberately small scraper. The important part is not BeautifulSoup. The important part is that it validates what it extracted before writing a successful observation.

from decimal import Decimal
import hashlib
import re
import httpx
from bs4 import BeautifulSoup

PRICE_RE = re.compile(r'(\d+[\d,]*\.\d{2})')


def parse_price(text: str) -> int:
    match = PRICE_RE.search(text.replace(',', ''))
    if not match:
        raise ValueError(f'no price found in: {text[:80]}')

    amount = Decimal(match.group(1))
    if amount <= 0:
        raise ValueError(f'invalid price: {amount}')

    return int(amount * 100)


def fetch_price(url: str) -> dict:
    try:
        response = httpx.get(
            url,
            timeout=10,
            headers={'User-Agent': 'pricing-monitor/1.0'}
        )
        response.raise_for_status()

        html = response.text
        soup = BeautifulSoup(html, 'html.parser')
        node = soup.select_one('[data-testid="product-price"]')

        if node is None:
            return {
                'extraction_status': 'failed_selector',
                'error': 'missing [data-testid="product-price"]',
                'raw_hash': hashlib.sha256(html.encode()).hexdigest()
            }

        return {
            'price_cents': parse_price(node.get_text(' ', strip=True)),
            'currency': 'USD',
            'extraction_status': 'ok',
            'raw_hash': hashlib.sha256(html.encode()).hexdigest()
        }

    except httpx.HTTPStatusError as e:
        return {
            'extraction_status': 'blocked' if e.response.status_code in (403, 429) else 'http_error',
            'error': f'{e.response.status_code} {url}'
        }
    except Exception as e:
        return {
            'extraction_status': 'parse_error',
            'error': str(e)
        }

Without these states, a broken selector often turns into one of three bad symptoms: nulls silently disappear from aggregates, zeroes pull averages down, or yesterday's price gets reused and nobody notices the data is stale.

Normalize before comparing

The number on the page is not always the number you should compare.

A $40 two-pack is not cheaper than a $25 single item. A $99 product with $18 shipping is not equal to a $99 product with free shipping. A monthly subscription and annual subscription need different handling.

At minimum, calculate an effective unit price:

SELECT
  competitor,
  sku,
  observed_at,
  (price_cents + COALESCE(shipping_cents, 0)) / NULLIF(unit_quantity, 0) AS unit_price_cents
FROM price_observation
WHERE extraction_status = 'ok'
  AND currency = 'USD';

This is where many pricing dashboards start giving useful signals. You can spot products where you look overpriced but actually include shipping, or products where you look competitive only because you sell a smaller quantity.

Look for patterns, not single differences

One competitor undercutting you once is noise. A competitor discounting the same SKU every Friday for six weeks is a pattern.

For example, this query looks for competitors whose price changes unusually often:

WITH daily AS (
  SELECT
    competitor,
    sku,
    date_trunc('day', observed_at) AS day,
    min(price_cents) AS price_cents
  FROM price_observation
  WHERE extraction_status = 'ok'
  GROUP BY competitor, sku, day
), changes AS (
  SELECT
    competitor,
    sku,
    day,
    price_cents,
    lag(price_cents) OVER (
      PARTITION BY competitor, sku ORDER BY day
    ) AS previous_price_cents
  FROM daily
)
SELECT competitor, sku, count(*) AS price_change_days
FROM changes
WHERE previous_price_cents IS NOT NULL
  AND price_cents <> previous_price_cents
GROUP BY competitor, sku
HAVING count(*) >= 5
ORDER BY price_change_days DESC;

That output can tell you more than "they are cheaper." Frequent discounts might mean inventory pressure, weak retention, or automated repricing. It might also mean your scraper is alternating between a logged-out price and a promo price, so check the raw observations before drawing conclusions.

Wire fits this kind of workflow when the useful output is not a screenshot or one-off scrape, but repeated price observations you can diff over time.

The hard part is product matching

Most pricing mistakes come from comparing the wrong products.

Names are not enough. Retailers add bundles, change pack sizes, include accessories, or sell regional variants. If you match on title similarity alone, "USB-C charger 30W" and "USB-C charger 30W 2-pack" may look close enough while being commercially different.

Use a match table with confidence and review status:

CREATE TABLE competitor_sku_match (
  internal_sku TEXT NOT NULL,
  competitor TEXT NOT NULL,
  competitor_sku TEXT NOT NULL,
  match_confidence NUMERIC NOT NULL,
  reviewed BOOLEAN NOT NULL DEFAULT false,
  PRIMARY KEY (internal_sku, competitor, competitor_sku)
);

Let automation propose matches, but require review for high-impact SKUs. This is boring operational work, and it prevents expensive mistakes.

What the analysis is actually good for

Once the data is clean enough, competitive pricing analysis helps with more than repricing.

It can show financial leaks, like a product priced above the market without a matching conversion rate. It can reveal white space, like a missing budget tier where competitors only sell premium bundles. It can support product decisions, like whether customers pay extra for a feature or whether everyone treats it as table stakes.

The tradeoff is that none of this works from a single scrape. You need history, normalized units, extraction failure tracking, and reviewed SKU matches.

A practical next step: pick 10 important SKUs, track 3 competitors for 30 days, store every observation, and review every anomaly by looking at the raw page. If the data still supports the same conclusion after that, it is probably safe enough to use in pricing discussions.

Competitive pricing is a data pipeline problem before it is a pricing strategy

Anakin — Mon, 07 Sep 2026 10:00:01 +0000

You get asked to build a feature that “matches competitor prices.” It sounds simple until the first dataset arrives. One competitor includes shipping, another shows a coupon only after login, one page says “from $19.99,” and your scraper quietly captured the price for a used item instead of the new one.

That is the part people usually skip when they talk about competitive pricing. The pricing rule is rarely the hard part. The hard part is collecting prices you can trust, normalizing them, and making sure automation does not start a race to the bottom.

Competitive pricing is not always undercutting

Competitive pricing means setting your price based on market prices, not only your internal cost model. That can mean a few different things:

Match the market price to reduce buyer friction.
Undercut by a small amount when price is the main decision factor.
Price above competitors when your product has brand, service, bundle, or availability advantages.
Sell one item cheaply and make money later, like the classic razor-and-blades model.

For developers, the business question usually turns into this:

Given our SKU, our cost, margin rules, stock status, and competitor offers,
return a recommended price and explain why.

That “explain why” matters. If a buyer or category manager asks why a product dropped from $49.99 to $38.20, “the script did it” is not a useful answer.

Collect prices you can defend

A basic competitor price collector might start like this:

pip install httpx beautifulsoup4

from decimal import Decimal
import re
import httpx
from bs4 import BeautifulSoup

TARGETS = [
    {
        "sku": "USB-C-CHARGER-65W",
        "competitor": "example-shop",
        "url": "https://example.com/products/65w-usb-c-charger",
        "price_selector": ".product-price",
    }
]

PRICE_RE = re.compile(r"(\d+[\d,]*(?:\.\d{2})?)")


def parse_price(text: str) -> Decimal:
    match = PRICE_RE.search(text.replace(",", ""))
    if not match:
        raise ValueError(f"price_not_found: {text[:80]!r}")
    return Decimal(match.group(1))


def fetch_price(target: dict) -> dict:
    headers = {"User-Agent": "PriceMonitor/1.0 contact@example.com"}

    with httpx.Client(timeout=10, follow_redirects=True, headers=headers) as client:
        response = client.get(target["url"])
        response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    node = soup.select_one(target["price_selector"])

    if node is None:
        raise ValueError(
            f"selector_not_found: {target['competitor']} {target['sku']}"
        )

    return {
        "sku": target["sku"],
        "competitor": target["competitor"],
        "url": target["url"],
        "price": parse_price(node.get_text(" ", strip=True)),
    }


for target in TARGETS:
    try:
        print(fetch_price(target))
    except httpx.HTTPStatusError as exc:
        print(f"http_error: {exc.response.status_code} {target['url']}")
    except Exception as exc:
        print(f"collection_error: {exc}")

This is enough to show the pattern, but it is not enough to run pricing for a real catalog.

Here is how it fails in production:

A page starts rendering prices with JavaScript, so selector_not_found spikes.
The site returns 403 Forbidden or a CAPTCHA page, and your parser extracts no price.
The captured price is “from $19.99,” not the actual matching variant.
The competitor shows a marketplace offer from a third-party seller with slow shipping.
The product is a 2-pack, while your SKU is a single unit.
The page shows a coupon, but only for logged-in users or specific regions.

If you do not store these failures explicitly, you will treat missing data as cheap data, which is worse than having no data. Wire fits this part of the workflow when competitor-price extraction needs structured fields, retries, and clear failure states instead of silent scraper drift.

Normalize before comparing

Do not compare raw page prices directly. Compare landed, equivalent prices.

A useful normalized record usually needs fields like these:

CREATE TABLE competitor_prices (
  sku TEXT NOT NULL,
  competitor TEXT NOT NULL,
  observed_at TEXT NOT NULL,
  item_price NUMERIC NOT NULL,
  shipping_price NUMERIC DEFAULT 0,
  currency TEXT NOT NULL,
  quantity INTEGER DEFAULT 1,
  in_stock BOOLEAN NOT NULL,
  product_condition TEXT NOT NULL,
  url TEXT NOT NULL,
  collection_status TEXT NOT NULL
);

Then compare on a derived value:

landed_unit_price = (item_price + shipping_price - instant_discount) / quantity

You may also need to exclude offers where product_condition != 'new', in_stock = false, or shipping time exceeds your threshold. Otherwise your repricer may match an out-of-stock listing or a used item.

This is also where product matching matters. Matching by title similarity alone will burn you. “iPhone 15 case” and “iPhone 15 Pro case” are not interchangeable. For important SKUs, store a manually approved competitor URL or a product identifier such as GTIN, UPC, EAN, or manufacturer part number.

Put guardrails around the pricing rule

Once the data is clean enough, the repricing logic can stay boring. That is a good thing.

from decimal import Decimal


def recommend_price(
    cost: Decimal,
    current_price: Decimal,
    competitor_price: Decimal,
    min_margin: Decimal,
    max_price: Decimal,
    undercut_by: Decimal = Decimal("0.01"),
) -> dict:
    floor = cost * (Decimal("1") + min_margin)
    candidate = competitor_price - undercut_by

    if candidate < floor:
        return {
            "price": current_price,
            "action": "hold",
            "reason": "competitor_below_margin_floor",
            "margin_floor": floor,
            "competitor_price": competitor_price,
        }

    return {
        "price": min(candidate, max_price),
        "action": "update",
        "reason": "undercut_competitor_within_margin",
        "margin_floor": floor,
        "competitor_price": competitor_price,
    }


print(
    recommend_price(
        cost=Decimal("30.00"),
        current_price=Decimal("49.99"),
        competitor_price=Decimal("42.00"),
        min_margin=Decimal("0.20"),
        max_price=Decimal("59.99"),
    )
)

The important bit is not the one-cent undercut. It is the hold case. If the competitor sells below your margin floor, your system should refuse to follow and record why.

You probably also want:

A maximum daily price change, for example 5%.
A minimum observation count before acting.
Human approval for high-revenue SKUs.
A cooldown period after each change.
Alerts when many competitors suddenly disappear from the dataset.

Without those controls, a bad scrape can become a bad price.

The real strategy depends on your business model

Amazon-style undercutting works when scale, logistics, and supplier terms support it. Best Buy uses price matching to stop customers from checking out elsewhere. Apple can price above similar hardware because the ecosystem and brand carry value. Aldi undercuts by redesigning operations, not by randomly accepting lower margins.

The implementation should reflect that strategy. A premium brand may monitor competitors only to avoid being wildly out of range. A commodity seller may update prices daily. A marketplace seller may need near-real-time monitoring, but only for SKUs where inventory and margin justify it.

A practical first version is small: pick 20 important SKUs, collect competitor prices twice a day, store failures as first-class records, calculate landed unit prices, and require manual approval before changing anything. After two weeks, inspect the failures and bad matches before you automate the next step.

Competitive Intelligence Is a Freshness Problem, Not a Dashboard Problem

Anakin — Fri, 04 Sep 2026 10:00:01 +0000

If your competitor pricing table refreshes every Monday, but the market changes on Tuesday morning, your repricing logic is optimizing against history. The problem usually is not that the dashboard looks bad. The problem is that the data pipeline does not tell you what changed, what failed, and what is too old to trust.

Freshness needs to be part of the data model

A lot of competitive intelligence systems start as reports. Someone scrapes prices, dumps them into a table, and builds a dashboard over the latest row per SKU.

That works until you need to make operational decisions from the data.

For example:

A grocery app needs to know where a competitor is out of stock before routing inventory.
A marketplace needs to adjust promotions before traffic peaks.
A ride-hailing app needs competitor fares while the user is still in the session.

At that point, latest known price is not enough. You need to know when you observed it, when your system fetched it, whether the fetch succeeded, and whether the data is still inside an acceptable SLA.

A useful schema stores observations, not just current state:

CREATE TABLE competitor_offer_observations (
  competitor text NOT NULL,
  sku text NOT NULL,
  region text NOT NULL,
  observed_at timestamptz NOT NULL,
  fetched_at timestamptz NOT NULL DEFAULT now(),
  status_code int,
  price_cents int,
  currency text,
  in_stock boolean,
  source_url text,
  error_code text,
  raw_hash text,
  PRIMARY KEY (competitor, sku, region, observed_at)
);

This lets you distinguish between three very different states:

The competitor lowered the price.
Your crawler failed and you are looking at old data.
The page rendered, but the selector broke and price parsed as null.

Those states should not drive the same business action.

Missing data is a state, not an absence

The dangerous failure mode in competitive data is silent incompleteness. A 403 response, a timeout, a changed DOM node, or a region-specific redirect can all look like no competitor price available if you collapse failures into nulls.

That creates false confidence. Your dashboard still loads. Your model still runs. It just makes decisions from partial market coverage.

A simple freshness query catches a lot of this:

WITH latest AS (
  SELECT DISTINCT ON (competitor, sku, region)
    competitor,
    sku,
    region,
    fetched_at,
    status_code,
    price_cents,
    in_stock,
    error_code
  FROM competitor_offer_observations
  ORDER BY competitor, sku, region, observed_at DESC
)
SELECT
  competitor,
  sku,
  region,
  price_cents,
  in_stock,
  CASE
    WHEN status_code IS DISTINCT FROM 200 THEN 'failed_fetch'
    WHEN price_cents IS NULL AND error_code IS NOT NULL THEN 'parse_failed'
    WHEN now() - fetched_at > interval '15 minutes' THEN 'stale'
    ELSE 'fresh'
  END AS data_state
FROM latest;

The important part is not the exact interval. It is making freshness explicit enough that downstream systems can refuse bad input.

For teams that need competitor prices and stockouts as an operational feed rather than a scraping project, Wire focuses on reliable extraction, normalization, and visible failure handling for this kind of market data.

Delivery format changes who can act on the data

A dashboard is fine for weekly review. It is a poor interface for automated pricing, inventory routing, or churn prediction.

If engineers need the data, expose it like a product API:

GET /competitive-offers?sku=ABC-123&region=brooklyn&fresh_within_seconds=900

A useful response should include state and timestamps, not just values:

{
  "sku": "ABC-123",
  "region": "brooklyn",
  "competitor": "example_market",
  "price_cents": 1299,
  "in_stock": true,
  "observed_at": "2026-08-18T10:14:03Z",
  "fetched_at": "2026-08-18T10:14:08Z",
  "data_state": "fresh"
}

That shape lets a pricing service reject stale rows, lets analysts measure coverage, and lets product teams debug why a rule fired.

For non-technical users, natural language querying can help, but only if it sits on top of the same governed dataset. If a sales lead asks where competitors are out of stock, the answer should come from the same freshness-aware observations that feed the API. Otherwise you end up with two competing versions of reality.

Build versus buy is mostly about maintenance

It is tempting to build this internally. The first version often looks manageable: a scraper, a cron job, a warehouse table, and a dashboard.

The maintenance cost shows up later:

Target sites change markup and break parsers.
Dynamic rendering returns empty HTML unless you execute client-side code.
Region and device behavior create different prices for the same SKU.
Rate limits and bot defenses turn a working job into intermittent 403s.
Product matching becomes messy when titles, bundles, and pack sizes differ.

If your team builds the pipeline, track coverage as a first-class metric. Do not just count successful jobs. Count expected observations versus usable observations per competitor, SKU, and region.

If you buy the feed, ask whether Wire or any other provider can return raw timestamps, failure states, and coverage metrics rather than only cleaned final prices.

That question matters because cleaned data without failure context can hide the same problem as a brittle internal scraper.

A practical starting point

Pick one decision that depends on competitor data, such as repricing a category or reacting to regional stockouts. Define the maximum acceptable data age for that decision. Then instrument your pipeline so every row can be classified as fresh, stale, failed, or parse_failed.

Once you can see those states, you will know whether you have a dashboard problem, a collection problem, or a decision system consuming data it should not trust.

Building an E-Commerce Web Data Pipeline That Does Not Lie to You

Anakin — Thu, 03 Sep 2026 10:00:02 +0000

You scrape a competitor product page, load the price into a dashboard, and everything looks fine. Then someone asks why the alert fired at 3 a.m. for a “70% price drop” that never happened. The page changed currency based on location, your parser grabbed the crossed-out price, and the pipeline treated bad data as a market signal.

Public web data is useful in e-commerce, but the hard part is not just collecting it. The hard part is making sure it means what you think it means.

Treat public web data as an unreliable input

E-commerce teams use public web data for common jobs:

Price and promotion monitoring
Product assortment tracking
Stock availability checks
Review and rating analysis
Search result and marketplace ranking observation
Ad placement verification

All of those sound straightforward until you look at the actual pages.

The same product may have different prices by region, login state, device type, or shipping destination. A product card may show a sale price, list price, unit price, subscription price, and marketplace seller price in the same DOM subtree. Availability can mean “in stock online,” “available for pickup,” or “ships in 3 weeks.”

If you store only price: 19.99, you lose the context needed to debug later.

A better record looks more like this:

{
  "url": "https://example.com/products/sku-123",
  "sku": "sku-123",
  "title": "Example Running Shoe",
  "price": 79.99,
  "currency": "USD",
  "price_type": "sale",
  "availability": "in_stock",
  "seller": "Example Store",
  "country": "US",
  "captured_at": "2026-08-18T12:30:00Z",
  "source": "product_page",
  "raw_hash": "f4b8..."
}

That extra context makes the data less convenient to model, but much easier to trust.

Parse for evidence, not just fields

A fragile scraper grabs the first price-looking string from the page. A safer scraper checks structured data first, validates the value, then falls back to HTML selectors only when needed.

Here is a small Python example using JSON-LD and a few basic checks:

import json
import re
from decimal import Decimal, InvalidOperation

import requests
from bs4 import BeautifulSoup

PRICE_RE = re.compile(r"[^0-9.]")


def parse_decimal(value):
    if value is None:
        return None
    cleaned = PRICE_RE.sub("", str(value))
    try:
        return Decimal(cleaned)
    except InvalidOperation:
        return None


def extract_product(url):
    response = requests.get(
        url,
        headers={"User-Agent": "Mozilla/5.0 price-monitor/1.0"},
        timeout=15,
    )
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    for script in soup.select('script[type="application/ld+json"]'):
        try:
            data = json.loads(script.string or "{}")
        except json.JSONDecodeError:
            continue

        items = data if isinstance(data, list) else [data]
        for item in items:
            if item.get("@type") != "Product":
                continue

            offers = item.get("offers") or {}
            if isinstance(offers, list):
                offers = offers[0] if offers else {}

            price = parse_decimal(offers.get("price"))
            currency = offers.get("priceCurrency")

            if price is None or price <= 0:
                raise ValueError(f"Invalid price parsed from {url}: {offers.get('price')}")

            return {
                "url": url,
                "title": item.get("name"),
                "price": str(price),
                "currency": currency,
                "availability": offers.get("availability"),
                "source": "json_ld"
            }

    raise ValueError(f"No Product JSON-LD found for {url}")

This code is not enough for every store. Some pages render prices client-side. Some expose stale JSON-LD. Some marketplace pages contain multiple products. But the pattern matters: prefer structured evidence, validate what you parse, and fail loudly when the page does not match your assumptions.

A bad failure mode looks like this:

price = None
currency = None
job_status = success

That is how dashboards learn to lie.

Expect blocking, drift, and partial data

Public pages change. Selectors break. Anti-bot systems return CAPTCHAs, 403 responses, or soft blocks where the server returns HTTP 200 with a page that says “verify you are human.” If your collector only checks status codes, it will store the CAPTCHA page as if it were a product page.

Add checks for symptoms:

def looks_blocked(html):
    markers = [
        "captcha",
        "verify you are human",
        "access denied",
        "unusual traffic",
    ]
    text = html.lower()
    return any(marker in text for marker in markers)

Then classify the result:

success: parsed and validated
blocked: CAPTCHA, access denied, or bot challenge
not_found: product removed or URL invalid
parse_error: page loaded but schema changed
validation_error: parsed data failed business rules

For teams that need managed e-commerce extraction rather than maintaining selectors, retries, and block detection themselves, Wire treats product collection as structured jobs with trackable failures instead of silent CSV gaps.

The important part is the failure taxonomy. Whether you build or buy collection, your downstream systems need to know the difference between “competitor is out of stock” and “our scraper got blocked.”

Dedupe products before comparing prices

Competitor monitoring often fails because teams compare URLs instead of products. The same item may appear under multiple URLs, colors, bundles, sellers, or tracking parameters.

Use a matching key when you have one:

GTIN, UPC, EAN, ISBN
Manufacturer part number
Marketplace ASIN or item ID
Brand plus normalized title plus size

Do not over-match. “iPhone 15 case” and “iPhone 15 Pro case” are different products, even if a fuzzy title match says they are close.

For fuzzy matching, store the confidence score and require review around the boundary:

from rapidfuzz import fuzz

score = fuzz.token_set_ratio(
    "Nike Pegasus 41 Mens Running Shoes Black Size 10",
    "Nike Air Zoom Pegasus 41 Men's Road Running Shoe - Black - 10"
)

if score >= 92:
    match_status = "auto_match"
elif score >= 80:
    match_status = "needs_review"
else:
    match_status = "no_match"

A wrong match can be worse than no data because it creates fake price gaps.

Put guardrails around alerts

Real-time competitor data tempts teams to automate every reaction. Be careful. Web data is noisy, and e-commerce pages contain short-lived promotions, personalization, and regional differences.

Before triggering an alert or repricing action, require multiple signals:

Same price observed in two consecutive runs
Price change exceeds a minimum absolute and percentage threshold
Currency and region match your baseline
Product match confidence is high
Page was not blocked or partially parsed

For scheduled competitor monitoring, Wire can return repeated price and availability snapshots, which still need the same validation rules before they drive alerts or pricing changes.

Public web data works best when you treat it like production telemetry from someone else's system: useful, incomplete, and occasionally wrong.

Start by taking one product category you already track manually. Define the fields, failure states, validation rules, and matching keys before scaling the crawler. If the small pipeline can explain its own bad data, the larger one has a chance.

Building a real-time price monitoring pipeline that does not poison your data

Anakin — Mon, 31 Aug 2026 10:00:02 +0000

A common failure mode in pricing systems is not that the scraper stops. It is that it keeps running and starts returning believable garbage: a price from the wrong variant, a sale price without the coupon condition, or a placeholder value from a partially rendered page.

If that bad data flows straight into a pricing engine, you do not have real-time intelligence. You have an automated way to make bad decisions faster.

Real-time data is mostly a freshness and trust problem

Teams usually talk about real-time web data as if the hard part is speed. Speed matters, but freshness without validation is dangerous.

For a competitor price monitor, you usually need to know four things:

What product did we observe?
What price did we observe?
When did we observe it?
How confident are we that the observation is correct?

That last field is the one many systems skip.

A simple pipeline might look like this:

Fetch competitor product pages or APIs
Extract price, availability, and metadata
Normalize currencies, units, and variants
Validate the observation against rules and history
Store the raw response and normalized record
Publish accepted changes to pricing or analytics systems

The important part is that extraction and decision-making stay separate. A scraper should not decide that your company should match a price. It should produce an observation with enough context for another service to evaluate it.

If your bottleneck is reliable extraction across changing product pages, Wire can sit in the extraction layer where teams need current competitor prices, availability signals, and failure visibility rather than another pricing rule engine.

Store observations, not just current state

A lot of teams start with a table like this:

CREATE TABLE competitor_prices (
  competitor TEXT NOT NULL,
  sku TEXT NOT NULL,
  price_cents INTEGER NOT NULL,
  currency TEXT NOT NULL,
  updated_at TIMESTAMP NOT NULL,
  PRIMARY KEY (competitor, sku)
);

That works for a dashboard showing current prices, but it destroys evidence. When a price changes from $99 to $19, you need to know whether that was a real discount, a parsing bug, or a product mismatch.

A better model keeps append-only observations:

CREATE TABLE price_observations (
  id BIGSERIAL PRIMARY KEY,
  competitor TEXT NOT NULL,
  sku TEXT NOT NULL,
  source_url TEXT NOT NULL,
  observed_price_cents INTEGER,
  currency TEXT,
  availability TEXT,
  confidence NUMERIC NOT NULL,
  raw_hash TEXT NOT NULL,
  observed_at TIMESTAMP NOT NULL,
  accepted BOOLEAN NOT NULL DEFAULT false,
  rejection_reason TEXT
);

CREATE INDEX idx_price_observations_lookup
ON price_observations (competitor, sku, observed_at DESC);

Then your “current price” becomes a query or materialized view over accepted observations, not the only copy of the truth.

Validate before publishing

Here is a simplified Python example. It rejects missing prices, unexpected currency changes, and large price movements unless the same value appears twice in a short window.

from dataclasses import dataclass
from datetime import datetime, timedelta

@dataclass
class Observation:
    competitor: str
    sku: str
    price_cents: int | None
    currency: str | None
    observed_at: datetime
    source_url: str

@dataclass
class ValidationResult:
    accepted: bool
    reason: str | None = None


def validate_observation(obs: Observation, history: list[Observation]) -> ValidationResult:
    if obs.price_cents is None:
        return ValidationResult(False, "missing_price")

    if obs.price_cents <= 0:
        return ValidationResult(False, "non_positive_price")

    previous = history[0] if history else None
    if not previous:
        return ValidationResult(True)

    if previous.currency and obs.currency != previous.currency:
        return ValidationResult(False, "currency_changed")

    change = abs(obs.price_cents - previous.price_cents) / previous.price_cents

    if change > 0.40:
        recent_matches = [
            h for h in history
            if h.price_cents == obs.price_cents
            and obs.observed_at - h.observed_at <= timedelta(minutes=30)
        ]

        if len(recent_matches) < 1:
            return ValidationResult(False, "large_price_change_needs_confirmation")

    return ValidationResult(True)

This is not enough for every business, but it shows the shape of the guardrail. You can add checks for pack size, variant name, shipping inclusion, coupon text, marketplace seller, and stock status.

The point is not to block every unusual price. The point is to stop one bad scrape from becoming an automated pricing action.

Treat freshness as a product requirement

“Real time” should have a number attached to it. A grocery delivery app, an airline fare monitor, and a B2B catalog do not need the same refresh interval.

For each data source, define:

Maximum acceptable age, for example 5 minutes or 6 hours
Retry policy when extraction fails
Whether stale data can still be shown
Whether stale data can drive automated decisions

A useful pattern is to publish data with an explicit freshness field:

{
  "competitor": "example-shop",
  "sku": "ABC-123",
  "price_cents": 12999,
  "currency": "USD",
  "observed_at": "2026-08-18T10:15:00Z",
  "fresh_until": "2026-08-18T10:45:00Z",
  "confidence": 0.94
}

Downstream systems can then make their own decision. A dashboard may show stale data with a warning. An automated repricing service may refuse to act on it.

Failure modes worth logging explicitly

Do not collapse every problem into scrape_failed. You want enough detail to know whether the issue is temporary, structural, or business-related.

Useful failure categories include:

http_403: blocked or unauthorized
http_429: rate limited
selector_missing: page structure changed
render_timeout: client-side page did not finish loading
product_unavailable: valid page, no purchasable item
variant_ambiguous: multiple possible matches
validation_rejected: extracted data failed quality checks

These categories make alerting less noisy. A few http_429 responses might trigger backoff. A sudden spike in selector_missing probably means someone needs to update the extractor.

Keep humans in the loop for high-impact changes

Fully automated pricing can work, but only when the risk is bounded. If a competitor drops a price by 3%, automation may be fine. If your system sees a 70% drop on a top-selling SKU, route it for review or require confirmation from another source.

You can encode that as policy:

def action_for_change(percent_change: float, revenue_rank: int) -> str:
    if percent_change < 0.05:
        return "auto_publish"

    if percent_change < 0.20 and revenue_rank > 100:
        return "auto_publish"

    if percent_change < 0.40:
        return "notify_category_manager"

    return "require_manual_review"

This keeps the system fast for routine changes while reducing the blast radius of bad or surprising data.

A practical next step

Pick one competitor, one product category, and one downstream decision. Build the pipeline with append-only observations, validation reasons, freshness metadata, and explicit failure categories before you connect it to pricing automation. You will learn more from a narrow system with good evidence than from a broad scraper that nobody trusts.

Building a Quote Monitoring Pipeline for Insurance Pricing

Anakin — Fri, 28 Aug 2026 10:00:04 +0000

A pricing team asks for competitor intelligence, and the first version usually looks simple: request a few quotes, put the premiums in a spreadsheet, compare numbers. Then the quote form changes, one site starts returning captchas, another hides fees behind a dropdown, and your dataset quietly fills with nulls.

Insurance pricing is not fully hidden. Every quote flow leaks signals about how inputs affect premiums: age, postcode, deductible, vehicle type, coverage limits, policy duration, and so on. The hard part is not noticing one quote. The hard part is collecting comparable observations often enough that the data can support a decision.

Treat quotes as observations, not facts

A quote is an observation taken at a specific time with a specific input profile. Store it that way. Do not just keep insurer name and premium.

A useful quote row usually needs fields like these:

CREATE TABLE quote_observations (
  id INTEGER PRIMARY KEY,
  captured_at TEXT NOT NULL,
  source TEXT NOT NULL,
  product TEXT NOT NULL,
  insurer TEXT NOT NULL,
  premium_cents INTEGER NOT NULL,
  currency TEXT NOT NULL,
  deductible_cents INTEGER,
  coverage_limit_cents INTEGER,
  policy_duration_days INTEGER,
  geography TEXT,
  profile_hash TEXT NOT NULL,
  raw_payload TEXT NOT NULL
);

The profile_hash matters because you need to compare like with like. If yesterday's driver age was 35 and today's is 36, the delta may be age-related rather than a competitor repricing. The raw_payload matters because parsers break. When they do, you need to reprocess old captures without rerunning the quote journey.

In this kind of workflow, Wire fits as the extraction layer for collecting live insurance quote fields across regions, demographics, and time windows.

Vary one input at a time

The common mistake is generating a large random sample and hoping analysis will explain it later. That gives you coverage, but it makes causality messy. For pricing intelligence, a controlled sampling matrix is easier to debug.

Start with a baseline profile, then change one input at a time:

import hashlib
import json
from copy import deepcopy

baseline = {
    'product': 'auto',
    'age': 35,
    'postcode': '94110',
    'deductible_cents': 50000,
    'coverage_limit_cents': 10000000,
    'vehicle': '2019 Toyota Camry',
    'policy_duration_days': 365,
}

variants = []

for age in [25, 35, 45, 60]:
    p = deepcopy(baseline)
    p['age'] = age
    variants.append(p)

for deductible in [25000, 50000, 100000]:
    p = deepcopy(baseline)
    p['deductible_cents'] = deductible
    variants.append(p)

for postcode in ['94110', '90011', '10001']:
    p = deepcopy(baseline)
    p['postcode'] = postcode
    variants.append(p)

def profile_hash(profile):
    stable = json.dumps(profile, sort_keys=True)
    return hashlib.sha256(stable.encode()).hexdigest()

for profile in variants:
    print(profile_hash(profile), profile)

This is not statistically complete, but it is understandable. If a premium jumps when only the postcode changes, you know where to investigate. You can add a larger sample later once the pipeline proves it can collect clean data.

Fail loudly when extraction breaks

Silent failure is worse than no data. If your parser cannot find the premium, raise an error and mark the capture as failed. Do not insert a zero, an empty string, or the previous value.

from bs4 import BeautifulSoup
import re

class QuoteParseError(Exception):
    pass

def parse_premium(html):
    soup = BeautifulSoup(html, 'html.parser')
    node = soup.select_one('[data-testid=monthly-premium]')

    if node is None:
        raise QuoteParseError('premium selector not found')

    text = node.get_text(' ', strip=True)
    match = re.search(r'\$([0-9,]+)(?:\.([0-9]{2}))?', text)

    if not match:
        raise QuoteParseError(f'premium text did not match currency pattern: {text}')

    dollars = int(match.group(1).replace(',', ''))
    cents = int(match.group(2) or '00')
    return dollars * 100 + cents

This will fail when the site changes markup, when the quote is hidden behind a login, when a bot wall returns a page that still has HTTP 200, or when the premium appears as an annual total instead of monthly. Those are good failures. They tell you the observation is not comparable.

Also record the response status, final URL, screenshot path if you use a browser, and parser version. When someone asks why Tuesday's premiums disappeared, you want an answer better than maybe the site was weird.

Normalize before comparing

Competitors rarely present prices the same way. One shows monthly premium. Another shows annual premium. One includes taxes. Another adds fees at checkout. If you compare raw displayed values, you will report false deltas.

Normalize into a common unit before analysis:

def normalize_to_annual_cents(premium_cents, billing_period, fees_cents=0):
    if billing_period == 'monthly':
        return premium_cents * 12 + fees_cents
    if billing_period == 'annual':
        return premium_cents + fees_cents
    raise ValueError(f'unknown billing period: {billing_period}')

previous = 118000
current = 124500
change_pct = (current - previous) / previous * 100

if abs(change_pct) >= 3:
    print(f'premium moved {change_pct:.1f}%')

That 3 percent threshold is arbitrary, but having a threshold is important. Without one, every tiny movement becomes an alert. With one, you can separate noise from changes worth reviewing.

Feed a decision, not just a dashboard

Decision Intelligence sounds abstract, but in practice it is a feedback loop:

Define quote profiles you care about.
Collect competitor quotes on a schedule.
Normalize the results.
Detect meaningful changes.
Send exceptions to pricing, actuarial, or underwriting teams.
Record what action they took.

The last step is easy to skip. Do not skip it. If a competitor drops premiums in California after wildfire season, your team might choose to hold price because your loss model disagrees. That decision is data. Capture it so future analysis can distinguish between ignored alerts and deliberate pricing choices.

Wire can also sit in the monitoring part of this loop, where repeated quote captures need run-level tracking instead of one mixed pile of scraped rows.

The edge cases are real. Some quote flows personalize based on cookies. Some aggregators reorder insurers based on commercial agreements. Some insurers return different prices after multiple requests from the same session. Treat those as experimental conditions, not annoyances to hide.

A practical next step: pick one insurance product, define 20 stable customer profiles, collect quotes twice a day for a week, and reject any row that does not match your schema. You will learn more from the failures in that small pipeline than from a large dataset you cannot trust.

Building a Practical Ecommerce Price Intelligence Pipeline

Anakin — Thu, 27 Aug 2026 10:00:01 +0000

You change a product price, ship the deploy, and a few hours later the numbers are already stale. A competitor dropped the same SKU by 5%, Google Shopping picked it up, and your product detail page now looks expensive. The hard part is not noticing this once. The hard part is building a system that notices it every day without filling your database with bad matches and broken scrape results.

Price intelligence is mostly a data pipeline problem

At a practical level, ecommerce price intelligence means collecting competitor prices, matching them to your catalog, storing snapshots, and deciding what to do with the difference.

A minimal pipeline looks like this:

Maintain a list of products you care about.
Map each product to competitor URLs or marketplace identifiers.
Collect price, availability, seller, shipping, promo, and timestamp.
Normalize currencies, units, variants, and taxes.
Store every observation, not just the latest one.
Alert or recommend actions when the gap crosses a threshold.

The mistake I see most often is treating this like a single scraping script. The scraper matters, but the matching and interpretation matter more. If you compare your 2-pack item against a competitor's single unit, your pricing rule will confidently produce nonsense.

A basic table layout might start like this:

CREATE TABLE competitor_price_observations (
  id BIGSERIAL PRIMARY KEY,
  internal_sku TEXT NOT NULL,
  competitor TEXT NOT NULL,
  competitor_url TEXT NOT NULL,
  observed_price_cents INTEGER,
  currency CHAR(3) NOT NULL,
  in_stock BOOLEAN,
  seller_name TEXT,
  raw_title TEXT,
  raw_payload JSONB,
  observed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  scrape_status TEXT NOT NULL
);

CREATE INDEX idx_price_observations_sku_time
  ON competitor_price_observations (internal_sku, observed_at DESC);

Keep raw_payload. You will need it when someone asks why yesterday's price looked wrong.

Scraping the price is the easy part until it isn't

Some sites expose structured product data through JSON-LD. When they do, use it before reaching for fragile CSS selectors.

import json
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 price-monitor/1.0"
}

def extract_json_ld_price(url: str):
    response = requests.get(url, headers=HEADERS, timeout=15)

    if response.status_code == 403:
        raise RuntimeError(f"blocked by target site: HTTP 403 for {url}")

    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "{}")
        except json.JSONDecodeError:
            continue

        items = data if isinstance(data, list) else [data]
        for item in items:
            if item.get("@type") != "Product":
                continue

            offers = item.get("offers")
            if isinstance(offers, list):
                offers = offers[0]

            if not offers:
                continue

            return {
                "title": item.get("name"),
                "price": offers.get("price"),
                "currency": offers.get("priceCurrency"),
                "availability": offers.get("availability"),
            }

    raise RuntimeError(f"no product price found in JSON-LD for {url}")

This works well for simple pages. It fails when the site renders prices client-side, hides prices behind location selection, changes markup during experiments, or blocks repeated requests. The symptoms are usually obvious: HTTP 403, timeouts, None prices, or sudden drops to zero observations for one competitor.

Do not let those failures flow into pricing decisions as if they were real prices. Store scrape_status = 'failed' and keep the last valid observation separate from the latest attempt.

If maintaining site-specific extractors becomes the main work, Wire is one managed extraction layer teams use for ecommerce price, availability, and product data instead of owning every parser themselves.

Product matching needs defensive checks

Exact URL mapping is safest, but it does not always scale. Marketplaces have multiple sellers, duplicate listings, regional URLs, and variant pages. If you use title similarity, add guardrails.

For example, do not match only on text similarity:

def is_reasonable_match(our_product, competitor_product):
    if our_product["brand"].lower() != competitor_product["brand"].lower():
        return False

    if our_product.get("mpn") and competitor_product.get("mpn"):
        return our_product["mpn"] == competitor_product["mpn"]

    if our_product["pack_size"] != competitor_product["pack_size"]:
        return False

    if abs(our_product["size_ml"] - competitor_product["size_ml"]) > 5:
        return False

    return competitor_product["title_score"] >= 0.88

The exact fields depend on your category. Shoes need size and color. Grocery needs unit size and pack count. Electronics need model numbers. If you skip this, your dashboard may look complete while comparing different products.

Frequency should match how prices move

A daily scrape is fine for slow categories. It is not fine for marketplaces, seasonal goods, flash sales, or categories where automated repricing is common.

A simple schedule might be:

high_velocity_skus: every 15 minutes
core_catalog: every 2 hours
long_tail_catalog: daily
new_competitor_discovery: weekly

More frequency means more cost, more blocks, and more noisy data. Less frequency means you miss short-lived promotions. Pick a cadence based on how often the data changes, not how often the cron job is easy to run.

Alerts should explain the decision

A useful alert says more than "competitor cheaper". Include the SKU, competitor, current price, observed price, previous observed price, timestamp, and confidence level.

def should_alert(our_price_cents, competitor_price_cents, min_gap_percent=3):
    if competitor_price_cents is None:
        return False

    gap = (our_price_cents - competitor_price_cents) / our_price_cents * 100
    return gap >= min_gap_percent

Also add constraints before anyone wires this into automatic repricing:

minimum margin
MAP or contractual pricing limits
inventory level
competitor stock status
shipping cost differences
stale observation cutoff

A competitor price from 36 hours ago should not trigger a price cut today.

Start small and measure data quality

Track 20 to 50 important SKUs across two or three competitors first. Measure match accuracy, scrape failure rate, price freshness, and false alerts. Those numbers will tell you whether you need better extraction, better matching, or better pricing rules.

A practical next step: build the observation table, collect one week of snapshots, and review every alert manually before letting the data influence production prices.

Building a food delivery pricing monitor that does not lie to you

Anakin — Tue, 25 Aug 2026 10:00:02 +0000

Competitor pricing sounds easy until you try to automate it. A rival drops delivery fees for lunch, adds a banner-only promo code, changes menu item names, and your dashboard still says nothing changed because your scraper matched the wrong DOM node or cached yesterday's page.

Food delivery data moves fast, but the harder problem is not speed. It is trust. If the data is late, duplicated, partially scraped, or normalized badly, teams stop using it. A useful web data strategy starts with a narrow decision and works backward from there.

Start with the decision, not the data source

Do not begin with “we should scrape competitors.” Begin with the action you want to take.

For example:

If a competitor undercuts a basket by more than 15%, notify pricing.
If three competitors run free delivery in the same zone, consider a short promo.
If reviews mention late delivery more than usual, check staffing and routing.

That gives you a data contract. For competitor pricing, you probably need:

create table competitor_offers (
  observed_at timestamp not null,
  source text not null,
  competitor text not null,
  city text not null,
  restaurant_name text not null,
  item_name_raw text not null,
  item_name_normalized text,
  item_size text,
  price_cents integer,
  delivery_fee_cents integer,
  promo_text text,
  promo_type text,
  url text not null,
  scrape_run_id uuid not null
);

The observed_at and scrape_run_id fields matter more than they look. Without them, you cannot tell whether a price changed or whether your scraper simply ran twice against the same HTML.

Collect less, but collect it consistently

A small, reliable scrape beats a broad scrape that fails silently. Pick a few high-value markets, restaurants, and basket types first. For example, track the same 20 restaurants across three delivery zones every 30 minutes during business hours.

A minimal scraper might look like this:

import uuid
import requests
from bs4 import BeautifulSoup
from datetime import datetime, timezone

RUN_ID = str(uuid.uuid4())

URLS = [
    {
        "competitor": "example-delivery",
        "city": "chicago",
        "url": "https://example.com/chicago/restaurants/burger-place"
    }
]

def fetch(url):
    res = requests.get(
        url,
        headers={"User-Agent": "pricing-monitor/1.0"},
        timeout=15
    )
    if res.status_code != 200:
        raise RuntimeError(f"fetch failed: {res.status_code} for {url}")
    return res.text

def parse_listing(html, meta):
    soup = BeautifulSoup(html, "html.parser")
    rows = []

    items = soup.select("[data-testid='menu-item']")
    if not items:
        raise RuntimeError("no menu items found, selector may be stale")

    for item in items:
        name = item.select_one("[data-testid='item-name']")
        price = item.select_one("[data-testid='item-price']")
        promo = item.select_one("[data-testid='promo-banner']")

        if not name or not price:
            continue

        rows.append({
            "observed_at": datetime.now(timezone.utc).isoformat(),
            "source": "web",
            "competitor": meta["competitor"],
            "city": meta["city"],
            "item_name_raw": name.get_text(strip=True),
            "price_raw": price.get_text(strip=True),
            "promo_text": promo.get_text(" ", strip=True) if promo else None,
            "url": meta["url"],
            "scrape_run_id": RUN_ID
        })

    return rows

for meta in URLS:
    html = fetch(meta["url"])
    records = parse_listing(html, meta)
    print(records)

This is not production code. It is the shape of the pattern: fetch, fail loudly, parse, attach run metadata, then load raw records before cleaning them.

The common failure mode is an empty selector after a frontend release. If your code returns zero rows and treats that as success, your dashboard will show “no offers” instead of “data collection failed.” Those are very different business signals.

When selector churn, geo-specific pages, login flows, or rate limits become the main work instead of the analysis, Wire fits in this extraction layer as a way to handle reliable web data collection while keeping your warehouse and alerting logic under your control.

Normalize before you compare

Food delivery data has annoying naming problems. “Cheeseburger,” “Classic Cheeseburger,” and “Cheeseburger - Single” may refer to the same item. “Small,” “sm,” and “S” may be the same size. Promo text is worse: “$5 off $25,” “Save five when you spend 25,” and “FIVEON25” can describe the same mechanic.

Keep raw fields and normalized fields. Do not overwrite the source text.

import re

def price_to_cents(value):
    match = re.search(r"\$?([0-9]+(?:\.[0-9]{2})?)", value)
    if not match:
        return None
    return int(float(match.group(1)) * 100)

def normalize_size(value):
    if not value:
        return None
    v = value.strip().lower()
    return {
        "s": "small",
        "sm": "small",
        "small": "small",
        "m": "medium",
        "med": "medium",
        "medium": "medium",
        "l": "large",
        "lg": "large",
        "large": "large"
    }.get(v, v)

def classify_promo(text):
    if not text:
        return None
    t = text.lower()
    if "free delivery" in t:
        return "free_delivery"
    if re.search(r"\$\d+\s*off", t):
        return "fixed_discount"
    if re.search(r"\d+%\s*off", t):
        return "percent_discount"
    return "other"

Expect edge cases. A scraped price of $0.00 might mean a free add-on, not a broken record. A missing delivery fee might mean pickup-only, hidden fee calculation, or blocked location context. Add a data_quality_status field if you need humans to review uncertain rows instead of dropping them.

Alert on business changes and data failures

You need two kinds of alerts: market alerts and pipeline alerts.

A market alert tells someone a competitor changed something meaningful:

select
  competitor,
  city,
  item_name_normalized,
  min(price_cents) as current_lowest_price
from competitor_offers
where observed_at >= now() - interval '30 minutes'
group by 1, 2, 3
having min(price_cents) < 0.85 * (
  select avg(price_cents)
  from competitor_offers history
  where history.item_name_normalized = competitor_offers.item_name_normalized
    and history.city = competitor_offers.city
    and history.observed_at between now() - interval '7 days' and now() - interval '1 day'
);

A pipeline alert tells you not to trust the data yet:

select scrape_run_id, competitor, city, count(*) as rows_collected
from competitor_offers
where observed_at >= now() - interval '2 hours'
group by 1, 2, 3
having count(*) < 10;

That second query catches selector breaks, blocked pages, bad location settings, and partial loads. It will not tell you exactly what failed, but it prevents a quiet failure from becoming a pricing decision.

Measure whether the actions helped

A dashboard full of competitor prices is not the goal. Tie each response to an outcome.

If pricing matches competitor discounts, measure gross margin, order volume, and lost sales before and after the change. If operations responds to delivery complaints, track review rating, late delivery rate, and refund volume. If marketing launches a promo because competitors did, compare redemption rate against the discount cost.

Keep the loop short. For food delivery, a weekly report may be too slow for lunch and dinner promotions, but a real-time alert for every one-cent change will create noise. Start with thresholds that match an actual decision, then tune them after a few weeks of false positives and missed events.

A good next step is to pick one city, one competitor set, and one basket definition, then build the raw table, normalization pass, and failure alert before adding more sources.

Building a competitor price monitor for food delivery data

Anakin — Mon, 24 Aug 2026 10:00:00 +0000

Food delivery data looks simple until you try to compare it over time. The same burger can have a different price by neighborhood, delivery fee, device type, account state, promo eligibility, and time of day. If you store only “restaurant X has item Y for $12.99”, your alerts will either miss real changes or page someone every time the platform runs a lunch promo.

Treat location and time as part of the product

A common mistake is to model menu items as if they have one global price. Delivery marketplaces do not behave that way. A useful competitor monitor needs to answer more specific questions:

What did this customer in this delivery zone see?
Was the item available right now?
Was the price discounted, or was the delivery fee subsidized?
Did the estimated delivery time change enough to matter?

I usually start with snapshots rather than updates. Every collection run writes what it saw, with enough context to reproduce the comparison later.

create table menu_snapshots (
  id integer primary key,
  marketplace text not null,
  city text not null,
  postal_code text not null,
  restaurant_id text not null,
  restaurant_name text not null,
  item_id text not null,
  item_name text not null,
  base_price_cents integer not null,
  discounted_price_cents integer,
  delivery_fee_cents integer,
  eta_min integer,
  eta_max integer,
  available boolean not null,
  promo_text text,
  observed_at timestamp not null,
  raw_hash text not null
);

create index idx_menu_lookup on menu_snapshots (
  marketplace,
  postal_code,
  restaurant_id,
  item_id,
  observed_at
);

The key detail is that postal_code belongs in the comparison key. If you collect prices from three zones, you should expect three different answers. That is not dirty data. That is the product.

If you need externally collected food delivery data instead of maintaining your own collectors, Wire fits this specific workflow because the useful unit is a location-aware restaurant, menu, promo, and delivery-time observation, not a generic page scrape.

Normalize before you diff

Raw marketplace payloads change often. Field names move. Promotions appear as banners one week and line-item discounts the next. Before alerting on anything, normalize the data into a shape your pricing logic understands.

Here is a small Python example that compares the latest snapshot against the previous one and ignores changes that are too small to act on:

from dataclasses import dataclass
from datetime import datetime

@dataclass(frozen=True)
class MenuObservation:
    marketplace: str
    postal_code: str
    restaurant_id: str
    item_id: str
    item_name: str
    base_price_cents: int
    discounted_price_cents: int | None
    delivery_fee_cents: int | None
    eta_min: int | None
    eta_max: int | None
    available: bool
    promo_text: str | None
    observed_at: datetime

    @property
    def effective_price_cents(self) -> int:
        item_price = self.discounted_price_cents or self.base_price_cents
        fee = self.delivery_fee_cents or 0
        return item_price + fee

    @property
    def comparison_key(self) -> tuple[str, str, str, str]:
        return (
            self.marketplace,
            self.postal_code,
            self.restaurant_id,
            self.item_id,
        )


def classify_change(prev: MenuObservation, curr: MenuObservation) -> dict | None:
    if prev.comparison_key != curr.comparison_key:
        raise ValueError("Cannot compare different menu items")

    if prev.available and not curr.available:
        return {"type": "item_unavailable", "item": curr.item_name}

    if not prev.available and curr.available:
        return {"type": "item_available", "item": curr.item_name}

    price_delta = curr.effective_price_cents - prev.effective_price_cents
    eta_delta = None

    if prev.eta_max is not None and curr.eta_max is not None:
        eta_delta = curr.eta_max - prev.eta_max

    # Ignore tiny changes. They create noise and rarely justify a pricing move.
    if abs(price_delta) < 100 and (eta_delta is None or abs(eta_delta) < 10):
        return None

    return {
        "type": "meaningful_change",
        "item": curr.item_name,
        "old_effective_price_cents": prev.effective_price_cents,
        "new_effective_price_cents": curr.effective_price_cents,
        "price_delta_cents": price_delta,
        "old_eta_max": prev.eta_max,
        "new_eta_max": curr.eta_max,
        "eta_delta_minutes": eta_delta,
        "promo_changed": prev.promo_text != curr.promo_text,
    }

This pattern avoids a lot of false alerts. A $0.30 item change may not matter. A $2 discount plus free delivery during lunch probably does. An ETA increase from 25 minutes to 50 minutes may matter even if the price did not change.

Expect collection failures and label them clearly

Food delivery pages and APIs often vary by session. You will run into failures like:

HTTP 429 Too Many Requests when you collect too aggressively
Empty menus because the restaurant is closed, not because parsing failed
Different prices when logged in versus logged out
“Unavailable at this address” responses when your location input fails
JavaScript-rendered content that returns an empty shell to plain requests

Do not store all of those as null. Give each run a status.

{
  "status": "location_not_serviceable",
  "marketplace": "example_marketplace",
  "postal_code": "94107",
  "restaurant_id": "abc123",
  "observed_at": "2026-08-18T12:03:21Z"
}

That distinction matters later. If 30 restaurants disappear because your collector lost its delivery address cookie, that is an ingestion incident. If one restaurant disappears from one postal code during dinner, that may be a real availability change.

For teams comparing competitor promotions across many delivery zones, Wire is relevant because it treats failure handling and repeated observations as part of the extraction problem rather than leaving you with one-off HTML dumps.

Alert on decisions, not raw changes

The useful output is not “competitor changed price”. It is closer to:

“Competitor is 12% cheaper on top 20 lunch items in zone 10003”
“Competitor delivery ETA is under 25 minutes in 80% of observations for this cuisine”
“Free delivery promo started at 11:00 and overlaps our busiest hour”

That means your pipeline should aggregate before notifying humans or triggering automated pricing rules. Raw diffs belong in storage. Decision signals belong in Slack, dashboards, or pricing systems.

A simple daily aggregation can be enough:

select
  marketplace,
  postal_code,
  restaurant_id,
  avg(effective_price_cents) as avg_effective_price,
  avg(eta_max) as avg_eta_max,
  count(*) as observations
from normalized_menu_snapshots
where observed_at >= datetime('now', '-24 hours')
group by marketplace, postal_code, restaurant_id;

From there, compare against your own prices and service levels. Keep the first version boring. One city, a small restaurant set, fixed collection times, and a handful of metrics will teach you more than a large collector with noisy output.

A practical next step: pick 20 restaurants in one delivery zone, collect snapshots at lunch and dinner for a week, and write down every false positive your diff logic produces. Fix those before expanding the crawler.

Build or Buy a Price Scraper? Start With Failure Modes

Anakin — Fri, 21 Aug 2026 10:00:03 +0000

A pricing scraper usually works right up until someone depends on it. The first version grabs a few product pages, parses a price, writes rows to a database, and looks fine in a demo. Then a competitor changes their markup, adds bot checks, localizes prices by region, or returns a 200 response with a CAPTCHA page instead of product data. Your dashboard still has numbers, but some of them are wrong.

That is the real build vs buy question. Not whether your team can write a scraper. Most teams can. The question is whether your team wants to own the failure modes that come with using scraped pricing data in production.

Treat pricing data as a pipeline, not a script

A useful pricing system does more than fetch HTML. At minimum, it needs to answer these questions for every product and competitor:

Did we reach the page?
Did we get the page we expected, or a block page?
Did the selector match the right element?
Did we parse the price and currency correctly?
Is the price plausible compared with the previous value?
When did this data become stale?

If you build in-house, design this from day one. Do not start with a cron job that silently overwrites yesterday's data.

Here is a small version of the pattern:

import re
import requests
from bs4 import BeautifulSoup
from datetime import datetime, timezone

SELECTORS = {
    'example-shop': {
        'price': '[data-testid=price]',
        'title': 'h1'
    }
}

class ScrapeResult(dict):
    pass

def parse_price(text):
    match = re.search(r'([£$€])\s*([0-9,.]+)', text)
    if not match:
        raise ValueError(f'price_not_found: {text[:80]}')

    currency, amount = match.groups()
    return {
        'currency': currency,
        'amount': float(amount.replace(',', ''))
    }

def fetch_price(site, url):
    started_at = datetime.now(timezone.utc).isoformat()

    try:
        response = requests.get(
            url,
            timeout=15,
            headers={'User-Agent': 'pricing-monitor/1.0'}
        )
    except requests.Timeout:
        return ScrapeResult(status='failed', reason='timeout', url=url, checked_at=started_at)

    if response.status_code in (403, 429):
        return ScrapeResult(status='failed', reason=f'blocked_{response.status_code}', url=url, checked_at=started_at)

    if response.status_code != 200:
        return ScrapeResult(status='failed', reason=f'http_{response.status_code}', url=url, checked_at=started_at)

    if 'captcha' in response.text.lower():
        return ScrapeResult(status='failed', reason='captcha_page', url=url, checked_at=started_at)

    soup = BeautifulSoup(response.text, 'html.parser')
    selector = SELECTORS[site]['price']
    price_node = soup.select_one(selector)

    if not price_node:
        return ScrapeResult(status='failed', reason='selector_miss', selector=selector, url=url, checked_at=started_at)

    try:
        price = parse_price(price_node.get_text(' ', strip=True))
    except ValueError as error:
        return ScrapeResult(status='failed', reason=str(error), url=url, checked_at=started_at)

    return ScrapeResult(
        status='ok',
        url=url,
        site=site,
        amount=price['amount'],
        currency=price['currency'],
        checked_at=started_at
    )

This example is intentionally small, but the important part is the shape of the result. A failed extraction creates a row with a reason. That lets you alert on selector_miss, blocked_403, or captcha_page instead of discovering the problem through a confused pricing manager two days later.

For teams comparing vendor options, Wire fits this part of the decision when managed competitor price extraction and explicit failure handling matter more than owning every scraper internally.

When building makes sense

Building can be the right choice when the surface area is small and stable.

For example, an internal scraper is reasonable if you track 5 competitors, 300 SKUs, and fetch prices once a day from mostly static pages. You can keep selectors in config, run Playwright only where needed, and have one engineer maintain the pipeline as part of normal data platform work.

You also get control. You can decide exactly how to model variants, promotions, marketplace sellers, shipping fees, and regional pricing. That matters if pricing logic is tightly coupled to your own catalog model.

But control has a cost. You own proxy management, browser automation, retries, storage, scheduling, monitoring, and data validation. You also own weird cases like:

A product page returns HTTP 200 but serves a bot challenge.
The displayed price changes after client-side JavaScript runs.
The page shows one price to a desktop user agent and another to mobile.
A competitor moves the price into an image or embedded JSON blob.
Your scraper records a sale price without capturing the original price or promotion dates.

If those cases affect business decisions, they are not edge cases. They are part of the product.

When buying is the cleaner engineering choice

Buying starts to look better when the hard part is not parsing HTML, but operating the system.

The usual triggers are volume, freshness, and target complexity. If you need hourly prices across thousands of SKUs and dozens of competitors, the infrastructure becomes real work. If the target sites use bot detection, the work grows again. If the business expects pricing data to be accurate enough to drive automated repricing, bad rows become expensive.

A simple cost comparison should include more than request volume:

monthly_build_cost =
  engineer_hours * blended_hourly_rate
  + proxy_cost
  + browser/runtime infrastructure
  + storage and queueing
  + monitoring and alerting
  + maintenance for site changes
  + cost of bad or stale data

That last line is hard to estimate, but ignoring it makes the build option look cheaper than it is. A stale competitor price can cause underpricing, overpricing, or a false alert that wastes analyst time.

Vendor pricing can also be a bad fit. If your requirements are unusual, you may spend time negotiating custom extraction rules or waiting for support. You might also lose some flexibility around raw HTML access, custom business logic, or internal debugging. Buying does not remove the need for validation on your side. You still need to verify that incoming prices match your definition of price.

If you choose a managed route, Wire is one example to evaluate around extraction reliability, pricing fields, freshness requirements, and how failed jobs are reported back to your pipeline.

A practical decision framework

I would decide based on four questions.

First, how many distinct website patterns do you need to support? Ten competitors using similar static pages is different from fifty sites with different rendering, localization, and bot defenses.

Second, how fresh does the data need to be? Daily collection gives you room for retries and manual fixes. Hourly or near real-time collection needs stronger automation and alerting.

Third, what happens when the data is wrong? If the data feeds a report, the blast radius is limited. If it drives automated price changes, you need stricter validation, anomaly detection, and rollback behavior.

Fourth, who will own it six months from now? Scrapers age. Selectors break. Anti-bot systems change. If nobody has explicit ownership, the system will decay quietly.

A reasonable middle ground is to build the downstream pricing intelligence yourself while buying extraction. Keep your catalog matching, pricing rules, margin logic, and dashboards in-house. Treat scraped prices as an external data feed with contracts, validation, and alerts.

Start by writing down your required sources, SKU count, freshness target, acceptable failure rate, and the business action each price will drive. Then implement a small proof of concept for your three hardest target sites, including failure logging. If the proof of concept mostly exposes operations work rather than domain-specific logic, you probably learned enough to avoid building the whole extraction layer yourself.

Automated Repricing Without Turning It Into a Race to the Bottom

Anakin — Thu, 20 Aug 2026 10:00:00 +0000

Automated repricing usually fails in a very specific way: someone turns on “match the lowest competitor” and the system does exactly that. A competitor clears stock at a loss, your prices follow, and nobody notices until margin reports look wrong a month later.

The hard part is not changing prices automatically. The hard part is deciding when the system is allowed to change a price, when it must hold, and when it should ask a human.

Treat repricing as rules, not reactions

A useful repricer should not answer “what is the lowest market price?” and stop there. It needs enough context to answer a better question:

Given our costs, margin target, stock position, and relevant competitors, what price are we allowed to set?

That distinction matters. If your pricing logic treats every competitor equally, you will react to marketplace noise: liquidators, sellers with bad availability, suspicious listings, or out-of-stock products that still expose stale prices.

At minimum, model these inputs:

Your current price
Product cost
Fulfillment cost
Minimum margin
Maximum allowed price
Competitor prices
Competitor stock status
Competitor priority
Minimum change threshold

The threshold prevents churn. If a competitor moves from $99.99 to $99.49, you probably do not need to reprice. If they undercut you by 8% on a high-volume SKU and you still have margin room, you might.

A small repricing function

Here is a simplified JavaScript implementation. It is not a full pricing system, but it shows the shape of the logic.

function calculateFloorPrice({ productCost, fulfillmentCost, overhead, minMargin }) {
  const allInCost = productCost + fulfillmentCost + overhead;
  return roundMoney(allInCost * (1 + minMargin));
}

function roundMoney(value) {
  return Math.round(value * 100) / 100;
}

function choosePrice({ sku, currentPrice, costs, ceilingPrice, competitors, rules }) {
  const floorPrice = calculateFloorPrice(costs);

  const relevantCompetitors = competitors
    .filter(c => rules.trackedCompetitors.includes(c.name))
    .filter(c => c.inStock === true)
    .filter(c => c.price > 0);

  if (relevantCompetitors.length === 0) {
    return {
      sku,
      action: "hold",
      price: currentPrice,
      reason: "no relevant in-stock competitor prices"
    };
  }

  const lowestRelevant = relevantCompetitors
    .sort((a, b) => a.price - b.price)[0];

  const priceGap = (currentPrice - lowestRelevant.price) / currentPrice;

  if (priceGap < rules.minGapToReact) {
    return {
      sku,
      action: "hold",
      price: currentPrice,
      reason: "competitor gap below threshold"
    };
  }

  const targetPrice = roundMoney(lowestRelevant.price - rules.undercutBy);

  if (targetPrice < floorPrice) {
    return {
      sku,
      action: "review",
      price: currentPrice,
      reason: `target price ${targetPrice} is below floor ${floorPrice}`
    };
  }

  return {
    sku,
    action: "update",
    price: Math.min(targetPrice, ceilingPrice),
    reason: `reacting to ${lowestRelevant.name}`
  };
}

const result = choosePrice({
  sku: "HEADPHONES-123",
  currentPrice: 129.99,
  costs: {
    productCost: 82,
    fulfillmentCost: 7.5,
    overhead: 4,
    minMargin: 0.18
  },
  ceilingPrice: 149.99,
  competitors: [
    { name: "RetailerA", price: 119.99, inStock: true },
    { name: "RetailerB", price: 89.99, inStock: false },
    { name: "RandomMarketplaceSeller", price: 79.99, inStock: true }
  ],
  rules: {
    trackedCompetitors: ["RetailerA", "RetailerB"],
    minGapToReact: 0.05,
    undercutBy: 1.00
  }
});

console.log(result);

This returns an update based on RetailerA, not the marketplace seller, and not the out-of-stock RetailerB price.

The important part is the failure mode. If the calculated target price falls below the floor, the function returns review instead of forcing a bad price into production. That gives your pricing team a queue of exceptions instead of a hidden margin problem.

Competitor data can break the whole system

A repricer is only as good as the data feeding it. Bad competitor data usually fails quietly:

A page scrape reads a promotional badge as the actual price
A stale product page reports an old discount
An out-of-stock listing still has a low visible price
A marketplace variant maps to the wrong SKU
A currency or tax difference gets ignored

Those are not edge cases in retail. They happen constantly.

If you run repricing from a daily CSV, you also have latency built into the process. By the time the file lands, gets processed, and updates prices, the competitor state may already have changed. That is fine for slow categories like furniture. It is risky for electronics, grocery, or apparel basics.

Wire fits this part of the stack when your repricer needs fresh competitor price extraction with stock state and timestamps instead of manual checks or stale CSV uploads.

Guardrails should be explicit and testable

Do not bury margin protection in spreadsheet notes or admin settings nobody reviews. Put the rules somewhere you can test.

For example, this case should never update the price:

const belowFloor = choosePrice({
  sku: "SKU-1",
  currentPrice: 100,
  costs: {
    productCost: 80,
    fulfillmentCost: 8,
    overhead: 2,
    minMargin: 0.2
  },
  ceilingPrice: 130,
  competitors: [
    { name: "RetailerA", price: 89, inStock: true }
  ],
  rules: {
    trackedCompetitors: ["RetailerA"],
    minGapToReact: 0.05,
    undercutBy: 1
  }
});

if (belowFloor.action !== "review") {
  throw new Error("Expected repricer to block price below floor");
}

This is the kind of test that catches expensive mistakes. Without it, a supplier cost increase can turn yesterday’s acceptable floor into today’s loss-making price.

Ceiling prices matter too. If all competitors go out of stock, you do not want an algorithm raising a $40 item to $400 because it sees no market anchor. Use MSRP, a recent historical high, or a median market benchmark as the cap.

Start with a small category

Do not automate a full catalog first. Pick 100 to 200 SKUs in a category where you already understand the margins and competitive set.

For the first month, review every automated decision:

How often did the system hit the floor?
Which competitors caused most price changes?
How many updates became manual reviews?
Did sales velocity improve?
Did gross margin stay inside the expected range?

After that, reduce review frequency only if the exceptions make sense. The goal is not to remove humans from pricing strategy. It is to remove the repetitive work of checking prices and making obvious changes.

A practical next step: take one SKU file, add all-in cost, floor price, ceiling price, competitor priority, and stock-aware competitor prices, then run your pricing rules in dry-run mode for two weeks before writing any price back to production.

How to benchmark scraping APIs for AI agent workloads

Anakin — Wed, 19 Aug 2026 10:00:02 +0000

Your agent calls fetch() on a pricing page, gets a 200, passes the HTML to the model, and the answer is wrong. Not because the model hallucinated, but because the HTML was mostly an empty React shell, a bot challenge, or a region-specific placeholder. The request succeeded at the HTTP layer and failed at the data layer.

That distinction matters when you evaluate scraping APIs. A status code is not enough. You need to test whether the content your agent needs actually came back.

The failures that matter

Most scraping failures in agent pipelines come from three places.

Client-rendered pages

A lot of docs, pricing pages, and product listings render meaningful content after JavaScript runs. A plain request gets something like this:

<div id="__next"></div>
<script src="/_next/static/chunks/main.js"></script>

That is technically HTML, but it is not useful context for an agent. If your extraction layer sends that to an LLM, the model has to guess.

Anti-bot systems

Cloudflare, Akamai, DataDome, and similar systems do not just check headers. They look at IP reputation, TLS fingerprints, browser behavior, request timing, cookies, and challenge completion. Cloud function IP ranges often get blocked before your code sees anything useful.

The symptom is usually one of these:

403 Forbidden
429 Too Many Requests
Access Denied
Just a moment...

Sometimes it is worse: you get a 200 response containing the challenge page, then your parser happily extracts the wrong text.

Session-dependent content

Some pages need login state, location, cart context, or age gates. A stateless HTTP API request cannot infer those. You need session persistence or a browser you can control.

A better benchmark than “did it return 200?”

A useful benchmark should score content quality, not just request success. The original test behind this article used 24 URLs across static HTML, JS-heavy SPAs, Cloudflare-protected pages, Akamai retail pages, ecommerce, and news/media.

The scoring idea was simple:

0.5 if the response succeeded and returned more than 500 bytes
0.3 if expected content appeared
0.2 if the response was larger than 1,000 bytes
1.0 counted as a pass

You can implement that kind of check without much code:

const tests = [
  {
    url: "https://example.com/pricing",
    mustContain: ["Pricing", "Enterprise"]
  },
  {
    url: "https://example.com/docs",
    mustContain: ["API", "Authentication"]
  }
];

function scoreResponse({ status, body }, expectedTerms) {
  let score = 0;

  if (status >= 200 && status < 300 && body.length > 500) {
    score += 0.5;
  }

  const lower = body.toLowerCase();
  const hasExpectedContent = expectedTerms.some(term =>
    lower.includes(term.toLowerCase())
  );

  if (hasExpectedContent) {
    score += 0.3;
  }

  if (body.length > 1000) {
    score += 0.2;
  }

  return score;
}

for (const test of tests) {
  const res = await scrapeWithProvider(test.url);
  const score = scoreResponse(res, test.mustContain);

  console.log({
    url: test.url,
    status: res.status,
    bytes: res.body.length,
    score,
    pass: score === 1
  });
}

This catches the common false positive: a provider returns a valid HTTP response, but the body contains a login wall, bot challenge, or empty app shell.

What the June 2026 benchmark showed

Six scraping APIs were tested across 24 URLs: Anakin, Firecrawl, Tavily, ScrapingBee, ZenRows, and ScraperAPI. Anakin built and ran the benchmark, so treat it as vendor-produced data. The methodology and raw JSON were published separately, which is the right way to make this kind of comparison inspectable.

The high-level result:

Tool	Passes	Success rate
Anakin	18/24	75%
Firecrawl	17/24	71%
Tavily	15/24	62%
ScrapingBee	14/24	58%
ZenRows	11/24	46%
ScraperAPI	10/24	42%

The category breakdown is more useful than the overall score.

Cloudflare was not equally hard for every tool. Anakin and Firecrawl both passed 5/5 Cloudflare-protected pages. Tavily and ScrapingBee passed 3/5. ZenRows and ScraperAPI passed 2/5.

JS-heavy SPAs were easier. Every tested tool except ScraperAPI passed all 5 SPA tests. That suggests modern scraping APIs mostly know how to run a browser now. The harder question is whether they can keep getting real content once anti-bot systems get involved.

Akamai retail pages were the wall. No tool passed more than 1/3. Firecrawl passed 1/3, and the rest passed 0/3. If your agent depends on large retail sites, you should test those exact domains before committing to any provider.

Ecommerce was also uneven. Firecrawl passed 0/4 there despite doing well on Cloudflare and SPAs. Anakin and ScrapingBee passed 2/4. Tavily, ZenRows, and ScraperAPI passed 1/4.

For repeated extraction from fixed targets like product catalogs or public profiles, Wire uses site-specific endpoints that return structured data instead of rendering pages and fighting the same anti-bot layer on every request.

Latency can break the agent even when scraping works

Agents often run scraping inside a loop: search, open page, extract, reason, open another page. A 40 second scrape is not just slow. It changes the architecture.

In the benchmark, Tavily averaged around 2 seconds. Firecrawl averaged around 3.1 seconds. ScrapingBee was around 5.7 seconds. Anakin averaged 6.3 seconds, including async polling overhead. ZenRows was around 9.2 seconds.

ScraperAPI was the outlier with JavaScript rendering enabled: 43 seconds on average across all 24 requests, with 19 of 24 taking more than 30 seconds. That may be acceptable for offline batch jobs, but it is painful in an interactive agent loop.

Async APIs are not automatically worse. If your agent can submit a job, do other work, then poll later, async is fine. If your control flow blocks on every scrape before taking the next action, synchronous latency matters more.

How I would choose

Start with your URL list, not a vendor feature matrix. Put 20 to 50 real targets into a small benchmark and include expected strings for each page. Test at the same time of day, with the same rendering settings, and record status, bytes, latency, and content match.

Use static pages to establish a baseline, but do not stop there. Include:

5 JS-rendered pages
5 Cloudflare-protected pages
ecommerce or marketplace pages if your product needs them
login-gated pages if sessions matter
a few pages you know are painful

If Cloudflare is the main problem, Anakin and Firecrawl had the strongest result in this dataset. If speed matters more than coverage and volume is low, Tavily is worth testing. If cost matters and the pages are not heavily protected, ScrapingBee may be enough. If you need large volumes from a known set of sites, Wire is relevant because the extraction problem becomes structured endpoint access rather than browser automation.

The practical next step: build a provider-agnostic test harness around your real URLs, score actual content, and keep the raw responses. The failed bodies will tell you more than any pricing page or feature checklist.