DEV Community: Anakin

Browser automation vs network calls for sites without public APIs

Anakin — Mon, 19 Oct 2026 10:00:02 +0000

You need data from a portal that never shipped a public API. The first version is usually a browser script: open the page, log in, click the button, scrape the table. It works until the site ships a harmless UI change and your job starts failing with TimeoutError: locator.click: Timeout 30000ms exceeded because the button text changed.

The real choice is the layer

There are two common ways to automate a site that only exposes a web UI.

Browser automation operates at the presentation layer. Tools like Playwright, Puppeteer, Selenium, and managed platforms such as CloudCruise drive a real browser. They click buttons, type into inputs, wait for navigation, handle downloads, and read data from the DOM.

Network automation operates one layer lower. Instead of clicking the UI, you inspect the requests the frontend already makes and call those endpoints directly. The page might not have a public API, but the JavaScript app usually talks to some backend API.

The difference matters because the failure modes are different.

A browser script depends on the page structure. If the app changes a label, moves a button, delays hydration, or swaps a table for virtualized rows, your script can break even if the underlying business action still works.

A network call depends on request shape, auth, and backend behavior. If the endpoint path changes, a CSRF token expires, the API starts requiring a new header, or the response schema changes, your integration breaks there instead.

For example, Wire is a maintained network-layer catalog for calling site actions directly when the useful behavior sits behind the page rather than in a public API.

What browser automation failure looks like

This is a typical Playwright flow:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();

await page.goto('https://example-portal.test/login');
await page.getByLabel('Email').fill(process.env.PORTAL_USER);
await page.getByLabel('Password').fill(process.env.PORTAL_PASS);
await page.getByRole('button', { name: 'Sign in' }).click();

await page.getByRole('link', { name: 'Reports' }).click();
await page.getByRole('button', { name: 'Download CSV' }).click();

await browser.close();

This can fail even when the portal still works for a person:

TimeoutError: locator.click: Timeout 30000ms exceeded.
Call log:
  - waiting for getByRole('button', { name: 'Download CSV' })

Maybe the product team renamed the button to Export. Maybe the button is now an icon with no accessible name. Maybe a cookie banner covers it. The backend endpoint that generates the CSV may not have changed at all.

The browser approach is still the right tool when you need the actual user flow. Login with SSO, TOTP prompts, healthcare payer portals, EHR systems, visual confirmations, file pickers, and workflows that only exist as UI interactions all fit here. CloudCruise, for example, focuses on managed browser workflows for healthcare portals, including credential handling and 2FA support. That is a real use case, not something you should replace with a fragile copied request.

What network automation failure looks like

If you inspect DevTools while clicking Download CSV, you might find a request like this:

curl 'https://example-portal.test/api/reports/claims?month=2026-01' \
  -H 'authorization: Bearer eyJhbGciOi...' \
  -H 'x-csrf-token: 7d1b0...' \
  -H 'accept: application/json'

A safer implementation checks status codes and schema instead of assuming the request worked:

async function fetchClaimsReport({ month, token, csrf }) {
  const res = await fetch(
    `https://example-portal.test/api/reports/claims?month=${encodeURIComponent(month)}`,
    {
      headers: {
        authorization: `Bearer ${token}`,
        'x-csrf-token': csrf,
        accept: 'application/json'
      }
    }
  );

  if (res.status === 401 || res.status === 403) {
    throw new Error(`auth failed: ${res.status}`);
  }

  if (res.status === 429) {
    throw new Error('rate limited');
  }

  if (!res.ok) {
    throw new Error(`report request failed: ${res.status}`);
  }

  const data = await res.json();

  if (!Array.isArray(data.claims)) {
    throw new Error('unexpected response shape: claims is missing');
  }

  return data.claims;
}

This version avoids DOM breakage, browser startup time, screenshots, and selector maintenance. It also gives you clearer operational errors: 401, 403, 429, or unexpected response shape instead of a generic click timeout.

But it has edges. You must understand authentication. You need to respect rate limits. You may need to refresh CSRF tokens. Some endpoints are intentionally tied to short-lived browser sessions. Some sites prohibit automated access in their terms. Direct calls are not automatically safer or more acceptable just because they are faster.

Long-running actions need a job pattern

Browser jobs and network jobs can both outlive a single HTTP request. A report might take 45 seconds to generate. A search might fan out across multiple pages. Do not keep one request open forever if the caller can lose the connection.

Use a task creation endpoint and poll by job ID:

async function waitForJob(jobId, getStatus, { timeoutMs = 120000 } = {}) {
  const started = Date.now();
  let delay = 1000;

  while (Date.now() - started < timeoutMs) {
    const job = await getStatus(jobId);

    if (job.status === 'succeeded') return job.result;
    if (job.status === 'failed') throw new Error(job.error || 'job failed');

    await new Promise(resolve => setTimeout(resolve, delay));
    delay = Math.min(delay * 1.5, 5000);
  }

  throw new Error(`job timed out: ${jobId}`);
}

Wire models long-running site actions this way: create a task, keep the returned job ID, then poll until the action returns structured JSON or a failure.

How to choose

Use browser automation when:

The workflow requires real user interaction, MFA, SSO, or visual confirmation
The site does not expose a clean backend request for the action
You need screenshots, video replay, or DOM-level debugging
Compliance requirements point you toward a managed browser platform built for that domain

Use network automation when:

The frontend calls a stable JSON endpoint for the data or action you need
You care about latency and concurrency more than visual fidelity
You want structured errors based on HTTP status and response schema
UI changes should not break the integration

For your next portal automation, record one successful manual run with DevTools open. If the important action maps to a clear backend request, prototype the network call and validate auth, rate limits, and response shape. If it does not, write the browser workflow and make selector failure an expected operational case rather than a surprise.

A scraping API should do more than start a scraper you built somewhere else

Anakin — Fri, 16 Oct 2026 10:00:02 +0000

A common trap with scraping tools is assuming that “has an API” means “can be driven entirely from code.” Sometimes it only means the API can start a project you already built in a GUI. That difference matters once scraping becomes part of a backend job, data pipeline, agent workflow, or monitoring system.

The difference between a trigger API and a scraping API

A trigger API wraps an existing scraper. You build the scraper somewhere else, usually by clicking through pages in a desktop or web UI, then your code starts that saved project.

The API flow usually looks like this:

# Start a saved scraping project
curl -X POST "https://api.example.com/v2/projects/PROJECT_TOKEN/run" \
  -d "api_key=$API_KEY"

# Poll for results
curl "https://api.example.com/v2/runs/RUN_TOKEN/data?api_key=$API_KEY"

That can be useful. If a non-developer built a scraper once and wants it scheduled or triggered from another system, this model works fine.

But it has a hard boundary: your code does not define the extraction. It only presses run.

A scraping API should let your code describe the job at runtime:

curl -X POST "https://api.example.com/v1/scrape" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/widget-123",
    "renderJavascript": true,
    "extract": {
      "title": "product name",
      "price": "current price",
      "availability": "stock status"
    }
  }'

That distinction changes how you maintain the system. With the first model, a new URL pattern may require someone to reopen the visual builder and edit selectors. With the second, the extraction contract lives next to the code that consumes the result.

Where the trigger model breaks down

The trigger model usually fails in boring production ways.

A selector changes and the job still “succeeds,” but returns an empty CSV. A site adds a login step and the run starts timing out. A product page template changes and half the rows get shifted into the wrong columns. Your pipeline sees HTTP 200, then discovers that price is missing downstream.

The failure mode is not always an exception. Often it is bad data.

That makes observability harder. If your code only knows “run project 123,” it cannot easily explain which selector failed, which extraction field was missing, or whether the target page changed. You end up debugging in the visual tool instead of in logs, tests, and version control.

For authenticated or repeated site-specific workflows, Wire treats those actions as API-callable units rather than desktop projects, which is the model you want when job inputs, credentials, and outputs need to stay inside a pipeline.

A better contract for scraper jobs

For code-driven scraping, I want the job request to contain enough information to reproduce the run:

{
  "url": "https://example.com/products/widget-123",
  "javascript": true,
  "country": "US",
  "timeoutMs": 30000,
  "schema": {
    "title": "string",
    "price": "number",
    "currency": "string",
    "inStock": "boolean"
  }
}

And I want the response to separate transport success from extraction success:

{
  "status": "completed",
  "httpStatus": 200,
  "extractionStatus": "partial",
  "data": {
    "title": "Widget 123",
    "price": null,
    "currency": null,
    "inStock": true
  },
  "errors": [
    {
      "field": "price",
      "message": "No matching price text found"
    }
  ]
}

That gives your pipeline something useful to do. You can reject partial results, retry with a different region, alert on missing fields, or store the raw HTML for inspection.

You can also write tests against the contract:

import assert from "node:assert";

function validateProduct(row) {
  assert.equal(typeof row.title, "string");
  assert.ok(row.title.length > 0);
  assert.equal(typeof row.inStock, "boolean");

  if (row.inStock) {
    assert.equal(typeof row.price, "number");
    assert.ok(row.price > 0);
  }
}

This is much harder when the real extraction logic lives in a GUI project that your test suite cannot inspect.

Security is part of the API shape

Scraping often starts with public pages, then drifts into authenticated pages: admin dashboards, partner portals, internal tools, vendor accounts.

That is where visual scrapers can get uncomfortable. If the tool asks someone to type credentials into a recorded login step, you need to know exactly how those secrets are stored, who can view them, whether they are encrypted, and whether the API can ever return them.

At minimum, look for these properties:

Secrets are stored outside the scraper definition
The API can reference a credential by ID, not by raw password
Secrets are never returned by read endpoints
Runs produce audit logs
Free or shared projects are not public by default

A safer request shape looks like this:

{
  "url": "https://portal.example.com/invoices",
  "auth": {
    "credentialId": "cred_vendor_portal_readonly"
  },
  "extract": {
    "invoiceNumber": "string",
    "total": "number",
    "dueDate": "date"
  }
}

The scraper gets access at run time. The pipeline never handles the password directly.

Wire is relevant here because its prebuilt actions can use vault-backed credentials without returning the secret through the API, which avoids mixing login material with extraction code.

When a visual scraper is still the right tool

Visual scrapers are not bad. They are often the right choice for a bounded task, especially when the person doing the work does not write code.

If someone needs to scrape one directory once a week, export a spreadsheet, and adjust the workflow manually when the site changes, a point-and-click builder may be faster than building an internal tool.

The tradeoff appears when the scraper becomes infrastructure. If the output feeds billing, search, analytics, alerts, model evaluation, or customer-facing features, you probably want the scrape definition, job inputs, validation, retries, and errors in code.

What to check before choosing

Before you depend on a scraping API, answer these questions with actual requests and responses:

Can I define a new scrape entirely through the API?
Can I pass a schema or extraction instructions at runtime?
What does a partial extraction failure look like?
Are JavaScript rendering, proxy region, timeout, and retries request parameters?
Can I run the same job in CI with fixture URLs?
Where are credentials stored, and can any endpoint read them back?
Does the pricing model punish failed runs or repeated cached fetches?
Are there page-per-run or worker caps that affect batch jobs?

The practical next step: take one real target page, write the API request you wish you could send, then compare tools against that contract instead of comparing feature lists.

Your competitor's price is not a float

Anakin — Thu, 15 Oct 2026 10:00:01 +0000

You add competitor pricing to a dashboard and start with the obvious model: SKU, competitor, price. It works for retailer pages. Then someone asks why the same product costs $6.49 on one site, $7.29 on a delivery app, and $14.74 at checkout. The bug is not in your scraper. The bug is the assumption that a product has one competitor price.

A price is an observation, not a property

For a nationally distributed product, customers may see different prices on your site, Amazon, Walmart, Instacart in one zip code, Instacart in another zip code, and DoorDash or Uber Eats from a nearby store.

Those are not copies of the same decision. Different systems set them:

Your site price is usually controlled by you.
Amazon may expose a Buy Box price affected by third-party sellers and marketplace rules.
A retailer site reflects that retailer's margin and promo calendar.
Instacart can vary by store, banner, zip code, fulfillment path, and shopper context.
Delivery apps often add platform markup, delivery fees, service fees, small-basket fees, and sometimes demand-based charges.

If your schema stores products.price, you will overwrite useful data or average it into something no customer actually saw.

A better primitive is a price observation:

create table price_observations (
  id bigserial primary key,
  sku text not null,
  competitor_sku text,
  surface text not null,              -- amazon, walmart, instacart, doordash
  retailer text,
  postcode text,
  observed_at timestamptz not null,
  subscriber_context text not null,   -- none, prime, instacart_plus, dashpass
  item_price_cents integer not null,
  delivery_fee_cents integer,
  service_fee_cents integer,
  surcharge_cents integer,
  currency char(3) not null default 'USD',
  raw_url text,
  raw_payload jsonb
);

This looks more annoying than price numeric(10,2), because it is. It also matches the thing you are measuring.

For teams collecting these delivery and quick-commerce prices as structured data, Wire treats the surface, postcode, fee components, and shopper context as part of the price record rather than metadata to recover later.

The checkout total can reverse the signal

On delivery surfaces, the listed item price is not the number the customer pays. That matters when you compare competitors.

Say your item is $7.29 on a grocery delivery app. A competitor's item is $5.99. If you track only item price, you report a 17.8% gap.

But the checkout may look like this:

from dataclasses import dataclass

@dataclass
class DeliveryPrice:
    item: int
    delivery_fee: int
    service_fee: int
    surcharge: int = 0

    @property
    def delivered_total(self) -> int:
        return self.item + self.delivery_fee + self.service_fee + self.surcharge

mine = DeliveryPrice(item=729, delivery_fee=399, service_fee=146, surcharge=200)
theirs = DeliveryPrice(item=599, delivery_fee=399, service_fee=146, surcharge=200)

item_gap = (mine.item - theirs.item) / theirs.item
total_gap = (mine.delivered_total - theirs.delivered_total) / theirs.delivered_total

print(f"item gap: {item_gap:.1%}")
print(f"delivered gap: {total_gap:.1%}")

Output:

item gap: 21.7%
delivered gap: 9.1%

The fee stack did not eliminate the difference, but it changed its size. In some baskets it compresses the competitive gap. In others it amplifies it, especially when fees depend on basket size or subscription status.

The common failure mode is a dashboard that says "we are 20% more expensive" while a customer sees a much smaller checkout difference. Or the reverse: item prices look close, but one platform's local fees make the delivered total much worse.

Location is part of the key

A national average is often useful for retailer sites. It is much less useful for delivery apps.

A delivery price depends on the local store, local platform markup, delivery supply, fee rules, and sometimes the exact address. Two zip codes in the same metro area can produce different item prices and different fees for the same SKU.

That means this query is usually wrong:

select sku, avg(item_price_cents) as avg_price
from price_observations
where surface = 'instacart'
group by sku;

It creates a number that no shopper paid. Use it only if you explicitly want a statistical summary.

For pricing decisions, keep geography in the grouping key:

select
  sku,
  postcode,
  percentile_cont(0.5) within group (order by item_price_cents) as median_item_price,
  percentile_cont(0.5) within group (
    order by item_price_cents
      + coalesce(delivery_fee_cents, 0)
      + coalesce(service_fee_cents, 0)
      + coalesce(surcharge_cents, 0)
  ) as median_delivered_total
from price_observations
where surface in ('instacart', 'doordash', 'ubereats')
group by sku, postcode;

This is not just a reporting detail. If a pricing team can only change prices regionally, collect data regionally. If it can act at store or zip-code level, collect at that level. The measurement grain should match the decision grain.

Subscription context is another split

Instacart+, DashPass, Uber One, Walmart+, and Prime can create different effective prices for the same product at the same time.

Sometimes the item price changes. Sometimes the delivery fee disappears. Sometimes the platform applies member-only promotions at checkout. If your scraper runs from a logged-in account, you may collect a subscriber price and label it as the public price. If it runs logged out, you may miss the price your most loyal customers see.

This is the kind of bug that does not throw an exception. It just poisons comparisons quietly.

Store the context explicitly:

{
  "sku": "DOGFOOD-12LB",
  "surface": "doordash",
  "postcode": "77002",
  "subscriber_context": "dashpass",
  "item_price_cents": 1899,
  "delivery_fee_cents": 0,
  "service_fee_cents": 285,
  "surcharge_cents": 0
}

If a vendor or internal pipeline cannot tell you whether an observation came from a subscriber or non-subscriber session, do not mix those rows without labeling the result.

Wire separates subscriber and non-subscriber delivery observations so downstream pricing analysis can compare like with like.

What I would build first

Before building a large monitoring system, run a small audit.

Pick one SKU. Collect prices from your own site, Amazon, one major retailer, Instacart in two distant zip codes, and one delivery app. For delivery surfaces, record item price, delivery fee, service fee, surcharge, postcode, timestamp, and subscription context.

Then calculate two spreads:

item price spread
delivered total spread

If those numbers are close, your current item-price tracker may be good enough for that category. If they differ materially, your data model needs to capture fees, location, and shopper context before the dashboard can answer the pricing question correctly.

Turning Ride-Hailing Scrapes Into Useful Pricing Signals

Anakin — Wed, 14 Oct 2026 10:00:01 +0000

Ride-hailing pricing changes quickly enough that a scrape from ten minutes ago can already be misleading. A competitor may surge in one neighborhood, show no cars in another, and offer a coupon only to a new account. If you treat those values as clean facts, your pricing or forecasting model will learn the wrong thing.

The useful pattern is to treat competitor data as timestamped observations with context, uncertainty, and failure states. That sounds obvious, but many scraping pipelines lose this information early.

Scraped fares are observations, not ground truth

A ride quote is not just price = 18.42.

It depends on at least:

pickup and dropoff coordinates
request time
service type, such as economy, comfort, XL, electric, accessibility
account state, such as new user, frequent rider, corporate account
app version, locale, currency, and payment method
whether the quote included surge, fees, taxes, discounts, or credits

Two scrapes can disagree and both can be correct if they used different accounts or slightly different coordinates.

A useful raw schema keeps those dimensions intact:

CREATE TABLE competitor_quote_observations (
  id BIGSERIAL PRIMARY KEY,
  competitor TEXT NOT NULL,
  observed_at TIMESTAMPTZ NOT NULL,
  pickup_lat DOUBLE PRECISION NOT NULL,
  pickup_lng DOUBLE PRECISION NOT NULL,
  dropoff_lat DOUBLE PRECISION NOT NULL,
  dropoff_lng DOUBLE PRECISION NOT NULL,
  service_type TEXT NOT NULL,
  persona TEXT NOT NULL,
  currency TEXT,
  total_fare_cents INTEGER,
  base_fare_cents INTEGER,
  distance_fare_cents INTEGER,
  time_fare_cents INTEGER,
  surge_multiplier NUMERIC,
  promo_cents INTEGER,
  eta_seconds INTEGER,
  raw_response JSONB NOT NULL,
  status TEXT NOT NULL,
  error_code TEXT
);

The status column matters. A failed scrape is still data. If one platform returns quotes in most areas but times out or hides availability in a specific zone, that tells you something different from “fare is null.”

For extraction teams that do not want to build every app and web collection path themselves, Wire fits this problem when the output needs to preserve quote context like location, persona, timestamp, ETA, and failure state rather than just a flattened price.

Normalize before feeding models

A pricing model usually needs a smaller table than the raw scrape table. You might aggregate by geohash, hour, competitor, and service type.

The important part is not to average stale and fresh observations together as if they have equal value. A quote observed 90 seconds ago should usually count more than one observed 25 minutes ago.

Here is a simple Python example that applies a freshness weight before producing features:

from datetime import datetime, timezone
import math

HALF_LIFE_SECONDS = 300  # after 5 minutes, weight is 0.5

def freshness_weight(observed_at: datetime, now: datetime) -> float:
    age = max(0, (now - observed_at).total_seconds())
    return 0.5 ** (age / HALF_LIFE_SECONDS)


def weighted_average(rows, field):
    numerator = 0.0
    denominator = 0.0

    now = datetime.now(timezone.utc)

    for row in rows:
        if row["status"] != "ok":
            continue
        value = row.get(field)
        if value is None:
            continue

        w = freshness_weight(row["observed_at"], now)
        numerator += value * w
        denominator += w

    return numerator / denominator if denominator else None


quotes = [
    {
        "competitor": "example_ride",
        "observed_at": datetime(2026, 8, 18, 10, 0, 0, tzinfo=timezone.utc),
        "status": "ok",
        "total_fare_cents": 2140,
        "eta_seconds": 360,
        "surge_multiplier": 1.4,
    },
    {
        "competitor": "example_ride",
        "observed_at": datetime(2026, 8, 18, 9, 42, 0, tzinfo=timezone.utc),
        "status": "ok",
        "total_fare_cents": 1750,
        "eta_seconds": 780,
        "surge_multiplier": 1.0,
    },
]

print(weighted_average(quotes, "total_fare_cents"))
print(weighted_average(quotes, "eta_seconds"))

This is not a production pricing model. It is the kind of guardrail you want before modeling starts. Without it, old low-fare quotes can dampen a current surge signal and make your system react late.

The useful features are usually boring

For ride-hailing, the most useful competitor features tend to be straightforward:

median competitor fare by area, route bucket, and hour
competitor ETA by area and service type
observed surge multiplier
quote success rate
promo availability by persona
fare component split, if visible
variance across accounts for the same route

The last one catches persona-based pricing. If a new account sees a lower fare than an existing account for the same trip, your model should not collapse both into one number without recording the persona.

A simple demand model might use your own demand plus competitor signals:

predicted_demand(area, hour) =
  baseline
  + beta_1 * own_completed_rides(area, hour - 1)
  + beta_2 * competitor_median_eta(area, hour)
  + beta_3 * competitor_surge_multiplier(area, hour)
  + beta_4 * competitor_quote_failure_rate(area, hour)

The sign of each coefficient depends on your market. A high competitor ETA can mean unmet demand that you can capture. It can also mean bad weather or an event that affects everyone, including your own drivers.

How scraping fails in this domain

The common failures are not subtle.

You get HTTP 429 responses or app-level throttling when you sample too aggressively. You get stale cached quotes that look valid but repeat the same fare and ETA for too long. You get partial responses where fare exists but ETA is missing. You get account contamination when a test account receives a promo and then stops representing a normal user.

You should log these as first-class states:

{
  "status": "failed",
  "error_code": "rate_limited",
  "competitor": "example_ride",
  "observed_at": "2026-08-18T10:04:12Z",
  "pickup_geohash": "dr5ru",
  "service_type": "economy"
}

Do not hide these rows from analysts. A spike in rate_limited may mean your scraper broke, but a spike in no_drivers_available in one area can be a useful competitive signal.

This is also where vendor output quality matters. Wire is relevant here if you need ride-hailing extraction that distinguishes “no quote available,” “blocked,” “timed out,” and “valid zero promo” instead of returning a blank field for all of them.

Use competitor data carefully

Competitor scraping should not directly set prices without constraints. Feed it into a pricing system as one group of features, then cap changes and monitor outcomes.

A safer control loop looks like this:

Collect quotes and ETAs for fixed route samples.
Validate freshness, persona, and failure rates.
Aggregate features by area and time window.
Use those features in pricing, supply allocation, or promotion models.
Compare outcomes against holdout areas before broad rollout.

The practical next step is to define one route grid, one service type, and one persona, then build the raw observation table before touching any pricing logic. If the data model cannot explain where a fare came from, the model built on top of it will not be trustworthy.

Treat product assortment as a data problem, not a catalog spreadsheet

Anakin — Tue, 13 Oct 2026 10:00:01 +0000

A catalog usually gets messy slowly. Someone adds five near-identical variants because a supplier sent them. Another team keeps a slow seller because it had one good month last year. Search results fill up with products nobody buys, while customers still complain that you do not carry the thing they came for.

That is a product assortment problem, but for developers it often shows up as a data problem: unclear SKU relationships, missing category mappings, stale inventory, and dashboards that count products without explaining whether the catalog is useful.

Start with a catalog model you can query

Before you can reason about assortment, you need a few basic relationships to be explicit. At minimum, separate the sellable SKU from the product family and category taxonomy.

create table categories (
  category_id bigint primary key,
  parent_category_id bigint,
  name text not null
);

create table products (
  product_id bigint primary key,
  parent_sku text not null,
  category_id bigint not null references categories(category_id),
  brand text,
  title text not null,
  status text not null check (status in ('active', 'inactive'))
);

create table skus (
  sku_id bigint primary key,
  product_id bigint not null references products(product_id),
  color text,
  size text,
  price numeric(10,2) not null,
  inventory_count int not null,
  created_at timestamp not null
);

create table order_items (
  order_id bigint,
  sku_id bigint not null references skus(sku_id),
  quantity int not null,
  unit_price numeric(10,2) not null,
  created_at timestamp not null
);

If your catalog cannot answer “which SKUs are variants of the same product?” then depth metrics will lie. If category_id is nullable or overloaded with values like misc, breadth metrics will tell you that you sell in more categories than you really do.

The failure mode is familiar: the dashboard says you have 12,000 active SKUs, but customers see 40 copies of the same black phone case and still cannot find a charger for their laptop.

Measure breadth, depth, and length separately

Product assortment has a few useful dimensions:

Breadth: how many categories you cover
Depth: how many variants you carry inside a product family or category
Length: total active SKUs
Quality mix: distribution across price bands
Consistency: how related the categories are to each other

You do not need a complicated model to start. These queries give you a baseline.

-- Catalog length
select count(*) as active_skus
from skus s
join products p on p.product_id = s.product_id
where p.status = 'active';

-- Breadth by active category
select count(distinct p.category_id) as active_categories
from products p
join skus s on s.product_id = p.product_id
where p.status = 'active'
  and s.inventory_count > 0;

-- Depth by category
select
  c.name as category,
  count(distinct p.parent_sku) as product_families,
  count(s.sku_id) as sku_count,
  round(count(s.sku_id)::numeric / nullif(count(distinct p.parent_sku), 0), 2) as avg_skus_per_family
from categories c
join products p on p.category_id = c.category_id
join skus s on s.product_id = p.product_id
where p.status = 'active'
group by c.name
order by avg_skus_per_family desc;

The last query is where problems often appear. A high avg_skus_per_family can be good for apparel, where size and color matter. It can be noise for commodity accessories, where 18 tiny variations split demand and make search results worse.

Find over-assortment with sales and inventory together

Counting SKUs is not enough. You need to know whether variants sell, sit, or cannibalize each other.

This query flags product families where many variants exist, but most have low sales in the last 90 days.

with sku_sales as (
  select
    s.sku_id,
    p.parent_sku,
    p.category_id,
    s.inventory_count,
    coalesce(sum(oi.quantity), 0) as units_90d
  from skus s
  join products p on p.product_id = s.product_id
  left join order_items oi
    on oi.sku_id = s.sku_id
   and oi.created_at >= now() - interval '90 days'
  where p.status = 'active'
  group by s.sku_id, p.parent_sku, p.category_id, s.inventory_count
), family_stats as (
  select
    parent_sku,
    count(*) as variant_count,
    count(*) filter (where units_90d = 0 and inventory_count > 0) as stocked_dead_variants,
    sum(inventory_count) as total_stock
  from sku_sales
  group by parent_sku
)
select *
from family_stats
where variant_count >= 8
  and stocked_dead_variants::numeric / variant_count >= 0.5
order by stocked_dead_variants desc;

Do not automatically delete these products. The query creates a review list. A variant might have zero sales because it launched yesterday, because it is seasonal, or because the product page is broken. The data points to a question, not the answer.

Detect under-assortment from demand signals

Under-assortment is harder because the missing product is not in your catalog. You need demand signals from searches, product listing views, support tickets, or competitor catalogs.

Suppose you track search events like this:

create table search_events (
  search_id bigint primary key,
  user_id bigint,
  query text not null,
  result_count int not null,
  created_at timestamp not null
);

A simple under-assortment signal is repeated searches with poor results.

select
  lower(query) as normalized_query,
  count(*) as searches,
  avg(result_count) as avg_results
from search_events
where created_at >= now() - interval '30 days'
group by lower(query)
having count(*) >= 50
   and avg(result_count) < 3
order by searches desc;

This catches cases like customers searching for “usb c gan charger” while your taxonomy only has a generic “chargers” category. The symptom is not just low conversion. It is demand with no useful landing place.

External catalog and price data can help here, but only if you store it as structured observations with timestamps and source URLs. Wire fits this part of the workflow when you need competitor assortment and pricing data as rows you can join against your own SKU and category tables.

Normalize competitor data before comparing it

Competitor matching fails in boring ways. Titles differ. Sizes use different units. One site says “midnight,” another says “black.” Bundles look cheaper until you compare unit counts.

A practical table for external observations might look like this:

create table competitor_offers (
  observed_at timestamp not null,
  source text not null,
  source_url text not null,
  external_title text not null,
  normalized_brand text,
  normalized_category text,
  normalized_size text,
  price numeric(10,2),
  in_stock boolean,
  match_confidence numeric(4,3),
  matched_sku_id bigint references skus(sku_id)
);

Keep external_title even after normalization. When a match looks wrong, the raw title is usually the fastest way to debug it.

Also store match_confidence. A low-confidence match should not drive repricing or assortment decisions. It should go into a manual review queue.

Turn metrics into an assortment review queue

The useful output is not a dashboard with five charts. It is a list of actions someone can inspect.

create table assortment_review as
select
  'possible_over_assortment' as reason,
  parent_sku as entity_id,
  json_build_object(
    'variant_count', variant_count,
    'stocked_dead_variants', stocked_dead_variants,
    'total_stock', total_stock
  ) as evidence,
  now() as created_at
from family_stats
where variant_count >= 8
  and stocked_dead_variants::numeric / variant_count >= 0.5;

You can add other reasons over time: missing category demand, price gaps, stale inventory, poor variant coverage, or competitor-only products with repeated search demand.

The important part is to keep the evidence with the recommendation. “Remove SKU 123” is not useful. “Review this product family because 9 of 12 stocked variants sold zero units in 90 days” is useful.

A good next step is to run the breadth, depth, and dead-variant queries against one category you know well. If the output surprises the category owner, fix the data model before you build more assortment logic on top of it.

Retail pricing data is messier than a price field

Anakin — Mon, 12 Oct 2026 10:00:03 +0000

You can have a clean pricing model and still make bad pricing decisions because the input data is wrong. In retail, the problem usually starts with a simple assumption: one product page has one price. Then you discover variants, coupons, member pricing, out-of-stock items, regional pricing, marketplace sellers, and JavaScript-rendered pages that sometimes return nothing at all.

Retail teams talk about competitive pricing, dynamic pricing, discount pricing, and value-based pricing. Developers usually inherit the operational part: collect enough reliable data so those strategies do not run on guesses.

Treat price as an observation, not a product attribute

A common first version of a price monitor stores this:

CREATE TABLE competitor_products (
  sku TEXT PRIMARY KEY,
  competitor_url TEXT NOT NULL,
  price_cents INTEGER,
  currency TEXT,
  updated_at TIMESTAMP
);

That works until the same URL shows different prices depending on size, location, login state, promotion, or seller. Updating one row destroys useful history and hides uncertainty.

A better pattern is to store every scrape result as an observation:

CREATE TABLE price_observations (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  internal_sku TEXT NOT NULL,
  competitor TEXT NOT NULL,
  competitor_url TEXT NOT NULL,
  observed_at TIMESTAMP NOT NULL,

  variant_key TEXT,
  seller_name TEXT,
  price_cents INTEGER,
  list_price_cents INTEGER,
  currency TEXT,
  in_stock BOOLEAN,
  promo_text TEXT,

  http_status INTEGER,
  extraction_status TEXT NOT NULL,
  raw_hash TEXT,
  error_message TEXT
);

This schema lets you answer different questions later:

What was the lowest available price today?
Did the competitor discount the item or change the base price?
Was the product actually purchasable?
Did our scraper fail, or did the site stop showing the product?

That distinction matters. A missing price is not the same thing as a zero price, and an out-of-stock product should not always trigger an aggressive price drop.

Capture the effective price, not just the visible number

Retail pages often expose several numbers:

List price: $129.99
Sale price: $99.99
Coupon: Save 10% at checkout
Loyalty price: $89.99 with membership
Bundle price: 2 for $170

If your parser grabs the first currency-looking string, it will eventually make a bad decision. You might parse the crossed-out list price instead of the sale price. Or you might treat a coupon as unavailable because it appears outside the price container.

A simple extractor should make the decision explicit:

from bs4 import BeautifulSoup
from decimal import Decimal
import re

PRICE_RE = re.compile(r"(\d+[\d,]*\.\d{2})")

def cents(text: str) -> int | None:
    if not text:
        return None
    match = PRICE_RE.search(text.replace(",", ""))
    if not match:
        return None
    return int(Decimal(match.group(1)) * 100)

def extract_price(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")

    sale = soup.select_one('[data-testid="sale-price"]')
    list_price = soup.select_one('[data-testid="list-price"]')
    stock = soup.select_one('[data-testid="stock-status"]')
    promo = soup.select_one('[data-testid="promo-message"]')

    price_cents = cents(sale.get_text(" ")) if sale else None
    list_price_cents = cents(list_price.get_text(" ")) if list_price else None

    if price_cents is None:
        raise ValueError("sale price selector matched nothing or contained no parseable price")

    return {
        "price_cents": price_cents,
        "list_price_cents": list_price_cents,
        "in_stock": bool(stock and "in stock" in stock.get_text(" ").lower()),
        "promo_text": promo.get_text(" ", strip=True) if promo else None,
    }

This is intentionally boring code. The important part is that extraction failure is explicit. If the selector breaks, you get a ValueError instead of silently inserting NULL and letting downstream pricing code interpret it however it wants.

Expect matching to be harder than scraping

Competitive pricing depends on comparing the same product. That sounds obvious, but matching breaks in small ways:

A competitor lists a 12-pack while you sell a single unit.
Color names differ: navy, midnight blue, blue black.
Product titles include accessories or bundles.
A marketplace page changes the default seller.
Variant URLs canonicalize back to the parent product.

Do not match only on title similarity. Store the evidence you used for the match:

{
  "internal_sku": "SHOE-123-BLK-10",
  "competitor": "example-store",
  "competitor_url": "https://example.com/products/running-shoe",
  "match_confidence": 0.92,
  "matched_fields": ["brand", "model", "color", "size"],
  "unit_quantity": 1
}

If a price suddenly drops by 70%, the first debugging question is not “what did the pricing algorithm do?” It is “did we compare the right item?”

Separate collection failures from business signals

Scrapers fail for normal reasons. You get 403 Forbidden, 429 Too Many Requests, stale cached HTML, bot checks, changed markup, and pages that render prices only after client-side API calls. These are collection failures, not market events.

Your pipeline should classify them separately:

try:
    response = fetch(url)
    data = extract_price(response.text)
    status = "ok"
    error = None
except TimeoutError as exc:
    data = {}
    status = "fetch_timeout"
    error = str(exc)
except ValueError as exc:
    data = {}
    status = "extraction_failed"
    error = str(exc)

Then add a rule: pricing decisions can use only observations where extraction_status = 'ok' and observed_at is recent enough for the category. Grocery prices might need hourly checks. Furniture might not.

If you are buying rather than building this extraction layer, Wire fits this part of the problem: competitor price monitoring where variant matching, promotion capture, and extraction failure handling all affect the final dataset.

Make pricing rules conservative by default

Even if the business wants dynamic pricing, the first implementation should avoid automatic overreaction. Put guardrails around changes:

SELECT internal_sku,
       MIN(price_cents) AS competitor_min_price
FROM price_observations
WHERE extraction_status = 'ok'
  AND in_stock = 1
  AND observed_at > datetime('now', '-6 hours')
GROUP BY internal_sku;

Then apply limits outside SQL:

Do not change price if fewer than two competitors have fresh observations.
Do not match an out-of-stock competitor.
Do not drop below margin floor.
Do not react to one anomalous observation without a second confirmation.
Log the input observations used for every price change.

That last point saves time. When someone asks why an item changed from $49.99 to $44.99, you need more than “the job ran.”

A practical next step

Pick ten SKUs and track them across two competitors for a week. Store observations instead of overwriting rows. Include stock, promo text, variant, seller, extraction status, and raw response hashes. Before writing any pricing algorithm, inspect the weird cases manually. Those weird cases are the system requirements.

Real-time market data is mostly a freshness problem

Anakin — Fri, 09 Oct 2026 10:00:01 +0000

A competitor changes price at 2:15 PM. Your scraper runs at midnight. The dashboard looks clean the next morning, but the business already lost a few hours of margin or conversion. The bug is not that you lack data. The bug is that your data describes the wrong moment.

Real-time usually means fresh enough

Most teams do not need true real-time market intelligence. They need data that is fresh enough for the decision it drives.

A pricing bot may need competitor changes within 5 minutes. An assortment gap report may be fine at 6 hours. A weekly category review does not need streaming infrastructure at all.

The first useful design question is not how do we make this real-time? It is:

What is the maximum age of this data before acting on it becomes unsafe?

That gives you a freshness SLO. For example:

competitor_price_freshness_seconds < 900 for 95% of tracked SKUs
stock_status_freshness_seconds < 300 for top 500 SKUs

Track freshness as a first-class metric. Job success is not enough. A scrape can succeed every hour while still being useless if the source changed 10 minutes after each run.

Wire fits this specific problem when reliable extraction, freshness tracking, and failure handling matter more than running one-off scrapers.

The shape of the system

A simple near-real-time market monitor has four parts:

Fetch the source
Normalize the result
Compare it with the last known state
Emit a change event

Do not plug dashboards directly to raw scrapes. Raw pages are unstable. Prices move between list price, sale price, coupon price, member price, and regional price. Normalize before anyone consumes the data.

Here is a small Python example that watches one product page and emits an event only when the normalized state changes:

import hashlib
import json
import re
import sys
from datetime import datetime, timezone
from decimal import Decimal
from pathlib import Path

import requests
from bs4 import BeautifulSoup

STATE_FILE = Path('product_state.json')


def parse_price(text):
    cleaned = re.sub(r'[^0-9.]', '', text)
    if not cleaned:
        raise ValueError(f'could not parse price from: {text!r}')
    return str(Decimal(cleaned))


def fetch_product(url):
    res = requests.get(
        url,
        timeout=10,
        headers={'User-Agent': 'market-monitor/1.0'}
    )

    if res.status_code == 429:
        raise RuntimeError('rate limited by source: HTTP 429')

    res.raise_for_status()

    soup = BeautifulSoup(res.text, 'html.parser')
    price_el = soup.select_one('[data-testid=price]')
    title_el = soup.select_one('h1')
    sold_out_el = soup.select_one('[data-testid=sold-out]')

    if price_el is None:
        raise RuntimeError('price selector missing, page layout may have changed')

    if title_el is None:
        raise RuntimeError('title selector missing, page may be blocked or changed')

    return {
        'url': url,
        'title': title_el.get_text(strip=True),
        'price': parse_price(price_el.get_text()),
        'in_stock': sold_out_el is None,
        'observed_at': datetime.now(timezone.utc).isoformat()
    }


def fingerprint(record):
    comparable = {
        'url': record['url'],
        'price': record['price'],
        'in_stock': record['in_stock']
    }
    payload = json.dumps(comparable, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()


def load_state():
    if not STATE_FILE.exists():
        return {}
    return json.loads(STATE_FILE.read_text())


def save_state(state):
    STATE_FILE.write_text(json.dumps(state, indent=2))


def main(url):
    state = load_state()
    record = fetch_product(url)
    key = record['url']
    new_hash = fingerprint(record)
    previous = state.get(key)

    if previous is None or previous['hash'] != new_hash:
        event = {
            'type': 'product_changed',
            'previous': previous['record'] if previous else None,
            'current': record
        }
        print(json.dumps(event, indent=2))

    state[key] = {'hash': new_hash, 'record': record}
    save_state(state)


if __name__ == '__main__':
    main(sys.argv[1])

Run it on a schedule:

python monitor.py 'https://example.com/products/sku-123'

In production, the printed event would go to Kafka, SQS, Pub/Sub, or a webhook. The important part is the event boundary. Downstream systems should react to product_changed, not scrape HTML themselves.

How these systems fail

The boring failures cause most of the damage.

A CSS selector stops matching and your parser returns null. If you silently write null into the database, the dashboard shows missing prices as real prices. Raise an error instead.

A source returns a bot challenge with HTTP 200. Your parser sees a valid HTML document, but it is not the product page. This is why the example checks for both price and title. In real systems, also check canonical URLs, currency, region, and known page markers.

A source rate limits you with 429. Retrying immediately makes the block worse. Back off, reduce concurrency per domain, and record the gap in freshness.

A product has multiple prices. List price, sale price, coupon price, and cart price can all be different. Pick the one that matches the business decision and name it clearly. competitor_price is vague. competitor_checkout_price_usd is much harder to misuse.

Speed without precision creates bad automation

Fast bad data is worse than slow bad data because it triggers action before a human notices the mistake.

If you connect price changes directly to automated pricing, add guardrails:

reject price decrease > 20% unless confirmed by two consecutive observations
reject currency changes unless source region also changed
ignore out-of-stock changes that flip back within 2 minutes
alert when freshness exceeds SLO instead of reusing stale values

These checks reduce false reactions. They also make incidents easier to debug because the system can explain why it did or did not act.

Build for loops, not reports

A report answers what happened. A decision loop answers what changed and what should happen next.

That loop can still start manually. Send a Slack alert when a top competitor drops price. Create a ticket when a rival goes out of stock. Notify merchandising when a category gap appears.

Once the signal quality is good, automate the low-risk actions first. For example, refresh an internal recommendation, flag a SKU for review, or pause a stale campaign. Leave margin-sensitive pricing changes behind stricter checks.

The practical next step: pick one decision your team already makes from stale external data, define its freshness SLO, and instrument the age of that data before changing the scraper itself.

Public Web Data Is Useful When You Treat It Like a Pipeline

Anakin — Wed, 07 Oct 2026 10:00:01 +0000

You can get a lot of value from public web data, but the useful part is rarely the first scrape. The real problem starts a week later: selectors stop matching, prices contain currency symbols from three regions, your job silently writes zero rows, and nobody notices until a dashboard looks wrong.

Public data needs boring data engineering

Public web data is attractive because it is already out there: product pages, public directories, reviews, job listings, government portals, documentation sites, release notes, and market pages. Smaller teams can answer questions that used to require expensive data vendors.

But public does not mean ready to use.

If you want to track competitor pricing, the hard part is not fetching HTML once. The hard parts are:

deciding exactly which pages answer the business question
collecting data without hammering someone else's site
detecting when extraction breaks
normalizing messy fields into stable types
storing observations over time instead of overwriting history
keeping enough metadata to debug bad data later

That is the difference between a scraper and a pipeline.

Start with the question, not the scraper

A common mistake is to start with, can we scrape this site? A better first question is, what decision will this data support?

For example, competitor pricing sounds specific, but it usually needs more definition:

Which competitors matter?
Which SKUs can be matched reliably?
Do we care about list price, discounted price, shipping, or availability?
How often does the data need to change before it affects a decision?
What failure rate is acceptable?

If the answer only needs weekly trends, a daily scrape with manual review may be enough. If the answer feeds an automated repricing system, you need stricter validation, alerting, and rollback behavior.

This matters because collection cost scales with freshness, coverage, and reliability. Scraping 50 pages once is simple. Scraping 50,000 pages every hour while proving the data is still correct is a different system.

A minimal pipeline shape

Here is a small Python example that shows the pattern. It fetches a public product listing page, parses product cards, validates required fields, and stores each observation in SQLite.

import datetime as dt
import sqlite3
import requests
from bs4 import BeautifulSoup

URL = 'https://example.com/products'
HEADERS = {
    'User-Agent': 'market-monitor/1.0 contact: data-team@example.com'
}


def fetch_html(url):
    response = requests.get(url, headers=HEADERS, timeout=10)

    if response.status_code == 403:
        raise RuntimeError('blocked: 403 Forbidden, check site policy or bot protection')

    if response.status_code == 429:
        raise RuntimeError('rate limited: 429 Too Many Requests, back off before retrying')

    response.raise_for_status()
    return response.text


def parse_products(html):
    soup = BeautifulSoup(html, 'html.parser')
    cards = soup.select('.product-card')

    if not cards:
        raise RuntimeError('selector matched 0 product cards, page structure may have changed')

    rows = []
    observed_at = dt.datetime.utcnow().isoformat(timespec='seconds') + 'Z'

    for card in cards:
        product_id = card.get('data-product-id')
        name_el = card.select_one('.product-name')
        price_el = card.select_one('.price')

        if not product_id or not name_el or not price_el:
            continue

        raw_price = price_el.get_text(strip=True)
        price = float(raw_price.replace('$', '').replace(',', ''))

        rows.append({
            'source': URL,
            'product_id': product_id,
            'name': name_el.get_text(strip=True),
            'price': price,
            'observed_at': observed_at
        })

    return rows


def save(rows):
    with sqlite3.connect('market.db') as db:
        db.execute('''
            create table if not exists price_observations (
                source text not null,
                product_id text not null,
                name text not null,
                price real not null,
                observed_at text not null,
                primary key (source, product_id, observed_at)
            )
        ''')

        db.executemany('''
            insert into price_observations
            (source, product_id, name, price, observed_at)
            values (:source, :product_id, :name, :price, :observed_at)
        ''', rows)


html = fetch_html(URL)
products = parse_products(html)

if len(products) < 5:
    raise RuntimeError(f'only parsed {len(products)} products, expected at least 5')

save(products)
print(f'saved {len(products)} product observations')

This is intentionally small, but it includes a few important habits:

It fails loudly when the page returns 403 or 429.
It treats zero matched elements as a broken extractor, not an empty dataset.
It stores observations with timestamps instead of keeping only the latest value.
It validates minimum row count before writing data.

For workflows where the main pain is keeping extraction stable across changing public pages, Wire can sit in the collection part of the pipeline while your application code handles validation, storage, and analysis.

Store raw and normalized data separately

Do not throw away the messy version too early. If a price parser turns 1.299,00 € into 1.299, you will want the original string when someone asks why the chart dropped by 99 percent.

A practical schema often has two layers:

create table raw_pages (
    source_url text not null,
    fetched_at text not null,
    status_code integer not null,
    html text not null,
    primary key (source_url, fetched_at)
);

create table normalized_prices (
    source_url text not null,
    product_id text not null,
    price_cents integer not null,
    currency text not null,
    observed_at text not null
);

Raw storage helps you replay parsers after a bug fix. Normalized storage helps analysts and application code query clean fields without parsing HTML every time.

You do not need to keep raw HTML forever. Set a retention period based on debugging needs, storage cost, and any legal or privacy requirements.

Make failures visible

A scraper that crashes is easier to fix than one that quietly writes bad data. Track operational metrics like:

pages fetched
non-200 responses
rows extracted
rows rejected during validation
duplicate rate
parse duration
last successful run time

Even a simple cron job can report useful signals:

0 * * * * /usr/bin/python3 /app/collect_prices.py >> /var/log/price-job.log 2>&1

Then alert on symptoms that imply broken data, not just process failure. For example, alert if the job exits successfully but extracts fewer than 80 percent of the usual row count.

If a team needs public web data as a recurring feed rather than one-off HTML parsing, Wire is relevant because recurring extraction only becomes useful when failures, retries, and structured outputs are part of the design.

Be careful with access and ethics

Publicly visible data still has rules around it. Before collecting, check the site's terms, robots.txt, API options, and applicable laws. Avoid authenticated areas unless you have permission. Do not collect personal data just because it appears on a page. Rate limit your requests and identify your crawler with a useful user agent.

Also prefer official APIs when they meet the need. APIs usually provide clearer contracts, better stability, and fewer surprises. Scraping makes sense when no suitable API exists, the data is public, and your collection pattern is reasonable.

A practical next step

Pick one public source you already use manually. Write down the exact decision it supports, then build a small pipeline that fetches it, validates row counts, stores timestamped observations, and alerts when extraction drops to zero. That exercise will teach you more about public web data than a much larger scrape with no validation.

Product Matching Is Mostly About Avoiding Confident Wrong Answers

Anakin — Tue, 06 Oct 2026 10:00:01 +0000

You have two product feeds that look close enough to compare prices. Then someone notices you matched a 256 GB phone against the 128 GB version, or a monitor bundle against the bare monitor. The pricing logic was fine. The bad decision happened earlier, when the catalog said two listings were the same product.

Product matching sounds simple until you deal with real ecommerce data. Titles are inconsistent, sellers stuff keywords into descriptions, variants hide in different fields, and regional listings use different units. If you treat matching as string similarity, you will get convincing false positives.

Matching is not search

Search can return similar things. Product matching needs to decide whether two listings represent the same sellable item.

These two are probably the same product:

Apple AirPods Pro 2nd Generation with MagSafe Charging Case
Apple AirPods Pro (2nd Gen), MagSafe Case, USB-C

These two are not the same product, even though the titles overlap heavily:

HP Pavilion 15 Laptop, Ryzen 5, 8GB RAM, 512GB SSD
HP Pavilion 15 Laptop, Intel i5, 8GB RAM, 512GB SSD

The difference is small in text and large in business impact. If you price the Ryzen SKU against the Intel SKU, your downstream reports will look precise while being wrong.

A practical matcher usually has four stages:

Normalize raw fields
Extract product attributes
Generate likely candidates
Score candidates with explicit mismatch rules

The important part is that candidate generation and final matching are different jobs. Candidate generation can be fuzzy. Final matching should be strict.

Normalize before comparing anything

Raw titles carry useful signals, but they also carry noise. Normalize units, casing, punctuation, common abbreviations, and known brand aliases before you compute similarity.

Here is a small example in Python. It is not a complete matcher, but it shows the shape of the problem:

import re
from dataclasses import dataclass

@dataclass
class Product:
    title: str
    brand: str | None = None
    model: str | None = None
    storage_gb: int | None = None
    color: str | None = None
    cpu: str | None = None

BRAND_ALIASES = {
    "hewlett packard": "hp",
    "apple inc": "apple",
}

COLORS = {"black", "white", "silver", "blue", "red", "space gray"}

CPU_PATTERNS = [
    (r"\bryzen\s*5\b", "ryzen 5"),
    (r"\bryzen\s*7\b", "ryzen 7"),
    (r"\bi5\b|\bintel\s+core\s+i5\b", "intel i5"),
    (r"\bi7\b|\bintel\s+core\s+i7\b", "intel i7"),
]

def normalize_text(value: str) -> str:
    value = value.lower()
    value = value.replace("-", " ")
    value = re.sub(r"[^a-z0-9\s]", " ", value)
    value = re.sub(r"\s+", " ", value).strip()
    return value

def extract_product(title: str) -> Product:
    text = normalize_text(title)

    brand = None
    for candidate in ["hewlett packard", "hp", "apple inc", "apple", "samsung"]:
        if candidate in text:
            brand = BRAND_ALIASES.get(candidate, candidate)
            break

    storage_gb = None
    storage_match = re.search(r"(\d+)\s*(gb|tb)\b", text)
    if storage_match:
        amount = int(storage_match.group(1))
        storage_gb = amount * 1024 if storage_match.group(2) == "tb" else amount

    cpu = None
    for pattern, normalized in CPU_PATTERNS:
        if re.search(pattern, text):
            cpu = normalized
            break

    color = next((c for c in COLORS if c in text), None)

    model_match = re.search(r"\b([a-z]+\s?\d{2,4}[a-z]*)\b", text)
    model = model_match.group(1).replace(" ", "") if model_match else None

    return Product(title=title, brand=brand, model=model, storage_gb=storage_gb, color=color, cpu=cpu)

p1 = extract_product("HP Pavilion 15 Laptop, Ryzen 5, 8GB RAM, 512GB SSD")
p2 = extract_product("HP Pavilion 15 Laptop, Intel Core i5, 8GB RAM, 512GB SSD")

print(p1)
print(p2)

This code intentionally extracts attributes instead of producing one big similarity score. That lets you treat some differences as fatal. CPU, storage size, pack size, region, and included accessories often matter more than title similarity.

Use blocking so you do not compare everything

If catalog A has 100,000 products and catalog B has 100,000 products, a full pairwise comparison means 10 billion comparisons. You need blocking, which means generating a smaller candidate set using cheap keys.

Common blocking keys include:

brand + model
brand + normalized title tokens
GTIN / UPC / EAN if available
brand + category + storage size
image hash bucket for visually similar listings

Blocking can miss matches if you make it too strict. For example, one seller may omit the model number from the title but include it in a specs table. A good pipeline usually creates multiple candidate sources, then merges them.

This is also where data extraction quality matters. If your scraper misses the specs table or drops bundled accessories from the description, the matcher never gets a chance to make the right decision. Wire is relevant in this part of the workflow when competitor catalog extraction needs to preserve attributes, variants, and match confidence instead of returning only raw listing rows.

Score matches with hard vetoes

After candidate generation, compute a score. But do not rely only on the score.

A pair can have a 0.94 title similarity and still be wrong. Add hard vetoes for attributes that define the SKU:

def is_match(a: Product, b: Product) -> tuple[bool, str]:
    if a.brand and b.brand and a.brand != b.brand:
        return False, "brand mismatch"

    if a.model and b.model and a.model != b.model:
        return False, "model mismatch"

    if a.storage_gb and b.storage_gb and a.storage_gb != b.storage_gb:
        return False, "storage mismatch"

    if a.cpu and b.cpu and a.cpu != b.cpu:
        return False, "cpu mismatch"

    return True, "candidate match"

This looks basic, but it prevents a common failure mode: a matcher that says yes because most tokens overlap. In ecommerce, the rare token often matters most.

You can still use embeddings or a trained classifier for the final score. Just make the model explain itself in terms of extracted attributes. If it cannot tell you why two products match, you will struggle to debug pricing errors later.

Treat bundles and regional differences as product data

Bundles are easy to miss because the title often stays almost identical:

Dell 27 Monitor S2721QS 4K UHD
Dell 27 Monitor S2721QS 4K UHD with HDMI Cable and Cleaning Kit

If the competitor includes accessories, matching their price directly may make your bare listing look overpriced. Extract included items into a separate field and compare them explicitly.

Regional differences cause a different kind of bug. A product may be the same even when units differ:

10 inch tablet
25.4 cm tablet

Normalize measurements before matching. But be careful with compliance labels, chargers, warranties, and radio bands. Those may represent real product differences, not formatting differences.

Counterfeits and unauthorized imports add another layer. The title and image can match perfectly while the seller, warranty, or fulfillment channel changes the risk. In that case, the output should not be just match=true. Use something like:

{
  "match": true,
  "confidence": 0.982,
  "risk_flags": ["unauthorized_seller", "warranty_missing"]
}

Build for review, not blind automation

A useful matcher should produce three buckets:

match: safe to use for pricing or catalog merge
no_match: safe to ignore
review: too close to discard, too risky to trust

The review bucket is not a failure. It is how you avoid pretending uncertain data is certain. Track review reasons like variant_conflict, missing_model, bundle_unclear, or regional_difference. Those labels help you improve extraction and rules over time.

A practical next step: take 200 known product pairs from your own catalog, label them as match or no match, then run a simple attribute-based matcher before trying embeddings or ML. The mistakes in that first pass will tell you which attributes actually define identity in your category.

Pricing Bugs in Ecommerce Are Usually Data Bugs

Anakin — Tue, 29 Sep 2026 10:00:01 +0000

A lot of ecommerce pricing problems do not start with a pricing team making a bad call. They start with a cron job, a stale competitor feed, a missing cache key, or a checkout calculation that does not match the product page. The symptom looks commercial: margin drops, conversion falls, carts get abandoned. The cause is often technical.

Matching competitors without context breaks margins

The most obvious mistake is also the easiest to automate badly: competitor X dropped the price, so we drop ours too.

That works until the competitor price is tied to something you did not capture. Maybe they are clearing inventory. Maybe the product is out of stock. Maybe the low price only appears for logged-in users, mobile users, or one postal code. If your system stores only sku and price, you cannot tell the difference.

A useful competitor observation needs context:

{
  "sku": "EARBUDS-001",
  "competitor": "example-store",
  "price": 45.00,
  "currency": "USD",
  "in_stock": false,
  "shipping": 6.99,
  "channel": "web",
  "location": "US-CA",
  "observed_at": "2026-08-18T10:15:00Z"
}

That in_stock field matters. If the competitor is out of stock, copying their price may throw away revenue. If they are running a temporary clearance sale, matching them for a week may train your own customers to wait.

For teams that need competitor price and stock extraction across messy pages, Wire fits this part of the problem as a structured extraction layer rather than just another price number in a spreadsheet.

Put guardrails around dynamic pricing

Dynamic pricing is not just “update prices often.” It is updating prices within rules that protect margin, trust, and sanity.

A basic pricing function should refuse to make changes when the inputs are incomplete or suspicious. For example:

from decimal import Decimal

MIN_MARGIN = Decimal("0.18")
MAX_DAILY_CHANGE = Decimal("0.12")


def propose_price(current_price, unit_cost, competitor_observation):
    if competitor_observation is None:
        return current_price, "no_competitor_data"

    if not competitor_observation["in_stock"]:
        return current_price, "competitor_out_of_stock"

    competitor_price = Decimal(str(competitor_observation["price"]))
    floor_price = unit_cost / (Decimal("1.00") - MIN_MARGIN)

    proposed = min(current_price, competitor_price - Decimal("0.50"))
    proposed = max(proposed, floor_price)

    max_down = current_price * (Decimal("1.00") - MAX_DAILY_CHANGE)
    max_up = current_price * (Decimal("1.00") + MAX_DAILY_CHANGE)

    if proposed < max_down:
        return max_down, "capped_daily_decrease"

    if proposed > max_up:
        return max_up, "capped_daily_increase"

    return proposed, "ok"

This is not a complete pricing engine, but it shows the pattern: competitor data is an input, not the authority. Margin floor, stock status, and max daily movement all matter.

Without these checks, failure is quiet. Nobody gets a stack trace. You just sell 2,000 units at a price that looked competitive and later discover that shipping and payment fees pushed the order below contribution margin.

Static prices are stale data with a nice UI

The opposite problem is leaving prices fixed while demand, inventory, and competitors move around you.

If a price was set manually three months ago, it may still look valid in the admin panel. That does not mean it is valid in the market. The technical issue is usually that price is treated as a field, not a decision backed by observations.

A better model stores price history and the reason for each change:

create table price_events (
  id bigserial primary key,
  sku text not null,
  old_price numeric(10,2) not null,
  new_price numeric(10,2) not null,
  reason text not null,
  source text not null,
  created_at timestamptz not null default now()
);

When conversion drops, you can inspect what changed. Was it a competitor match? A seasonal rule? A margin correction? A manual override?

That audit trail also helps you spot bad automation. If 80 percent of changes say competitor_match but sales did not improve, your matching rule probably needs more context.

Segmented pricing needs explicit cache keys

Customer bucketing can be useful: first-time visitors, repeat buyers, wholesale accounts, loyalty members, regional markets. It can also create nasty bugs.

The classic failure is showing one price on the product page and another at checkout because the systems used different segmentation logic. Another common one is CDN caching a logged-in or regional price and serving it to everyone.

If price depends on segment, your cache key must include the segment inputs:

price:{sku}:{currency}:{country}:{channel}:{customer_segment}

Do not hide this logic inside three services with slightly different definitions of “repeat buyer.” Put the segment calculation somewhere testable and version it. If you change the rules, you should know which orders used which version.

Price sensitivity should be measured, not guessed

Developers often get asked to “test a lower price.” That request needs an experiment design, not just a database update.

At minimum, track units sold, conversion rate, gross margin, and cart abandonment for each price variant. Revenue alone can mislead you. A lower price may increase orders while reducing total contribution margin.

A simple sensitivity check looks like this:

sensitivity = (new_quantity - old_quantity) / (new_price - old_price)

If earbuds move from $50 to $45 and monthly units increase from 1,000 to 1,300:

(1300 - 1000) / (45 - 50) = -60

The negative sign is expected: price went down, demand went up. The size tells you demand moved a lot for a $5 change. That does not automatically mean $45 is better. You still need margin math.

Checkout surprises are pricing defects

Shipping, tax, handling fees, and payment surcharges should not appear as a surprise at the last step. Users read that as a bait-and-switch, even if your backend thinks it calculated everything correctly.

This is worth testing like any other contract between services:

import { test, expect } from "@playwright/test";

test("product page estimated total matches checkout total", async ({ page }) => {
  await page.goto("/products/earbuds-001?postalCode=94107");

  const productTotal = await page.locator("[data-testid=estimated-total]").innerText();

  await page.click("[data-testid=add-to-cart]");
  await page.goto("/checkout");

  const checkoutTotal = await page.locator("[data-testid=order-total]").innerText();

  expect(checkoutTotal).toBe(productTotal);
});

This test will fail when tax rules, shipping estimates, or promotions drift between page and checkout code. That is exactly the point.

Automation needs failure modes you can see

Manual competitor checks do not scale, but automation without visibility creates a different problem. You need to know when extraction failed, when a selector changed, when a competitor blocked requests, and when the data looks statistically weird.

A pricing job should emit counts like:

competitor_observations_total
competitor_observations_failed
price_changes_proposed
price_changes_rejected_margin_floor
price_changes_rejected_missing_stock

If competitor pages vary by location, account state, or device type, Wire can help collect those observations with the surrounding context that pricing rules need.

Start with one SKU category. Store contextual competitor observations, add margin and movement guardrails, write one checkout total test, and log every price change with a reason. That small slice will show whether your pricing problem is strategy, data quality, or code.

Price matching is a bad default: model the pricing decision instead

Anakin — Wed, 23 Sep 2026 10:00:01 +0000

A common retail pricing bug is not a syntax error. It is a pricing service that does exactly what you told it to do: scrape a competitor price, set your price to match it, and slowly erase your margin because the competitor price was out of stock, coupon-gated, bundled, or part of a weekend promo.

Price matching is a control loop

If you build pricing software, it helps to think of price matching as a control loop:

Observe competitor prices
Normalize those observations
Decide whether they matter
Apply a price change
Measure what happened

Most broken implementations spend all their effort on step 1 and almost none on steps 2 through 5.

A naive implementation looks like this:

def match_lowest_price(our_sku, competitor_offers):
    lowest = min(offer["price"] for offer in competitor_offers)
    return lowest

That function is easy to explain and dangerous to run. It treats every competitor offer as equivalent. It ignores shipping, stock, coupons, minimum advertised price rules, product condition, seller reputation, and whether you can make money at the returned price.

The failure mode is predictable. One competitor drops a SKU from $299 to $249 for a clearance sale. Your system matches it. Another competitor matches you. Your system sees the market at $249 and keeps the new price. Nobody encoded the fact that the original signal was temporary.

Store observations, not just prices

The first fix is to stop storing a single price field as if it represents the market. A competitor offer is a bundle of facts.

A useful schema starts closer to this:

CREATE TABLE competitor_offer_observations (
    id BIGSERIAL PRIMARY KEY,
    observed_at TIMESTAMPTZ NOT NULL,
    competitor TEXT NOT NULL,
    competitor_sku TEXT NOT NULL,
    matched_internal_sku TEXT NOT NULL,
    shelf_price NUMERIC(10, 2),
    checkout_price NUMERIC(10, 2),
    shipping_price NUMERIC(10, 2),
    coupon_value NUMERIC(10, 2),
    currency CHAR(3) NOT NULL,
    in_stock BOOLEAN,
    delivery_days INT,
    seller_name TEXT,
    product_condition TEXT,
    promo_label TEXT,
    scrape_status TEXT NOT NULL,
    raw_payload JSONB NOT NULL
);

This gives your pricing code enough context to reject bad comparisons. A price from an out-of-stock item should usually not trigger a reaction. A marketplace seller with a two-week delivery estimate may not matter if your value proposition is next-day shipping. A coupon visible only after login may matter a lot, but you need to model it explicitly.

This is also where scraping reliability becomes part of pricing correctness, not just data plumbing. If your extractor misses coupons or silently fails on checkout pages, your pricing engine receives false market data. Wire fits this part of the workflow when you need extraction jobs that return price, availability, promotion details, and failure states instead of only a shelf-price field.

Add guardrails before automation

Before you reach for machine learning, write down the rules a human pricing manager would enforce. The rules do not need to be perfect. They need to prevent obvious damage.

Here is a simple Python example:

from dataclasses import dataclass
from decimal import Decimal
from typing import Optional

@dataclass
class Offer:
    competitor: str
    checkout_price: Decimal
    shipping_price: Decimal
    in_stock: bool
    delivery_days: int
    product_condition: str
    promo_label: Optional[str]

@dataclass
class ProductPolicy:
    current_price: Decimal
    unit_cost: Decimal
    min_margin_pct: Decimal
    max_single_change_pct: Decimal
    ignore_delivery_over_days: int


def landed_price(offer: Offer) -> Decimal:
    return offer.checkout_price + offer.shipping_price


def is_comparable(offer: Offer, policy: ProductPolicy) -> bool:
    if not offer.in_stock:
        return False
    if offer.product_condition.lower() != "new":
        return False
    if offer.delivery_days > policy.ignore_delivery_over_days:
        return False
    return True


def floor_price(policy: ProductPolicy) -> Decimal:
    return policy.unit_cost / (Decimal("1.0") - policy.min_margin_pct)


def recommended_price(policy: ProductPolicy, offers: list[Offer]) -> Decimal:
    comparable = [o for o in offers if is_comparable(o, policy)]

    if not comparable:
        return policy.current_price

    market_price = min(landed_price(o) for o in comparable)
    margin_floor = floor_price(policy)

    target = max(market_price, margin_floor)

    max_drop = policy.current_price * policy.max_single_change_pct
    lowest_allowed_today = policy.current_price - max_drop

    return max(target, lowest_allowed_today)

This still matches prices, but it no longer does so blindly. It refuses to compare against irrelevant offers. It protects margin. It limits daily price movement so one bad scrape cannot cut a product by 40%.

The tradeoff is that rules create false negatives. You will sometimes ignore a competitor move that a human would care about. That is better than automatically following every noisy signal. You can tune rules after you inspect misses.

Separate signal from noise

A useful pricing system should ask questions like these before changing a price:

Did the competitor price persist across multiple observations?
Is the product actually in stock?
Is the lower price caused by a coupon, bundle, loyalty discount, or financing offer?
Does the competitor regularly discount this category?
Will matching violate margin, MAP, or brand constraints?
Are we seeing the same movement across several competitors?

The persistence check alone catches many bad updates. For example:

SELECT matched_internal_sku,
       competitor,
       MIN(checkout_price + shipping_price) AS min_landed_price,
       COUNT(*) AS observations
FROM competitor_offer_observations
WHERE observed_at > now() - interval '6 hours'
  AND scrape_status = 'ok'
  AND in_stock = true
GROUP BY matched_internal_sku, competitor
HAVING COUNT(*) >= 3;

That query does not prove a price is meaningful, but it filters out one-off observations. You can then combine it with category-specific rules. Electronics may need faster reaction times than furniture. Private-label products may not need direct matching at all.

Checkout price is another edge many teams underestimate. The shelf price may be $299, but the customer sees $279 after an auto-applied coupon and $14.99 shipping at checkout. If your data model cannot represent those fields separately, your decision code has to guess. Wire is relevant here because checkout-level extraction and explicit job errors make it easier to tell the difference between “no discount exists” and “the scraper never reached the discount step.”

Treat automation as execution, not strategy

Once you have clean observations and explicit guardrails, automation becomes much safer. A model can estimate elasticity, predict competitor behavior, or choose among pricing strategies, but it should not be the first layer of defense.

In practice, I would log every recommendation before applying it:

{
  "sku": "TV-55-OLED-123",
  "current_price": "899.00",
  "recommended_price": "879.00",
  "reason_codes": [
    "competitor_landed_price_lower",
    "price_seen_3_times",
    "above_margin_floor",
    "within_daily_change_limit"
  ],
  "ignored_offers": [
    {
      "competitor": "example-shop",
      "reason": "out_of_stock"
    }
  ]
}

Those reason codes matter when someone asks why revenue dropped in a category, or why you did not match a visible competitor price. Without them, pricing automation becomes hard to debug.

A good next step is to take one high-volume SKU category and build the observation table, comparability rules, margin floor, and recommendation log for that category only. Run it in shadow mode for two weeks, compare its recommendations with human decisions, then decide which rules deserve automation.

Model food delivery pricing as data, not a single scraped number

Anakin — Tue, 22 Sep 2026 10:00:01 +0000

A common mistake in food delivery pricing analysis is scraping the checkout total and treating it as the price. That number is useful, but it hides the reason the price changed. Was it the restaurant markup, delivery fee, service fee, a promotion, subscription eligibility, surge pricing, or a different address? If you only store the final total, every downstream dashboard turns into guesswork.

Food delivery pricing is a stack. Your data model needs to preserve the stack.

The price is not one field

For the same basket, two platforms can differ by 20% at checkout while showing similar menu prices. The difference usually comes from layers that move independently:

Restaurant item prices and markups
Platform service fees
Delivery fees
Small order fees
Promotions and credits
Subscription benefits
Driver or demand incentives that affect availability and delivery estimates
Taxes and local fees

If your scraper or ingestion job flattens this into total_price, you lose the ability to answer basic questions later.

A better raw table stores both the observed total and each component you can identify:

create table delivery_quotes (
  quote_id text primary key,
  platform text not null,
  restaurant_id text not null,
  restaurant_name text not null,
  captured_at timestamptz not null,
  country text not null,
  city text not null,
  postal_code text not null,
  latitude numeric,
  longitude numeric,
  basket_hash text not null,
  subscriber boolean not null,

  item_subtotal_cents integer,
  menu_markup_cents integer,
  service_fee_cents integer,
  delivery_fee_cents integer,
  small_order_fee_cents integer,
  promo_discount_cents integer,
  tax_cents integer,
  tip_cents integer,
  total_cents integer not null,

  delivery_eta_min integer,
  available boolean not null,
  raw_payload jsonb not null
);

The raw_payload column matters. Delivery platforms change labels, reorder fee rows, and sometimes collapse fees into vague names like other fees. If you only store parsed fields, parser bugs become permanent data loss.

Normalize baskets before comparing platforms

Competitive pricing data only works when the basket stays stable. Comparing a burger meal on one platform with a different combo on another platform creates noise that looks like pricing strategy.

One practical approach is to define canonical baskets, then map platform-specific item IDs into them.

from dataclasses import dataclass

@dataclass(frozen=True)
class BasketItem:
    canonical_sku: str
    quantity: int

@dataclass(frozen=True)
class Basket:
    basket_id: str
    items: tuple[BasketItem, ...]

LUNCH_BASKET = Basket(
    basket_id="burger_lunch_v1",
    items=(
        BasketItem("cheeseburger", 1),
        BasketItem("fries_regular", 1),
        BasketItem("coke_500ml", 1),
    ),
)

PLATFORM_SKU_MAP = {
    "ubereats": {
        "cheeseburger": "item_82716",
        "fries_regular": "item_19302",
        "coke_500ml": "item_77291",
    },
    "doordash": {
        "cheeseburger": "sku_cb_001",
        "fries_regular": "sku_fr_003",
        "coke_500ml": "sku_dr_014",
    },
}

def build_platform_cart(platform: str, basket: Basket) -> list[dict]:
    sku_map = PLATFORM_SKU_MAP[platform]
    return [
        {"platform_item_id": sku_map[item.canonical_sku], "quantity": item.quantity}
        for item in basket.items
    ]

This mapping looks boring, but it is where a lot of pricing analysis succeeds or fails. You need versioned baskets because menus change. If a restaurant replaces coke_500ml with coke_330ml, you should create burger_lunch_v2, not silently mutate historical comparisons.

Geography needs to be below city level

City-level averages hide the interesting part. Delivery fees and promotions often vary by postal code, neighborhood, traffic corridor, or even address. A platform can look competitive across New York while being consistently expensive in a high-value set of Manhattan postal codes.

For data collection, treat location as part of the primary key. These two observations are not duplicates:

platform=ubereats restaurant_id=abc basket=burger_lunch_v1 postal_code=10001 captured_at=2026-08-18T18:00Z
platform=ubereats restaurant_id=abc basket=burger_lunch_v1 postal_code=10011 captured_at=2026-08-18T18:00Z

They may return different delivery fees, ETAs, availability, and promotions.

This is also where scraping jobs become fragile. You need stable address fixtures, session handling, and checks that the platform actually priced the quote for the intended location. Tools like Wire sit in this part of the stack: geo-aware extraction, promotion capture, and failure handling for delivery platforms whose pages change often.

Promotions expire faster than weekly reporting

If you collect data once per week, you are not monitoring promotions. You are sampling history.

Flash discounts and cuisine-specific offers can appear for a few hours during peak demand. If a competitor runs 30% off sushi from 6pm to 9pm on Friday, a Monday morning scrape will miss the entire event.

For promotion monitoring, store every observed offer separately from the quote:

create table delivery_promotions (
  promo_observation_id text primary key,
  platform text not null,
  restaurant_id text,
  postal_code text not null,
  captured_at timestamptz not null,
  promo_type text not null,
  description text not null,
  discount_percent numeric,
  discount_cents integer,
  minimum_order_cents integer,
  funded_by text,
  expires_at timestamptz,
  raw_payload jsonb not null
);

Then query by demand window, not calendar week:

select
  platform,
  postal_code,
  count(*) as promo_count,
  avg(discount_percent) as avg_discount_percent
from delivery_promotions
where captured_at >= '2026-08-14 17:00:00+00'
  and captured_at <  '2026-08-14 22:00:00+00'
  and promo_type = 'percentage_discount'
group by platform, postal_code;

The failure mode here is quiet. Nothing crashes. Your dashboard just says there were no promotions because your crawler slept through them.

Segment subscriber and non-subscriber prices

Subscriptions change the meaning of price. A non-subscriber may see a $3.99 delivery fee while a subscriber sees zero. If you average those together, you get a number nobody actually paid.

Collect both views when possible:

Logged out or non-subscriber session
Subscriber session
Same basket
Same restaurant
Same address
Same time window

Store subscriber as a dimension, not as metadata. Analysts will need to filter on it constantly.

This also changes promotion analysis. Free delivery promotions barely matter to users who already get free delivery, but percentage discounts still do. If your model does not segment subscriptions, you can overestimate or underestimate the effect of a competitor promotion.

Build checks for bad quotes

Delivery pricing data has plenty of bad states. Some are obvious, like HTTP 403s. Others look valid unless you check them.

Examples:

Restaurant is closed, but the page still returns menu prices
Address geocodes to the wrong city
Basket item is unavailable and the cart silently drops it
Promotion requires a minimum order your basket does not meet
Currency changes because the platform redirects to another market
Subscriber session expires and starts returning non-subscriber fees

Add validation before writing analytics tables:

def validate_quote(q: dict) -> list[str]:
    errors = []

    if not q["available"]:
        errors.append("restaurant_unavailable")

    if q["currency"] != q["expected_currency"]:
        errors.append("currency_mismatch")

    if q["requested_postal_code"] != q["priced_postal_code"]:
        errors.append("location_mismatch")

    if q["cart_item_count"] != q["expected_item_count"]:
        errors.append("basket_incomplete")

    component_sum = sum(q.get(k, 0) or 0 for k in [
        "item_subtotal_cents",
        "service_fee_cents",
        "delivery_fee_cents",
        "small_order_fee_cents",
        "tax_cents",
    ]) - (q.get("promo_discount_cents", 0) or 0)

    if abs(component_sum - q["total_cents"]) > 2:
        errors.append("total_does_not_match_components")

    return errors

Keep invalid observations, but mark them. They are useful for debugging collection quality, not for pricing decisions. Wire is relevant here because reliable delivery extraction is less about fetching pages and more about detecting these bad quote states before they pollute the dataset.

The useful benchmark is explainable

A checkout total tells you who is cheaper. A layered quote tells you why.

That difference matters when someone asks what to do next. Lowering delivery fees, funding a restaurant promo, changing subscription benefits, and adjusting commission terms are different actions. They should not come from the same flattened metric.

If you are building this internally, start with one market, three postal codes, two baskets, and two collection windows per day. Store raw payloads, parsed fee components, subscriber state, and validation errors from the beginning. It is much easier to widen a clean dataset than to explain six months of totals that cannot be decomposed.