Competitive Intelligence Is a Freshness Problem, Not a Dashboard Problem

If your competitor pricing table refreshes every Monday, but the market changes on Tuesday morning, your repricing logic is optimizing against history. The problem usually is not that the dashboard looks bad. The problem is that the data pipeline does not tell you what changed, what failed, and what is too old to trust.

Freshness needs to be part of the data model

A lot of competitive intelligence systems start as reports. Someone scrapes prices, dumps them into a table, and builds a dashboard over the latest row per SKU.

That works until you need to make operational decisions from the data.

For example:

• A grocery app needs to know where a competitor is out of stock before routing inventory.
• A marketplace needs to adjust promotions before traffic peaks.
• A ride-hailing app needs competitor fares while the user is still in the session.

At that point, latest known price is not enough. You need to know when you observed it, when your system fetched it, whether the fetch succeeded, and whether the data is still inside an acceptable SLA.

A useful schema stores observations, not just current state:

CREATE TABLE competitor_offer_observations (
  competitor text NOT NULL,
  sku text NOT NULL,
  region text NOT NULL,
  observed_at timestamptz NOT NULL,
  fetched_at timestamptz NOT NULL DEFAULT now(),
  status_code int,
  price_cents int,
  currency text,
  in_stock boolean,
  source_url text,
  error_code text,
  raw_hash text,
  PRIMARY KEY (competitor, sku, region, observed_at)
);

This lets you distinguish between three very different states:

• The competitor lowered the price.
• Your crawler failed and you are looking at old data.
• The page rendered, but the selector broke and price parsed as null.

Those states should not drive the same business action.

Missing data is a state, not an absence

The dangerous failure mode in competitive data is silent incompleteness. A 403 response, a timeout, a changed DOM node, or a region-specific redirect can all look like no competitor price available if you collapse failures into nulls.

That creates false confidence. Your dashboard still loads. Your model still runs. It just makes decisions from partial market coverage.

A simple freshness query catches a lot of this:

WITH latest AS (
  SELECT DISTINCT ON (competitor, sku, region)
    competitor,
    sku,
    region,
    fetched_at,
    status_code,
    price_cents,
    in_stock,
    error_code
  FROM competitor_offer_observations
  ORDER BY competitor, sku, region, observed_at DESC
)
SELECT
  competitor,
  sku,
  region,
  price_cents,
  in_stock,
  CASE
    WHEN status_code IS DISTINCT FROM 200 THEN 'failed_fetch'
    WHEN price_cents IS NULL AND error_code IS NOT NULL THEN 'parse_failed'
    WHEN now() - fetched_at > interval '15 minutes' THEN 'stale'
    ELSE 'fresh'
  END AS data_state
FROM latest;

The important part is not the exact interval. It is making freshness explicit enough that downstream systems can refuse bad input.

For teams that need competitor prices and stockouts as an operational feed rather than a scraping project, Wire focuses on reliable extraction, normalization, and visible failure handling for this kind of market data.

Delivery format changes who can act on the data

A dashboard is fine for weekly review. It is a poor interface for automated pricing, inventory routing, or churn prediction.

If engineers need the data, expose it like a product API:

GET /competitive-offers?sku=ABC-123&region=brooklyn&fresh_within_seconds=900

A useful response should include state and timestamps, not just values:

{
  "sku": "ABC-123",
  "region": "brooklyn",
  "competitor": "example_market",
  "price_cents": 1299,
  "in_stock": true,
  "observed_at": "2026-08-18T10:14:03Z",
  "fetched_at": "2026-08-18T10:14:08Z",
  "data_state": "fresh"
}

That shape lets a pricing service reject stale rows, lets analysts measure coverage, and lets product teams debug why a rule fired.

For non-technical users, natural language querying can help, but only if it sits on top of the same governed dataset. If a sales lead asks where competitors are out of stock, the answer should come from the same freshness-aware observations that feed the API. Otherwise you end up with two competing versions of reality.

Build versus buy is mostly about maintenance

It is tempting to build this internally. The first version often looks manageable: a scraper, a cron job, a warehouse table, and a dashboard.

The maintenance cost shows up later:

• Target sites change markup and break parsers.
• Dynamic rendering returns empty HTML unless you execute client-side code.
• Region and device behavior create different prices for the same SKU.
• Rate limits and bot defenses turn a working job into intermittent 403s.
• Product matching becomes messy when titles, bundles, and pack sizes differ.

If your team builds the pipeline, track coverage as a first-class metric. Do not just count successful jobs. Count expected observations versus usable observations per competitor, SKU, and region.

If you buy the feed, ask whether Wire or any other provider can return raw timestamps, failure states, and coverage metrics rather than only cleaned final prices.

That question matters because cleaned data without failure context can hide the same problem as a brittle internal scraper.

A practical starting point

Pick one decision that depends on competitor data, such as repricing a category or reacting to regional stockouts. Define the maximum acceptable data age for that decision. Then instrument your pipeline so every row can be classified as fresh, stale, failed, or parse_failed.

Once you can see those states, you will know whether you have a dashboard problem, a collection problem, or a decision system consuming data it should not trust.