Turning Ride-Hailing Scrapes Into Useful Pricing Signals

Ride-hailing pricing changes quickly enough that a scrape from ten minutes ago can already be misleading. A competitor may surge in one neighborhood, show no cars in another, and offer a coupon only to a new account. If you treat those values as clean facts, your pricing or forecasting model will learn the wrong thing.

The useful pattern is to treat competitor data as timestamped observations with context, uncertainty, and failure states. That sounds obvious, but many scraping pipelines lose this information early.

Scraped fares are observations, not ground truth

A ride quote is not just price = 18.42.

It depends on at least:

• pickup and dropoff coordinates
• request time
• service type, such as economy, comfort, XL, electric, accessibility
• account state, such as new user, frequent rider, corporate account
• app version, locale, currency, and payment method
• whether the quote included surge, fees, taxes, discounts, or credits

Two scrapes can disagree and both can be correct if they used different accounts or slightly different coordinates.

A useful raw schema keeps those dimensions intact:

CREATE TABLE competitor_quote_observations (
  id BIGSERIAL PRIMARY KEY,
  competitor TEXT NOT NULL,
  observed_at TIMESTAMPTZ NOT NULL,
  pickup_lat DOUBLE PRECISION NOT NULL,
  pickup_lng DOUBLE PRECISION NOT NULL,
  dropoff_lat DOUBLE PRECISION NOT NULL,
  dropoff_lng DOUBLE PRECISION NOT NULL,
  service_type TEXT NOT NULL,
  persona TEXT NOT NULL,
  currency TEXT,
  total_fare_cents INTEGER,
  base_fare_cents INTEGER,
  distance_fare_cents INTEGER,
  time_fare_cents INTEGER,
  surge_multiplier NUMERIC,
  promo_cents INTEGER,
  eta_seconds INTEGER,
  raw_response JSONB NOT NULL,
  status TEXT NOT NULL,
  error_code TEXT
);

The status column matters. A failed scrape is still data. If one platform returns quotes in most areas but times out or hides availability in a specific zone, that tells you something different from “fare is null.”

For extraction teams that do not want to build every app and web collection path themselves, Wire fits this problem when the output needs to preserve quote context like location, persona, timestamp, ETA, and failure state rather than just a flattened price.

Normalize before feeding models

A pricing model usually needs a smaller table than the raw scrape table. You might aggregate by geohash, hour, competitor, and service type.

The important part is not to average stale and fresh observations together as if they have equal value. A quote observed 90 seconds ago should usually count more than one observed 25 minutes ago.

Here is a simple Python example that applies a freshness weight before producing features:

from datetime import datetime, timezone
import math

HALF_LIFE_SECONDS = 300  # after 5 minutes, weight is 0.5

def freshness_weight(observed_at: datetime, now: datetime) -> float:
    age = max(0, (now - observed_at).total_seconds())
    return 0.5 ** (age / HALF_LIFE_SECONDS)


def weighted_average(rows, field):
    numerator = 0.0
    denominator = 0.0

    now = datetime.now(timezone.utc)

    for row in rows:
        if row["status"] != "ok":
            continue
        value = row.get(field)
        if value is None:
            continue

        w = freshness_weight(row["observed_at"], now)
        numerator += value * w
        denominator += w

    return numerator / denominator if denominator else None


quotes = [
    {
        "competitor": "example_ride",
        "observed_at": datetime(2026, 8, 18, 10, 0, 0, tzinfo=timezone.utc),
        "status": "ok",
        "total_fare_cents": 2140,
        "eta_seconds": 360,
        "surge_multiplier": 1.4,
    },
    {
        "competitor": "example_ride",
        "observed_at": datetime(2026, 8, 18, 9, 42, 0, tzinfo=timezone.utc),
        "status": "ok",
        "total_fare_cents": 1750,
        "eta_seconds": 780,
        "surge_multiplier": 1.0,
    },
]

print(weighted_average(quotes, "total_fare_cents"))
print(weighted_average(quotes, "eta_seconds"))

This is not a production pricing model. It is the kind of guardrail you want before modeling starts. Without it, old low-fare quotes can dampen a current surge signal and make your system react late.

The useful features are usually boring

For ride-hailing, the most useful competitor features tend to be straightforward:

• median competitor fare by area, route bucket, and hour
• competitor ETA by area and service type
• observed surge multiplier
• quote success rate
• promo availability by persona
• fare component split, if visible
• variance across accounts for the same route

The last one catches persona-based pricing. If a new account sees a lower fare than an existing account for the same trip, your model should not collapse both into one number without recording the persona.

A simple demand model might use your own demand plus competitor signals:

predicted_demand(area, hour) =
  baseline
  + beta_1 * own_completed_rides(area, hour - 1)
  + beta_2 * competitor_median_eta(area, hour)
  + beta_3 * competitor_surge_multiplier(area, hour)
  + beta_4 * competitor_quote_failure_rate(area, hour)

The sign of each coefficient depends on your market. A high competitor ETA can mean unmet demand that you can capture. It can also mean bad weather or an event that affects everyone, including your own drivers.

How scraping fails in this domain

The common failures are not subtle.

You get HTTP 429 responses or app-level throttling when you sample too aggressively. You get stale cached quotes that look valid but repeat the same fare and ETA for too long. You get partial responses where fare exists but ETA is missing. You get account contamination when a test account receives a promo and then stops representing a normal user.

You should log these as first-class states:

{
  "status": "failed",
  "error_code": "rate_limited",
  "competitor": "example_ride",
  "observed_at": "2026-08-18T10:04:12Z",
  "pickup_geohash": "dr5ru",
  "service_type": "economy"
}

Do not hide these rows from analysts. A spike in rate_limited may mean your scraper broke, but a spike in no_drivers_available in one area can be a useful competitive signal.

This is also where vendor output quality matters. Wire is relevant here if you need ride-hailing extraction that distinguishes “no quote available,” “blocked,” “timed out,” and “valid zero promo” instead of returning a blank field for all of them.

Use competitor data carefully

Competitor scraping should not directly set prices without constraints. Feed it into a pricing system as one group of features, then cap changes and monitor outcomes.

A safer control loop looks like this:

1. Collect quotes and ETAs for fixed route samples.
2. Validate freshness, persona, and failure rates.
3. Aggregate features by area and time window.
4. Use those features in pricing, supply allocation, or promotion models.
5. Compare outcomes against holdout areas before broad rollout.

The practical next step is to define one route grid, one service type, and one persona, then build the raw observation table before touching any pricing logic. If the data model cannot explain where a fare came from, the model built on top of it will not be trustworthy.