Travel Fare Monitoring Architecture: The Geography and Timing Dimensions of Price Data

Last spring I spent two weeks chasing a bug that wasn't a bug. A fare-monitoring job I'd built kept reporting that a route had "dropped 18% overnight" — then popped right back up. I re-checked selectors, verified timestamps, dumped raw HTML. Everything was correct. What I'd actually built was a measurement instrument pointing at a different part of the market every few hours, mistaking its own inconsistency for market movement.

Here's the thing about travel prices that took me embarrassingly long to internalize: the price of a flight or hotel room isn't a property of the itinerary. It's a property of the itinerary plus where you observe from plus when you observe. Change the exit IP's country, and the point-of-sale changes, and the currency changes, and often the fare bucket itself changes. Poll once a day at a fixed time, and you're sampling a market that moves intraday and behaves differently on weekends.

The thesis of this post: travel price data is two-dimensional — the WHERE dimension (observation geography) and the WHEN dimension (observation timing). An architecture that treats both as first-class design dimensions, not implementation accidents, produces data you can actually make decisions on. Everything else follows from that.

The WHERE Dimension: POS Country, Currency, and Session Pinning

Most travel pricing engines are point-of-sale aware. The price you're quoted depends on what market the request appears to come from — driven by IP geolocation, currency parameters, and sometimes locale headers. The same seat on the same plane can quote differently from a US IP, a UK IP, and a German IP, not just because of currency denomination but because fare inventory is allocated per market and per selling channel.

The first architectural consequence: a market isn't a website, it's a (site, country) pair. If you monitor "the route" from whatever random exit IP your proxy pool gives you, you're blending three or four different price series into one noisy soup. Every apparent price spike is unfalsifiable — was it the market, or the exit IP?

The fix is to pin a session per market: one sticky proxy session per (site, country) pair, held stable across the life of the monitoring job, so consecutive observations are actually comparable. The shape that's worked for me:

import time
import random
import requests
from dataclasses import dataclass

@dataclass
class Market:
    site: str                  # e.g. "site-a" booking site handle
    country: str               # ISO country code, e.g. "us"
    city: str | None           # optional city-level pin, e.g. "newyork"
    currency: str              # expected quote currency, e.g. "USD"
    pos_label: str             # canonical market key

class MarketSession:
    """One session pinned to one market, held stable so
    consecutive observations stay comparable."""

    def __init__(self, market: Market, proxy_user: str, proxy_pass: str):
        self.market = market
        geo = f"-cc-{market.country}" + (f"-city-{market.city}" if market.city else "")
        proxy = f"http://{proxy_user}{geo}:{proxy_pass}@p.thordata.com:9000"
        self.session = requests.Session()
        self.session.proxies = {"http": proxy, "https": proxy}
        # Locale must match the IP
        self.session.headers.update({
            "Accept-Language": {"us": "en-US", "gb": "en-GB", "de": "de-DE"}.get(
                market.country, "en"),
        })
        self._last_hit = 0.0

    def get(self, url: str) -> requests.Response:
        elapsed = time.time() - self._last_hit
        if elapsed < 5:
            time.sleep(5 - elapsed + random.uniform(0.5, 2.0))
        self._last_hit = time.time()
        r = self.session.get(url, timeout=30, headers={"X-Currency": self.market.currency})
        r.raise_for_status()
        return r

    def healthcheck(self) -> str | None:
        """Verify the exit IP matches the market — pin, then prove it."""
        r = self.session.get("https://ipinfo.io/json", timeout=15)
        info = r.json()
        if info.get("country", "").lower() != self.market.country:
            return f"drift: exit={info.get('country')} expected={self.market.country}"
        return None

Two details in there are doing more work than they look like. The healthcheck matters because a "pinned" session is a claim, and claims drift — proxy sessions rotate, providers reassign exits, and a silent drift from a US exit to a Canadian one poisons your US price series with Canadian quotes. I run the check at session startup and every N requests after; drift kills the session and rotates a new pinned one.

The locale headers matter because pricing backends cross-check signals. A German IP with Accept-Language: en-US and a USD currency parameter can land you in an ambiguous fallback market — quotes that match neither the US nor the DE series. Geo pinning only works if the whole request is coherent.

Currency normalization

Once you observe from multiple markets, you're collecting quotes in EUR, GBP, USD, and friends. Comparing them raw is meaningless; converting them badly is worse. The rules I've settled on:

FX_CACHE: dict[str, float] = {"USD": 1.0}   # refresh daily from an FX API

def to_usd(amount: float, currency: str, observed_at: str) -> float:
    rate = FX_CACHE.get(currency)
    if rate is None:
        raise ValueError(f"no FX rate for {currency} at {observed_at}")
    return round(amount * rate, 2)

def normalize(observation: dict) -> dict:
    return {
        "market": observation["market"],           # pos_label, preserved!
        "price_local": observation["price"],
        "currency": observation["currency"],
        "price_usd": to_usd(observation["price"], observation["currency"],
                            observation["observed_at"]),
        "taxes_fees_included": observation.get("taxes_fees_included", None),
        "observed_at": observation["observed_at"],
    }

Keep price_local alongside price_usd — an "increase" entirely explained by currency movement is not a market signal, and you can only distinguish that if you stored the local price and the rate used. Also preserve the market label on every row forever. The most useful analysis I've run on travel data is per-market trend comparison: the same itinerary observed from three POS countries diverging for a week is a genuinely interesting event, and you can't see it if you collapsed the market dimension at ingest.

The WHEN Dimension: Fares Are a Time Series, Not a Snapshot

Travel pricing is dynamic — continuously repriced against demand forecasts, competitor moves, and booking curves. Two failure modes follow.

Failure mode one: point-in-time snapshots. A single daily fetch tells you what the fare was at 09:14 UTC — one sample of a series that can swing meaningfully within a day. Any price alert built on single samples will alternate between panic and relief.

Failure mode two: uniform polling. Fares don't move uniformly across the week. Business-heavy routes track weekday business demand; leisure routes behave differently on weekends. Sampling at one fixed time systematically aliases whatever pattern lives at that time.

The scheduler below treats timing as a design variable — sampling more during known-volatile windows, and sampling differently on weekends:

import datetime as dt
import random
from collections import defaultdict

def next_poll_times(route: str, day: dt.date, samples_per_day: int = 4) -> list[dt.datetime]:
    """A day's sampling schedule with time-of-day and weekday structure."""
    is_weekend = day.weekday() >= 5

    # Heavier sampling where fares move more.
    windows = [
        (dt.time(6, 0),  dt.time(9, 30)),    # overnight repricing settles
        (dt.time(11, 0), dt.time(14, 0)),    # midday competitive adjustments
        (dt.time(17, 0), dt.time(20, 0)),    # evening demand-driven moves
    ]
    if is_weekend:
        # Leisure browsing peaks later on weekends; shift one window
        windows[1] = (dt.time(13, 0), dt.time(16, 0))

    per_window = max(1, samples_per_day // len(windows))
    times = []
    for start, end in windows:
        span = (end.hour * 60 + end.minute) - (start.hour * 60 + start.minute)
        for _ in range(per_window):
            offset = random.randint(0, span)
            t = (dt.datetime.combine(day, start)
                 + dt.timedelta(minutes=offset))
            times.append(t + dt.timedelta(seconds=random.randint(0, 300)))
    return sorted(times)

class PollingPlanner:
    def __init__(self, jitter_pct: float = 0.25):
        self.jitter_pct = jitter_pct
        self.last_run: dict[str, dt.datetime] = defaultdict(lambda: dt.datetime.min)

    def due_jobs(self, routes: list[str], now: dt.datetime) -> list[str]:
        due = []
        for route in routes:
            schedule = next_poll_times(route, now.date())
            previous = [t for t in schedule if t <= now]
            if not previous:
                continue
            scheduled = previous[-1]
            if self.last_run[route] < scheduled:
                due.append(route)
                self.last_run[route] = now
        return due

Every sample carries its timestamp; nothing downstream ever sees a price without its WHEN.

Why Averaging Beats Snapshots

With geography pinned and timing structured, you have something rare in scraping: a comparable time series. And once observations are comparable, the right unit of analysis stops being "the latest price" and becomes "the recent distribution of prices." This is the point where a fare monitor stops being a scraper and becomes instrumentation.

from statistics import median, pstdev

def fare_signal(samples: list[dict], window_hours: int = 48) -> dict | None:
    """Collapse recent samples from ONE market into a stable signal.
    Each sample has 'price_usd' and 'observed_at'."""
    if not samples:
        return None
    prices = [s["price_usd"] for s in samples if s["price_usd"] is not None]
    if len(prices) < 3:
        return None        # not enough evidence yet
    med = median(prices)
    return {
        "median_usd": round(med, 2),
        "spread_pct": round(pstdev(prices) / med * 100, 1) if med else None,
        "n": len(prices),
    }

Median-over-recent-samples absorbs the intraday flapping that made my client's alerts useless. The spread_pct field is a bonus: when the spread suddenly widens, the market itself is volatile — often more actionable than the level. And the n floor matters: three samples is my minimum before claiming anything about a fare. A confident claim from one observation is exactly the bug I opened with.

Deduplicating Itineraries Across Channels

One more dimension that bites everyone: the same flight sells through multiple booking channels — the airline's own site, aggregators, OTAs — and often at different prices. If you monitor five channels, you have five price series for one sellable product. You need a canonical itinerary identity to join them.

import hashlib

def itinerary_id(origin: str, destination: str,
                 depart_dt: dt.datetime, carrier_code: str,
                 flight_number: str, cabin: str) -> str:
    raw = "|".join([
        origin.upper(), destination.upper(),
        depart_dt.strftime("%Y-%m-%dT%H:%M"),
        carrier_code.upper(), flight_number.upper(), cabin.upper(),
    ])
    return hashlib.sha1(raw.encode()).hexdigest()[:16]

# OBS-A1F3 == OBS-A1F3 regardless of channel.
# Channel and market are observation attributes.

The key discipline: identity comes from the physical product (route, schedule, flight, cabin), never from the channel. Then "best available price for this itinerary" becomes a query, not a data-modeling crisis — and a price gap between channels becomes a reportable signal in its own right.

Putting It Together

Layer	Decision	Why it matters
Market definition	(site, country) pairs, not sites	POS country changes the price series
Sessions	One pinned proxy session per market, healthchecked	Comparability across time
Currency	Store local price + rate + USD, per row	Separate FX noise from market moves
Timing	Time-of-day windows, weekend-aware schedules	Sample the market, don't alias it
Signal	Median over recent samples, minimum n	Decisions on distributions, not points
Identity	Physical itinerary hash, channel as attribute	Cross-channel comparison without chaos

None of these layers is individually hard. The discipline is refusing to collapse WHERE and WHEN into "just a scrape." A fare monitor that observes from a consistent place, on a schedule that respects how fares actually move, and reports distributions instead of snapshots, produces data you can make decisions on. I learned that one phantom 18% fare drop at a time.

Disclosure: I use Thordata's residential proxies for the country- and city-level geo-pinning that travel fare monitoring depends on. If you want to try them, they're at thordata.com, and the code **thor020* gets you 10% off.*